We prove an explicit inverse Chevalley formula in the equivariant $K$-theory of semi-infinite flag manifolds of simply-laced type. By an inverse Chevalley formula, we mean a formula for the product of an equivariant scalar with a Schubert class, expressed as a $\mathbb{Z}[q^{\pm 1}]$-linear combination of Schubert classes twisted by equivariant line bundles. Our formula applies to arbitrary Schubert classes in semi-infinite flag manifolds of simply-laced type and equivariant scalars $e^λ$, where $λ$ is an arbitrary minuscule weight. By a result of Stembridge, our formula completely determines the inverse Chevalley formula for arbitrary weights in simply-laced type, except for type $E_8$. The combinatorics of our formula is governed by the quantum Bruhat graph, and the proof is based on a limit from the double affine Hecke algebra. As such, our formula also provides an explicit determination of all nonsymmetric $q$-Toda operators for minuscule weights in ADE type.