We consider random walks in dynamic random environments given by Markovian dynamics on $\mathbb{Z}^d$. We assume that the environment has a stationary distribution $μ$ and satisfies the Poincaré inequality w.r.t. $μ$. The random walk is a perturbation of another random walk (called "unperturbed"). We assume that also the environment viewed from the unperturbed random walk has stationary distribution $μ$. Both perturbed and unperturbed random walks can depend heavily on the environment and are not assumed to be finite-range. We derive a law of large numbers, an averaged invariance principle for the position of the walker and a series expansion for the asymptotic speed. We also provide a condition for non-degeneracy of the diffusion, and describe in some details equilibrium and convergence properties of the environment seen by the walker. All these results are based on a more general perturbative analysis of operators that we derive in the context of $L^2$-bounded perturbations of Markov processes by means of the so-called Dyson-Phillips expansion.