We examine the limiting behavior of multiple ergodic averages associated with arithmetic progressions whose differences are elements of a fixed integer sequence. For each $\ell$, we give necessary and sufficient conditions under which averages of length $\ell$ of the aforementioned form have the same limit as averages of $\ell$-term arithmetic progressions. As a corollary, we derive a sufficient condition for the presence of arithmetic progressions with length $\ell+1$ and restricted differences in dense subsets of integers. These results are a consequence of the following general theorem: in order to verify that a multiple ergodic average is controlled by the degree $d$ Gowers-Host-Kra seminorm, it suffices to show that it is controlled by some Gowers-Host-Kra seminorm, and that the degree $d$ control follows whenever we have degree $d+1$ control. The proof relies on an elementary inverse theorem for the Gowers-Host-Kra seminorms involving dual functions, combined with novel estimates on averages of seminorms of dual functions. We use these estimates to obtain a higher order variant of the degree lowering argument previously used to cover averages that converge to the product of integrals.