In this paper we show how polynomial walks can be used to establish a twisted recurrence for sets of positive density in $\mathbb{Z}^d$. In particular, we prove that if $Γ\leq \operatorname{GL}_d(\mathbb{Z})$ is finitely generated by unipotents and acts irreducibly on $\mathbb{R}^d$, then for any set $B \subset \mathbb{Z}^d$ of positive density, there exists $k \geq 1$ such that for any $v \in k \mathbb{Z}^d$ one can find $γ\in Γ$ with $γv \in B - B$. Our method does not require the linearity of the action, and we prove a twisted recurrence for semigroups of maps from $\mathbb{Z}^d$ to $\mathbb{Z}^d$ satisfying some irreducibility and polynomial assumptions. As one of the consequences, we prove a non-linear analog of Bogolubov's theorem -- for any set $B \subset \mathbb{Z}^2$ of positive density, and $p(n) \in \mathbb{Z}[n]$, with $p(0) = 0$ and $\operatorname{deg}(p) \geq 2$, there exists $k \geq 1$ such that $k \mathbb{Z} \subset \{ x - p(y) \, | \, (x,y) \in B-B \}$. Unlike the previous works on twisted recurrence that used recent results of Benoist-Quint and Bourgain-Furman-Lindenstrauss-Mozes on equidistribution of random walks on automorphism groups of tori, our method relies on the classical Weyl equidistribution for polynomial orbits on tori.