We make quantitative improvements to recently obtained results on the structure of the image of a large difference set under certain quadratic forms and other homogeneous polynomials. Previous proofs used deep results of Benoist-Quint on random walks in certain subgroups of $\operatorname{SL}_r(\mathbb{Z})$ (the symmetry groups of these quadratic forms) that were not of a quantitative nature. Our new observation relies on noticing that rather than studying random walks, one can obtain more quantitative results by considering polynomial orbits of these group actions that are not contained in cosets of submodules of $\mathbb{Z}^r$ of small index. Our main new technical tool is a uniform Furstenberg-Sárközy theorem that holds for a large class of polynomials not necessarily vanishing at zero, which may be of independent interest and is derived from a density increment argument and Hua's bound on polynomial exponential sums.