Consider a degree-$d$ polynomial $f(ξ_1,\dots,ξ_n)$ of independent Rademacher random variables $ξ_1,\dots,ξ_n$. To what extent can $f(ξ_1,\dots,ξ_n)$ concentrate on a single point? This is the so-called polynomial Littlewood-Offord problem. A nearly optimal bound was proved by Meka, Nguyen and Vu: the point probabilities are always at most about $1/\sqrt n$, unless $f$ is "close to the zero polynomial" (having only $o(n^d)$ nonzero coefficients). In this paper we prove several results supporting the general philosophy that the Meka-Nguyen-Vu bound can be significantly improved unless $f$ is "close to a polynomial with special algebraic structure", drawing some comparisons to phenomena in analytic number theory. In particular, one of our results is a corrected version of a conjecture of Costello on multilinear forms (in an appendix with Ashwin Sah and Mehtaab Sawhney, we disprove Costello's original conjecture).