We give a structural description of the finite subsets $A$ of an arbitrary group $G$ which obey the polynomial growth condition $|A^n| \leq n^d |A|$ for some bounded $d$ and sufficiently large $n$, showing that such sets are controlled by (a bounded number of translates of) a coset nilprogression in a certain precise sense. This description recovers some previous results of Breuillard-Green-Tao and Breuillard-Tointon concerning sets of polynomial growth; we are also able to describe the subsequent growth of $|A^m|$ fairly explicitly for $m \geq n$, at least when $A$ is a symmetric neighbourhood of the identity. We also obtain an analogous description of symmetric probability measures $μ$ whose $n$-fold convolutions $μ^{*n}$ obey the condition $\| μ^{*n} \|_{\ell^2}^{-2} \leq n^d \|μ\|_{\ell^2}^{-2}$. In the abelian case, this description recovers the inverse Littlewood-Offord theorem of Nguyen-Vu, and gives a variant of a recent nonabelian inverse Littlewood-Offord theorem of Tiep-Vu. Our main tool to establish these results is the inverse theorem of Breuillard, Green, and the author that describes the structure of approximate groups.