Given a finite group $G$ and a generating set $S \subseteq G$, the diameter $diam(G,S)$ is the least integer $n$ such that every element of $G$ is the product of at most $n$ elements of $S$. In this paper, for bounded $|S|$, we characterize groups with polynomial diameter as the groups with a large abelian section close to the top, precisely of size an exponential portion of the size of the full group. This complements a key result of Breuillard and Tointon. As a consequence, groups with polynomial diameter have many conjugacy classes, and contain a large nilpotent subgroup of class at most $2$.