In this note we characterize when non-classical polynomials are necessary in the inverse theorem for the Gowers $U^k$-norm. We give a brief deduction of the fact that a bounded function on $\mathbb F_p^n$ with large $U^k$-norm must correlate with a classical polynomial when $k\leq p+1$. To the best of our knowledge, this result is new for $k=p+1$ (when $p>2$). We then prove that non-classical polynomials are necessary in the inverse theorem for the Gowers $U^k$-norm over $\mathbb F_p^n$ for all $k\geq p+2$, completely characterizing when classical polynomials suffice.