We discover that tautological intersection numbers on $\bar{\mathcal{M}}_{g, n}$, the moduli space of stable genus $g$ curves with $n$ marked points, are evaluations of Ehrhart polynomials of partial polytopal complexes. In order to prove this, we realize the Virasoro constraints for tautological intersection numbers as a recursion for integer-valued polynomials. Then we apply a theorem of Breuer that classifies Ehrhart polynomials of partial polytopal complexes by the nonnegativity of their $f^*$-vector. In dimensions 1 and 2, we show that the polytopal complexes that arise are \emph{inside-out polytopes} i.e. polytopes that are dissected by a hyperplane arrangement.