We give two new constructions of the harmonic algebra of a lattice polytope $P$, a bigraded algebra whose character is the $q$-Ehrhart series of $P$ defined by Reiner and Rhoades. First, we show that the harmonic algebra is the associated graded algebra of the semigroup algebra of $P$ with respect to a certain natural filtration, clarifying it's relationship with the more classical semigroup algebra. We then give a geometric interpretation of the harmonic algebra as a quotient of the ring of global sections of a certain family of line bundles on the blowup of the toric variety associated to $P$ at a generic point. Using this connection to toric geometry we resolve one the main conjectures of Reiner and Rhoades by showing that the harmonic algebra is not finitely generated in general.