We study the class of continuous polynomial Volterra processes, which we define as solutions to stochastic Volterra equations driven by a continuous semimartingale with affine drift and quadratic diffusion matrix in the state of the Volterra process. To demonstrate the versatility of possible state spaces within our framework, we construct polynomial Volterra processes on the unit ball. This construction is based on a stochastic invariance principle for stochastic Volterra equations with possibly singular kernels. Similarly to classical polynomial processes, polynomial Volterra processes allow for tractable expressions of the moments in terms of the unique solution to a system of deterministic integral equations, which reduce to a system of ODEs in the classical case. By applying this observation to the moments of the finite-dimensional distributions we derive a uniqueness result for polynomial Volterra processes. Moreover, we prove that the moments are polynomials with respect to the initial condition, another crucial property shared by classical polynomial processes. The corresponding coefficients can be interpreted as a deterministic dual process and solve integral equations dual to those verified by the moments themselves. Additionally, we obtain a representation of the moments in terms of a pure jump process with killing, which corresponds to another non-deterministic dual process.