In this paper, we make a novel connection between Stein's method and noncommutative algebra by viewing polynomial Stein operators (Stein operators with polynomial coefficients) as elements of the first Weyl algebra. Through this connection we study the algebraic structure of classes of polynomial Stein operators. In the case of the standard Gaussian distribution, we provide a complete description of the corresponding class of polynomial Stein operators by (i) identifying it as a vector space over $\mathbb{R}$ with an explicit given basis and (ii) by showing that this class is a principal right ideal of the first Weyl algebra generated by the classical Gaussian Stein operator $\partial -x$, with $\partial$ denoting the usual differential operator. We also study the characterising property of polynomial Stein operators for the standard Gaussian distribution, and give examples of general classes of polynomial Stein operators that are characterising, as well as classes that are not characterising unless additional distributional assumptions are made. By appealing to a standard property of Weyl algebras, we shown that the non-characterising property possessed by a wide class of polynomial Stein operators for the standard Gaussian distribution is a consequence of a general result that is perhaps surprising from a probabilistic perspective: the intersection between the class of polynomial Stein operators for any two target distributions with holonomic densities or holonomic characteristic functions is non-trivial.