We establish upper bounds on the size of the largest subset of $\{1,2,\dots,N\}$ lacking nonzero differences of the form $h(p_1,\dots,p_{\ell})$, where $h\in \mathbb{Z}[x_1,\dots,x_{\ell}]$ is a fixed polynomial satisfying appropriate conditions and $p_1,\dots,p_{\ell}$ are prime. The bounds are of the same type as the best-known analogs for unrestricted integer inputs, due to Bloom-Maynard and Arala for $\ell=1$, and to the authors for $\ell \geq 2$.