We consider the parabolic Anderson problem with random potentials having inverse-square singularities around the points of a standard Poisson point process in $\mathbb{R}^d$, $d \geq 3$. The potentials we consider are obtained via superposition of translations over the points of the Poisson point process of a kernel $\mathfrak{K}$ behaving as $\mathfrak{K}(x) \approx θ|x|^{-2}$ near the origin, where $θ\in (0,(d-2)^2/16]$. In order to make sense of the corresponding path integrals, we require the potential to be either attenuated (meaning that $\mathfrak{K}$ is integrable at infinity) or, when $d=3$, renormalized, as introduced by Chen and Kulik in [8]. Our main results include existence and large-time asymptotics of non-negative solutions via Feynman-Kac representation. In particular, we settle for the renormalized potential in $d=3$ the problem with critical parameter $θ= 1/16$, left open by Chen and Rosinski in [arXiv:1103.5717].