We explore connections between covariance representations, Bismut-type formulas and Stein's method. First, using the theory of closed symmetric forms, we derive covariance representations for several well-known probability measures on $\mathbb{R}^d$, $d \geq 1$. When strong gradient bounds are available, these covariance representations immediately lead to $L^p$-$L^q$ covariance estimates, for all $p \in (1, +\infty)$ and $q = p/(p-1)$. Then, we revisit the well-known $L^p$-Poincaré inequalities ($p \geq 2$) for the standard Gaussian probability measure on $\mathbb{R}^d$ based on a covariance representation. Moreover, for the nondegenerate symmetric $α$-stable case, $α\in (1,2)$, we obtain $L^p$-Poincaré and pseudo-Poincaré inequalities, for $p \in (1, α)$, via a detailed analysis of the various Bismut-type formulas at our disposal. Finally, using the construction of Stein's kernels by closed forms techniques, we obtain quantitative high-dimensional CLTs in $1$-Wasserstein distance when the limiting Gaussian probability measure is anisotropic. The dependence on the parameters is completely explicit and the rates of convergence are sharp.