We establish existence of Stein kernels for probability measures on $\mathbb{R}^d$ satisfying a Poincaré inequality, and obtain bounds on the Stein discrepancy of such measures. Applications to quantitative central limit theorems are discussed, including a new CLT in Wasserstein distance $W_2$ with optimal rate and dependence on the dimension. As a byproduct, we obtain a stability version of an estimate of the Poincaré constant of probability measures under a second moment constraint. The results extend more generally to the setting of converse weighted Poincaré inequalities. The proof is based on simple arguments of calculus of variations. Further, we establish two general properties enjoyed by the Stein discrepancy, holding whenever a Stein kernel exists: Stein discrepancy is strictly decreasing along the CLT, and it controls the skewness of a random vector.