In this manuscript, we consider the Langevin dynamics on $\mathbb{R}^d$ with an overdamped vector field and driven by multiplicative Brownian noise of small amplitude $\sqrtε$, $ε>0$. Under suitable assumptions on the vector field and the diffusion coefficient, it is well-known that it possesses a unique invariant probability measure $μ^ε$. As $ε$ tends to zero, we prove that the probability measure $ε^{d/2} μ^ε(\sqrtε\mathrm{d} x)$ converges in the $p$-Wasserstein distance for $p\in [1,2]$ to a Gaussian measure with zero-mean vector and non-degenerate covariance matrix which solves a Lyapunov matrix equation. Moreover, the error term is estimated. We emphasize that generically no explicit formula for $μ^ε$ can be found.