Let $\{u(t,x)\}_{t>0,x\in{{\mathbb R}^{d}}}$ denote the solution to the linear (fractional) stochastic heat equation. We establish rates of convergence with respect to the uniform distance between the density of spatial averages of solution and the density of the standard normal distribution in some different scenarios. We first consider the case that $u_0\equiv1$, and the stochastic fractional heat equation is driven by a space-time white noise. When $α=2$ (parabolic Anderson model, PAM for short) and the stochastic heat equation is driven by colored noise in space, we present the rates of convergence respectively in the case that $u_0\equiv1$, $d\geq1$ and $u_0=δ_0$, $d=1$ under an additional condition $\hat f(\mathbb{R}^d)<\infty$. Our results are obtained by using a combination of the Malliavin calculus and Stein's method for normal approximations.