We prove that in the full $L^2$-regime the partition function of the directed polymer model in dimensions $d\geq 3$, if centered, scaled and averaged with respect to a test function $\varphi \in C_c(\mathbb{R}^d)$, converges in distribution to a Gaussian random variable with explicit variance. Introducing a new idea in this context of a martingale difference representation, we also prove that the log-partition function, which can be viewed as a discretisation of the KPZ equation, exhibits the same fluctuations, when centered and averaged with respect to a test function. Thus, the two models fall within the Edwards-Wilkinson universality class in the full $L^2$-regime, a result that was only established, so far, for a strict subset of this regime in $d\geq 3$.