The aim of this paper is to justify in dimensions two and three the ansatz of Caracciolo et al. stating that the displacement in the optimal matching problem is essentially given by the solution to the linearized equation i.e. the Poisson equation. Moreover, we prove that at all mesoscopic scales, this displacement is close in suitable negative Sobolev spaces to a curl-free Gaussian free field. For this we combine a quantitative estimate on the difference between the displacement and the linearized object, which is based on the large-scale regularity theory recently developed in collaboration with F. Otto, together with a qualitative convergence result for the linearized problem.