We consider the assignment problem between two sets of $N$ random points on a smooth, two-dimensional manifold $Ω$ of unit area. It is known that the average cost scales as $E_Ω(N)\sim\frac{1}{2π}\ln N$ with a correction that is at most of order $\sqrt{\ln N\ln\ln N}$. In this paper, we show that, within the linearization approximation of the field-theoretical formulation of the problem, the first $Ω$-dependent correction is on the constant term, and can be exactly computed from the spectrum of the Laplace--Beltrami operator on $Ω$. We perform the explicit calculation of this constant for various families of surfaces, and compare our predictions with extensive numerics.