We derive new formulas for the expectation and variance of Wilson loops for any contractible simple loop on a compact orientable surface of genus $1$ and higher, in the model of two-dimensional Yang--Mills theory with structure group $\mathrm{U}(N)$. They are written in terms of a Gaussian measure on the dual of $\mathrm{U}(N)$ introduced recently by the author and M. Maïda \cite{LM3}. From these formulas, we prove a quantitative result on the convergence of the expectation and variance as $N$ tends to infinity, refining a result of the author and A. Dahlqvist \cite{DL}. We finally derive the large $g$ limit of the Wilson loop expectation and variance, by analogy with the study of integrals on moduli spaces of compact hyperbolic surfaces. Surprisingly, the variance does not vanish in this regime, but there are no nontrivial fluctuations of any order.