This article gives an explicit formula for the leading term of the free energy of three-dimensional $U(N)$ lattice gauge theory for any $N$, as the lattice spacing tends to zero. The proof is based on a novel technique that avoids phase cell renormalization. The technique also yields a similar formula for the four-dimensional theory, but only in the weak coupling~limit.