We consider the convergence of the empirical spectral measures of random $N \times N$ unitary matrices. We give upper and lower bounds showing that the Kolmogorov distance between the spectral measure and the uniform measure on the unit circle is of the order $\log N / N$, both in expectation and almost surely. This implies in particular that the convergence happens more slowly for Kolmogorov distance than for the $L_1$-Kantorovich distance. The proof relies on the determinantal structure of the eigenvalue process.