Let $\{U^N_t\}_{t\ge 0}$ be a standard Brownian motion on $\mathbb{U}(N)$. For fixed $N\in\mathbb{N}$ and $t>0$, we give explicit bounds on the $L_1$-Wasserstein distance of the empirical spectral measure of $U^N_t$ to both the ensemble-averaged spectral measure and to the large-$N$ limiting measure identified by Biane. We are then able to use these bounds to control the rate of convergence of paths of the measures on compact time intervals. The proofs use tools developed by the first author to study convergence rates of the classical random matrix ensembles, as well as recent estimates for the convergence of the moments of the ensemble-average spectral distribution.