We prove that the convergence of the real and imaginary parts of the logarithm of the characteristic polynomial of unitary Brownian motion toward Gaussian free fields on the cylinder, as the matrix dimension goes to infinity, holds in certain suitable Sobolev spaces, which we believe to be optimal. This is the natural dynamical analogue of the result for a fixed time by Hughes, Keating and O'Connell [1]. A weak kind of convergence is known since the work of Spohn [2], which was widely improved recently by Bourgade and Falconet [3]. In the course of this research we also proved a Wick-type identity, which we include in this paper, as it might be of independent interest.