We study the potential functions that determine the optimal density for $\varepsilon$-entropically regularized optimal transport, the so-called Schrödinger potentials, and their convergence to the counterparts in classical optimal transport, the Kantorovich potentials. In the limit $\varepsilon\to0$ of vanishing regularization, strong compactness holds in $L^{1}$ and cluster points are Kantorovich potentials. In particular, the Schrödinger potentials converge in $L^{1}$ to the Kantorovich potentials as soon as the latter are unique. These results are proved for all continuous, integrable cost functions on Polish spaces. In the language of Schrödinger bridges, the limit corresponds to the small-noise regime.