We analyze the rate of convergence of the local statistics of Dyson Brownian motion to the GOE/GUE for short times $t=o(1)$ with deterministic initial data V . Our main result states that if the density of states of $V$ is bounded both above and away from $0$ down to scales $\ell \ll t$ in a small interval of size $G \gg t$ around an energy $E_0$, then the local statistics coincide with the GOE/GUE near the energy $E_0$ after time $t$. Our methods are partly based on the idea of coupling two Dyson Brownian motions from [6], the parabolic regularity result of [15], and the eigenvalue rigidity results of [21].