We study a natural random walk on the $n \times n$ upper triangular matrices, with entries in $\mathbb{Z}/m \mathbb{Z}$, generated by steps which add or subtract a uniformly random row to the row above. We show that the mixing time of this random walk is $O(m^2n \log n+ n^2 m^{o(1)})$. This answers a question of Stong and of Arias-Castro, Diaconis, and Stanley.