We study random walks on sub-Riemannian manifolds using the framework of retractions, i.e., approximations of normal geodesics. We show that such walks converge to the correct horizontal Brownian motion if normal geodesics are approximated to at least second order. In particular, we (i) provide conditions for convergence of geodesic random walks defined with respect to normal, compatible, and partial connections and (ii) provide examples of computationally efficient retractions, e.g., for simulating anisotropic Brownian motion on Riemannian manifolds.