Synchrony is one of the most common dynamical states emerging on networks. The speed of convergence towards synchrony provides a fundamental collective time scale for synchronizing systems. Here we study the asymptotic synchronization times for directed networks with topologies ranging from completely ordered, grid-like, to completely disordered, random, including intermediate, partially disordered topologies. We extend the approach of Master Stability Functions to quantify synchronization times. We find that the synchronization times strongly and systematically depend on the network topology. In particular, at fixed in-degree, stronger topological randomness induces faster synchronization, whereas at fixed path length, synchronization is slowest for intermediate randomness in the small-world regime. Randomly rewiring real-world neural, social and transport networks confirms this picture.