Interactions in time-varying complex systems are often very heterogeneous at the topological level (who interacts with whom) and at the temporal level (when interactions occur and how often). While it is known that temporal heterogeneities often have strong effects on dynamical processes, e.g. the burstiness of contact sequences is associated with slower spreading dynamics, the picture is far from complete. In this paper, we show that temporal heterogeneities result in temporal sparsity} at the time scale of average inter-event times, and that temporal sparsity determines the amount of slowdown of Susceptible-Infectious (SI) spreading dynamics on temporal networks. This result is based on the analysis of several empirical temporal network data sets. An approximate solution for a simple network model confirms the association between temporal sparsity and slowdown of SI spreading dynamics. Since deterministic SI spreading always follows the fastest temporal paths, our results generalize -- paths are slower to traverse because of temporal sparsity, and therefore all dynamical processes are slower as well.