The emergence of SARS-CoV-2 and the coronavirus infectious disease (COVID-19) has become a pandemic. Social (physical) distancing is a key non-pharmacologic control measure to reduce the transmission rate of SARS-COV-2, but high-level adherence is needed. Using daily travel distance and stay-at-home time derived from large-scale anonymous mobile phone location data provided by Descartes Labs and SafeGraph, we quantify the degree to which social distancing mandates have been followed in the U.S. and its effect on growth of COVID-19 cases. The correlation between the COVID-19 growth rate and travel distance decay rate and dwell time at home change rate was -0.586 (95% CI: -0.742 ~ -0.370) and 0.526 (95% CI: 0.293 ~ 0.700), respectively. Increases in state-specific doubling time of total cases ranged from 1.04 ~ 6.86 days to 3.66 ~ 30.29 days after social distancing orders were put in place, consistent with mechanistic epidemic prediction models. Social distancing mandates reduce the spread of COVID-19 when they are followed.