Despite the rising importance of enhancing community resilience to disasters, our understanding on how communities recover from catastrophic events is limited. Here we study the population recovery dynamics of disaster affected regions by observing the movements of over 2.5 million mobile phone users across three countries before, during and after five major disasters. We find that, although the regions affected by the five disasters have significant differences in socio-economic characteristics, we observe a universal recovery pattern where displaced populations return in an exponential manner after all disasters. Moreover, the heterogeneity in initial and long-term displacement rates across communities across the three countries were explained by a set of key universal factors including the community's median income level, population size, housing damage rate, and the connectedness to other cities. These universal properties of recovery dynamics extracted from large scale evidence could impact efforts on urban resilience and sustainability across various disciplines.