The COVID-19 pandemic has significantly impacted daily activity rhythms and life routines. Understanding the dynamics of these impacts on different groups of people is essential for creating environments where people's lives and well-being are least disturbed during such circumstances. Starting in June 2021, we conducted a year-long study to collect high-resolution data from fitness trackers as well as answers to monthly questionnaires from 128 working adults. Using questionnaires, we investigate how routines of exercising and working have changed throughout the pandemic for different people. In addition to that, for each person in the study, we build temporal distributions of daily step counts to quantify their daily movement rhythms and use the inverse of the Earth mover's distance between different movement rhythms to quantify the movement consistency over time. Throughout the pandemic, our cohort shows a shift in exercise routines, manifested in a decrease in time spent on non-walking physical exercises as opposed to the unchanged amount of time spent on walking. In terms of daily rhythms of movement, we show that migrants and those who live alone demonstrate a lower level of consistency of daily rhythms of movement compared to their counterparts. We also observe a relationship between movement and on-site work attendance, as participants who go to work (as opposed to working remotely) also tend to maintain more consistent daily rhythms of movement. Men and migrants show a faster pace in going back to work after the decrease in restriction measures that were set in place due to the pandemic. Our results quantitatively demonstrate the unequal effect of the pandemic among different sub-populations and inform organizations and policymakers to provide more adequate support and adapt to the different needs of different groups in the post-pandemic era.