Musical tastes reflect our unique values and experiences, our relationships with others, and the places where we live. But as each of these things changes, do our tastes also change to reflect the present, or remain fixed, reflecting our past? Here, we investigate how where a person lives shapes their musical preferences, using geographic relocation to construct quasi-natural experiments that measure short- and long-term effects. Analyzing comprehensive data on over 16 million users on Spotify, we show that relocation within the United States has only a small impact on individuals' tastes, which remain more similar to those of their past environments. We then show that the age gap between a person and the music they consume indicates that adolescence, and likely their environment during these years, shapes their lifelong musical tastes. Our results demonstrate the robustness of individuals' musical identity, and shed new light on the development of preferences.