Segregation is a key challenge in promoting more diverse and inclusive cities. Research based on large-scale mobility data indicates that segregation between majority and minority groups persists in daily activities beyond residential areas, like visiting shops and restaurants. Aspects including lifestyle differences, homophily, and mobility constraints have been proposed as drivers of this phenomenon, but their contributions remain poorly quantified. Here, we elucidate how different mechanisms influence segregation outside home, looking at the distinctive segregation experienced by native and foreign-born individuals. Our study is based on the movement of ~320,000 individual smartphone devices collected in Sweden, where immigration creates profound divides. We find that while day-to-day activities lead to mixing for native-born individuals, foreign-born individuals remain segregated in their out-of-home activities. Using counterfactual simulations, we show that this heterogeneous effect of mobility on experienced segregation results mainly from two mechanisms: homophily and limited travel, i.e. foreign-born individuals (i) tend to visit destinations visited by similar individuals, and (ii) have limited mobility ranges. We show that homophily, as represented by destination preference, plays a minor role, while limited mobility, associated with reduced transport access, limits opportunities for foreign-born minorities to diversify their encounters. Our findings suggest that enhancing transport accessibility in foreign-born concentrated areas could reduce social segregation.