Despite modest declines in residential segregation levels since the Civil Rights Era, segregation remains a defining feature of the U.S. landscape. This study highlights the importance of considering physical barriers--features of the urban environment that disconnect locations--when measuring segregation. We use population and geographic data for 20 U.S. Rustbelt cities from the 2010 decennial census and a novel approach for measuring and analyzing segregation that incorporates the connectivity of roads and the excess distance imposed by physical barriers, such as highways, railroad tracks, and dead-end streets. We find that physical barriers divide urban space in ways that reinforce or exacerbate segregation, but there is substantial variation in the extent to which they increase segregation both within and across these cities and for different ethnoracial groups. By uncovering a new source of variation in the segregation experienced by city residents, the findings have implications for understanding the mechanisms that contribute to the persistence of segregation and the consequences of segregation.