Recent studies suggest that social media usage -- while linked to an increased diversity of information and perspectives for users -- has exacerbated user polarization on many issues. A popular theory for this phenomenon centers on the concept of "filter bubbles": by automatically recommending content that a user is likely to agree with, social network algorithms create echo chambers of similarly-minded users that would not have arisen otherwise. However, while echo chambers have been observed in real-world networks, the evidence for filter bubbles is largely post-hoc. In this work, we develop a mathematical framework to study the filter bubble theory. We modify the classic Friedkin-Johnsen opinion dynamics model by introducing another actor, the network administrator, who filters content for users by making small changes to the edge weights of a social network (for example, adjusting a news feed algorithm to change the level of interaction between users). On real-world networks from Reddit and Twitter, we show that when the network administrator is incentivized to reduce disagreement among users, even relatively small edge changes can result in the formation of echo chambers in the network and increase user polarization. We theoretically support this observed sensitivity of social networks to outside intervention by analyzing synthetic graphs generated from the stochastic block model. Finally, we show that a slight modification to the incentives of the network administrator can mitigate the filter bubble effect while minimally affecting the administrator's target objective, user disagreement.