Personalization is pervasive in the online space as, when combined with learning, it leads to higher efficiency and revenue by allowing the most relevant content to be served to each user. However, recent studies suggest that such personalization can propagate societal or systemic biases, which has led to calls for regulatory mechanisms and algorithms to combat inequality. Here we propose a rigorous algorithmic framework that allows for the possibility to control biased or discriminatory personalization with respect to sensitive attributes of users without losing all of the benefits of personalization.