Polarization between groups is a major topic of contemporary societal debate as well as of research into intergroup relations. Formal modelers of opinion dynamics try to explain how intergroup polarization can arise from simple first principles of interactions within and between groups. Models have been proposed in which intergroup attitudes affect social influence in the form of homophily or xenophobia, elaborated as fixed tendencies of individuals to interact more with in-group members, be more open to influence from in-group members and perhaps even distance oneself from attitudes of outgroup members. While these models can generate polarization between groups, their underlying assumptions curiously neglect a central insight from research on intergroup attitudes. Intergroup attitudes are themselves subject to social influence in interactions with both in- and outgroup members. I extend an existing model of opinion formation with intergroup attitudes, by adding this feedback-effect. I show how this changes model predictions about the process and the conditions of polarization between groups. In particular, it is demonstrated how the model implies that intergroup polarization can become less likely if intergroup attitudes change under social influence; and how more complex patterns of intergroup relations emerge. Especially, a renegade minority (outgroup-lovers) can have a key role in avoiding mutually negative intergroup relations and even elicit attitude reversal, resulting in a majority of individuals developing a negative attitude towards their in-group and a positive one of the outgroup. Interpretations of these theoretical results and directions for future research are further discussed.