Polarization of opinions about vaccination can have a negative impact on pandemic control. In this work we quantify this negative impact for the transmission of COVID-19, using an agent based simulation in an heterogeneous population with multi-type networks, representing different types of social interactions. We show that the clustering of unvaccinated individuals, associated with polarization of opinion, can lead to significant differences in the evolution of the pandemic compared to deterministic model predictions. Under our realistic baseline scenario these differences are a 33pc increase of the effective reproduction number, a 157pc increase of infections at the peak and a 30pc increase in the final cumulative attack rate.