Zealots are the vertices in a social network who do not change their opinions under social pressure, and are crucial to the study of opinion dynamics on complex networks. In this paper, we study the effect of zealots on the polarization dynamics of a deterministic majority-rule model using the configuration model as a substrate. To this end, we propose a novel quantifier, called `correlated polarization', for measuring the amount of polarization in the network when vertices can exists in two opposite states. The quantifier takes into account not only the fraction of vertices with each opinion, but also how they are connected to each other. We then show that the presence of zealots does not have a fixed effect on the polarization, and can change it in positive, negative or neutral way depending upon their topological characteristics like degree, their total fraction in the network, density and degree heterogeneity of the network, and the type of initial conditions of the dynamics. Our results particularly highlight the importance of the role played by the initial conditions in drifting the polarization towards lower or higher values as the total number of zealots is increased.