We study the impact of phase noise on the downlink performance of a multi-user multiple-input multiple-output system, where the base station (BS) employs a large number of transmit antennas $M$. We consider a setup where the BS employs $M_{\mathrm{osc}}$ free-running oscillators, and $M/M_{\mathrm{osc}}$ antennas are connected to each oscillator. For this configuration, we analyze the impact of phase noise on the performance of the regularized zero-forcing (RZF), when $M$ and the number of users $K$ are asymptotically large, while the ratio $M/K=β$ is fixed. We analytically show that the impact of phase noise on the signal-to-interference-plus-noise ratio (SINR) can be quantified as an effective reduction in the quality of the channel state information available at the BS when compared to a system without phase noise. As a consequence, we observe that as $M_{\mathrm{osc}}$ increases, the SINR performance of all considered precoders degrades. On the other hand, the variance of the random phase variations caused by the BS oscillators reduces with increasing $M_{\mathrm{osc}}$. Through Monte-Carlo simulations, we verify our analytical results, and compare the performance of the precoders for different phase noise and channel noise variances. For all considered precoders, we show that when $β$ is small, the performance of the setup where all BS antennas are connected to a single oscillator is superior to that of the setup where each BS antenna has its own oscillator. However, the opposite is true when $β$ is large and the signal-to-noise ratio at the users is low.