The zigzag process is a variant of the telegraph process with position dependent switching intensities. A characterization of the $L^2$-spectrum for the generator of the one-dimensional zigzag process is obtained in the case where the marginal stationary distribution on $\R$ is unimodal and the refreshment intensity is zero. Sufficient conditions are obtained for a spectral mapping theorem, mapping the spectrum of the generator to the spectrum of the corresponding Markov semigroup. Furthermore results are obtained for symmetric stationary distributions and for perturbations of the spectrum, in particular for the case of a non-zero refreshment intensity. In the examples we consider (including a Gaussian target distribution) a slight increase of the refreshment intensity above zero results in a larger $L^2$-spectral gap, corresponding to an improved convergence in $L^2$.