In this paper, we consider optimal control problems of stochastic Volterra equations (SVEs) with singular kernels, where the control domain is not necessarily convex. We establish a global maximum principle by means of the spike variation technique. To do so, we first show a Taylor type expansion of the controlled SVE with respect to the spike variation, where the convergence rates of the remainder terms are characterized by the singularity of the kernels. Next, assuming additional structure conditions for the kernels, we convert the variational SVEs appearing in the expansion to their infinite dimensional lifts. Then, we derive first and second order adjoint equations in form of infinite dimensional backward stochastic evolution equations (BSEEs) on weighted $L^2$ spaces. Moreover, we show the well-posedness of the new class of BSEEs on weighted $L^2$ spaces in a general setting.