For stochastic Volterra equations driven by standard Brownian and with singular kernels $K(u)=u^{H-\frac{1}{2}}/Γ(H+1/2), H\in (0,1/2)$, it is known that the Milstein scheme has a convergence rate of $n^{-2H}$. In this paper, we show that this rate is optimal. Moreover, we show that the error normalized by $n^{-2H}$ converge stably in law to the (nonzero) solution of a certain linear Volterra equation of random coefficients with the same fractional kernel.