We derive the joint asymptotic distribution of empirical quantiles and expected shortfalls under general conditions on the distribution of the underlying observations. In particular, we do not assume that the distribution function is differentiable at the quantile with strictly positive derivative. Hence the rate of convergence and the asymptotic distribution for the quantile can be non-standard, but our results show that the expected shortfall remains asymptotically normal with a $\sqrt{n}$-rate, and we even give the joint distribution in such non-standard cases. In the derivation we use the bivariate scoring functions for quantile and expected shortfall as recently introduced by Fissler and Ziegel (2016). The main technical issue is to deal with the distinct rates for quantile and expected shortfall when applying the argmax-continuity theorem. We also consider spectral risk measures with finitely-supported spectral measures, and illustrate our results in a simulation study.