The security of quantum key distribution has traditionally been analyzed in either the asymptotic or non-asymptotic regimes. In this paper, we provide a bridge between these two regimes, by determining second-order coding rates for key distillation in quantum key distribution under collective attacks. Our main result is a formula that characterizes the backoff from the known asymptotic formula for key distillation -- our formula incorporates the reliability and security of the protocol, as well as the mutual information variances to the legitimate receiver and the eavesdropper. In order to determine secure key rates against collective attacks, one should perform a joint optimization of the Holevo information and the Holevo information variance to the eavesdropper. We show how to do so by analyzing several examples, including the six-state, BB84, and continuous-variable quantum key distribution protocols (the last involving Gaussian modulation of coherent states along with heterodyne detection). The technical contributions of this paper include one-shot and second-order analyses of private communication over a compound quantum wiretap channel with fixed marginal and key distillation over a compound quantum wiretap source with fixed marginal. We also establish the second-order asymptotics of the smooth max-relative entropy of quantum states acting on a separable Hilbert space, and we derive a formula for the Holevo information variance of a Gaussian ensemble of Gaussian states.