The existence of moments of first downward passage times of a spectrally negative Lévy process is governed by the general dynamics of the Lévy process, i.e. whether the Lévy process is drifting to $+\infty$, $-\infty$ or oscillates. Whenever the Lévy process drifts to $+\infty$, we prove that the $κ$-th moment of the first passage time (conditioned to be finite) exists if and only if the $(κ+1)$-th moment of the Lévy jump measure exists. This generalises a result shown earlier by Delbaen for Cramér-Lundberg risk processes \cite{Delbaen1990}. Whenever the Lévy process drifts to $-\infty$, we prove that all moments of the first passage time exist, while for an oscillating Lévy process we derive conditions for non-existence of the moments and in particular we show that no integer moments exist.