We consider the early time regime of the Kardar-Parisi-Zhang (KPZ) equation in $1+1$ dimensions in curved (or droplet) geometry. We show that for short time $t$, the probability distribution $P(H,t)$ of the height $H$ at a given point $x$ takes the scaling form $P(H,t) \sim \exp{\left(-Φ_{\rm drop}(H)/\sqrt{t} \right)}$ where the rate function $Φ_{\rm drop}(H)$ is computed exactly. While it is Gaussian in the center, i.e., for small $H$, the PDF has highly asymmetric non-Gaussian tails which we characterize in detail. This function $Φ_{\rm drop}(H)$ is surprisingly reminiscent of the large deviation function describing the stationary fluctuations of finite size models belonging to the KPZ universality class. Thanks to a recently discovered connection between KPZ and free fermions, our results have interesting implications for the fluctuations of the rightmost fermion in a harmonic trap at high temperature and the full couting statistics at the edge.