The early time regime of the Kardar-Parisi-Zhang (KPZ) equation in $1+1$ dimension, starting from a Brownian initial condition with a drift $w$, is studied using the exact Fredholm determinant representation. For large drift we recover the exact results for the droplet initial condition, whereas a vanishingly small drift describes the stationary KPZ case, recently studied by weak noise theory (WNT). We show that for short time $t$, the probability distribution $P(H,t)$ of the height $H$ at a given point takes the large deviation form $P(H,t) \sim \exp{\left(-Φ(H)/\sqrt{t} \right)}$. We obtain the exact expressions for the rate function $Φ(H)$ for $H<H_{c2}$. Our exact expression for $H_{c2}$ numerically coincides with the value at which WNT was found to exhibit a spontaneous reflection symmetry breaking. We propose two continuations for $H>H_{c2}$, which apparently correspond to the symmetric and asymmetric WNT solutions. The rate function $Φ(H)$ is Gaussian in the center, while it has asymmetric tails, $|H|^{5/2}$ on the negative $H$ side and $H^{3/2}$ on the positive $H$ side.