Seppäläinen and Valkó showed in \cite{SV} that for a suitable choice of parameters, the variance growth of the free energy of the stationary O'Connell-Yor polymer is governed by the exponent $2/3$, characteristic of models in the KPZ universality class. We develop exact formulas based on Gaussian integration by parts to relate the cumulants of the free energy, $\log Z_{n,t}^θ$, to expectations of products of quenched cumulants of the time of the first jump from the boundary into the system, $s_0$. We then use these formulas to obtain estimates for the $k$-th central moment of $\log Z_{n,t}^θ$ as well as the $k$-th annealed moment of $s_0$ for $k> 2$, with nearly optimal exponents $(1/3)k+ε$ and $(2/3)k+ε$, respectively.