We give formulae for the cumulants of complex Wishart (LUE) and inverse Wishart matrices (inverse LUE). Their large-$N$ expansions are generating functions of double (strictly and weakly) monotone Hurwitz numbers which count constrained factorisations in the symmetric group. The two expansions can be compared and combined with a duality relation proved in [F. D. Cunden, F. Mezzadri, N. O'Connell and N. J. Simm, arXiv:1805.08760] to obtain: i) a combinatorial proof of the reflection formula between moments of LUE and inverse LUE at genus zero and, ii) a new functional relation between the generating functions of monotone and strictly monotone Hurwitz numbers. The main result resolves the integrality conjecture formulated in [F. D. Cunden, F. Mezzadri, N. J. Simm and P. Vivo, J. Phys. A 49 (2016)] on the time-delay cumulants in quantum chaotic transport. The precise combinatorial description of the cumulants given here may cast new light on the concordance between random matrix and semiclassical theories.