We derive an explicit formula for the connected $(n,m)$-point functions associated to an arbitrary diagonal tau-function $τ_f(\boldsymbol{t}^+,\boldsymbol{t}^-)$ of the 2d Toda lattice hierarchy using fermionic computations and the boson-fermion correspondence. Then for fixed $\boldsymbol{t}^-$, we compute the KP-affine coordinates of $τ_f(\boldsymbol{t}^+,\boldsymbol{t}^-)$. As applications, we present a unified approach to compute various types of connected double Hurwitz numbers, including the ordinary double Hurwitz numbers, the double Hurwitz numbers with completed $r$-cycles, and the mixed double Hurwitz numbers. We also apply this method to the computation of the stationary Gromov-Witten invariants of $\mathbb P^1$ relative to two points.