We prove that the isomonodromic tau function on a torus with Fuchsian singularities and generic monodromies in $GL(N,\mathbb{C})$ can be written in terms of a Fredholm determinant of Cauchy-Plemelj operators. We further show that the minor expansion of this Fredholm determinant is described by a series labeled by charged partitions. As an example, we show that in the case of $SL(2,\mathbb{C})$ this combinatorial expression takes the form of a dual Nekrasov-Okounkov partition function, or equivalently of a free fermion conformal block on the torus. Based on these results, we also propose a definition of the tau function of the Riemann-Hilbert problem on a torus with generic jump on the A-cycle.