In this paper, we introduce the notion of spectral genus $\widetilde{p}_{g}$ of a germ of an isolated hypersurface singularity $(\mathbb{C}^{n+1}, 0) \to (\mathbb{C}, 0)$, defined as a sum of small exponents of monodromy eigenvalues. The number of these is equal to the geometric genus $p_{g}$, and hence $\widetilde{p}_g$ can be considered as a secondary invariant to it. We then explore a secondary version of the Durfee conjecture on $p_{g}$, and we predict an inequality between $\widetilde{p}_{g}$ and the Milnor number $μ$, to the effect that $$\widetilde{p}_g\leq\frac{μ-1}{(n+2)!}.$$ We provide evidence by confirming our conjecture in several cases, including homogeneous singularities and singularities with large Newton polyhedra, and quasi-homogeneous or irreducible curve singularities. We also show that a weaker inequality follows from Durfee's conjecture, and hence holds for quasi-homogeneous singularities and curve singularities. Our conjecture is shown to relate closely to the asymptotic behavior of the holomorphic analytic torsion of the sheaf of holomorphic functions on a degeneration of projective varieties, potentially indicating deeper geometric and analytic connections.