In 2013, Zhai proved that most numerical semigroups of a given genus have depth at most $3$ and that the number $n_g$ of numerical semigroups of a genus $g$ is asymptotic to $S\varphi^g$, where $S$ is some positive constant and $\varphi \approx 1.61803$ is the golden ratio. In this paper, we prove exponential upper and lower bounds on the factors that cause $n_g$ to deviate from a perfect exponential, including the number of semigroups with depth at least $4$. Among other applications, these results imply the sharpest known asymptotic bounds on $n_g$ and shed light on a conjecture by Bras-Amorós (2008) that $n_g \geq n_{g-1} + n_{g-2}$. Our main tools are the use of Kunz coordinates, introduced by Kunz (1987), and a result by Zhao (2011) bounding weighted graph homomorphisms.