Hypergeometric functions of complex matrices were introduced by James in multivariate statistics. These special functions play many roles in random matrix theory. The main goal of this paper is to suggest a new use for them as holomorphic observables of the Circular Unitary Ensemble. We analyze the high-dimensional behavior of the expected derivatives of these random analytic functions, and show that they admit asymptotic expansions which can be described in terms of quasimodular forms, giving an apparently new connection between the CUE and number theory.