We consider the graphs involved in the theoretical physics model known as the colored Sachdev-Ye-Kitaev (SYK) model. We study in detail their combinatorial properties at any order in the so-called $1/N$ expansion, and we enumerate these graphs asymptotically. Because of the duality between colored graphs involving $q+1$ colors and colored triangulations in dimension $q$, our results apply to the asymptotic enumeration of spaces that generalize unicellular maps - in the sense that they are obtained from a single building block - for which a higher-dimensional generalization of the genus is kept fixed.