Ultra-dense (UD) wireless networks and cloud radio access networks (CRAN) are two promising network architectures for the emerging fifth-generation (5G) wireless communication systems. By jointly employing them, a new appealing network solution is proposed in this paper, termed UD-CRAN. In a UD-CRAN, millimeter-wave (mmWave) wireless fronthaul is preferred for information exchange between the central processor and the distributed remote radio heads (RRHs), due to its lower cost and higher flexibility in deployment, compared to fixed optical links. This motivates our study in this paper on the downlink transmission in a mmWave fronthaul enabled, orthogonal frequency division multiple access (OFDMA) based UD-CRAN. In particular, the fronthaul is shared among the RRHs via time division multiple access (TDMA), while the RRHs jointly transmit to the users on orthogonal frequency subchannels using OFDMA. The joint resource allocation over the TDMA-based mmWave fronthaul and OFDMA-based wireless transmission is investigated to maximize the weighted sum rate of all users. Although the problem is non-convex, we propose a Lagrange duality based solution, which can be efficiently computed with good accuracy. To further reduce the complexity, we also propose a greedy search based heuristic, which achieves close to optimal performance under practical setups. Finally, we show the significant throughput gains of the proposed joint resource allocation approach compared to other benchmark schemes by simulations.