The subspace-based techniques are widely utilized in various scientific fields, and they need accurate estimation of the signal subspace dimension. The classic RMT estimator for model order estimation based on random matrix theory assumes that the noise is white Gaussian, and performs poorly in the presence of colored noise with unknown covariance matrix. In the presence of colored noise, the multivariate regression (MV-R) algorithm models the source detection as a multivariate regression problem and infers the model order from the covariance matrix of the residual error. However, the MV-R algorithm requires that the noise is sufficiently weaker than the signal. In order to deal with these problems, this paper proposes a novel signal number estimation algorithm in the presence of colored noise based on the analysis of the behavior of information theoretic criteria. Firstly, a first criterion is defined as the ratio of the current eigenvalue and the mean of the next ones, and its properties is analyzed with respect to the over-modeling and under-modeling. Moreover, a second criterion is designed as the ratio of the current value and the next value of the first criterion, and its properties is analyzed with respect to the over-modeling and under-modeling. Then, a novel enhanced RMT estimator is proposed for signal number estimation by analyzing the detection properties among the signal number estimates obtained by these two criteria, the MV-R estimator and the RMT estimator to sequentially determine whether the eigenvalue being tested is arising from a signal or from noise. Finally, simulation results are presented to illustrate that the proposed enhanced RMT estimator has better estimation performance than the existing methods.