This paper studies the impact of imperfect channel covariance information on the uplink (UL) and downlink (DL) spectral efficiencies (SEs) of a time-division duplexed (TDD) massive multiple-input multiple-output (MIMO) system. We derive closed-form expressions for the UL and DL average SEs by considering linear minimum mean squared (LMMSE)-type and element-wise LMMSE-type channel estimation that represent LMMSE and element-wise LMMSE with estimated covariance matrices, respectively. The closed-form expressions of these average SEs are functions of the number of observations used for estimating the spatial covariance matrices of individual and contaminated channels of a target user, and thus enable us to select these key parameters to achieve the desired SE. We present a theoretical analysis of SE behavior for different values of these parameters, followed by simulations, which also demonstrate and validate this behavior. Specifically, we present the SEs computed using estimated covariance matrices and show the accurate agreement between the theoretical and simulated SEs as functions of the number of observations for estimating the covariance matrices of individual and contaminated channels of a user. We also compare these SEs across channel estimation techniques using analytical and simulation studies.