On the time-varying channel estimation, the traditional downlink (DL) channel restoration schemes usually require the reconstruction for the covariance of downlink process noise vector, which is dependent on DL channel covariance matrix (CCM). However, the acquisition of the CCM leads to unacceptable overhead in massive MIMO systems. To tackle this problem, in this paper, we propose a novel scheme for the DL channel tracking. First, with the help of virtual channel representation (VCR), we build a dynamic uplink (UL) massive MIMO channel model with the consideration of off-grid refinement. Then, a coordinate-wise maximization based expectation maximization (EM) algorithm is adopted for capturing the model parameters, including the spatial signatures, the time-correlation factors, the off-grid bias, the channel power, and the noise power. Thanks to the angle reciprocity, the spatial signatures, timecorrelation factors and off-grid bias of the DL channel model can be reconstructed with the knowledge of UL ones. However, the other two kinds of model parameters are closely related with the carrier frequency, which cannot be perfectly inferred from the UL ones. Instead of relearning the DL model parameters with dedicated training, we resort to the optimal Bayesian Kalman filter (OBKF) method to accurately track the DL channel with the partially prior knowledge. At the same time, the model parameters will be gradually restored. Specially, the factor-graph and the Metropolis Hastings MCMC are utilized within the OBKF framework. Finally, numerical results are provided to demonstrate the efficiency of our proposed scheme.