Massive multiple-input multiple-output (MIMO) systems offer significant potential to enhance wireless communication performance, yet accurate and timely channel state information (CSI) acquisition remains a key challenge. Existing works on CSI estimation and radio map applications typically rely on stationary CSI statistics and accurate location labels. However, the CSI process can be discontinuous due to user mobility and environmental variations, and inaccurate location data can degrade the performance. By contrast, this paper studies radio-map-embedded CSI tracking and radio map construction without the assumptions of stationary CSI statistics and precise location labels. Using radio maps as the prior information, this paper develops a radio-map-embedded switching Kalman filter (SKF) framework that jointly tracks the location and the CSI with adaptive beamforming for sparse CSI observations under reduced pilots. For radio map construction without precise location labels, the location sequence and the channel covariance matrices are jointly estimated based on a Hidden Markov Model (HMM). An unbiased estimator on the channel covariance matrix is found. Numerical results on ray-traced MIMO channel datasets demonstrate that using 1 pilot in every 10 milliseconds, an average of over 97% of capacity over that of perfect CSI can be achieved, while a conventional Kalman filter (KF) can only achieve 76%. Furthermore, the proposed radio-map-embedded CSI model can reduce the localization error from 30 meters from the prior to 6 meters for radio map construction.