In the beyond fifth-generation (B5G) and upcoming sixth-generation (6G) wireless communication systems, millimeter (mmWave) wave technology is a promising solution for offering additional bandwidth resources and mitigating spectrum congestion. Beam tracking is an essential procedure for providing reliable communication services in the mmWave communication system, with the challenge of providing consistent and accurate tracking performance. In this study, we introduce a low-overhead beam tracking algorithm based on channel charting, which significantly reduces beam scanning times during the tracking process. By projecting the beam information to the channel chart, the beam tracking problem is transformed into the acquisition of the beam cluster in the channel chart. Leveraging contrastive learning, the proposed channel chart projects high-dimensional channel state information into a low-dimensional feature space that preserves spatial proximities. Using a dynamic candidate beam acquisition strategy, the complexity of our beam tracking algorithm is significantly reduced. The proposed algorithm significantly reduces scanning complexity while maintaining high prediction accuracy, achieving an accuracy of 98.27\% in simulation environments. Compared to existing methods, the proposed method can reduce beam scanning times by up to 55.9\%. In addition, we also performed field tests, and the measured results demonstrated excellent communication quality during mobility.