Due to severe signal attenuation at millimeter-wave (mmWave) frequencies large antenna arrays are required at both base station and user equipment to achieve necessary beamfoming gain and compensate for the signal power loss. The initial access and beamforming algorithms are typically designed assuming sparsity of mmWave channels, resulting from a very few significant multipath clusters, and considering fixed locations of terminals and scatterers. Channel tracking algorithms have been proposed to account for channel variations due to user mobility. Existing works did not consider mobility of the scatterers, which adds new challenges and opportunities into a channel tracking problem. In this work, we consider a more realistic assumption of mobile scatterers and their impact on channel tracking algorithms. We propose a novel channel tracking algorithm that takes into account the dynamics of cluster evolution, and adaptively tracks channel parameters with the objective to reduce training overhead. We also propose a simple implementation of aperiodic tracking to accommodate tracking to different channel variations. We analyze the performance of the proposed tracking algorithm under highly dynamic channels, and compare it to existing channel tracking algorithms with respect to tracking accuracy, achievable rate, and required training overhead, when aperiodic and periodic trackings are used.