In this paper, we present a novel mobile user tracking (UT) scheme for codebook-based intelligent reflecting surface (IRS) aided millimeter wave (mmWave) systems. The proposed UT scheme exploits the temporal correlation of the direction from the IRS to the mobile user for selecting IRS phase shifts that provide reflection towards the user. To this end, the user's direction is periodically estimated based on a generalized likelihood ratio test (GLRT) and the user's movement trajectory is extrapolated from several past direction estimates. The efficiency of the proposed UT scheme is evaluated in terms of the average effective rate, which accounts for both the required signaling overhead and the achieved signal to noise ratio (SNR). Our results show that for medium to high SNR, the proposed codebook based UT scheme achieves a higher effective rate than two reference approaches based on full codebook search and optimization of the individual IRS unit cells, respectively.