Let $q$ be a positive integer. Consider an infinite word $ω=w_0w_1w_2\cdots$ over an alphabet of cardinality $q$. A finite word $u$ is called an arithmetic factor of $ω$ if $u=w_cw_{c+d}w_{c+2d}\cdots w_{c+(|u|-1)d}$ for some choice of positive integers $c$ and $d$. We call $c$ the initial number and $d$ the difference of $u$. For each such $u$ we define its arithmetic index by $\lceil\log_q d\rceil$ where $d$ is the least positive integer such that $u$ occurs in $ω$ as an arithmetic factor with difference $d$. In this paper we study the rate of growth of the arithmetic index of arithmetic factors of a generalization of the Thue-Morse word defined over an alphabet of prime cardinality. More precisely, we obtain upper and lower bounds for the maximum value of the arithmetic index in $ω$ among all its arithmetic factors of length $n$.