We present a new number theoretic definition of discrete fractional Fourier transform (DFrFT) . In this approach the DFrFT is defined as the $N \times N$ dimensional unitary representation of the generator of the arithmetic rotational group $SO_{2}[\mathbb{Z}_N]$, which is the finite set of $\bmod N$ integer, $2\times 2$ matrices acting on the points of the discrete toroidal phase space lattice $\mathbb{Z}_N \times \mathbb{Z}_N$, preserving the euclidean distance $\bmod N$. We construct explicitly, using techniques of the Finite Quantum Mechanics (FQM), the $p^n$ dimensional unitary matrix representation of the group $SO_{2}[\mathbb{Z}_{p^n}]$ and especially we work out in detail the one which corresponds to the generator. This is our definition of the arithmetic fractional Fourier transform (AFrFT). Following this definition, we proceed to the construction of efficient quantum circuits for the AFrFT, on sets of $n$ $p$-dimensional qudits with $p$ a prime integer, by introducing novel quantum subcircuits for diagonal operators with quadratic phases as well as new quantum subcircuits for multipliers by a constant. The quantum subcircuits that we introduce provide a set capable to construct quantum circuits for any element of a more general group, the group of Linear Canonical Transformations (LCT), $SL_{2}[\mathbb{Z}_N]$ of the toroidal phase space lattice. As a byproduct, extensions of the diagonal and multiplier quantum circuits for both the qudit and qubit case are given, which are useful alone in various applications. Also, we analyze the depth, width and gate complexity of the efficient AFrFT quantum circuit and we estimate its gate complexity which is of the order $O(n^2)$, its depth which is of the order $O(n)$ with depth $n$, while at the same time it has a structure permitting local interactions between the qudits.