Relying on the Classification of Finite Simple Groups it was shown by Feng and Xu (Discrete Math., 2005) that every quartic Cayley graph of a regular $p$-group, $p \neq 2,5$, is normal. In this paper a CFSG-free proof of Feng-Xu theorem is given. Along the way it is also proved that for an arbitrary $p$-group $G$ with a minimum set $\{a,b\}$ of two generators, in the corresponding Cayley graph $\mathrm{Cay}(G,\{a,a^{-1},b,b^{-1}\})$ the induced action of vertex stabilizer on the neighbors' set is contained in the dihedral group $D_8$.