An automorphism of a graph is called quasi-semiregular if it fixes a unique vertex of the graph and its remaining cycles have the same length. This kind of symmetry of graphs was first investigated by Kutnar, Malnič, Martínez and Marušič in 2013, as a generalization of the well-known semiregular automorphism of a graph. Symmetric graphs of valency three or four, admitting a quasi-semiregular automorphism, have been classified in recent two papers. Let $p\geq 5$ be a prime and $Γ$ a connected symmetric graph of valency $p$ admitting a quasi-semiregular automorphism. In this paper, we first prove that either $Γ$ is a connected Cayley graph $\rm{Cay}(M,S)$ such that $M$ is a $2$-group admitting a fixed-point-free automorphism of order $p$ with $S$ as an orbit of involutions, or $Γ$ is a normal $N$-cover of a $T$-arc-transitive graph of valency $p$ admitting a quasi-semiregular automorphism, where $T$ is a non-abelian simple group and $N$ is a nilpotent group. Then in case $p=5$, we give a complete classification of such graphs $Γ$ such that either $\rm{Aut}(Γ)$ has a solvable arc-transitive subgroup or $Γ$ is $T$-arc-transitive with $T$ a non-abelian simple group. We also construct the first infinite family of symmetric graphs that have a quasi-semiregular automorphism and an insolvable full automorphism group.