A graph $\Ga=(V,E)$ is called a Cayley graph of some group $T$ if the automorphism group $\Aut(\Ga)$ contains a subgroup $T$ which acts on regularly on $V$. If the subgroup $T$ is normal in $\Aut(\Ga)$ then $\Ga$ is called a normal Cayley graph of $T$. Let $r$ be an odd prime. Fang et al. \cite{FMW} proved that, with a finite number of exceptions for finite simple group $T$, every connected symmetric Cayley graph of $T$ of valency $r$ is normal. In this paper, employing maximal factorizations of finite almost simple groups, we work out a possible list of those exceptions for $T$.