In 2011, Fang et al. in (J. Combin. Theory A 118 (2011) 1039-1051) posed the following problem: Classify non-normal locally primitive Cayley graphs of finite simple groups of valency $d$, where either $d\leq 20$ or $d$ is a prime number. The only case for which the complete solution of this problem is known is of $d=3$. Except this, a lot of efforts have been made to attack this problem by considering the following problem: Characterize finite nonabelian simple groups which admit non-normal locally primitive Cayley graphs of certain valency $d\geq4$. Even for this problem, it was only solved for the cases when either $d\leq 5$ or $d=7$ and the vertex stabilizer is solvable. In this paper, we make crucial progress towards the above problems by completely solving the second problem for the case when $d\geq 11$ is a prime and the vertex stabilizer is solvable.