A normal variety $X$ is called $H$-spherical for the action of the complex reductive group $H$ if it contains a dense orbit of some Borel subgroup of $H$. We resolve a conjecture of Hodges--Yong by showing that their spherical permutations are characterized by permutation pattern avoidance. Together with results of Gao--Hodges--Yong this implies that the sphericality of a Schubert variety $X_w$ with respect to the largest possible Levi subgroup is characterized by this same pattern avoidance condition.