The minimal degree of a permutation group $G$ is the minimum number of points not fixed by non-identity elements of $G$. Lower bounds on the minimal degree have strong structural consequences on $G$. In 2014 Babai proved that the automorphism group of a strongly regular graph with $n$ vertices has minimal degree $\geq c n$, with known exceptions. Strongly regular graphs correspond to primitive coherent configurations of rank 3. We extend Babai's result to primitive coherent configurations of rank 4. We also show that the result extends to non-geometric distance-regular graphs of bounded diameter. The proofs combine structural and spectral methods.