The motion of a graph is the minimum number of vertices that are moved by a non-trivial automorphism. Equivalently, it can be defined as the minimal degree of its automorphism group (as a permutation group on the vertices). In this paper we develop some results on permutation groups (primitive and imprimitive) with small minimal degree. As a consequence of such results we classify vertex-transitive graphs whose motion is $4$ or a prime number.