We characterise the primitive 2-closed groups $G$ of rank at most four that are not the automorphism group of a graph or digraph and show that if the degree is at least 2402 then there are just two infinite families or $G\leqslant \mathrm{A}Γ\mathrm{L}_1(p^d)$, the 1-dimensional affine semilinear group. These are the first known examples of non-regular 2-closed groups that are not the automorphism group of a graph or digraph.