Let $E$ be a subset of the affine plane over a finite field $\mathbb{F}_q$. We bound the size of the subgroup of $SL_2(\mathbb{F}_q)$ that preserves $E$. As a consequence, we show that if $E$ has size $\ll q^α$ and is preserved by $\gg q^β$ elements of $SL_2(\mathbb{F}_q)$ with $β\geq 3α/2$, then $E$ is contained in a line. This result is sharp in general, and will be proved by using combinatorial arguments and applying a point-line incidence bound in $\mathbb{F}_q^3$ due to Mockenhaupt and Tao (2004).