The main result of this paper is that, if $Γ$ is a finite connected $4$-valent arc-transitive graph, then either $Γ$ is part of a well-understood family of graphs, or every non-identity automorphism of $Γ$ fixes at most $1/3$ of the vertices. As a corollary, we get a similar result for $3$-valent vertex-transitive graphs. Based on these results we propose a conjecture on the number of fixed points of non-identity automorphisms of vertex-transitive graphs of bounded valency.