A set $S$ of permutations is forcing if for any sequence $\{Π_i\}_{i \in \mathbb{N}}$ of permutations where the density $d(π,Π_i)$ converges to $\frac{1}{|π|!}$ for every permutation $π\in S$, it holds that $\{Π_i\}_{i \in \mathbb{N}}$ is quasirandom. Graham asked whether there exists an integer $k$ such that the set of all permutations of order $k$ is forcing; this has been shown to be true for any $k\ge 4$. In particular, the set of all twenty-four permutations of order $4$ is forcing. We provide the first non-trivial lower bound on the size of a forcing set of permutations: every forcing set of permutations (with arbitrary orders) contains at least four permutations.