For a set of permutations (patterns) $Π$ in $S_k$, consider the set of all permutations in $S_n$ that avoid all patterns in $Π$. An important problem in current algebraic combinatorics is to find pattern sets $Π$ such that the corresponding quasi-symmetric function is symmetric for all $n$. Recently, Bloom and Sagan proved that for any $k \ge 4$, the size of such $Π$ must be at least $3$ unless $Π\subseteq \{[1, 2, \dots, k],\; [k, \dots, 1]\}$, and asked for a general lower bound. We prove that the minimal size of such $Π$ is exactly $k - 1$. The proof applies a new generalization of a theorem of Bose from extremal combinatorics. This generalization is proved using the multilinear polynomial approach of Alon, Babai and Suzuki to the extension by Ray-Chaudhuri and Wilson to Bose's theorem.