It is widely believed that the parity of the partition function $p(n)$ is ``random.'' Contrary to this expectation, in this note we prove the existence of infinitely many congruence relations modulo 4 among its values. For each square-free integer $1<D\equiv 23\pmod{24},$ we construct a weight 2 meromorphic modular form that is congruent modulo 4 to a certain twisted generating function for the numbers $p\big(\frac{Dm^2+1}{24}\big)\pmod 4$. We prove the existence of infinitely many linear dependence congruences modulo 4 among suitable sets of holomorphic normalizations of these series. These results rely on the theory of class numbers and Hilbert class polynomials, and {\it generalized twisted Borcherds products} developed by Bruinier and the author.