There is a conjecture that the second largest maximal cliques in Paley graphs of square order $P(q^2)$ have size $\frac{q+ε}{2}$, where $q \equiv ε\pmod 4$, and split into two orbits under the full group of automorphisms whenever $q \ge 25$ (a symmetric description for these two orbits is known). However, some extra second largest maximal cliques (of this size) exist in $P(q^2)$ whenever $q \in \{9,11,13,17,19,23\}$. In this paper we analyse the algebraic and geometric structure of the extra cliques.