Extremal problems on the $4$-cycle $C_4$ played a heuristic important role in the development of extremal graph theory. A fundamental theorem of Füredi states that the Turán number $ex(q^2+q+1, C_4)\leq \frac12 q(q+1)^2$ holds for every $q\geq 14$, which matches with the classic construction of Erdős-R{é}nyi-Sós and Brown from finite geometry for prime powers $q$. Very recently, we obtained the first stability result on Füredi's theorem, by showing that for large even $q$, every $(q^2+q+1)$-vertex $C_4$-free graph with more than $\frac12 q(q+1)^2-0.2q$ edges must be a spanning subgraph of a unique polarity graph. Using new technical ideas in graph theory and finite geometry, we strengthen this by showing that the same conclusion remains true if the number of edges is lowered to $\frac12 q(q+1)^2-\frac12 q+o(q)$. Among other applications, this gives an immediate improvement on the upper bound of $ex(n,C_4)$ for infinitely many integers $n$. A longstanding conjecture of Erdős and Simonovits states that every $n$-vertex graph with $ex(n,C_4)+1$ edges contains at least $(1+o(1))\sqrt{n}$ 4-cycles. We proved an exact result and confirmed Erdős-Simonovits conjecture for infinitely many integers $n$. As the second main result of this paper, we further characterize all extremal graphs for which achieve the $\ell$th least number of copies of $C_4$ for any fixed positive integer $\ell$. This can be extended to more general settings and provides enhancements on the understanding of the supersaturation problem of $C_4$.