We prove that there exists a constant $C$ so that, for all $s,k \in \mathbb{N}$, if $G$ has average degree at least $k^{Cs^3}$ and does not contain $K_{s,s}$ as a subgraph then it contains an induced subgraph which is $C_4$-free and has average degree at least $k$. It was known that some function of $s$ and $k$ suffices, but this is the first explicit bound. We give several applications of this result, including short and streamlined proofs of the following two corollaries. We show that there exists a constant $C$ so that, for all $s,k \in \mathbb{N}$, if $G$ has average degree at least $k^{Cs^3}$ and does not contain $K_{s,s}$ as a subgraph then it contains an induced subdivision of $K_k$. This is the first quantitative improvement on a well-known theorem of Kühn and Osthus; their proof gives a bound that is triply exponential in both $k$ and $s$. We also show that for any hereditary degree-bounded class $\mathcal{F}$, there exists a constant $C=C_\mathcal{F}$ so that $C^{s^3}$ is a degree-bounding function for $\mathcal{F}$. This is the first bound of any type on the rate of growth of such functions. It is open whether there is always a polynomial degree-bounding function.