We deduce, as a consequence of the arithmetic removal lemma, an almost-all version of the Balog-Szemerédi-Gowers theorem: For any $K\geq 1$ and $\varepsilon > 0$, there exists $δ= δ(K,\varepsilon)>0$ such that the following statement holds: if $|A+_ΓA| \leq K|A|$ for some $Γ\geq (1-δ)|A|^2$, then there is a subset $A' \subset A$ with $|A'| \geq (1-\varepsilon)|A|$ such that $|A'+A'| \leq |A+_ΓA| + \varepsilon |A|$. We also discuss issues around quantitative bounds in this statement, in particular showing that when $A \subset \mathbb{Z}$ the dependence of $δ$ on $ε$ cannot be polynomial for any fixed $K>2$.