Fix $α\in (0,1/3)$. We show that, from a topological point of view, almost all sets $A\subseteq \mathbb{N}$ have the property that, if $A^\prime=A$ for all but $o(n^α)$ elements, then $A^\prime$ is not a nontrivial sumset $B+C$. In particular, almost all $A$ are totally irreducible. In addition, we prove that the measure analogue holds with $α=1$.