We show for $A,B\subset\mathbb{R}^d$ of equal volume and $t\in (0,1/2]$ that if $|tA+(1-t)B|< (1+t^d)|A|$, then (up to translation) $|\text{co}(A\cup B)|/|A|$ is bounded. This establishes the sharp threshold for Figalli and Jerison's quantative stability of the Brunn-Minkowski inequality. We additionally establish a similar sharp threshold for iterated sumsets.