We prove a sharp stability result concerning how close homothetic sets attaining near-equality in the Brunn-Minkowski inequality are to being convex. In particular, resolving a conjecture of Figalli and Jerison, we show there are universal constants $C_n,d_n>0$ such that for $A \subset \mathbb{R}^n$ of positive measure, if $|\frac{A+A}{2}\setminus A| \le d_n |A|$, then $|\operatorname{co}(A)\setminus A| \le C_n |\frac{A+A}{2}\setminus A|$ for $\operatorname{co}(A)$ the convex hull of $A$.