Let $d(\cdot)$ denote the natural density on the positive integers. We characterize all sets $A,B$ with positive density satisfying $d(A+B)=d(A)+d(B)$, under the assumption that the two sets are not both contained in a proper finite union of residue classes. This gives a new inverse theorem for Kneser's sumset inequality in the integers, and provides a partial answer to a long-standing open question of Erdős and Graham.