Let $d \geq 3$ be a natural number. We show that for all finite, non-empty sets $A \subseteq \mathbb{R}^d$ that are not contained in a translate of a hyperplane, we have \[ |A-A| \geq (2d-2)|A| - O_d(|A|^{1- δ}),\] where $δ>0$ is an absolute constant only depending on $d$. This improves upon an earlier result of Freiman, Heppes and Uhrin, and makes progress towards a conjecture of Stanchescu.