Let $d \geq 4$ be a natural number and let $A$ be a finite, non-empty subset of $\mathbb{R}^d$ such that $A$ is not contained in a translate of a hyperplane. In this setting, we show that \[ |A-A| \geq \bigg(2d - 2 + \frac{1}{d-1} \bigg) |A| - O_{d}(|A|^{1- δ}), \] for some absolute constant $δ>0$ that only depends on $d$. This provides a sharp main term, consequently answering questions of Ruzsa and Stanchescu up to an $O_{d}(|A|^{1- δ})$ error term. We also prove new lower bounds for restricted type difference sets and asymmetric sumsets in $\mathbb{R}^d$.