We study the Falconer distance set problem in Euclidean space and obtain improved dimensional estimates under natural Fourier analytic assumptions cast in terms of the Fourier dimension and spectrum. Interestingly, under reasonably mild assumptions, we are able to beat the $d/2$ dimension threshold in dimensions $d \geq 5$. For example, we show that (in any ambient spatial dimension $d$) a Borel set with Fourier dimension at least $2$ has a distance set of full Hausdorff dimension. We also show that (in any ambient spatial dimension $d$) a Borel set with Fourier spectrum at least $d/4+1$ at $θ=1/2$ has a distance set of full Hausdorff dimension. In particular, this can hold for sets with Fourier dimension zero (provided $d \geq 4$). We also consider pinned variants of these problems and construct examples that demonstrate the sharpness (or near sharpness) of our results.