The random geometric graph $\mathsf{RGG}(n,\mathbb{S}^{d-1}, p)$ is formed by sampling $n$ i.i.d. vectors $\{V_i\}_{i = 1}^n$ uniformly on $\mathbb{S}^{d-1}$ and placing an edge between pairs of vertices $i$ and $j$ for which $\langle V_i,V_j\rangle \ge τ^p_d,$ where $τ^p_d$ is such that the expected density is $p.$ We study the low-degree Fourier coefficients of the distribution $\mathsf{RGG}(n,\mathbb{S}^{d-1}, p)$ and its Gaussian analogue. Our main conceptual contribution is a novel two-step strategy for bounding Fourier coefficients which we believe is more widely applicable to studying latent space distributions. First, we localize the dependence among edges to few fragile edges. Second, we partition the space of latent vector configurations $(\mathsf{RGG}(n,\mathbb{S}^{d-1}, p))^{\otimes n}$ based on the set of fragile edges and on each subset of configurations, we define a noise operator acting independently on edges not incident (in an appropriate sense) to fragile edges. We apply the resulting bounds to: 1) Settle the low-degree polynomial complexity of distinguishing spherical and Gaussian random geometric graphs from Erdos-Renyi both in the case of observing a complete set of edges and in the non-adaptively chosen mask $\mathcal{M}$ model recently introduced by [MVW24]; 2) Exhibit a statistical-computational gap for distinguishing $\mathsf{RGG}$ and the planted coloring model [KVWX23] in a regime when $\mathsf{RGG}$ is distinguishable from Erdos-Renyi; 3) Reprove known bounds on the second eigenvalue of random geometric graphs.