We show that for an $n\times n$ random symmetric matrix $A_n$, whose entries on and above the diagonal are independent copies of a sub-Gaussian random variable $ξ$ with mean $0$ and variance $1$, \[\mathbb{P}[s_n(A_n) \le ε/\sqrt{n}] \le O_ξ(ε^{1/8} + \exp(-Ω_ξ(n^{1/2}))) \quad \text{for all } ε\ge 0.\] This improves a result of Vershynin, who obtained such a bound with $n^{1/2}$ replaced by $n^{c}$ for a small constant $c$, and $1/8$ replaced by $(1/8) + η$ (with implicit constants also depending on $η> 0$). Furthermore, when $ξ$ is a Rademacher random variable, we prove that \[\mathbb{P}[s_n(A_n) \le ε/\sqrt{n}] \le O(ε^{1/8} + \exp(-Ω((\log{n})^{1/4}n^{1/2}))) \quad \text{for all } ε\ge 0.\] The special case $ε= 0$ improves a recent result of Campos, Mattos, Morris, and Morrison, which showed that $\mathbb{P}[s_n(A_n) = 0] \le O(\exp(-Ω(n^{1/2}))).$ The main innovation in our work are new notions of arithmetic structure -- the Median Regularized Least Common Denominator and the Median Threshold, which we believe should be more generally useful in contexts where one needs to combine anticoncentration information of different parts of a vector.