Let $d\geq 3$ be a fixed integer and $A$ be the adjacency matrix of a random $d$-regular directed or undirected graph on $n$ vertices. We show there exist constants $\mathfrak d>0$, \begin{align*} {\mathbb P}(\text{$A$ is singular in $\mathbb R$})\leq n^{-\mathfrak{d}}, \end{align*} for $n$ sufficiently large. This answers an open problem by Frieze [12] and Vu [28,19]. The key idea is to study the singularity probability of adjacency matrices over a finite field ${\mathbb F}_p$. The proof combines a local central limit theorem and a large deviation estimate.