In this paper, we discuss the adjacency matrices of finite undirected simple graphs over a finite prime field $\mathbb{F}_p$. We apply symmetric (row and column) elementary transformations to the adjacency matrix over $\mathbb{F}_p$ in order to get a direct sum decomposition by other adjacency matrices. In this paper, we give a complete description of the direct sum decomposition of the adjacency matrix of any graph over $\mathbb{F}_p$ for any odd prime $p$. Our key tool is quadratic residues of $\mathbb{F}_p$.