In 2003, van Dam and Haemers posed a fundamental question in spectral graph theory: does there exist a ``sensible'' matrix whose spectrum determines a random graph up to isomorphism? This paper introduces the class of {\em natural graph matrices}, which are matrices defined by applying a fixed sequence of elementary operations to the adjacency matrix. This class includes many standard matrices such as the adjacency matrix, the Seidel matrix, the Laplacian matrix, and the distance matrix. We give an affirmative answer to the question of van Dam and Haemers by proving the existence of a natural graph matrix whose spectrum determines random graphs up to isomorphism. The proof introduces a new algebraic framework called {\em double algebras}, which provides a simple sufficient condition for spectral determination. This sufficient condition is then shown to hold for random graphs.