In finite group theory, studying the prime graph of a group has been an important topic for almost the past half-century. Recently, prime graphs of solvable groups have been characterized in graph theoretical terms only. This now allows the study of these graphs without any knowledge of the group theoretical background. In this paper we study prime graphs from a linear algebra angle and focus on the class of minimally connected prime graphs introduced in earlier work on the subject. As our main results, we determine the determinants of the adjacency matrices and the spectra of some important families of these graphs.