Paley graphs form a nice link between the distribution of quadratic residues and graph theory. These graphs possess remarkable properties which make them useful in several branches of mathematics. Classically, for each prime number $p$ we can construct the corresponding Paley graph using quadratic and non-quadratic residues modulo $p$. Therefore, Paley graphs are naturally associated with the Legendre symbol at $p$ which is a quadratic Dirichlet character of conductor $p$. In this article, we introduce the generalized Paley graphs. These are graphs that are associated with a general quadratic Dirichlet character. We will then provide some of their basic properties. In particular, we describe their spectrum explicitly. We then use those generalized Paley graphs to construct some new families of Ramanujan graphs. Finally, using special values of $L$-functions, we provide an effective upper bound for their Cheeger number.