The generalized distance spectral radius of a connected graph $G$ is the spectral radius of the generalized distance matrix of $G$, defined by $$D_α(G)=αTr(G)+(1-α)D(G), \;\;0\leα\le 1,$$ where $D(G)$ and $Tr(G)$ denote the distance matrix and diagonal matrix of the vertex transmissions of $G$, respectively. This paper characterizes the unique graph with minimum generalized distance spectral radius among the connected graphs with fixed chromatic number, which answers a question about the generalized distance spectral radius in spectral extremal theories. In addition, we also determine graphs with minimum generalized distance spectral radius among the $n$-vertex trees and unicyclic graphs, respectively. These results generalize some known results about distance spectral radius and distance signless Laplacian spectral radius of graphs.