A dissociation set of a graph is a set of vertices which induces a subgraph with maximum degree less than or equal to one. The dissociation number of a graph is the maximum cardinality of its dissociation sets. In this paper, we study the connected graphs of order $n$ with a given dissociation number that attains the minimum spectral radius. We characterize these graphs when the dissociation number is in $\{n-1,~n-2,~\lceil2n/3\rceil,~\lfloor2n/3\rfloor,~2\}$. We also prove that these graphs are trees when the dissociation number is larger than $\lceil {2n}/{3}\rceil$.