A dissociation set in a graph is a subset of vertices which induces a subgraph with maximum degree at most one. The dissociation number of a graph is the maximum cardinality of its dissociation sets. In this paper, we consider the $n$-vertex connected graphs with a given dissociation number that attain the minimum spectral radius. By using structure analysis and constructing difference equations, we characterize the extremal graphs with dissociation number $n-3$.