Brualdi and Hoffman proposed a well-known problem of determining the graph with maximum adjacency spectral radius among all graphs with given size $m$. Early work by Friedland and Stanley addressed some specific cases. This problem was later completely solved by Rowlinson and recently revisited by Cheng and Weng. Pioneering work on the distance matrix was carried out by Graham and Pollak, as well as by Graham and Lovász. The distance spectral radius $ρ(G)$ of a connected graph $G$ is the largest eigenvalue of its distance matrix. In this paper, we completely solve the problem of characterizing the connected graph with minimum distance spectral radius among all graphs with size $m$. Let $\mathcal{G}(m)$ be the class of connected graphs with $m$ edges. For every $m \ge 3$, let $n$ be the unique integer satisfying $\binom{n-1}{2} < m \le \binom{n}{2}$, and we write $m = \binom{n-1}{2} + s$ with $1 \le s \le n-1$. Recently, Lin and Zhou [Adv. in Appl. Math. 173 (2026)] investigated the graph in $\mathcal{G}(m)$ that minimizes $ρ(G)$ in the range $s \ge \max\{ \frac{n-6}{2}, 1\}$. However, the problem is much more difficult in the remaining range $1 \le s \le \frac{n-7}{2}$, and they conjectured that the unique minimizer is $\overline{P_{n,s+1}}$, the complement of a balanced disjoint union of paths. Using novel matrix analysis, we solve this conjecture in the affirmative. Moreover, we provide a new unified proof for the entire range $1 \le s \le n-1$. The key ingredients in our proof argument include an innovative comparison principle for the distance spectral radius, an increment analysis of $Φ$-functions on paths and cycles, an argument for balancing path lengths, and a walk enumeration technique via the Neumann series.