Let $m$ be a positive integer. Brualdi and Hoffman proposed the problem to determine the (connected) graphs with maximum spectral radius in a given graph class and they posed a conjecture for the class of graphs with given size $m$. After partial results due to Friedland and Stanley, Rowlinson completely confirmed the conjecture. The distance spectral radius of a connected graph is the largest eigenvalue of its distance matrix. We investigate the problem to determine the connected graphs with minimum distance spectral radius in the class of graphs with size $m$. Given $m$, there is exactly one positive integer $n$ such that ${n-1\choose 2} <m\leq {n\choose 2}$. We establish some structural properties of the extremal graphs for all $m$ and solve the problem for ${n-1\choose 2}+\max\{\frac{n-6}{2},1\}\le m\leq {n\choose 2}$. We give a conjecture for the remaining case. To prove the main results, we also determine the the complements of forests of fixed order with large and small distance spectral radius.