The toughness $τ(G)=\mathrm{min}\{\frac{|S|}{c(G-S)}: S~\mbox{is a cut set of vertices in}~G\}$ for $G\ncong K_n.$ The concept of toughness initially proposed by Chv$\mathrm{\acute{a}}$tal in 1973, which serves as a simple way to measure how tightly various pieces of a graph hold together. A graph $G$ is called $t$-tough if $τ(G)\geq t.$ It is very interesting to investigate the relations between toughness and eigenvalues of graphs. Fan, Lin and Lu [European J. Combin. 110 (2023) 103701] provided sufficient conditions in terms of the spectral radius for a graph to be 1-tough with minimum degree $δ$ and $t$-tough with $t\geq 1$ being an integer, respectively. By using some typical distance spectral techniques and structural analysis, we in this paper present sufficient conditions based on the distance spectral radius to guarantee a graph to be 1-tough with minimum degree $δ.$ Moreover, we also prove sufficient conditions with respect to the distance spectral radius for a graph to be $t$-tough, where $t$ or $\frac{1}{t}$ is a positive integer.