Given a finite group $G$, the difference graph of $G$, denoted by $\mathcal{D}(G)$, is the difference of the enhanced power graph of $G$ and the power graph of $G$, with all isolated vertices removed. This paper mainly studies the dominating sets of the difference graph of a finite group. In particular, we prove that the diameter of the difference graph of a nilpotent group has an upper bound of $4$. Furthermore, we generalize and refine the result by Biswas et al. by classifying all nilpotent groups whose difference graph has diameter $k$, for each $k\le 4$.