The Difference graph $\mathcal{D}(G)$ of a finite group $G$ is the difference of the enhanced power graph $\mathcal{P}_{E}(G)$ and the power graph $\mathcal{P}(G)$ with all the isolated vertices removed. In this paper, we characterize the vertex set of the difference graph of finite nilpotent groups and obtain its cardinality. Consequently, we obtain the metric dimension of the difference graph of finite nilpotent groups. Moreover, this paper determines the metric dimension of the difference graphs of certain non-nilpotent groups, namely: dihedral groups, the generalized quaternion groups, and the semi-dihedral groups.