The power graph of a finite group $G$ is a simple undirected graph with vertex set $G$ and two vertices are adjacent if one is a power of the other. The enhanced power graph of a finite group $G$ is a simple undirected graph whose vertex set is the group $G$ and two vertices $a$ and $b$ are adjacent if there exists $c \in G$ such that both $a$ and $b$ are powers of $c$. In this paper, we investigate the difference graph $\mathcal{D}(G)$ of a finite group $G$, which is the difference of the enhanced power graph and the power graph of $G$ with all isolated vertices removed. We study the difference graphs of finite groups with forbidden subgraphs among other results. We first characterize an arbitrary finite group $G$ such that $\mathcal{D}(G)$ is a chordal graph, star graph, dominatable, threshold graph, and split graph. From this, we conclude that the latter four graph classes are equivalent for $\mathcal{D}(G)$. By applying these results, we classify the nilpotent groups $G$ such that $\mathcal{D}(G)$ belong to the aforementioned five graph classes. This shows that all these graph classes are equivalent for $\mathcal{D}(G)$ when $G$ is nilpotent. Then, we characterize the nilpotent groups whose difference graphs are cograph, bipartite, Eulerian, planar, and outerplanar. Finally, we consider the difference graph of non-nilpotent groups and determine the values of $n$ such that the difference graphs of the symmetric group $S_n$ and alternating group $A_n$ are cograph, chordal, split, and threshold.