The power graph and the enhanced power graph of a group $\mathbf G$ are simple graphs with vertex set $G$; two elements of $G$ are adjacent in the power graph if one of them is a power of the other, and they are adjacent in the enhanced power graph if they generate a cyclic subgroup. The difference graph of a group $\mathbf G$, denoted by $\mathcal D(\mathbf G)$, is the difference of the enhanced power graph and the power graph of group $\mathbf G$ with all the isolated vertices removed. In this paper, we prove that, if a pair of finite abelian groups of order divisible by at most two primes have isomorphic difference graphs, then they are isomorphic.