The enhanced power graph of a group $G$ is the graph $\mathcal{G}_E(G)$ with vertex set $G$ and edge set $ \{(u,v): u, v \in \langle w \rangle,~\mbox{for some}~ w \in G\}$. In this paper, we compute the spectrum of the distance matrix of the enhanced power graph of non-abelian groups of order $pq$, dihedral groups, dicyclic groups, elementary abelian groups $\mathrm{El}(p^n)$ and the non-cyclic abelian groups $\mathrm{El}(p^n)\times\mathrm{El}(q^m)$ and $\mathrm{El}(p^n)\times \mathbb{Z}_m$, where $p$ and $q$ are distinct primes. For the non-cyclic abelian group $\mathrm{El}(p^n)\times \mathrm{El}(q^m)$, we also compute the spectrum of the adjacency matrix of its enhanced power graph and the spectrum of the adjacency and the distance matrix of its power graph.