The power graph of a group $G$ is a simple and undirected graph with vertex set $G$ and two distinct vertices are adjacent if one is a power of the other. In this article, we characterize (non-cyclic) finite groups of prime exponent and finite elementary abelian $2$-groups (of rank at least $2$) in terms of their power graphs.