Let $Γ$ be an undirected and simple graph. A set $ S $ of vertices in $Γ$ is called a {cyclic vertex cutset} of $Γ$ if $Γ- S$ is disconnected and has at least two components containing cycles. If $Γ$ has a cyclic vertex cutset, then it is said to be {cyclically separable}. The {cyclic vertex connectivity} of $Γ$ is the minimum of cardinalities of the cyclic vertex cutsets of $Γ$. The {power graph} $\mathcal{P}(G)$ of a group $G$ is the undirected and simple graph whose vertices are the elements $G$ and two vertices are adjacent if one of them is the power of other in $G$. In this paper, we first characterize the finite $ p $-groups ($p$ is a prime number) whose power graphs are cyclically separable in terms of their maximal cyclic subgroups. Then we characterize the finite $ p $-groups whose power graphs have equal vertex connectivity and cyclic vertex connectivity.