Let $G$ be a graph with vertex set $\{v_1,\dots,v_n\}$. The Seidel matrix of $G$ is an $ n\times n$ matrix whose diagonal entries are zero, $ij$-th entry is $-1$ if $v_i$ and $v_j$ are adjacent, and otherwise is $1$. The $p$-Seidel energy of the graph $G$ is defined as the sum of the absolute values of the $p$-th powers of all eigenvalues of the Seidel matrix of $G$ and introduced in [European Journal of Combinatorics, (86) (2020), 103078]. In this article, we characterize the graph that minimizes the $p$-Seidel energy among all graphs with fixed order $n$, for $p>2$. We also characterize the graph that maximizes the $p$-Seidel energy among all graphs with fixed order $n$, for $0<p<2$. In addition, for every $p>2$, we characterize the graph that minimizes the $p$-Seidel energy among all $r$-regular graphs with fixed order $n$, where $n$ is a prime power with $n\equiv 1\pmod 4$, $r=\frac{n-1}{2}$. For every $p>2$, we also characterize the graph that maximizes the $p$-Seidel energy among all $r$-regular graphs with fixed order $n=2r$. Finally, we pose several open problems concerning the $p$-Seidel energy for different values of $p$.