A skew-Hadamard difference set (SHDS) in a finite group $G$ is a classical combinatorial object with deep connections to design theory, coding theory, group theory, and the construction of Hadamard matrices. Even though the abelian case has been extensively studied -- with strong structural constraints known, such as the necessity of $G$ being a $p$-group for some prime $p \equiv 3 \pmod{4}$ -- there are still some open questions regarding existence of SHDSs for the abelian case. The non-abelian case remains largely unexplored, despite the known existence of non-abelian SHDSs. In this paper, we establish new necessary conditions on the order and structure of a finite group $G$ that admits an SHDS. These results provide the first general structural restrictions for SHDSs in non-abelian groups. In particular, we prove that if a group $G$ is nilpotent and admits an SHDS, then $G$ is a $p$-group. Our method makes use of the structure of the rational group algebra, and completely avoids the use of the group characters.