In 1988, Duval introduced the concept of directed strongly regular graphs, which can be viewed as a directed graph version of strongly regular graphs. Such directed graphs have similar structural and algebraic properties to strongly regular graphs. In the past three decades, it was found that Cayley graphs, especially those over dihedral groups, play a key role in the construction of directed strongly regular graphs. In this paper, we focus on the characterization of directed strongly regular Cayley graphs over more general groups. Let $G$ be a non-abelian group with an abelian subgroup of index $2$. We give some necessary conditions for a Cayley graph over $G$ to be directed strongly regular, and characterize the directed strongly regular Cayley graphs over $G$ satisfying specified conditions. This extends some previous results of He and Zhang (2019).