Distance-regular graphs are a class of regualr graphs with pretty combinatorial symmetry. In 2007, Miklavič and Potočnik proposed the problem of charaterizing distance-regular Cayley graphs, which can be viewed as a natural extension of the problem of characterizing strongly-regular Cayley graphs (or equivalently, regular partial difference sets). In this paper, we provide a partial characterization for distance-regular Cayley graphs over semi-dihedral groups and pseudo-semi-dihedral groups, both of which are $2$-groups with a cyclic subgroup of index $2$.