We study bipartite distance-regular Cayley graphs with diameter three or four. We give sufficient conditions under which a bipartite Cayley graph can be constructed on the semidirect product of a group -- the part of this bipartite Cayley graph which contains the identity element -- and $\mathbb{Z}_{2}$. We apply this to the case of bipartite distance-regular Cayley graphs with diameter three, and consider cases where the sufficient conditions are not satisfied for some specific groups such as the dihedral group. We also extend a result by Miklavič and Potočnik that relates difference sets to bipartite distance-regular Cayley graphs with diameter three to the case of diameter four. This new case involves certain partial geometric difference sets and -- in the antipodal case -- relative difference sets.