A bipartite graph $G=(V,E)$ with $V=V_1\cup V_2$ is biregular if all the vertices of each stable set, $V_1$ and $V_2$, have the same degree, $r$ and $s$, respectively. This paper studies difference sets derived from both Abelian and non-Abelian groups. From them, we propose some constructions of bipartite biregular graphs with diameter $d=3$ and asymptotically optimal order for given degrees $r$ and $s$. Moreover, we find some biMoore graphs, that is, bipartite biregular graphs that attain the Moore bound.