Evra, Feigon, Maurischat, and Parzanchevski (2023) introduced a biregular extension of Cayley graphs. In this paper, we reformulate their definition and provide some basic properties. We also show how these Cayley incidence graphs relate to various notions of Cayley hypergraphs. We further establish connections between Cayley incidence graphs and certain geometric and combinatorial structures, including coset geometries, difference sets and cages.