Let $Γ$ be a Cayley graph, or a Cayley sum graph, or a twisted Cayley graph, or a twisted Cayley sum graph, or a vertex-transitive graph. Denote the degree of $Γ$ by $d$, its edge Cheeger constant by $\mathfrak{h}_Γ$, and its vertex Cheeger constant by $h_Γ$. Assume that $Γ$ is undirected, non-bipartite. We prove that the edge bipartiteness constant of $Γ$ is $Ω({\mathfrak{h}_Γ}/{d})$, the vertex bipartiteness constant of $Γ$ is $Ω(h_Γ)$, and the smallest eigenvalue of the normalized adjacency operator of $Γ$ is $-1 + Ω({h_Γ^2}/{d^2})$. This answers in the affirmative a question of Moorman, Ralli and Tetali on the lower spectral gap of Cayley sum graphs.