Let $G$ be a finite, undirected $d$-regular graph and $A(G)$ its normalized adjacency matrix, with eigenvalues $1 = λ_1(A)\geq \dots \ge λ_n \ge -1$. It is a classical fact that $λ_n = -1$ if and only if $G$ is bipartite. Our main result provides a quantitative separation of $λ_n$ from $-1$ in the case of Cayley graphs, in terms of their expansion. Denoting $h_{out}$ by the (outer boundary) vertex expansion of $G$, we show that if $G$ is a non-bipartite Cayley graph (constructed using a group and a symmetric generating set of size $d$) then $λ_n \ge -1 + ch_{out}^2/d^2\,,$ for $c$ an absolute constant. We exhibit graphs for which this result is tight up to a factor depending on $d$. This improves upon a recent result by Biswas and Saha who showed $λ_n \ge -1 + h_{out}^4/(2^9d^8)\,.$ We also note that such a result could not be true for general non-bipartite graphs.