Let $G$ be a finite group with $|G|\geq 4$ and $S$ be a subset of $G$. Given an automorphism $σ$ of $G$, the twisted Cayley graph $C(G, S)^σ$ (resp. the twisted Cayley sum graph $C_Σ(G, S)^σ$) is defined as the graph having $G$ as its set of vertices and the adjacent vertices of a vertex $g\in G$ are of the form $σ(gs)$ (resp. $σ(g^{-1} s)$) for some $s\in S$. If the twisted Cayley graph $C(G, S)^σ$ is undirected and connected, then we prove that the nontrivial spectrum of its normalised adjacency operator is bounded away from $-1$ and this bound depends only on its degree, the order of $σ$ and the vertex Cheeger constant of $C(G, S)^σ$. Moreover, if the twisted Cayley sum graph $C_Σ(G, S)^σ$ is undirected and connected, then we prove that the nontrivial spectrum of its normalised adjacency operator is bounded away from $-1$ and this bound depends only on its degree and the vertex Cheeger constant of $C_Σ(G, S)^σ$. We also study these twisted graphs with respect to anti-automorphisms, and obtain similar results. Further, we prove an analogous result for the Schreier graphs satisfying certain conditions.