The ferromagnetic Ising model on an $n\times n$ square lattice region $Λ$ with mixed boundary conditions can exhibit a phase transition as temperature varies. For this spin system, if we fix the spins on the top and bottom sides of the square to be $+$ and the left and right sides to be~$-$, a standard Peierls argument based on energy shows that below some critical temperature~$t_c$, any local Markov chain $\mathcal{M}$ requires time exponential in $n$ to mix. Spin glasses are magnetic alloys that generalize the Ising model by specifying the strength of nearest neighbor interactions on the lattice, including whether they are ferromagnetic or antiferromagnetic. Whenever a face of the lattice is bounded by an odd number of edges with ferromagnetic interactions, the face is considered {\it frustrated} because the local competing objectives cannot be simultaneously satisfied. We consider spin glasses with exactly four well-separated frustrated faces that are symmetric around the center of the lattice region under $90$ degree rotations. We show that local Markov chains require exponential time for all spin glasses in this class. This argument extends to the ferromagnetic Ising model with mixed boundary conditions described above, which behaves like spin glasses with frustrated faces on the boundary. The standard Peierls argument breaks down when the frustrated faces are on the interior of $Λ$ and yields weaker results when they are on the boundary of $Λ$ but not near the corners. We show that there is a universal temperature $T$ below which $\mathcal{M}$ will be slow for all spin glasses with four well-separated frustrated faces. Our argument shows that there is an exponentially small cut indicated by the {\it free energy}, carefully exploiting both entropy and energy to establish a small bottleneck in the state space to establish slow mixing.