A quantum Latin square is an $n \times n$ array of unit vectors where each row and column forms an orthonormal basis of a fixed complex vector space. We introduce the notion of $(G,G')$-invariant quantum Latin squares for finite groups $G$ and $G'$. These are quantum Latin squares with rows and columns indexed by $G$ and $G'$ respectively such that the inner product of the $a,b$-entry with the $c,d$-entry depends only on $a^{-1}c \in G$ and $b^{-1}d \in G'$. This definition is motivated by the notion of group invariant bijective correlations introduced in [Roberson \& Schmidt (2020)], and every group invariant quantum Latin square produces a group invariant bijective correlation, though the converse does not hold. In this work we investigate these group invariant quantum Latin squares and their corresponding correlations. Our main result is that, up to applying a global isometry to every vector in a $(G,G')$-invariant quantum Latin square, there is a natural bijection between these objects and trace and conjugate transpose preserving isomorphisms between the group algebras of $G$ and $G'$. This in particular proves that a $(G,G')$-invariant quantum Latin square exists if and only if the multisets of degrees of irreducible representations are equal for $G$ and $G'$. Another motivation for this line of work is that whenever Cayley graphs for groups $G$ and $G'$ are quantum isomorphic, then there is a $(G,G')$-invariant quantum correlation witnessing this, and thus it suffices to consider such correlations when searching for quantum isomorphic Cayley graphs. Given a group invariant quantum correlation, we show how to construct all pairs of graphs for which it gives a quantum isomorphism.