We define $G$-cospectrality of two $G$-gain graphs $(Γ,ψ)$ and $(Γ',ψ')$, proving that it is a switching isomorphism invariant. When $G$ is a finite group, we prove that $G$-cospectrality is equivalent to cospectrality with respect to all unitary representations of $G$. Moreover, we show that two connected gain graphs are switching equivalent if and only if the gains of their closed walks centered at an arbitrary vertex $v$ can be simultaneously conjugated. In particular, the number of switching equivalence classes on an underlying graph $Γ$ with $n$ vertices and $m$ edges, is equal to the number of simultaneous conjugacy classes of the group $G^{m-n+1}$. We provide examples of $G$-cospectral non-switching isomorphic graphs and we prove that any gain graph on a cycle is determined by its $G$-spectrum. Moreover, we show that when $G$ is a finite cyclic group, the cospectrality with respect to a faithful irreducible representation implies the cospectrality with respect to any other faithful irreducible representation, and that the same assertion is false in general.