Given a finite abelian group $G$ and a subset $J\subset G$ with $0\in J$, let $D_{G}(J,N)$ be the maximum size of $A\subset G^{N}$ such that the difference set $A-A$ and $J^{N}$ have no non-trivial intersection. Recently, this extremal problem has been widely studied for different groups $G$ and subsets $J$. In this paper, we generalize and improve the relevant results by Alon and by Hegedűs by building a bridge between this problem and cyclotomic polynomials with the help of algebraic graph theory. In particular, we construct infinitely many non-trivial families of $G$ and $J$ for which the current known upper bounds on $D_{G}(J, N)$ can be improved exponentially.