The concept of matrix rigidity was first introduced by Valiant in 1977. Roughly speaking, a matrix is rigid if its rank cannot be reduced significantly by changing a small number of entries. There has been extensive interest in rigid matrices as Valiant showed in his MFCS'77 paper that rigidity can be used to prove arithmetic circuit lower bounds. In a surprising result, Alman and Williams (FOCS'19) showed that the (real valued) Hadamard matrix, which was conjectured to be rigid, is actually not very rigid. This line of work was extended by Dvir and Edelman (\emph{Theory of Computing}, 2019) to a family of matrices related to the Hadamard matrix, but over finite fields. In our work, we take another step in this direction and show that for any abelian group $G$ and function $f:G \rightarrow \mathbb{C}$, the matrix given by $M_{xy} = f(x - y)$ for $x,y \in G$ is not rigid. In particular, we get that complex valued Fourier matrices, circulant matrices, and Toeplitz matrices are all not rigid and cannot be used to carry out Valiant's approach to proving circuit lower bounds. Our results also hold when we consider matrices over a fixed finite field instead of the complex numbers. This complements a recent result of Goldreich and Tal (\emph{Comp. Complexity}, 2018) who showed that Toeplitz matrices are nontrivially rigid (but not enough for Valiant's method). Our work differs from previous non-rigidity results in that those works considered matrices whose underlying group of symmetries was of the form $\mathbb{F}_p^n$ with $p$ fixed and $n$ tending to infinity, while in the families of matrices we study, the underlying group of symmetries can be any abelian group and, in particular, the cyclic group $\mathbb{Z}_N$, which has very different structure. Our results also suggest natural new candidates for rigidity in the form of matrices whose symmetry groups are highly non-abelian.