In this paper we introduce a new notion of a sequence of symmetry groups of an infinite word. Given a subgroup $G_n$ of the symmetric group $S_n$, it acts on the set of finite words of length $n$ by permutation. We associate to an infinite word $w$ a sequence $(G_n(w))_{n\geq 1}$ of its symmetry groups: For each $n$, a symmetry group of $w$ is a subgroup $G_n(w)$ of the symmetric group $S_n$ such that $g(v)$ is a factor of $w$ for each permutation $g \in G_n(w)$ and each factor $v$ of length $n$ of $w$. We study general properties of the symmetry groups of infinite words and characterize the sequences of symmetry groups of several families of infinite words. We show that for each subgroup $G$ of $S_n$ there exists an infinite word $w$ with $G_n(w)=G$. On the other hand, the structure of possible sequences $(G_n(w))_{n\geq 1}$ is quite restrictive: we show that they cannot contain for each order $n$ certain cycles, transpositions and some other permutations. The sequences of symmetry groups can also characterize a generalized periodicity property. We prove that symmetry groups of Sturmian words and more generally Arnoux-Rauzy words are of order two for large enough $n$; on the other hand, symmetry groups of certain Toeplitz words have exponential growth.