We introduce coloring groups, which are permutation groups obtained from a proper edge coloring of a graph. These groups generalize the generalized toggle groups of Striker (which themselves generalize the toggle groups introduced by Cameron and Fon-der-Flaass). We present some general results connecting the structure of a coloring group to the structure of its graph coloring, providing graph-theoretic characterizations of the centralizer and primitivity of a coloring group. We apply these results particularly to generalized toggle groups arising from trees as well as coloring groups arising from the independence posets introduced by Thomas and Williams.