The fixing number of a graph $G$ is the smallest cardinality of a set of vertices $S$ such that only the trivial automorphism of $G$ fixes every vertex in $S$. The fixing set of a group $Γ$ is the set of all fixing numbers of finite graphs with automorphism group $Γ$. Several authors have studied the distinguishing number of a graph, the smallest number of labels needed to label $G$ so that the automorphism group of the labeled graph is trivial. The fixing number can be thought of as a variation of the distinguishing number in which every label may be used only once, and not every vertex need be labeled. We characterize the fixing sets of finite abelian groups, and investigate the fixing sets of symmetric groups.