If $G$ is a group and $S$ a generating set, $G$ canonically embeds into the automorphism group of its Cayley graph and it is natural to try to minimize, over all generating sets, the index of this inclusion. This infimum is called the Cayley index of the group. In a recent series of works, we have characterized the infinite finitely generated groups with Cayley index $1$. We complement this characterization by showing that the Cayley index is $2$ in the remaining cases and is attained for a finite generating set.