We study finite groups that occur as combinatorial automorphism groups or geometric symmetry groups of convex polytopes. When $Γ$ is a subgroup of the combinatorial automorphism group of a convex $d$-polytope, $d\geq 3$, then there exists a convex $d$-polytope related to the original polytope with combinatorial automorphism group exactly $Γ$. When $Γ$ is a subgroup of the geometric symmetry group of a convex $d$-polytope, $d\geq 3$, then there exists a convex $d$-polytope related to the original polytope with both geometric symmetry group and combinatorial automorphism group exactly $Γ$. These symmetry-breaking results then are applied to show that for every abelian group $Γ$ of even order and every involution $σ$ of $Γ$, there is a centrally symmetric convex polytope with geometric symmetry group $Γ$ such that $σ$ corresponds to the central symmetry.