The conformal loop ensemble (CLE) is the canonical conformally invariant probability measure on non-crossing loops in a simply connected domain in $\mathbb C$ and is indexed by a parameter $κ\in (8/3,8)$. We consider CLE$_κ$ on the whole-plane in the regime in which the loops are self-intersecting ($κ\in (4,8)$) and show that it is invariant under the inversion map $z \mapsto 1/z$. This shows that whole-plane CLE$_κ$ for $κ\in (4,8)$ defines a conformally invariant measure on loops on the Riemann sphere. The analogous statement in the regime in which the loops are simple ($κ\in (8/3,4]$) was proven by Kemppainen and Werner and together with the present work covers the entire range $κ\in (8/3,8)$ for which CLE$_κ$ is defined. As an intermediate step in the proof, we show that CLE$_κ$ for $κ\in (4,8)$ on an annulus, with any specified number of inner-boundary-surrounding loops, is well-defined and conformally invariant.