Consider the strata of primitive $k$-differentials on the Riemann sphere whose singularities, except for two, are poles of order divisible by $k$. The map that assigns to each $k$-differential the $k$-residues at these poles is a ramified cover of its image. Generalizing results known in the case of abelian differentials, we describe the ramification locus of this cover and provide a formula, involving the $k$-factorial function, for the cardinality of each fiber. We prove this formula using intersection calculations on the multi-scale compactification of the strata of $k$-differentials. In special cases, we also give alternative proofs using flat geometry. Finally, we present an application to cone spherical metrics with dihedral monodromy.