In this work we extend the recently introduced group-theoretical approach to moment-cumulant relations in non-commutative probability theory to the notion of conditionally free cumulants. This approach is based on a particular combinatorial Hopf algebra which may be characterised as a non-cocommutative generalisation of the classical unshuffle Hopf algebra. Central to our work is the resulting non-commutative shuffle algebra structure on the graded dual. It implies an extension of the classical relation between the group of Hopf algebra characters and its Lie algebra of infinitesimal characters and, among others, the appearance of new forms of adjoint actions of the group on its Lie algebra which happens to play a key role in the new algebraic understanding of conditionally free cumulants.