Holevo's just-as-good fidelity is a similarity measure for quantum states that has found several applications. One of its critical properties is that it obeys a data processing inequality: the measure does not decrease under the action of a quantum channel on the underlying states. In this paper, I prove a refinement of this data processing inequality that includes an additional term related to recoverability. That is, if the increase in the measure is small after the action of a partial trace, then one of the states can be nearly recovered by the Petz recovery channel, while the other state is perfectly recovered by the same channel. The refinement is given in terms of the trace distance of one of the states to its recovered version and also depends on the minimum eigenvalue of the other state. As such, the refinement is universal, in the sense that the recovery channel depends only on one of the states, and it is explicit, given by the Petz recovery channel. The appendix contains a generalization of the aforementioned result to arbitrary quantum channels.