It was predicted and recently experimentally confirmed that systems with microscopically reversible dynamics in locally quadratic potentials warm up faster than they cool down. This thermal relaxation asymmetry challenged the local-equilibrium paradigm valid near equilibrium. Because the intuition and proof hinged on the dynamics obeying detailed balance, it was not clear whether the asymmetry persists in systems with irreversible dynamics. To fill this gap, we here prove the relaxation asymmetry for systems driven out of equilibrium by a general linear drift. The asymmetry persists due to a non-trivial isomorphism between driven and reversible processes. Moreover, rotational motions emerge that, strikingly, occur in opposite directions during heating and cooling. This highlights that noisy systems do not relax by passing through local equilibria.