In enumerative geometry, Virasoro constraints were first conjectured in Gromov-Witten theory with many new recent developments in the sheaf theoretic context. In this paper, we rephrase the sheaf-theoretic Virasoro constraints in terms of primary states coming from a natural conformal vector in Joyce's vertex algebra. This shows that Virasoro constraints are preserved under wall-crossing. As an application, we prove the conjectural Virasoro constraints for moduli spaces of torsion-free sheaves on any curve and on surfaces with only $(p,p)$ cohomology classes by reducing the statements to the rank 1 case.