In the domain of Specific Emitter Identification (SEI), it is recognized that transmitters can be distinguished through the impairments of their radio frequency front-end, commonly referred to as Radio Frequency Fingerprint (RFF) features. However, modulation schemes can be deliberately coupled into signal-level data to confound RFF information, often resulting in high susceptibility to failure in SEI. In this paper, we propose a domain-invariant feature oriented Margin Disparity Discrepancy (MDD) approach to enhance SEI's robustness in rapidly modulation-varying environments. First, we establish an upper bound for the difference between modulation domains and define the loss function accordingly. Then, we design an adversarial network framework incorporating MDD to align variable modulation features. Finally, We conducted experiments utilizing 7 HackRF-One transmitters, emitting 11 types of signals with analog and digital modulations. Numerical results indicate that our approach achieves an average improvement of over 20\% in accuracy compared to classical SEI methods and outperforms other UDA techniques. Codes are available at https://github.com/ZhangYezhuo/MDD-SEI.