Many fault diagnosis methods of rotating machines are based on discriminative features extracted from signals collected from the key components such as bearings. However, under complex operating conditions, periodic impulsive characteristics in the signal related to weak fault information are often obscured by noise interference. Consequently, existing approaches struggle to learn interpretable fault-related features in such scenarios. This paper proposes a novel transformer framework (FE-MCFormer) to extract interpretable time-frequency features, with the aim of improving the fault detection accuracy and intrepretability of rotating machines under strong noise. First, a Fourier adaptive reconstruction embedding layer is introduced as a global information encoder in the model. Subsequently, a time-frequency fusion module is designed, further improve the model robustness and interpretability. The effectiveness of FE-MCFormer in machine fault diagnosis is validated through three case studies.