Semantic communication (SemCom) has emerged as a transformative paradigm for efficient information transmission by emphasizing the exchange of task-relevant meaning rather than raw data. While diffusion-based SemCom models have demonstrated remarkable generative capabilities, existing studies predominantly focus on point-to-point links, overlooking the potential of multi-user (MU) cooperation in MU wireless environments. To address this limitation, we propose a Cooperative Time-Division Diffusion (CTD-Diff) framework. Unlike traditional approaches that view channel noise solely as a detriment, our framework innovatively integrates the noisy wireless transmission process directly into the forward diffusion chain. Specifically, we design a multi-user cooperation mechanism based on Time-Division Multiple Access (TDMA), where idle users overhearing the active transmitter act as semantic collaborators. To maximize the signal fidelity, the receiver employs direct signal aggregation to fuse the direct signal with cooperative copies. This aggregated noisy semantic representation serves as the condition for the reverse diffusion process, allowing the receiver to reconstruct high-fidelity data by mitigating the cumulative channel distortions. By effectively converting physical channel noise into diffusion noise, the proposed method significantly enhances the transmission reliability. Extensive experiments demonstrate that CTD-Diff outperforms various baselines regarding the reconstruction accuracy and the perceptual quality, particularly under challenging low signal-to-noise ratio (SNR) conditions.