Semantic communication (SemCom) is accelerating its momentum to catch up with the massive increase in users' demands in both quantity and quality, with the assistance of advanced deep learning (DL) techniques. Specifically, SemCom can actively embed the semantic meaning of the data into the transmission process, while eliminating statistical redundancy to preserve bandwidth resources for other users. Therefore, the transmitter encodes the message in the most concise way, while the receiver tries to interpret the message with the DL model and its knowledge of the transmitter's intended meaning. Most existing works only consider one transmitter and one receiver, which limits their ability to address the diversity in users' models and capabilities. Therefore, in this paper, we propose a multi-user semantic communication system where each user is equipped with a distinct DL-based joint source-channel decoder architecture, reflecting the diversity in computing capacity. The challenging issue with the proposed system is the catastrophic forgetting property of neural networks, where the DL-based encoder fails to encode the data for the previous user when being trained with a new user. To address this, we propose an anchor decoder with an architecture that is symmetric to the encoder. The symmetric decoder has the same computational capacity as the encoder, providing feedback that aligns with the encoder's extraction capabilities and enhances optimization efficiency. The parameters of the optimized encoder are then frozen and used to train decoders for various users, aligning them with the encoder outputs. Finally, we conduct a series of simulation experiments to validate the proposed framework against other benchmarks.