As the number of antennas in frequency-division duplex (FDD) multiple-input multiple-output (MIMO) systems increases, acquiring channel state information (CSI) becomes increasingly challenging due to limited spectral resources and feedback overhead. In this paper, we investigate the impact of the feedback channel on CSI feedback in a multi-user MIMO orthogonal frequency-division multiplexing (OFDM) scenario, where the received downlink pilot signal is directly utilized as the source for CSI feedback in a joint design with CSI feedback and precoding. Considering the influence of the feedback channel, we propose an end-to-end joint CSI estimation-feedback-precoding network based on a deep joint source-channel coding architecture with an adaptive number of users. Experimental results demonstrate that, under the same feedback and CSI estimation overheads, the proposed joint multi-module end-to-end network achieves a higher multi-user downlink spectral efficiency than traditional algorithms based on separate architecture and partially separated artificial intelligence-based network architectures under comparable channel quality. Furthermore, compared to conventional separate architecture, the proposed network architecture with joint architecture reduces the computational burden and model storage overhead at the UE side, facilitating the deployment of low-overhead multi-module joint architectures in practice. Meanwhile, the network designed at the BS achieves user-number adaptability without increasing the number of trainable parameters, thereby reducing both model storage and distribution overhead by requiring only a single set of parameters for different numbers of users. While slightly increasing storage requirements at the base station, it reduces computational complexity and precoding design delay, effectively reducing the effects of channel aging challenges.