Massive multiple input and multiple output (MIMO) systems with orthogonal frequency division multiplexing (OFDM) are foundational for downlink multi-user (MU) communication in future wireless networks, for their ability to enhance spectral efficiency and support a large number of users simultaneously. However, high user density intensifies severe inter-user interference (IUI) and pilot overhead. Consequently, existing blind and semi-blind channel estimation (CE) and signal detection (SD) algorithms suffer performance degradation and increased complexity, especially when further challenged by frequency-selective channels and high-order modulation demands. To this end, this paper proposes a novel semi-blind joint channel estimation and signal detection (JCESD) method. Specifically, the proposed approach employs a hybrid precoding architecture to suppress IUI. Furthermore we formulate JCESD as a non-convex constellation fitting optimization exploiting constellation affine invariance. Few pilots are used to achieve coarse estimation for initialization and ambiguity resolution. For high-order modulations, a data augmentation mechanism utilizes the symmetry of quadrature amplitude modulation (QAM) constellations to increase the effective number of samples. To address frequency-selective channels, CE accuracy is then enhanced via an iterative refinement strategy that leverages improved SD results. Simulation results demonstrate an average throughput gain of 11\% over widely used pilot-based methods in MU scenarios, highlighting the proposed method's potential to improve spectral efficiency.