End-to-end (E2E) learning has recently been proposed to jointly design the modulator and symbol detector by using deep neural networks (DNNs). However, existing schemes lack sufficient capability to cancel multi-user interference (MUI) in uplink multi-user multiple-input multiple-output (MU-MIMO) systems. In this paper, we propose a graph neural network (GNN)-based E2E learning scheme that employs a GNN-based modulator to generate learned constellation points, and a GNN-based detector to cancel MUI. They are jointly optimized to minimize the symbol error rate (SER) performance loss. Simulation results demonstrate that the proposed E2E outperforms existing schemes with a predefined modulator. Specifically, it achieves an approximate 2 dB gain in a high MUI environment and surpasses even the maximum-likelihood (ML) detector in a low MUI condition.