Sparse code multiple access (SCMA) is a promising non-orthogonal multiple access scheme for enabling massive connectivity in next generation wireless networks. However, current SCMA codebooks are designed with the same size, leading to inflexibility of user grouping and supporting diverse data rates. To address this issue, we propose a variable modulation SCMA (VM-SCMA) that allows users to employ codebooks with different modulation orders. To guide the VM-SCMA design, a VM matrix (VMM) that assigns modulation orders based on the SCMA factor graph is first introduced. We formulate the VM-SCMA design using the proposed average inverse product distance and the asymptotic upper bound of sum-rate, and jointly optimize the VMM, VM codebooks, power and codebook allocations. The proposed VM-SCMA not only enables diverse date rates but also supports different modulation order combinations for each rate. Leveraging these distinct advantages, we further propose an adaptive VM-SCMA (AVM-SCMA) scheme which adaptively selects the rate and the corresponding VM codebooks to adapt to the users' channel conditions by maximizing the proposed effective throughput. Simulation results show that the overall designs are able to simultaneously achieve a high-level system flexibility, enhanced error rate results, and significantly improved throughput performance, when compared to conventional SCMA schemes.