Group recommendation aims to recommend tailored items to groups of users, where the key challenge is modeling a consensus that reflects member preferences. Although several existing deep learning models have achieved performance improvements, they still fail to capture consensus in various aspects: (1) Capturing consensus in small-group (2~5 members) recommendation systems, which align more closely with real-world scenarios, remains a significant challenge; (2) Most existing models significantly enhance the overall group performance but struggle with balancing individual and group performance. To address these issues, we propose Capturing Consensus with Contrastive Learning in Group Recommendation (C$^3$), which focuses on exploring the consensus behind group decision-making. A Transformer encoder is used to learn both group and user representations, and contrastive learning mitigates overfitting for users with many interactions, yielding more robust group representations. Experiments on four public datasets demonstrate that C$^3$ significantly outperforms state-of-the-art baselines in both user and group recommendation tasks.