Abstract:A data mixture refers to how different data sources are combined to train large language models, and selecting an effective mixture is crucial for optimal downstream performance. Existing methods either conduct costly searches directly on the target model or rely on mixture scaling laws that fail to extrapolate well to large model sizes. We address these limitations by introducing a compute-efficient pipeline for data mixture scaling. First, we propose CAMEL, a capacity-aware mixture law that models validation loss with the nonlinear interplay between model size and mixture. We also introduce a loss-to-benchmark prediction law that estimates benchmark accuracy from validation loss, enabling end-to-end performance prediction for the target model. Next, we study how to allocate a fixed compute budget across model scales to fit the law and reduce prediction error. Finally, we apply our method to Mixture-of-Experts models with up to 7B-A150M parameters to fit the law, and verify the optimal mixture derived from the law by extrapolating to a 55B-A1.2B target model. Compared to prior methods, we reduce mixture optimization costs by 50\% and improves downstream benchmark performance by up to 3\%.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2603.08022 [cs.LG] |
| (or arXiv:2603.08022v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2603.08022 arXiv-issued DOI via DataCite |
From: Jingwei Li [view email]
[v1]
Mon, 9 Mar 2026 06:58:00 UTC (2,413 KB)
[v2]
Wed, 6 May 2026 03:36:45 UTC (2,414 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。