Abstract:Deploying large language models to heterogeneous hardware is often constrained by memory, not compute. We introduce MCAP (Monte Carlo Activation Profiling), a load-time per-layer importance estimator that enables dynamic precision and memory placement decisions on the target device. MCAP produces a lightweight per-layer signal that drives both precision dispatch (W4A8 vs. W4A16) and residency tier (GPU, RAM, SSD), allowing a single set of weights to operate across diverse memory budgets. Our system, NVE, achieves 1.5-1.8x higher decode throughput than this http URL Q4_0 on NVIDIA T4 and enables models to run in memory regimes previously infeasible without modifying weights.
| Comments: | Code available at this https URL |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.21026 [cs.LG] |
| (or arXiv:2604.21026v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.21026 arXiv-issued DOI via DataCite (pending registration) |
From: Anurita Das [view email]
[v1]
Wed, 22 Apr 2026 19:18:30 UTC (2,332 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。