Abstract:Bandwidth-constrained distributed reinforcement learning (RL) post-training of large language models is bottlenecked by two channels: weight synchronization from trainers to inference workers, and gradient or pseudo-gradient synchronization across trainers. We find that approximately 99% of per-step weight updates are invisible after the BF16 cast used by standard training and inference forward passes. We explain this sparsity by showing that, at typical RL post-training learning rates, Adam updates often fall below the local BF16 rounding threshold. We turn this observation into an algorithmic principle called compute-visible sparsification: transmit only updates that would change the next forward pass. PULSE (Precision-gated Updates for Low-precision Sparse Exchange) turns this principle into two communication algorithms: PULSESync sends lossless sparse BF16 weight patches from trainers to inference workers, and PULSELoCo sparsifies DiLoCo-style FP32 pseudo-gradient synchronization with error feedback. Over bandwidth-constrained commodity networks, PULSESync cuts weight-synchronization communication by over 100x while reconstructing trainer weights bit-identically. PULSELoCo matches DiLoCo across four models while reducing trainer-to-trainer communication by over 17x versus DiLoCo and over 100x versus DDP in the largest evaluated setting.
| Comments: | 40 pages, 19 figures, 14 tables |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2602.03839 [cs.LG] |
| (or arXiv:2602.03839v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.03839 arXiv-issued DOI via DataCite |
From: Erfan Miahi [view email]
[v1]
Tue, 3 Feb 2026 18:56:48 UTC (527 KB)
[v2]
Tue, 19 May 2026 16:03:06 UTC (1,001 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。