Abstract:Hadamard transforms have become a key tool for stabilizing low-precision training, but existing methods apply them uniformly across tensors and computation paths. We show that this one-size-fits-all strategy is inherently limited: Hadamard smoothing reduces quantization error only when its direction is properly aligned with the operand's outlier structure. Through a systematic study of weights, activations, and gradients in LLM training, we identify three stable outlier patterns, Row-wise, Column-wise, and None, and show that each outlier pattern pair in matrix multiplication requires a distinct transform or outlier-handling strategy. We propose AdaHOP, Adaptive Hadamard transform with Outlier-Pattern-aware strategy, which applies Inner Hadamard Transform (IHT) when inner-dimension mixing properly suppresses the operands' outliers, and selectively applies Outlier Extraction (OE) that extracts dominant outlier rows or columns into a high-precision path when it does not. With fused, hardware-aware Triton kernels, AdaHOP enables training from scratch at MXFP4 precision with BF16-level quality, while achieving up to 3.6X memory compression, 1.46X end-to-end training speedup over BF16.
| Comments: | 21 pages, 10 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.02525 [cs.LG] |
| (or arXiv:2604.02525v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.02525 arXiv-issued DOI via DataCite |
From: Seonggon Kim [view email]
[v1]
Thu, 2 Apr 2026 21:24:15 UTC (11,266 KB)
[v2]
Thu, 7 May 2026 21:33:53 UTC (7,643 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。