Abstract:In Transformer models, non-GEMM (non-General Matrix Multiplication) operations -- especially Softmax and Layer Normalization (LayerNorm) -- often dominate hardware cost due to their nonlinear nature. To address this, previous approximation studies mainly target rank-oriented tasks, which is acceptable for classification. However, edge Natural Language Processing (NLP) applications and edge generative AI are largely evaluated based on score-oriented tasks, so normalization-guaranteed non-GEMM operations are essential. We propose a hardware-efficient Softmax and LayerNorm with Guaranteed Normalization for Edge devices. Our design employs hardware-efficient approximation methods while preserving the normalization (Softmax: $\sum p = 1$, LayerNorm: $\sigma = 1$). Our architecture is described in Verilog HDL and synthesized using the Samsung 28nm CMOS process. In accuracy evaluation, we achieve high accuracy with minimal degradation: GLUE +0.07%, SQuAD -0.01%, perplexity -0.09%. Implementation results show that our architecture is small: $942\,\mu m^2$ for Softmax, $1199\,\mu m^2$ for LayerNorm. Compared to the state of the art, we achieve up to 11x and 14x reduction in area, respectively.
| Comments: | Accepted by 2026 IEEE International Symposium on Circuits and Systems (ISCAS) |
| Subjects: | Hardware Architecture (cs.AR); Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.23647 [cs.AR] |
| (or arXiv:2604.23647v1 [cs.AR] for this version) | |
| https://doi.org/10.48550/arXiv.2604.23647 arXiv-issued DOI via DataCite (pending registration) |
From: Ji-Hoon Kim [view email]
[v1]
Sun, 26 Apr 2026 10:34:04 UTC (234 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。