Abstract:As privacy concerns in AI technologies continue to grow, Homomorphic Encryption (HE) offers a way to perform computations on encrypted data without the need of decryption during operations. However, HE is limited to addition and multiplication, making non-linear functions incompatible in their original form. This limitation has become more critical with the widespread use of Large Language Models (LLMs), where the non-linearity of activation functions such as the Rectified Linear Unit (ReLU) poses challenges for deployment in privacy-preserving Natural Language Processing (NLP) settings. This paper proposes a kernel-based approximation of ReLU, enabling its use within HE-constrained settings and thus contributing a critical step toward supporting privacy-preserving LLMs. A smooth kernel-based function, mimicking ReLU, is approximated using a second-degree polynomial, inspired by Jackson's theorem, to achieve low multiplicative depth. The proposed method is trained and assessed directly on token embeddings from pre-trained LLMs and evaluated in various scenarios, from simulated and tokenized data to deep learning and transformer models. Results show improved approximation fidelity, supporting the method's suitability for secure and privacy-preserving inference in various tasks.
| Comments: | 10 pages, 3 figures, submitted to the 33rd ACM Conference on Computer and Communications Security (CCS) |
| Subjects: | Cryptography and Security (cs.CR) |
| ACM classes: | I.2.7; E.3; G.1.2 |
| Cite as: | arXiv:2605.23641 [cs.CR] |
| (or arXiv:2605.23641v1 [cs.CR] for this version) | |
| https://doi.org/10.48550/arXiv.2605.23641 arXiv-issued DOI via DataCite (pending registration) |
From: Dimitra Papatsaroucha [view email]
[v1]
Fri, 22 May 2026 13:54:37 UTC (142 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。