View PDF HTML (experimental)

Abstract:Recent work using Fully Homomorphic Encryption (FHE) has made non-interactive privacy-preserving inference of deep Convolutional Neural Networks (CNNs) possible. However, the performance of these methods remains limited by their heavy reliance on bootstrapping, a costly FHE operation applied across multiple layers that severely slows inference. Moreover, they depend on high-degree polynomial approximations of non-linear activations, which increase multiplicative depth and reduce accuracy by 2-5% compared with plaintext ReLU models. In this work, we close the accuracy gap between FHE-based non-interactive CNNs and their plaintext counterparts while achieving faster inference than existing methods. We propose a quadratic polynomial approximation of ReLU that achieves the theoretical minimum multiplicative depth for non-linear activations, together with a penalty-based training strategy. We further introduce structural optimizations that reduce the required FHE levels in CNNs by a factor of five compared with prior work, allowing us to run deep CNN models under leveled FHE without bootstrapping. To further accelerate inference and recover accuracy typically lost with polynomial approximations, we introduce parameter clustering together with a joint strategy involving data layout and ensemble techniques. Experiments with VGG and ResNet models on CIFAR and Tiny-ImageNet datasets show that our approach achieves up to 4x faster private inference than prior work, with accuracy comparable to plaintext ReLU models.

Submission history

From: Manaar Alam [view email]
[v1] Fri, 26 Sep 2025 19:10:23 UTC (223 KB)
[v2] Mon, 9 Feb 2026 06:07:32 UTC (1,926 KB)
[v3] Wed, 9 Sep 2026 13:17:42 UTC (2,297 KB)