





























Abstract:We study the problem of propagating the mean and covariance of a general multivariate Gaussian distribution through a deep (residual) neural network using layer-by-layer moment matching. We close a longstanding gap by deriving exact moment matching for the probit, GeLU, ReLU (as a limit of GeLU), Heaviside (as a limit of probit), and sine activation functions; for both feedforward and generalized residual layers. On random networks, we find orders-of-magnitude improvements in the KL divergence error metric, up to a millionfold, over popular alternatives. On a variational Bayes neural network, we show that our method attains hundredfold improvements in KL divergence from Monte Carlo ground truth over a state-of-the-art deterministic inference method. We also give a smooth-distance error bound showing that, under regularity assumptions, moment matching removes the leading low-variance errors and propagates higher-order local accuracy through the layers of a network.
| Comments: | new theoretical result on higher-order accuracy |
| Subjects: | Machine Learning (cs.LG); Numerical Analysis (math.NA) |
| Cite as: | arXiv:2601.22307 [cs.LG] |
| (or arXiv:2601.22307v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2601.22307 arXiv-issued DOI via DataCite |
From: Simon Kuang [view email]
[v1]
Thu, 29 Jan 2026 20:47:55 UTC (1,031 KB)
[v2]
Thu, 7 May 2026 21:21:56 UTC (1,116 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。