Abstract:Random-feature neural networks (RFNNs), including architectures with fixed hidden layers and analytically determined output weights, offer fast training but often suffer from issues due to dense representations of the hidden layer activation. Their reliance on dense feature mappings and least squares solvers can limit scalability and numerical stability, particularly for high-dimensional or stiff systems. Specifically, the activation matrix is observed to be low-rank and extremely ill-conditioned. In this work, we propose a sparse framework for RFNNs that integrates structured sparsity into the hidden layer activations that increases the rank and employs Sparse Singular Value Decomposition (sSVD) for solving the resulting linear least squares problem scalably and efficiently while catering to the bad condition number. We explore the theory behind Lanczos-Golub-Kahan Bidiagonalization technique for sparse SVD and conduct some experiments to identify some limitations and justify the requirement for orthogonalization step in our application. Then, we demonstrate that the proposed method maintains or improves solution accuracy for solving the benchmark one-dimensional steady convection-diffusion equations case having stronger advection, while achieving substantial gains in training efficiency and robustness compared to standard dense implementations.
| Subjects: | Numerical Analysis (math.NA); Machine Learning (cs.LG); Computational Physics (physics.comp-ph) |
| Cite as: | arXiv:2605.07286 [math.NA] |
| (or arXiv:2605.07286v1 [math.NA] for this version) | |
| https://doi.org/10.48550/arXiv.2605.07286 arXiv-issued DOI via DataCite (pending registration) |
From: Siddharth Rout [view email]
[v1]
Fri, 8 May 2026 05:48:43 UTC (831 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。