View PDF HTML (experimental)

Abstract:This paper presents the study of a Bayesian estimation procedure for single-hidden-layer neural networks using $\ell_{1}$ controlled neuron weight vectors. We study the structure of the posterior density and provide a representation that makes it amenable to rapid sampling via Markov Chain Monte Carlo (MCMC). Let the neural network have $K$ neurons with internal weights of dimension $d$ and fix the outer weights. Thus there are $Kd$ parameters overall. With $N$ data observations, use a gain parameter or inverse temperature of $\beta$ in the posterior density for the internal weights.
The posterior is intrinsically multi-modal and not naturally suited to rapid mixing of direct MCMC algorithms. For a continuous uniform prior on the $\ell_{1}$ ball, we demonstrate that the posterior density can be written as a mixture density with suitably defined auxiliary random variables, where the mixture components are log-concave. Furthermore, when the total number of model parameters $Kd$ is large enough that $Kd \geq C(\beta N)^{2}$, the mixing distribution of the auxiliary random variables is also log-concave. Thus, neuron parameters can be sampled from the posterior by only sampling log-concave densities. The authors refer to the pairing of weights with such auxiliary random variables as a log-concave coupling.

Submission history

From: Curtis McDonald [view email]
[v1] Tue, 26 Nov 2024 18:29:14 UTC (48 KB)
[v2] Wed, 15 Jan 2025 21:32:35 UTC (51 KB)
[v3] Tue, 18 Mar 2025 18:58:21 UTC (81 KB)
[v4] Tue, 18 Aug 2026 03:11:05 UTC (64 KB)