A General Bias‑Variance Decomposition for Proper Scoring Rules – Finally!

Or: Why your ensemble works, how to build confidence regions in logit space, and what Bregman information really does for uncertainty estimation.

If you’ve ever trained a classifier, you’ve heard the mantra:

Bias‑variance trade‑off.

Until now.

• Explain ensembles via a law of total Bregman variance.
• Build confidence regions directly in logit space.
• Detect out‑of‑distribution inputs better than raw softmax confidence.

Let’s dive in.

---

The problem: Uncertainty under domain drift

Missing piece: A general bias‑variance decomposition for strictly proper scoring rules.

---

Background: Bregman divergences & proper scoring rules

Bregman divergence

Given a differentiable convex function $\phi$ , the Bregman divergence is

$$d_\phi(x, y) = \phi(y) - \phi(x) - \langle \nabla \phi(x), y-x \rangle.$$

Example: $\phi(x)=x^2$ gives $d_\phi(x,y)=(x-y)^2$ (squared error).

Example: $\phi(x)=x\ln x$ gives the KL divergence.

Strictly proper scoring rule

A scoring rule $S(P, y)$ is strictly proper if the expected score is maximised only when $P$ equals the true data distribution $Q$ .

Common examples:

• Log score: $S(P,y)=\log p(y)$
• Brier score: $S(P,y)=-|\delta_y - P|^2$
• CRPS (continuous ranked probability score)

Every strictly proper scoring rule corresponds to a Bregman divergence generated by the negative entropy $G$ (Ovcharov, 2018).

---

The main result: A general bias‑variance decomposition

Theorem (Gruber & Buettner, 2023)

ParseError: KaTeX parse error: Expected group after '^' at position 98: …\underbrace{B{G^̲}[S(\hat{f})]}{…

What does each term mean?

• $B_{G^*}[X]$ – Bregman information (generalised variance). For $\phi(x)=x^2$ , $B_\phi[X] = \mathrm{Var}(X)$ .
• $d_{G^*, S^{-1}}$ – Bregman divergence in the dual space – that’s the squared bias.

So the classical MSE decomposition ( $\text{error} = \text{noise} + \text{var} + \text{bias}^2$ ) is a special case of this theorem.

---

Bregman information – the “variance” term

Definition (Banerjee et al., 2005):

$$B_\phi[X] = \mathbb{E}[d_\phi(\mathbb{E}[X], X)] = \mathbb{E}[\phi(X)] - \phi(\mathbb{E}[X]).$$

It measures spread around the mean in the sense of a Bregman divergence.

Figure 2 in the paper shows $B_{\sigma_+}$ for the softplus function $\sigma_+(x)=\ln(1+e^x)$ – this controls variance for binary classification in logit space.

[!NOTE]
When $\phi$ is the squared function, $B_\phi$ is the classical variance.

When $\phi$ is the log‑sum‑exp function (LSE), $B_\phi$ is the variance in logit space.

---

Special case: Exponential families

For an exponential family $p_\theta(y) = \exp(\langle \theta, T(y)\rangle - A(\theta))h(y)$ , the decomposition becomes:

$$\mathbb{E}[-\ln p_{\hat{\theta}}(Y)] = A(\theta) + B_A[\hat{\theta}] + d_A(\theta, \mathbb{E}[\hat{\theta}]).$$

• $B_A[\hat{\theta}]$ – variance in the natural parameter space (classical variance weighted by the log‑partition function $A$ ).
• Perfectly recovers the classical MSE case when $A(\theta)=\theta^2/2$ .

---

Special case: Classification (logit space) – this is huge

Corollary:

$$\mathbb{E}[-\ln \text{sm}Y(\hat{z})] = H(Q) \;+\; B{\mathrm{LSE}}[\hat{z}] \;+\; d_{\mathrm{LSE}}\big(\mathrm{sm}^{-1}(Q),\,\mathbb{E}[\hat{z}]\big),$$

where $\mathrm{LSE}(x) = \ln\sum e^{x_i}$ (LogSumExp).

Why is this surprising?

• The variance term $B_{\mathrm{LSE}}[\hat{z}]$ is computed directly on the logits, without applying softmax.
• No normalisation to probabilities needed – numerically stable and conceptually clean.

This is perfect for deep neural networks:

To estimate predictive uncertainty, just compute the Bregman information of the logits over an ensemble or multiple forward passes.

---

Applications

1. Why ensembles reduce uncertainty

The law of total Bregman information:

$$B_G[X] = \mathbb{E}[B_G[X \mid Y]] + B_G[\mathbb{E}[X \mid Y]].$$

For an ensemble that averages over random initialisations $W$ :

$$B_A[\hat{\theta}D^{(n)}] \to B_A[\mathbb{E}_W[\hat{\theta}{W,D}]],$$

i.e., the variance due to $W$ disappears.

The expected score strictly improves.

This is the first general theoretical justification for why ensembles are almost always beneficial.

---

2. Confidence regions via Markov’s inequality

Using Markov’s inequality on the Bregman divergence:

$$P\Big(d_G(\mathbb{E}[X], X) \ge \frac{1}{\alpha} B_G[X]\Big) \le \alpha.$$

Thus a $(1-\alpha)$ -confidence region is:

ParseError: KaTeX parse error: Expected group as argument to '\big' at position 92: …]}{\alpha} \big}̲.

Figure 3 & 4 in the paper:

• Binary classification – confidence intervals on the probability simplex.
• Iris dataset – convex confidence regions for three classes.

No need for normality assumptions – works with any proper score.

3. Out‑of‑distribution detection (CIFAR‑10C / ImageNet‑C)

Result (Figure 1 in the paper):

• To reach 90% validation accuracy, using max softmax confidence you must discard ≈14% of data.
• Using Bregman information $B_{\mathrm{LSE}}$ you only discard ≈7% of data.

→ Bregman information is a superior uncertainty measure under domain drift.

---

Limitations (real talk)

• Computational cost: Estimating Bregman information requires multiple predictions per input (ensemble, MC dropout, or multi‑epoch sampling).
• Proper scoring rules only: Doesn’t directly apply to 0‑1 loss (accuracy). But for probabilistic forecasting that’s fine – use log‑loss.
• Not Bayesian: It gives a frequentist variance measure, not a full posterior.

Future work: extend to Bayesian neural networks and large language models (uncertainty for hallucinations).

---

Take‑away

• First general closed‑form bias‑variance decomposition for strictly proper scoring rules.
• Bregman information emerges as the universal variance term – generalising classical variance.
• Logit‑space formulation makes it practical for deep learning.
• Demonstrated benefits: ensembling theory, confidence regions, OOD detection.

Code available: GitHub – MLO‑lab/Uncertainty_Estimates_via_BVD

---

---