Abstract:Ensuring that predicted probabilities align with observed frequencies is critical in high-stakes domains such as clinical decision support, autonomous driving and financial risk assessment. Existing calibration methods typically apply a single global transformation or rely on post-hoc binning over predicted confidences, limiting their ability to exploit heterogeneous reliability across sub-populations. We propose Clustered Calibration, a representation-aware framework that identifies sub-populations via clustering in learned feature spaces (e.g., coverage vectors, SHAP values, CNN activations, Transformer embeddings) and fits a soft mixture of cluster-specific parametric calibrators under hierarchical shrinkage toward a global mapping. This design yields context-specific calibration while maintaining global stability. Across six tabular datasets and additional image and text benchmarks, clustered calibration consistently improves or matches strong global calibrators in terms of negative log-likelihood and Brier score, while preserving AUC and accuracy. We further show, both analytically and empirically, that fixed-bin Expected Calibration Error (ECE) can mis-rank soft, region-aware calibrators even when proper scoring rules improve, and we advocate for log-loss and Brier as more reliable bases for model selection in such settings.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2510.19328 [cs.LG] |
| (or arXiv:2510.19328v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2510.19328 arXiv-issued DOI via DataCite |
From: Tomer Lavi [view email]
[v1]
Wed, 22 Oct 2025 07:41:30 UTC (887 KB)
[v2]
Sat, 23 May 2026 15:34:18 UTC (1,712 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。