Abstract:Online A/B testing at scale relies on proxy metrics -- short-term, easily-measured signals used in place of slow-moving long-term outcomes. When the proxy-outcome relationship is heterogeneous across user segments, aggregate correlation can mask directional failures akin to Simpson's Paradox, leading to costly ship/no-ship errors. We introduce PROXIMA (Proxy Metric Validation Framework for Online Experiments), a lightweight diagnostic framework that scores proxy reliability through a composite of three complementary dimensions: normalised effect correlation, directional accuracy, and segment-level fragility rate. Unlike surrogate-index approaches that predict long-term treatment effects, PROXIMA directly audits whether a candidate proxy leads to correct launch decisions and flags the user segments where it fails. We validate PROXIMA on two public datasets -- the Criteo Uplift corpus (14M observations, advertising) and KuaiRec (7K users, video recommendation) -- using 80 simulated A/B tests. Early engagement metrics achieve a composite reliability of 0.80 on Criteo and 0.62 on KuaiRec, yielding 98.4% average decision agreement with an oracle policy. Fragility analysis reveals that recommendation domains exhibit substantially higher segment-level heterogeneity (68% fragility) than advertising (13%), yet directional accuracy remains above 96% in both cases. A sensitivity analysis over the weight space confirms that no single component suffices and that the composite provides substantially better discrimination between reliable and unreliable proxies than correlation alone. Code and reproduction scripts are available at: this https URL
| Comments: | 14 pages. Sole-author submission. Independent research. Companion code at this https URL. Zenodo archive: https://doi.org/10.5281/zenodo.15483241. Related US provisional patent application: 63/974,569 (filed Feb 3, 2026) |
| Subjects: | Methodology (stat.ME); Machine Learning (cs.LG); Applications (stat.AP) |
| MSC classes: | 62-07, 62P25, 62L05 |
| Cite as: | arXiv:2604.14352 [stat.ME] |
| (or arXiv:2604.14352v1 [stat.ME] for this version) | |
| https://doi.org/10.48550/arXiv.2604.14352 arXiv-issued DOI via DataCite (pending registration) |
From: Avinash Amudala [view email]
[v1]
Wed, 15 Apr 2026 19:10:53 UTC (4,444 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。