Abstract:Sequential tests that allow continuous monitoring are common in A/B experimentation. Power calculations for these tests require simulations that are hard to scale across many metrics on an experimentation platform. Instead, a common sizing heuristic inflates the fixed-sample size until the marginal rejection probability at the planned endpoint reaches $1-\beta$. This last-point rule is conservative because always-valid (AV) power is the probability of a boundary crossing at any time during the run, not at the endpoint alone. We give a closed-form correction factor $k^(\alpha, \beta, t_0)$ expressed in elementary functions and the bivariate normal CDF, where $t_0 = m/n_z$ is the burn-in fraction. The closed-form approximation depends on the boundary only through its value and slope at the planned endpoint and can be evaluated for any smooth concave boundary. We work out three cases: the confidence sequences of Waudby-Smith et al. (2023) and Maharaj et al. (2023), and the mixture sequential probability ratio test of Johari et al. (2022). Setting the total sample size to $k^ \cdot n_z$, where $n_z$ is the fixed-sample size for allocation ratio $r$, hits empirical power within approximately 3 percentage points of target in Gaussian simulations. The correction factor depends on the allocation ratio $r$ only through $t_0 = m/n_z(r)$. We study sensitivity to the burn-in parameter and show that the correction saves 8--20% of the last-point sample budget across the operating range.
From: Mårten Schultzberg [view email]
[v1]
Tue, 16 Jun 2026 18:14:27 UTC (40 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。