View PDF HTML (experimental)

Abstract:Flaky tests, tests that pass or fail nondeterministically without changes to code or environment, pose a threat to software reliability. While classical software engineering has developed a body of techniques to study flakiness, evidence for quantum software remains limited. Prior work relies on static analysis or small sets of manually reported incidents, leaving open questions about their prevalence, characteristics, and detectability.
This paper presents a large-scale dynamic characterization of flaky tests in quantum software through a longitudinal case study of the Qiskit Terra core library. We executed the Qiskit Terra test suite 10,000 times across 23 releases in controlled environments. For each release, we measured test-outcome variability, identified flaky tests, estimated empirical failure probabilities, analyzed recurrence across versions, used Wilson confidence intervals to quantify baseline rerun budgets, and mapped flaky tests to Terra subcomponents.
Across 27,026 qualified test identifiers, we identified 62 unique flaky tests. Although flakiness rates were low (0-0.17%), recurrence was substantial: 52 of 62 flaky tests (83.87%) reappeared in multiple releases, while only 10 tests (16.13%) were confined to a single release. Failure probabilities spanned several orders of magnitude, with a median of $9 \times 10^{-4}$ and 34 tests (54.84%) at or below $10^{-3}$, implying that thousands to tens of thousands of executions may be required for confident detection.
These results show that flakiness observed in the studied Terra releases is rare under our controlled protocol but difficult to detect under typical continuous integration budgets. To support future research and replication, we release a dataset of per-test execution outcomes.

Submission history

From: Andriy Miranskyy [view email]
[v1] Fri, 19 Dec 2025 21:47:31 UTC (328 KB)
[v2] Fri, 26 Dec 2025 16:02:19 UTC (350 KB)
[v3] Fri, 5 Jun 2026 17:00:21 UTC (170 KB)
[v4] Tue, 8 Sep 2026 14:29:20 UTC (209 KB)