Abstract:Background. The reliability paradox describes the empirical observation that cognitive tasks producing robust group-level effects often yield poor between-individual reliability. Existing approaches rely predominantly on the intraclass correlation coefficient (ICC), which captures only linear, second-moment dependence between test and retest.
Methods. We introduce a normalized, information-theoretic complement to ICC, NLR{\Delta}, defined as the difference between empirically estimated mutual information and the analytic Gaussian baseline implied by the test-retest correlation. We pair NLR{\Delta} with ICC(2,1), bias-corrected and accelerated (BCa) bootstrap intervals, Benjamini-Hochberg false discovery rate (FDR) control, and a 24-cell multiverse over the KSG nearest-neighbour parameter, correlation method, and minimum-sample threshold. The full pipeline is governed by pre-specified claim contracts, content-addressed provenance, and SHA-256-verified raw data ingestion, and is released as the MixMind Reliability Framework.
Results. Across 50 estimable primary measures from the Flanker, Stroop, Stop-Signal, Go/No-Go, and Posner task families, the median NLR{\Delta} is -0.138 nats, with interquartile range [-0.257, -0.034]. Zero of 50 primary measures exceed the headline rule. The companion ICC(2,1) analysis recovers the classical reliability paradox pattern, and the 24-specification multiverse yields 0 of 1,200 estimable cells passing the headline rule.
Conclusions. On these two public datasets, replacing or augmenting ICC with an information-theoretic reliability measure does not rescue cognitive tasks from the reliability paradox. The robust null is invariant to the analytic choices examined here. We release the full pipeline, raw-data hashes, and contracts to enable exact replication and extension to other datasets and tasks.
| Comments: | 12 pages, 2 figures, 3 tables; software and reproducibility materials archived at Zenodo DOI https://doi.org/10.5281/zenodo.20207371 |
| Subjects: | Methodology (stat.ME) |
| Cite as: | arXiv:2605.24995 [stat.ME] |
| (or arXiv:2605.24995v1 [stat.ME] for this version) | |
| https://doi.org/10.48550/arXiv.2605.24995 arXiv-issued DOI via DataCite (pending registration) |
From: Maria Westrin [view email]
[v1]
Sun, 24 May 2026 10:48:31 UTC (143 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。