






















Abstract:Evaluating the true forecasting ability of AI agents requires environments resistant to overfitting, free from centralized trust, and grounded in incentive-compatible scoring. Existing benchmarks either rely on static datasets vulnerable to training-data contamination, or measure trading PnL -- a metric conflating predictive accuracy with timing, sizing, and risk appetite. We introduce Foresight Arena, the first permissionless, on-chain benchmark for evaluating AI forecasting agents on real-world prediction markets. Agents submit probabilistic forecasts on binary Polymarket markets via a commit-reveal protocol enforced by Solidity smart contracts on Polygon PoS; outcomes are resolved trustlessly through the Gnosis Conditional Token Framework. Performance is measured by the Brier Score and a novel Alpha Score -- proper scoring rules that incentivize honest probability reporting and isolate predictive edge over market consensus. We provide a formal analysis: closed-form variance for per-market Alpha, the connection to Murphy's classical Brier decomposition, and a power analysis characterizing the number of rounds required to reliably distinguish agents of different skill levels. We show that detecting a true edge of $\alpha^* = 0.02$ at 80% power requires approximately 350 resolved binary predictions (50 rounds of 7 markets), while $\alpha^* = 0.01$ requires four times more. We complement these analytical results with a 50-round live evaluation of five frontier LLM agents plus a random baseline. Murphy decomposition distinguishes well-calibrated agents from market-tracking agents that fail through reduced resolution. All smart contracts and evaluation infrastructure are open-source.
| Comments: | 27 pages, 5 figures, 10 tables. Project page: this https URL. Code: this https URL |
| Subjects: | Multiagent Systems (cs.MA); Machine Learning (cs.LG); General Finance (q-fin.GN) |
| MSC classes: | 62F03, 62P05, 91B26, 68T07 |
| ACM classes: | I.2.6; I.2.11; H.4.2; G.3; J.4 |
| Cite as: | arXiv:2605.00420 [cs.MA] |
| (or arXiv:2605.00420v1 [cs.MA] for this version) | |
| https://doi.org/10.48550/arXiv.2605.00420 arXiv-issued DOI via DataCite (pending registration) |
From: Maksym Nechepurenko [view email]
[v1]
Fri, 1 May 2026 05:33:10 UTC (34 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。