Abstract:Socio-economic causal effects depend heavily on their institutional and environmental contexts. The same intervention can produce different, even opposite, effects across regulatory regimes, market conditions, time periods, or populations. This poses a challenge for large language models (LLMs) in decision-support roles: can they infer the direction of a causal effect under a specified context, and revise that judgment when the context changes?
To address this, we introduce EconCausal, a large-scale benchmark of 10,490 context-annotated causal triplets extracted from 2,595 high-quality empirical studies in top-tier economics and finance journals, constructed through a rigorous four-stage pipeline with multi-run consensus, context refinement, and multi-critic filtering.
Across models, LLMs often fail to condition their predictions on context. While top models reach 88% accuracy in fixed, explicit contexts, accuracy falls by 32.6~pp on cases that require revising the sign across contexts (73.9% to 41.3%), and drops below 50% once misleading signed evidence is introduced. Models also over-commit to directional (+/-) signs, recognizing null effects only 13.8% of the time while remaining poorly calibrated on these categories. The dataset and benchmark are publicly available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2510.07231 [cs.CL] |
| (or arXiv:2510.07231v4 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2510.07231 arXiv-issued DOI via DataCite |
From: Donggyu Lee [view email]
[v1]
Wed, 8 Oct 2025 17:00:49 UTC (369 KB)
[v2]
Thu, 9 Oct 2025 16:46:30 UTC (369 KB)
[v3]
Mon, 23 Feb 2026 05:21:42 UTC (35,854 KB)
[v4]
Tue, 26 May 2026 12:27:46 UTC (1,215 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。