Abstract:Demographic information is often used to model annotator perspectives in subjective tasks such as hate speech detection, but its benefit is inconsistent: it improves performance in some settings and behaves as noise in others. This paper asks when demographic features help. We analyze demographic gain as a function of both data split properties and modeling frameworks. For data splits, we measure annotator disagreement, namely how often annotators assign different labels to the same example, along with training size and train-test demographic coverage. We find that demographic gains concentrate in regimes with low training disagreement, high test disagreement, fine-grained ambiguity measurement, sufficient training data, and greater demographic overlap. Motivated by these regimes, we introduce a gated demographic residual model that treats demographics as a selective adjustment to text-only predictions. Experiments on MHS and POPQUORN show that this design is effective, especially on high disagreement or low confidence examples. Overall, our results suggest that demographics should not be assumed useful by default; their value depends jointly on the data regime and the modeling framework.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.27313 [cs.CL] |
| (or arXiv:2605.27313v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.27313 arXiv-issued DOI via DataCite (pending registration) |
From: Weibin Cai [view email]
[v1]
Tue, 26 May 2026 17:24:41 UTC (708 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。