Abstract:The next-token prediction (NTP) objective trains language models to predict a single token at each step, even though many continuations can express the same meaning. For example, in the sentence ``this sticker can be placed here'', positioned, attached, or put are all plausible alternatives. While standard NTP training treats these alternatives as mutually exclusive targets, we explore a self-supervised framework that encourages models to predict concepts, approximated as sets of semantically equivalent tokens. Models trained with this concept supervision align better with human similarity judgments, improve classification, clustering, and reranking performance, and achieve comparable or stronger downstream reasoning. These gains come with lower perplexity on semantically meaningful words (Section 3.2) and only minimal increases in global perplexity, suggesting that concepts enhance semantic alignment while preserving language modeling quality. Our code is available at this https URL .
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.29123 [cs.CL] |
| (or arXiv:2603.29123v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.29123 arXiv-issued DOI via DataCite |
From: Chen Shani [view email]
[v1]
Tue, 31 Mar 2026 01:20:03 UTC (7,242 KB)
[v2]
Sun, 24 May 2026 18:26:27 UTC (5,837 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。