Abstract:We propose Autolearn, a framework that enables language models to learn from documents they read, with no external supervision. Passages that produce anomalously high per-token loss are flagged, verified through a self-generated Q&A chain, and trained on with conviction-proportional $\beta_2$ adjustment. We introduce the perturbation gap (paraphrase-to-original perplexity ratio) as a metric that distinguishes memorization from understanding. The key mechanism is the training data format: Q&A-format training drives the perturbation gap below the pre-trained baseline (2.098 vs. 2.204, $\Delta = -0.106$, $> 10\sigma$), suppressing token-sequence memorization, while standard fine-tuning's best attempt remains within noise ($\Delta = -0.010$, $< 1\sigma$). Across four models spanning Qwen3 and Phi-4 families, Autolearn is the only method that enters this regime. Stochastic evaluation reveals passage-specific knowledge acquisition: the probability of generating a correct novel fact rises from 6% to 54% after training ($p < 10^{-4}$), and Q&A format outperforms standard fine-tuning on genuinely novel facts. The system is self-extinguishing: learned content reduces surprisal below threshold and is skipped on re-encounter.
| Comments: | 21 pages, 2 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2604.01951 [cs.LG] |
| (or arXiv:2604.01951v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2604.01951 arXiv-issued DOI via DataCite |
From: Kang Sin Choi [view email]
[v1]
Thu, 2 Apr 2026 12:17:10 UTC (300 KB)
[v2]
Thu, 7 May 2026 11:05:07 UTC (66 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。