Abstract:Discrete diffusion models (DMs) have achieved strong performance in language and other discrete domains, offering a compelling alternative to autoregressive modeling. Yet this performance typically depends on large training datasets, challenging the performance of DMs in small-data regimes -- common under real-world constraints. Aimed at this challenge, recent work in continuous DMs suggests that transfer learning via classifier ratio-based guidance can adapt a pretrained DM to a related target distribution, often outperforming alternatives such as full-weight fine-tuning on the target data. By contrast, transfer learning for discrete DMs remains unexplored. We address this gap by exploring practical analogues of ratio-based transfer learning for discrete DMs. Our theoretical analysis shows that a direct extension of existing ratio-based guidance is computationally prohibitive, scaling with vocabulary size. To overcome this limitation, we introduce a scheduling mechanism that yields a practical algorithm, Guided Transfer Learning for discrete diffusion models (GTL). GTL enables sampling from a target distribution without modifying the pretrained denoiser and reduces the cost to linear scaling in vocabulary size, which in turn supports longer sequence generation. We evaluate GTL on sequential data, including synthetic Markov chains and language modeling tasks, and provide a detailed empirical analysis of its behavior. The results highlight a clear trade-off: when target datasets are large, weight fine-tuning is often preferable, whereas GTL becomes increasingly effective as target data shrinks. Finally, we experimentally demonstrate a key failure mode of GTL: when the source and target distributions overlap poorly, the ratio-based classifier required for guidance becomes unreliable, limiting transfer performance.
| Comments: | Accepted at ICLR 2026 ReALM-GEN Workshop 9 pages (main text) + appendix |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2512.10877 [cs.LG] |
| (or arXiv:2512.10877v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2512.10877 arXiv-issued DOI via DataCite |
|
| Journal reference: | ICLR 2026 ReALM-GEN Workshop |
From: Julian Kleutgens [view email]
[v1]
Thu, 11 Dec 2025 18:05:55 UTC (685 KB)
[v2]
Fri, 20 Feb 2026 13:58:35 UTC (930 KB)
[v3]
Fri, 20 Mar 2026 18:11:14 UTC (1,037 KB)
[v4]
Tue, 14 Apr 2026 21:32:45 UTC (1,024 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。