





















Abstract:Chemical Language Models (CLMs) pre-trained on large scale molecular data are widely used for molecular property prediction. However, the common belief that increasing training resources such as model size, dataset size, and training compute improves both pretraining loss and downstream task performance has not been systematically validated in the chemical domain. In this work, we evaluate this assumption by pretraining CLMs while scaling training resources and measuring transfer performance across diverse molecular property prediction (MPP) tasks. We find that while pretraining loss consistently decreases with increased training resources, downstream task performance shows limited improvement. Moreover, alternative metrics based on the Hessian or loss landscape also fail to estimate downstream performance in CLMs. We further identify conditions under which downstream performance saturates or degrades despite continued improvements in pretraining metrics, and analyze the underlying task dependent failure modes through parameter space visualizations. These results expose a gap between pretraining based evaluation and downstream performance, and emphasize the need for model selection and evaluation strategies that explicitly account for downstream task characteristics.
| Subjects: | Machine Learning (cs.LG); Quantitative Methods (q-bio.QM) |
| Cite as: | arXiv:2602.11618 [cs.LG] |
| (or arXiv:2602.11618v4 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.11618 arXiv-issued DOI via DataCite |
From: Tatsuya Sagawa [view email]
[v1]
Thu, 12 Feb 2026 06:14:34 UTC (8,073 KB)
[v2]
Tue, 17 Feb 2026 12:54:31 UTC (7,963 KB)
[v3]
Wed, 1 Apr 2026 08:39:47 UTC (4,944 KB)
[v4]
Wed, 13 May 2026 07:35:21 UTC (4,952 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。