Abstract:Linear properties are ubiquitous in the representations of language models; however, testing them experimentally remains a challenging task. This work focuses on relational linearity: the hypothesis that, for a fixed relation (e.g., "plays"), the unembedding of an object (e.g., "trumpet") can be predicted from the embedding of its subject (e.g.,"Miles Davis") by a linear map. We present an experimental method to test the formulation of relational linearity by Marconato et al. (2025). Specifically, we introduce a probing method, based on Kullback-Leibler divergence, to evaluate this property and examine its variation across layers and paraphrased relational queries. It is also more efficient than previous work; for example, it avoids the crude Jacobian approximations used in Linear Relational Embeddings by Hernandez et al. (2024). Our findings across four datasets show that relational linearity varies across models, exhibits layer-wise patterns consistent with prior observations about linguistic information in model representations, and is differently affected by changes in how the relation is phrased.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.22532 [cs.LG] |
| (or arXiv:2605.22532v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.22532 arXiv-issued DOI via DataCite (pending registration) |
From: Emanuele Marconato [view email]
[v1]
Thu, 21 May 2026 14:22:27 UTC (1,991 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。