Abstract:Tabular foundation models with different architectures converge in accuracy across a range of classification and regression tasks. This raises questions a leaderboard cannot answer: (i) whether the models execute the same in-context algorithm, (ii) where row, column, and class-permutation invariances originate, and (iii) how robust they are under perturbations engineered against the inferred mechanism. We characterize all three. The model families realize qualitatively distinct similarity-based readouts: from an attention-weighted vote over context labels to a class-conditional mean readout, each confirmed by causal intervention. We find that the representation collapse highlighted in prior work is not a practical concern for them. Each model's permutation invariances trace to specific positional parameters whose removal preserves accuracy and makes approximate invariance exact. Perturbations engineered against each readout reproduce predicted failure modes; hub and rank attacks isolate them from refit baselines. Together these results give a mechanistic account of contemporary tabular foundation models and identify which inductive biases govern both their accuracy and characteristic failures.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.21288 [cs.LG] |
| (or arXiv:2605.21288v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.21288 arXiv-issued DOI via DataCite (pending registration) |
From: Marin Biloš [view email]
[v1]
Wed, 20 May 2026 15:23:16 UTC (196 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。