Abstract:Tabular foundation models are becoming increasingly popular for low-resource tabular problems. These models make up for small training datasets by pretraining on large volumes of synthetic data. The prior knowledge obtained via pretraining provides the exceptional performance, but the resulting model becomes a black box that is difficult to interpret and costly for inference. In this work, we explore an alternative strategy: using reasoning-capable LLMs to induce decision trees for small tabular datasets in an agentic setup. We design a minimal set of tools for constructing, analyzing, and manipulating decision trees. Equipped with these tools, the LLM combines its prior knowledge with learning from data to produce a lightweight decision tree that outperforms CART and recent non-greedy tree learners and remains competitive with tree ensembles on low-resource tabular problems. While a single agentic decision tree is competitive with state-of-the-art black box models, it also comes with a human-readable reasoning trace that can be checked for biases and data leaks. Furthermore, the reasoning-based LLM's creation process allows for additional human input to be incorporated into the tree without it being captured in data.
| Comments: | Preprint, code at this https URL |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2509.21465 [cs.LG] |
| (or arXiv:2509.21465v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2509.21465 arXiv-issued DOI via DataCite |
From: George Yakushev [view email]
[v1]
Thu, 25 Sep 2025 19:30:39 UTC (1,029 KB)
[v2]
Wed, 4 Mar 2026 11:39:38 UTC (4,768 KB)
[v3]
Fri, 15 May 2026 11:35:18 UTC (1,162 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。