The Rise of Tabular Foundation Models and the Shift in Benchmarking Performance for Data Science

For more than a decade, the consensus within the data science community has remained largely unchallenged: when dealing with structured, tabular data, gradient-boosted decision trees (GBDTs) are the superior choice. Frameworks such as XGBoost, LightGBM, and CatBoost have dominated Kaggle competitions and industrial applications alike due to their efficiency, interpretability, and predictive power. However, a significant paradigm shift is currently underway. The emergence of "tabular foundation models"—pretrained transformers designed to predict missing columns in spreadsheets in a zero-shot manner—is fundamentally altering the competitive landscape.

On the TabArena leaderboard, a prominent community benchmark for tabular machine learning, every single-model entry outperforming the most highly tuned gradient-boosted trees is now a tabular foundation model. This development marks a pivotal moment in the evolution of machine learning, suggesting that the era of manual feature engineering and intensive hyperparameter tuning for GBDTs may be giving way to the era of large-scale, in-context learning for structured data.

The Architecture of Tabular Foundation Models

Tabular foundation models (FMs) operate on a principle similar to the large language models (LLMs) used for text. However, while a text-based LLM processes word-pieces from a fixed vocabulary, a tabular model must handle cells, columns, and rows consisting of numbers and categories with no fixed vocabulary. These models are typically pretrained on millions of synthetic tables generated by random structural causal models (SCMs). This pretraining process instills a general procedure for interpreting a new table and inferring the relationships between its features and the target variable, rather than memorizing specific world knowledge.

One of the most prominent examples of this new class of model is TabICLv2. Developed by the SODA team at Inria, TabICLv2 is a relatively compact model with approximately 28 million parameters. Unlike traditional models that require a "training" phase on a specific dataset, TabICLv2 utilizes in-context learning. To make a prediction, the model is provided with rows of known labels as context—much like providing examples in an LLM prompt—and it predicts the labels for held-out rows in a single forward pass.

The architecture of TabICLv2 consists of three stacked transformers:

1. Column-Wise Transformer: This stage processes each column independently to convert raw values into embeddings. By learning the distribution of a column, the model can distinguish between identical digits that represent different concepts, such as a price versus a postal code.
2. Row-Wise Transformer: This stage collapses the per-feature embeddings of a row into a single fixed-length vector. It uses learned query tokens to attend over the members of the row, effectively summarizing the information within a single example.
3. In-Context Learning Transformer: This final stage performs the actual prediction. Unlabeled test rows attend over the labeled training rows provided in the context. The model identifies the most similar training examples and reads off an answer, acting as a highly sophisticated, learned version of a k-nearest-neighbors algorithm.

Independent Verification and Benchmarking

The validity of these performance gains has been subjected to rigorous independent auditing. Using the TabArena benchmark, which comprises 51 datasets spanning binary classification, multiclass classification, and regression, researchers have sought to verify if the reported Elo ratings hold up under scrutiny. TabArena uses an Elo-based system where a standard Random Forest is anchored at 1000 points. A 400-point gap indicates 10:1 win odds.

In an independent audit conducted on AWS A10G hardware, TabICLv2 was re-run from scratch across all 51 datasets in the benchmark’s Lite protocol. The results confirmed the model’s strength: the independent run yielded an Elo rating of 1559, closely matching the official rating of 1575. The median relative difference in metric values per task was a mere 0.08%, a gap attributed to standard GPU non-determinism rather than methodology errors.

The efficiency of these models is also a significant factor. The entire evaluation sweep for 51 datasets was completed in just 2.1 hours, costing approximately $2 in compute resources. This challenges the traditional defense of GBDTs—that they are significantly cheaper to deploy—by showing that tabular FMs can offer superior accuracy with manageable serving costs.

Comparative Performance: Foundation Models vs. GBDTs

The TabArena standings reveal a clear hierarchy. At the top sits Google Research’s TabFM, which currently leads the board by a significant margin. Below it are various configurations of AutoGluon ensemble pipelines and other foundation models like TabPFN and TabICLv2.

The disparity between these models and traditional trees is stark. Even when GBDTs are given extensive tuning budgets—often involving hours of hyperparameter search and post-hoc ensembling—they struggle to match the zero-shot performance of foundation models. For instance, a fully tuned and ensembled LightGBM reaches an Elo of 1432, while the untuned TabICLv2 sits 158 points higher.

However, the "tree family" still maintains dominance in specific regimes. Analysis suggests that GBDTs remain the preferred choice in two primary scenarios:

• High Dimensionality: In datasets with more than 100 features, GBDTs frequently outperform foundation models. In the TabArena benchmark, foundation models won only one of the six datasets that exceeded this feature threshold.
• High-Cardinality Categoricals and Large Scale: On very large datasets (exceeding 20,000 rows) with complex categorical variables, the win rate for foundation models begins to decline.

The Contamination Debate: Synthetic vs. Real Data

As tabular foundation models evolve, the question of data provenance and benchmark contamination has become central to the discussion. The cleanest models from an evidential standpoint are those trained purely on synthetic data, such as TabICLv2 and the original TabPFN. Because these models have never "seen" a real-world table during pretraining, there is no risk of benchmark leakage.

In contrast, newer models like RealTabPFN-2.5 and TabDPT have achieved higher accuracy by incorporating real-world datasets from sources like OpenML and Kaggle into their pretraining corpora. While these models employ sophisticated deduplication pipelines to prevent contamination, the incentive to achieve higher Elo ratings through real-world data exposure creates an "epistemic challenge" for evaluators. Users seeking absolute certainty in model performance on their specific data often favor synthetic-only models, whereas those seeking maximum accuracy may opt for models trained on mixed data, provided they trust the developer’s deduplication claims.

Hybrid Approaches and Model Routing

One of the most promising avenues for future development is the use of hybrid systems or "routers." Since GBDTs and foundation models often excel on different types of datasets, a system that can intelligently choose the best model for a given table could achieve performance levels higher than any single model.

Research into "oracle routers"—which cheat by picking the better model based on known test errors—shows a potential Elo ceiling of 1707 when combining TabICLv2 and TabPFN. A more practical, deployable router that selects models based on validation error has also shown success when choosing between different foundation models. However, routing between a foundation model and a GBDT remains difficult, as the validation error metrics of the two families are often not directly comparable.

Broader Implications for the Industry

The shift toward tabular foundation models has profound implications for how organizations approach data science. Traditionally, a significant portion of a data scientist’s time is spent on feature engineering, handling missing values, and optimizing model parameters. Tabular FMs automate much of this process. By accepting pandas DataFrames as-is and providing predictions in seconds without a traditional training phase, these models lower the barrier to entry for high-performance machine learning.

Furthermore, the ability of these models to provide full predictive distributions—rather than just point estimates—offers greater utility for risk assessment and decision-making. TabICLv2’s regression head, for example, emits 999 quantiles, allowing users to understand the median and the uncertainty intervals of a prediction.

While gradient-boosted trees are far from obsolete—particularly for very wide tables or resource-constrained CPU environments—the TabArena results suggest they are no longer the default answer for every tabular problem. The rise of models like TabICLv2 and TabFM represents a maturation of the field, where the lessons learned from language modeling are being successfully translated to the structured data that powers much of the global economy.

As the community continues to audit these models and refine the benchmarks, the focus is likely to shift toward scaling these transformers to handle even larger and more complex spreadsheets, further narrowing the gap where traditional trees still hold their ground. For now, the evidence is clear: for most modest-sized tables, the foundation model era has arrived.