P-008 · 2026 · Hackathon entry
TabPFN-3.5 vs Bayesian Pareto/NBD customer models
When does the TabPFN-3.5 foundation model beat Bayesian Pareto/NBD models at predicting whether customers return? A benchmark on Online Retail II and CDNOW.
Study design
Common inputs, then a predeclared expanded feature set
The question
Retention analysts often predict repeat buying with Pareto/NBD, a 1987 model that needs no labelled examples. TabPFN-3.5 is a tabular foundation model that learns in context from labelled examples, without task-specific training. I built this benchmark for the Prior Labs TabPFN-3.5 Hackathon to find out when it predicts better. At monthly forecast origins, every existing customer gets two predictions: the probability of no purchase in the next 90 days and the number of purchases in that window.
Approach
TabPFN-3.5 runs untuned against simple reference rules, logistic and Poisson regression, tuned LightGBM, Pareto/NBD, and Abe’s (2009) hierarchical Bayesian version, which is only scored if its MCMC sampler converges. The supervised models first see only the three inputs Pareto/NBD uses (frequency, recency and customer age), then a predeclared expanded set with recent activity, gaps, trends and spend. I fixed the protocol on development origins, then scored three untouched holdout origins once, with confidence intervals from a bootstrap that resamples whole customers. Learning curves repeat the comparison with 50 to 2,000 labelled customers.
Results
On Online Retail II with the three Pareto/NBD inputs, the label-free Pareto/NBD beats TabPFN-3.5 on Brier score, a measure of probability error where lower is better (0.169 against 0.177), and is better calibrated. With expanded features, TabPFN-3.5 ties with the GLM, LightGBM and Pareto/NBD on Brier score, ranks inactive customers best (PR-AUC 0.858) and has the lowest error on purchase counts. It is also the best supervised model at every learning-curve size. On CDNOW, the Bayesian models’ home dataset, Abe’s model converges and wins on the common inputs. In short, TabPFN-3.5 wins when it gets richer features and when labels are scarce.
Limits
The data is non-contractual, so 90-day inactivity is not churn, and the models rank risk rather than estimate the effect of contacting a customer. TabPFN-3.5 had the least calibrated probabilities, because the inactivity rate in its training period differed from the holdout’s. The study covers two public retail datasets with overlapping holdout origins. The repository includes a local Streamlit workbench, the saved predictions and one command that regenerates every table.