Capability research on foundation models. What makes one good in the first place; what it was pretrained on; how you would know, from its behavior, which part of the data was responsible. Most of this is with the Tabular Foundation Models Lab at Lexsi Labs.
* equal contribution.
Synthetic relational pretraining data is supposed to teach a tabular foundation model to use relations. This traces the predictive structure from the pretraining distribution through to model behavior, and separates the cases where the model genuinely learns to use relations from the cases where it only appears to.
Attributing what a tabular foundation model does at inference back to the synthetic data it was pretrained on: behavior to provenance, rather than the usual direction.
The first logit-lens implementation for tabular in-context learning: a classification phase transition at Layer 8, representation hijacking as early as Layer 4, and total hijacking from 1.2% poisoned context. Larger models are more vulnerable, not less. QK/OV decomposition and direct logit attribution show that mechanistically-informed defenses (PCA subspace scrubbing, head ablation) fail because of manifold redundancy.
Introduces O'Prior, a compositional realism prior for synthetic pretraining of tabular foundation models. I ran the interpretability and data-quality ablations, which confirm that mechanism diversity, realism composition, and shift-aware stress each contribute independently to downstream quality.
Neural operators for ocean temperature prediction, with the physics constraint and the explanation methods carrying equal weight: a forecast you cannot interrogate is not much use to the people who have to act on it.