Talking SOTA

The idea in one line: State-of-the-art fluency is a short list of numbers with their caveats attached, a few questions that separate real results from marketing, and an honest list of what nobody has solved.

SOTA (state of the art) means the best published result on a named test at a named date. Experts know which numbers survive scrutiny and which question to ask next. That skill is worth an order of magnitude in a conversation, because the same model can look 444x better or 8x better depending on which comparator you accept, and its calibration error can sit anywhere from 0.011 to 0.245 depending on the task. All rankings and prices are as of October 2026; leaderboards here have a half-life of about six weeks.

Every number has three hidden parts: the test (which data), the comparator (against what), and the date (before or after a rival shipped).

Worked example. Someone says “Jev’s calibration error is 0.023”. Ask which test. That figure is Arize’s hallucination task (Arize, 2026-10-07). On Vals’s 10-K claim checks it was 0.011; on a LegalBench slice, 0.108 (Vals AI, 2026-10-06). On the community Decision Index it was 0.080 overall and 0.245 on retrieval tasks (Decision Index, 2026-10-28). Same model, a 20x spread. The precise statement is “Jev’s calibration is task-dependent, from 0.011 to 0.245 in published tests”, and TypeSafe itself publishes no calibration metric.

A decision model reads input and returns a probability over answers you defined, in one pass, with no generated text (anatomy of Jev, how decision models work). The state: the contract won and the model commoditized. Jev’s three production types, Choice (up to 255 options), Score (2-10 ordered levels) and Noul (yes/no), became a convention copied by OpenAI, Microsoft, Cloudflare, Perplexity and others within a month.

Questions that reveal understanding:

• “Is accuracy measured against ground truth or against agreement with frontier models?” TypeSafe’s own reference labels are averages of two frontier models.
• “Are you reading probabilities or the confidence field?” Jev’s confidence is a fixed function of the distribution, not a probability of being right. For a Choice it is (p_max − 1/n) / (1 − 1/n): with 4 options and a top probability of 0.7, confidence is (0.7 − 0.25) / 0.75 = 0.6.
• “What does RLCD (TypeSafe’s unpublished training method) add beyond log loss on outcomes?” A proper-scoring reward on a one-pass distribution is close to supervised log loss.

Open problems: Jev’s architecture and size are undisclosed. Single-pass models remain bad at counting, arithmetic and date comparison. A security vendor says Jev is more vulnerable to some attacks than LLMs, with no success rates published.

Calibration means stated probabilities match observed frequencies (calibration). ECE (expected calibration error) is the average gap between stated confidence and actual accuracy. The state: base models are well calibrated, post-training breaks it, and one parameter or a proper-scoring reward repairs most of it in domain, but not out of domain.

The threshold formula in words: act when the probability exceeds the cost of a false positive divided by the total cost of both kinds of mistake. With illustrative costs, if a wrong “yes” costs $400 and a wrong “no” costs $40, act only when p ≥ 400 / 440 ≈ 0.91.

Questions: “What is the noise floor of that ECE estimate?” “Is the coverage guarantee marginal (on average) or per subgroup?” “Under which elicitation protocol?” Whether verbalized or token-probability confidence wins flips with the protocol (Kim and Kang, 2026).

Open problems: whether reasoning improves calibration (better in 33 of 36 settings in one study, worse with more thinking budget in another); calibration out of domain after proper-scoring RL; trained routers that over-escalate to the strongest model at high budgets.

A tabular foundation model predicts on your table from rows supplied as context, with no training run, because it was pretrained on millions of synthetic tables (in-context learning and PFNs). The state: they now lead the main benchmark on small and medium tables, and the leader is trained on synthetic data only. Rankings use Elo, a chess-style rating built from head-to-head wins across datasets.

Source: TabArena leaderboard data.

Questions: “Up to how many rows, and on whose data?” TabArena tops out around 150,000 rows; the 1M-row claims rest on Prior Labs’ internal datasets. “Fit time or predict time?” Foundation models remove the tuning bill but pay per prediction. “Is that entry carrying a leak warning?”

Open problems: performance beyond about 150K rows; whether real data still matters (TabDPT says yes); whether in-context learning is really Bayesian (Nagler explains the gains as variance reduction instead).

A time-series foundation model forecasts a new series from its history in context, zero-shot (TabPFN, TimesFM and friends). The state: tiny models are nearly as good as big ones, the leaderboard turns over every few weeks, and the top of the boards is dominated by ensembles. CRPS (continuous ranked probability score) grades a probabilistic forecast; GIFT-Eval divides a model’s CRPS by seasonal naive’s.

Source: GIFT-Eval, updated 2026-10-23.

Questions: “Zero-shot, or fine-tuned per site?” On ERCOT solar, zero-shot TimesFM had 9.25% error, fine-tuned 3.71%. “Single model, or an ensemble tuned on the benchmark’s training split?” “Does the model carry a leak flag?” “Can I use it commercially?” Chronos-2, Toto 2.0 and TimesFM-2.5 are Apache 2.0.

Open problems: TimesFM-3’s compute and data mix are undisclosed; whether agentic ensembles transfer to production; no public zero-shot benchmark exists for remaining-useful-life prediction on vibration data.

Weather is the clearest win; single-cell biology is the clearest overstatement (vertical foundation models).

Questions: “Inference-only speedup, or end to end?” “Against a linear or pseudobulk (averaged-expression) baseline?” “Binned by similarity to training data?” “Does the licence allow training on outputs?” AlphaFold 3’s terms bar it.

Open problems: virtual cells against linear baselines; out-of-distribution weather and heavy precipitation; “stable in simulation” versus “novel and makeable” materials; manufacturing data that is scarce and rights-encumbered.

The state: labels are nearly free, gold is not, and the best labels are the ones the world already wrote (labels from AI and from the world, verifiable rewards and evals).

Questions: “Where did the labels come from, and how correlated are the labellers’ errors?” “Did you run a random-reward control on two base-model families?” “Which licence covers the model that made your labels?” Anthropic, Google and OpenAI bar training competing models on their outputs.

Open problems: where a non-generative decision model sits under fair use after Thomson Reuters v. Ross (2026-09-29); how much human gold an LLM-labelled set needs; how severe collapse really is.

The state: training the model is the cheap part; labels, evaluation and failed runs are the expensive parts (the bootstrapper’s stack).

What it buys at serving time, in one estimate that assumes 500 input and 30 output tokens per decision and full utilization for self-hosting:

Cost per 1M decisions (estimate, late October 2026)

flagship frontier LLM writing JSON      $6,500
Sonnet-class LLM                        $1,300
small frontier tier (Haiku-class)       $35-65
Jev at list price                       $21
150M encoder, serverless CPU            $0.76-3.74
150M encoder, busy RTX 4090             $0.08-0.84

The “orders of magnitude” claim holds against flagship models (350-16,000x for the encoder) but shrinks to 10-800x against the cheapest frontier tier, and Jev’s list price sits within about 1-3x of batched or cached Haiku-class calls. Utilization is the hidden multiplier: a GPU busy 5% of the time costs 20x more per decision.

Questions: “Was most of the budget GPU time or labels?” “At what utilization was that per-decision cost measured?” “Does the teacher’s licence allow distillation?” Anthropic, Google and OpenAI restrict training competing models on their outputs; DeepSeek explicitly permits distillation.

Open problems: whether “competing models” clauses cover a narrow decision classifier is undefined; most frontier vertical models do not disclose training compute (TimesFM-3, Aurora), so cost comparisons rest on the few that do.

Jev launched as “193.6x faster, 444.6x cheaper”. The arithmetic on TypeSafe’s own table:

Jev launch multiples, from TypeSafe's own per-case table

model      time/case   cost/case   accuracy   multiple
Jev          0.4 s     $0.0004     67.8%
Sonnet 5    78.1 s        ..       67.8%      78.1/0.4     = 195x
Opus 5        ..       $0.1761     73.1%      0.1761/0.0004 = 440x
Luna        12.9 s     $0.0033     66.8%      32x faster, 8x cheaper

That reproduces the headlines to within rounding, and shows the trick: each headline uses a different worst-case comparator: the slowest model for speed, the most expensive for cost. Against the cheapest model of equal accuracy (Luna, one point below) the gap is about 8x on cost and 32x on time (TypeSafe evals). The homepage chart’s own numbers give 171x and 75x (Eden AI). Users reported a median 7x speedup in one scan of 12,759 launch tweets, a single-source figure (InfoQ). The rule: always compare against the cheapest model of equal accuracy.

Weather’s speedups are real, but check three things (vertical foundation models):

• Inference only. Every headline ratio compares running a trained network with running a physics model. None counts producing ERA5 (decades of physics reanalysis) or training. GraphCast’s training took about 21,500 TPU-v4-hours, an estimate from 4 weeks on 32 devices.
• Different hardware. GraphCast’s “under a minute” is one TPU v4; HRES’s “hours” are hundreds of supercomputer nodes. Wall-clock time is not energy.
• Different resolution. FourCastNet’s “45,000x faster, 12,000x less energy” is a node-to-node estimate for a hypothetical 18 km model (arXiv 2202.11214). NeuralGCM’s “about 100,000x” is against a climate model on about 13,000 CPUs. AIFS runs at 28 km against the physics model’s 9 km.

Only ECMWF’s roughly 1,000x energy figure comes from an operational side-by-side (ECMWF, 2025-02-25). That is the number to quote.

• Leak flags. GIFT-Eval marks Chronos, Chronos-Bolt, TimesFM 1 and 2, TTM-R2 and Lag-Llama as having seen test data.
• Memorization. A TabArena issue shows LimiX-2 near-perfect on one dataset until the column order is reversed; TabArena added a warning on 2026-10-27.
• Private tests. The Decision Index moved to private tests in edition 0.4 because public items leak.
• Systems, not models. Agentic ensembles tuned on benchmark training splits hold most top spots on GIFT-Eval.
• Self-reporting. Almost every tabular and time-series SOTA claim comes from the model’s maker; the raw result files are the independent anchor.
• Editions differ. JevBench v1.3.0 ranks Jev #1; the Decision Index ranks it around 48th; scores across editions are not comparable.

Here is what an expert would say if you asked for the state of the field in a minute:

“Typed decision models, which return a probability instead of text, went from one vendor to a dozen in a month. The contract stuck and the model commoditized: $0.042 per million input tokens, hosted leaders within a point of each other, open clones close behind. Calibration is the real differentiator, and it’s trainable but task-dependent: Jev’s error ranges from 0.011 to 0.245 across published tests, so you recalibrate on your own data. Tabular foundation models now beat tuned gradient boosting by about 600 Elo on small and medium tables, trained on synthetic data alone. Time-series models roughly halve seasonal-naive error, tiny ones are nearly as good as big ones, and the leaderboard turns over every six weeks. In verticals, weather is the real win, about 1,000x less energy operationally, while single-cell models still lose to linear baselines. The moat isn’t the model: it’s a verifier, outcome-labelled data and the rights to use it. Training is cheap; a 2B decision model trains in under an hour. Always ask for the cheapest equal-accuracy comparator, a tuned baseline, and data after the cut-off.”

• Arize: decision model benchmark — independent accuracy, calibration, latency and cost for the hosted leaders.
• Vals AI: independent evaluation of Jev — how calibration fails to transfer between tasks.
• GIFT-Eval leaderboard — time-series standings with leak flags.
• TabArena leaderboard — the living tabular benchmark and its warnings.
• ECMWF: AIFS goes operational — the operational weather comparison behind the 1,000x energy figure.
• Damani et al., RLCR — calibration as a training objective.