Distill a decision model

The idea in one line: Pay a big model once to label your decisions, train a small model to copy those labels, calibrate its probabilities, and check it against the world, not only against its teacher.

Distillation means training a small “student” model on the outputs of a big “teacher” model. It is the economic core of a Jev-like product: label once, then run the cheap student forever.

The evidence is strong. In a 2024 case study, Mixtral labelled 1,811 financial sentences and a RoBERTa-base student reached 94% accuracy, the same as its teacher and GPT-4, at about $2.70 per million inferences on a T4 against $3,061 for GPT-4; fine-tuning cost under $1 in 15 minutes (Hugging Face blog, 2024-02-16). At web scale, FineWeb-Edu had Llama-3-70B score 500,000 pages, trained a small regressor on those scores (its F1, a score that balances missed and wrong keeps, was 82% at the keep threshold), and ran it over 15 trillion tokens in about 6,000 H100-hours (FineWeb-Edu card, v1.4.0, 2025-07-11).

The 10x lever is “distil, don’t call”. The choice between two good encoders (0.5-2 points on GLUE, a standard language-understanding benchmark) is 10% polish. What decides success is the question design, the teacher you are allowed to learn from, and a gold set that tells you the truth.

Picture an apprentice who learns by watching a master make thousands of calls. The apprentice gets fast, but can only be as good as the master, unless a referee who knows the real outcome grades them both.

In a decision model, the teacher is a frontier LLM, the student is a 150M encoder or a 1B decoder, and the referee is the world: did the build pass, was the bug really a duplicate (labels from AI and from the world).

The student can learn from hard labels (just the teacher’s answer) or soft labels (the teacher’s probabilities). Recall from probability is the product that log loss charges -ln(p) for the probability p you gave the right answer.

Worked example. The teacher says P(duplicate) = 0.7, P(valid) = 0.3. The student says 0.6 and 0.4.

• Hard label “duplicate”: loss = -ln(0.6) = 0.51.
• Soft label: loss = -(0.7 ln 0.6 + 0.3 ln 0.4) = 0.358 + 0.275 = 0.633. Its floor, reached when the student copies the teacher exactly, is the teacher’s own entropy -(0.7 ln 0.7 + 0.3 ln 0.3) = 0.611. The gap, 0.022, is the KL divergence: how far the student’s distribution is from the teacher’s.

The precise version: a student trained only on teacher labels has a ceiling at teacher agreement. It inherits the teacher’s errors and blind spots, and the probabilities of RLHF-tuned teachers (chat models tuned with reinforcement learning from human feedback) tend to be poorly calibrated, so calibrate the student on human or outcome labels instead (calibration). The only escape is labels from outside the teacher: outcomes, humans or verifiers. Even Jev’s launch “accuracy” was agreement with frontier models, about 68% by one report (NYU Shanghai RITS, single source).

1. Choose the questions. Write each recurring decision as a typed question with a closed answer set: a Choice among options, a Score on an ordered scale, or a yes/no (Jev calls it “Noul”). The designing the questions lesson covers the label guide: one-sentence definitions, examples, tie-breaks, “none of the above” and “unsure”.

   FineWeb-Edu is the template: an additive 0-5 “educational value” rubric, one point per satisfied criterion, then a keep/remove threshold at 3. Label 100-200 real items yourself, rewrite the guide, then label 50 more blind and check your own consistency, because people discover their real criteria while grading (Shankar et al., 2024-04-18). Store each row with label_source (outcome, LLM jury, human or synthetic), guide_version and the time the decision would have been made.

2. Pick a teacher you are allowed to learn from. Anthropic, Google and OpenAI restrict using their outputs to build “competing” models (OpenAI’s exception covers internal classifiers, wording seen only in a search snippet). DeepSeek’s terms explicitly permit “training other models (such as model distillation)” (DeepSeek ToS, 2026-04-29), and open weights under MIT or Apache-2.0 (DeepSeek V4, gpt-oss-120b, Qwen3.5) carry no such clause. Qwen3.8’s new community licence needs a separate deal for Model-as-a-Service businesses. The practical rule: label data you will sell with DeepSeek or an open-weight teacher, use frontier labs for evaluation and adjudication of a small subset, and tag every label with the model ID so you can re-label later. See data rights.

3. Label with a batch API. A batch API takes thousands of requests at once, returns them within hours, and charges half price. Anthropic’s batches hold up to 100,000 requests, “most” finish within an hour, they expire after 24 hours and results are kept 29 days (Anthropic batch docs, read 2026-10-28). Prompt caching stores your long, fixed instructions so later requests pay a fraction for them; Anthropic says the cache discount stacks with the batch discount.

   Worked example (Claude Opus 5.5 batch, prices as of October 2026): a 300-token cached guide at $0.10 per million tokens costs $0.00003; 300 fresh tokens at $2 per million cost $0.0006; a 20-token JSON label at $10 per million costs $0.0002. Total: $0.00083 per item, $8.30 per 10,000.

   Cost per 1 million items, derived from the Anthropic, OpenAI, Gemini, DeepSeek and DeepInfra pricing pages (read 2026-10-28), with that cached prefix, 20 output tokens for a label or 200 with a short rationale, and no hidden reasoning:

   Labeller (batch or off-peak)	Label only	With rationale
   GPT-5-nano	$12	$48
   Gemini 2.5 Flash-Lite	$22	$58
   Claude Haiku 5.5 or GPT-6-luna	$22	$66
   DeepSeek-V4-Flash (open weights, on DeepInfra)	$36	$68
   DeepSeek flash, off-peak (no batch tier)	$58	$166
   Gemini 3.8 Flash (price doubles 2027-01-01)	$161	$499
   Qwen3.5-397B (open weights, on DeepInfra)	$261	$801
   Sonnet 5.5, GPT-6.1-sol or Gemini 4 Argon	$415	$1,315
   Claude Opus 5.5	$830	$2,630
   Claude Fable 5.1 or GPT-6-astra	about $2,040-2,150	about $6,540-6,650

   Two traps. Thinking tokens: reasoning models bill hidden reasoning as output, and 1,000 extra per item adds $10,000 per million on Opus 5.5 batch, more than the labels themselves, so turn reasoning off or cap it and buy rationales only on a 10% subset. Tokenizers: Claude 4.7 and later produce “approximately 30% more tokens for the same text”, so compare cost per item, not per token.

   # Label one item with a typed schema via any OpenAI-compatible endpoint.
   from typing import Literal
   from openai import OpenAI
   from pydantic import BaseModel, ConfigDict
   
   class Triage(BaseModel):
       model_config = ConfigDict(extra="forbid")  # strict JSON schema
       label: Literal["duplicate", "needs_info", "valid_bug", "unsure"]
       confidence: Literal["low", "medium", "high"]
   
   GUIDE = open("label_guide_v1.md").read()  # long and fixed: first, cached
   client = OpenAI()  # set base_url for DeepSeek or a self-hosted teacher
   
   def label(item: str, model: str) -> Triage:
       schema = {"name": "triage", "strict": True,
                 "schema": Triage.model_json_schema()}
       r = client.chat.completions.create(
           model=model, temperature=0,
           messages=[{"role": "system", "content": GUIDE},
                     {"role": "user", "content": item}],
           response_format={"type": "json_schema", "json_schema": schema})
       return Triage.model_validate_json(r.choices[0].message.content)

   At scale, write the same request bodies to JSONL and submit them to the provider’s batch endpoint.

4. Use a jury, and build a gold set. A jury is several models from different families labelling independently. A panel of smaller models beat one large judge at over 7x lower cost (Verga et al., 2024-04-29). Diversity matters because LLM errors are correlated: across 350+ models, two models agree 60% of the time when both are wrong (Kim et al., 2025-06-09). The recipe: three families at temperature 0; keep unanimous, high-confidence labels; send 2-of-3 splits to a stronger adjudicator; send three-way splits to yourself, because they usually mean the guide has a hole. For 80,000 items × 3 cheap models that is about $23 in batch, plus about $61 to adjudicate 16,000 splits with a mid-tier model.

   # Keep unanimous high-confidence jury labels; route the rest.
   import pandas as pd
   
   votes = pd.read_parquet("votes.parquet")  # item_id, model, label, confidence
   wide = votes.pivot(index="item_id", columns="model", values="label")
   top = wide.mode(axis=1)[0]  # majority label per item
   agree = wide.eq(top, axis=0).sum(axis=1) / wide.notna().sum(axis=1)
   high = votes.confidence.eq("high").groupby(votes.item_id).all()
   keep = agree.eq(1.0) & high.reindex(agree.index, fill_value=False)
   accepted = top[keep]                                  # train on these
   adjudicate = agree[(agree < 1.0) & (agree >= 0.5)].index  # stronger model
   to_human = agree[agree < 0.5].index                   # guide gap

   A gold set is a sample labelled carefully by you or by a trusted outcome, used to measure everything else. To estimate an accuracy p within ±e at 95% confidence you need n = 1.96² × p(1-p) / e² items. With p = 0.5 and e = 0.05: 3.84 × 0.25 ÷ 0.0025 ≈ 385 items for ±5 points. With p = 0.9 and e = 0.02: about 865 for ±2 points. Outcome-labelled items are free gold, as long as the outcome post-dates the teacher’s training data.

5. Choose the backbone. Recall from how decision models work that a decision model is a shared backbone plus a scoring head per answer type. Two shapes fit a bootstrapper (estimates from the reference design):

   	Encoder plus pairwise head	Small decoder scoring options
   Backbone	Ettin-150M or ModernBERT-base (English), mmBERT-base (multilingual), up to about 400M	Qwen3-0.6B or 1.7B (NanoJev’s base); Qwen3.5-0.8B for long context
   Options go in as	one sequence: options and questions with marker tokens, then the state (up to 8K tokens)	a shared state prefix, then block-masked suffixes per question and option
   Cost per training run	$3-15 (150M), $10-40 (400M)	$10-45 (0.6B), $30-130 (1.7B)
   Latency	about 20 ms on a GPU; 130-208 ms on a CPU (GLiNER2, 205M)	tens of ms on a GPU; 177-1,049 ms unoptimised on Apple MPS (NanoJev)
   Weak spots	little world knowledge; 8K context (32K with UltraBERT)	2-4x slower at equal quality; label-token biases until fine-tuned; start from a base checkpoint, not a chat one

   Start with the encoder: with identical data, “a 400M encoder outperforms a 1B decoder on MNLI” (a sentence-pair reasoning benchmark) (Ettin, 2025-07-15). Move to the decoder when states exceed 8K tokens, when questions need world knowledge, or when options change per request and must be read from context, which is the case Jev targets.

6. Train. Three paths, cheapest first.

   (a) A plain encoder fine-tune. About 13 minutes on a $0.34/h RTX 4090 for 100k examples × 3 epochs (derived), or about 3.2 hours on an M4 Pro laptop (from 26 examples/s measured for this course under heavy background load, so an idle Mac may well be faster).

   # Fully fine-tune a 150M encoder as a 5-way classifier (Transformers).
   from datasets import load_dataset
   from transformers import (AutoModelForSequenceClassification,
                             AutoTokenizer, Trainer, TrainingArguments)
   
   name = "answerdotai/ModernBERT-base"
   tok = AutoTokenizer.from_pretrained(name)
   ds = load_dataset("parquet", data_files={"train": "train.parquet",
                                            "validation": "val.parquet"})
   ds = ds.map(lambda b: tok(b["text"], truncation=True, max_length=128),
               batched=True)  # rows need "text" and an int "label"
   model = AutoModelForSequenceClassification.from_pretrained(
       name, num_labels=5)
   args = TrainingArguments(output_dir="out", num_train_epochs=3,
                            learning_rate=3e-5, eval_strategy="epoch",
                            per_device_train_batch_size=32)
   trainer = Trainer(model=model, args=args, train_dataset=ds["train"],
                     eval_dataset=ds["validation"], processing_class=tok)
   trainer.train()
   val_logits = trainer.predict(ds["validation"]).predictions  # step 7

   (b) Unsloth’s decision trainer for a small decoder. It trains a LoRA adapter, a small low-rank patch on frozen weights (the bootstrapper’s stack explains it). Its page claims Qwen3.5-0.8B went from 36% to 73% on a typed-decisions benchmark using 4 GB and 42 minutes, and Qwen3.5-2B reached 81% on its test set with 8 GB in 40 minutes (vendor claims; the GPU is not named, and a T4 runs the free Kaggle notebook). The calls below are as written on its docs page (read 2026-10-28); each training row holds state, questions and gold.

   # Unsloth decision model: load, add LoRA, train, calibrate.
   from unsloth import FastDecisionModel, DecisionTrainer
   from transformers import TrainingArguments
   from datasets import load_dataset
   
   model, tok = FastDecisionModel.from_pretrained(
       model_name="unsloth/Qwen3.5-4B", max_seq_length=2048, load_in_4bit=True)
   model = FastDecisionModel.get_peft_model(
       model, r=16, lora_alpha=16, lora_dropout=0,
       use_gradient_checkpointing="unsloth", random_state=3407)
   rows = load_dataset("LocalLLaMA/typed-decisions", "all", split="train")
   items, report = FastDecisionModel.build_dataset(rows, tok, model)
   train, held = FastDecisionModel.split_holdout(items, seed=3407)
   trainer = DecisionTrainer(
       model=model, processing_class=tok, train_dataset=train,
       eval_dataset=held, args=TrainingArguments(
           output_dir="out", per_device_train_batch_size=2,
           gradient_accumulation_steps=4, max_steps=240,
           learning_rate=2e-5))  # page's code; its guide says 2e-4
   trainer.train()
   print(FastDecisionModel.calibrate(model, tok, held))  # accuracy, ECE, loss

   (c) MLX on your Mac. mlx-lm trains on JSONL in a data folder holding train.jsonl (and optionally valid.jsonl); a model path that points to a 4-bit model makes it QLoRA (LoRA on a 4-bit base) (mlx-lm LORA.md). Train the decoder to write the answer’s label token, then read that token’s probability instead of generating (generating vs deciding). LoRA on Qwen3-0.6B ran at about 1,380 tokens/s on the M4 Pro (measured).

   # data/train.jsonl, one object per line:
   {"prompt": "<state> <question> A) ... B) ...", "completion": " B"}
   
   # LoRA on a small decoder; loss on the completion only
   mlx_lm.lora --model Qwen/Qwen3-0.6B --train --data data \
     --batch-size 8 --iters 600 --mask-prompt
   
   # QLoRA on a 4-bit base, checkpointing to save memory
   mlx_lm.lora --model mlx-community/Qwen3-4B-4bit --train \
     --data data --grad-checkpoint
   
   # Merge the adapter into the base weights
   mlx_lm.fuse --model Qwen/Qwen3-0.6B

   On rented NVIDIA GPUs, TRL’s SFTConfig(packing=True, packing_strategy="bfd") packs short examples into full rows (it needs FlashAttention, a fast attention kernel). Whatever the path, shuffle option order and vary the state’s format (JSON, YAML, key-value lines) during training so the model doesn’t learn position or formatting shortcuts.

7. Calibrate with temperature scaling. A model is calibrated when its 0.8s come true about 80% of the time (calibration). Temperature scaling divides every logit (the raw score before softmax) by one number T, fitted on held-out data; it works “surprisingly well” (Guo et al., 2017-06-14). Worked example: logits [2.0, 1.0, 0.1] give probabilities [0.66, 0.24, 0.10]. With T = 2 they become [1.0, 0.5, 0.05], so exp gives [2.72, 1.65, 1.05], summing to 5.42, and the probabilities soften to [0.50, 0.30, 0.19]. The ranking never changes; only the confidence does.

   # Fit one temperature T on held-out logits by minimising log loss.
   import numpy as np
   
   def log_loss(logits, y, T):
       z = logits / T
       z = z - z.max(axis=1, keepdims=True)  # numerical safety
       logp = z - np.log(np.exp(z).sum(axis=1, keepdims=True))
       return -logp[np.arange(len(y)), y].mean()
   
   def fit_temperature(logits, y):
       grid = np.linspace(0.25, 5.0, 96)  # steps of 0.05; T = 1 is no change
       return grid[np.argmin([log_loss(logits, y, T) for T in grid])]
   
   T = fit_temperature(val_logits, val_labels)  # (n, k) logits, int labels
   probs = np.exp(test_logits / T)
   probs /= probs.sum(axis=1, keepdims=True)

   Fit one T per question type. The open clone decider-2b fitted 2.465 for Choice, 2.833 for Score and 3.185 for yes/no; on 399 yes/no items its expected calibration error (ECE, the average gap between stated confidence and actual accuracy) was 0.341 at its shipped setting and 0.095 at the fitted temperature. Yet its author kept the sharper default, because on a held-out set the JevBench-style score fell from 29.6 to -40.1 at the fitted temperature (decider-2b card, read 2026-10-28). Know which scoring rule you are optimising.

8. Evaluate against the teacher and against outcomes. Keep two scoreboards. Agreement with the teacher tells you how well you copied; accuracy against gold and outcome labels tells you the truth. Split by time (the latest 20% as test) and by entity (package, repo, customer), and hold out whole question schemas to measure transfer to new tasks (verifiable rewards and evals). Report accuracy, macro-F1 (F1 averaged over classes, so rare classes count), log loss, the Brier score (mean squared error of the probabilities), ECE, accuracy on the 80% of items you are most confident about, and consistency when you shuffle options or reword the question. Rewording matters: when Arize reversed a hallucination question, Liquid’s d1 fell to an ROC AUC (how well scores rank right answers above wrong ones; 0.5 is chance) of 0.17 while Jev, Clef and OpenAI held at 0.89-0.93 (Arize, 2026-10-07). Finally, compare with a zero-shot Haiku-class model on the same 500 gold items (under $1), and remember that even famous test sets carry at least 3.3% label errors on average (Northcutt et al., 2021-03-26). The thresholds and fallbacks that turn these numbers into automation are in from probability to automation.

C4 · L2 Container · Distillation pipeline

+-------------------+   +-------------------+   +-------------------+
| [Person]          |   | [External system] |   | [External system] |
| You               |   | World outcomes    |   | Teacher LLM APIs  |
| guide + gold set  |   | CI, bug trackers  |   | batch, half price |
+-------------------+   +-------------------+   +-------------------+
          |                       |                  ^         |
          | gold labels           | outcome labels   | items   | labels
          | (CSV)                 | (REST/JSON)      | (JSONL) |
..........|.......................|..................|.........|.......
:         v                       v                  |         v      :
:  +---------------------------------+ items +----------------------+ :
:  | [Database: Parquet]             |------>| [Container: Python]  | :
:  | Item store                      |<------| Labeller + jury      | :
:  | items, votes, gold, outcomes    | votes | batch job            | :
:  +---------------------------------+       +----------------------+ :
:         |                                                           :
:         | train / val / test splits (Parquet)                       :
:         v                                                           :
:  +----------------------+  weights  +---------------------------+   :
:  | [Container: GPU/Mac] |---------->| [Container: Python]       |   :
:  | Trainer              |  (files)  | Calibrator + eval harness |   :
:  | Transformers/Unsloth |           | fits T, scores test set   |   :
:  +----------------------+           +---------------------------+   :
:                                                 |                   :
:........ Distillation pipeline [system] .........|...................:
                                                  | model, T and
                                                  v report (files)
                                       +-------------------+
                                       | [Software system] |
                                       | Decision service  |
                                       | (next lesson)     |
                                       +-------------------+

Three sources of labels (you, the world, the teachers) land in one item store; a batch job runs the jury, a trainer on your Mac or a rented GPU learns from the agreed labels, and a separate calibrator fits T and scores the held-out test before anything ships. New to these boxes? See how to read the diagrams.

Within weeks of Jev’s launch, small teams published working clones, each a recipe you can read.

• NanoJev (MIT) adds decision heads to Qwen3-0.6B: Choice scores 2-255 candidates with a shared head and a softmax, yes/no is a sigmoid, Score is a distribution over 2-10 levels plus its expectation. It trained on about 10,900 stored questions for four narrow game tasks, so it shows the mechanism rather than broad accuracy, and a shared-prefix cache with block attention masks gave a 2.84x speedup at 64 candidates with logit drift under 1e-5 (NanoJev; PR #14, 2026-09-21). It proves the wiring fits in a small repo.
• decider-2b (Apache-2.0): Qwen3.5-2B-Base plus a rank-64 LoRA, 39,500 text and 18,129 image rows, 56 minutes on one B300 GPU, and about 95,000 downloads in the month to 2026-10-28 (model card). It proves one person can ship a clone in an hour of compute. Its card also flags that some training sets (IconQA, ScienceQA, TQA) are non-commercial: a licence-chain warning for everyone.
• Unsloth’s decision trainer turns the recipe into a library call (step 6b), with a calibration step that reports ECE; vendor numbers, and a page that contradicts itself on LoRA rank and learning rate.
• Cloudflare’s Clef shows the industrial version: a frozen backbone, a rank-256 LoRA and a routing head, trained with label-smoothed cross-entropy plus a Brier term on synthetic data, then reinforcement learning (remio).
• Cygnet (blockbrain), a frozen Gemma-4-12B with a decision head, tied for first on JevBench 1.5 at 73.70, above Jev 1.13 at 72.13 (BenchLM).

Sometimes the teacher isn’t an LLM but a simulator. A PFN (prior-data fitted network, from in-context learning and PFNs) is a transformer pre-trained on millions of synthetic datasets drawn from a prior, a recipe for generating plausible datasets. At use time it reads your labelled rows as context and predicts new rows in one forward pass, with no training.

The recipe, from the original paper (Müller et al., 2021-12-20):

1. Write a prior you can sample from, such as random causal graphs that generate tables.
2. Sample a dataset, hide some labels, and train the transformer to predict them from the rest.
3. Repeat millions of times. That loss equals the expected KL divergence to the true Bayesian answer under the prior, up to a constant, so the network learns to do Bayes in one pass.

The toy version runs in about a minute. nanoTabPFN is “Train your own small TabPFN in less than 500 LOC and a few minutes”: a 3-layer transformer with 96-dimensional embeddings, 4 attention heads and a 192-wide MLP, trained for 2,500 steps of 32 datasets on a prior dump of 150-row, 5-feature tables (nanoTabPFN README, Apache-2.0, read 2026-10-28). Its paper reports it “converged after one minute on a single NVIDIA GeForce RTX 2080 Ti”, beat classic baselines (k-nearest neighbours, decision trees and random forests) on small binary tasks from TabArena, and used about 160,000x less compute than TabPFN v2 (Pfefferle et al., 2025-11-05).

git clone https://github.com/automl/nanoTabPFN && cd nanoTabPFN
pip install numpy torch schedulefree h5py scikit-learn openml seaborn
curl -L -o 300k_150x5_2.h5 "<prior dump link from the README>"
python train.py

Then climb the ladder:

TabICLv2 is the bootstrapper’s prize: BSD-3 licensed, it released its synthetic-data generator and pre-training code, and untuned it beats tuned Real-TabPFN-2.5 on TabArena, a public tabular benchmark, narrowly (Elo 1591 against 1587 on the official board). Prior Labs did not release TabPFN v2’s generator. That is the lesson: compute is cheap, the prior is the moat. A vertical PFN starts from nanoTabPFN or TabICLv2 code with your domain’s simulator as the prior (see vertical foundation models).

• Reinforcement learning on outcomes. When the world supplies outcomes (did the build pass?), RL whose reward is a proper scoring rule such as the Brier score provably yields calibration as well as accuracy (RLCR, 2025-07-22). Jev’s “Reinforcement Learning for Calibrated Decisions” claims this direction but is undocumented, and for one-pass distribution outputs it may add little over supervised log loss on outcomes.
• “Competing” is undefined in the frontier labs’ terms, so whether a narrow classifier competes is a legal question, not a technical one.
• Calibration is fragile under rewording and distribution shift; recalibrate on each customer’s data.
• Language-conditioned PFNs, models that read both labelled rows and a natural-language question, are an open direction: no published source combines them for typed decisions yet.

• FineWeb-Edu dataset card — the clearest large-scale example of “LLM labels, small classifier, run on everything”.
• Synthetic data: save money, time and carbon — the RoBERTa-matches-GPT-4 distillation case study with costs.
• Verga et al., Replacing judges with juries — why a panel of small, different models beats one big judge.
• Guo et al., On calibration of modern neural networks — the paper behind temperature scaling.
• Unsloth, Train your own decision model — the library path for a decoder decision model.
• Pfefferle et al., nanoTabPFN — train a PFN yourself in minutes and see why the prior matters.