How decision models work

The idea in one line: A decision model is a scorer, not a writer: one shared network reads the state, the question and every allowed answer in a single forward pass, gives each answer one number, and a small head per answer type turns those numbers into probabilities.

Two design choices account for nearly all the gains in this field, and neither needs a bigger model.

1. Score instead of generate. A February 2026 study found fine-tuned BERT-family encoders matched or beat GPT-4o and Claude Sonnet 4.5 prompting on 3 of 4 classification datasets, at $5 to $33 per million requests against $192 to $2,702 (Valdes Gonzalez, 2026-02-06).
2. Pack every option into one pass. GLiClass lost about 7% throughput going from 1 to 128 labels, while a model that runs once per label got about 52x slower (GLiClass, 2025-08-11).

Almost everything else (which encoder, which fix for label bias) is 10% polish, and none of the pieces is new.

A refresher from Part 1 (generating vs deciding):

• A logit is the raw score a model gives one possible output. Softmax turns logits into probabilities that sum to 1, p_i = exp(z_i) / Σ exp(z_j), so [2.0, 1.0, 0.1] becomes 0.66, 0.24 and 0.10. A sigmoid, 1 / (1 + exp(−z)), turns one logit into a yes-probability: 0 gives 0.5, 2 gives 0.88.
• A chat model generates: pick a token, append it, repeat, each step waiting for the last. A decision model scores: the answers are known, so one forward pass (a single run of the network over the input) yields a number for each.

That pass costs about 2 × N FLOPs (single adds or multiplies) per input token for a model with N parameters (Kaplan et al., 2020-01-23). A 150M encoder reading 1,000 tokens: 2 × 1.5e8 × 1,000 = 3e11 FLOPs, once. A 0.6B decoder: 1.2e12 for the same read, plus 40 sequential steps if it then writes a 40-token JSON answer.

Natural language inference (NLI) is deciding whether a premise sentence entails, contradicts or is neutral towards a hypothesis. Yin, Hay and Roth recast classification as NLI: the text is the premise, and each label becomes a hypothesis such as “This text is about politics.” (Yin et al., 2019-08-31). Zero-shot means the model was never trained on those labels; it only reads their names.

facebook/bart-large-mnli (0.4B parameters, 2.4M downloads in the last month) made this popular (model card, read 2026-10-28). Its flaw is cost: one pass per label, so 100 labels cost 100 passes. Better data raised accuracy: Laurer’s DeBERTa-v3-large “zeroshot-v2.0” reached a mean macro-F1 (a per-class accuracy score averaged so rare classes count equally) of about 0.68 across 28 tasks, against 0.497 for bart-large-mnli (model card, metrics as of 2024-04-03).

The lesson that carried forward: phrase every task as the same kind of question, and one output head serves them all.

A cross-encoder reads the state and one option together, so each can look at the other: most accurate, one pass per pair. A bi-encoder turns each into a vector separately and compares them, so option vectors can be cached. Sentence-BERT’s example: finding the most similar pair among 10,000 sentences takes about 65 hours with a BERT cross-encoder and about 5 seconds with bi-encoder embeddings (Reimers & Gurevych, 2019). Late interaction (ColBERT) keeps one vector per token and is about 100x faster than BERT rerankers at competitive quality (Khattab & Zaharia, 2020-04-27).

Bi-encoders struggle with negation, conditions and “none of the above”, so decision models let options look at the state (GLiClass). Options are short and few (Jev allows up to 255), so that is affordable.

A decoder is a left-to-right transformer, the shape of every chat model. A logprob is the log of the probability it gives a token. Three ways to read a decision out of one:

1. Cloze scoring: add up the logprobs of each option’s text after the question. One pass per option, unless passes share a prefix.
2. Multiple-choice prompting: list options as A, B, C and read the next-token probability of each letter. One pass, but only if the model reliably ties options to letters, a skill that “varies greatly” (Robinson et al., 2022-10-22).
3. Yes/no scoring: ask a closed question and read P(“yes”) from one position.

The traps, each with a known fix:

• Surface-form competition. “computer” and “PC” split the probability, so the right idea can lose. Domain-conditional PMI divides each option’s probability by its probability in a neutral context (Holtzman et al., EMNLP 2021).
• Label bias from the prompt. Few-shot GPT-3 swung “from near chance to near state-of-the-art” with prompt format and example order. Contextual calibration asks about a content-free input such as “N/A” and rescales so it gets an even spread: up to 30.0 points of accuracy (Zhao et al., 2021-02-19).
• First-token probabilities disagree with the written answer more than 60% of the time in instruction-tuned models (Wang et al., 2024-02-22).
• Chat tuning damages calibration. Victor Dibia measured ECE (the average gap between stated confidence and accuracy) of 0.02 to 0.05 for base Qwen2.5 and 0.10 to 0.18 for the instruct versions (Dibia, 2026-09-28).

The verdict: zero-shot logprob scoring of a chat model is a prototype, not a product. Fine-tuning on the exact scoring format removes most traps, and a fitted temperature fixes most of the rest (calibration).

The lineage closest to Jev: a decoder turned into a typed classifier by reading one next-token distribution, then fine-tuned for that reading.

• monoT5 reads a relevance word’s logits as a probability (Nogueira et al., 2020-03-14).
• Qwen3-Reranker softmaxes the last-position logits for “yes” and “no”. The 0.6B scored 65.80 on the MTEB-R retrieval benchmark against 57.03 for BGE-reranker-v2-m3 (Qwen3 Embedding, 2025-06-05).
• Llama Guard scores “the probability of the first output token being ‘unsafe’” (Inan et al., 2023-12-07); ShieldGemma’s 2B reached 0.887 AU-PRC (area under the precision-recall curve; 1.0 is perfect) on OpenAI’s moderation set (Zeng et al., 2024-07-31).
• Qwen3Guard adds a three-way answer (safe, controversial, unsafe) and a “Stream” variant with a dedicated token-level classification head (Qwen3Guard, 2025-10-16): the step past reading the vocabulary.

# Illustrative: the yes/no readout behind rerankers and guards.
# The prompt is simplified; real models use a fixed chat template.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen3-0.6B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

def p_yes(state: str, question: str) -> float:
    prompt = f"{state}\nQuestion: {question}\nAnswer yes or no:"
    ids = tok(prompt, return_tensors="pt")
    with torch.no_grad():
        logits = model(**ids).logits[0, -1]       # next-token logits
    yes = tok.convert_tokens_to_ids("yes")
    no = tok.convert_tokens_to_ids("no")
    pair = torch.stack([logits[no], logits[yes]])
    return torch.softmax(pair, dim=0)[1].item()  # P(yes) out of no/yes

A reward model outputs one number for “how good is this answer”, learned from people’s preferences. The Bradley-Terry model says answer 1 beats answer 2 with probability exp(r1) / (exp(r1) + exp(r2)), where r is a hidden quality score (DPO paper, 2023-05-29). With r1 = 1 and r2 = 0: 2.72 / 3.72 = 0.73, so a one-point lead means a 73% chance of winning.

InstructGPT used such a model to fine-tune GPT-3, and its 1.3B model was preferred over the 175B GPT-3 (Ouyang et al., 2022-03-04). Underneath, it is a decoder with one scalar head instead of a vocabulary; a Choice head is the K-way version, a softmax over K option scores. Judges complete the family: GenRM scores P(“Yes”) to “Is the answer correct?”, lifting best-of-N on GSM8K maths from 73% to 93.4% (Zhang et al., 2024-08-27); G-Eval weights each 1-to-5 rating by its probability (Liu et al., 2023-03-29). That expected value over score buckets is exactly Jev’s Score.

Pairwise scoring makes a 255-option Choice cost 255 passes. Four tricks fix that.

GLiClass puts every label in the same input, each marked with a <<LABEL>> token, before the text. One pass yields one vector per label, and a small scorer turns each into a logit (GLiClass). Examples per second (ex/s) on one A6000 GPU at batch size 1:

The price: options see each other and their order, which brings position bias, so shuffle option order in training. GLiNER2 (205M) packs the same way: 163 ms against 6,758 ms pairwise at 20 labels (GLiNER2, 2025-07-24).

The KV cache stores each token’s attention keys and values for reuse (see transformers in plain words). Read [state][question] once and reuse its cache for every option. vLLM does this across requests (vLLM docs); SGLang reports up to 6.4x throughput (2023-12-12), Hydragen up to 32x (2024-02-07). This is the likely reason Jev’s latency barely moves from 1 to 64 questions (0.10 s to 0.12 s, Vals AI, 2026-10-06).

An attention mask is the table of which tokens may look at which. Pack all options after one shared prefix, and mask so each option sees the prefix and itself but never another option. Each option’s position numbers restart where the prefix ended, so every option “thinks” it came straight after the prefix.

Block mask: prefix p1 p2 p3, then option A (a1 a2) and option B (b1 b2).
Row = the token looking; column = a token it may look at (1 = yes).

        p1 p2 p3 a1 a2 b1 b2    position id
  p1     1  .  .  .  .  .  .    0
  p2     1  1  .  .  .  .  .    1
  p3     1  1  1  .  .  .  .    2
  a1     1  1  1  1  .  .  .    3
  a2     1  1  1  1  1  .  .    4
  b1     1  1  1  .  .  1  .    3   <- restarts after the prefix
  b2     1  1  1  .  .  1  1    4

NanoJev, an open “nano replica” of Jev on Qwen3-0.6B, implemented exactly this (NanoJev PR #14, 2026-09-21). On a Mac’s GPU (Apple MPS, float32):

Logits matched the one-at-a-time version to within 1e-5. The same PR reports the downside: 2 candidates ran slower (0.65x), and yes/no batches slowed to 0.40x. It pays off only with many options or a long prefix.

Label options A, B, C and read one position’s probability over the letters. On a 7B instruct model, Dibia found this within about 2 points of full-option scoring up to 25 options, 5.4 points behind at 77, and 25x cheaper at 20 questions with 77 options (0.54 s against 13.6 s) (Dibia). An unconfirmed hypothesis: Jev’s 255-option cap is one byte of codes, and its first-option bias is what letter read-outs produce.

A head is the small final layer that turns the backbone’s vectors into a typed answer.

Ordinal means the levels have an order, so being one level off should cost less than three (CORAL, 2019-01-20). For squared-error metrics, the best single reading of a distribution is its mean, not its most likely level (Lukasik et al., 2024-03-07), which is why Score returns an average.

# Illustrative: the three heads, applied to logits a backbone produced
import numpy as np

def choice_head(option_logits, T=1.0):        # softmax over K options
    z = np.asarray(option_logits) / T
    e = np.exp(z - z.max())
    return e / e.sum()

def noul_head(logit, T=1.0):                  # sigmoid on one logit
    return 1 / (1 + np.exp(-logit / T))

def score_head(level_logits, T=1.0):          # K levels, then the mean
    p = choice_head(level_logits, T)
    return p, float((np.arange(len(p)) * p).sum())

print(choice_head([2.0, 1.0, 0.1]).round(2))  # [0.66 0.24 0.1 ]
print(round(noul_head(2.0), 2))               # 0.88
print(score_head([0.0, 0.7, -5.0]))           # about (0.33, 0.67, 0.0), 0.67

T is the temperature, one number that softens or sharpens every distribution; fitting it is the main calibration fix.

A separate confidence. The distribution captures aleatoric uncertainty, ambiguity in the case itself. A separate confidence should capture epistemic uncertainty, the model’s ignorance of inputs like this. Cheapest first: read it off the distribution (top probability, or entropy H = −Σ p log p, 0.69 for (0.5, 0.5) and 0.33 for (0.9, 0.1)), as Jev’s confidence does, though that cannot tell “ambiguous” from “unfamiliar”; train a correctness head (DeVries & Taylor, 2018-02-13); use the disagreement of 3 to 5 ensemble members (Lakshminarayanan et al., 2016-12-05); or use conformal set size (from probability to automation).

An encoder reads the whole input with attention in both directions: the natural shape for scoring, though it cannot generate. At equal size, encoders win at classification. Ettin trained matched pairs on the same public data: the 150M encoder scored 89.2 on the MNLI benchmark against 88.2 for the 400M decoder, and a 400M encoder beats a 1B decoder (Ettin, 2025-07-15; HF blog).

Leading open encoders as of October 2026, scored on GLUE, a 0-100 suite of English understanding tasks:

mmBERT (2025-09-08) covers 1,800+ languages. Two counterpoints. Trained on the same data, the older DeBERTa-v3 “remains superior in sample efficiency” to ModernBERT, whose wins are speed and long context (Antoun et al., 2025-04-11); GLiClass agrees (0.719 against 0.620 average zero-shot F1). And decoders know more of the world: on the Decision Index as of October 2026, 202 of 240+ entries are decoders and 33 encoders, and the best encoder (GLiClass v4 edge, 0.4B) scores 31.21 against Jev’s 60.24 (Decision Index). Picking between top encoders moves GLUE by 0.5 to 2 points, which is polish; picking encoder or decoder for your task is the real lever.

On general classification, small zero-shot models trail frontier LLMs: GPT-4o averaged 0.84 zero-shot accuracy, GLiNER2 (205M) 0.72 and a DeBERTa-v3 NLI model 0.69 (GLiNER2). But frontier zero-shot can collapse on label schemes it has not seen. Small models fine-tuned on each dataset (minus a 200-example test set) against GPT-4 zero-shot, accuracy (Bucher & Martini, 2024-06-12):

On German anger, GPT-4 did far worse than blind guessing. RoBERTa-large passed zero-shot BART on every dataset after about 200 training examples, with a “sweet spot” at 200 to 500. On AG News, fine-tuned RoBERTa scored 94.63 macro-F1 at $10 per million requests against 87.93 for GPT-4o zero-shot at $276 (Valdes Gonzalez).

Distillation, training a small student on a big teacher’s labels, makes those labels cheap. A RoBERTa-base student trained on 1,811 Mixtral-labelled sentences matched GPT-4’s 94% accuracy at about $2.70 per million inferences against $3,061, after under $1 of fine-tuning (Laurer, 2024-02-16).

Answering new questions zero-shot depends on a different lever: how many distinct tasks the model trained on. FLAN used 60+ datasets, Super-NaturalInstructions 1,616 tasks, and Laurer’s universal classifier 33 datasets and 389 classes for +9.4% zero-shot (Laurer et al., 2023-12-29). For a Jev-like model: thousands of question schemas beat millions of examples of a few.

C4 · L3 Component · Decision model server (reference design)

............... Decision model server [container] ................
:                                                                :
:  request (JSON): state + questions + options or levels         :
:        |                                                       :
:        v                                                       :
:  +-----------------------+  text    +------------------------+ :
:  | [Component: Python]   |--------->| [Component: tokenizer] | :
:  | Serializer            | (UTF-8)  | Packer                 | :
:  | canonical JSON, maths |          | one row: state + every | :
:  | done in code          |          | question and option    | :
:  +-----------------------+          +------------------------+ :
:                                       | ids + block mask       :
:                                       | (tensors)              :
:                                       v                        :
:               +--------------------------------------+         :
:               | [Component: PyTorch]                 |         :
:               | Backbone: encoder or small decoder   |         :
:               | one forward pass for everything      |         :
:               +--------------------------------------+         :
:                     | one vector per question and option       :
:                     | (tensors), sent to its type's head       :
:                     v                                          :
:   +----------------------+    +----------------------+         :
:   | [Component: PyTorch] |    | [Component: PyTorch] |         :
:   | Choice head          |    | Noul head            |         :
:   | MLP, then softmax    |    | sigmoid on one logit |         :
:   +----------------------+    +----------------------+         :
:   +----------------------+    +----------------------+         :
:   | [Component: PyTorch] |    | [Component: PyTorch] |         :
:   | Score head           |    | Confidence head      |         :
:   | K levels, then mean  |    | spread or learned    |         :
:   +----------------------+    +----------------------+         :
:                     | logits (tensors)                         :
:                     v                                          :
:            +-----------------------------------+               :
:            | [Component: Python]               |               :
:            | Calibrator: one temperature per   |               :
:            | type, plus conformal thresholds   |               :
:            +-----------------------------------+               :
:                     | distributions (JSON)                     :
:                     v                                          :
:                  response                                      :
:................................................................:

One request flows through a serializer, a packer, a single backbone pass, the heads for each answer type, and a calibrator. Only the heads differ by type; everything else is shared across tasks. New to C4? See how to read the diagrams.

• Serializer: one canonical layout with stable key order, trained on the same state as JSON, YAML and key-value lines. Compute counts and version comparisons in code; prompt format alone swung GPT-3.5 by up to 40% on one task (He et al., 2024-11-15).
• Choice scorer: s_i = MLP([h_q; h_i; h_q ⊙ h_i]), then p = softmax(s / T). In words: join the question vector, the option vector and their element-wise product, and a small shared network gives one score per option, so new options need no new weights.
• Training: log loss on labels, a pull toward teacher probabilities where you have them, optionally a Brier term; shuffle option order every time.
• Evaluation: hold out whole question schemas to measure true zero-shot transfer; report accuracy, log loss, Brier, ECE, accuracy at 80% coverage, and whether answers survive shuffled options.

Three variants. Training and serving costs are the researcher’s estimates, assuming H100 rental at $1.99 to $4.29 an hour (IntuitionLabs, 2026-10-11):

For comparison, Jev at list price is about $42 per million such decisions. Build order: start with (c) feeding (a): label 100K to 200K decisions with a cheap teacher, re-label the 10 to 20% where teachers disagree with a strong one, hand-check 1,000, train Ettin-150M, about $300 to $800 in all (distill a decision model). Move to (b) when states pass 8K tokens or need world knowledge. Add RL only when outcomes come from the world (did the build pass?): RLCR showed a proper-scoring-rule reward yields calibrated models (Damani et al., 2025-07-22). The researcher’s analysis adds a caution: for one-pass distribution outputs, such a reward has the same optimum as plain supervised log loss, so ask what Jev’s “RLCD” adds.

• Open blueprints exist. H2O-Lightning-4B (Apache-2.0) is a Qwen3.5-4B fine-tune on 2.22M examples where “each decision is one forward pass and one output token”. Its “two-way reading”, attention in both directions over the input on some layers, scored 206 of 231 on JevBench against 202 one-way, by H2O’s own count (H2O card, v1.4.1).
• Hybrid backbones break block masks. Qwen3.5 mixes linear-attention layers that carry a running state, not a per-token cache (HF card); the researcher’s untested idea is to fork that state at the end of the prefix.
• What is Jev? Size, backbone and RLCD are undisclosed; the leading guess is a fine-tuned open-weight multimodal decoder read in parallel per question.

• GLiClass (Knowledgator), 2025-08-11 — packing every label into one encoder pass, with throughput tables.
• NanoJev PR #14 (2026-09-21) — a small, readable implementation of block-masked option scoring.
• Calibrate Before Use, 2021-02-19 — why prompt-read probabilities are biased, and the fix.
• Bucher & Martini, 2024-06-12 — when a few hundred labels beat GPT-4 zero-shot, with learning curves.
• Ettin, 2025-07-15 — matched encoders and decoders on the same data.
• Victor Dibia: How Jev works (2026-09-28) — letter read-outs against full-option scoring, measured.