Generating vs deciding

The idea in one line: A generator writes its answer one token at a time and you parse what it wrote; a decider reads the question and every allowed answer once and hands back a probability for each, which is cheaper, faster and impossible to get off-schema.

Most people first meet language models as chat: you ask, it writes. If your real question is “which of these five failure classes explains this build log?”, writing is wasted work. You wanted five numbers.

The cost gap is not 10%. A February 2026 study found fine-tuned small encoder models matched or beat GPT-4o and Claude Sonnet 4.5 prompting on 3 of 4 text-classification datasets, at $5-33 per million requests against $192-2,702 (Valdes Gonzalez, 2026-02-06). That is one to two orders of magnitude, and it comes mostly from one design choice: score the options instead of generating an answer. This lesson shows you where that factor comes from, number by number, and what you give up for it.

A token is a chunk of text, often a word or part of one, that a model reads and writes. A vocabulary is the fixed list of all tokens a model knows. In Jev’s documentation, a quick-start request with three questions used 428 input tokens (TypeSafe docs: Quick start, read 2026-10-28). An independent benchmark measured an average of 950 input tokens per decision (JevBench, v1.6.1, 2026-10-06).

At the end of reading its input, a language model produces one raw score for every token in its vocabulary. That raw score is a logit: any number, positive or negative, where bigger means “more likely”. Logits are not probabilities yet. They don’t sit between 0 and 1, and they don’t add up to 1.

Softmax turns a list of logits into probabilities that add up to 1:

p_i = exp(z_i) / (exp(z_1) + exp(z_2) + ... + exp(z_n))

In words: raise e (about 2.718) to the power of each logit, which makes every number positive and stretches the gaps, then divide each by the total so they sum to 1.

Worked example. A model reads a failed build log and gives three logits for the three answers you allowed:

(The logits and the arithmetic come from the course’s decision-architectures research notes.)

Two properties you can check by hand:

1. Only the gaps matter. Add 5 to every logit and you get the same 0.66, 0.24 and 0.10, because every exp gets multiplied by the same exp(5) and it cancels in the division.
2. Dividing by a temperature softens the answer. Divide every logit by T = 2 and they become 1.0, 0.5 and 0.05. Their exps are 2.72, 1.65 and 1.05, which sum to 5.42, so the probabilities become 0.50, 0.30 and 0.19. Same ranking, less certainty. Fitting that one number T on held-out labels is the most common way to fix an overconfident model; you will meet it again in calibration.

For a single yes-or-no answer there is a one-logit version, the sigmoid: p = 1 / (1 + exp(−z)). A logit of 0 gives 0.5; a logit of 2 gives 0.88.

# Softmax and temperature by hand (numbers from the table above).
import math

def softmax(logits, T=1.0):
    exps = [math.exp(z / T) for z in logits]
    total = sum(exps)
    return [round(e / total, 2) for e in exps]

logits = [2.0, 1.0, 0.1]
print(softmax(logits))                  # [0.66, 0.24, 0.1]
print(softmax([z + 5 for z in logits])) # same: only gaps matter
print(softmax(logits, T=2.0))           # [0.5, 0.3, 0.19]: softer

A chat model generates with an autoregressive loop. “Autoregressive” means each output depends on the outputs before it:

1. Read the whole input and compute a softmax over the entire vocabulary.
2. Pick one token from that distribution.
3. Append it to the input.
4. Go back to step 1, until the model emits a stop token.

Generating: one model pass per output token, each waits for the last

  prompt                    -> [model] -> {
  prompt {                  -> [model] -> "class"
  prompt {"class"           -> [model] -> :
  prompt {"class":          -> [model] -> "compile"
  ... more passes ...       -> [model] -> }
  then: parse the text, and hope it matches your schema

Inference has two phases with very different speeds (Splitwise, 2023-11-30):

• Prefill reads the input. All input tokens are processed together, in parallel, so it keeps the hardware busy. It is “compute-intensive”.
• Decode writes the output, one token per step. Each step must wait for the previous one, and each step re-reads all of the model’s weights from memory to produce a single token. It is “memory-intensive” and slow per token.

That is why APIs charge more for output. When Jev launched in September 2026, GPT-5.6 Terra was reported at $2 per million input tokens and $12 per million output tokens (The Register, 2026-09-16): output cost 6 times as much.

A 40-token JSON answer means 40 sequential decode steps after the prefill. A reasoning model that “thinks” first adds hundreds or thousands more.

Then you parse. The model might write "compile error" when your schema says "compile", or wrap the JSON in a sentence, or invent a fourth class. Constrained decoding, which forbids any token that would break the schema, fixes the format. It still gives you one sampled answer, not a distribution. And forcing a format has a measured cost: format restrictions made LLM reasoning worse in one 2024 study (Tam et al., “Let Me Speak Freely?”, 2024-08-05). Jev’s CEO put the deeper objection bluntly: “simply masking logits is insufficient because if ever a model was assigning probability to an invalid token, the model is by definition confused” (Hacker News, 2026-09-15).

When you already know the allowed answers, you don’t need the loop. You need one number per answer, and all of those numbers can come out of a single prefill.

Deciding: one model pass, one number per allowed answer

  state (the build log) + question + all allowed answers
                       |
                       v
                    [model]          one forward pass
                       |
                       v
  logits         2.0        1.0        0.1
  softmax        0.66       0.24       0.10
                 compile    missing    infra
                            dependency

There are three standard ways to read a decision out of a model (decision-architectures research notes):

1. Score each option’s text. Put an option after the question and read how probable the model finds that text. Simple, but naively one pass per option.
2. Multiple choice by letter. List all options as A, B, C and read the probabilities of the next token being “A”, “B” or “C”. One pass covers everything. It only works well if the model reliably ties each option to its letter, which “varies greatly” across models (Robinson et al., 2022-10-22).
3. Yes or no. Ask a closed question and read P(yes) = softmax([z_no, z_yes])[1] from one position. This is the production standard for binary decisions: Qwen3-Reranker, Google’s ShieldGemma and Meta’s Llama Guard all read their answer from one next-token distribution this way.

Here is what option 3 looks like in code. It is a sketch with a real library API, not a tuned product:

# Sketch: read P("yes") from one next-token distribution (no generation).
# Real transformers API; prompt and model choice are illustrative.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen3-0.6B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

prompt = ("Build log: fatal error: foo.h: No such file or directory\n"
          "Question: is this a missing build dependency? yes or no: ")
ids = tok(prompt, return_tensors="pt")
with torch.no_grad():
    logits = model(**ids).logits[0, -1]       # one forward pass
yes, no = tok.encode("yes")[0], tok.encode("no")[0]
p_yes = torch.softmax(logits[[no, yes]], dim=0)[1]
print(f"P(yes) = {p_yes:.2f}")

Nothing to parse. No off-schema answer is possible, because the model is never asked to write anything. And you get a full distribution, not one sample.

Use the rule of thumb from what a model is: reading costs about 2 × N FLOPs per input token for a model with N parameters (Kaplan et al., 2020-01-23). A FLOP is one arithmetic operation.

• A 150M-parameter encoder reading 1,000 tokens of state, question and options: 2 × 1.5e8 × 1,000 = 3e11 FLOPs, once. Done.
• A 0.6B decoder doing the same prefill: 1.2e12 FLOPs. Generating a 40-token answer then adds 40 sequential steps, each one waiting on memory.

Measured, not estimated:

Source: Valdes Gonzalez, 2026-02-06, at 2026-01-22 prices. On AG News the fine-tuned encoders were also more accurate (macro-F1 94.6 for RoBERTa against 87.9 for zero-shot GPT-4o).

Hosted decision models have pushed the price further. As of October 2026, Jev lists $0.042 per million input tokens with output free (TypeSafe docs: Models). JevBench’s 950-token average puts 1,000 decisions at about $0.04.

Scoring beats generating, but naive scoring has its own trap: one pass per option. The original NLI zero-shot classifier (facebook/bart-large-mnli) asks “does this text entail ‘This text is about X’?” once per label, so 100 labels cost 100 passes. A 255-option question would cost 255 passes.

The 2025-2026 move is to pack every option into one pass:

• Encoders. GLiClass writes all labels and the text into one input, and every label comes out of one pass. Going from 1 label to 128, its base model lost about 7% of its throughput (49.4 to 45.9 examples per second). A pairwise model that scores one label per pass got about 52 times slower (24.6 to 0.47) (GLiClass, 2025-08-11). At 20 labels, GLiNER2 took 163 ms where the pairwise approach took 6,758 ms (GLiNER2, 2025-07-24).
• Decoders. Read the shared state once, keep its working memory (the KV cache, stored attention data that later tokens can reuse), and attach every option after it with an attention mask that stops options from seeing each other. NanoJev, an open replica of Jev built on Qwen3-0.6B, measured 2.84x faster at 64 candidates with logits unchanged to within 1e-5, but 0.65x (slower) at 2 candidates (NanoJev PR #14, 2026-09-21). The trick pays off only with many options or a long shared state.
• Many questions about one state. The same idea applies across questions. Vals AI measured Jev at 0.10 s for 1 judgment per document and 0.12 s for 64 (Vals AI, 2026-10-06). TypeSafe’s own cookbook asks 13 questions about a 54,000-character article: one call cost $0.000497 and took 0.27 s, against $0.006090 and 2.71 s for 13 separate calls, with the same answers (TypeSafe cookbook). That’s 12.2x cheaper and 10x faster.

Why does this change cost by orders of magnitude rather than by a few percent? Because the savings multiply:

  no decode loop          the output costs nothing extra
x one pass for K options  K passes become ~1 (52x at K = 128)
x one state, Q questions  Q reads of the state become 1
x a small model           cost per token scales with size
  ---------------------------------------------------------
  = cost per decision falls by 10x, then 10x again

Each factor alone is modest; stacked, they are the gap between a few dollars and a few thousand dollars per million decisions. How the heads, packing and training fit together is the subject of how decision models work.

Keep the headline numbers honest. TypeSafe advertises Jev as “193.6x faster, 444.6x cheaper” on its own four workflow tests, which is a vendor claim, and TypeSafe itself calls those multipliers “on the higher end of real world gains”. An analysis of 12,759 launch tweets found a median user-reported speedup of 7x and a median cost saving of 30x (InfoQ, 2026-10-01). Real workflows do other work besides the decision.

Deciding is not free of problems. It swaps one set of failures for another.

The last row is a warning: zero-shot scoring of an off-the-shelf chat model is a prototype, not a product. Prompt wording alone moved few-shot GPT-3 accuracy “from near chance to near state-of-the-art”, and rescaling against a content-free “N/A” input improved accuracy by up to 30 points (Zhao et al., “Calibrate Before Use”, 2021-02-19). The fix is to fine-tune the model on the exact scoring format, which is what decision-model builders do.

When generating is still the right tool. When the answer is open-ended, when it needs knowledge the small model lacks, or when it needs several steps of reasoning. In a narrow test by Every, Fable 5.1 caught 7 of 7 planted writing defects where Jev caught 6 (NYU Shanghai RITS, 2026-09-16). The pattern the field is converging on: decide fast with a probability, and escalate to a generator only when the probability says “not sure”. That routing is the subject of from probability to automation.

• Cost-Aware Model Selection for Text Classification (Valdes Gonzalez, 2026): the accuracy, latency and price tables behind “one to two orders of magnitude”.
• GLiClass (Knowledgator, 2025): packing every label into one encoder pass, with the 7% versus 52x measurement.
• NanoJev PR #14 (2026): shared-prefix, block-masked option scoring on a 0.6B decoder, with honest speedups and slowdowns.
• Splitwise (Patel et al., 2023): why prefill and decode behave so differently on hardware.
• “My Answer is C” (Wang et al., 2024): why reading probabilities off a chat model needs care.