Calibration

The idea in one line: A model is calibrated when its probabilities mean what they say: of all the times it says 0.9, about 9 in 10 turn out right.

A calibrated probability is a number you can do arithmetic with: expected cost, expected error rate, how many cases a human must review. An uncalibrated one can still rank cases, but you cannot plan with it. In the next lesson the same model, with the same accuracy, either loses or saves money on every automated decision depending only on whether its 0.97 really means 97%.

The fix is often an order of magnitude for almost nothing. One fitted number, a temperature, took a ResNet-110’s calibration error on CIFAR-100 from 16.53% to 1.26% with accuracy unchanged (Guo et al., 2017-06-14). Simple recalibration took 18 guardrail models from 6-28% calibration error to under 2.5% (Cornacchia et al., ICML 2026 workshop). That is the cheapest 10x in machine learning, for a few hundred to a thousand labels.

A forecaster says “30% chance of rain” on 100 different days. If it rains on about 30 of them, the forecaster is calibrated. If it rains on 60, their 30% is a lie you cannot plan around, however good they are at telling wet days from dry ones.

Three properties get mixed up, so pin them down now:

• Calibration: stated probabilities match observed frequencies.
• Sharpness: forecasts commit (3% or 97%, not 50%). The governing principle is “maximizing the sharpness of the predictive distributions subject to calibration” (Gneiting, Balabdaoui & Raftery, 2007).
• Discrimination: scores rank right answers above wrong ones. It is usually measured by AUROC, the chance that a random positive case scores higher than a random negative one (1.0 is perfect, 0.5 is a coin).

Two toy models show they are different things:

Model A ranks perfectly but its numbers are useless as probabilities; model B is calibrated and useless. You want both, and the metrics below check both.

A reliability diagram groups predictions by stated confidence into bins and plots, for each bin, how often the model was actually right (scikit-learn calibration guide, v1.9.1, read 2026-10-28). Points on the diagonal are calibrated; points below it are overconfident (says more than it delivers); points above it are underconfident.

Take 20 made-up predictions in two bins:

Reliability check, 20 made-up predictions in two bins (# = 0.1)

bin   n    stated   observed accuracy        gap    reading
0.6   10   0.60     ######       0.60        0.00   on the diagonal
0.9   10   0.90     #######      0.70        0.20   overconfident

ECE = (10/20) x 0.00 + (10/20) x 0.20 = 0.10

The model says 0.9 on ten cases but gets only seven right. On average its stated confidence is 0.10 too high.

Expected calibration error (ECE) squashes the diagram into one number: the gap between confidence and accuracy in each bin, weighted by that bin’s share of predictions.

ECE = Σ over bins b of  (n_b / n) × |accuracy_b − confidence_b|

Guo et al. used 15 equal-width bins on [0, 1]; they also define maximum calibration error, the worst single bin, for high-stakes settings (Guo et al.). Always plot how many predictions fall in each bin next to the diagram: a big gap in a bin with three items means little.

# ECE of top-label confidences, equal-width or equal-mass bins
import numpy as np

def ece(conf, correct, n_bins=15, equal_mass=False):
    """conf: probability of the chosen answer; correct: 1 or 0."""
    conf, correct = np.asarray(conf, float), np.asarray(correct, float)
    if equal_mass:  # same number of examples per bin (less biased)
        edges = np.quantile(conf, np.linspace(0, 1, n_bins + 1))
    else:           # Guo et al. 2017: equal-width bins on [0, 1]
        edges = np.linspace(0, 1, n_bins + 1)
    idx = np.clip(np.searchsorted(edges, conf, side="right") - 1,
                  0, n_bins - 1)
    total = 0.0
    for b in range(n_bins):
        m = idx == b
        if m.any():  # |accuracy - confidence|, weighted by bin share
            total += m.mean() * abs(correct[m].mean() - conf[m].mean())
    return total

conf = [0.6] * 10 + [0.9] * 10
correct = [1] * 6 + [0] * 4 + [1] * 7 + [0] * 3
print(round(ece(conf, correct, n_bins=10), 2))  # 0.1, as worked by hand

ECE is the most quoted calibration number and the most criticised:

• It is biased, and the bias depends on how many examples you have. Equal-mass bins (the same count per bin) are less biased than equal-width ones (Roelofs et al., 2020-12-15).
• Changing the bins can reorder which method “wins”. “The rank ordering of recalibration methods is drastically affected” by which calibration measure you pick (Nixon et al., 2019-04-02).
• It jumps around as bins change. Smooth ECE replaces bins with kernel smoothing and ships as the relplot package with no settings to tune (Błasiok & Nakkiran, 2023-09-21).
• It can be gamed. Always predicting the base rate (model B above) scores ECE near 0. Report a proper scoring rule and a ranking metric beside it.
• Scaling fixes look better on binned ECE than they are (Kumar, Liang & Ma, 2019-09-23).

The noise floor is the ECE a perfectly calibrated model shows by chance on a finite test set. In the researcher’s own simulation (confidences uniform between 0.5 and 1, labels drawn from those exact probabilities, 10 equal-mass bins, 300 trials each), the mean ECE of a perfect model was:

So an ECE of 0.03 on 1,000 examples is indistinguishable from perfect. Never compare ECE against zero; compare it against the floor for your sample size.

A scoring rule gives a forecast a number once the outcome is known. It is proper if a forecaster gets the best expected score by reporting their true belief (Gneiting & Raftery, 2007-03-01). In plain words: a proper rule makes honesty the best strategy. That is why proper rules serve as training losses, as RL rewards, and as vendor comparisons that cannot be gamed.

The two you need, each for one yes/no case with outcome y (1 or 0) and forecast p:

• Brier score = (p − y)², averaged over cases. Bounded between 0 and 1. A coin-flip forecast of 0.5 scores 0.25.
• Log loss = −log p(what actually happened), averaged. Unbounded: one confident mistake costs a lot.

Worked numbers:

Now see why Brier is proper. Suppose you truly believe the chance is 0.7. Report 0.7, and your expected Brier is 0.7 × 0.09 + 0.3 × 0.49 = 0.21. Exaggerate to 0.9: 0.7 × 0.01 + 0.3 × 0.81 = 0.25. Claim certainty at 1.0: 0.7 × 0 + 0.3 × 1 = 0.30. Lower is better, so the honest 0.7 wins.

Brier splits into three parts, the Murphy decomposition: reliability (miscalibration), minus resolution (how usefully forecasts vary), plus uncertainty (noise you cannot remove). So a lower Brier does not by itself mean better calibration: a sharper, better-ranking model can win on Brier while being worse calibrated (scikit-learn). Report Brier and ECE and AUROC.

Guo et al. found that “modern neural networks, unlike those from a decade ago, are poorly calibrated”. The mechanism: networks keep lowering classification error after their log loss starts to overfit. On CIFAR-100, test error fell from 29% to 27% while log loss got worse, so “the network learns better classification accuracy at the expense of well-modeled probabilities” (Guo et al., 2017-06-14). The disagreement worth knowing: four years later, Vision Transformers and MLP-Mixers were “among the best calibrated”, and “architecture is a major determinant of calibration properties” (Minderer et al., 2021-06-15). The 2017 finding describes a training recipe and an era, not a law.

• GPT-4: “the pre-trained model is highly calibrated”. After post-training (labelled “ppo” in the figure), ECE on an MMLU subset rose from 0.007 to 0.074, about 10x worse (GPT-4 Technical Report, v1 2023-03-15).
• Anthropic’s 52B RLHF model was poorly calibrated, and “simply adjusting the temperature … to T = 2.5 largely fixes calibration issues”. Base models were “very well calibrated” on multiple-choice questions with visible lettered options (Kadavath et al., 2022-07-11).
• Small open models show the same pattern. Victor Dibia measured ECE of 0.02 to 0.05 for base Qwen2.5 against 0.10 to 0.18 for the instruct versions (Dibia, 2026-09-28).

Why? Pre-training minimises cross-entropy (log loss), which is a proper scoring rule, so it rewards honest next-token probabilities. Post-training “may deviate from cross-entropy in favor of reinforcement learning” (Kalai et al., 2025-09-04). Reward models add “an inherent bias towards high-confidence scores regardless of the actual quality of responses” (Leng et al., 2024-10-13). And benchmarks grade right or wrong with no credit for “I don’t know”, so models are “optimized to be good test-takers” (Kalai et al.).

Reasoning-model RL makes honesty about uncertainty worse still. Reinforcement fine-tuning “could reduce model refusal rates by more than 80%”; mixing 10% unanswerable problems into training largely restores them (Song, Shi & Zhao, 2025-05-20). Across 20 frontier models, reasoning fine-tuning “degrades abstention by 24% on average” (AbstentionBench, 2025-06-10). TypeSafe’s case for Jev starts here: it argues RLHF rewards “confident-sounding hallucinations” (TypeSafe docs: AI primer, read 2026-10-28).

Temperature scaling keeps the trained model frozen and divides all its logits (raw scores before softmax, from generating vs deciding) by one number T, fitted on held-out labelled data. T above 1 softens an overconfident model; the top answer never changes, so accuracy is untouched.

Worked example: logits [2.0, 1.0, 0.1] give probabilities 0.66, 0.24 and 0.10. Divide by T = 2 and you get [1.0, 0.5, 0.05], whose exponentials 2.72, 1.65 and 1.05 sum to 5.42, giving 0.50, 0.30 and 0.19. Same winner, humbler number.

# Fit one temperature T on held-out data by minimising log loss
import numpy as np
from scipy.optimize import minimize_scalar
from scipy.special import log_softmax, softmax

def fit_temperature(logits, labels):
    """Return T > 0 minimising the NLL of softmax(logits / T)."""
    def nll(log_t):  # optimise log T so T stays positive
        lp = log_softmax(logits / np.exp(log_t), axis=1)
        return -lp[np.arange(len(labels)), labels].mean()
    res = minimize_scalar(nll, bounds=(-3, 3), method="bounded")
    return float(np.exp(res.x))

# A black-box API that returns probabilities works too:
# logits = np.log(probs + 1e-12)   (log-probs are logits up to a constant)
# T = fit_temperature(val_logits, val_labels)   # T > 1: was overconfident
# test_probs = softmax(test_logits / T, axis=1) # same argmax and accuracy

On a synthetic, overconfident 10-class model (5,000 validation and 5,000 test items), this code found T = 2.96 and cut ECE from 0.326 to 0.015, with accuracy unchanged (the researcher’s own run).

How it compares with other post-hoc maps, ECE % from Guo et al.’s Table 1:

One parameter beats many-parameter maps when there are many classes: matrix scaling, which learns a full linear layer, made CIFAR-100 worse. Platt scaling (a two-parameter sigmoid) suits yes/no outputs; isotonic regression (any non-decreasing step function) “will perform as well as or better than sigmoid when there is enough data (greater than ~1000 samples)” (scikit-learn). Training-time options exist too: focal loss, label smoothing and deep ensembles, the last of which holds up best under distribution shift (Ovadia et al., 2019-06-06).

If the RL reward for a stated confidence q is a proper scoring rule, the model earns the most only by stating its true chance of being right. RLCR uses reward = 1[correct] − (q − 1[correct])²: correctness minus the Brier score. Worked: right at q = 0.8 earns 1 − 0.04 = 0.96; wrong at q = 0.8 earns 0 − 0.64 = −0.64; wrong at q = 1.0 earns −1. Bluffing costs.

On Qwen2.5-7B (Damani et al., 2025-07-22):

Same accuracy, 2.6x to 12x better calibration in domain, and better but still imperfect out of domain. Without RL, calibration tuning found “a thousand graded examples are sufficient to outperform baseline methods” with LoRA on an open model (Kapoor et al., 2024-06-12).

Jev’s own method, RLCD (“Reinforcement Learning for Calibrated Decisions”), is described by TypeSafe as “optimised to produce honest probabilities rather than answers a human rater likes” (noze.it, 2026-09-19, one source, quoting the vendor). No method paper or calibration metric is public.

There are three ways to get a confidence out of an LLM: read its token probabilities (logprobs); ask it to say a number (“confidence: 85%”), called verbalized confidence; or sample many answers and use their agreement rate. The evidence is split:

• For RLHF chat models, verbalized confidences “often reduce the expected calibration error by a relative 50%” versus their token probabilities (Tian et al., 2023-05-24).
• But verbalized confidence is overconfident, “possibly imitating human patterns” (Xiong et al., 2023-06-22). On Humanity’s Last Exam in January 2025, frontier models showed 73% to 89% RMS calibration error at 2.7% to 13.4% accuracy (HLE, 2025-01-24).
• The protocol decides the winner: changing how token probabilities are read flips which method scores better in 8 of 20 settings (Kim & Kang, 2026-05-26).
• Verbalized numbers cost output tokens, and 7% to 32% of small models’ JSON outputs were invalid in Dibia’s test.
• Many reasoning APIs no longer return logprobs at all. Azure lists logprobs as unsupported for OpenAI reasoning models other than GPT-6 Astra (Microsoft Learn, updated 2026-10-26).

A typed-decision API that returns a probability vector by design removes the elicitation problem, though not the need to check the numbers.

TypeSafe publishes no reliability diagram, no ECE and no Brier score (TypeSafe evals, read 2026-10-28). Its evals score agreement with labels averaged from two frontier models, which cannot show calibration against ground truth. What independent testers found:

Sources: Vals AI, 2026-10-06; Decision Index (data generated 2026-10-28); Dibia, 2026-09-28, citing a community benchmark.

Two more data points pull the other way. Emil Lindfors found Jev somewhat underconfident at both ends (98% agreement in the 0.9 to 1.0 bin), and that careful, longer wording raised ECE from 0.040 to 0.116 (Lindfors, 2026-09-18). The only TypeSafe-published figure Dibia found was 27 of 30 correct above 0.9 confidence on an older model, a 95% interval of 74% to 97%: too few cases to say much. Several open decision models score lower ECE on the Decision Index, such as Fastino GLiDE 1.2 at 0.020.

The lesson is not “Jev is calibrated” or “Jev is not”. It is that calibration is a property of a model on a task, and it moved tenfold between two of Vals’s tasks. Keep the noise floor in mind too: on a few hundred items, small ECE differences are noise, while a gap like 0.011 against 0.108 is not.

1. Sample real traffic, stratified by input type and difficulty, plus a random slice so overall numbers stay honest.
2. Use ground-truth labels, not LLM consensus. Frontier-model labels are a fine first pass, but have people adjudicate a few hundred to measure label noise. Calibration against noisy labels is calibration against the noise.
3. Split three ways: a calibration set (to fit the temperature), a threshold set (to choose operating points) and a frozen test set touched once per model version. Vals used a 30/70 split and checked the realised error on the held-out part.
4. Tag each item with its slice (source, language, time) and what a mistake costs. Check calibration per slice; a good average hides a bad subgroup.
5. Add a newer, time-shifted slice to catch drift, and re-check on every vendor model update. Pin the model version.

Score the API’s probabilities (or Jev’s Noul value), not its confidence field, which is a spread statistic, not a probability (anatomy of Jev).

• Accuracy, plus log loss and Brier.
• ECE with equal-mass bins (10 to 15), its noise floor, and smooth ECE.
• A reliability diagram with per-bin counts.
• AUROC for “is the top answer right?”.
• Risk-coverage and coverage at your target error rate, the automation number (next lesson).

Put error bars on all of it with the bootstrap: recompute the metric on many resamples of your test set, drawn with replacement. When comparing two models, resample items and score both on the same resample, so shared difficulty cancels.

# Bootstrap CI and the ECE noise floor (uses ece() from above)
import numpy as np

def bootstrap_ci(metric, *arrays, n_boot=2000, seed=0):
    """95% percentile CI: resample whole examples with replacement."""
    rng = np.random.default_rng(seed)
    n = len(arrays[0])
    stats = [metric(*(np.asarray(a)[i] for a in arrays))
             for i in (rng.integers(0, n, n) for _ in range(n_boot))]
    return np.percentile(stats, [2.5, 97.5])

def ece_noise_floor(conf, metric, n_sim=500, seed=0):
    """ECE a perfectly calibrated model with these confidences shows."""
    rng = np.random.default_rng(seed)
    conf = np.asarray(conf, float)
    sims = [metric(conf, rng.random(len(conf)) < conf)
            for _ in range(n_sim)]
    return np.percentile(sims, [50, 95])

# lo, hi = bootstrap_ci(ece, conf, correct)
# floor_median, floor_95 = ece_noise_floor(conf, ece)
# call it miscalibrated only if lo > floor_95

In the researcher’s own run, a model 5 points overconfident on 1,000 items had ECE 0.069 (95% CI 0.050 to 0.096), against a perfect model’s floor of 0.028 (median) and 0.039 (95th percentile). The interval clears the floor, so the miscalibration is real.

The standard error of a rate p on n items is sqrt(p(1 − p) / n), so a ±ε interval at 95% needs n = 1.96² × p(1 − p) / ε². Worked: for ±5 points at p = 0.5, n = 3.84 × 0.25 / 0.0025 = 384.2, so 385 items.

Ten times tighter error bars cost a hundred times the labels. So spend labels where decisions are made: the automated slice, whose error rate is your contract, and the band near your threshold.

• Is ECE the right metric? Widely reported, widely criticised, no consensus replacement. Smooth ECE and proper scores lead.
• Verbalized or token probabilities? Unresolved, and protocol-sensitive.
• Does reasoning help calibration? Better in 33 of 36 settings in one study (Yoon et al., 2025-05-20); worse with more thinking budget on expert judgments in another (Lacombe et al., 2025-08-20).
• Out-of-domain calibration improves with RLCR but stays far from in-domain (0.21 to 0.25 against 0.03). How far a vertical model’s calibration carries to a new customer’s data is open.
• Is Jev calibrated? Unknown in general; the method is unpublished, and the evidence varies by task.

• Guo et al., On Calibration of Modern Neural Networks, 2017-06-14 — the paper that made temperature scaling the default.
• Kadavath et al., Language Models (Mostly) Know What They Know, 2022-07-11 — base models calibrate, RLHF breaks it, T = 2.5 fixes it.
• Roelofs et al., Mitigating Bias in Calibration Error Estimation, 2020-12-15 — why binned ECE misleads and what to use instead.
• Damani et al., RLCR, 2025-07-22 — a proper scoring rule in the RL reward, with the numbers.
• scikit-learn: Probability calibration — the practical guide, with temperature, Platt and isotonic built in.
• Vals AI: An Independent Evaluation of TypeSafe’s Jev (2026-10-06) — calibration measured per task on a real decision API.