Serving for a fraction of a cent

The idea in one line: Cost per decision is the price of an hour divided by the decisions you serve in that hour; a small encoder on one CPU core makes that under a dollar per million, thousands of times below flagship LLMs, but only 10-800 times below the cheapest LLM APIs.

The prompt behind this course sets the bar at “orders of magnitude less inference compute”, cheap enough “to run on everything”. This lesson checks that claim with real prices and measured speeds, and it comes out true against flagship models and only partly true against the cheapest tiers. Knowing exactly where the gap is 10,000x and where it is 10x is what lets you pitch honestly and price without losing money.

The 10x levers in serving are model shape (an encoder that reads once instead of a decoder that writes JSON), utilisation (an idle GPU costs 20x more per decision), and the fallback rate to an expensive model. Quantization, thread counts and runtime flags are mostly 10% polish, and one popular trick turned out to be no faster at all.

Think of a taxi meter: you pay for time, not distance. A server bills by the hour, so what matters is how many decisions fit in that hour.

From generating vs deciding: prefill is the single pass in which a model reads its input, and decode is writing output one token at a time, each step waiting for the last. A decision model that scores known options needs only prefill. A model that writes a JSON answer pays for decode too.

cost per decision = price per hour
                    / (decisions per hour x utilisation)

compute per decision ~ 2 x parameters x input tokens   (FLOPs)

In words: halve the time per decision and you halve the cost; leave the machine idle 80% of the time and you pay 5x. The second line, from transformers in plain words, says compute grows with model size times tokens read.

Worked example. A 150M encoder reading 128 tokens needs 2 × 150M × 128 ≈ 3.8e10 FLOPs. A 1.24B decoder reading 500 tokens needs about 1.24e12, 32x more (derived). On the CPU, the encoder took 49.7 ms on one thread (measured). Modal charges $0.0000131 per core-second plus $0.00000222 per GiB-second, so one core with 1 GiB is $0.055 an hour (Modal pricing, read 2026-10-28). One core does 3,600 ÷ 0.0497 ≈ 72,400 decisions an hour, so a million cost $0.055 ÷ 72,400 × 1,000,000 ≈ $0.76.

# Cost per 1M decisions from a price per hour and a measured speed.
def cost_per_million(price_per_hour, seconds_per_decision, utilisation=1.0):
    decisions_per_hour = 3600 / seconds_per_decision * utilisation
    return price_per_hour / decisions_per_hour * 1_000_000

core_hour = (0.0000131 + 0.00000222) * 3600  # Modal: 1 core + 1 GiB, ~$0.055
print(cost_per_million(core_hour, 0.0497))   # encoder, 128 tokens: ~$0.76
print(cost_per_million(core_hour, 0.244))    # encoder, 512 tokens: ~$3.74
print(cost_per_million(core_hour, 1.51))     # 1B decoder, 256 tokens: ~$23
print(cost_per_million(core_hour, 500 / 170 + 30 / 56))  # 500 + JSON: ~$53
print(cost_per_million(core_hour, 0.0497, utilisation=0.2))  # idle: ~$3.81

Quantization stores a model’s weights in fewer bits, like rounding prices to whole dollars to fit a smaller receipt. The mechanism: pick a scale, store each weight as a small integer, and multiply back by the scale when computing. Worked example: with a scale of 0.0005, the weight 0.0137 becomes 0.0137 ÷ 0.0005 = 27.4, stored as the integer 27; reading it back gives 27 × 0.0005 = 0.0135, an error of 0.0002. An 8-bit integer takes 1 byte against 4 for a 32-bit float, so the file shrinks about 4x: ModernBERT-base went from 571 MB (fp32) to 144 MB (int8) (measured).

The vocabulary you’ll meet:

• ONNX is an open file format for trained models, and ONNX Runtime is an engine that runs it on CPUs and GPUs. llama.cpp is an engine for running LLMs on ordinary hardware, and Metal is Apple’s GPU interface.
• int8 dynamic quantization (ONNX Runtime) computes the activation scales during each inference: more accurate, with some overhead. Static quantization fixes them beforehand from calibration data (ONNX Runtime docs, read 2026-10-28).
• GGUF is llama.cpp’s file format; Q8_0 is its 8-bit type and Q4_K_M a 4-bit one.
• GPTQ and AWQ are 4-bit weight-only methods mostly used on GPUs.

What the measurements showed for scoring-only models:

Why: reading a prompt is compute-bound, and Q8 is cheaper to unpack than Q4, so heavier quantization helps memory and decode, not decision latency. For a model that only scores options, use Q8_0. Keep the disagreement visible on int8: ONNX Runtime’s docs say Arm processors can generally get better int8 performance, but warn “it is not rare to get worse performance on old devices”; on this Arm chip int8 bought no speed. x86 CPUs with VNNI (instructions that speed up int8 dot products) are where int8 is expected to help; they were not measured here. Int8 still matters for size: a 4x smaller download is what makes a browser model practical.

# Shrink an encoder to int8 and serve it on one CPU thread (ONNX Runtime).
import numpy as np
import onnxruntime as ort
from onnxruntime.quantization import quantize_dynamic

quantize_dynamic("model.onnx", "model_int8.onnx")  # ~4x smaller file

so = ort.SessionOptions()
so.intra_op_num_threads = 1  # many 1-thread workers beat 1 wide worker
so.inter_op_num_threads = 1
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
sess = ort.InferenceSession("model.onnx", so,
                            providers=["CPUExecutionProvider"])
ids = np.random.randint(0, 50_000, size=(1, 128), dtype=np.int64)
# Input names depend on the export: check sess.get_inputs().
logits = sess.run(None, {"input_ids": ids,
                         "attention_mask": np.ones_like(ids)})[0]

Latency per decision, what one call waits for:

Two lessons from the CPU rows. The encoder needs about 12-15x fewer core-seconds than the 1B decoder for the same text; more threads narrow the latency gap to about 4x but not the cost gap. And the encoder scales poorly with threads at batch size 1, so for throughput run many single-thread workers. Scoring options in one pass instead of writing JSON saves about 0.2 s per decision on a 1B model, roughly halving its latency.

A small GPU (derived). ModernBERT-base reads about 147,000 tokens/s on an RTX 4090 (ModernBERT, Table 2). That is about 1,150 decisions/s at 128 tokens, so $0.08-0.21 per million at $0.34-0.89/h, if you keep it busy. Assuming a 1.24B decoder achieves the same FLOP rate (about 44 TFLOP/s) gives about 35 decisions/s at 500 tokens, $2.7-7.0 per million.

Serverless platforms start a container when a request arrives. A cold start is the delay while a fresh container boots and loads the model. Modal containers boot in “about one second”, and its default 60-second idle window before scaling down is billed (Modal cold start guide); RunPod’s default idle timeout is 5 seconds (RunPod docs). Modal says GPU memory snapshots cut the start of vLLM (a GPU serving engine) from 45 s to 5 s (Modal blog, 2025-07-30, vendor claim). For a 150M encoder (about 600 MB in fp32), the cold start is dominated by the container and Python, not the weights: keep one warm CPU worker instead of a GPU.

Edge and browser. transformers.js runs ONNX models in the browser: on the CPU through WebAssembly with q8 as the default, or on WebGPU, which its docs call “still experimental in many browsers”; it supports ModernBERT and Qwen3.5 and publishes no speed numbers (transformers.js docs, read 2026-10-28). Ship an int8 ModernBERT-base (about 144 MB) or a tiny Ettin-17M or -32M encoder. Inference then costs you $0, because the user’s device does the work; the binding cost is download size.

Context models need one trick. TabPFN re-reads its labelled rows on every call unless you cache them. With 500 rows of context on CPU, one prediction took 1.29 s uncached and 88 ms with fit_with_cache; with 5,000 rows, 50.7 s against 66 ms (measured). Batched, it costs 2-4 ms per row. The package refuses more than 1,000 context rows on CPU for v2 unless you set TABPFN_ALLOW_CPU_LARGE_DATASET=1.

# TabPFN v2 on CPU: cache the fitted context, then batch the queries.
from tabpfn import TabPFNClassifier

clf = TabPFNClassifier(model_path="tabpfn-v2-classifier.ckpt",  # v2: commercial OK
                       device="cpu", n_estimators=1,
                       fit_mode="fit_with_cache")
clf.fit(X_context, y_context)         # your labelled rows become context
proba = clf.predict_proba(X_queries)  # send about 1,000 rows per call

Assumptions (change them and the ratios move, but the orders of magnitude don’t):

• One decision is 500 input tokens (a 300-token instruction, schema and options prefix plus a 200-token item) and 30 output tokens of JSON ({label, probability}), with no reasoning tokens. API rows use real-time standard prices; decision APIs bill input only.
• A fine-tuned specialist doesn’t need the 300-token instruction, because the task lives in its weights, so encoder rows read 128-512 tokens.
• Self-hosted CPU is Modal’s $0.055 per core-hour with the measured M4 Pro speeds; x86 cloud cores may be up to 2x slower or faster. Self-hosted GPU is an RTX 4090 at $0.34-0.89/h.
• Self-hosted rows assume full utilisation; the next column shows 20% (pay for idle time: ×5).

Sources: Anthropic, OpenAI, Gemini, DeepInfra and Modal pricing (read 2026-10-28); RunPod (updated 2026-10-08); Jev’s price from TypeSafe AI (2026-09-15); Decision-1 from InfoWorld (2026-10-13); OpenAI Decisions from DataCamp (2026-10-13); Decider Flash from UsagePricing.

Reading the table honestly:

• Against flagship and Sonnet-class JSON, the specialist is three to four orders of magnitude cheaper. Sonnet-class at $1,300 against the encoder on CPU ($0.76-3.74) is 350-1,700x; against the encoder on a busy 4090 ($0.08-0.84) it is 1,500-16,000x. Latency drops from seconds to tens of milliseconds.
• Against the small frontier tier, the gap is one to two orders, not four. Haiku-class at $35-65 against the encoder is 10-800x (about 10-85x if you self-host on CPU). That is still large, but it is not 10,000x.
• Against decision APIs at the price floor, self-hosting barely wins on price. Decider Flash at $5 is only 1.3-6.6x above the CPU encoder; Jev’s $21 is within about 1-3x of batched or cached Haiku. Here the case rests on latency, calibration, privacy, on-premises deployment and owning your margin, not price alone.
• A self-hosted decoder only wins on a busy GPU. On serverless CPU, a 1B decoder costs about the same as the Haiku-class APIs.
• Accuracy is the caveat under every row. The table assumes the specialist matches the teacher on your task. Verify that on your gold set (distill a decision model) before you quote any ratio.

Vendor headlines need the same care: Jev’s “444.6x cheaper” doesn’t reproduce from the example figures on its own launch post (they give about 171x by one calculation), the post itself calls its ratios “likely on the higher end of real world gains” (TypeSafe AI, 2026-09-15), and users reported a median 30x saving (InfoQ, 2026-10-01).

The fallback dominates the bill. Production systems send low-confidence cases to a bigger model (from probability to automation); Browserbase put Jev behind a 0.7 threshold with an LLM fallback and cut median latency from 1.97 s to 0.46 s (Browserbase). Illustration: if 80% of decisions stay on a $3.74 encoder and 20% fall back to a $65 Haiku-class call, the blend is 0.8 × 3.74 + 0.2 × 65 ≈ $16 per million, and $13 of it is the fallback. Moving the automation rate from 80% to 90% saves more than any quantization.

Price per decision fell to the floor within weeks of Jev’s launch: $0.042 per million input tokens for Jev, Decision-1 and d1-ultra, then $0.02 and $0.01 from Perplexity. Because decision models only prefill and are priced per input token, serving cost scales with parameter count, and that sets the margin. Gross margin is revenue minus the direct cost of serving it, divided by revenue.

Estimate on rented H100s (derived from the landscape research; assumptions: $3.99 per GPU-hour (Lambda), dense peaks of 989.5 TFLOPS in BF16 and 1,979 in FP8, the 16- and 8-bit number formats (NVIDIA), 40% MFU (the share of that peak actually used), no idle time):

Worked example (2B in FP8). Each token costs 2 × 2B = 4e9 FLOPs, so a million tokens is 4e15 FLOPs. At 40% of 1,979 TFLOPS, the H100 does about 7.9e14 FLOPs a second, so a million tokens take about 5 seconds: 5 ÷ 3,600 × $3.99 ≈ $0.0056. At a $0.042 price that leaves (0.042 - 0.0056) ÷ 0.042 ≈ 87%.

At 50% real utilisation, double every cost. Prompt caching (many questions over one cached state), mixture-of-experts backbones (which use only part of their weights per token) and owned hardware lower it; whether Qwen3.8-27B, the base of several 27B decision models, is dense is not stated in any source found. Still, a $0.02 price on a 27B model looks like a loss leader or a platform play. For an independent seller, shrinking the model while holding accuracy is a 10x lever on margin. For scale, Bessemer’s 2025 data put fast-growing AI apps at about 25% gross margin, often negative, and steadier ones at about 60% (Bessemer, 2025-08-13).

C4 · L2 Container · Decision service

+---------------------+                       +---------------------+
| [External system]   |                       | [External system]   |
| Your application    |                       | Frontier LLM API    |
| typed questions     |                       | hard cases, labels  |
+---------------------+                       +---------------------+
     |          ^                                ^            ^
     | state +  | choice + p                     | fallback   | 1% re-
     | questions| (HTTPS/JSON)                   | (HTTPS)    | labels
.....|..........|................................|............|........
:    v          |                                |            |       :
:  +---------------------+  below threshold      |            |       :
:  | [Container: Python] |-----------------------+            |       :
:  | Decision API        |                                    |       :
:  | thresholds, routing |----------------+                   |       :
:  +---------------------+  writes (SQL)  |                   |       :
:     |          ^                        v                   |       :
:     | score    | logits    +---------------------+          |       :
:     v (HTTP)   |           | [Database: SQLite]  |          |       :
:  +---------------------+   | Decision log        |          |       :
:  | [Container: ONNX]   |   | inputs, p, outcomes |          |       :
:  | Model workers, CPU  |   +---------------------+          |       :
:  | 150M encoder        |              | reads (SQL)         |       :
:  +---------------------+              v                     |       :
:                            +---------------------+  batch   |       :
:                            | [Container: Python] |----------+       :
:                            | Drift job, monthly  |  (JSONL)         :
:                            +---------------------+                  :
:...................... Decision service [system] ....................:

Your application sends a state and typed questions; the Decision API asks single-thread ONNX workers for logits, applies calibrated thresholds, and sends only low-confidence cases to a frontier model. Every decision and its later outcome lands in a log, and a monthly job re-labels about 1% of traffic with a teacher in batch to catch drift (for a million decisions a month, 10,000 labels cost about $8-26 with Opus batch). New to these boxes? See how to read the diagrams.

• One pass for many questions. Packing every option into one sequence keeps cost nearly flat as options grow: GLiClass loses about 7% throughput from 1 to 128 labels, while a pairwise cross-encoder, which reads the input once per option, gets about 52x slower (GLiClass, 2025-08-11). Decoders get the same effect from a shared-prefix cache: NanoJev measured 2.84x at 64 candidates (PR #14).
• Prefill-only serving on GPUs: batch the prefill, cache the shared state, and quantize to FP8; cost then scales with parameters times tokens.
• Distilling context models: Prior Labs ships an engine that turns TabPFN-2.5 into a compact MLP or tree ensemble with “orders-of-magnitude lower” latency (vendor claim, no numbers).
• Open questions: int8 speed on x86 CPUs with VNNI was not measured; independent frontier-API latency under load is scarce; WebGPU speed for transformers.js is unpublished; Clef’s price is reported as both $0.09 and $0.24 per million for its flash model; and whether the 27B decision models are dense changes the margin maths.

• ONNX Runtime, quantization — dynamic versus static int8, and when it can be slower.
• Warner et al., ModernBERT — throughput tables that anchor every GPU number here.
• Valdes Gonzalez, cost-aware classification study — fine-tuned encoders against GPT-4o and Claude prompting, with 2026 prices and latencies.
• Arize, decision model benchmark — the first independent head-to-head of hosted decision APIs, including the rewording failure.
• transformers.js — running ONNX decision models in the browser.
• Modal, cold starts — what serverless start-up and idle billing really cost.