Transformers in plain words

The idea in one line: A transformer turns each token into a list of numbers, lets every token look at the others to update its numbers (attention), repeats that through a stack of layers, and reads the answer off the result; what it costs to run is set mostly by how many parameters it has and how many tokens it reads.

Nearly every model in this course is a transformer: Jev’s open replicas, the encoders behind zero-shot classifiers, TimesFM-3 and TabPFN. You don’t need to build one from scratch. You do need to know three things about one, because each moves cost or accuracy by large factors:

• Size sets the bill. Reading costs about 2 × N FLOPs per token for a model with N parameters. On the course’s research estimate, a 2-billion-parameter decision model sold at Jev’s price keeps about 87% gross margin, while a 27-billion-parameter one loses money on every call. That is not polish; it is whether the business exists.
• Shape sets the accuracy per dollar. With the same data and recipe, a 150M encoder beat a 400M decoder on a standard reasoning-about-sentences benchmark (Ettin, 2025-07-15). Picking the right shape beats picking a bigger model.
• Context sets the ceiling. How much a model can read at once limits what state you can hand it.

This lesson walks through one transformer from text in to scores out. Here is the whole path as a C4 component diagram (new to these? see how to read the diagrams):

C4 · L3 Component · Transformer model

................ Transformer model [container] ...............
:                                                            :
:  text in (UTF-8)                                           :
:     |                                                      :
:     v                                                      :
:  +------------------------+      +-----------------------+ :
:  | [Component: tokenizer] |      | [Component: table]    | :
:  | Tokenizer              |----->| Embedding lookup      | :
:  | text -> token ids      | ids  | one vector per token  | :
:  +------------------------+(int) +-----------------------+ :
:                                     | vectors (floats)     :
:                                     v                      :
:                      +------------------------------+      :
:                      | [Component: L layers]        |      :
:                      | Attention + feed-forward     |      :
:                      | each token updates its       |      :
:                      | vector from the others       |      :
:                      +------------------------------+      :
:                                     | vectors (floats)     :
:                                     v                      :
:                      +------------------------------+      :
:                      | [Component: head]            |      :
:                      | Output head                  |      :
:                      | vectors -> logits -> softmax |      :
:                      +------------------------------+      :
:............................................................:

Text becomes token IDs, IDs become vectors, a stack of layers refines the vectors, and a small head turns them into the scores you read. Everything except the head is shared by every task the model does.

A token is a chunk of text, often a word piece. The tokenizer splits text into tokens and gives each a number, its ID. Jev’s quick-start request was 428 tokens (TypeSafe docs: Quick start, read 2026-10-28).

An ID is just a label; “token 512” isn’t closer in meaning to “token 513”. So the model looks each ID up in a table and replaces it with a list of numbers. That list is an embedding (also called a vector), and the table is part of the model’s parameters, learned in training. Tokens used in similar ways end up with similar lists.

A toy with three numbers per token (the 2017 Transformer used 512, TimesFM-3 uses 1,280):

To compare two embeddings, take the dot product: multiply matching positions and add.

failed · error  = 0.9×0.8 + 0.1×0.2 + 0.0×0.1 =  0.74
failed · passed = 0.9×(−0.7) + 0.1×0.3 + 0×0  = −0.60

High means similar, negative means opposite. Remember the dot product; attention is built from it.

Learned word vectors are older than transformers. A 2013 Google paper showed high-quality word vectors could be learned from a 1.6-billion-word dataset in under a day (Mikolov et al., 2013-01-16).

The embedding table can be a large share of a small model. mmBERT, a multilingual encoder covering more than 1,800 languages, has 307M parameters in total but only 110M outside its embeddings (mmBERT, 2025-09-08). So about 197M, roughly two-thirds, is the vocabulary table.

The word “failed” means something different in “the akmod failed” and “the retry never failed”. To get the right meaning, each token needs to look at its neighbours. Attention is how it does that. It is the core idea of the 2017 paper that introduced the transformer, which described an architecture “based solely on attention mechanisms” (Vaswani et al., 2017-06-12).

Each token makes three vectors from its embedding, using learned parameters:

• a query: what am I looking for?
• a key: what do I contain, for others to match against?
• a value: what do I hand over if someone attends to me?

Then, for each token:

1. Dot its query with every token’s key. That gives one score per token: how relevant is that token to me?
2. Softmax the scores into weights that add up to 1.
3. Blend the value vectors using those weights. The blend is the token’s updated vector.

The paper’s formula packs all three steps into one line:

Attention(Q, K, V) = softmax(Q Kᵀ / √d_k) V

Q, K and V hold every token’s queries, keys and values; d_k is the length of a key. Dividing by √d_k shrinks all scores by the same factor, which works exactly like the softmax temperature in generating vs deciding.

The token “failed” attends over three tokens. We skip the √d_k division to keep the numbers round.

# Attention for one query token, by hand (toy 2-number vectors).
import numpy as np

q = np.array([1.0, 1.0])                 # query of "failed"
K = np.array([[1.5, 0.5],                # key of "akmod"
              [0.5, 0.5],                # key of "kernel"
              [0.1, 0.0]])               # key of "the"
V = np.array([[1.0, 0.0],                # value of "akmod"
              [0.0, 1.0],                # value of "kernel"
              [0.0, 0.0]])               # value of "the"

scores = K @ q                           # [2.0, 1.0, 0.1]
weights = np.exp(scores) / np.exp(scores).sum()   # softmax
print(weights.round(2))                  # [0.66 0.24 0.1 ]
print((weights @ V).round(2))            # [0.66 0.24]: new "failed"

The scores are 2.0, 1.0 and 0.1, the same three numbers as the softmax table in the previous lesson, so the weights are again 0.66, 0.24 and 0.10. “failed” now carries 66% of “akmod”’s value and 24% of “kernel”’s, and almost nothing from “the”. It has learned from context that this is a kernel-module failure.

All tokens do this at the same time. The paper’s Table 1 puts the number of sequential steps per attention layer at a constant, O(1), against O(n) for the older recurrent networks that read one word after another. That parallelism is what made transformers practical to train on big hardware: the 2017 model reached a then-record translation score after 3.5 days on eight GPUs.

The price is that every token scores every other token. With n tokens that is n × n scores per layer, which the paper writes as O(n²·d).

tokens read      scores per attention layer
      1,000      1,000 × 1,000   =         1,000,000
     32,000     32,000 × 32,000  =     1,024,000,000

Thirty-two times the input, about a thousand times the attention work. For short inputs this term is small next to the rest of the model, which is why the cost rule later in this lesson can ignore it. For long ones it is not.

A layer is one round of attention followed by a small per-token feed-forward network (a few multiplications by learned matrices). A transformer stacks many layers, and each refines every token’s vector a little more. The 2017 base model had a vector width of 512 and about 65 million parameters (Vaswani et al., Table 3). TimesFM-3 has 20 layers, a width of 1,280 and 16 attention “heads”, which are parallel copies of attention that can each look for different things, for 330 million parameters (research notes on in-context models, from Google Research, 2026-08-31 and the model card).

Transformers come in two main shapes, and the difference is one rule about who may look at whom. That rule is an attention mask: a table of which tokens may attend to which.

Who can look at whom (each row looks at the columns marked x)

Encoder (bidirectional)          Decoder (causal)
         the  akmod  failed               the  akmod  failed
the       x     x      x         the       x
akmod     x     x      x         akmod     x     x
failed    x     x      x         failed    x     x      x

• An encoder lets every token see every other token, before and after it. BERT (2018) made this famous; its abstract describes “deep bidirectional representations” that “can be fine-tuned with just one additional output layer” (Devlin et al., 2018-10-11). Encoders output one vector per token and are the natural shape for reading and scoring. They cannot write text.
• A decoder lets each token see only the tokens before it. That is what makes generation possible: the model predicts the next token from everything so far. Every chat model is a decoder. Decoders can still score, as the yes/no example in the previous lesson showed.

Which is better for decisions? A 2025 study trained matched encoder and decoder pairs on the same open data with the same recipe. The 150M encoder scored 89.2 on MNLI (a large test of whether one sentence follows from another) against 88.2 for the 400M decoder, and a 400M encoder beat a 1B decoder. Retraining a decoder as an encoder, or the reverse, on 50 billion more tokens “still generally trails” the native shape (Ettin paper, 2025-07-15; blog).

So why build decision models on decoders at all? Because the big open decoders carry far more world knowledge, and they handle long inputs. The research notes’ advice: start with an encoder; move to a 0.6-1.7B decoder when the state is very long or the question needs knowledge an encoder lacks.

A context window is the most tokens a model can read in one go. Some figures, all from the research notes:

Bigger is not automatically better. Attention cost grows with the square of the length, and Jev’s own documentation reports accuracy loss on large, noisy state. For a build log, the research notes’ practical rule is to cut it down in code first, for example to the failing stanza or the last N lines, and give the model only that.

Two numbers describe a model’s size, and people mix them up:

• Parameters are the stored dials, the numbers in the embedding table, attention and feed-forward matrices. They decide memory. At 2 bytes per parameter (the common 16-bit format), Qwen3-0.6B needs about 1.2 GB just for its weights.
• Compute is the arithmetic done per call, counted in FLOPs. It decides speed and cost.

They are linked by the rule of thumb from what a model is: reading costs about 2 × N FLOPs per token (Kaplan et al., 2020-01-23). Each parameter takes part in about one multiply and one add for every token.

The landscape research notes work this out for hosted decision models. Their assumptions: an H100 GPU rented on demand at $3.99 an hour (Lambda pricing, read 2026-10-28), a peak of 989.5 trillion FLOPs per second in 16-bit arithmetic (NVIDIA H100), and 40% MFU (model FLOPs utilization, the share of that peak you actually get). So the GPU does about 0.4 × 989.5e12 ≈ 3.96e14 useful FLOPs a second.

For a 9B-parameter model reading a million tokens:

FLOPs   = 2 × 9e9 × 1,000,000        = 1.8e16
seconds = 1.8e16 / 3.96e14           ≈ 45
dollars = 45 / 3,600 × $3.99         ≈ $0.050

Repeat for other sizes and switch to 8-bit arithmetic (FP8, about twice as fast), and you get the notes’ table. These are their own estimates, not published costs:

Gross margin is revenue minus the direct cost of serving it, divided by revenue. $0.042 per million input tokens is the list price of Jev, Microsoft Decision-1 and Liquid d1-ultra as of October 2026. The table is why the research notes say size is margin: when the market price sits at the floor, the seller with the smallest model that still holds accuracy survives. The full cost-per-decision story is in serving for a fraction of a cent.

• Train small models for longer. For a fixed training budget, Chinchilla (70B) beat Gopher (280B) by using more data per parameter (Hoffmann et al., 2022-03-29). When a model will be used heavily, it now pays to go much further: Llama 3 8B’s compute-optimal point was about 200 billion tokens, and Meta trained it on 15 trillion, about 75 times more, and kept seeing gains (Meta, 2024-04-18). Over-training small models is standard practice in 2026.
• Small keeps getting better. One study finds the parameters needed to match a reference model halve about every three months (Xiao et al., 2024-12-05). Datadog’s Toto-2.0-4m forecaster fits in 16 MB of weights and beats a 205M-parameter Chronos model on one benchmark (research notes on in-context models).
• Modern encoders are close to each other. ModernBERT, Ettin, NeoBERT and the new UltraBERT differ by 0.5-2 points on the GLUE benchmark. Picking among them is 10% polish; picking encoder versus decoder, and training on the right task mix, matters more.
• Attention is being replaced in places. Qwen3.5-0.8B mixes Gated DeltaNet layers, a cheaper “linear attention” that carries a running summary instead of storing every token’s keys and values, with ordinary attention layers (HF card, 2026-02). The decision-architectures notes point out, as their own inference, that the masking tricks for scoring many options in one pass don’t carry over directly to those layers. The notes found no published method for combining the two.
• Open: whether 32K-token encoders hold up once someone replicates UltraBERT, and how much long context actually helps decisions versus cutting the state down in code.

• Attention Is All You Need (Vaswani et al., 2017): the original transformer; Table 1 is the cost comparison used above.
• BERT (Devlin et al., 2018): the encoder that made “pre-train once, add one output layer” standard.
• Seq vs Seq: Ettin (2025): the cleanest encoder-versus-decoder comparison at equal data and size.
• Scaling Laws for Neural Language Models (Kaplan et al., 2020): where the 2N and 6N FLOP rules come from.
• Training Compute-Optimal Large Language Models (Hoffmann et al., 2022): Chinchilla, the starting point for “how big, trained on how much”.