Labels from AI and from the world

The idea in one line: the best labels are the ones the world already wrote (a build passed, a bug was closed as a duplicate). A jury of LLMs from different families fills the gaps for about ten cents per thousand labels, and a few hundred human gold labels keep both honest.

Labels used to be the expensive part of machine learning. In October 2026 the cheapest batch LLMs label for $0.07-0.11 per 1,000, while 2023 data implies about $60 per 1,000 for crowdworkers on Amazon Mechanical Turk (MTurk): roughly 500-900× cheaper (the dataset report’s arithmetic, from Gilardi et al., 2023-03-27 and price pages). The money has moved to the schema (designing the questions), the guide, and a few hundred checked gold items.

The 10x lever is where labels come from, not how you word the prompt. Labels the world writes cost nothing per row, carry no annotator bias, arrive timestamped, and are the only way to learn whether your model’s “70%” really happens 70% of the time (calibration). LLM labels fill the gaps cheaply, but their errors are correlated in ways that can fool you.

An outcome label is recorded by a process that would have happened anyway, after the moment of decision. A bug gets closed. An update gets pushed or pulled. A build passes or fails. Nobody had to annotate anything.

A worked example: base rates. Of the 844,142 Bodhi updates in that table (796,835 + 43,225 + 4,082), only 4,082 were unpushed, which is 0.48%. A model that always answers “this update will stay pushed” is 99.5% accurate and completely useless. For rare outcomes, report per-class recall and the Brier score. Never quote bare accuracy.

Human annotation can’t keep up: Fedora’s Log Detective holds about 1,228 annotated reports, while Koji produces about 780 failures a day. Two tricks from the course’s distro report fill the gap. The retry: a failed job that passes when re-run with identical inputs was transient (in one kernel-module repo, 85% of retries passed). The fix diff: diff a failed build against the next good build of the same package, and simple rules map the diff to a failure class (label guide example).

Mining at scale. Store raw JSON beside a normalised row with each item’s last_change_time, and join the outcome to the input as it looked at decision time: the bug’s first comment, not its final state.

Leakage is any path by which training data contains information that would not be available at prediction time. Kapoor and Narayanan catalogue eight types and found leakage errors in 329 papers across 17 fields (Kapoor & Narayanan, 2022-07-14). Six bite outcome-labelled data hardest:

1. Post-decision fields. resolution, dupe_of, final karma and closing comments come after the decision; in an input or a labeller’s prompt, they let the label be copied, not inferred. Drop them.
2. Random splits. Near-duplicates land on both sides, so the test measures memory. Split by time and group by entity.
3. Memorisation. Models may have read the outcome in pre-training: on SWE-bench Verified they picked the buggy file from the issue text alone with up to 76% accuracy, against about 53% elsewhere (SWE-Bench Illusion, 2025-06-14). Test on items resolved after every model’s cutoff.
4. Proxy mismatch. “Merged” is not “correct”. One Bodhi update the dataset report pulled went stable in four minutes with karma 0 and test gating ignored, likely an automatic push: “stable” encoded policy, not quality.
5. Censoring. Only resolved items have labels, and speed correlates with the label (duplicates close fast). Report the unresolved pool’s base rate.
6. Policy drift. Karma rules and CI configs change label meanings. Keep a policy_version and evaluate on the latest period.

The split that defeats traps 2 and 3 looks like this. The dates are the case study’s own plan:

Time split with a gap, plus an entity split

     train: decided before 2026-07-01    test: after 2026-08-01
  |-------------------------------------|  gap  |---------------|
   packages in group A only                      group B only

  The month-long gap stops one fix from landing on both sides.
  No package appears in both train and test.

The evidence on LLMs as annotators splits into two camps, and both are right.

• The optimists (2023). ChatGPT beat MTurk crowdworkers on four of five tweet-annotation tasks, with higher intercoder agreement than crowdworkers and trained annotators, at under $0.003 per annotation, about 20× cheaper (Gilardi et al., 2023-03-27).
• The cautious. Across 27 tasks from 11 social-science datasets, LLM performance was “highly contingent on both the dataset and the type of annotation task” (Pangakis et al., 2023-05-31). The Alternative Annotator Test offers a statistical procedure for justifying an LLM in place of humans; closed models such as GPT-4o sometimes passed it (Calderon et al., 2025-01-19).
• The sharpest warning (2025). Across 37 tasks, about 13 million labels from 18 LLMs and 2,361 hypotheses, LLM annotation led to wrong conclusions for about 31% of hypotheses with state-of-the-art models, and about half with small ones. Human annotations were the key protection (Baumann et al., 2025-09-10).

High average accuracy and systematic, correlated errors coexist. Use LLM labels, and check them against something that is not an LLM.

What a label costs, at 1,500 input tokens (guide plus item) and 150 output tokens per label, batch prices as of October 2026 (OpenAI, Gemini, Claude, DeepSeek, read 2026-10-28):

Check the first row by hand: 1.5M input tokens at $0.025 per million ($0.0375) plus 0.15M output tokens at $0.20 per million ($0.03) makes $0.0675 per thousand. One trap multiplies that bill: hidden reasoning tokens are billed as output. The course’s stack report computes that 1,000 extra thinking tokens per example add $10,000 per million examples on Opus 5.5 batch. Turn reasoning off when you label.

Ask for typed output: JSON matching a schema, with one label from an enum, a confidence bucket and an evidence quote. Structured-output modes in OpenAI-compatible APIs constrain the reply. Validate it with Pydantic anyway (instructor and outlines are maintained helpers).

# Label one item with a typed schema via any OpenAI-compatible endpoint.
from typing import Literal
from openai import OpenAI
from pydantic import BaseModel, ConfigDict, Field
class Triage(BaseModel):
    """One triage decision; extra='forbid' keeps the schema strict."""
    model_config = ConfigDict(extra="forbid")
    label: Literal["duplicate", "needs_info", "upstream", "valid_bug",
                   "not_a_bug", "none_of_the_above", "unsure"]
    confidence: Literal["low", "medium", "high"]
    evidence: str = Field(description="Shortest quote that decides the label")

GUIDE = open("label_guide_v3.md").read()  # definitions, examples, tie-breaks
client = OpenAI()  # base_url=... for a self-hosted open-weights endpoint

def label(report: str, model: str) -> Triage:
    schema = {"name": "triage", "strict": True,
              "schema": Triage.model_json_schema()}
    r = client.chat.completions.create(
        model=model, temperature=0,
        messages=[{"role": "system", "content": GUIDE},
                  {"role": "user", "content": report}],
        response_format={"type": "json_schema", "json_schema": schema})
    return Triage.model_validate_json(r.choices[0].message.content)

The dataset report did not run this snippet against a live API; its schema was checked separately and comes out strict. At scale, submit the same requests as JSONL to a batch endpoint (half price at OpenAI, Anthropic and Google), guide first so prompt caching applies. The label set holds both none_of_the_above (a real class) and unsure (an abstention you route, never train on).

A jury (or panel) is several models labelling the same item independently, combined by vote. A panel of smaller models from different families beat a single large judge across six datasets at over 7× lower cost (Verga et al., 2024-04-29). Self-consistency (sampling one model several times and taking the majority) is cheaper and gained 17.9 points on the GSM8K maths benchmark (Wang et al., 2022-03-21), but every sample shares one model’s blind spots.

The catch is that models’ errors are not independent. Across 350+ models on one leaderboard, when two models are both wrong they pick the same wrong answer 60% of the time, and larger, more accurate models have more correlated errors, even across providers (Kim et al., 2025-06-09). Judges also favour models similar to themselves (Goel et al., 2025-02-06), and recognise and prefer their own writing (Panickssery et al., 2024-04-15).

A worked example: what correlation does to a jury. Take two labellers, each 90% accurate, on a task with 6 labels. If their errors were independent and spread evenly over the 5 wrong labels, two wrong answers would match 1 time in 5.

                         independent errors   correlated (60%)
both wrong               0.1 x 0.1 = 1%       at least 1%
...and pick the same
wrong label              1 in 5 (5 wrong)     3 in 5
unanimous but wrong      1% x 0.2 = 0.2%      1% x 0.6 = 0.6%

Correlation triples the rate of confidently wrong, unanimous labels, and that is before counting that correlated models are also wrong together more often than 1%. So a unanimous jury is weaker evidence than it looks, and only labels from outside the models (outcomes, humans, verifiers) break the correlation.

The working recipe from the dataset report: three models from different families at temperature 0, with typed output including a confidence and an evidence quote. Keep unanimous, high-confidence labels; adjudicate 2-of-3 splits with a stronger model; send three-way splits to a human, because they usually mean the guide has a hole.

# Filter a 3-model jury: keep unanimous high-confidence labels, route the rest.
import pandas as pd
from sklearn.metrics import cohen_kappa_score

votes = pd.read_parquet("votes.parquet")  # item_id, model, label, confidence
wide = votes.pivot(index="item_id", columns="model", values="label")
top = wide.mode(axis=1)[0]  # majority label per item
agree = wide.eq(top, axis=0).sum(axis=1) / wide.notna().sum(axis=1)
high = votes.assign(h=votes.confidence.eq("high")).groupby("item_id").h.all()
keep = agree.eq(1.0) & high.reindex(agree.index, fill_value=False)
accepted = pd.DataFrame({"label": top[keep], "label_source": "llm_jury"})
adjudicate = agree[~keep & (agree >= 0.5)].index  # stronger model
human = agree[agree < 0.5].index  # three-way split: usually a guide gap
m = list(wide.columns)
kappas = {(a, b): round(cohen_kappa_score(wide[a], wide[b]), 2)
          for i, a in enumerate(m) for b in m[i + 1:]}
print(len(accepted), len(adjudicate), len(human), kappas)

This is the report’s jury filter with one change: in the original, unanimous items without all-high confidence matched no route and silently dropped out. Here they go to the adjudicator, so every item lands somewhere. The pairwise Cohen’s kappa (agreement corrected for chance) shows which models are near-copies; drop the most correlated one before you scale up.

Keep one dissent in view: the FineWeb-Edu card notes that the jury-based annotation it tried retained more low-quality samples (FineWeb-Edu card). Measure juries against your gold set. When you need valid statistics, not just a training set, prediction-powered inference corrects estimates made with model labels using a small human set (Angelopoulos et al., 2023-01-23), and confidence-driven inference cut human labels by over 25% on three tasks while keeping valid confidence intervals (Gligorić et al., 2024-08-27).

Picture an expensive expert who grades a sample of essays, and an apprentice who learns to grade the way the expert does and then grades the whole pile. Distillation is that: a big “teacher” model labels a sample, and a small “student” model trained on those labels runs on everything else, at a tiny fraction of the cost.

FineWeb-Edu is the cleanest public example. Llama-3-70B-Instruct scored 500k web pages from 0 to 5. A small regression head on an embedding model (Snowflake-arctic-embed) learned those scores, with an F1 of 82% at the keep threshold of 3. That student then classified all 15 trillion tokens in about 6,000 H100-hours, keeping 1.3 trillion tokens at threshold 3, or 5.4 trillion at threshold 2 (FineWeb-Edu card, v1.4.0, 2025-07-11). The dataset report derives about 0.3 H100-hours per million documents for the student’s pass.

The same economics hold for a decision task. In a 2024 case study, Mixtral-8x7B labelled 1,811 financial sentences, and a RoBERTa-base student reached 94% accuracy, the same as its teacher and GPT-4. The student cost about $2.70 per million sentences on a T4 GPU, against $3,061 for GPT-4, and fine-tuning took 15 minutes and under $1 (Hugging Face blog, 2024-02-16). This is the economic core of a Jev-like product (distillation recipe).

The distillation ceiling. A student trained only on LLM labels inherits its teacher’s errors. One outside write-up reports that Jev’s launch “accuracy” meant agreement with the average of two frontier models, putting Jev at “roughly 68%” (RITS), a single source. That measures distilled frontier judgement, not truth. Only labels from outside the teacher (outcomes, humans, verifiers) get you past the ceiling (verifiable rewards).

Weak supervision labels data with many cheap, noisy rules instead of hand labels. A labelling function is a small program that votes or abstains: “if the component is kernel and the text mentions akmods, vote upstream”. Data programming models each function’s accuracy and correlations and outputs a probabilistic label per item. In Snorkel’s user study, experts built models 2.8× faster than with seven hours of hand labelling, and predictive performance improved by 45.5% on average (Ratner et al., 2017-11-28).

The library is fading: Snorkel’s open-source repo last released v0.10.0 on 2024-02-27, and the team now focuses on the commercial Snorkel Flow. The idea outlives it. Today an LLM prompt is just another labelling function, alongside your regexes and outcome joins.

Even famous datasets are wrong more often than you would guess. Applied to 10 popular test sets, confident learning found at least 3.3% label errors on average, and at least 6% in the ImageNet validation set. After correction, ResNet-18 overtakes ResNet-50 on ImageNet if the share of originally mislabelled test examples rises by just 6% (Northcutt et al., 2021-03-26). At 3-6% noise, your evaluation labels, not your model, may decide which model looks best.

Confident learning estimates which labels are wrong from a model’s own predictions (Northcutt et al., JAIR 2021). The intuition: if a model that never saw an item is confident it belongs to another class, look again. In its author’s words, “the thresholds for each class are the average predicted probability of examples in that class”, and an example whose probability for a class clears that threshold is confidently counted as that class (Northcutt, 2019-11-03). Items counted into a class other than their label are the suspects.

A worked example. Items labelled duplicate get an average predicted p(duplicate) of 0.80, so the duplicate threshold is 0.80. One item is labelled valid_bug, but the model gives it p(duplicate) = 0.91 and p(valid_bug) = 0.05. 0.91 clears 0.80, so it is counted as a duplicate against its given label, and goes to the review queue. The probabilities must be out-of-fold: each row scored by a model trained without it, or the model just echoes the labels you are checking.

# Rank likely label errors with cleanlab from out-of-fold probabilities.
from cleanlab.filter import find_label_issues
from sentence_transformers import SentenceTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict

texts, labels = load_rows()  # your rows; labels: ints 0..K-1
X = SentenceTransformer("all-MiniLM-L6-v2").encode(texts, batch_size=64)
# Out-of-fold probabilities: each row is scored by a model that never saw it.
pred_probs = cross_val_predict(LogisticRegression(max_iter=2000), X, labels,
                               cv=5, method="predict_proba")
issues = find_label_issues(labels=labels, pred_probs=pred_probs, n_jobs=1,
                           return_indices_ranked_by="self_confidence")
print(f"{len(issues)} suspected errors ({len(issues) / len(labels):.1%})")
review_queue = issues[:500]  # humans re-check the worst first

A gotcha the dataset report hit with cleanlab 2.9 on macOS: find_label_issues uses multiprocessing by default, and a script without an if __name__ == "__main__": guard re-runs itself in every worker and never finishes. Pass n_jobs=1. With that fix, on 600 synthetic rows with 30 deliberately flipped labels, it flagged 31 rows, including all 30 flips. cleanlab’s last release is v2.9.0 (2026-01-13). When the queue shows the same confusion again and again, fix the guide, not just the rows.

Synthetic data is training data written by a model. For a decision dataset it is a scalpel, not a backbone.

• Subjectivity predicts failure. The more subjective the task, the worse models trained on synthetic data perform (Li et al., 2023-10-11). Keep synthetic data for objective, checkable labels.
• Force diversity with attributes. Plain “write an example of class X” prompts produced biased data. Prompts with sampled attributes (length, style, topic) matched them at 5% of the query cost, and attribute diversity drove performance (Yu et al., 2023-06-28). Cosmopedia found the same: rewording alone did not stop duplicates (Hugging Face, 2024-03-20).
• Hard negatives and edge cases are the sweet spot. A hard negative is a near-miss: a minimal edit to a real item that flips its correct label, such as a duplicate report rewritten so the root cause differs, or a Containerfile with one wrong base-image tag. Models scored up to 25% lower on such contrast sets than on the original test sets (Gardner et al., 2020-04-06).

Four rules keep synthetic data safe. Generate with one model family and label with the jury, never with the generator (self-preference). Verify with a checker when one exists: does the edited Containerfile really fail to build? Tag every row label_source=synthetic and cap its share per class. Never put synthetic items in the test set.

Model collapse is degradation from training on model outputs: the tails of the original distribution vanish when each generation trains only on the previous one’s outputs (Shumailov et al., 2023-05-27; later in Nature, 2024). But it depends on the workflow:

Replace: each generation trains on the last one's output only
  gen 0: real -> gen 1: synth(0) -> gen 2: synth(1) -> ...
  the tails vanish; test error grows                 collapse

Accumulate: keep all real data, add synthetic data each round
  gen 0: real -> gen 1: real + s1 -> gen 2: real + s1 + s2
  test error stays bounded                           stable

Accumulating real plus synthetic data kept test error bounded across language, diffusion and VAE models (Gerstgrasser et al., 2024-04-01); a fixed-size subsample of the pool degrades slowly, not explosively (Kazdan et al., 2024-10-22). The dissent deserves a hearing: “strong model collapse” finds that even 1% synthetic data can cause collapse in a supervised regression setting, and larger training sets do not fix it (Dohmatob et al., 2024-10-07). Either way the operational answer is the same: real data is the anchor, synthetic data is augmentation, evaluation is real and time-split, and a model’s own labels never loop back without fresh outside labels.

Active learning chooses the next items to label by model uncertainty, instead of at random. Common signals are the lowest top-class probability, the smallest margin between the top two classes, or jury disagreement.

A worked example with margins. Item A gets p = 0.48 for duplicate and 0.45 for valid_bug, a margin of 0.03, so the model is torn: label it first. Item B gets 0.95 and 0.03, a margin of 0.92: skip it for now.

It is fragile as your only sampling scheme: benefits were inconsistent across tasks, and data picked with one model did not reliably help a successor (Lowell et al., 2018-07-12). OpenAI did find it significantly improved process supervision when collecting PRM800K (Lightman et al., 2023-05-31). Keep a random slice (say 20%) of every round so estimates stay unbiased.

A gold set is a sample labelled carefully by humans, or by an outcome you trust, and used to measure everything else. How big must it be? To estimate an accuracy p within plus or minus e at 95% confidence:

n = 1.96^2 x p(1 - p) / e^2

worst case p = 0.5, e = 0.05:  3.8416 x 0.25 / 0.0025 = 385
p = 0.9,            e = 0.02:  3.8416 x 0.09 / 0.0004 = 865

Per-class claims need that many items per class. A class at 5% prevalence needs about 20× more random items, so sample gold stratified by the jury’s predicted label and by disagreement, then reweight. In practice: 300-500 gold items to choose labellers and catch guide bugs; 1,000+ (stratified) to publish per-class accuracy; two independent humans on 200-300 items to measure agreement. At about 30 seconds an item, 1,000 gold labels are about 8 hours of expert time (stack report). Outcome-labelled items are free gold, provided the outcome post-dates the models’ training data.

The running example is Fedora bug triage, with the labels duplicate, needs_info, upstream, valid_bug, not_a_bug and none_of_the_above, and Bugzilla resolutions as outcome labels. Swap in any domain. Prices as of October 2026.

Day 1: schema, guide, rights ($0). Write the decision, the labels and guide v0. Label 150 real reports yourself, revise to v1, label 50 more blind and check self-agreement (designing the questions). Fill in the rights inventory, and choose labellers whose terms allow your use (data rights).

Day 2: source and clean ($0-99). Pull about 120k closed Fedora bugs through the REST API with include_fields, keeping only fields known at filing time, with outcomes in a separate column. Map resolutions to labels: DUPLICATE to duplicate, UPSTREAM to upstream, NOTABUG to not_a_bug, INSUFFICIENT_DATA to needs_info, the fixed family to valid_bug, EOL to unlabelled. Optionally add 100k web pages through Common Crawl’s URL index or Firecrawl Standard ($99 for 100k pages). Then exact and MinHash dedup, a PII scrub, and a split by time (the latest 20% is test) and by component (sourcing).

# Near-duplicate removal with MinHash LSH (datasketch 2.x).
from datasketch import MinHash, MinHashLSH

def shingles(text: str, n: int = 5) -> set[str]:
    """Word n-grams of a normalised text."""
    words = text.lower().split()
    return {" ".join(words[i:i + n]) for i in range(max(1, len(words) - n + 1))}

def signature(text: str, num_perm: int = 128) -> MinHash:
    """MinHash signature of a text's shingle set."""
    m = MinHash(num_perm=num_perm)
    for s in shingles(text):
        m.update(s.encode("utf8"))
    return m

def dedup(docs: dict[str, str], threshold: float = 0.8) -> list[str]:
    """Keep the first document of each near-duplicate cluster; return kept ids."""
    lsh, keep = MinHashLSH(threshold=threshold, num_perm=128), []
    for doc_id, text in docs.items():
        sig = signature(text)
        if not lsh.query(sig):  # nothing similar kept yet
            lsh.insert(doc_id, sig)
            keep.append(doc_id)
    return keep

Day 3: pilot the jury (under $5). Draw a 300-item gold set: label it blind, then compare with the outcome labels. Run 2,000 items through four cheap labellers (for example gpt-oss-120b, DeepSeek V4 Flash, Gemini 2.5 Flash-Lite, Claude Haiku 5.5). Score each against gold and against outcomes resolved after the models’ cutoffs, measure inter-model kappa, and fix the guide where they all fail. Keep the three most accurate and least correlated, from different families.

Day 4: label at scale (about $40). Send about 80k unlabelled items to batch APIs, three models each: 80 × ($0.068 + $0.105 + $0.113) ≈ $23. Generate about 5k synthetic hard negatives with a mid-tier model ($1 / $5 per million tokens in batch, about 800 tokens in and 500 out each): 5,000 × ($0.0008 + $0.0025) ≈ $17. Tag them synthetic; the jury labels them.

Day 5: agree, adjudicate, clean (about $60). Run the jury filter. Adjudicate the 2-of-3 cases (assumed about 20% of items) with a stronger model: 16,000 × (1,800 tokens in at $1/M + 400 out at $5/M) ≈ $61. Three-way splits come to you. Run the cleanlab pass on your laptop and review the top 500 issues yourself (about 4 hours).

Day 6: quality and agreement (about $150-200, optional). Pay a second annotator with domain knowledge to label 300 gold items blind; the report’s 4-5 hours at $40/h is an assumption, not a sourced price. Compute Krippendorff’s alpha against yourself and the jury, decontaminate the splits, and write the datasheet.

Day 7: prove it ($0). Train a baseline: logistic regression, or SetFit, which is competitive with parameter-efficient fine-tuning while training an order of magnitude faster (Tunstall et al., 2022-09-22). Evaluate on the time-split, outcome-labelled test set, calibrate, and report accuracy, macro-F1, Brier score and a reliability diagram per label (evals). Freeze v1.0 with its guide, gold set and dataset card.

For a general decision model like Jev, task diversity is what generalises: Flan-PaLM 540B, fine-tuned on 1.8K tasks, beat PaLM 540B by 9.4 points on average (Chung et al., 2022-10-20). The dataset report infers (an inference, not a measurement) that fifty tasks of 1,000 examples beat one task of 50,000.

The labelling pipeline as a C4 diagram (how to read the diagrams). First the context:

C4 · L1 System context · Labelling pipeline

+----------------+  +-------------------+  +----------------+
| [Person]       |  | [External system] |  | [Person]       |
| Builder (you)  |  | Sourcing pipeline |  | Second         |
|                |  |                   |  | annotator      |
+----------------+  +-------------------+  +----------------+
   | labels gold,      | hands over           | labels 300
   | reviews splits    | clean rows           | gold items
   v                   v                      v
+--------------------------------------------------------------+
| [Software system]                                            |
| Labelling pipeline                                           |
| clean rows in; labelled, versioned dataset out               |
+--------------------------------------------------------------+
    | reads         | sends          | runs labels    | hands over
    v outcomes      v batches        v on             v dataset
+-------------+ +--------------+ +---------------+ +-------------+
| [External   | | [External    | | [External     | | [External   |
|  system]    | |  system]     | |  system]      | |  system]    |
| Bugzilla,   | | LLM batch    | | Open-weight   | | Training    |
| Bodhi, Koji | | APIs, three  | | labeller on a | | pipeline    |
|             | | families     | | rented GPU    | |             |
+-------------+ +--------------+ +---------------+ +-------------+

You and a second annotator supply gold labels and review the hard cases. The pipeline takes clean rows from sourcing, reads outcomes from the world, buys LLM labels, and hands a labelled, versioned dataset to training.

Zoomed in to the containers:

C4 · L2 Container · Labelling pipeline

+-------------------+  +-------------------+  +--------------------+
| [External system] |  | [External system] |  | [External system]  |
| Sourcing pipeline |  | Bugzilla, Bodhi,  |  | LLM batch APIs or  |
+-------------------+  | Koji APIs         |  | open-weight host   |
     |                 +-------------------+  +--------------------+
     | clean rows          ^                    batches (JSONL) ^
     v (Parquet)           | reads (HTTPS/JSON)                 |
.....|.....................|... Labelling pipeline [system] ....|.....
:    |                     |                                    |    :
:  +----------------------+                 +----------------------+ :
:  | [Container: Python]  |  rows with      | [Container: Python]  | :
:  | Outcome miner        |---------------->| LLM labeller         | :
:  | joins rows to world  |  no outcome     | typed schema; jury   | :
:  | outcomes as at       |  (Parquet)      | of 3 families plus   | :
:  | decision time        |                 | an adjudicator       | :
:  +----------------------+                 +----------------------+ :
:       | outcome labels                       | votes      ^ 2 of 3 :
:       v (Parquet)                            v (Parquet)  |        :
:  +----------------------+                 +---------------+------+ :
:  | [Database: Parquet]  |  accepted       | [Container: pandas]  | :
:+>| Label store + gold   |<----------------| Jury filter          | :
:| | label_source,        |  (Parquet)      | unanimous: keep;     | :
:| | guide_version        |                 | 2 of 3: adjudicate   | :
:| +----------------------+                 +----------------------+ :
:|                                                                   :
:|      | reads labels                         | 3-way splits        :
:|      v (Parquet)                            v (JSON)              :
:| +----------------------+                 +----------------------+ :
:| | [Container: Python]  |  flags          | [Container:          | :
:| | cleanlab pass        |---------------->|  Label Studio]       | :
:| | out-of-fold          |  (JSON)         | Review UI            | :
:| | probabilities        |                 | you and annotator    | :
:| +----------------------+                 +----------------------+ :
:|                                             |                     :
:|                        gold labels, fixes   |                     :
:|                        (Parquet)            |                     :
:+---------------------------------------------+                     :
:....................................................................:

Outcomes go straight into the label store; rows without one go to the LLM labeller (a three-family jury plus an adjudicator, in batch). The jury filter keeps unanimous, confident labels, sends 2-of-3 splits back for adjudication and three-way splits to review. cleanlab flags suspects from the store, and reviewed gold labels and fixes flow back. A synthetic generator, not drawn, feeds tagged hard negatives to the labeller; it never labels them itself.

• FineWeb-Edu dataset card (v1.4.0, 2025-07-11) — the distillation pattern at web scale, including what didn’t work.
• Verga et al., Replacing judges with juries (2024-04-29) — why diverse small panels beat one big judge, and at what cost.
• Kim et al., Correlated errors in large language models (2025-06-09) — the evidence that LLM mistakes move together.
• Baumann et al., Large Language Model Hacking (2025-09-10) — how LLM annotation can flip research conclusions, and how human labels protect you.
• Gerstgrasser et al., Is model collapse inevitable? (2024-04-01) — replace versus accumulate, with the evidence.