Designing the questions

The idea in one line: before you collect a single byte, decide which questions your model answers, what every answer means, and when “none of the above” or “unsure” applies, because a wrong schema is the one dataset mistake no tool can clean away later.

Every later stage in this part has a tool for its mistakes. Duplicates have MinHash, noisy labels have juries and cleanlab, leaks have time splits. A wrong question has no tool. Every label is an answer to the wrong question, so the only fix is to relabel everything. When your labels come from the world (a bug closed as duplicate, a build that failed), you may also have to wait months for the world to produce new ones. The basics go wrong more often than you would think: a survey of 591 dataset papers found about 30% with subpar quality management, and common misuse of agreement statistics (Klie et al., 2023-07-16).

Here is the 10x lever. The way you phrase the question decides where the labels come from. Phrase it so an event in the world settles it, and labels arrive free, by the hundred thousand, with no annotator at all. Phrase it as a matter of taste, and you buy every label one at a time. Polishing your labeller’s prompt is a 10% improvement next to that. That is why the first day of a data project goes on the schema, the guide and a 200-item pilot. It is the cheapest day of the project.

Think of a decision dataset as a stack of marked exam papers. Each paper has a situation, a question, a fixed list of allowed answers, and the answer that turned out to be right. A decision model (a model that returns one answer from a closed set plus a probability, instead of free text) learns from that stack. Each row is:

(input, typed question, answer from a closed set, probability)

The answer type is the shape of the allowed answers. Four types cover almost every decision, and they map onto the three types Jev exposes (anatomy of Jev): Choice picks one of up to 255 options, Score rates on 2 to 10 ordered levels, and Noul gives P(yes).

One pattern from the web-data world is worth copying. FineWeb-Edu asked a large model to score web pages for “educational value” on an additive rubric: one point for each criterion the page meets, 0 to 5 in total. It then turned the score into a binary keep-or-remove at 3, and the classifier trained on those labels reached an F1 of 82% at that threshold (FineWeb-Edu card, v1.4.0, 2025-07-11). The additive rubric turns one vague judgement (“is this educational?”) into five small checkable ones. Whenever a question feels like taste, try to break it into criteria like that.

“Help me ship a distro 5x faster” is a goal, not a question. No model can answer it. You turn it into the small decisions a maintainer makes again and again, and put each one through three filters.

1. Can it be answered from the input alone? If a careful human would need facts that are not in the input (say, the whole git history), it is a retrieval task, not a decision. A high “unsure” rate in your pilot will expose it.
2. What event in the world settles it? A retry passes, a bug gets closed as a duplicate, an update gets pulled. If such an event exists, your labels can come free (see labels from AI and from the world).
3. Does an exact, fast oracle already exist? An oracle is a program that gives the right answer every time. If one exists, call it. Training a model to imitate it is a mistake.

Here is the distro job run through those filters. The rows come from the case study’s decision map (the decision map).

Five of the case study’s 23 decisions have exact, fast oracles. The best model targets have three things at once: they come up often, the oracle is slow, and the labels arrive for free.

A worked example of “comes up often”. Fedora’s build system, Koji, ran 284,204 failed per-architecture build tasks in the 12 months to 2026-10-28, about 780 a day (a measurement in the course’s distro report). The same report estimates 5 to 30 minutes of human time to triage one failure. Even at the low end, 780 × 5 minutes is 3,900 minutes, or 65 hours of attention a day across the project. A question that comes up that often is worth a model. A question that comes up twice a month is not, however clever the model.

The taxonomy is the list of allowed answers. Four rules keep it healthy.

Labels must not overlap, and together they must cover everything, at one level of detail. This is often called MECE (mutually exclusive, collectively exhaustive). If two labels overlap, annotators split arbitrarily between them, and agreement collapses. If you need finer detail, ask a second question that depends on the first (a hierarchy). Don’t grow one flat list.

Keep the world’s taxonomy when one exists, and merge synonyms. Fedora’s bug tracker (Red Hat Bugzilla) already closes each bug with one of twelve resolutions. On 2026-10-28 its public REST API reported these counts for product Fedora (first-hand queries of bugzilla.redhat.com/rest/bug, field total_matches, in the course’s dataset report):

Four of these (ERRATA, RAWHIDE, CURRENTRELEASE, NEXTRELEASE) all mean “fixed somewhere”. Add them up: 199,720 + 125,419 + 47,012 + 27,525 = 399,676, which is 52% of the 764,880 bugs in the table. Merged, they give you one well-defined class holding half the data. Left apart, they give you four classes whose differences are about release timing, not about the bug.

Some outcomes are not labels at all. EOL (97,055 bugs, 13% of the table) means the Fedora release reached end of life before anyone decided anything. That is a censored outcome: the clock ran out before the answer arrived. Treat it as “no label”, never as “not a bug”.

The same field can be a label for one question and not for another. INSUFFICIENT_DATA says nothing about whether the bug was real, so it is “no label” for the question “is this a real bug?”. It is exactly the answer to “does this report need more information?”, which is why the course’s triage recipe maps it to a needs_info label. Write down, per question, which world fields count as answers. WONTFIX, WORKSFORME and CANTFIX each need a written rule before you use them.

Expect heavy tails, and plan for them. In Fedora’s GCC 15 mass prebuild, 1,438 of 8,711 builds failed. Of the 914 failures classified by hand, 665 were one class (C23, the new C language default), 160 C++, 22 internal compiler errors, 16 -Werror, 19 other and 32 unclassified (Fedora wiki, 2025-01-10). That adds up to 914. One class holds 73%, so a model that always answers “C23” looks 73% accurate while being useless. Keep the real class mix in your test set and report results per class.

These two look alike. They are opposites.

• “None of the above” is a property of the item. The item is real and in scope, but none of your classes fits. It is a real class, and the model must learn it, because production will send inputs your taxonomy never imagined. In the build-failure guide below it is called other.
• “Unsure” is a property of the annotator, human or model. The input lacks the information, or the guide doesn’t say what to do. It is an abstention (a refusal to answer that is routed elsewhere; see from probability to automation). Route it to review, count it, and never train on it as a class.

A rising “unsure” rate on one label is the cheapest signal you will ever get that the guide or the taxonomy is wrong. Here is an illustration with made-up numbers: in a 200-item pilot you mark 12 items unsure, and 8 of the 12 sit between upstream and valid_bug. Your annotators are not the problem. The definition of upstream is. Fix the guide, not the people.

Pilot tally (illustrative numbers, 200 items)

label pair you hesitated between       unsure items
upstream  vs valid_bug                            8
duplicate vs valid_bug                            2
needs_info vs not_a_bug                           2
                                       ------------
                                                 12  (6% of 200)

You don’t know your own rules until you have applied them. Shankar and colleagues watched people build grading criteria for model outputs and found a loop: you need criteria to grade outputs, but grading outputs is what defines the criteria, and some criteria only appear once you have seen the outputs (Who Validates the Validators?, 2024-04-18). They call it criteria drift. The same happens to a label guide. Your first draft is a guess. The guide you can trust is the one written after you have argued with 200 real items.

The practical rule from the course’s dataset report is this:

1. Write guide v0: the question, the labels, and one sentence per label.
2. Label 100-200 real items yourself, drawn at random (not hand-picked), and write down every hesitation.
3. Rewrite the guide as v1. Merge labels you could not tell apart. Add a tie-break rule for each hesitation that came back. Paste in real examples.
4. A few days later, label 50 new items blind and measure how far you agree with yourself.
5. Freeze v1, give it a version number, and log the guide version on every label. Only then does anyone else, human or model, start labelling.

How to measure agreement with yourself. Cohen’s kappa measures agreement between two labellings, corrected for the agreement you would get by chance:

kappa = (p_o - p_e) / (1 - p_e)

p_o = share of items where the two passes agree
p_e = chance agreement = sum over labels of
      (share in pass 1) x (share in pass 2)

A worked example with three labels. You relabel 50 items and agree on 44, so p_o = 0.88. In both passes the labels split 50% / 30% / 20%, so p_e = 0.5×0.5 + 0.3×0.3 + 0.2×0.2 = 0.38. Then kappa = (0.88 - 0.38) / (1 - 0.38) = 0.81. The chance correction matters. 88% raw agreement sounds great, but a third of it you would get by luck.

For more than two annotators, use Krippendorff’s alpha, a close cousin that also handles missing labels. By convention, an alpha of 0.800 or more is reliable, 0.667 to 0.800 supports only tentative conclusions, and below 0.667 the labels are discarded (Wikipedia, read 2026-10-28). If you cannot agree with yourself at that level, no model will agree with you either.

Every tool in that table assumes the label set is right. cleanlab, for example, finds items whose given label disagrees with what a model expects among your classes. If two of your classes overlap, it will flag the overlap forever and never tell you to merge them. Even with a sound schema, label noise is large enough to matter. Famous test sets carry at least 3.3% label errors on average (ImageNet’s validation set at least 6%), enough to change which model looks best (Northcutt et al., 2021-03-26).

An honest nuance: LLM labels have made relabelling text cheap. The course’s one-week recipe labels 80,000 items with three models for about $23 (as of October 2026). So the real cost of a schema change is no longer the relabelling. It is everything you cannot rerun: the human gold set and the guide, the mapping from world outcomes (for new outcomes you wait on the world), and everything downstream, such as trained models, thresholds and your eval history.

To keep a later change cheap, store enough on every row that a schema change becomes a migration, not a restart. This is the row the course’s dataset report recommends:

id              stable item id
task_id         which typed question this row answers
schema_version  version of the label set
input           the item, or a pointer to it
label           one value from the closed set
label_source    outcome | llm_jury | human | synthetic
label_conf      confidence, or jury agreement
guide_version   label guide used for this label
decision_time   when the decision would have been made
outcome_time    when the world settled it
source_url      where the item came from
licence         licence or terms of the source
split           train | dev | test

decision_time and outcome_time look like bookkeeping. They are what let you build leak-free time splits later. licence and source_url are what let you honour a takedown (data rights).

The case study’s second decision is “Which failure class is this build log?”, a Choice with 12 classes. The class list below is the case study’s own design, not a Fedora standard. What makes it trainable is that most labels come from the world. Two tricks from the course’s distro report do the work:

• The retry. A failed job re-run with identical inputs that then passes was transient. In Universal Blue’s kernel-module repo, 40 of 47 re-attempted runs (85%) passed on retry.
• The fix diff. For a failed Koji build, take the next successful build of the same package and diff the packaging commits. A BuildRequires change means a missing dependency. A new patch touching C sources means a toolchain error. %check edits mean a test failure. %files edits mean packaging. No diff means transient.

The guide is what a human, or an LLM, reads when no outcome exists yet. It opens like this:

LABEL GUIDE   build-failure-class   v1
              (frozen after a 200-item pilot)

Question  Which class of problem stopped this build?
Input     A log snippet of at most ~2k tokens, cut around the
          first fatal error, plus the package and build target.
Answer    Exactly one label from the table. Quote the shortest
          log line that decides it in the evidence field.
Outcome   If a retry result or fix diff exists, it decides the
          label. Your reading only covers items without one.

Then one row per label. In the real guide, each row also gets two or three real positive and negative examples from your pilot; they are left out here.

The outcome signals for the first five labels come from the case study’s fix-diff and retry rules. The others are this guide’s proposals, which you would test in the pilot before trusting them.

Then the rules that turn hesitations into decisions:

Tie-breaks
  1. The first fatal error wins. Later errors are usually
     knock-on effects of the first.
  2. A test killed by OOM or a timeout is `resource`, not
     `test-failure`.
  3. A missing header that only disappeared because of a new
     toolchain is `new-toolchain-error`; a header that was
     never declared is `missing-build-dependency`.

Unsure (not a label: route to review, do not train on it)
  - the snippet was cut before the first fatal error;
  - two labels fit and no tie-break decides;
  - the log is not a build log at all.

Bookkeeping
  - log guide_version on every label;
  - every new tie-break bumps the version (v1 -> v1.1).

How you test the guide. You test it against the world, not against your own taste. Fedora’s Log Detective project has a 640-row test split of human-annotated failures. Map it to your classes and add 200 items you label by hand. Use the GCC 15 classification (914 failures) as a second check. Then compare three things: your guide-based labels, the fix-diff labels, and an LLM jury reading the same guide. Where all three agree, the guide works. Where the jury and the fix diff disagree on one pair of labels, the guide has a hole there. You fix it and bump the version. The next two lessons, sourcing and labels, build the pipeline that runs this at scale.

• Shankar et al., Who Validates the Validators? (2024-04-18) — the evidence for criteria drift, and why you label before you freeze.
• Klie et al., Annotation quality management in the wild (2023-07-16) — 591 dataset papers, and how often agreement and quality checks go wrong.
• FineWeb-Edu dataset card (v1.4.0, 2025-07-11) — an additive rubric and a threshold, turned into a classifier at web scale.
• Fedora wiki: F42 GCC 15 mass prebuild (2025-01-10) — a real, hand-made failure taxonomy with its counts.
• Gebru et al., Datasheets for datasets (2018-03-23) — what to write down about a dataset, starting with its schema.