Building the co-pilot

The idea in one line: Model three decisions first: “will a retry pass?”, “which failure class is this?” and “will this bump break the image?”. Label them from what the build farm did next, and climb from rules to a small calibrated model only as far as the eval demands.

The decision map listed 23 typed questions. This lesson picks three and builds them. The surprise is where the effort goes. The model is the easy part. The label miners (jobs that join each failure to what happened next), the outcome store and the eval are the real work. They’re also the moat, because anyone can rent the same GPU.

The whole plan fits in under $500 plus about four weeks of evenings. The first useful milestone needs no language model at all. That’s the bootstrapper’s 10x: a rules-plus-trees model that auto-retries transient failures removes most 3 a.m. wake-ups, while a fancier model on hard logs would only trim minutes.

C4 · L1 System context · Distro co-pilot

+-------------------+                         +-----------------------+
| [Person]          |  asks, approves         | [Software system]     |
| Distro maintainer |------------------------>| Distro co-pilot       |
| ships the image   |<------------------------| triages, routes, odds |
+-------------------+  escalations            +-----------------------+
                                                          |
            +----------------------+----------------------+
            | reads runs, retries  | asks for fixes       | asks odds
            v                      v                      v
  +-------------------+  +-------------------+  +-------------------+
  | [External system] |  | [External system] |  | [External system] |
  | GitHub Actions,   |  | Coding agent      |  | Decision API      |
  | Koji, COPR, forge |  | writes the fixes  |  | rung 1, optional  |
  +-------------------+  +-------------------+  +-------------------+

The co-pilot sits between the maintainer and the build systems (how to read the diagrams). It reads every run, answers typed questions with probabilities, hands real fixes to a coding agent and escalates only what it’s unsure of. Early on it can borrow a hosted decision API for its odds; later it runs its own small model.

Why these three? Each one is frequent, each has a slow oracle (a 20-180-minute CI run), and each gets its labels for free.

Decision 1: “will a retry pass?” It has the cheapest label, the highest volume and an instant payoff. Recall that 40 of 47 re-run Universal Blue akmods jobs passed (85%). Start with baseline rules that already catch most transient failures: All mirrors were tried, HTTP 403, 429 and 5xx errors, timeouts, and a full runner disk.

How strict should the threshold be? Use the cost of being wrong, as in from probability to automation. Here’s a worked example with small numbers:

A wrong retry costs one extra CI run: machine time, no human.
A missed transient costs a human look: say 10 minutes.

Model flags 100 failures with P(retry passes) >= 0.7.
If it's calibrated, at least 70 of them pass on retry.
  saved:  70 human looks x 10 min = 700 minutes
  wasted: at most 30 extra CI runs, and those 30 still get a look

Wasted retries cost almost nothing, so the threshold can be loose (P ≥ 0.7). Compare “safe to auto-merge”, where a wrong yes can ship a broken image to users. That threshold must be strict, around P ≥ 0.98, and Fedora’s AI policy wants a human to stay accountable anyway.

Decision 2: “which failure class?” Use 12 classes, each mapped to a playbook or a fix template the coding agent can apply:

transient-network           missing-build-dependency
new-toolchain-error         test-failure
packaging-files             repo-resolution-conflict
kmod-vs-kernel              signing / Secure Boot
lint / policy (bootc lint)  resource (OOM, disk, timeout)
config-option-drift         other

Remember the GCC 15 prebuild from where the time goes: 73% of 914 classified failures were one class. With a distribution that heavy-tailed, a good classifier plus a template clears most of a campaign.

Decision 3: “will this bump break the image?” Its inputs are plain facts in a table: the package, the version gap, days since the upstream release, the size of the kernel jump, whether Universal Blue has already published kernel modules for that kernel, and Repology’s status. It gates terra-fedora’s nightly auto-updates. For NVIDIA and ZFS modules, add a 5-level risk Score.

The trick from the decision map is to take labels from the retry and the fix, not from annotators.

The sizes are the research’s targets for four weeks of one person’s work, not measurements.

Here are the fix-diff rules as code. Each failed build gets a label from the change that made the next build pass:

# Illustrative rules: label a failed build from the fix that followed
# it, i.e. the diff to the next good build of the same package.
def label_from_fix(spec_added, spec_sections, patched_files):
    """spec_added: added spec lines; spec_sections: spec sections
    edited; patched_files: source files touched by any new patch."""
    if not spec_added and not patched_files:
        return "transient"                  # same inputs later passed
    if any(l.startswith("BuildRequires:") for l in spec_added):
        return "missing-build-dependency"
    if any(f.endswith((".c", ".h", ".cc", ".cpp")) for f in patched_files):
        return "new-toolchain-error"
    if "%check" in spec_sections:
        return "test-failure"
    if "%files" in spec_sections:
        return "packaging-files"
    return "other"                          # send to the LLM jury


print(label_from_fix(["BuildRequires: pkgconfig(gtk4)"], ["preamble"], []))

It prints missing-build-dependency. In an RPM spec, BuildRequires: lists what the build needs, %check runs the tests and %files lists what the package ships. Check the rules against a few hundred hand-checked cases before you trust them. “No diff” can also mean the fix landed in a different package.

Practical rules for mining:

• Start collecting GitHub logs now. Public Actions logs vanish after 90 days.
• Use APIs and dumps with a plain, identified user agent. Browser-like clients hit anti-bot walls on Fedora and AUR pages.
• Keep snippets, not logs. Store only Drain-extracted snippets of at most about 2,000 tokens. Full logs in Fedora’s own dataset reach about 77 million characters, and Jev accepts 64k tokens per request.
• Use an LLM jury only for the ambiguous rows. 02’s price table puts labelling 500,000 items at about $75 with a Haiku-class model through batch APIs (prices fetched 2026-10-28, so as of October 2026). That makes 50,000 ambiguous items cost under $10. The labels lesson covers juries.
• Drop post-decision fields (anything written after the failure, such as the bug’s final resolution), so the label can’t leak into the input.

Don’t start with a neural network. Climb only as far as the eval forces you.

Rung 0: rules plus gradient boosting, on a CPU, in minutes. Gradient boosting builds many small decision trees, each correcting the last; LightGBM is the common library. Turn each log into features: which Drain templates appear, their counts, the failed step’s name. This is Fedora’s own precedent: rpmeta uses LightGBM and XGBoost for its build-time predictions. For decision 3, whose inputs are already a table, rung 0 may be all you ever need.

Rung 1: a decision API, zero-shot or few-shot. Put the 12-class label guide in the instructions and send the snippet; no training at all. Jev costs $0.042 per million input tokens as of October 2026, with output free, and independent tests measured about 0.1 to 0.65 seconds per request (Vals AI, 2026-10-06). Its calibration varies by task, though: one benchmark measured an ECE of 0.011, another 0.108. So fit your thresholds on distro data, never the vendor’s defaults.

Rung 2: distil into a small open model. Distillation means training a small model to copy the answers of a bigger one, or here, the answers the world gave. Follow the reference design: an encoder backbone, one head per answer type, and a calibrator.

• Backbone. A long-context encoder such as ModernBERT (149M or 395M parameters, 8,192 tokens, code in its pre-training) or an Ettin encoder (six sizes from 17M to 1B). Fedora’s SmolLM2-1.7B-Instruct-LogDetective is another possible start.
• Heads. A softmax head over the 12 classes for decision 2, and a sigmoid head (one probability) for decision 1.
• Calibration. Temperature scaling fits one number, T, that divides the model’s raw scores before the softmax: p = softmax(z / T). Worked example with two classes and raw scores z = [2.0, 0.0]: at T = 1, p = e²/(e² + 1) = 0.88. At T = 2, p = e/(e + 1) = 0.73. If answers given at 0.88 confidence are right only 73% of the time, T = 2 makes the number honest. See calibration.
• Cost. About $3-15 per training run for a 150M encoder, or $10-40 at 400M, on a rented H100 at $1.99-4.29 an hour (as of October 2026). These are estimates from 02’s 6·N·D FLOPs arithmetic.

Two more rungs exist, but you may never need them. Move to a small decoder (Qwen3-0.6B or 1.7B in the reference design) only when states exceed 8,000 tokens or the questions need world knowledge, such as package ecosystems or kernel versions. Add a reinforcement-learning stage with a proper scoring rule as the reward (a score, such as log loss, that is best only when the probability is honest) only once outcomes arrive from the world. Distill a decision model walks through the recipe.

C4 · L2 Container · Distro co-pilot

                      +-------------------------+
                      | [External system]       |
                      | GitHub Actions, Koji,   |
                      | COPR: runs and logs     |
                      +-------------------------+
                        |                   ^ webhooks in,
          pulls (HTTPS) |                   | retries out (HTTPS)
........................|...................| Distro co-pilot [system] .
:                       v                   v                          :
:  +------------------------+           +------------------------+     :
:  | [Container: Python]    |           | [Container: Python]    |     :
:  | Label miner, cron      |           | Action worker          |--+  :
:  | joins fail, retry, fix |           | acts on each failure   |  |  :
:  +------------------------+           +------------------------+  |  :
:         |                                   |           ^         |  :
:         | writes rows (SQL)            asks |           | odds    |  :
:         v                       (HTTP/JSON) v           | (JSON)  |  :
:  +------------------------+           +------------------------+  |  :
:  | [Database: SQLite]     |           | [Container: ONNX]      |  |  :
:  | Outcome store          |           | Decision server        |  |  :
:  | labels, timestamps     |           | three typed questions  |  |  :
:  +------------------------+           +------------------------+  |  :
:         |                                         ^               |  :
:         | reads (SQL)                             |               |  :
:         v                                         |               |  :
:  +------------------------+                       |               |  :
:  | [Container: Python]    | ships model (ONNX)    |               |  :
:  | Trainer, batch job     |-----------------------+               |  :
:  | CPU; rented GPU rung 2 |                                       |  :
:  +------------------------+                                       |  :
:                                                                   |  :
:                                                                   |  :
:...................................................................|..:
                                    asks for fixes (CLI)  escalates |
                                  +-----------------------+---------+
                                  v                       v (issue)
                       +---------------------+ +---------------------+
                       | [External system]   | | [Person]            |
                       | Coding agent        | | Maintainer          |
                       | writes the diffs    | | the uncertain tail  |
                       +---------------------+ +---------------------+

Two loops run side by side. The left column is offline: a nightly label miner joins each failure to its retry and its fix, and a batch trainer turns those rows into a model file. The right column is online. When a CI webhook arrives, the action worker asks the decision server for odds, then retries, hands the job to the coding agent, or opens an issue for the maintainer.

C4 · L3 Component · Decision server

................... Decision server [container] ....................
:                                                                  :
:  request (HTTP/JSON)                                             :
:         | a log                           | a bump's details     :
:         v                                 v                      :
:  +-------------------------+      +--------------------------+   :
:  | [Component: Python]     |      | [Component: Python]      |   :
:  | Log extractor           |      | Feature builder          |   :
:  | Drain3, csgrep          |      | version gap, kernel jump |   :
:  +-------------------------+      +--------------------------+   :
:         |                                 |                      :
:         | snippet (text)                  | features (array)     :
:         v                                 v                      :
:  +-------------------------+      +--------------------------+   :
:  | [Component: ONNX]       |      | [Component: LightGBM]    |   :
:  | Log classifier          |      | Bump model               |   :
:  | encoder, 12 classes     |      | gradient-boosted trees   |   :
:  +-------------------------+      +--------------------------+   :
:         |                                 |                      :
:         | logits (array)                  | raw score            :
:         +----------------+----------------+                      :
:                          |                                       :
:                          v                                       :
:            +---------------------------+                         :
:            | [Component: Python]       |                         :
:            | Calibrator                |                         :
:            | temperature per question  |                         :
:            +---------------------------+                         :
:                          | p (JSON)                              :
:                          v                                       :
:            +---------------------------+                         :
:            | [Component: Python]       |                         :
:            | Policy                    |                         :
:            | threshold, abstain, route |                         :
:            +---------------------------+                         :
:                          |                                       :
:                          v answer, p, action (JSON)              :
:                                                                  :
:..................................................................:

A log takes the left path. The extractor cuts it to the informative lines with Drain3 (a Python version of Drain) and csgrep (a tool that parses compiler messages), and the classifier scores the 12 classes. A bump takes the right path through a tree model. Both meet at the calibrator, which turns raw scores into honest probabilities. The policy then acts, abstains or escalates.

An eval you can trust needs the right split, the right metrics and the right baselines. Verifiable rewards and evals covers the general method.

Splits.

• Split by time. Train on events before 2026-07-01 and test on events after 2026-08-01. The month-long gap stops a fix that lands on both sides from leaking into the test.
• Group by package. No package appears in both train and test, so the model can’t just memorize names.
• Watch for memorization. Test the rung 1 API and any LLM baseline on failures that happened after their training cutoff, since a model may have read the outcome.

Metrics.

• Macro-F1: the F1 score (a balance of precision and recall) averaged over classes, so rare classes count as much as common ones. Report per-class recall alongside it.
• ECE (expected calibration error, the average gap between stated confidence and actual accuracy), plus reliability curves.
• Coverage at fixed precision, the operational one: the share of failures the model handles alone while at least 98% of those automatic actions are right.

# Coverage at fixed precision: the share of cases the model may handle
# alone while at least 98% of those automatic answers are right.
import numpy as np


def coverage_at_precision(p_max, correct, target=0.98):
    """p_max: confidence per case; correct: 1 if the answer was right."""
    order = np.argsort(-np.asarray(p_max))          # most confident first
    hits = np.cumsum(np.asarray(correct)[order])
    n = np.arange(1, len(order) + 1)
    ok = np.nonzero(hits / n >= target)[0]          # prefixes that pass
    return 0.0 if ok.size == 0 else (ok[-1] + 1) / len(order)


# Ten test failures: the five most confident answers are all right.
p = [0.99, 0.98, 0.97, 0.95, 0.93, 0.90, 0.80, 0.70, 0.60, 0.55]
y = [1, 1, 1, 1, 1, 0, 1, 0, 1, 0]
print(coverage_at_precision(p, y))                  # 0.5

Read it by hand. Sorted by confidence, the first five answers are all right (5 of 5 = 100%). The sixth is wrong, so precision drops to 5 of 6 = 83%, and it never climbs back above 98%. So you may automate the top half: coverage 0.5.

Baselines. The model must beat three:

1. the rules alone;
2. Log Detective’s explanation, mapped to a class;
3. a frontier LLM, zero-shot.

To win, it must beat the rules on coverage at 98% precision, at under a hundredth of the frontier LLM’s cost per decision.

How big a gold set? By binomial arithmetic, 385 hand-checked items pin accuracy to within about ±5 points at 95% confidence. That’s why the plan uses 300-640 gold items per decision.

1. A weekend: the first milestone. Train a rules-plus-LightGBM “will a retry pass?” model on Universal Blue runs and wire it to auto-retry in terra-fedora’s workflow. No LLM needed.

2. Weeks 1-4: mine. Run the label miners nightly and build the three datasets and gold sets above.

3. Try rung 1. Point a decision API at the gold sets. Stop rule: if rung 1 meets the target, don’t train anything.

4. Shadow mode, two weeks. The co-pilot predicts but doesn’t act. You compare its answers with what you did.

5. Live, four weeks. Run it on terra-fedora plus a mirror of Universal Blue’s CI events.

Success means all three of these:

• At least 50% of failed runs are resolved without human attention (an auto-retry, or a templated fix that passes the build), with no more than 2% wrong automatic actions.
• Median maintainer minutes per failure cut 3x. Log each touch to measure it.
• ECE at most 0.05 on the test split.

Prices as of October 2026; the budget is the research’s estimate.

Where this sits. Fedora already runs both halves of the idea separately: an LLM that explains failures with no published accuracy (Log Detective) and a tree model that predicts build time (rpmeta). What doesn’t exist publicly is a typed, calibrated failure classifier scored against build-farm outcomes. The plan above is that missing piece, at bootstrapper scale. The reference design’s default recipe labels 100-200K decisions with a cheap LLM teacher and trains a small encoder, for about $300-800 all-in. Distro work can skip most of that labelling bill, because the build farm is a free teacher.

Open problems:

• Class drift. New toolchains create new failure classes; the C23 class only appeared in bulk once GCC 15 made C23 the default. Keep “other” and re-cluster it every release.
• Merge isn’t correctness. For bump-safety labels, bot PRs merge 92-99% of the time, so negatives are scarce. Add reverts, rollbacks and later CI failures as negatives.
• Vendor calibration doesn’t transfer. Jev’s ECE ranges from 0.011 to 0.108 across benchmarks, so every threshold has to be refitted on your own distro data.
• No head-to-head found. The research found no published comparison of Log Detective, a frontier LLM and a small classifier on the same outcome-labelled failures. It also couldn’t verify recent academic work on predicting build failures, so treat this as a gap, not proof that none exists.

• fedora-copr/logdetective: the Drain-based snippet extraction you can reuse, and the data your eval should map from.
• Hugging Face: log_detective_qna: the 640-row test split, under CDLA-Permissive-2.0.
• fedora-copr/rpmeta: the LightGBM precedent for rung 0 inside a real build system.
• ModernBERT, arXiv 2412.13663: the long-context encoder suggested for rung 2.
• Vals AI: independent evaluation of Jev: latency and calibration numbers you should check before trusting rung 1.
• GitHub Actions log retention: why the label miner has to start today.