The decision map

The idea in one line: Most recurring distro chores hide a typed question. Call a tool when one gives the exact answer fast, and model only the questions that are frequent, slow to check and labelled for free.

Choosing which questions to model is the biggest decision in this case study. Get it wrong one way and you train a model to imitate a tool that already answers exactly, in seconds, for free. Get it wrong the other way and you skip a question whose labels the build farm writes for you every day.

Where the time goes showed the hours pile up in diagnosis, waiting and selection. This lesson turns them into a map of 23 typed questions, marks which have exact oracles and free labels, and covers the AI Fedora already runs and the policies that limit what a co-pilot may do.

Recall the contract from anatomy of Jev. A decision model takes a state (text or JSON describing the situation) plus typed questions, and returns a probability for each answer you defined in advance. Jev has three production types:

• Choice: pick one of up to 255 options; returns a distribution over them.
• Score: rate on 2 to 10 ordered levels; returns a distribution and an expected score.
• Noul (short for Bernoulli): a yes/no question; returns one probability of “yes”.

A worked example. It’s 3 a.m. and an akmods CI run fails at its “Test Image” step. A maintainer asks three things, often without noticing they’re separate:

State: the failed step's last 2,000 tokens of log, plus run metadata

Noul    "Will a plain retry pass?"               P(yes) = 0.85
Choice  "Which failure class is this?"
          transient-network 0.80 | kmod-vs-kernel 0.10 | other 0.10
Score   "How risky is this kernel bump for kmods, 1-5?"
          expected score 1.4

The numbers are illustrative, but 0.85 is a sensible start: 40 of 47 re-run akmods jobs passed, so it’s the base rate before the model reads the log. Each answer maps to an action (retry, run the network playbook, hold the kernel), and each action has a cost when it’s wrong. That’s how probabilities become automation. Writing good questions and label guides is its own craft: see designing the questions.

Here are 23 recurring decisions in distro work as typed questions. The “time each” column is the researcher’s estimate, since no time logs exist. “Verifiable” says how cheaply and reliably the true answer arrives.

A few terms. Bodhi is Fedora’s update system, where testers give each update karma (+1 or −1 votes). EPSS is a machine-learning model that scores how likely a CVE is to be exploited. Stack Linux is the operator’s planned public distro: stack install would pick an install method for a name, and stack doctor would gather Secure Boot state, the GPU driver and failed services into a report (tsaqib/terra-fedora/STACK-CLI.md:100-151).

An oracle here is a tool that gives the true answer, not a guess. Five rows have exact, fast oracles:

• #1, the solver. Dependency solving is NP-complete in theory (no known method is fast for every case) (Abate et al., arXiv 2011.07851), but libsolv inside dnf5 answers “does this set resolve?” exactly, in seconds.
• #4, file-provides queries. dnf provides and pacman -F say which package ships a file.
• #15, patch application. Either the patch applies or it doesn’t.
• #21, config validators. mango -c <file> -p reports bad keys (tsaqib/mycachy/CLAUDE.md:76).
• The lookups in #22, which are repository queries.

Why modelling these is a mistake, in numbers: terra-fedora has 187 package rows. A “will this resolve?” model at 95% accuracy gets about 187 × 0.05 ≈ 9 wrong. The solver gets 0 wrong, in seconds. Call the tool and spend the model budget elsewhere.

So the rule is simple:

Is there a tool that gives the exact answer?
|
+-- yes, in seconds -----------> call the tool. No model.
|                                (solver, provides, validator)
|
+-- yes, but it takes 20-180 min -> the model predicts now,
|                                   the tool confirms later
|
+-- no (hardware, taste) -------> the model only routes;
                                   a human decides

The best model targets have three things at once: high frequency, a slow oracle, and free labels. On the map those are #2, #3, #8, #9, #10, #12 and #16. Two rows can’t be automated: hardware truth (#11) needs a real laptop, and taste (#7) is the product itself, since the 25 rejected features are the distro’s identity. A model can only pre-screen and route those.

A naturally labelled outcome is a label the world produces anyway, such as “this build failed”, “this PR merged” or “this retry passed”, so no annotator is needed. The labels lesson explains why this is the 10x data source. Distro work is unusually rich in it.

Counts were measured from each public API or dump on 2026-10-28. See Log Detective’s dataset, rpmeta’s, Buildroot stats and OSV data. One near-miss: Renovate’s Merge Confidence crowd data would suit #8, but it covers npm, PyPI and other language ecosystems, not distro packages, and its algorithm is private (docs).

Two labels the build farm writes for you. Human annotation won’t scale: Log Detective’s 1,228 reports are under half a percent of a single year’s 284,204 failed Koji tasks. But the farm records what happened next:

1. The retry outcome. A failed job re-run with identical inputs that then passes was transient.
2. The fix diff. Diff the recipe against the next successful build of the same package, and map the change to a class. Building the co-pilot turns this into code.

Access is narrowing. Anubis, an anti-bot wall that challenges browser-like clients, now fronts Fedora’s web pages, the AUR, Buildroot and Log Detective’s site; plain API clients still pass. GitHub deletes public Actions logs after 90 days. The research reports even disagree on Koji: one pulled full-year counts, another timed out on its count queries. Plan for pagination and a nightly job.

The traps. Free labels leak in predictable ways; one survey found leakage errors in 329 papers across 17 fields (Kapoor & Narayanan, 2022-07-14). Drop every field written after the decision, such as a bug’s final resolution. Split by time and group by package, or near-duplicate failures land on both sides. Remember that merged isn’t correct and stable isn’t tested: one Fedora update went from submitted to stable in four minutes with karma 0. And keep the date with every label, because policies drift. More in verifiable rewards and evals and, for the licence column, data rights.

Log Detective is a Fedora service that explains failed RPM build logs with a large language model, aimed at newer packagers. It first cuts the log down with Drain, an algorithm that groups log lines into templates so the rare, informative lines stand out (fedora-copr/logdetective, read 2026-10-28).

• Deployment. Since May 2026, Packit (the service that runs builds for Fedora’s dist-git pull requests) requests an analysis after every failed scratch build (Fedora Magazine, 2026-05-13). The setup example loads a 3.9 GB Granite 4.0 model file. Its authors say an explanation takes about a minute (DevConf.CZ 2025 talk).
• Models and data. The fedora-copr Hugging Face org holds 13 models, including SmolLM2-1.7B-Instruct-LogDetective, and the log_detective_qna dataset (3,367 Q&A pairs, 640 of them a test split).
• Results. None published. The Magazine article says the tool likely offers little to people with years of experience, and the SmolLM2 model card reports no evaluation.

The deeper problem is the output type. An explanation is free text, so you can’t score it automatically. A typed answer (“which class?”, “transient?”) can be scored against outcomes the build farm already records.

rpmeta predicts how long an RPM build will take from package metadata and builder hardware. It’s a gradient-boosting model (LightGBM and XGBoost: many small decision trees, each correcting the errors of the last), trained on 705,766 COPR build records and served over a REST API (fedora-copr/rpmeta, read 2026-10-28). No accuracy is published. So Fedora runs two models with two philosophies: an LLM explainer and a cheap, typed predictor. The second is the Jev-like one.

• Kernel backports, 2018. Sasha Levin trained a neural network on “should this commit go to the stable kernels?”, labelled by which commits had reached stable. It reached about 85% precision (of the commits it picked, how many were right) and 80% recall (of the right commits, how many it picked), for about $2,000 of GPU time (LWN, 2018-09-12). That’s a single source.
• Moderation. Universal Blue ran GitHub’s ai-moderator action to label issues spam or ai-generated; Bazzite removed it on 2026-09-29 as an “action that’s being sunset” (ublue-os/bazzite commit d3c4d31f25).
• CVE scoring. EPSS v3 claims to be 82% better than earlier EPSS models at spotting exploited vulnerabilities (arXiv 2302.14172), again from a single source.

Distro projects have written AI policies, and they decide what a co-pilot may do upstream.

What it means. Every ban or restriction targets content in contributions. A decision model that classifies a log, retries a job, ranks CVEs or flags spam contributes no content, and Fedora names that zone explicitly. The generative half of a co-pilot carries the policy risk, and only when its patches flow upstream. For a personal derivative like terra-fedora, none of these rules binds until work goes to Fedora, COPR or Terra.

The frontier here is the plumbing, not the model. Typed decisions in kernel maintenance worked in 2018. Fedora’s newest models are an LLM explainer and a boosted-tree regressor. The strongest “agent” by volume is a bot with no language model, behind 41% of merged nixpkgs pull requests. The research found no published typed failure classifier scored against build-farm outcomes.

Open problems:

• No accuracy for Log Detective. Its DevConf talk aims for near-perfect explanations; Fedora Magazine says it offers little to experts. No numbers settle it.
• Merge is a skewed label. Bot PRs merge 92-99% of the time, so “safe to merge” needs revert and regression signals too.
• Hardware truth has no dataset linking image versions to boot success on specific GPUs.
• Research gaps. Recent academic work on predicting distro build failures, and Canonical’s AI experiments in packaging, couldn’t be verified.

• Fedora Magazine: Log Detective in Packit: how Fedora wired an LLM failure explainer into its PR CI, and its own caveats.
• fedora-copr/rpmeta: the existing typed, non-generative model inside Fedora’s build system.
• Fedora AI-assisted contributions policy: the clearest statement of what automated validation may do.
• LWN: Machine learning and stable kernels: the 2018 precedent for a typed decision model in kernel maintenance.
• Kapoor & Narayanan: Leakage and the reproducibility crisis: the leakage traps that bite outcome-labelled data.
• Debian vote 2026/002: the options, the tallies and the wording Debian settled on.