ADR 010: Discovery — what is in this tree, and whose code is it

• Status: Accepted. Implementation progress is tracked in the decision register, which is the single place it is stated
• Date: 2026-08-12
• Scope: What gets scanned, what gets scored, which analyzers are applicable, what the report states about the tree
• Related: ADR 005, ADR 006, ADR 007

Context

Nothing looked at the repository before deciding what to run and what to score. The pool was resolved from the operator's concerns, depth and licence policy; the scan walked whatever include_extensions listed; and the score was computed over whatever that produced. At no point did the tool ask what is actually in this tree.

Two defects came out of that gap, both measured, both invisible on every repository the tool had previously been tested against.

Tools claimed coverage of languages that are not present

kilo is one C file. complexipy, mypy, pydocstyle, radon, ruff and vulture all ran, produced zero measurements and zero findings between them, and the coverage section reported documentation, style, types and dead-code as examined. Identical on gson and ripgrep.

The rule that produced this is correct in general and wrong here: a tool that ran and found nothing has examined its concern. That is what stops "vulture found no dead code" collapsing into "nobody looked". It holds only when the tool had input. A Python linter on a C repository examined nothing, and recording it as coverage is absence-as-value arriving in the section written to prevent absence-as-value.

Other people's code was scored as the team's own

Adding .mjs/.cjs to the include list — a correct fix for genuinely unread source — pulled 10,759 files from packages/mui-icons-material/lib/ into the calibration corpus. Six-line SVG icon wrappers, one per icon, written by builder.mjs in the same package. They dropped material-ui's duplication from 3.729 to 2.321 and moved CALIBRATION_C from 2.6279 to 2.6414.

The same class elsewhere: 5,090 of whisper.cpp's findings sit in ggml/, a library synced from another repository. 16,719 of lapack's sit in TESTING/, which is_test_path missed because it matched test and tests but not testing. transformers' duplication read 25.567 with its generated modular files included and 13.690 without.

The existing defence was a list of directory names — vendor/, third_party/, node_modules/, migrations/. ggml/ was not on it, so a vendored tree counted as first-party because of what its authors happened to call it. Extending the list is not a fix; it is the same mechanism with one more entry, waiting for the next project to choose a different word.

Decision

A discovery pass runs before anything else, and classifies every file by language and provenance from evidence the repository itself provides.

1. Classification is by evidence, never by name

Each rule points at something checkable, and the evidence is recorded beside the verdict so a reader can disagree with it:

test_every_classification_carries_the_evidence_for_it refuses a classification with no evidence. A verdict nobody can check is a guess with better manners. The same reasoning forbids a graphics/ name in the exclude list: an asset directory is recognised by what it holds, never by what its authors called it — the exact mistake ggml/ was.

Ignore rules are evidence only where the repository wrote them (D175). Only the per-directory .gitignore files are read, never .git/info/exclude or a user's core.excludesFile: those differ by machine, and one commit would score differently on two laptops (P1). A tracked file stays first-party whatever a rule matches, because git tracking it is the repository saying it is source; an untracked file no rule matches stays first-party too, because that is uncommitted work (D56). A classified tree's every scanned file leaves the score — its HTML, XML and JSON as well as its source — and is counted in summary.generated_files.

2. Where there is no evidence, the answer is first-party

This module removes code from someone's audit. A false positive silently deletes their own work from their own report, which is the worse direction of error, so the bias is deliberately toward leaving code in.

That bias was tested immediately and in the right direction. The first banner rule matched substrings, and classified material-ui's hand-written formatMuiErrorMessage.ts as generated because its docstring says "It's imported by the code generated by @mui/babel-plugin". Prose about generated code, mid-sentence. Two further iterations were needed: requiring the phrase to open a line rejected curl's genuine /*** Sample code generated by the curl command line tool ***, which does not. The rule that holds is position, not presence — a declaration comes before the prose, so nothing that reads like a finished sentence may precede it, on its line or the one above.

3. A tool with no input is not-applicable, not ran

A tool whose catalog languages do not intersect the tree records not-applicable with a reason naming both halves — "reads python; this tree is C, so it had nothing to examine" — and claims no concern coverage.

Any overlap is enough to run a tool. lizard reads a dozen languages and must not be excluded from a C repository for the absence of Kotlin. Suppressing a tool removes real evidence, so an inapplicable verdict has to be positively supported by a language list sharing nothing with the tree.

Running a tool and covering a language are different questions, and treating them as one was wrong. This section originally stopped here, and the validation sample showed what that costs: json holds 6 Python files among 305 C++ ones and lapack 1 among 2,884 C files, so a single build script made every Python-only tool applicable and both repositories claimed types and style for the whole tree. Binary applicability is right at the extremes — curl, with no Python at all, correctly reported 3 of 11 — and wrong everywhere in between. See §3a.

The two vocabularies are mapped explicitly. The inventory says C++ and C#; the catalog says cpp and csharp. Comparing them directly finds no overlap and marks every C++ tool inapplicable — the same defect as concerns against concepts, and the third instance of it in this codebase. CATALOG_LANGUAGE is held complete against KNOWN_SOURCE_SUFFIXES by a test.

3a. Coverage is claimed per language, without a threshold

Amended 2026-08-13, superseding the repository-wide coverage claim above.

The obvious fix for §3's gap is a share threshold — a language must be at least N% of the tree for its tools to count — and it is the wrong fix. It tunes a number until the two repositories in front of you behave, and that number then applies to every repository forever, distorting a genuinely polyglot tree to make a mostly-C++ one read correctly.

Per-language coverage needs no cutoff. "mypy covers types for Python" is true when Python is 2% of the tree and true when it is 98%, so there is nothing to tune and no way for a rule fitted to one repository to distort another. test_the_answer_does_not_depend_on_the_language_mix holds exactly that property.

The repository-wide claim is then scoped to the languages that were scored, not composed across every language present. The first attempt intersected across all of them and erased click's coverage entirely — a healthy Python library holding one shell script that no tool reads, where one unscanned file rewrote the answer for six hundred scanned ones. A language the scan never opened is reported by the unread-source rule and must not also rewrite the coverage claim.

Scoping is necessary and not sufficient: concepts_covered: [types] still reads as "this repository is type-checked" on a tree that is 98% unread C++. So the record carries scored_languages beside it and the report renders the full matrix. A claim that cannot be read without its scope is a claim that will be misread.

4. Generated and vendored code leaves the scored population

It is counted and reported, never silently dropped: summary.generated_files, summary.vendored_files and summary.classifications state what was set aside and why. Excluding code without saying so would replace one dishonesty with another.

5. The corpus is re-derived only after this lands

Recorded because the sequencing is load-bearing. Widening what the tool reads widens what contaminates it, in the same act. Any constant fitted before provenance detection embeds material-ui's icon build in the scale.

Options considered

A. Extend the exclusion list. Add ggml/, TESTING/, lib/. Rejected: lib/ is an ordinary source directory in most repositories, and the approach has already failed once for exactly the reason it will fail again — it can only know the names it was told.

B. Detect generated code by structure — a directory of many tiny mutually-near-identical files. Would have caught material-ui's icons. Rejected: it is a heuristic over shape rather than a fact the repository states, and it would misclassify legitimate repetitive code. This project's whole method is evidence over inference.

C. Ask the operator to configure it. Rejected as the only mechanism — a tool that requires per-repository setup before its score means anything is a tool nobody runs on a repository they did not write. Configuration remains available and overrides discovery.

D. Score everything and report provenance alongside. Rejected: it leaves the number wrong and asks the reader to correct it mentally. The score is the artifact people quote.

Consequences

• Scores move on any repository containing generated or vendored code, and move correctly. This is a public change and rides with the schema bump.
• CALIBRATION_C was re-derived after this landed and kept at 2.6279: a 2,000-resample bootstrap puts the 95% interval at [2.2522, 3.4718], both candidates roll the corpus median to exactly 4.0000, and they differ on one repository in forty by 0.1. See tools/calibration/sampling_error.py.
• Coverage shrinks honestly on non-Python repositories. gson, cobra, ripgrep and curl went from reporting 9 of 11 analyzers contributing to 3 of 11, because the other six had nothing to read.
• Detection is partial and will stay partial. Ecosystems this does not yet recognise report their code as first-party, which is the safe direction. New rules are added when a real repository demonstrates the need, not speculatively.
• One more pass over the tree per audit. Measured at under a second on babel's 17,818 files.

Invariants

1. Every non-first-party classification carries the evidence that produced it.
2. A file with no evidence against it is first-party. Doubt leaves code in the audit.
3. A tool whose languages do not appear in the tree records not-applicable; any overlap makes it applicable to run. 3a. Coverage is claimed per language and never by a share threshold. The repository-wide claim covers only the scored languages, and is published with those languages named.
4. Generated and vendored files are excluded from the scored population and counted in the report.
5. Classification reads the repository's own declarations. No rule may be a bare list of directory names.

Amendment, 2026-09-24 — Shell is scored, so click's case now narrows the claim

§3's motivating case was a Python library holding one shell script that no tool reads and the scan never opened. From 3.9.0 Shell is parsed and in the default include_extensions, decided by the operator when Shell shipped: parsed by default, like every language with a scanner, rather than only on request. A shell script is therefore a scored language, and the scoped intersection correctly narrows the repository-wide claim for it — no tool checks types in shell, so a repository with scored scripts cannot claim types for the whole of it, while coverage_by_language still says Python had it. The rule in §3 is unchanged; its example moved from "unscanned" to "scanned". The property that one unscanned file must not rewrite the claim is kept, with Elixir standing in (test_one_stray_unscored_file_does_not_erase_a_python_library).