Product intent

The governing document. When another document, a code comment, or a release note disagrees with this one, this one is right and the other is a bug.

What this is

A deterministic maintainability audit that produces a bounded work order for an AI coding agent.

The role, stated once so the rest follows from it: this is the independent check on a generator's output — the role a generator cannot fill for itself. A tool that writes code and also grades it is reporting on its own work. This one writes nothing, runs no model, and re-derives the same verdict from the same inputs, which is what makes its answer usable as evidence rather than as an opinion. That is a separation-of-duties property before it is a technical one, and it is why determinism is load-bearing rather than a preference: a checker that cannot reproduce last quarter's verdict cannot attest to anything. A model reviewing a generator's output does not supply that independence either. It can find real problems, and hostile audits by a model are part of how this tool is built, but a model's verdict is an opinion until something deterministic reproduces it. Model review is a finder that sits on this tool's evidence, never a substitute for it.

The corollary is a boundary, not a modesty statement. The better a code-generating agent becomes, the more an independent check on it is worth — so agents and the platforms around them are the customer for what this produces, not the competition. What this tool does not compete on is listed in the roadmap: generation, security scanning, compliance profiles, comprehension of languages it does not parse, IDE experience, model routing.

Not a linter (those exist and are better at it), not an AI reviewer, not a quality dashboard. The scanner and the score exist to aim the remediation prompt. If the prompt were removed, the rest would be a worse version of tools that already ship.

The primary surface is an AI chat host — an IDE assistant or chat window driving the MCP server. The CLI is the automation surface for CI and scripts. Almost no user runs the audit at a bare terminal, so a capability that exists only behind a terminal prompt does not exist for most users. The individual wiring defects found when this contract first met a real host—and their closing tests—are recorded in the chat-surface defect register.

Chat, MCP, and an interactive CLI TTY are one setup: the same questions. MCP is the chat transport, not a third questionnaire. The questions are the set in first run and questions (setup_questions in source). There is not a CLI setup, a chat setup, and an MCP setup. A surface that asks a subset is a bug. CI, a non-TTY, and an explicit --config never ask — that is not a different question set; there is no operator. MCP allowed-root grants are a server boundary, not first-run setup.

The job it does

Agents write code faster than humans can read it. The ratio of code-written to code-reviewed has collapsed, and unmaintainable code that used to accumulate over years can now accumulate in an afternoon. The same speed is the way out: an agent pointed at specific, deterministic findings can fix them at the same rate they appear.

The loop this product closes:

1. Measure pressure points deterministically, with no LLM involved.
2. Apply one uniform standard, so "better" and "worse" are not an argument.
3. Emit a prompt scoped to those findings only, with explicit instructions not to refactor beyond them.
4. Hand it to the agent. Review a scoped diff instead of a speculative rewrite.
5. Check the diff against the work order that produced it — did it stay inside the named paths, did it silence anything, did any dimension move backwards.

Step 3 is the product. Steps 1 and 2 are in service of it. Step 5 is what makes step 3 a constraint rather than a request.

For most of this project's life the loop ended at step 4, and the honest statement of that limit was: the work order bounds the agent by instruction, and nothing verifies the diff came back inside it. That is no longer true. --conformance compares the diff against the paths the report named and answers two separate questions — stayed in scope, and silenced nothing — while --fail-on-regression ratchets the dimension scores across scan history. --attestation-output composes the three into one record. See the remediation hole.

What the loop still does not do, and is not going to: step 5 reads the diff's shape, never its correctness. It establishes that a change went where it was told and hid nothing. Whether the change is right is a claim this tool does not make, and the attestation says so in its own text rather than leaving a reader to assume otherwise.

The writable oracle

A check that reproduces is still a self-report if the work under review could author the record. This package is a separate process from the generator — it writes no source and runs no model — so an agent fixing a target cannot rewrite the rubric by editing the target. The remaining hole is the oracle that does live in the tree: the tests.

The pairing rule lets a test file into a remediating diff even when the work order never named it. That is the right rule for writing the failing test with the fix, and it is the file an agent can poison — skip, alias a matcher, replace an assertion with assert True, delete the assertion. clean fails on a suppression directive the diff added to a file the work order named, and — since 3.11.0 — on any test file in the diff made unable to fail: a skip or suppression, an assertion that cannot fail, the assertion API replaced, a failing assertion swallowed, or assertions deleted with nothing in their place (_oracle, reported as oracle_weakened). Held to a precision bar frozen before the detector was written.

Three homes, decided 2026-09-26. The first was built in 3.11.0, behind a precision bar frozen before the detector was written.

1. This package, --conformance clean. Detect oracle-weakening in the remediating diff as the same second verdict as a skipped test. Still shape. Still not correctness. Implementation is gated on a frozen precision bar; the existing marker list is deliberately narrow because a check that cries wolf gets switched off.

2. tools/prove_falsifiers.py stays this repository's gate. It already proves a newly cited test fails at the base commit. Generalizing that to a stranger's suite — every remediation, new tests must fail on the left side of the revspec — is the capability the roadmap names as "the test that defends nothing." It remains unscheduled. It does not become a first-order step in the product loop, and it does not license a claim that the tests would catch a regression.

3. Freeze, a fresh seat, and a human on the test diff sit outside this package. This auditor does not sequence remediation, freeze the suite before an agent writes, or launch a review that never saw the work. Those are jobs for the seats that already split test-writing, implementation, and audit, and for the operator reading the test hunk. This package's job at that seam is clean on the diff.

See the decision register.

The ask beside the diff (Decision 13, 2026-09-27). The other failure leaves the tests alone. Frozen tests that were wrong before the agent arrived, and production code bent until they pass: nothing weakened, the record untouched, and the diff still wrong because nobody asked for that code. --conformance reads a diff against an ask when the ask is this tool's own work order. Since 3.11.0 it also takes an ask it did not write — --ask FILE, paths and globs from the operator — and reports every change outside it — other than a test pairing to an asked file — production and test apart, with any glob that matched nothing listed. Whether code inside the ask is what was wanted stays a judgment this package does not make.

Semantic judgment and economic priority

The next useful boundary is not a larger generic smell catalog. It is a way to make repository knowledge explicit enough that a deterministic tool can act on it. Source code alone can observe that the same bare strings are repeated in dispatch, validation and capability checks. It cannot universally know whether those strings should be constants, an enum, a value object, or reified operations with behavior and generic result types. That choice contains domain intent.

ADR 003 accepts three honestly different outputs: facts established by a language type system, violations of checked-in repository policy, and design-review candidates inferred from structural or historical friction. Determinism means that the same tree, analyzer versions and policy produce the same result; it does not require the tool to pretend that one universal rule contains every repository's semantics. The current increment is TypeScript only. These results aim the bounded prompt; they do not change the standard score, range, grade, or evidence status.

ADR 004 v1 is shipped. Change history supplies recurrence and hotspot churn; the user may supply loaded labor cost (low/base/high) and optional horizon, reliability tier, review burden and incident cost. The result is a labeled scenario range in a separate report block and a reordered work order, not a modification of the uniform score and not a prediction. The product direction is:

deterministic finding + explicit repository semantics + configured economic context -> a bounded work order ordered by estimated exposure

ADR 003 option C is accepted and its TypeScript precision increment is current work. Both paths preserve the offline, no-LLM audit path and the evidence standard below.

Who it is for

• Teams running agents in the dev loop who are tired of unbounded cleanup PRs.
• Repos that want a maintainability gate without a SaaS analyzer or shipping code to a third party.
• Solo developers who want one deterministic audit pinnable in a Makefile or CI script.

What it promises

Each promise is falsifiable, and named so a failure can be reported against it.

The developer's own analyzers are read, not run

Decided 2026-10-12 (Decision 17). The catalog holds some 760 analyzers and this tool runs a pinned handful. It does not set out to run the rest: the developer runs their own tools where they already run them, in their own build or pipeline, and this tool reads the SARIF they wrote and states what it examined. Running a project's build to obtain that output is not this tool's job — a build executes the repository it builds.

Why P1 was amended

P1 read "no network, no LLM" when the tool was pure computation over a file tree. Under ADR 006 it invokes external analyzers, and some of those live in ecosystems whose normal installation path is a fetch — npx for the Node tools most obviously.

Two honest options existed: refuse to acquire tools, which makes the multi-language story depend on every user hand-installing a dozen ecosystems; or provide an explicit acquisition choice and disclose it. The latter was chosen, so the promise separates analysis from acquisition. Analysis still touches no network and no model: this agent does not transmit the audited source. Acquisition defaults off. When a user enables analyzers.acquire_tools, a missing tool may be fetched and its version is recorded in the report so the run remains reproducible.

Why P1 names these excepted fields. An audit ran the same tree twice and found the reports differing by one millisecond, on an analyzer's seconds. The promise said "same report out" and the determinism check had been quietly stripping root, git_status_short and every seconds for as long as it had existed — so the promise was false of the JSON a consumer actually diffs, and true only of the test's own projection.

The exceptions are legitimate: a duration is a fact about the run, not about the repository, and an absolute path is not portable. generated_at — the wall-clock time the audit ran, added so the report can state under its title when it was produced — is the same kind: a fact about the run, which the determinism check strips because two audits of one tree legitimately differ on it. What was wrong when only the first three were stripped was that they were undisclosed, which is exactly the shape this table exists to prevent — a promise kept green by a check narrower than the promise. They are named above now, and test_the_determinism_exceptions_are_exactly_the_disclosed_ones fails if the strip list grows without this page growing with it.

Why P1 names the history window. The run-describing fields above are recorded and not compared. The history window is an exception of a different kind, and it went undisclosed longer: DEFAULT_SINCE is 12 months ago, which git resolves against the wall clock at the moment the audit runs. It is not configurable.

So "same tree, config, pinned analyzer versions and scan history in, same score out" is true within a run and false across days. A commit ages out of the window overnight and the churn, hotspot, coupling and ownership rates move underneath an unchanged tree — no input changed, and the grade did. That is a real determinism limit, not a rounding one, and it is the same shape as the three fields above: legitimate behaviour kept invisible by a check narrower than the promise. The determinism test compares two runs seconds apart, which is precisely the interval over which this cannot fail.

Named rather than removed, because a fixed absolute window would make every report a different question over time, and pinning the window per repository is a config decision nobody has asked for. test_the_history_window_is_disclosed_as_clock_relative fails if the window changes without this paragraph changing with it.

Why P1 says "pinned" and CI splits gating from drift detection. P1's determinism is conditional: same tree, config, pinned analyzer versions and scan history in, same evidence out. The condition is real and the report records each analyzer's version so two runs can be compared on it.

The PR gates now do pin it. verify and audit install the twelve pip analyzers through checked-in constraints/analyzers.txt, so a green gating run certifies P1 against a fixed analyzer input set rather than against whatever PyPI served that day.

The weekly scheduled run keeps the old drift signal on purpose. It installs the same top-level analyzer pool unpinned in a throwaway virtualenv, freezes what pip resolved, and fails visibly on a diff against constraints/analyzers.txt. That preserves the D48 signal — an unchanged main going red because an analyzer shipped — without asking the merge gate to certify a floating environment. test_the_gating_pipeline_pins_the_analyzers_and_the_scheduled_run_detects_drift holds the workflow and this page together.

What P1 does not police. External analyzers run as local child processes (_runner). This package does not wrap them in a network sandbox, and it does not inspect whether they open sockets. A future SaaS CLI adapter would be a different product. The host that prints a report into an IDE chat is also outside this process. P1 is enforced as determinism plus no HTTP client in src/ that uploads the tree — not as “no packet can leave the box.”

The prerequisites this creates are listed in the analyzer pool rather than left for a user to discover from a failure.

Why P7 and P8 were added

P1–P6 are all self-consistency promises: the machine agrees with itself, the arithmetic checks, nothing is concealed. A repository holding one function and one test passed all six while being reported as 5.0 / A+, evidence complete, verified — deterministically, under the same rubric, with the arithmetic correct and nothing withheld. Six for six, and the result was worthless.

Nothing in P1–P6 asks whether a number corresponds to anything about the code, so four days of hostile audits could not see it. Worse, the evidence standard classifies the rubric as Tier 2 — a judgment that "needs no validation study and no apology" — which is right for weights and became a principled-sounding reason never to ask whether a specific score was defensible.

P7 is that missing question, stated as a falsifiable property. P8 makes it checkable: a score cannot be judged sufficient unless the report says what produced it. See ADR 005 and ADR 006.

What it must never claim

This list exists because the project has already broken it once and retracted. Adding a claim in these shapes requires deleting it from this list first, with the evidence that earns it.

• That it detects AI-written code, or that any metric distinguishes AI authorship. Tested with a matched control and retracted. Any revival needs a pre-registered design with commit-level authorship.
• That it detected TDD/BDD chronology or test quality. It may report TDD-shaped file pairing and constructs. It may not say tests were written first, or that they would catch a regression, unless the operator opted into suite execution and coverage was parsed.
• That the score predicts business outcomes — defect rates, delivery speed, cost. No outcome study has been run. The rubric is a standard, not a model of the future.
• That a configured economic scenario is a prediction or observed saving. User inputs make assumptions explicit; they do not validate the model. Scenario ranges must remain separate from the score until held-out outcomes establish predictive performance.
• That a heuristic can recover experienced engineering judgment. A repeated string dispatch or repeatedly patched abstraction can justify a design-review candidate. It does not prove which abstraction is correct.
• That a passing grade means the code is maintainable. It means nothing this tool measures is out of band.
• That the architecture, or any ADR, is "fully implemented" while its consumers and invariants have not migrated.
• That a bug class is closed because the reported instance stopped reproducing. See the closure rules below.
• That third-party tools cannot use the network, or that a run is kernel-air-gapped. This agent does not transmit the audited source; it does not sandbox children.

The evidence standard

Six audit rounds went in circles for one reason: claims of different kinds were held to the same loose bar. A judgment was defended as if it needed a study, and a study was asserted as if it were a judgment. Every claim in this repository is exactly one of these, and its type determines what it needs.

Tier 1 — Deterministic property

"maintainability_estimate is the mean of the printed categories." A statement about what the code does.

Requires: a test that fails if it stops being true. Not an example — a property over the real field set or input space. No study, no debate.

Tier 2 — Standard (judgment)

"Duplication is 25% of modularity." "A file over 400 lines warns."

Requires: that it be explicit, deterministic, uniform across repositories, and readable in source. It requires no validation study, and no apology. Every standard — ISO 25010, a building code, a style guide — is judgment made explicit and applied uniformly. Anyone may disagree with a weight; they can read it and argue. Legitimacy comes from being stated and applied evenly, not from being proven optimal.

Calibration against the reference corpus is what keeps these judgments anchored to real code. It does not convert them into Tier 3 facts.

Tier 3 — Empirical claim

"AI-assisted fixes touch more files." A statement about the world.

Requires: pinned inputs checked in, a matched control, a pre-registered primary outcome, correction for multiple comparisons, and re-derivation offline. Absent any of those, it is reported as an exploratory direction and never as a finding. A result that crosses significance depending on a specification choice is fragile by demonstration and must be labeled so.

The failure mode this prevents: stating a Tier 3 claim in a document whose genre implies Tier 1 or 2. That is precisely how the retracted claim reached the README headline.

When a finding is closed

Adapted from ADR 001, which earned these the hard way. All conditions, not a majority:

1. The original reproduction no longer fails.
2. The governing invariant is stated independently of that reproduction.
3. Tests exercise the production model, not a hand-built fixture that happens to carry the right keys.
4. The equivalent missing, invalid, zero, and not-applicable states are all covered.
5. Public claims do not exceed what the invariant and tests establish.
6. Any changed empirical result is re-derived from pinned inputs and corrected everywhere it is quoted.

Passing the test suite and the self-audit is necessary and is not sufficient. Two consecutive audit rounds were rejected for fixing the demonstrated instance and leaving structurally identical paths untouched.

What counts as a defect, and what does not

This tool audits itself, so it will always have findings. Without this section "no known defects" is unreachable by construction, every audit reopens the question of whether the work is finished, and the honest answer to "is it done" is permanently no.

A defect is a false claim. Behaviour that contradicts what this repository says it does: a promise the code breaks, a document that describes something the code does not do, a citation naming a function that does not exist, a check that passes vacuously. Those get filed, and they get a severity.

A finding is not a defect. The tool's own output about its own source — a declaration in the warn band, a near-duplicate pair, a file approaching the size limit, a style nit — is the product working. It is reported, it is weighed by a human, and it blocks nothing. A warn band that had to be empty would be a gate, and this project deliberately separates the two.

Restyling this repository to raise its own grade is explicitly not work. The score is second class; a tool that games its own metric has broken the only promise that matters.

The release bar

No open High and no open Medium. That is the whole rule.

A Low may stand open through a release when it is recorded, understood and not worth holding a version for. The release workflow reads the register for open entries; a Low left open must say in the entry why it is not blocking, so the decision is visible rather than assumed.

This is a stopping rule, not a quality ceiling. Nothing here says stop looking — the audits are the point, and finding something is the tool doing its job rather than evidence the job is unfinished.

What success looks like

• An agent handed the bounded work order acts on the standard instead of wandering.
• A maintainer can explain any grade by pointing at a named measurement.
• A hostile audit finds nothing that a stated invariant did not already cover.

What the evidence currently supports

studies.md is the source of record. This section holds no table and no interpretation — an earlier revision duplicated both and the copies promptly disagreed. What it does hold is the single approved public sentence, because a document whose job is to say what the product may claim cannot do that job while quoting nothing. Every figure below also appears in studies.md, and a test enforces that.

One pre-registered experiment has been run: does the bounded prompt work?. Its registered verdict is INCONCLUSIVE and stays recorded that way; the registered hypothesis was that bounded prompts produce narrower diffs, and they did not.

The defensible public sentence, and the only one:

Generic prompting made 2 of 6 repositories worse; bounded prompting made 1 of 6 worse and improved 5 of 6, under this tool's own finding count.

An earlier revision read "bounded ones did not", which concealed the bounded arm's own failure; an audit flagged it as a misleading public claim and it is corrected here rather than quietly rewritten. Two limits travel with that sentence wherever it goes: the bounded arm is told which findings to close and then scored on closing them, and "improved" means "closed the findings this tool names", which no outcome study has yet connected to real maintenance cost.

What failure looks like

• The score becomes something to game rather than something to act on.
• The tool grows into a worse SonarQube.
• A claim outruns its evidence tier — the single failure this project has actually committed.

Non-goals

Replacing mature analyzers; rewriting repositories automatically; sending code to an LLM by default; treating maintainability as purely numeric; preserving undocumented behavior for hypothetical consumers; carrying the work out. This tool measures and emits a bounded prompt; acting on that prompt is the agent's job and the operator's decision, not this package's. Nothing in this repository describes a successor, rename or execution layer — a reader who infers one is reading a plan that does not exist.