A production rename can break the PDF bake-off's research probes with the whole test suite still green

[willhea]

What's wrong

A refactor of production code can break the scripts that reproduce a published study's numbers, and every check in the repository still passes. The PDF backend bake-off study under docs/research/pdf-backend-bakeoff/ ships 79 Python probes; 24 of them make 75 from deltatrack... import references, several to private names (_is_unnumbered_layout, _MIN_LINES_FOR_GUARD, _pdf_tree_payload) that are exactly the ones a refactor moves. Nothing resolves those imports: no test imports or executes the probes, and ruff does not resolve cross-module names either, so lint would not catch it even where it runs.

The repository already has the gate for this, built for a different study. tests/test_research_probes.py statically resolves every from deltatrack... import in the provision-matching probes, executing nothing. Its scope is one directory:

PROBES = PROJECT_ROOT / "docs" / "research" / "provision-matching" / "probes"

The bake-off probes are outside it.

How it surfaced

Found during a repository-hygiene review of PR #553 (the PDF backend bake-off study), after merging develop brought tests/test_research_probes.py onto the branch. That file was added in response to the same exposure in the other study, and its docstring records what it cost there:

a pure rename in #492 broke the deltatrack imports of thirteen of them, #308 moved the corpus out from under all fifteen, and nobody noticed for months, because nothing ran them. paper.md's "every number can be reproduced from the scripts named in Appendix A" was false at review time in the strongest sense -- none of them executed at all.

(#492 consolidated the text-similarity cutoffs that were written in five places; #308 changed how test fixtures are committed and moved the corpus.)

The bake-off study makes the same promise about itself. docs/research/pdf-backend-bakeoff/validation/README.md states "Every number in all three documents comes from those files; none is transcribed by hand."

The bake-off probes are not currently broken. I ran a static mirror of the existing gate over the whole study directory, on spike/pdf-backend-bakeoff at 1901f6c:

79 python files, 75 deltatrack imports checked
all deltatrack imports resolve

So this is an unguarded exposure, not a live defect. That is also the cheapest moment to close it: the gate can be added green rather than as part of a repair.

Why it matters

The study is the evidence base for an architectural decision about where the seam between the PDF engine and DeltaTrack's own code should sit, and it is still open work. Two named follow-ups depend on re-running these probes: a heading-level correctness oracle and a fresh structure-rich holdout, both listed as blocking an ADR. If a rename silently orphans the probes in the meantime, the study becomes unreproducible in the same way and on the same timescale as the provision-matching one did, and the failure is invisible because a green suite is the expected state.

Excluding a directory from lint is a decision about style. Nothing in that decision implies the code should stop being checked for whether it resolves.

What to do

Options, roughly in increasing order of coverage:

1. Add the bake-off directory to the existing gate. Turn PROBES into a list of research roots and parametrize the existing import check over it. Smallest change; each new study still needs a line.
2. Discover research probe roots. Parametrize over docs/research/*/ and check every *.py beneath. No per-study edit, and a future study is covered on the day it lands. Needs a rule for what counts as a probe directory, and the bake-off keeps its probes in two places (probes/ and validation/, including validation/phase2/ and validation/phase3/), so a probes-only glob would miss most of this study.
3. Also check that named result files have a producer. The same review found g07_extraction_cost.json committed with no g07_*.py in the tree while two documents promised probes g01-g07 (fixed in PR spike: PDF backend bake-off, the hybrid seam, and an adversarial validation of it #553). That is a different check from import resolution and could be a separate issue if it is worth having.

Option 2 looks better suited to how this repository actually accumulates studies, but option 1 is a defensible smaller step. Worth deciding rather than assuming.

Verification

"The suite passes" cannot demonstrate this fix, because it passes now. The gate must be shown to fail on a known-bad case: rename or remove a symbol that a bake-off probe imports (for example _MIN_LINES_FOR_GUARD in src/deltatrack/compare/pdf.py), confirm the new check goes red, then restore and confirm it goes green.

Unverified

docs/research/ currently holds exactly two studies, provision-matching (covered by the gate) and pdf-backend-bakeoff (not covered), so the gap is fully enumerated today. What I did not check:

• whether the JavaScript probes (*.mjs, 14 files in the bake-off) have an equivalent unresolved-import exposure against their npm dependencies;
• whether the bake-off probes' imports of each other (they import siblings such as contract, reconstruct and score_hybrid by path manipulation, not only deltatrack) would be caught by the same static approach.

Refs #553