What needs doing
Add a check that runs the commands and code snippets the documentation tells people to run, in a shell that has not activated the project environment, so an instruction that no longer works fails the build instead of reaching a reader.
The repository already checks its documentation, but only for completeness and internal consistency. tests/test_docs_consistency.py verifies that every discovered CLI command has a row in the README command reference, that a row documents only its own command, that setup commands are spelled with an explicit path, and that no documented command names a retired fixture directory. None of those checks execute anything. A documented command can therefore be correct in form and broken in fact, indefinitely, with CI green.
Three such defects were found in a single review pass of the running instructions, none of them visible to any existing check:
How it surfaced
A review of the documented instructions for running the differ, carried out by executing each documented line in a fresh shell and reading the output, rather than by reading the prose. Every one of the three was found by running something, and none by reading it.
Why it matters
This class of drift is invisible by construction. Anyone with an established checkout never runs the setup path again, so the only people who meet a broken instruction are new contributors and new users, which is exactly the population least likely to file an issue about it and most likely to leave instead.
It is also the same failure shape the repository already tracks for its test suite in #288 (the test suite fails open: green runs that check nothing), applied to documentation instead of code. The suite passing tells you nothing about the docs, because the suite never touches them.
What to do
Options, roughly in increasing cost:
- Import-and-help smoke only: for each fenced Python block, run its imports; for each documented command line, run it with
--help. Cheap and safe. Catches both verified defects above. Misses semantic drift, for example a flag that still exists but changed meaning.
- Execute extracted fenced blocks in a subprocess with no activation. Catches everything above including the Quickstart shell-scoping failure. Needs an opt-out marker, see the constraint below.
- Script the Quickstart as a clean-clone run on a schedule. Highest fidelity for the on-ramp, slowest, and needs network, so it belongs in a scheduled workflow rather than the normal test run.
One constraint applies to every option: some documented commands must not be executed by a gate. tools/fetch_bill_archives.py downloads every GovInfo BILLSTATUS archive for congresses 112 to 119, hundreds of megabytes, immediately and with no prompt or flags. Any block-extraction approach needs a way to mark a block as illustrative before it runs anything.
Verification
A new gate of this kind passes on the first run by construction, which proves nothing about whether it can fail. Before trusting it, point it at one of the two verified defects above while that defect is still present, and confirm it goes red.
Refs #288, #315, #605, #606
What needs doing
Add a check that runs the commands and code snippets the documentation tells people to run, in a shell that has not activated the project environment, so an instruction that no longer works fails the build instead of reaching a reader.
The repository already checks its documentation, but only for completeness and internal consistency.
tests/test_docs_consistency.pyverifies that every discovered CLI command has a row in the README command reference, that a row documents only its own command, that setup commands are spelled with an explicit path, and that no documented command names a retired fixture directory. None of those checks execute anything. A documented command can therefore be correct in form and broken in fact, indefinitely, with CI green.Three such defects were found in a single review pass of the running instructions, none of them visible to any existing check:
docs/web-compare.mdimportscompare.pdf, a module path removed when the engine became an installable package. Filed as Documented snippet for running the PDF comparison locally fails on its first line #606.tools/fetch_bills.pyreports the un-activated case as a rawModuleNotFoundErrortraceback. Filed as Downloading a bill fails with a raw Python traceback when the environment is not activated #605.How it surfaced
A review of the documented instructions for running the differ, carried out by executing each documented line in a fresh shell and reading the output, rather than by reading the prose. Every one of the three was found by running something, and none by reading it.
Why it matters
This class of drift is invisible by construction. Anyone with an established checkout never runs the setup path again, so the only people who meet a broken instruction are new contributors and new users, which is exactly the population least likely to file an issue about it and most likely to leave instead.
It is also the same failure shape the repository already tracks for its test suite in #288 (the test suite fails open: green runs that check nothing), applied to documentation instead of code. The suite passing tells you nothing about the docs, because the suite never touches them.
What to do
Options, roughly in increasing cost:
--help. Cheap and safe. Catches both verified defects above. Misses semantic drift, for example a flag that still exists but changed meaning.One constraint applies to every option: some documented commands must not be executed by a gate.
tools/fetch_bill_archives.pydownloads every GovInfo BILLSTATUS archive for congresses 112 to 119, hundreds of megabytes, immediately and with no prompt or flags. Any block-extraction approach needs a way to mark a block as illustrative before it runs anything.Verification
A new gate of this kind passes on the first run by construction, which proves nothing about whether it can fail. Before trusting it, point it at one of the two verified defects above while that defect is still present, and confirm it goes red.
Refs #288, #315, #605, #606