Repository navigation
Conversation
check-tla.sh traceability now implements ADR-4326 decision 6. An area is
enrolled when its directory holds antithesis-mutants.tsv. Every table in an
enrolled area's traceability.md must then carry a sixth "Antithesis
property" column whose cell is sut: <path.rs> -- <message> (the file holds
the literal), driver: <workload> -- <message> (the literal occurs under
antithesis/<workload>/), or none: <reason> with a non-empty reason. A
five-column table in an enrolled area fails, and so does a six-column table
in an area with no .tsv. The .tsv needs one line per negative/*.cfg, naming
either the mutant <area>/<stem>, found in Rust as inv_mutant!("<area>/<stem>"),
and a message a sut: or driver: cell carries, or none: <reason>.
A table header is now recognised by its exact first cell, "TLA+ action or
property", instead of a pattern that also skipped any data row mentioning an
action and a property; such rows are checked now. Every data row must have
the header's cell count. No area is enrolled by this change, and the real
tree still resolves all 202 rows.
formal/tla/README.md and TRACEABILITY.md describe the column and the .tsv,
and TRACEABILITY.md's counts now match the checker (agents 54, total 202).
ADR-1113 gains an amendment for D8 with an inline pointer, and its index row
in docs/adrs/README.md now says Accepted as the ADR does.
Refs: #4330
Refs: #4326
Changelog: none
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Review fixes for the Antithesis property column checker. A row of only pipes, spaces, dashes and colons was skipped as a separator anywhere in a table, so an all-empty row or "| - | - | ... |" passed. Only the row directly under the header is a separator now; any later such row is a data row and fails, in enrolled and unenrolled areas alike. A sut: or driver: message now counts only as the whole double-quoted string literal, and driver: searches only .rs files under antithesis/<workload>/, so README prose and a substring of a longer literal no longer satisfy it. Mutants are searched under antithesis/*/driver/ as well as crates/ and services/, where ADR-4326 decision 3 puts the common area's broken store. A literal on a comment line or after a file's first #[cfg(test)] line does not count, nor, for driver: and mutants, one in a file under tests/. none: needs a space before its reason, in the cell and in the .tsv. New cases cover each of these and the cell-count, duplicate stem, unknown stem, " -- ", unknown prefix, sixth header name, empty .tsv none: reason and <area>/<stem> name rules; each was shown failing with its rule broken. ADR-4326 gains an amendment recording that #4330 corrected the ADR-1113 index row and the TRACEABILITY.md count, and narrowing decision 6. Refs: #4330 Refs: #4326 Changelog: none Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
The driver: check matches a message only as a whole string literal in the workload's .rs files, but no case failed when that match dropped its quotes. Two cases now cover it: a message that occurs in the driver only as an identifier, and one that is only part of a longer literal. Both fail against a match without the quotes. The README described a later separator-like row as failing for want of a Rust source ref; it fails on whatever cell is not a valid entry. Refs: #4330 Refs: #4326 Changelog: none Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
|
@claude-fleet review |
|
Review task queued. Task: 22bce31c-fd3e-4a3f-95c4-9845051dd244 |
There was a problem hiding this comment.
Reviewed the whole diff, ran the suite and the doc guards, and probed the new rules against the real tree.
Request changes: one should_fix, three nits.
Findings:
- should_fix, scripts/check-tla.sh:760 —
rust_literal_livestops at the first#[cfg(test)]line, hiding production literals in most repo files - nit, scripts/check-tla.sh:761 — a block-comment body line without a leading
*counts as live - nit, scripts/check-tla.sh:817 —
sut:accepts a path undertests/, unlikedriver:and mutants - nit, scripts/check-tla.sh:958 — a separator row's column count is never checked
Verified:
- 202 rows total, 54 in agents, matching the new TRACEABILITY.md figures —
check-tla.sh traceabilityreports 54+25+21+9+29+45+19 - no area is enrolled and every area still passes five-column — same run, exit 0
- the header is matched on its exact first cell and only the row under it is a separator — scripts/check-tla.sh:942-958, proved by cases (ao2), (au1)-(au4)
- the
sut:path stays a whole token for the existingcrates/...sweep — scripts/check-tla.sh:988-991, case (ai2) fails onmissing source 'crates/gone.rs' - the mutant literal is accepted from
antithesis/*/driver/and rejected outside it — scripts/check-tla.sh:870-873, case (ax1)/(ax2) - ADR-1113's status line reads
Accepted (2026-09-04), so the index row change matches it — docs/adrs/1113-tla-verification-suite.md:3 - both amendment markers name real sections (
Context|6. The traceability columnat docs/adrs/4326-...md:145,D8. Traceability names symbols, not lines)
Checks run:
bash scripts/check-tla.test.sh→148 passed, 0 failedbash scripts/check-tla.sh traceability→ exit 0, all seven areas PASSbash scripts/guards/check-amendment-integrity.sh→ clean (565 amendments)bash scripts/guards/check-adr-band-sources.sh→ cleanpython3 scripts/check_docs.py→docs gate: clean.- Probe: sourced the lib against a scratch repo to reproduce the four findings below
Smaller things:
strip_code_spanunwraps exactly one backtick pair, so a message written as a double-backtick span (needed when the message itself contains a backtick) will not match. Not reachable today.- A
sut:ordriver:message that happens to contain acrates/...token will be picked up by the trailingcrates/sweep and resolved as a source ref. Pre-existing sweep behaviour, but the new column makes free text in a row more likely.
Not checked:
- Nothing in the diff was left unread. I did not run TLC (the traceability lane needs no Java) or any cargo gate; the change is scripts and docs only, as the description says.
Task 22bce31c-fd3e-4a3f-95c4-9845051dd244 on ci-16gb-fsn1-1, model claude-opus-5, effort medium.
| # never as awk program text or -v, so backslashes stay literal. | ||
| rust_literal_live() { | ||
| RUST_LITERAL="$1" awk ' | ||
| /^[[:space:]]*#\[cfg\(test\)\]/ { exit } |
There was a problem hiding this comment.
should_fix: rust_literal_live treats the file's first #[cfg(test)] line as the start of the test region and stops scanning there, but in this repo that attribute very often sits on a single test-only item in the middle of a file, with hundreds of lines of production code after it.
Failure: an area is enrolled and a row's cell is sut: crates/ravel-otlp/src/label_projection.rs -- <message> for an inv_always! added anywhere after line 105. crates/ravel-otlp/src/label_projection.rs:105 is #[cfg(test)] pub(crate) fn for_test, and real production items follow at lines 126, 161, 182, 223 and 288 of 1514. The checker reports message '<m>' not found as the string literal "<m>" in '<path>' for a literal that is plainly live. The same shape holds for crates/ravel-cache/src/lib.rs (first #[cfg(test)] at 107 of 980), crates/ravel-fleet/src/query_workers.rs (637 of 1283) and many others; a scan of the repo found dozens.
Evidence: scripts/check-tla.sh:760 /^[[:space:]]*#\[cfg\(test\)\]/ { exit }. Reproduced by sourcing the lib against a scratch tree whose crates/a.rs is fn helper(), #[cfg(test)] fn only_for_test(), then fn publish() { inv_always!(ok, "commit never precedes data"); }: check_traceability returns 1 with exactly that diagnostic.
The failure direction is safe (it rejects, it does not accept), and no area is enrolled yet, so nothing is broken today. But this is the rule the first enrollment runs into. Matching only a #[cfg(test)] that introduces a module (#[cfg(test)] immediately followed by a mod line), or skipping the gated item rather than the rest of the file, would hold the intent without the false rejection.
There was a problem hiding this comment.
Fixed in 559a843. The scan now skips only the item after #[cfg(test)] (and any further attributes): one line when the item ends there, otherwise every line until its braces balance. New cases ay2-after-test-mod, ay2-after-cfg-use and ay2-after-deref put a live literal after a test module, after a #[cfg(test)] use, and on a line starting with a dereference. All three fail against the old exit-at-first-attribute scan.
| rust_literal_live() { | ||
| RUST_LITERAL="$1" awk ' | ||
| /^[[:space:]]*#\[cfg\(test\)\]/ { exit } | ||
| /^[[:space:]]*(\/\/|\/\*|\*)/ { next } |
There was a problem hiding this comment.
nit: The comment skip matches only //, /* or * after leading whitespace, so a line inside a /* ... */ block that is not star-prefixed counts as live code.
Failure: a file holding
/*
if inv_mutant!("enr/block-no-star") {}
let dead = "a dead message";
*/
satisfies both the mutant rule and a sut: cell naming a dead message. Probed against a scratch tree: check_traceability returns 0 and reports PASS (2 rows resolve) with the mutant and the message living only inside that comment. Case (ay) covers the //, single-line /* ... */ and * forms, so this one slips past.
Evidence: scripts/check-tla.sh:761. formal/tla/README.md:238-241 and the ADR-4326 amendment both state the prefix rule exactly, so the behaviour is documented rather than a surprise. It is worth noting because this direction is the unsafe one: it accepts a mutant that does not exist in live code.
There was a problem hiding this comment.
Fixed in 559a843. The scan tracks /* ... */ blocks instead of matching a leading *, so an unprefixed body line is skipped (case ay-block-bare) and a line starting with a dereference is no longer dropped.
| *.rs) : ;; | ||
| *) note "$where: sut: path '$target' is not a Rust (.rs) path"; return 1 ;; | ||
| esac | ||
| [ -f "$REPO_ROOT/$target" ] || { note "$where: sut: no such file '$target'"; return 1; } |
There was a problem hiding this comment.
nit: A sut: cell is checked with rust_literal_live directly, which applies no tests/ exclusion, while driver: cells and mutants go through rust_literal_live_under and skip tests/ (scripts/check-tla.sh:770-772).
Failure: sut: crates/m/tests/it.rs -- tests only message passes. Probed against a scratch tree with that file and no other occurrence: PASS (1 rows resolve). A property whose only carrier is an integration test is not carried in the SUT, which is what decision 6 is asserting.
Evidence: scripts/check-tla.sh:817-819 versus scripts/check-tla.sh:770-772. The asymmetry is stated in formal/tla/README.md:241-243, so this is a question about the rule rather than a mismatch with it; rejecting a sut: path under tests/ alongside the existing .rs and repo-relative checks would close it.
There was a problem hiding this comment.
Fixed in 559a843: a sut: path under tests/ now fails (case ay3).
| # later row of pipes and dashes is a data row and is checked as one. | ||
| if [ "$after_header" -eq 1 ]; then | ||
| after_header=0 | ||
| [[ "$line" =~ $TRACE_DELIMITER_ROW ]] && continue |
There was a problem hiding this comment.
nit: The row directly under the header is skipped whole when it matches TRACE_DELIMITER_ROW, with no check that it has the table's column count.
Failure: a five-column table whose separator is |---|---| passes. Probed: check_traceability on a header of five cells, a two-cell separator and one valid data row returns 0, PASS (1 rows resolve). The table renders wrong in markdown and the checker says nothing, while the ADR amendment states that a table's rows must have the header's column count.
Evidence: scripts/check-tla.sh:955-958. One trace_cells count comparison on the delimiter row before continue would cover it.
There was a problem hiding this comment.
Fixed in 559a843: the separator row under a header must have the table's cell count (case ay4).
rust_literal_live stopped scanning at a file's first #[cfg(test)] line. In this repository that attribute often sits on a single use or helper near the top of a file, so every production literal below it was hidden and a valid sut:, driver: or mutant entry failed. The scan now skips only the item that follows the attribute (and any further attributes): one line when the item ends there, else every line until its braces balance. The review of the first version also found three smaller gaps, fixed here. A line inside a /* ... */ block without a leading * counted as live code, and a line starting with a dereference (*slot = ...) was dropped as a comment, so the scan now tracks block comments instead of matching a * prefix. A sut: path under tests/ was accepted, unlike driver: and mutant literals. A separator row's column count was never checked. Each new case fails against the rule it pins: a scan that exits at the first #[cfg(test)] fails the three ay2 cases, a scan that does not skip block bodies fails ay-block-body and ay-block-bare, and removing the tests/ or separator checks fails ay3 and ay4. Refs: #4330 Refs: #4326 Changelog: none Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
|
@claude-fleet review |
|
Review task queued. Task: 28998fa2-3619-49a0-9266-ebfb58620752 |
There was a problem hiding this comment.
Previous round at fda2868: 4 findings.
| Previous finding | Classification | Cited line |
|---|---|---|
scripts/check-tla.sh:760 (should_fix): rust_literal_live treats the file's first #[cfg(test)] line as the start of the test region and stops scanning there, but in this repo that attribute very often sits on a single test-only item in the middle of a file, with hundreds of lines of production code after it. Failure: an area is enrolled and a row's cell is sut: crates/ravel-otlp/src/label_projection.rs -- <message> for an inv_always! added anywhere after line 105. crates/ravel-otlp/src/label_projection.rs:105 is #[cfg(test)] pub(crate) fn for_test, and real production items follow at lines 126, 161, 182, 223 and 288 of 1514. The checker reports message '<m>' not found as the string literal "<m>" in '<path>' for a literal that is plainly live. The same shape holds for crates/ravel-cache/src/lib.rs (first #[cfg(test)] at 107 of 980), crates/ravel-fleet/src/query_workers.rs (637 of 1283) and many others; a scan of the repo found dozens. Evidence: scripts/check-tla.sh:760 /^[[:space:]]*#\[cfg\(test\)\]/ { exit }. Reproduced by sourcing the lib against a scratch tree whose crates/a.rs is fn helper(), #[cfg(test)] fn only_for_test(), then fn publish() { inv_always!(ok, "commit never precedes data"); }: check_traceability returns 1 with exactly that diagnostic. The failure direction is safe (it rejects, it does not accept), and no area is enrolled yet, so nothing is broken today. But this is the rule the first enrollment runs into. Matching only a #[cfg(test)] that introduces a module (#[cfg(test)] immediately followed by a mod line), or skipping the gated item rather than the rest of the file, would hold the intent without the false rejection. |
fixed | scripts/check-tla.sh:770 |
scripts/check-tla.sh:761 (nit): The comment skip matches only //, /* or * after leading whitespace, so a line inside a /* ... */ block that is not star-prefixed counts as live code. Failure: a file holding /* if inv_mutant!("enr/block-no-star") {} let dead = "a dead message"; */ satisfies both the mutant rule and a sut: cell naming a dead message. Probed against a scratch tree: check_traceability returns 0 and reports PASS (2 rows resolve) with the mutant and the message living only inside that comment. Case (ay) covers the //, single-line /* ... */ and * forms, so this one slips past. Evidence: scripts/check-tla.sh:761. formal/tla/README.md:238-241 and the ADR-4326 amendment both state the prefix rule exactly, so the behaviour is documented rather than a surprise. It is worth noting because this direction is the unsafe one: it accepts a mutant that does not exist in live code. |
fixed | scripts/check-tla.sh:777 |
scripts/check-tla.sh:817 (nit): A sut: cell is checked with rust_literal_live directly, which applies no tests/ exclusion, while driver: cells and mutants go through rust_literal_live_under and skip tests/ (scripts/check-tla.sh:770-772). Failure: sut: crates/m/tests/it.rs -- tests only message passes. Probed against a scratch tree with that file and no other occurrence: PASS (1 rows resolve). A property whose only carrier is an integration test is not carried in the SUT, which is what decision 6 is asserting. Evidence: scripts/check-tla.sh:817-819 versus scripts/check-tla.sh:770-772. The asymmetry is stated in formal/tla/README.md:241-243, so this is a question about the rule rather than a mismatch with it; rejecting a sut: path under tests/ alongside the existing .rs and repo-relative checks would close it. |
fixed | scripts/check-tla.sh:830 |
scripts/check-tla.sh:958 (nit): The row directly under the header is skipped whole when it matches TRACE_DELIMITER_ROW, with no check that it has the table's column count. Failure: a five-column table whose separator is |---|---| passes. Probed: check_traceability on a header of five cells, a two-cell separator and one valid data row returns 0, PASS (1 rows resolve). The table renders wrong in markdown and the checker says nothing, while the ADR amendment states that a table's rows must have the header's column count. Evidence: scripts/check-tla.sh:955-958. One trace_cells count comparison on the delimiter row before continue would cover it. |
fixed | scripts/check-tla.sh:977 |
Verified the fix commit against all four earlier findings, then reviewed the full diff.
Request changes: one should_fix on the new #[cfg(test)] skip, one on a now-stale ADR bullet, one nit.
All four previous findings are fixed. The revert check proves it: with scripts/check-tla.sh reverted to fda2868 and the new test file in place, ay-block-bare, ay2-after-test-mod, ay2-after-cfg-use, ay2-after-deref, ay3 and ay4 fail (149 passed, 6 failed); on the head tree the suite is 155/155.
The new item-scoped skip is a real improvement over "stop at the first #[cfg(test)] line", and I confirmed it on the exact file the first finding cited (crates/ravel-otlp/src/label_projection.rs:105-109): production code after the gated for_test is scanned again. What it does not handle is a gated item whose signature spans lines, which is the common shape in this repo, and there the direction flips to accepting test-only text as SUT evidence.
Findings:
- should_fix, scripts/check-tla.sh:773 — a gated item with a multi-line signature ends its skip after one line, so its body counts as live
- should_fix, docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260 — the amendment still states the superseded comment/
cfg(test)/tests/rules - nit, scripts/check-tla.sh:777 — a
/* ... */block opened mid-line is not tracked
Verified:
- ADR-1113's status line reads Accepted, so the index row change matches it — docs/adrs/1113-tla-verification-suite.md header, docs/adrs/README.md:149
TRACEABILITY.md's new totals are what the checker reports: 54 agents rows, 202 overall (9+21+25+29+45+19+54) — formal/tla/TRACEABILITY.md:22-24- The separator cell-count check runs on the delimiter row before
continue, and only for the row directly under the header — scripts/check-tla.sh:972-981 - A
sut:path undertests/is now rejected before the file-exists check, matchingrust_literal_live_under's exclusion — scripts/check-tla.sh:830 vs 789 - Finding 2's case is closed: a non-star line inside a block comment that opens at line start is skipped — scripts/check-tla.sh:766, 777
Checks run:
bash scripts/check-tla.test.sh→ 155 passed, 0 failed- Same suite with
scripts/check-tla.shreverted to fda2868 → 149 passed, 6 failed (the six cases above) bash scripts/check-tla.sh traceability→ all areas pass, 202 rowsscripts/guards/check-amendment-integrity.sh→ exit 0 (565 amendments);scripts/guards/check-adr-band-sources.sh→ exit 0- Sourced the script's lib and ran
rust_literal_liveagainst real repo files and scratch files for each edge case below
Smaller things:
- The brace counting is textual, and the code comment and
formal/tla/README.md:242-244say so, but the early-end direction is the unsafe one: in a scratch file,#[cfg(test)] mod tests {whose first body line isfn t() { let s = "}"; }ends the skip there, and the rest of the module counts as live.services/ravel-server/src/ipcrypt.rs:521-547already solves this in Rust by matching the}at the attribute's own indent, which is immune to braces in strings and also fixes the multi-line-signature case below. Worth copying that rule rather than counting braces. - The PR body says the suite passes "148 of 148"; at this head it is 155 of 155.
Not checked: nothing in the diff was left unread; the Antithesis workload layout under antithesis/ does not exist yet in this tree, so the driver: path was exercised only through the test suite's scratch trees.
Not checked:
- The antithesis/ workload tree does not exist in this checkout, so the driver: branch of check_antithesis_cell was exercised only through the test suite's scratch trees, not against a real workload.
Task 28998fa2-3619-49a0-9266-ebfb58620752 on ci-16gb-fsn1-1, model claude-opus-5, effort medium.
| pending { | ||
| pending = 0 | ||
| depth = braces($0) | ||
| if (depth > 0 || ($0 !~ /[;}][[:space:]]*$/)) skipping = 1 |
There was a problem hiding this comment.
should_fix: A #[cfg(test)] item whose signature spans several lines sets skipping with depth = 0, so the very next line drives depth <= 0 and ends the skip one line into the item: the item's body is then scanned as live code.
Failure: sut: crates/ravel-ingest/src/read_backoff.rs -- provisioning GET passes, although "provisioning GET" occurs only at line 192, inside the #[cfg(test)] pub(crate) fn failing_then_passing_get_store( whose parameter list runs to line 186 (grep -rn '"provisioning GET"' --include='*.rs' crates services returns that one line). Sourced the script's lib and called rust_literal_live '"provisioning GET"' crates/ravel-ingest/src/read_backoff.rs against the real file: exit 0, live. Same for the literal at crates/ravel-catalog/src/fold.rs:2456, whose only home is the #[cfg(test)] pub(crate) fn build_part_key_index_leaves( starting at line 2380. This is the unsafe direction: a property whose only carrier is a test helper reads as carried in the SUT.
Evidence: scripts/check-tla.sh:772-773. Line 772 counts braces on the head line only, and 773 arms skipping at depth 0 whenever the head neither opens a brace nor ends in ;/} — which is every wrapped signature and every gated struct field. Running the same awk state machine over all 1535 .rs files under crates/ and services/ gives 1028 skips, 72 of which end on a line that is not the item's closing brace; 46 of those end on a parameter line such as &self,, mut self, or store: &dyn ObjectStoreBackend,. One of them (crates/ravel-ingest/src/write_gate.rs:79) is itself a #[cfg(test)] attribute swallowed by the skip, so the field it gates is scanned as live too.
services/ravel-server/src/ipcrypt.rs:521-547 already implements this rule in Rust by skipping to the } at the attribute's own indentation, which handles both this case and the brace-in-a-string case the comment documents.
There was a problem hiding this comment.
Fixed in cc639b7: the skip now runs through the first line ending in ; when no brace has opened, else until the braces balance. Case ay-cfg-test-multiline (a fn with a three-line signature) fails against the previous scan.
| - A mutant from decision 3 occurs as `inv_mutant!("<area>/<stem>")` in a | ||
| `.rs` file under `crates/`, `services/` or `antithesis/*/driver/`, the last | ||
| for the common area's driver-side broken store. | ||
| - For a message and a mutant alike, a literal on a comment line (`//`, `/*` |
There was a problem hiding this comment.
should_fix: This PR's last commit changed two rules and updated formal/tla/README.md and ADR-1113's amendment, but not this bullet, so the ADR-4326 amendment now states behaviour the checker does not have.
Failure: a reader enrolling an area follows "a literal ... after the file's first #[cfg(test)] line does not count" and concludes a literal below the first gated item in a file is invisible; scripts/check-tla.sh:766-777 skips only the gated item, and test case ay2 in scripts/check-tla.test.sh pins the opposite. The same bullet says only driver: and mutants exclude tests/, while scripts/check-tla.sh:830 now rejects a sut: path under tests/ outright.
Evidence: docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260-263 versus formal/tla/README.md:240-246 and docs/adrs/1113-tla-verification-suite.md:617-620, which both carry the new wording.
There was a problem hiding this comment.
Fixed in cc639b7: the amendment bullet now states the current rules (comments incl. mid-line /*, #[cfg(test)] items, tests/ files, and the sut: refusal under tests/), and formal/tla/README.md matches.
| next | ||
| } | ||
| /^[[:space:]]*\/\// { next } | ||
| /^[[:space:]]*\/\*/ { if (!index(substr($0, index($0, "/*") + 2), "*/")) in_block = 1; next } |
There was a problem hiding this comment.
nit: in_block is armed only by a line whose first non-space characters are /*, so a block comment opened after code on the same line is never tracked and its body counts as live.
Failure: a file holding
fn f() {
let s = "}"; /* opens here
let dead = "MSG";
*/
}
makes a sut: cell naming MSG pass. Probed by sourcing the lib against a scratch file of exactly that shape: rust_literal_live '"MSG"' <file> exits 0. The star-prefixed and line-start forms that case (ay) covers are now handled; this is the remaining opening form, and it accepts dead text.
Evidence: scripts/check-tla.sh:777. formal/tla/README.md:241-242 states the rule as written, so this is a question about the rule rather than a mismatch with it.
There was a problem hiding this comment.
Fixed in cc639b7: text after a /* on any line is dropped and the scan skips to the matching */. Cases ay-midline-block and ay-midline-block-body fail against the previous scan.
A #[cfg(test)] item whose signature spans several lines ended its skip on the first line, because no brace had opened yet, so the rest of the signature and the whole body counted as live code. The skip now runs through the first line ending in ; when no brace has opened, and otherwise until the braces balance. A /* ... */ comment opened after code on a line was not tracked, so its text and the lines that followed it up to */ counted as live. The scan now drops the text after a /* on any line and skips to the matching */. The ADR-4326 amendment and formal/tla/README.md still described the earlier rules; both now state the current ones. New cases ay-cfg-test-multiline, ay-midline-block and ay-midline-block-body fail against the previous scan. Refs: #4330 Refs: #4326 Changelog: none Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
|
@claude-fleet review |
|
Review task queued. Task: 1e103d99-3223-4810-b977-5f5fa097a1d0 |
There was a problem hiding this comment.
Previous round at 559a843: 7 findings.
| Previous finding | Classification | Cited line |
|---|---|---|
scripts/check-tla.sh:770 (should_fix): rust_literal_live treats the file's first #[cfg(test)] line as the start of the test region and stops scanning there, but in this repo that attribute very often sits on a single test-only item in the middle of a file, with hundreds of lines of production code after it. Failure: an area is enrolled and a row's cell is sut: crates/ravel-otlp/src/label_projection.rs -- <message> for an inv_always! added anywhere after line 105. crates/ravel-otlp/src/label_projection.rs:105 is #[cfg(test)] pub(crate) fn for_test, and real production items follow at lines 126, 161, 182, 223 and 288 of 1514. The checker reports message '<m>' not found as the string literal "<m>" in '<path>' for a literal that is plainly live. The same shape holds for crates/ravel-cache/src/lib.rs (first #[cfg(test)] at 107 of 980), crates/ravel-fleet/src/query_workers.rs (637 of 1283) and many others; a scan of the repo found dozens. Evidence: scripts/check-tla.sh:760 /^[[:space:]]*#\[cfg\(test\)\]/ { exit }. Reproduced by sourcing the lib against a scratch tree whose crates/a.rs is fn helper(), #[cfg(test)] fn only_for_test(), then fn publish() { inv_always!(ok, "commit never precedes data"); }: check_traceability returns 1 with exactly that diagnostic. The failure direction is safe (it rejects, it does not accept), and no area is enrolled yet, so nothing is broken today. But this is the rule the first enrollment runs into. Matching only a #[cfg(test)] that introduces a module (#[cfg(test)] immediately followed by a mod line), or skipping the gated item rather than the rest of the file, would hold the intent without the false rejection. |
fixed | scripts/check-tla.sh:775 |
scripts/check-tla.sh:777 (nit): The comment skip matches only //, /* or * after leading whitespace, so a line inside a /* ... */ block that is not star-prefixed counts as live code. Failure: a file holding /* if inv_mutant!("enr/block-no-star") {} let dead = "a dead message"; */ satisfies both the mutant rule and a sut: cell naming a dead message. Probed against a scratch tree: check_traceability returns 0 and reports PASS (2 rows resolve) with the mutant and the message living only inside that comment. Case (ay) covers the //, single-line /* ... */ and * forms, so this one slips past. Evidence: scripts/check-tla.sh:761. formal/tla/README.md:238-241 and the ADR-4326 amendment both state the prefix rule exactly, so the behaviour is documented rather than a surprise. It is worth noting because this direction is the unsafe one: it accepts a mutant that does not exist in live code. |
fixed | scripts/check-tla.sh:781 |
scripts/check-tla.sh:830 (nit): A sut: cell is checked with rust_literal_live directly, which applies no tests/ exclusion, while driver: cells and mutants go through rust_literal_live_under and skip tests/ (scripts/check-tla.sh:770-772). Failure: sut: crates/m/tests/it.rs -- tests only message passes. Probed against a scratch tree with that file and no other occurrence: PASS (1 rows resolve). A property whose only carrier is an integration test is not carried in the SUT, which is what decision 6 is asserting. Evidence: scripts/check-tla.sh:817-819 versus scripts/check-tla.sh:770-772. The asymmetry is stated in formal/tla/README.md:241-243, so this is a question about the rule rather than a mismatch with it; rejecting a sut: path under tests/ alongside the existing .rs and repo-relative checks would close it. |
fixed | scripts/check-tla.sh:837 |
scripts/check-tla.sh:977 (nit): The row directly under the header is skipped whole when it matches TRACE_DELIMITER_ROW, with no check that it has the table's column count. Failure: a five-column table whose separator is |---|---| passes. Probed: check_traceability on a header of five cells, a two-cell separator and one valid data row returns 0, PASS (1 rows resolve). The table renders wrong in markdown and the checker says nothing, while the ADR amendment states that a table's rows must have the header's column count. Evidence: scripts/check-tla.sh:955-958. One trace_cells count comparison on the delimiter row before continue would cover it. |
fixed | scripts/check-tla.sh:984 |
scripts/check-tla.sh:773 (should_fix): A #[cfg(test)] item whose signature spans several lines sets skipping with depth = 0, so the very next line drives depth <= 0 and ends the skip one line into the item: the item's body is then scanned as live code. Failure: sut: crates/ravel-ingest/src/read_backoff.rs -- provisioning GET passes, although "provisioning GET" occurs only at line 192, inside the #[cfg(test)] pub(crate) fn failing_then_passing_get_store( whose parameter list runs to line 186 (grep -rn '"provisioning GET"' --include='*.rs' crates services returns that one line). Sourced the script's lib and called rust_literal_live '"provisioning GET"' crates/ravel-ingest/src/read_backoff.rs against the real file: exit 0, live. Same for the literal at crates/ravel-catalog/src/fold.rs:2456, whose only home is the #[cfg(test)] pub(crate) fn build_part_key_index_leaves( starting at line 2380. This is the unsafe direction: a property whose only carrier is a test helper reads as carried in the SUT. Evidence: scripts/check-tla.sh:772-773. Line 772 counts braces on the head line only, and 773 arms skipping at depth 0 whenever the head neither opens a brace nor ends in ;/} — which is every wrapped signature and every gated struct field. Running the same awk state machine over all 1535 .rs files under crates/ and services/ gives 1028 skips, 72 of which end on a line that is not the item's closing brace; 46 of those end on a parameter line such as &self,, mut self, or store: &dyn ObjectStoreBackend,. One of them (crates/ravel-ingest/src/write_gate.rs:79) is itself a #[cfg(test)] attribute swallowed by the skip, so the field it gates is scanned as live too. services/ravel-server/src/ipcrypt.rs:521-547 already implements this rule in Rust by skipping to the } at the attribute's own indentation, which handles both this case and the brace-in-a-string case the comment documents. |
fixed | scripts/check-tla.sh:769 |
docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260 (should_fix): This PR's last commit changed two rules and updated formal/tla/README.md and ADR-1113's amendment, but not this bullet, so the ADR-4326 amendment now states behaviour the checker does not have. Failure: a reader enrolling an area follows "a literal ... after the file's first #[cfg(test)] line does not count" and concludes a literal below the first gated item in a file is invisible; scripts/check-tla.sh:766-777 skips only the gated item, and test case ay2 in scripts/check-tla.test.sh pins the opposite. The same bullet says only driver: and mutants exclude tests/, while scripts/check-tla.sh:830 now rejects a sut: path under tests/ outright. Evidence: docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260-263 versus formal/tla/README.md:240-246 and docs/adrs/1113-tla-verification-suite.md:617-620, which both carry the new wording. |
fixed | docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260 |
scripts/check-tla.sh:777 (nit): in_block is armed only by a line whose first non-space characters are /*, so a block comment opened after code on the same line is never tracked and its body counts as live. Failure: a file holding fn f() { let s = "}"; /* opens here let dead = "MSG"; */ } makes a sut: cell naming MSG pass. Probed by sourcing the lib against a scratch file of exactly that shape: rust_literal_live '"MSG"' <file> exits 0. The star-prefixed and line-start forms that case (ay) covers are now handled; this is the remaining opening form, and it accepts dead text. Evidence: scripts/check-tla.sh:777. formal/tla/README.md:241-242 states the rule as written, so this is a question about the rule rather than a mismatch with it. |
fixed | scripts/check-tla.sh:779 |
No longer carried to the next round, settled in rounds 2 and 3:
- scripts/check-tla.sh:770 (should_fix):
rust_literal_livetreats the file's first#[cfg(test)]line as the start of the test region and stops scanning there, but in this repo that attribute very often sits on a single test-only item in the middle of a file, with hundreds of lines of production code after it. Failure: an area is enrolled and a row's cell issut: crates/ravel-otlp/src/label_projection.rs -- <message>for aninv_always!added anywhere after line 105.crates/ravel-otlp/src/label_projection.rs:105is#[cfg(test)] pub(crate) fn for_test, and real production items follow at lines 126, 161, 182, 223 and 288 of 1514. The checker reportsmessage '<m>' not found as the string literal "<m>" in '<path>'for a literal that is plainly live. The same shape holds forcrates/ravel-cache/src/lib.rs(first#[cfg(test)]at 107 of 980),crates/ravel-fleet/src/query_workers.rs(637 of 1283) and many others; a scan of the repo found dozens. Evidence: scripts/check-tla.sh:760/^[[:space:]]*#\[cfg\(test\)\]/ { exit }. Reproduced by sourcing the lib against a scratch tree whosecrates/a.rsisfn helper(),#[cfg(test)] fn only_for_test(), thenfn publish() { inv_always!(ok, "commit never precedes data"); }:check_traceabilityreturns 1 with exactly that diagnostic. The failure direction is safe (it rejects, it does not accept), and no area is enrolled yet, so nothing is broken today. But this is the rule the first enrollment runs into. Matching only a#[cfg(test)]that introduces a module (#[cfg(test)]immediately followed by amodline), or skipping the gated item rather than the rest of the file, would hold the intent without the false rejection. - scripts/check-tla.sh:777 (nit): The comment skip matches only
//,/*or*after leading whitespace, so a line inside a/* ... */block that is not star-prefixed counts as live code. Failure: a file holding /* if inv_mutant!("enr/block-no-star") {} let dead = "a dead message"; */ satisfies both the mutant rule and asut:cell naminga dead message. Probed against a scratch tree:check_traceabilityreturns 0 and reportsPASS (2 rows resolve)with the mutant and the message living only inside that comment. Case (ay) covers the//, single-line/* ... */and*forms, so this one slips past. Evidence: scripts/check-tla.sh:761. formal/tla/README.md:238-241 and the ADR-4326 amendment both state the prefix rule exactly, so the behaviour is documented rather than a surprise. It is worth noting because this direction is the unsafe one: it accepts a mutant that does not exist in live code. - scripts/check-tla.sh:830 (nit): A
sut:cell is checked withrust_literal_livedirectly, which applies notests/exclusion, whiledriver:cells and mutants go throughrust_literal_live_underand skiptests/(scripts/check-tla.sh:770-772). Failure:sut: crates/m/tests/it.rs -- tests only messagepasses. Probed against a scratch tree with that file and no other occurrence:PASS (1 rows resolve). A property whose only carrier is an integration test is not carried in the SUT, which is what decision 6 is asserting. Evidence: scripts/check-tla.sh:817-819 versus scripts/check-tla.sh:770-772. The asymmetry is stated in formal/tla/README.md:241-243, so this is a question about the rule rather than a mismatch with it; rejecting asut:path undertests/alongside the existing.rsand repo-relative checks would close it. - scripts/check-tla.sh:977 (nit): The row directly under the header is skipped whole when it matches
TRACE_DELIMITER_ROW, with no check that it has the table's column count. Failure: a five-column table whose separator is\|---\|---\|passes. Probed:check_traceabilityon a header of five cells, a two-cell separator and one valid data row returns 0,PASS (1 rows resolve). The table renders wrong in markdown and the checker says nothing, while the ADR amendment states that a table's rows must have the header's column count. Evidence: scripts/check-tla.sh:955-958. Onetrace_cellscount comparison on the delimiter row beforecontinuewould cover it.
Request changes: one should_fix finding. All seven previous findings are fixed; the latest commit traded the early-end bug for an over-skip in the other direction.
The #[cfg(test)] skip now ends only on balanced braces (once a brace opened) or on a line ending in ;. Two item shapes match neither, so the skip runs past the item into live code: an item whose braces open and close on the head line, and an item with no braces that ends in , (a gated struct field). Both swallow the rest of the file until some later line happens to end in ;. The direction is safe (reject, not accept) and no area is enrolled, but it is the direction previous finding 1 was about, and both shapes were handled correctly at 559a843.
Verified:
- Previous finding 5's case is gone:
rust_literal_live '"provisioning GET"' crates/ravel-ingest/src/read_backoff.rsexits 1 (the literal lives only in the gated multi-line helper at read_backoff.rs:184-192). - Previous finding 1's case is gone: live literals after
crates/ravel-otlp/src/label_projection.rs:105("_bucket","instance","otel.scope.name") all resolve live. - The ADR-4326 bullet (docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260-263) now matches scripts/check-tla.sh:771-784 and :837, and formal/tla/README.md:240-246 states the same rules.
- The separator cell-count check exists at scripts/check-tla.sh:983-986; the
sut:refusal undertests/at :837. - "202 rows, 54 in agents" matches the checker: 54+25+21+9+29+45+19 = 202, and formal/tla/TRACEABILITY.md:22,24 carry those numbers.
Checks run:
scripts/check-tla.sh traceability-> all areas pass, 202 rows.scripts/check-tla.test.sh-> 158 passed, 0 failed.- Revert check on the new cases: ran the 559a843 awk scan against the new
early.rsfixture shapes.cfg-test-multiline,midline-blockandmidline-block-bodyall read live under the old scan (exit 0) and dead under the new one (exit 1), so each new case fails without this commit. - Ran the current scan over every
.rsundercrates/andservices/with the skip instrumented: 95 gated items whose head line neither balances braces nor ends in;, plus one brace-balanced one-liner.
Smaller things:
- The description says the suite passes "148 of 148"; it now reports 158 passed.
- A line that closes a comment and then carries code is dropped whole:
/* note */ let m = "MSG";and a block whose*/is followed by code both report"MSG"dead (probed, exit 1 each). formal/tla/README.md:240-243 states the rule this way, so it is the rule rather than a mismatch, and the direction is a reject.
Not checked:
- docs/adrs/README.md and docs/adrs/1113-tla-verification-suite.md were read only for the ADR-1113 status row and the amendment wording; the amendment-integrity and band-source guards were not run.
- check_docs.py and the changelog guard, which the description reports as exit 0.
Task 1e103d99-3223-4810-b977-5f5fa097a1d0 on rp1, model claude-opus-5, effort medium.
| function skip_line(s) { | ||
| depth += braces(s) | ||
| if (depth > 0) opened = 1 | ||
| if ((opened && depth <= 0) || (!opened && s ~ /;[[:space:]]*$/)) skipping = 0 |
There was a problem hiding this comment.
should_fix: The skip ends only when braces balance after one opened, or on a line ending in ;, so a #[cfg(test)] item that matches neither keeps skipping on and swallows live code until some unrelated later line ends in ;.
Failure (gated struct field, ends in ,): sut: crates/ravel-object-store/src/azure.rs -- CredentialEnv fails, although "CredentialEnv" is live production code at azure.rs:334 inside impl fmt::Debug for CredentialEnv. The #[cfg(test)] pub(crate) authority_host: Option<String>, at azure.rs:328-329 arms the skip with depth = 0, opened = 0; the struct's closing } drives depth to -1 without ever setting opened, so the skip runs 85 lines. Sourced the lib and called rust_literal_live '"CredentialEnv"' crates/ravel-object-store/src/azure.rs: exit 1, dead. Same for "<redacted>" at azure.rs:340.
Failure (one-line item, braces balance on the head line): #[cfg(test)] fn only_for_test() { let _ = 1; } followed by fn publish() { let m = "live msg"; } reports "live msg" dead (probed: exit 1). braces() returns 0 for the head line, so opened is never set and the line ends in } rather than ;. The real tree has this shape at crates/ravel-sql/src/statement.rs:257-259 (fn lowering_at(&self, _site: super::LoweringSite) {}), where the skip runs to line 298 and hides the LoweringSite, LowerError and Statement definitions.
Both shapes worked at 559a843, whose head-line test was $0 !~ /[;}][[:space:]]*$/: the one-liner ended at its own line and the field ended at the struct's }. Neither shape is in the fixtures, so scripts/check-tla.test.sh passes 158/158 with the regression in. Over the whole tree the instrumented scan finds 95 gated items whose head line neither balances braces nor ends in ;, plus the one brace-balanced one-liner.
Evidence: scripts/check-tla.sh:767-769 and :775; crates/ravel-object-store/src/azure.rs:328-340; crates/ravel-sql/src/statement.rs:257-298.
Ending the skip when the head line's braces balance and at least one brace was seen, or when it ends in ; or ,, covers both; formal/tla/README.md:240-246 and the ADR-4326 bullet then need the matching wording, and early.rs needs a one-line gated item plus a gated struct field with live code after each.
There was a problem hiding this comment.
Fixed in dd493e9. The skip now ends on the item's first line when it ends in , (a field), once the braces balance on any line that opened one (so a one-line item ends at its own line), and never past a closing brace of the enclosing scope. Cases ay2-after-one-line and ay2-after-field fail against the previous scan; the field case is in its own fixture file so an earlier skip cannot swallow its attribute. On the real tree, "CredentialEnv" and "<redacted>" in azure.rs and the literals after statement.rs:257 now read as live. Per the owner, this lands on green CI without a fourth review round.
The previous change ended a #[cfg(test)] skip only on a line ending in ; before any brace, or once braces balanced after one opened. Two common shapes matched neither: a gated struct field (its line ends in ,) and a one-line item whose braces balance on its own line. The skip then ran on and hid live code, for example "CredentialEnv" in crates/ravel-object-store/src/azure.rs after a gated field and the definitions after a one-line gated fn in crates/ravel-sql/src/statement.rs. The skip now ends on the item's first line when that line ends in , (a field), on any line containing an opening brace once the braces balance, and never runs past a closing brace of the enclosing scope. A multi-line signature still stays hidden, since only the first line's trailing comma ends the item. Cases ay2-after-one-line and ay2-after-field fail against the previous scan; the field case sits in its own file so an earlier skip cannot swallow its attribute. Refs: #4330 Refs: #4326 Changelog: none Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
scripts/check-tla.sh traceabilitynow checks a sixth "Antithesis property" column in each area that opts in by holding anantithesis-mutants.tsv, as ADR-4326 decision 6 sets out. No area is enrolled yet, so every area stays five-column and the real tree still checks clean at 202 rows.The rules:
.tsvfails. A header is recognised by its exact first cell. Only the row directly under a header counts as its separator.sut: <path.rs> -- <message>: the file exists and holds the message as a whole quoted string literal.driver: <workload> -- <message>: the literal is in a.rsfile underantithesis/<workload>/.none: <reason>, with a non-empty reason..tsv. It needs one line pernegative/*.cfg. Each line names a mutant whoseinv_mutant!("<area>/<name>")literal is live Rust undercrates/,services/orantithesis/*/driver/. Comment lines,cfg(test)items andtests/do not count. The line's property must be one the table carries, ornone: <reason>.Docs:
formal/tla/README.mddescribes the column and the.tsv.TRACEABILITY.md's totals now match what the checker reports: 202 rows, 54 of them in agents.Tests:
scripts/check-tla.test.shpasses 148 of 148. Each rule was broken one at a time in a scratch tree, and a case failed every time.check_docs.py, the amendment, band-sources and changelog guards all exit 0.No cargo gate: the change is scripts and docs only.
Fixes: #4330
Refs: #4326