Skip to content

test(tla): check the Antithesis property column in traceability - #4359

Queued
pmoust wants to merge 6 commits into
mainfrom
task/14347065-3f87-412e-84af-a6a0ca63f43f/merge
Queued

pmoust wants to merge 6 commits into
mainfrom
task/14347065-3f87-412e-84af-a6a0ca63f43f/merge

Conversation

@pmoust

@pmoust pmoust commented Nov 16, 2026

Copy link
Copy Markdown
Member

scripts/check-tla.sh traceability now checks a sixth "Antithesis property" column in each area that opts in by holding an antithesis-mutants.tsv, as ADR-4326 decision 6 sets out. No area is enrolled yet, so every area stays five-column and the real tree still checks clean at 202 rows.

The rules:

  • Enrollment. An enrolled area's tables must have six columns. A six-column table in an area with no .tsv fails. A header is recognised by its exact first cell. Only the row directly under a header counts as its separator.
  • Cell forms. Each cell must be one of:
    • sut: <path.rs> -- <message>: the file exists and holds the message as a whole quoted string literal.
    • driver: <workload> -- <message>: the literal is in a .rs file under antithesis/<workload>/.
    • none: <reason>, with a non-empty reason.
  • The .tsv. It needs one line per negative/*.cfg. Each line names a mutant whose inv_mutant!("<area>/<name>") literal is live Rust under crates/, services/ or antithesis/*/driver/. Comment lines, cfg(test) items and tests/ do not count. The line's property must be one the table carries, or none: <reason>.

Docs:

  • formal/tla/README.md describes the column and the .tsv.
  • TRACEABILITY.md's totals now match what the checker reports: 202 rows, 54 of them in agents.
  • ADR-1113 is Accepted, so it gets an amendment with an inline pointer in D8. Its index row now says Accepted.
  • ADR-4326 gets an amendment recording that this change corrected the two stale facts its Context cited.

Tests: scripts/check-tla.test.sh passes 148 of 148. Each rule was broken one at a time in a scratch tree, and a case failed every time. check_docs.py, the amendment, band-sources and changelog guards all exit 0.

No cargo gate: the change is scripts and docs only.

Fixes: #4330
Refs: #4326

check-tla.sh traceability now implements ADR-4326 decision 6. An area is
enrolled when its directory holds antithesis-mutants.tsv. Every table in an
enrolled area's traceability.md must then carry a sixth "Antithesis
property" column whose cell is sut: <path.rs> -- <message> (the file holds
the literal), driver: <workload> -- <message> (the literal occurs under
antithesis/<workload>/), or none: <reason> with a non-empty reason. A
five-column table in an enrolled area fails, and so does a six-column table
in an area with no .tsv. The .tsv needs one line per negative/*.cfg, naming
either the mutant <area>/<stem>, found in Rust as inv_mutant!("<area>/<stem>"),
and a message a sut: or driver: cell carries, or none: <reason>.

A table header is now recognised by its exact first cell, "TLA+ action or
property", instead of a pattern that also skipped any data row mentioning an
action and a property; such rows are checked now. Every data row must have
the header's cell count. No area is enrolled by this change, and the real
tree still resolves all 202 rows.

formal/tla/README.md and TRACEABILITY.md describe the column and the .tsv,
and TRACEABILITY.md's counts now match the checker (agents 54, total 202).
ADR-1113 gains an amendment for D8 with an inline pointer, and its index row
in docs/adrs/README.md now says Accepted as the ADR does.

Refs: #4330
Refs: #4326
Changelog: none
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Review fixes for the Antithesis property column checker.

A row of only pipes, spaces, dashes and colons was skipped as a separator
anywhere in a table, so an all-empty row or "| - | - | ... |" passed. Only
the row directly under the header is a separator now; any later such row is
a data row and fails, in enrolled and unenrolled areas alike.

A sut: or driver: message now counts only as the whole double-quoted string
literal, and driver: searches only .rs files under antithesis/<workload>/,
so README prose and a substring of a longer literal no longer satisfy it.
Mutants are searched under antithesis/*/driver/ as well as crates/ and
services/, where ADR-4326 decision 3 puts the common area's broken store.
A literal on a comment line or after a file's first #[cfg(test)] line does
not count, nor, for driver: and mutants, one in a file under tests/.
none: needs a space before its reason, in the cell and in the .tsv.

New cases cover each of these and the cell-count, duplicate stem, unknown
stem, " -- ", unknown prefix, sixth header name, empty .tsv none: reason
and <area>/<stem> name rules; each was shown failing with its rule broken.
ADR-4326 gains an amendment recording that #4330 corrected the ADR-1113
index row and the TRACEABILITY.md count, and narrowing decision 6.

Refs: #4330
Refs: #4326
Changelog: none
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
The driver: check matches a message only as a whole string literal in the
workload's .rs files, but no case failed when that match dropped its
quotes. Two cases now cover it: a message that occurs in the driver only
as an identifier, and one that is only part of a longer literal. Both fail
against a match without the quotes.

The README described a later separator-like row as failing for want of a
Rust source ref; it fails on whatever cell is not a valid entry.

Refs: #4330
Refs: #4326
Changelog: none
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
@pmoust

pmoust commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

@claude-fleet review

@claude-fleet

claude-fleet Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Review task queued.

Task: 22bce31c-fd3e-4a3f-95c4-9845051dd244
Commit: fda2868
Model: claude-opus-5 (effort medium)
Executor: ci-16gb-fsn1-1
Status: done. Review posted.

@pmoust pmoust added the antithesis Worked with or found by Antithesis (deterministic simulation testing) label Nov 16, 2026

@claude-fleet claude-fleet Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the whole diff, ran the suite and the doc guards, and probed the new rules against the real tree.

Request changes: one should_fix, three nits.

Findings:

  • should_fix, scripts/check-tla.sh:760 — rust_literal_live stops at the first #[cfg(test)] line, hiding production literals in most repo files
  • nit, scripts/check-tla.sh:761 — a block-comment body line without a leading * counts as live
  • nit, scripts/check-tla.sh:817 — sut: accepts a path under tests/, unlike driver: and mutants
  • nit, scripts/check-tla.sh:958 — a separator row's column count is never checked

Verified:

  • 202 rows total, 54 in agents, matching the new TRACEABILITY.md figures — check-tla.sh traceability reports 54+25+21+9+29+45+19
  • no area is enrolled and every area still passes five-column — same run, exit 0
  • the header is matched on its exact first cell and only the row under it is a separator — scripts/check-tla.sh:942-958, proved by cases (ao2), (au1)-(au4)
  • the sut: path stays a whole token for the existing crates/... sweep — scripts/check-tla.sh:988-991, case (ai2) fails on missing source 'crates/gone.rs'
  • the mutant literal is accepted from antithesis/*/driver/ and rejected outside it — scripts/check-tla.sh:870-873, case (ax1)/(ax2)
  • ADR-1113's status line reads Accepted (2026-09-04), so the index row change matches it — docs/adrs/1113-tla-verification-suite.md:3
  • both amendment markers name real sections (Context|6. The traceability column at docs/adrs/4326-...md:145, D8. Traceability names symbols, not lines)

Checks run:

  • bash scripts/check-tla.test.sh → 148 passed, 0 failed
  • bash scripts/check-tla.sh traceability → exit 0, all seven areas PASS
  • bash scripts/guards/check-amendment-integrity.sh → clean (565 amendments)
  • bash scripts/guards/check-adr-band-sources.sh → clean
  • python3 scripts/check_docs.py → docs gate: clean.
  • Probe: sourced the lib against a scratch repo to reproduce the four findings below

Smaller things:

  • strip_code_span unwraps exactly one backtick pair, so a message written as a double-backtick span (needed when the message itself contains a backtick) will not match. Not reachable today.
  • A sut: or driver: message that happens to contain a crates/... token will be picked up by the trailing crates/ sweep and resolved as a source ref. Pre-existing sweep behaviour, but the new column makes free text in a row more likely.

Not checked:

  • Nothing in the diff was left unread. I did not run TLC (the traceability lane needs no Java) or any cargo gate; the change is scripts and docs only, as the description says.

Task 22bce31c-fd3e-4a3f-95c4-9845051dd244 on ci-16gb-fsn1-1, model claude-opus-5, effort medium.

Comment thread scripts/check-tla.sh Outdated
# never as awk program text or -v, so backslashes stay literal.
rust_literal_live() {
RUST_LITERAL="$1" awk '
/^[[:space:]]*#\[cfg\(test\)\]/ { exit }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should_fix: rust_literal_live treats the file's first #[cfg(test)] line as the start of the test region and stops scanning there, but in this repo that attribute very often sits on a single test-only item in the middle of a file, with hundreds of lines of production code after it.

Failure: an area is enrolled and a row's cell is sut: crates/ravel-otlp/src/label_projection.rs -- <message> for an inv_always! added anywhere after line 105. crates/ravel-otlp/src/label_projection.rs:105 is #[cfg(test)] pub(crate) fn for_test, and real production items follow at lines 126, 161, 182, 223 and 288 of 1514. The checker reports message '<m>' not found as the string literal "<m>" in '<path>' for a literal that is plainly live. The same shape holds for crates/ravel-cache/src/lib.rs (first #[cfg(test)] at 107 of 980), crates/ravel-fleet/src/query_workers.rs (637 of 1283) and many others; a scan of the repo found dozens.

Evidence: scripts/check-tla.sh:760 /^[[:space:]]*#\[cfg\(test\)\]/ { exit }. Reproduced by sourcing the lib against a scratch tree whose crates/a.rs is fn helper(), #[cfg(test)] fn only_for_test(), then fn publish() { inv_always!(ok, "commit never precedes data"); }: check_traceability returns 1 with exactly that diagnostic.

The failure direction is safe (it rejects, it does not accept), and no area is enrolled yet, so nothing is broken today. But this is the rule the first enrollment runs into. Matching only a #[cfg(test)] that introduces a module (#[cfg(test)] immediately followed by a mod line), or skipping the gated item rather than the rest of the file, would hold the intent without the false rejection.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 559a843. The scan now skips only the item after #[cfg(test)] (and any further attributes): one line when the item ends there, otherwise every line until its braces balance. New cases ay2-after-test-mod, ay2-after-cfg-use and ay2-after-deref put a live literal after a test module, after a #[cfg(test)] use, and on a line starting with a dereference. All three fail against the old exit-at-first-attribute scan.

Comment thread scripts/check-tla.sh Outdated
rust_literal_live() {
RUST_LITERAL="$1" awk '
/^[[:space:]]*#\[cfg\(test\)\]/ { exit }
/^[[:space:]]*(\/\/|\/\*|\*)/ { next }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: The comment skip matches only //, /* or * after leading whitespace, so a line inside a /* ... */ block that is not star-prefixed counts as live code.

Failure: a file holding

/*
    if inv_mutant!("enr/block-no-star") {}
    let dead = "a dead message";
*/

satisfies both the mutant rule and a sut: cell naming a dead message. Probed against a scratch tree: check_traceability returns 0 and reports PASS (2 rows resolve) with the mutant and the message living only inside that comment. Case (ay) covers the //, single-line /* ... */ and * forms, so this one slips past.

Evidence: scripts/check-tla.sh:761. formal/tla/README.md:238-241 and the ADR-4326 amendment both state the prefix rule exactly, so the behaviour is documented rather than a surprise. It is worth noting because this direction is the unsafe one: it accepts a mutant that does not exist in live code.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 559a843. The scan tracks /* ... */ blocks instead of matching a leading *, so an unprefixed body line is skipped (case ay-block-bare) and a line starting with a dereference is no longer dropped.

Comment thread scripts/check-tla.sh
*.rs) : ;;
*) note "$where: sut: path '$target' is not a Rust (.rs) path"; return 1 ;;
esac
[ -f "$REPO_ROOT/$target" ] || { note "$where: sut: no such file '$target'"; return 1; }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: A sut: cell is checked with rust_literal_live directly, which applies no tests/ exclusion, while driver: cells and mutants go through rust_literal_live_under and skip tests/ (scripts/check-tla.sh:770-772).

Failure: sut: crates/m/tests/it.rs -- tests only message passes. Probed against a scratch tree with that file and no other occurrence: PASS (1 rows resolve). A property whose only carrier is an integration test is not carried in the SUT, which is what decision 6 is asserting.

Evidence: scripts/check-tla.sh:817-819 versus scripts/check-tla.sh:770-772. The asymmetry is stated in formal/tla/README.md:241-243, so this is a question about the rule rather than a mismatch with it; rejecting a sut: path under tests/ alongside the existing .rs and repo-relative checks would close it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 559a843: a sut: path under tests/ now fails (case ay3).

Comment thread scripts/check-tla.sh Outdated
# later row of pipes and dashes is a data row and is checked as one.
if [ "$after_header" -eq 1 ]; then
after_header=0
[[ "$line" =~ $TRACE_DELIMITER_ROW ]] && continue

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: The row directly under the header is skipped whole when it matches TRACE_DELIMITER_ROW, with no check that it has the table's column count.

Failure: a five-column table whose separator is |---|---| passes. Probed: check_traceability on a header of five cells, a two-cell separator and one valid data row returns 0, PASS (1 rows resolve). The table renders wrong in markdown and the checker says nothing, while the ADR amendment states that a table's rows must have the header's column count.

Evidence: scripts/check-tla.sh:955-958. One trace_cells count comparison on the delimiter row before continue would cover it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 559a843: the separator row under a header must have the table's cell count (case ay4).

rust_literal_live stopped scanning at a file's first #[cfg(test)] line. In
this repository that attribute often sits on a single use or helper near
the top of a file, so every production literal below it was hidden and a
valid sut:, driver: or mutant entry failed. The scan now skips only the
item that follows the attribute (and any further attributes): one line
when the item ends there, else every line until its braces balance.

The review of the first version also found three smaller gaps, fixed here.
A line inside a /* ... */ block without a leading * counted as live code,
and a line starting with a dereference (*slot = ...) was dropped as a
comment, so the scan now tracks block comments instead of matching a *
prefix. A sut: path under tests/ was accepted, unlike driver: and mutant
literals. A separator row's column count was never checked.

Each new case fails against the rule it pins: a scan that exits at the
first #[cfg(test)] fails the three ay2 cases, a scan that does not skip
block bodies fails ay-block-body and ay-block-bare, and removing the tests/
or separator checks fails ay3 and ay4.

Refs: #4330
Refs: #4326
Changelog: none
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
@pmoust

pmoust commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

@claude-fleet review

@claude-fleet

claude-fleet Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Review task queued.

Task: 28998fa2-3619-49a0-9266-ebfb58620752
Commit: 559a843
Model: claude-opus-5 (effort medium)
Executor: ci-16gb-fsn1-1
Status: done. Review posted.

@claude-fleet claude-fleet Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous round at fda2868: 4 findings.

Previous finding Classification Cited line
scripts/check-tla.sh:760 (should_fix): rust_literal_live treats the file's first #[cfg(test)] line as the start of the test region and stops scanning there, but in this repo that attribute very often sits on a single test-only item in the middle of a file, with hundreds of lines of production code after it. Failure: an area is enrolled and a row's cell is sut: crates/ravel-otlp/src/label_projection.rs -- <message> for an inv_always! added anywhere after line 105. crates/ravel-otlp/src/label_projection.rs:105 is #[cfg(test)] pub(crate) fn for_test, and real production items follow at lines 126, 161, 182, 223 and 288 of 1514. The checker reports message '<m>' not found as the string literal "<m>" in '<path>' for a literal that is plainly live. The same shape holds for crates/ravel-cache/src/lib.rs (first #[cfg(test)] at 107 of 980), crates/ravel-fleet/src/query_workers.rs (637 of 1283) and many others; a scan of the repo found dozens. Evidence: scripts/check-tla.sh:760 /^[[:space:]]*#\[cfg\(test\)\]/ { exit }. Reproduced by sourcing the lib against a scratch tree whose crates/a.rs is fn helper(), #[cfg(test)] fn only_for_test(), then fn publish() { inv_always!(ok, "commit never precedes data"); }: check_traceability returns 1 with exactly that diagnostic. The failure direction is safe (it rejects, it does not accept), and no area is enrolled yet, so nothing is broken today. But this is the rule the first enrollment runs into. Matching only a #[cfg(test)] that introduces a module (#[cfg(test)] immediately followed by a mod line), or skipping the gated item rather than the rest of the file, would hold the intent without the false rejection. fixed scripts/check-tla.sh:770
scripts/check-tla.sh:761 (nit): The comment skip matches only //, /* or * after leading whitespace, so a line inside a /* ... */ block that is not star-prefixed counts as live code. Failure: a file holding /* if inv_mutant!("enr/block-no-star") {} let dead = "a dead message"; */ satisfies both the mutant rule and a sut: cell naming a dead message. Probed against a scratch tree: check_traceability returns 0 and reports PASS (2 rows resolve) with the mutant and the message living only inside that comment. Case (ay) covers the //, single-line /* ... */ and * forms, so this one slips past. Evidence: scripts/check-tla.sh:761. formal/tla/README.md:238-241 and the ADR-4326 amendment both state the prefix rule exactly, so the behaviour is documented rather than a surprise. It is worth noting because this direction is the unsafe one: it accepts a mutant that does not exist in live code. fixed scripts/check-tla.sh:777
scripts/check-tla.sh:817 (nit): A sut: cell is checked with rust_literal_live directly, which applies no tests/ exclusion, while driver: cells and mutants go through rust_literal_live_under and skip tests/ (scripts/check-tla.sh:770-772). Failure: sut: crates/m/tests/it.rs -- tests only message passes. Probed against a scratch tree with that file and no other occurrence: PASS (1 rows resolve). A property whose only carrier is an integration test is not carried in the SUT, which is what decision 6 is asserting. Evidence: scripts/check-tla.sh:817-819 versus scripts/check-tla.sh:770-772. The asymmetry is stated in formal/tla/README.md:241-243, so this is a question about the rule rather than a mismatch with it; rejecting a sut: path under tests/ alongside the existing .rs and repo-relative checks would close it. fixed scripts/check-tla.sh:830
scripts/check-tla.sh:958 (nit): The row directly under the header is skipped whole when it matches TRACE_DELIMITER_ROW, with no check that it has the table's column count. Failure: a five-column table whose separator is |---|---| passes. Probed: check_traceability on a header of five cells, a two-cell separator and one valid data row returns 0, PASS (1 rows resolve). The table renders wrong in markdown and the checker says nothing, while the ADR amendment states that a table's rows must have the header's column count. Evidence: scripts/check-tla.sh:955-958. One trace_cells count comparison on the delimiter row before continue would cover it. fixed scripts/check-tla.sh:977

Verified the fix commit against all four earlier findings, then reviewed the full diff.

Request changes: one should_fix on the new #[cfg(test)] skip, one on a now-stale ADR bullet, one nit.

All four previous findings are fixed. The revert check proves it: with scripts/check-tla.sh reverted to fda2868 and the new test file in place, ay-block-bare, ay2-after-test-mod, ay2-after-cfg-use, ay2-after-deref, ay3 and ay4 fail (149 passed, 6 failed); on the head tree the suite is 155/155.

The new item-scoped skip is a real improvement over "stop at the first #[cfg(test)] line", and I confirmed it on the exact file the first finding cited (crates/ravel-otlp/src/label_projection.rs:105-109): production code after the gated for_test is scanned again. What it does not handle is a gated item whose signature spans lines, which is the common shape in this repo, and there the direction flips to accepting test-only text as SUT evidence.

Findings:

  • should_fix, scripts/check-tla.sh:773 — a gated item with a multi-line signature ends its skip after one line, so its body counts as live
  • should_fix, docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260 — the amendment still states the superseded comment/cfg(test)/tests/ rules
  • nit, scripts/check-tla.sh:777 — a /* ... */ block opened mid-line is not tracked

Verified:

  • ADR-1113's status line reads Accepted, so the index row change matches it — docs/adrs/1113-tla-verification-suite.md header, docs/adrs/README.md:149
  • TRACEABILITY.md's new totals are what the checker reports: 54 agents rows, 202 overall (9+21+25+29+45+19+54) — formal/tla/TRACEABILITY.md:22-24
  • The separator cell-count check runs on the delimiter row before continue, and only for the row directly under the header — scripts/check-tla.sh:972-981
  • A sut: path under tests/ is now rejected before the file-exists check, matching rust_literal_live_under's exclusion — scripts/check-tla.sh:830 vs 789
  • Finding 2's case is closed: a non-star line inside a block comment that opens at line start is skipped — scripts/check-tla.sh:766, 777

Checks run:

  • bash scripts/check-tla.test.sh → 155 passed, 0 failed
  • Same suite with scripts/check-tla.sh reverted to fda2868 → 149 passed, 6 failed (the six cases above)
  • bash scripts/check-tla.sh traceability → all areas pass, 202 rows
  • scripts/guards/check-amendment-integrity.sh → exit 0 (565 amendments); scripts/guards/check-adr-band-sources.sh → exit 0
  • Sourced the script's lib and ran rust_literal_live against real repo files and scratch files for each edge case below

Smaller things:

  • The brace counting is textual, and the code comment and formal/tla/README.md:242-244 say so, but the early-end direction is the unsafe one: in a scratch file, #[cfg(test)] mod tests { whose first body line is fn t() { let s = "}"; } ends the skip there, and the rest of the module counts as live. services/ravel-server/src/ipcrypt.rs:521-547 already solves this in Rust by matching the } at the attribute's own indent, which is immune to braces in strings and also fixes the multi-line-signature case below. Worth copying that rule rather than counting braces.
  • The PR body says the suite passes "148 of 148"; at this head it is 155 of 155.

Not checked: nothing in the diff was left unread; the Antithesis workload layout under antithesis/ does not exist yet in this tree, so the driver: path was exercised only through the test suite's scratch trees.

Not checked:

  • The antithesis/ workload tree does not exist in this checkout, so the driver: branch of check_antithesis_cell was exercised only through the test suite's scratch trees, not against a real workload.

Task 28998fa2-3619-49a0-9266-ebfb58620752 on ci-16gb-fsn1-1, model claude-opus-5, effort medium.

Comment thread scripts/check-tla.sh Outdated
pending {
pending = 0
depth = braces($0)
if (depth > 0 || ($0 !~ /[;}][[:space:]]*$/)) skipping = 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should_fix: A #[cfg(test)] item whose signature spans several lines sets skipping with depth = 0, so the very next line drives depth <= 0 and ends the skip one line into the item: the item's body is then scanned as live code.

Failure: sut: crates/ravel-ingest/src/read_backoff.rs -- provisioning GET passes, although "provisioning GET" occurs only at line 192, inside the #[cfg(test)] pub(crate) fn failing_then_passing_get_store( whose parameter list runs to line 186 (grep -rn '"provisioning GET"' --include='*.rs' crates services returns that one line). Sourced the script's lib and called rust_literal_live '"provisioning GET"' crates/ravel-ingest/src/read_backoff.rs against the real file: exit 0, live. Same for the literal at crates/ravel-catalog/src/fold.rs:2456, whose only home is the #[cfg(test)] pub(crate) fn build_part_key_index_leaves( starting at line 2380. This is the unsafe direction: a property whose only carrier is a test helper reads as carried in the SUT.

Evidence: scripts/check-tla.sh:772-773. Line 772 counts braces on the head line only, and 773 arms skipping at depth 0 whenever the head neither opens a brace nor ends in ;/} — which is every wrapped signature and every gated struct field. Running the same awk state machine over all 1535 .rs files under crates/ and services/ gives 1028 skips, 72 of which end on a line that is not the item's closing brace; 46 of those end on a parameter line such as &self,, mut self, or store: &dyn ObjectStoreBackend,. One of them (crates/ravel-ingest/src/write_gate.rs:79) is itself a #[cfg(test)] attribute swallowed by the skip, so the field it gates is scanned as live too.

services/ravel-server/src/ipcrypt.rs:521-547 already implements this rule in Rust by skipping to the } at the attribute's own indentation, which handles both this case and the brace-in-a-string case the comment documents.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc639b7: the skip now runs through the first line ending in ; when no brace has opened, else until the braces balance. Case ay-cfg-test-multiline (a fn with a three-line signature) fails against the previous scan.

- A mutant from decision 3 occurs as `inv_mutant!("<area>/<stem>")` in a
`.rs` file under `crates/`, `services/` or `antithesis/*/driver/`, the last
for the common area's driver-side broken store.
- For a message and a mutant alike, a literal on a comment line (`//`, `/*`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should_fix: This PR's last commit changed two rules and updated formal/tla/README.md and ADR-1113's amendment, but not this bullet, so the ADR-4326 amendment now states behaviour the checker does not have.

Failure: a reader enrolling an area follows "a literal ... after the file's first #[cfg(test)] line does not count" and concludes a literal below the first gated item in a file is invisible; scripts/check-tla.sh:766-777 skips only the gated item, and test case ay2 in scripts/check-tla.test.sh pins the opposite. The same bullet says only driver: and mutants exclude tests/, while scripts/check-tla.sh:830 now rejects a sut: path under tests/ outright.

Evidence: docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260-263 versus formal/tla/README.md:240-246 and docs/adrs/1113-tla-verification-suite.md:617-620, which both carry the new wording.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc639b7: the amendment bullet now states the current rules (comments incl. mid-line /*, #[cfg(test)] items, tests/ files, and the sut: refusal under tests/), and formal/tla/README.md matches.

Comment thread scripts/check-tla.sh Outdated
next
}
/^[[:space:]]*\/\// { next }
/^[[:space:]]*\/\*/ { if (!index(substr($0, index($0, "/*") + 2), "*/")) in_block = 1; next }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: in_block is armed only by a line whose first non-space characters are /*, so a block comment opened after code on the same line is never tracked and its body counts as live.

Failure: a file holding

fn f() {
    let s = "}"; /* opens here
    let dead = "MSG";
    */
}

makes a sut: cell naming MSG pass. Probed by sourcing the lib against a scratch file of exactly that shape: rust_literal_live '"MSG"' <file> exits 0. The star-prefixed and line-start forms that case (ay) covers are now handled; this is the remaining opening form, and it accepts dead text.

Evidence: scripts/check-tla.sh:777. formal/tla/README.md:241-242 states the rule as written, so this is a question about the rule rather than a mismatch with it.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in cc639b7: text after a /* on any line is dropped and the scan skips to the matching */. Cases ay-midline-block and ay-midline-block-body fail against the previous scan.

A #[cfg(test)] item whose signature spans several lines ended its skip on
the first line, because no brace had opened yet, so the rest of the
signature and the whole body counted as live code. The skip now runs
through the first line ending in ; when no brace has opened, and otherwise
until the braces balance.

A /* ... */ comment opened after code on a line was not tracked, so its
text and the lines that followed it up to */ counted as live. The scan now
drops the text after a /* on any line and skips to the matching */.

The ADR-4326 amendment and formal/tla/README.md still described the
earlier rules; both now state the current ones.

New cases ay-cfg-test-multiline, ay-midline-block and
ay-midline-block-body fail against the previous scan.

Refs: #4330
Refs: #4326
Changelog: none
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
@pmoust

pmoust commented Nov 16, 2026

Copy link
Copy Markdown
Member Author

@claude-fleet review

@claude-fleet

claude-fleet Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Review task queued.

Task: 1e103d99-3223-4810-b977-5f5fa097a1d0
Commit: cc639b7
Model: claude-opus-5 (effort medium)
Executor: rp1
Status: done. Review posted.

@claude-fleet claude-fleet Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous round at 559a843: 7 findings.

Previous finding Classification Cited line
scripts/check-tla.sh:770 (should_fix): rust_literal_live treats the file's first #[cfg(test)] line as the start of the test region and stops scanning there, but in this repo that attribute very often sits on a single test-only item in the middle of a file, with hundreds of lines of production code after it. Failure: an area is enrolled and a row's cell is sut: crates/ravel-otlp/src/label_projection.rs -- <message> for an inv_always! added anywhere after line 105. crates/ravel-otlp/src/label_projection.rs:105 is #[cfg(test)] pub(crate) fn for_test, and real production items follow at lines 126, 161, 182, 223 and 288 of 1514. The checker reports message '<m>' not found as the string literal "<m>" in '<path>' for a literal that is plainly live. The same shape holds for crates/ravel-cache/src/lib.rs (first #[cfg(test)] at 107 of 980), crates/ravel-fleet/src/query_workers.rs (637 of 1283) and many others; a scan of the repo found dozens. Evidence: scripts/check-tla.sh:760 /^[[:space:]]*#\[cfg\(test\)\]/ { exit }. Reproduced by sourcing the lib against a scratch tree whose crates/a.rs is fn helper(), #[cfg(test)] fn only_for_test(), then fn publish() { inv_always!(ok, "commit never precedes data"); }: check_traceability returns 1 with exactly that diagnostic. The failure direction is safe (it rejects, it does not accept), and no area is enrolled yet, so nothing is broken today. But this is the rule the first enrollment runs into. Matching only a #[cfg(test)] that introduces a module (#[cfg(test)] immediately followed by a mod line), or skipping the gated item rather than the rest of the file, would hold the intent without the false rejection. fixed scripts/check-tla.sh:775
scripts/check-tla.sh:777 (nit): The comment skip matches only //, /* or * after leading whitespace, so a line inside a /* ... */ block that is not star-prefixed counts as live code. Failure: a file holding /* if inv_mutant!("enr/block-no-star") {} let dead = "a dead message"; */ satisfies both the mutant rule and a sut: cell naming a dead message. Probed against a scratch tree: check_traceability returns 0 and reports PASS (2 rows resolve) with the mutant and the message living only inside that comment. Case (ay) covers the //, single-line /* ... */ and * forms, so this one slips past. Evidence: scripts/check-tla.sh:761. formal/tla/README.md:238-241 and the ADR-4326 amendment both state the prefix rule exactly, so the behaviour is documented rather than a surprise. It is worth noting because this direction is the unsafe one: it accepts a mutant that does not exist in live code. fixed scripts/check-tla.sh:781
scripts/check-tla.sh:830 (nit): A sut: cell is checked with rust_literal_live directly, which applies no tests/ exclusion, while driver: cells and mutants go through rust_literal_live_under and skip tests/ (scripts/check-tla.sh:770-772). Failure: sut: crates/m/tests/it.rs -- tests only message passes. Probed against a scratch tree with that file and no other occurrence: PASS (1 rows resolve). A property whose only carrier is an integration test is not carried in the SUT, which is what decision 6 is asserting. Evidence: scripts/check-tla.sh:817-819 versus scripts/check-tla.sh:770-772. The asymmetry is stated in formal/tla/README.md:241-243, so this is a question about the rule rather than a mismatch with it; rejecting a sut: path under tests/ alongside the existing .rs and repo-relative checks would close it. fixed scripts/check-tla.sh:837
scripts/check-tla.sh:977 (nit): The row directly under the header is skipped whole when it matches TRACE_DELIMITER_ROW, with no check that it has the table's column count. Failure: a five-column table whose separator is |---|---| passes. Probed: check_traceability on a header of five cells, a two-cell separator and one valid data row returns 0, PASS (1 rows resolve). The table renders wrong in markdown and the checker says nothing, while the ADR amendment states that a table's rows must have the header's column count. Evidence: scripts/check-tla.sh:955-958. One trace_cells count comparison on the delimiter row before continue would cover it. fixed scripts/check-tla.sh:984
scripts/check-tla.sh:773 (should_fix): A #[cfg(test)] item whose signature spans several lines sets skipping with depth = 0, so the very next line drives depth <= 0 and ends the skip one line into the item: the item's body is then scanned as live code. Failure: sut: crates/ravel-ingest/src/read_backoff.rs -- provisioning GET passes, although "provisioning GET" occurs only at line 192, inside the #[cfg(test)] pub(crate) fn failing_then_passing_get_store( whose parameter list runs to line 186 (grep -rn '"provisioning GET"' --include='*.rs' crates services returns that one line). Sourced the script's lib and called rust_literal_live '"provisioning GET"' crates/ravel-ingest/src/read_backoff.rs against the real file: exit 0, live. Same for the literal at crates/ravel-catalog/src/fold.rs:2456, whose only home is the #[cfg(test)] pub(crate) fn build_part_key_index_leaves( starting at line 2380. This is the unsafe direction: a property whose only carrier is a test helper reads as carried in the SUT. Evidence: scripts/check-tla.sh:772-773. Line 772 counts braces on the head line only, and 773 arms skipping at depth 0 whenever the head neither opens a brace nor ends in ;/} — which is every wrapped signature and every gated struct field. Running the same awk state machine over all 1535 .rs files under crates/ and services/ gives 1028 skips, 72 of which end on a line that is not the item's closing brace; 46 of those end on a parameter line such as &self,, mut self, or store: &dyn ObjectStoreBackend,. One of them (crates/ravel-ingest/src/write_gate.rs:79) is itself a #[cfg(test)] attribute swallowed by the skip, so the field it gates is scanned as live too. services/ravel-server/src/ipcrypt.rs:521-547 already implements this rule in Rust by skipping to the } at the attribute's own indentation, which handles both this case and the brace-in-a-string case the comment documents. fixed scripts/check-tla.sh:769
docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260 (should_fix): This PR's last commit changed two rules and updated formal/tla/README.md and ADR-1113's amendment, but not this bullet, so the ADR-4326 amendment now states behaviour the checker does not have. Failure: a reader enrolling an area follows "a literal ... after the file's first #[cfg(test)] line does not count" and concludes a literal below the first gated item in a file is invisible; scripts/check-tla.sh:766-777 skips only the gated item, and test case ay2 in scripts/check-tla.test.sh pins the opposite. The same bullet says only driver: and mutants exclude tests/, while scripts/check-tla.sh:830 now rejects a sut: path under tests/ outright. Evidence: docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260-263 versus formal/tla/README.md:240-246 and docs/adrs/1113-tla-verification-suite.md:617-620, which both carry the new wording. fixed docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260
scripts/check-tla.sh:777 (nit): in_block is armed only by a line whose first non-space characters are /*, so a block comment opened after code on the same line is never tracked and its body counts as live. Failure: a file holding fn f() { let s = "}"; /* opens here let dead = "MSG"; */ } makes a sut: cell naming MSG pass. Probed by sourcing the lib against a scratch file of exactly that shape: rust_literal_live '"MSG"' <file> exits 0. The star-prefixed and line-start forms that case (ay) covers are now handled; this is the remaining opening form, and it accepts dead text. Evidence: scripts/check-tla.sh:777. formal/tla/README.md:241-242 states the rule as written, so this is a question about the rule rather than a mismatch with it. fixed scripts/check-tla.sh:779

No longer carried to the next round, settled in rounds 2 and 3:

  • scripts/check-tla.sh:770 (should_fix): rust_literal_live treats the file's first #[cfg(test)] line as the start of the test region and stops scanning there, but in this repo that attribute very often sits on a single test-only item in the middle of a file, with hundreds of lines of production code after it. Failure: an area is enrolled and a row's cell is sut: crates/ravel-otlp/src/label_projection.rs -- <message> for an inv_always! added anywhere after line 105. crates/ravel-otlp/src/label_projection.rs:105 is #[cfg(test)] pub(crate) fn for_test, and real production items follow at lines 126, 161, 182, 223 and 288 of 1514. The checker reports message '<m>' not found as the string literal "<m>" in '<path>' for a literal that is plainly live. The same shape holds for crates/ravel-cache/src/lib.rs (first #[cfg(test)] at 107 of 980), crates/ravel-fleet/src/query_workers.rs (637 of 1283) and many others; a scan of the repo found dozens. Evidence: scripts/check-tla.sh:760 /^[[:space:]]*#\[cfg\(test\)\]/ { exit }. Reproduced by sourcing the lib against a scratch tree whose crates/a.rs is fn helper(), #[cfg(test)] fn only_for_test(), then fn publish() { inv_always!(ok, "commit never precedes data"); }: check_traceability returns 1 with exactly that diagnostic. The failure direction is safe (it rejects, it does not accept), and no area is enrolled yet, so nothing is broken today. But this is the rule the first enrollment runs into. Matching only a #[cfg(test)] that introduces a module (#[cfg(test)] immediately followed by a mod line), or skipping the gated item rather than the rest of the file, would hold the intent without the false rejection.
  • scripts/check-tla.sh:777 (nit): The comment skip matches only //, /* or * after leading whitespace, so a line inside a /* ... */ block that is not star-prefixed counts as live code. Failure: a file holding /* if inv_mutant!("enr/block-no-star") {} let dead = "a dead message"; */ satisfies both the mutant rule and a sut: cell naming a dead message. Probed against a scratch tree: check_traceability returns 0 and reports PASS (2 rows resolve) with the mutant and the message living only inside that comment. Case (ay) covers the //, single-line /* ... */ and * forms, so this one slips past. Evidence: scripts/check-tla.sh:761. formal/tla/README.md:238-241 and the ADR-4326 amendment both state the prefix rule exactly, so the behaviour is documented rather than a surprise. It is worth noting because this direction is the unsafe one: it accepts a mutant that does not exist in live code.
  • scripts/check-tla.sh:830 (nit): A sut: cell is checked with rust_literal_live directly, which applies no tests/ exclusion, while driver: cells and mutants go through rust_literal_live_under and skip tests/ (scripts/check-tla.sh:770-772). Failure: sut: crates/m/tests/it.rs -- tests only message passes. Probed against a scratch tree with that file and no other occurrence: PASS (1 rows resolve). A property whose only carrier is an integration test is not carried in the SUT, which is what decision 6 is asserting. Evidence: scripts/check-tla.sh:817-819 versus scripts/check-tla.sh:770-772. The asymmetry is stated in formal/tla/README.md:241-243, so this is a question about the rule rather than a mismatch with it; rejecting a sut: path under tests/ alongside the existing .rs and repo-relative checks would close it.
  • scripts/check-tla.sh:977 (nit): The row directly under the header is skipped whole when it matches TRACE_DELIMITER_ROW, with no check that it has the table's column count. Failure: a five-column table whose separator is \|---\|---\| passes. Probed: check_traceability on a header of five cells, a two-cell separator and one valid data row returns 0, PASS (1 rows resolve). The table renders wrong in markdown and the checker says nothing, while the ADR amendment states that a table's rows must have the header's column count. Evidence: scripts/check-tla.sh:955-958. One trace_cells count comparison on the delimiter row before continue would cover it.

Request changes: one should_fix finding. All seven previous findings are fixed; the latest commit traded the early-end bug for an over-skip in the other direction.

The #[cfg(test)] skip now ends only on balanced braces (once a brace opened) or on a line ending in ;. Two item shapes match neither, so the skip runs past the item into live code: an item whose braces open and close on the head line, and an item with no braces that ends in , (a gated struct field). Both swallow the rest of the file until some later line happens to end in ;. The direction is safe (reject, not accept) and no area is enrolled, but it is the direction previous finding 1 was about, and both shapes were handled correctly at 559a843.

Verified:

  • Previous finding 5's case is gone: rust_literal_live '"provisioning GET"' crates/ravel-ingest/src/read_backoff.rs exits 1 (the literal lives only in the gated multi-line helper at read_backoff.rs:184-192).
  • Previous finding 1's case is gone: live literals after crates/ravel-otlp/src/label_projection.rs:105 ("_bucket", "instance", "otel.scope.name") all resolve live.
  • The ADR-4326 bullet (docs/adrs/4326-tla-invariants-as-antithesis-properties.md:260-263) now matches scripts/check-tla.sh:771-784 and :837, and formal/tla/README.md:240-246 states the same rules.
  • The separator cell-count check exists at scripts/check-tla.sh:983-986; the sut: refusal under tests/ at :837.
  • "202 rows, 54 in agents" matches the checker: 54+25+21+9+29+45+19 = 202, and formal/tla/TRACEABILITY.md:22,24 carry those numbers.

Checks run:

  • scripts/check-tla.sh traceability -> all areas pass, 202 rows.
  • scripts/check-tla.test.sh -> 158 passed, 0 failed.
  • Revert check on the new cases: ran the 559a843 awk scan against the new early.rs fixture shapes. cfg-test-multiline, midline-block and midline-block-body all read live under the old scan (exit 0) and dead under the new one (exit 1), so each new case fails without this commit.
  • Ran the current scan over every .rs under crates/ and services/ with the skip instrumented: 95 gated items whose head line neither balances braces nor ends in ;, plus one brace-balanced one-liner.

Smaller things:

  • The description says the suite passes "148 of 148"; it now reports 158 passed.
  • A line that closes a comment and then carries code is dropped whole: /* note */ let m = "MSG"; and a block whose */ is followed by code both report "MSG" dead (probed, exit 1 each). formal/tla/README.md:240-243 states the rule this way, so it is the rule rather than a mismatch, and the direction is a reject.

Not checked:

  • docs/adrs/README.md and docs/adrs/1113-tla-verification-suite.md were read only for the ADR-1113 status row and the amendment wording; the amendment-integrity and band-source guards were not run.
  • check_docs.py and the changelog guard, which the description reports as exit 0.

Task 1e103d99-3223-4810-b977-5f5fa097a1d0 on rp1, model claude-opus-5, effort medium.

Comment thread scripts/check-tla.sh Outdated
function skip_line(s) {
depth += braces(s)
if (depth > 0) opened = 1
if ((opened && depth <= 0) || (!opened && s ~ /;[[:space:]]*$/)) skipping = 0

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should_fix: The skip ends only when braces balance after one opened, or on a line ending in ;, so a #[cfg(test)] item that matches neither keeps skipping on and swallows live code until some unrelated later line ends in ;.

Failure (gated struct field, ends in ,): sut: crates/ravel-object-store/src/azure.rs -- CredentialEnv fails, although "CredentialEnv" is live production code at azure.rs:334 inside impl fmt::Debug for CredentialEnv. The #[cfg(test)] pub(crate) authority_host: Option<String>, at azure.rs:328-329 arms the skip with depth = 0, opened = 0; the struct's closing } drives depth to -1 without ever setting opened, so the skip runs 85 lines. Sourced the lib and called rust_literal_live '"CredentialEnv"' crates/ravel-object-store/src/azure.rs: exit 1, dead. Same for "<redacted>" at azure.rs:340.

Failure (one-line item, braces balance on the head line): #[cfg(test)] fn only_for_test() { let _ = 1; } followed by fn publish() { let m = "live msg"; } reports "live msg" dead (probed: exit 1). braces() returns 0 for the head line, so opened is never set and the line ends in } rather than ;. The real tree has this shape at crates/ravel-sql/src/statement.rs:257-259 (fn lowering_at(&self, _site: super::LoweringSite) {}), where the skip runs to line 298 and hides the LoweringSite, LowerError and Statement definitions.

Both shapes worked at 559a843, whose head-line test was $0 !~ /[;}][[:space:]]*$/: the one-liner ended at its own line and the field ended at the struct's }. Neither shape is in the fixtures, so scripts/check-tla.test.sh passes 158/158 with the regression in. Over the whole tree the instrumented scan finds 95 gated items whose head line neither balances braces nor ends in ;, plus the one brace-balanced one-liner.

Evidence: scripts/check-tla.sh:767-769 and :775; crates/ravel-object-store/src/azure.rs:328-340; crates/ravel-sql/src/statement.rs:257-298.

Ending the skip when the head line's braces balance and at least one brace was seen, or when it ends in ; or ,, covers both; formal/tla/README.md:240-246 and the ADR-4326 bullet then need the matching wording, and early.rs needs a one-line gated item plus a gated struct field with live code after each.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dd493e9. The skip now ends on the item's first line when it ends in , (a field), once the braces balance on any line that opened one (so a one-line item ends at its own line), and never past a closing brace of the enclosing scope. Cases ay2-after-one-line and ay2-after-field fail against the previous scan; the field case is in its own fixture file so an earlier skip cannot swallow its attribute. On the real tree, "CredentialEnv" and "<redacted>" in azure.rs and the literals after statement.rs:257 now read as live. Per the owner, this lands on green CI without a fourth review round.

The previous change ended a #[cfg(test)] skip only on a line ending in ;
before any brace, or once braces balanced after one opened. Two common
shapes matched neither: a gated struct field (its line ends in ,) and a
one-line item whose braces balance on its own line. The skip then ran on
and hid live code, for example "CredentialEnv" in
crates/ravel-object-store/src/azure.rs after a gated field and the
definitions after a one-line gated fn in crates/ravel-sql/src/statement.rs.

The skip now ends on the item's first line when that line ends in , (a
field), on any line containing an opening brace once the braces balance,
and never runs past a closing brace of the enclosing scope. A multi-line
signature still stays hidden, since only the first line's trailing comma
ends the item.

Cases ay2-after-one-line and ay2-after-field fail against the previous
scan; the field case sits in its own file so an earlier skip cannot swallow
its attribute.

Refs: #4330
Refs: #4326
Changelog: none
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
@pmoust
pmoust added this pull request to the merge queue Nov 17, 2026
Any commits made after this event will not be merged.
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 17, 2026
@pmoust
pmoust added this pull request to the merge queue Nov 17, 2026
Any commits made after this event will not be merged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

antithesis Worked with or found by Antithesis (deterministic simulation testing)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4326-T1: Sixth traceability column and antithesis-mutants.tsv in check-tla.sh

1 participant