Skip to content

test(compat): released-binary fence drill, certify teardown test and CI lane - #4372

Queued
pmoust wants to merge 7 commits into
mainfrom
land/4243-fence-drill
Queued

pmoust wants to merge 7 commits into
mainfrom
land/4243-fence-drill

Conversation

@pmoust

@pmoust pmoust commented Nov 17, 2026

Copy link
Copy Markdown
Member

The released-binary fence drill for ADR-4233 decision 4 and decision 9, plus the deterministic certify test and the CI lane that decision 9 promises.

What is here:

  • scripts/compat/released-binary-drill.sh and drill-lib.sh run the signed v0.33.0 release binary against local RustFS with isolated buckets.
    • It drives every v0.33.0 write path before the fence (positive control), fences the provisioning records, and parks the alert lease.
    • It then records every key that lands after the fence, warm and after a cold restart.
    • It judges the run by the class table the owner decision (delegated) set on T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243. The fence's narrowed scope must hold. Exempt advisory writes, unfenced audit records and prerequisite-only paths are counted and classified, not failed. Any unclassified write fails.
    • The ingest trust limit is taken from the last provisioning refresh before the fence. The read path compares v0.33.0 and main answers exactly.
  • scripts/compat/released-binary-drill.test.sh covers the drill's decision logic (154 cases, no network). Deliberately broken versions of each check fail it.
  • crates/ravel-compat-tests holds the bucket-fencing example and tests/certify_flags_teardown_pair.rs. The crate is excluded from the workspace; it enables the test-only format-drill window through its own dev-dependencies, the one place check-release-profile.sh allows it.
    • With an injected clock past certify's time gate, the test asserts that certify does not certify over a v0.33.0-shape teardown l0/commit pair, nor over an unrecorded teardown object.
    • It also asserts that the late-write pass marks a certificate contradicted when such a pair lands after it.
    • This replaces the drill's census row, which no run inside the executor ceiling can meet.
  • A compat-tests CI job runs the drill cases and the compat crate's fmt, clippy and tests on pull requests and in the merge queue. The drill test step moved here from doc-scripts.
  • ADR-4233 gets an amendment recording the audit class and the residual: a pre-baseline maintain process still runs audit retention with no version hold, and only the terminated-processes prerequisite controls it.

The drill's two runs and the owner decisions are on #4243. This PR does not run the drill; it is run by hand against a published release.

This is a workflow change, so it counts as done only when the Actions run on this PR is green.

Refs: #4243
Refs: #4233

Document scripts/compat/released-binary-drill.sh in scripts/README.md. The
drill and its scripts landed in the preceding commits on this branch. This
commit records the run against the published v0.33.0 assets on a local
RustFS process.

The drill exits 1 (DRILL FAIL). Two v0.33.0 write paths published into
fenced buckets after the grace horizon, which is the fence's hour plus 3 h
plus 60 s:

- The admission reconciler rewrites t/<h>/{m,l,s}/admission/<pid>.snapshot
  every 10 s, with no provisioning check. It did so in the warm bucket at
  22:02:33Z, past the 22:01:00Z horizon. A cold-restarted process that
  refused every ingest and query request did the same within 2 minutes of
  its own fence.
- At graceful shutdown, the ingest teardown bypass pass ("no trusted
  generation view") wrote a metrics l0 object and its commit record at
  22:02:39Z. The point it held was acknowledged before 21:00 and stayed
  buffered for an hour.

Reads agreed: v0.33.0 and main returned identical results for all 8
queries on the unfenced copy.

Refs: #4243
Refs: #4233
Changelog: none
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
ADR-4233 decision 4 was narrowed after the first drill run: the guarantee
is the attested prerequisite that every pre-baseline process is terminated
before compat activate, and the fence is defence in depth with a closed
list of exempt advisory key patterns. The drill now judges against that
rule.

EXEMPT_PATTERNS at the top of released-binary-drill.sh lists the only
classes exempted: t/<h>/<sig>/admission/<pid>.snapshot and
t/<h>/<sig>/maint/scrub/<shard>.cursor for m, l and s. Both are written by
v0.33.0 without a provisioning check (the scrub's check fails open), carry
no data, and a newer build reaps or overwrites them. Metric metadata, the
alert lease and memo, idempotency markers, data objects, commit records
and catalog parts are not exempt. A glob matches one path segment per "*".

- A fenced or alerts key outside the list landing after the horizon while
  the process runs, or after the fence from a restarted process, fails.
  Exempt keys are recorded with LastModified as "after" or "within".
- Writes at process stop are recorded. Each l0 data object must carry the
  trailer v0.33.0 writes (RSG1 7, RLG1 5 for logs and alerts, RSP1 5);
  a commit record is checked through its paired data object; any other
  key fails because no version rule covers it. drill_s3.py gains a
  ranged trailer read.
- The maintain process runs with --scrub-period 1h and the warm phase
  ends one 61 min rotation after the horizon. No scrub cursor write after
  the horizon fails the run, and the budget check includes that tail.

The test suite adds the shipped exempt list, an exempt key that passes, a
non-exempt key that fails, teardown writes at the wrong version, an
unpaired commit record and an unversioned key that fail. Five mutants of
the decision logic (a glob spanning "/", a magic-only trailer check,
unrecorded exempt writes, unpaired commit records accepted, no late
writes) each fail it.

The drill was not run in this change: the executor had no route to
github.com, so the signed v0.33.0 assets could not be fetched or verified.

Refs: #4243
Refs: #4233
Changelog: none
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
The released-binary fence drill now judges every key written or removed
under the tenant after the fence against the revised pre-registration on
#4243. Each post-fence event falls in exactly one class of
drill_post_fence_class, and each class count is a figure checked present
once: alerts-late, l0c-after-horizon, maintenance and unclassified must
be zero, the other classes are recorded. Further rows: no ingest 2xx
requested after the trust limit, no query 2xx after the fence, every
teardown data write and idem write at the trailer version v0.33.0 writes,
and every such write flagged by main's certify census or late-write pass.
The warm phase takes listings while it runs (warm-snap, and every 10 min
under all) so a key written and removed between two listings is seen.

crates/ravel-compat-tests is created as its own workspace, excluded from
the root, with the drill_census example that runs certify and
late_write_pass against the drill bucket.

released-binary-drill.test.sh covers each new function with a passing
and a failing case; 27 mutations of drill-lib.sh, at least two per
check, each make it fail.

Refs: #4243
Refs: #4233
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Per-query audit records written under u/l0/ and u/c/ get their own drill
class, audit-unfenced, recorded and not judged. u/l1, u/maint and audit
removals stay prerequisite-only.

The ingest trust limit of a warm v0.33.0 router is the hour of its last
provisioning refresh before the fence plus 2 h. The drill bounds that
refresh from below per router, from the last 2xx write that ended before
the fence began, minus the 60 s refresh interval and the 30 s request
bound, and judges each router's ingest against its own limit.

The certify census row and its drill_census_check are removed: certify's
time gate opens hours after the fence, so a deterministic test replaces
the row.

Refs: #4243
Refs: #4233
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Replaces the drill census example with a deterministic test against
MemoryStore: certify refuses to certify over a released RSEG 7 l0 and
commit pair, and the late-write pass marks a certificate contradicted when
the pair lands after it.

Refs: #4243
Refs: #4233
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Refs: #4243
Refs: #4233
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>
Refs: #4243
Refs: #4233
Signed-off-by: Panagiotis Moustafellos <pmoust@nofire.ai>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant