Repository navigation
Description
Make retained Ravel data safely upgradeable from a baseline release onward:
a new binary reads the supported older data and keeps emitting what the
deployed readers read until an operator activates the new write format;
activation is durable and fences processes that cannot read it; migration
converges old objects through the existing commit/catalog protocol and is
resumable under concurrency; and an old reader is retired only on
certification evidence that stays valid while writers and maintenance run.
Scope (owner, 2026-11-15): the contract starts at the release that ships
this epic and covers that release and later. No bridge is built for data
written in formats that earlier releases already retired (RSPAN v4, RLOG v4).
Stage 0
Skipped: this epic makes no performance claim. Its cost figures (requests
per certified object, census pages per shard) are stated in the ADR and
pinned by tests, not measured against a baseline.
ADR
docs/adrs/4233-retained-data-compatibility-contract.md (proposed; number is
this issue's).
Relationship to other issues
Supersedes the open tasks of #16 (T11 reader window flip, T6 rewrite-on-touch
for old parts, T10 mixed-version e2e): this epic replaces the single-version
pre-v1.0 policy of ADR-0531 with a contract that starts at the baseline
release.
Tasks
- T1 T1 (epic #4233): Fold refuses newer catalog parts and newer tenant config (H1, H11) #4235 (W1, medium): Fold refuses newer catalog parts and newer tenant config (H1, H11)
- T2 T2 (epic #4233): Maintenance hazard closures: audit version hold, cursor tags, migrate holds, strict listings #4236 (W1, medium): Maintenance hazard closures: audit version hold, cursor tags, migrate holds, strict listings
- T3 T3 (epic #4233): Compaction claim above the known version is never stolen (H8) #4237 (W1, low): Compaction claim above the known version is never stolen (H8)
- T4 T4 (epic #4233): Newer idempotency marker refuses instead of re-ingesting (H10) #4238 (W1, low): Newer idempotency marker refuses instead of re-ingesting (H10)
- T5 T5 (epic #4233): Alert evaluator refuses its tick on unreadable-version history (H3) #4239 (W1, medium): Alert evaluator refuses its tick on unreadable-version history (H3)
- T6 T6 (epic #4233): Format crates: typed nested-section versions, report unknown section kinds (H5, H6 reader side) #4240 (W1, low): Format crates: typed nested-section versions, report unknown section kinds (H5, H6 reader side)
- T7 T7 (epic #4233): sys/compat record, provisioning version 4 fence, write policy view #4241 (W2, high): sys/compat record, provisioning version 4 fence, write policy view
- T8 T8 (epic #4233): Versioned writers and the format-drill feature #4242 (W3, medium): Versioned writers and the format-drill feature
- T16 T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243 (W3, medium): Released-binary fence drill against v0.33.0 (kill criterion)
- T9 T9 (epic #4233): Ingest writers follow the activation #4244 (W4, high): Ingest writers follow the activation
- T10 T10 (epic #4233): Maintenance writers follow the activation; migrate at the active version; background migrate #4245 (W5, high): Maintenance writers follow the activation; migrate at the active version; background migrate
- T11 T11 (epic #4233): Server wiring: compat refresher, alert lease parking, born-fenced first touch, background migrate flag #4246 (W6, high): Server wiring: compat refresher, alert lease parking, born-fenced first touch, background migrate flag
- T12 T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 (W7, high): Certify, init census, activation fence walk, late-write pass
- T13 T13 (epic #4233): ravel-cli compat commands and the retired-reader startup gate #4248 (W8, high): ravel-cli compat commands and the retired-reader startup gate
- T14 T14 (epic #4233): Executable compatibility matrix, frozen corpus, end-to-end reachability #4249 (W9, high): Executable compatibility matrix, frozen corpus, end-to-end reachability
- T15 T15 (epic #4233): Operator procedure and the normative compatibility document #4250 (W9, low): Operator procedure and the normative compatibility document
- T17 T17 (epic #4233): maintenance hazard follow-ups (cursor past held bucket, re-encode holds, alert retention hold) #4284 (W5b, medium): Maintenance hazard follow-ups (after T10)
- T18 T18 (epic #4233): bound the blind compat view, fence provisioning under a stale view, kidx written version #4287 (W6b, high): Bound the blind compat view, fence provisioning under a stale view, kidx written version (after T10F; before compat activate)
- T19 T19 (epic #4233): install the ingest and maintenance write policies in ravel-server #4301 (W6c, high): Install the ingest and maintenance write policies in ravel-server; teardown flush under a policy (after T18; before compat activate)
- T20 T20 (epic #4233): migrate walk stall on a large held bucket, RLOG/RSPAN re-encode holds, hash-mismatch hold #4304 (W6d, medium): Migrate walk stall on a large held bucket, RLOG/RSPAN re-encode holds, hash-mismatch hold (after T17)
- T21 T21 (epic #4233): certify finalization waits for a late-write pass; strict tenant listing; shard name form #4344 (W7b, high): Certify finalization waits for a late-write pass; strict tenant listing; shard name form (after T12; before compat certify ships)
- T22 T22 (epic #4233): teardown flush obeys the write policy on every signal #4362 (W6e, high): Teardown flush obeys the write policy on every signal (release-gating)
- T24 Ingest: an acknowledged buffered row may stay unpublished for an hour after the trust view expires #4363 (W6f, high): Bound the untrusted-view hold of acknowledged rows (drop and count) and stop acknowledging past the horizon (owner: release-gating; after T22)
Ledger
Stage 1 - design (2026-11-15T13:40Z)
base: origin/main 90e7643; worktree store-wt-upgrade
ADR-4233 drafted (uncommitted, worktree); awaiting owner approval
scope (owner 2026-11-15): contract starts at the baseline release; no bridge for RSPAN 4 / RLOG 4
open at gate: adopt v0.24-v0.33 stores in place (recommended) or refuse
approval (2026-11-15T14:35Z): owner delegated the gate to an adversarial Fable review; rounds: 1 BLOCK (5 major, 5 minor), 2 BLOCK (2 major, 5 minor), 3 BLOCK (1 major, 5 minor), 4 APPROVE (3 minor, 2 fixed in text, 1 no change); all findings fixed in the ADR
adoption decision: recommendation (adopt in place via compat init) taken as approved with the ADR
ADR committed a0676df on docs/adr-4233-compat-contract; PR #4234 opened, @claude-fleet review requested
Stage 2 - wave plan (2026-11-15T14:38Z)
W1: T1 #4235, T2 #4236, T3 #4237, T4 #4238, T5 #4239, T6 #4240
W2: T7 #4241
W3: T8 #4242, T16 #4243
W4: T9 #4244
W5: T10 #4245
W6: T11 #4246
W7: T12 #4247
W8: T13 #4248
W9: T14 #4249, T15 #4250
rules: no two tasks in one crate per wave; high-risk tasks solo (W2, W4-W8); W9 pairs T14 (new crate) with T15 (docs only)
T16 runs locally (docker + published images); its result is the kill criterion for ADR-4233 decision 4
Wave 1 - dispatched (2026-11-15T15:20Z, from main 90e7643)
plan: T1 #4235, T2 #4236, T3 #4237, T4 #4238, T5 #4239, T6 #4240
- T1 (epic #4233): Fold refuses newer catalog parts and newer tenant config (H1, H11) #4235 task=ad64d3e1-a6f1-4e23-b9e3-fc92c71a7979 dispatched tier=executor floor=60
- T2 (epic #4233): Maintenance hazard closures: audit version hold, cursor tags, migrate holds, strict listings #4236 task=311b4466-41c8-48ee-b85d-1017945de984 dispatched tier=executor floor=60
- T3 (epic #4233): Compaction claim above the known version is never stolen (H8) #4237 task=a5c7741d-9e3d-490b-97f0-fca0eb309e87 dispatched tier=executor floor=60
- T4 (epic #4233): Newer idempotency marker refuses instead of re-ingesting (H10) #4238 task=f5590378-f34b-4980-ac62-a691a3ac2f60 dispatched tier=executor floor=60
- T5 (epic #4233): Alert evaluator refuses its tick on unreadable-version history (H3) #4239 task=00a328fa-eab6-4557-af15-7f3f7a3e42dd dispatched tier=executor floor=60
- T6 (epic #4233): Format crates: typed nested-section versions, report unknown section kinds (H5, H6 reader side) #4240 task=5cd8bf52-d063-49ae-9d73-bd60ce9b91c8 dispatched tier=executor floor=60
scope (2026-11-15): owner put everything before v0.33 out of scope; ADR second commit 824b3a7; T16 #4243 narrowed to v0.33.0
T1 dispatched with --paths fold.rs:L4100-L6200 after the guard named open PR #4231 (its fold.rs hunks are at 6268+ and 6915-7010)
T3 #4237 task=a5c7741d done result=28109c70e (2026-11-15T15:34Z)
T4 #4238 task=f5590378 done result=514dfeff6 (2026-11-15T15:41Z); open: server compile against MarkerLookupError.source, counter render in ravel-server metrics, server-side no-write test, consistency-model doc; fix at landing
T2 #4236 task=311b4466 done result=ac9990185 (2026-11-15T15:46Z); open: maintenance.md and observability.md doc updates (held buckets, UnaddressableKey, audit hold), ravel-cli MigrateHeld report and audit held count, reencode path still fails on newer part, reported retention.rs bucket_is_empty_but_tombstone unaddressable-key gap
T1 #4235 task=ad64d3e1 below-floor (ci-16gb-fsn1-1, 42 of 60 GB), nothing written (2026-11-15T16:01Z); redispatching once with host=ravel-clickbench-ubuntu-2
- T1 (epic #4233): Fold refuses newer catalog parts and newer tenant config (H1, H11) #4235 task=7492115e-c0cd-4b07-9a02-5621abc5804b dispatched tier=executor floor=60
T1 #4235 task=7492115e below-floor again (2026-11-15T16:05Z): host=ravel-clickbench-ubuntu-2 preference fell back to ci-16gb-fsn1-1 (42 GB) while that box's slots ran T5/T6; T1 blocked; plan: redispatch with the host label once T5 or T6 frees a slot there; spec floor corrected to 40 (dry-run lists no lane run) but held at the big box for link headroom
T6 #4240 task=5cd8bf52 done result=b9148bf07 (2026-11-15T16:48Z); open: ravel-cli reconstruct.rs test LogFooter literal missing unknown_section_kinds (E0063), fix at landing
- T1 (epic #4233): Fold refuses newer catalog parts and newer tenant config (H1, H11) #4235 task=a76212d8-f545-4b1b-bce7-35cbbe395058 dispatched tier=executor floor=40
ADR PR #4234 review round 1 (2026-11-15T17:21Z): 5 should_fix + 7 nits, all fixed in 008a7ef, replied, review re-requested
T1 #4235 task=a76212d8 done result=1884c7ba7 from c9575d8 (2026-11-15T17:53Z); reported: covering_postings.rs:323 scrubber skips a newer .npost silently; other tenant-config read errors still fold under defaults (existing decision, not reversed); no new CatalogError variants (exhaustive matches outside scope)
ADR PR #4234 review round 2 (2026-11-15T18:21Z): 1 should_fix (fenced prov record unrewritable by baseline) + 3 nits fixed in 1b167cf; T7 #4241 and T14 #4249 updated; review re-requested (round 3 is the last)
- T2 (epic #4233): Maintenance hazard closures: audit version hold, cursor tags, migrate holds, strict listings #4236 task=a7b784ee-8e3e-47ca-9d63-47c52948ae0c dispatched tier=executor floor=60
- T4 (epic #4233): Newer idempotency marker refuses instead of re-ingesting (H10) #4238 task=7aa9cb21-c4ba-4cba-a93b-0bd944d8349b dispatched tier=executor floor=60
- T6 (epic #4233): Format crates: typed nested-section versions, report unknown section kinds (H5, H6 reader side) #4240 task=eac4d49a-2964-4392-9501-7bbc4adaaf3c dispatched tier=executor floor=60
wave 1 fix rounds dispatched (2026-11-15T19:03Z): T2F task=a7b784ee from ac99901, T4F task=7aa9cb21 from 514dfef, T6F task=eac4d49a from b9148bf; filed #4260 (retention tombstone over an unaddressable-only bucket)
T6F task=eac4d49a below-floor (ci-16gb-fsn1-1, 36 of 60 GB) (2026-11-15T19:07Z); one-line fixture fix made locally instead (no build), branch land/t6-format-sections 153b914; CI compiles it
T5 #4239 task=00a328fa done result=805134c3a (2026-11-15T19:09Z) on aarch64 8 GB: all-features ravel-server integration binaries not linked (OOM), lib lane 1599 green, clippy all-targets per lane; open: ADR-2708 D6 still describes the skip, amend at landing
re-plan (2026-11-15T19:19Z, owner speed-ups): W1 T1-T6 (landing); W2 T7 (from main, fold part moved to T11), T8 (stacked on land/t6-format-sections); W3 T9, T10, T11, T16; W4 T12 then T13 (depends_on); W5 T14, T15. Each high-risk task gets its own Opus checkpoint; code PRs merge on CI + checkpoint, no fleet review rounds
- T7 (epic #4233): sys/compat record, provisioning version 4 fence, write policy view #4241 task=e5895fb9-6268-4854-878b-282745a63907 dispatched tier=executor floor=60
- T8 (epic #4233): Versioned writers and the format-drill feature #4242 task=f12d88c1-a77f-46cb-b38c-8fea21594f2a dispatched tier=executor floor=60
Wave 2 - dispatched (2026-11-15T19:30Z): T7 #4241 task=e5895fb9 from main 19108bb; T8 #4242 task=f12d88c1 from land/t6-format-sections 153b914
T2F #4236 task=a7b784ee done result=13a42d0 on ac99901 (2026-11-15T19:45Z); gates -p ravel-maintain -p ravel-cli EXIT=0
T4F #4238 task=7aa9cb21 done result=d71f64a9b on 514dfef (2026-11-15T20:04Z); gates passed in pieces, not one gates.sh run; 503/UNAVAILABLE; counter rendered
- Epic: retained-data compatibility contract (install, activate, migrate, retire) #4233 task=70a4116b-f665-40a0-964a-ea696c66fdc4 dispatched tier=script-only floor=10
Wave 1 - review (2026-11-15T20:05Z): checkpoint CP1 task=70a4116b (opus, read-only) over w1 results t1 1884c7b, t2 13a42d0, t3 28109c7, t4 d71f64a, t5 805134c, t6 153b914 against main
T7 #4241 task=e5895fb9 ended mid-gate (2026-11-15T20:17Z) on rp2: backgrounded its test build and ended its turn; result b368812 (3 wip commits, implementation present, no gate result); continuation T7C spec ready, dispatching when a ClickBench slot frees
- T7 (epic #4233): sys/compat record, provisioning version 4 fence, write policy view #4241 task=6d5959ad-315a-4a54-a580-9c24117e5bde dispatched tier=executor floor=60
review CP1 (2026-11-15T20:37Z): t1 block (unread parts still replaced; main 19108bb made kidx v3 known), t3 block (ravel-server test asserts steal; changelog; ADR-1029), t5 block (older unreadable history freezes alerting; retention remedy false); t2, t4, t6 pass with should_fix items routed to T10/T12/T15; ADR-4233 PR #4234 merged fba7e66; T7C task=6d5959ad from b368812
- T1 (epic #4233): Fold refuses newer catalog parts and newer tenant config (H1, H11) #4235 task=80769a87-bec7-4c38-a495-af822488cbcd dispatched tier=executor floor=60
wave 1 landing (2026-11-15T20:42Z): PRs #4264 (T2, 75cae38), #4265 (T4, 4baa6a3), #4266 (T6, d3ac8b6) opened on main fba7e66, merge on CI green; blocked t1/t3/t5 fix rounds T1F, T5F, T3F dispatching to host=ravel-fleet-amd64-1; T6 should_fix filed #4263
- T5 (epic #4233): Alert evaluator refuses its tick on unreadable-version history (H3) #4239 task=9e5ada63-af3b-4c0c-b58b-0d0c9c5abcce dispatched tier=executor floor=60
- T3 (epic #4233): Compaction claim above the known version is never stolen (H8) #4237 task=ed3f398a-1aec-40d1-aa18-11c8d6e5cb31 dispatched tier=executor floor=60
T7C #4241 task=6d5959ad below-floor (2026-11-15T20:42Z): fell back to ci-16gb-fsn1-1 (26 GB of 60); held until T8 frees a ClickBench slot; fsn1 free space falling 42 -> 36 -> 26 GB across the session
T3F #4237 task=ed3f398a below-floor on fsn1 (26 GB) (2026-11-15T20:46Z); T1F task=80769a87 and T5F task=9e5ada63 running (ravel-fleet-amd64-1 or ClickBench); queue for next free good slot: T7C then T3F; PR #4266 all-features failed (cargo exit 101), log pending run completion
landing (2026-11-15T21:19Z): #4264 green 45p/0f, enqueued; #4266 closed (landed only b9148bf, the changelog-only head commit of the T6 result), replaced by #4268 (efb333c, full 17-file result + fixture); #4265 rerun pending (promql-difftest deadline test, unrelated)
landing (2026-11-15T21:30Z): #4265 rerun green (flake confirmed, signature 6c3908449567d6e3 not repeated), 46p/0f, enqueued
- T7 (epic #4233): sys/compat record, provisioning version 4 fence, write policy view #4241 task=07fde44e-ded6-4f89-88f0-a8bc98c7ad3b dispatched tier=executor floor=60
T8 #4242 task=f12d88c1 done result=cf27ecefe on 153b914 (2026-11-15T21:50Z); gates -p segment,logseg,rspan,cli EXIT=0 (1826/1826), drill lanes EXIT=0, default bytes pinned at e7557ea; T7C redispatched task=07fde44e from b368812
landed (2026-11-15T21:58Z): #4264 merged 5261469 (T2 #4236)
landing (2026-11-15T22:02Z): #4268 green 38p/0f incl. all-features, enqueued
landed (2026-11-15T22:06Z): #4265 merged 5745822 (T4 #4238)
T1F #4235 task=80769a87 done result=c45196af7 on fba7e66 (2026-11-15T22:38Z), gate green; deviation: peek tolerant of store errors (checkpoint to rule); T8 landing branch land/4242-write-versions adc2e35 (+changelog fragment, +format-bump guard fix for format-drill twins)
- T1 (epic #4233): Fold refuses newer catalog parts and newer tenant config (H1, H11) #4235 task=74f84a85-29d7-42b6-bb97-8664c4614573 dispatched tier=script-only floor=10
- T8 (epic #4233): Versioned writers and the format-drill feature #4242 task=ac06899f-6b4a-4f25-a7b4-cf51376a9dd9 dispatched tier=script-only floor=10
- T3 (epic #4233): Compaction claim above the known version is never stolen (H8) #4237 task=7d298f47-474d-4b60-8218-04b3478c6d7f dispatched tier=executor floor=60
checkpoints dispatched (2026-11-15T22:39Z): CP-T1F task=74f84a85, CP-T8 task=ac06899f (read-only, opus); T3F redispatched task=7d298f47 from 28109c7; #4268 merged ffe3057 (T6 #4240)
T3F #4237 task=7d298f47 below-floor again (2026-11-15T22:42Z): ci-16gb-fsn1-1 at 18 GB of 60; held until T5F or T7C frees a good slot
CP-T1F (2026-11-15T22:44Z): block, 3 blockers (tolerant peek loses objects: sweep rule 5; read-path GET errors still degrade; rebuilt fold skips leaf peek) + 3 should_fix; second fix round T1F2 dispatching
- T1 (epic #4233): Fold refuses newer catalog parts and newer tenant config (H1, H11) #4235 task=4e1cbdfd-5d0c-495f-83bb-332c1fdfa371 dispatched tier=executor floor=40
CP-T8 pass (2026-11-15T22:46Z); guard scope + doc fixed on land/4242-write-versions 3bd23fe; other items routed to #4244/#4245/#4249; PR #4273 opened. T1F2 task=4e1cbdfd dispatched from c45196a
- T1 (epic #4233): Fold refuses newer catalog parts and newer tenant config (H1, H11) #4235 task=291f7ba4-f33f-41bc-b830-d8f78cfe7d97 dispatched tier=script-only floor=10
- T3 (epic #4233): Compaction claim above the known version is never stolen (H8) #4237 task=0da148ee-e68f-4718-a388-76d196a606ca dispatched tier=executor floor=60
T1F2 #4235 task=4e1cbdfd done result=80385bd on c45196a (2026-11-15T23:26Z), gates green; re-review CP-T1F2 task=291f7ba4; T3F retried task=0da148ee
landing (2026-11-15T23:28Z): #4273 (T8) green 46p/0f, enqueued
T7C #4241 task=07fde44e below-floor (2026-11-15T23:31Z): fell back to ci-16gb-fsn1-1 at 10 GB after the host label's grace; owner notified (fsn1 42 -> 10 GB since 15:00); T7C held for a confirmed-free large slot
CP-T1F2 pass (2026-11-15T23:35Z); doc fixes 4dde443; follow-ups #4275; PR #4276 opened (T1 #4235, land/4235-fold-refuses-newer)
#4276 CI (2026-11-15T23:45Z): ravel-cli fold_human_report_prints_every_fold_report_field failed (newer_part_peek_gets not in text report); fixed in 6b618a8
landed (2026-11-16T00:01Z): #4273 merged be285a7 (T8 #4242)
- T7 (epic #4233): sys/compat record, provisioning version 4 fence, write policy view #4241 task=de5ef765-19eb-4dde-a227-eb38ab41981e dispatched tier=executor floor=60
T3F #4237 task=0da148ee ended mid-gate (2026-11-16T00:07Z): backgrounded its gate (sonnet); result edba847 rebased on main, 3 commits; continuation T3C (opus); T7C redispatched task=de5ef765 to ClickBench
- T3 (epic #4233): Compaction claim above the known version is never stolen (H8) #4237 task=17b94c0d-f304-4fa5-8f86-67fe21a73c5d dispatched tier=executor floor=60
landing (2026-11-16T00:17Z): #4276 (T1) green 45p/0f, enqueued
T5F #4239 task=9e5ada63 failed on final push (2026-11-16T00:46Z); rescue bundle head = checkpoint a80f4f1 (nothing lost); gate killed by signal 9 on rp2; landing branch land/4239-alert-newer-history, PR CI as the gate, checkpoint CP-T5F
- T5 (epic #4233): Alert evaluator refuses its tick on unreadable-version history (H3) #4239 task=e9a9300a-140c-415a-856e-f86e3b7847f6 dispatched tier=script-only floor=10
CP-T5F pass (2026-11-16T00:53Z); rule description + memo test fixed in beb865e on #4277; H3 wording nit routed to #4250
landed (2026-11-16T00:53Z): #4276 merged 8e41292 (T1 #4235)
- T3 (epic #4233): Compaction claim above the known version is never stolen (H8) #4237 task=870eea38-c3d1-4bcd-9955-dae9cd937317 dispatched tier=script-only floor=10
T3C #4237 task=17b94c0d done result=470d1558d (2026-11-16T01:01Z), gate green; landing branch land/4237-claims-never-stolen f04e195 (AI trailer stripped from an executor message); re-review CP-T3 dispatched; #4276 merged 8e41292
#4277 CI (2026-11-16T01:03Z): dashboard-coverage test failed (rule description named the refusal counter); fixed in c73e95c
CP-T3 pass (2026-11-16T01:08Z); recheck liveness fix 9af70fa + docs f0cf8c1; nits #4278; T3 PR opened
landing (2026-11-16T01:40Z): #4279 (T3) green 45p/0f, enqueued
landing (2026-11-16T01:41Z): #4277 (T5) green 45p/0f, enqueued
landed (2026-11-16T02:23Z): #4279 merged 05dc749 (T3 #4237)
Wave 1 - landed (2026-11-16T02:23Z): T1 #4276 8e41292, T2 #4264 5261469, T3 #4279 05dc749, T4 #4265 5745822, T5 #4277 42bf6b8, T6 #4268 ffe3057; wave 2: T8 #4273 be285a7 landed, T7 continuation T7C running; closed #4235-#4240 #4242
T7C #4241 task=de5ef765 running ~3h (2026-11-16T03:04Z); checkpoint 57494c1 (00:40Z, one test wip); transcript 401; likely in its ravel-server-closure gate; 4h ceiling ~04:07Z, continue from checkpoint if killed
T7C #4241 task=de5ef765 still running at 04:35Z (~4.5h since dispatch, past the usual 4h ceiling); no change since checkpoint 57494c1 at 00:40Z; one more window, then cancel and continue from the checkpoint
T7C #4241 task=de5ef765 cancelled at 05:05Z (~5h, no change since checkpoint 57494c1); continuation from its result ref if one is pushed, else from the checkpoint
- T7 (epic #4233): sys/compat record, provisioning version 4 fence, write policy view #4241 task=73683d27-f24e-42c1-ae00-c35fee1304e6 dispatched tier=script-only floor=10
T7 landing (2026-11-16T05:12Z): folded 4 wip commits onto main 09d339d as d752995 (2 textual conflicts resolved: provisioning.rs variants beside T1's, classification comment beside T3's; 0 ProvisioningRecord literals missing fields); checkpoint CP-T7 task=73683d27; PR opened, CI as gate
CP-T7 pass (2026-11-16T05:19Z); adoption fence-time fix + doc fixes in 707b861 on #4283. RELEASE HOLD: no release cut until T9/T10/T11 wire sys/compat (CP-T7 should_fix 4)
re-plan (2026-11-16T05:20Z): T10 split; maintenance hazard follow-ups moved to T17 #4284 (after T10, same crate). Wave 3 (T9, T10, T11) to stack on land/4241-compat-record before T7 merges; T11 gates ravel-server lanes in PR CI (fleet cannot finish them)
- T9 (epic #4233): Ingest writers follow the activation #4244 task=2c3d20ad-d04f-4379-a371-6014a6f8eafc dispatched tier=executor floor=60
- T10 (epic #4233): Maintenance writers follow the activation; migrate at the active version; background migrate #4245 task=ad8072c2-360b-43bc-9614-4ab078ebb0b9 dispatched tier=executor floor=60
- T11 (epic #4233): Server wiring: compat refresher, alert lease parking, born-fenced first touch, background migrate flag #4246 task=6ef4020d-c984-4c00-95dc-7d3f1cb8ed52 dispatched tier=executor floor=60
Wave 3 - dispatched (2026-11-16T05:38Z), stacked on land/4241-compat-record 707b861: T9 #4244 task=2c3d20ad, T10 #4245 task=ad8072c2, T11 #4246 (--fix: the duplicate guard named #4283, the PR it is stacked on)
landing (2026-11-16T06:02Z): #4283 (T7) green 46p/0f, enqueued
- T11 (epic #4233): Server wiring: compat refresher, alert lease parking, born-fenced first touch, background migrate flag #4246 task=455bed6c-a10a-4ef5-847d-abfe52b55863 dispatched tier=script-only floor=10
T11 #4246 task=6ef4020d done result=cdb6355bc on 707b861 (2026-11-16T06:30Z); deviations: sys/compat keyed by tenant-hash key (unkeyed store: no view), compat init must record catalog families at current versions (noted on #4247/#4248); checkpoint CP-T11 dispatched
Wave 2 - landed (2026-11-16T06:39Z): T7 #4283 de80c40, T8 #4273 be285a7; closed #4241 #4242. RELEASE HOLD stands until T9/T10/T11 land
CP-T11 block (2026-11-16T06:40Z): IAM templates lack sys/compat GetObject (refresh AccessDenied -> ViewStale forever on an un-inited store); kidx/catalog-head need a ceiling not an exact version; + 3 should_fix; fix round T11F dispatching
- T11 (epic #4233): Server wiring: compat refresher, alert lease parking, born-fenced first touch, background migrate flag #4246 task=85557574-a604-4939-a27e-01532ccbfbc0 dispatched tier=executor floor=60
- T10 (epic #4233): Maintenance writers follow the activation; migrate at the active version; background migrate #4245 task=632061a7-3ee3-4e42-81f6-6eff25e09ee7 dispatched tier=script-only floor=10
T10 #4245 task=ad8072c2 done result=f75968366 on 707b861 (2026-11-16T06:46Z); executor gate 963/963; checkpoint CP-T10 dispatched
CP-T10 block (2026-11-16T06:56Z): floor/rewrite/force2 checks compare against CURRENT not the run version (breaks first activation in a two-version build); force-2 arm aborts on unknown section; before_put copies view freshness; H6 hold mislabelled; query/reshard audit unwired; compaction-record family missing. Fix round T10F dispatching
- T10 (epic #4233): Maintenance writers follow the activation; migrate at the active version; background migrate #4245 task=6289b478-493c-4a8f-80b7-826162d5d6f6 dispatched tier=executor floor=60
- T11 (epic #4233): Server wiring: compat refresher, alert lease parking, born-fenced first touch, background migrate flag #4246 task=4496982b-1ddf-4735-8fcc-98793ddd1466 dispatched tier=script-only floor=10
T11F task=85557574 done result=bc335cf7b (2026-11-16T07:32Z); landed as PR #4286 (land/4246-server-compat f6052ec, tree equal to result); re-review CP-T11F dispatched; PR CI is the gate for feature lanes
CP-T11F pass, no blocker (2026-11-16T07:41Z): should_fix items (blind view unbounded on any store error, provisioning born-fenced check ignores staleness, kidx metrics written version, docs) filed as T18 #4287, after T10F, before compat activate ships. RELEASE HOLD now also covers T18. #4286 merges on CI
- T9 (epic #4233): Ingest writers follow the activation #4244 task=60410f91-9f3d-4f7d-a9f4-603d5883a9a1 dispatched tier=executor floor=60
T9 #4244 task=2c3d20ad failed (2026-11-16T07:42Z): final push exit 128 mid-gate, no report; rescue bundle unreachable (Tailscale SSH check); recovery continuation T9R dispatched from checkpoint aeaa55d (6 wip commits)
coordination (2026-11-16T07:55Z): #4260 taken standalone by session chatgpt-recommendation (retention.rs bucket_is_empty_but_tombstone only); message it when T10F (#4245) merges
coordination (2026-11-16T08:05Z): #4260 dispatched by chatgpt-recommendation as task 6bf496c4 (retention.rs, server metrics.rs near held_by_lease, one line each in observability.md and maintenance.md); overlaps #4286, T10F and T18 docs files in other regions, rebase on land
landing (2026-11-16T08:15Z): #4286 (T11) green 46p/0f, CP-T11F pass, enqueued
- T9 (epic #4233): Ingest writers follow the activation #4244 task=3ca97b82-94da-47e0-a5c3-ce7c4e8b4642 dispatched tier=script-only floor=10
- T10 (epic #4233): Maintenance writers follow the activation; migrate at the active version; background migrate #4245 task=ee0d12a4-6b97-4791-8bbf-e200bebc830e dispatched tier=script-only floor=10
T9R task=60410f91 done result=9bb7df31b on de80c40; T10F task=6289b478 done result=099f06d28 on de80c40 (2026-11-16T08:20Z); T10F conflicts with #4286 in compat.rs check_fresh (needs bucket epoch): resolve in a landing task after #4286 merges; CP-T9 and CP-T10F dispatched
CP-T10F pass (2026-11-16T08:28Z): no blocker; should_fix (force-2 freshness/target, rewrite_and_publish ceiling, floor default-build test, docs metric) + check_fresh(now, bucket_epoch) conflict ruling go to landing round T10L, dispatched once #4286 lands; metric render to T18, drill lane to T14
CP-T9 block (2026-11-16T08:29Z): check_attempt copies view freshness (breaks every flush on a no-record store once #4246 lands); admission/flush-open floor uses compile-time version; logs/spans tests thin. Fix round T9F ready, dispatches once #4286 lands (with T10L)
landed (2026-11-16T08:55Z): T11 #4286 6a5361f, closed #4246. Dispatching T9F and T10L on it
- T9 (epic #4233): Ingest writers follow the activation #4244 task=86e2c796-25c5-4bd1-a6a0-16690a8b5220 dispatched tier=executor floor=60
- T10 (epic #4233): Maintenance writers follow the activation; migrate at the active version; background migrate #4245 task=620e8b56-ed17-4f17-8ba5-4140b42ca4c5 dispatched tier=executor floor=60
coordination (2026-11-16T09:48Z): #4260 is PR #4293 (chatgpt-recommendation): retention.rs, server metrics.rs counter, docs; no T10F files. #4286 already merged (6a5361f)
T10L task=620e8b56 done result=e3ae1e3db on 6a5361f (2026-11-16T10:08Z); check_fresh(now, bucket_epoch) resolution matches the CP-T10F ruling; landed as land/4245-maintain-compat aaafcfe (range diff equal to result); merges on CI
landing (2026-11-16T10:43Z): #4295 (T10) green 45p/0f, enqueued
landed (2026-11-16T11:30Z): T10 #4295 691c7d6, closed #4245. Dispatching T17 #4284 and T18 #4287 from main
- T17 (epic #4233): maintenance hazard follow-ups (cursor past held bucket, re-encode holds, alert retention hold) #4284 task=843bff0b-88f4-41cc-8f5a-0d9900c637b3 dispatched tier=executor floor=60
- T18 (epic #4233): bound the blind compat view, fence provisioning under a stale view, kidx written version #4287 task=a0ccd2a9-02f5-4723-b733-31485a5b0027 dispatched tier=executor floor=60
- T9 (epic #4233): Ingest writers follow the activation #4244 task=d6ea187a-524c-477b-81b7-802b94c200f2 dispatched tier=script-only floor=10
T9F task=86e2c796 done result=950f4a8af on b6646b0 (2026-11-16T11:42Z); landed as land/4244-ingest-compat 3a5c93b (range diff equal); re-review CP-T9F dispatched alongside CI
CP-T9F pass (2026-11-16T11:48Z): no blocker; should_fix (dropped-router teardown flush refused no_bucket_view, latent until a policy is installed) routed to T19 #4301. GAP FOUND: nothing in ravel-server installs the ingest or maintenance policy yet (only the catalog) -> T19 wires both, after T18. RELEASE HOLD now covers T18 and T19. #4300 merges on CI
landing (2026-11-16T12:36Z): #4300 (T9) green 45p/0f, CP-T9F pass, enqueued
- T17 (epic #4233): maintenance hazard follow-ups (cursor past held bucket, re-encode holds, alert retention hold) #4284 task=7909bb8d-70ba-4a0d-abbc-1e9e1c8782fa dispatched tier=script-only floor=10
T17 #4284 task=843bff0b done result=991b406b1 (2026-11-16T12:54Z); landed as land/4284-maintain-hazards def3d2f; checkpoint CP-T17 dispatched alongside CI
- T18 (epic #4233): bound the blind compat view, fence provisioning under a stale view, kidx written version #4287 task=eb5c6813-ed6b-4f12-8583-c0d5d3ef0e66 dispatched tier=script-only floor=10
CP-T17 pass (2026-11-16T13:19Z): should_fix (walk stall on a held bucket that exhausts the budget, RLOG/RSPAN re-encode classification, hash-mismatch aborts walk, section-version labels) -> T20 #4304; #4302 merges on CI. T18 #4287 task=a0ccd2a9 done result=1fa7ed7f7; landed as one commit land/4287-blind-view 3cec386, PR #4303; CP-T18 dispatched
- T18 (epic #4233): bound the blind compat view, fence provisioning under a stale view, kidx written version #4287 task=c2a2e7d5-cb40-4fc9-a280-a3c0bf30e585 dispatched tier=executor floor=60
CP-T18 block (2026-11-16T13:25Z): sticky non-AccessDenied refresh mark ends the blind exemption for the process lifetime on a never-activated store without a grant (one 5xx = outage); first-touch refusal also hits existing buckets. Fix round T18F dispatched (latest-outcome rule, refuse only on create). Activation must refuse while any writer role cannot read sys/compat -> #4247/#4248
Wave 3 - landed (2026-11-16T13:32Z): T9 #4300 3c7cdbb, T10 #4295 691c7d6, T11 #4286 6a5361f; closed #4244 #4245 #4246. RELEASE HOLD stands: T18 (#4303 + T18F) and T19 (server installs ingest/maintain policies) still open
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=fe10cc4d-7980-438e-8f61-a48c71e1ea84 dispatched tier=executor floor=60
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=c5f26b92-728e-452e-a7fd-9b7b2c368000 dispatched tier=executor floor=60
Wave 4 - dispatched (2026-11-16T13:38Z) from main a63aff6: T12 #4247 split in two: T12A init+activate task=fe10cc4d, T12B certify+late writes+H5 retention probe task=c5f26b92 (--fix, second task on the ticket). T20 #4304 waits for T12 (same crate). T13 #4248 after T12
landing (2026-11-16T13:47Z): #4302 (T17) green 45p/0f, CP-T17 pass, enqueued
- T18 (epic #4233): bound the blind compat view, fence provisioning under a stale view, kidx written version #4287 task=e3ff5faa-8f11-4422-bf4f-1dfe3d2d247e dispatched tier=script-only floor=10
T18F task=c2a2e7d5 done result=d8950a749 (2026-11-16T14:10Z); appended to #4303 as b20050b (latest-outcome rule, CheckOnly first touch); re-review CP-T18F dispatched alongside CI
CP-T18F pass (2026-11-16T14:15Z): should_fix (NotFound-then-AccessDenied safety test, outcome ordering ticket, server doc + HELP strings) -> T19 #4301; ADR decision-3 amendment for the exemption -> T15 #4250. #4303 merges on CI
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=9fea8f98-53f0-4858-ae7f-2a7e355b75c7 dispatched tier=executor floor=60
T12A task=fe10cc4d done-without-gate (2026-11-16T14:29Z): backgrounded its gate and ended; result 44d693c (2 wip). Continuation T12AC dispatched from it
landed (2026-11-16T14:31Z): T17 #4302 8d719e7, closed #4284
- T18 (epic #4233): bound the blind compat view, fence provisioning under a stale view, kidx written version #4287 task=14199318-ad76-491d-a791-8c08084191a3 dispatched tier=executor floor=60
#4303 CI red (2026-11-16T14:55Z): real semantic conflict with T9 (landed meanwhile): ingest test models a missing grant with FaultStore Permanent, not StoreError::AccessDenied, so T18's latest-outcome rule refuses it. Fix T18G dispatched: rebase on main + test fix; will reopen as a new PR (no force-push of #4303's published head)
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=c41b06bd-6c24-49d9-a7f2-0540f2822d5f dispatched tier=script-only floor=10
T12AC task=9fea8f98 done result=7ae3d3a22 (orphan root commit; applied as a diff against its base a63aff6) (2026-11-16T15:49Z); PR land/4247-compat-activate; CP-T12A dispatched alongside CI
T18G task=14199318 done (2026-11-16T15:51Z): rebased + ingest test fix; #4303 closed, replaced by #4311 (land/4287-blind-view-v2 a2b951a, range diff equal to result). Merges on CI
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=48683189-793f-49f1-b303-66ac629726b4 dispatched tier=executor floor=60
CP-T12A block (2026-11-16T15:56Z): lease-finding failure cursor skips unparked tenants then sets fence_complete (real); kidx=2 'blocker' is main-without-#4311 (verified: no production fold writes leaf v3; #4311 sets fold_write_version kidx=2), so #4311 must land before #4310. Fix round T12AF dispatched (append to #4310, no rebase)
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=3d0ba60b-e6d4-45de-8917-5b92b689c25a dispatched tier=script-only floor=10
T12B task=c5f26b92 done result=019c9d6be on a63aff6 (2026-11-16T16:03Z); conflicts with main in alert_retention.rs (T17 added a trailer hold) and will collide with T12A on changelog.d/4247.added.md: land via a rebase task after T12A lands; CP-T12B dispatched on the result now
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=d2baa25d-34c3-4994-81f2-c137a7686e9b dispatched tier=executor floor=60
CP-T12B block (2026-11-16T16:11Z): 4 blockers in certify (replacement CAS keeps max watermark; resume by index skips shifted units and reads fenced=0 as born fenced; cursor not keyed by target/epoch; trailer not checked against stamp) + 8 should_fix (RSPAN H5, legal-hold data objects, P/max_flush_lifetime source, ...). Fix round T12BF dispatched incl. rebase on main (alert_retention merge with T17, fragment 4247-2)
landing (2026-11-16T16:26Z): #4311 (T18) green 45p/0f, CP-T18F pass, enqueued
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=6d17892c-5d7e-44b1-be70-eec0118eaec5 dispatched tier=script-only floor=10
T12AF task=48683189 done result=0c6fd9da1 (2026-11-16T16:39Z); appended to #4310 as ae752ad; re-review CP-T12AF dispatched alongside CI. #4310 lands after #4311
CP-T12AF pass (2026-11-16T16:45Z): resume paths verified; should_fix activation-stamp race (pre-existing, CAS-side fix) -> T20 #4304 (now ravel-maintain + ravel-catalog). #4310 merges on CI after #4311
- T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243 task=e82451c4-2cc6-4e0f-abde-fa65aaea9f76 dispatched tier=executor floor=60
release coordination (2026-11-16T16:52Z): v0.34.0 release issue #4317 (store-3e runs it). Posted: #4247 dropped from the hold (unreachable without #4248); go waits on #4311, #4301 (T19), #4243 (T16). T16 drill dispatched task=e82451c4 (fleet, docker preflight)
- T19 (epic #4233): install the ingest and maintenance write policies in ravel-server #4301 task=1d871ad5-eb30-4f2a-b17a-d7c51b208c37 dispatched tier=executor floor=60
(2026-11-16T17:45Z) landed T18 #4311 f087752, closed #4287. T19 #4301 dispatched. #4310 (T12A) green 46p/0f, CP-T12AF pass, enqueued. T16 drill task=e82451c4 BLOCKED: no docker access for fleet user on ravel-fleet-amd64-1
- T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243 task=a2213a3a-e6eb-492f-abf0-ce4284a85826 dispatched tier=executor floor=60
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=53dd5a0f-7016-4be7-ac63-25a1a60840a2 dispatched tier=script-only floor=10
(2026-11-16T17:46Z) T12BF task=d2baa25d done result=3fa9b5b23 on ac1c2bf (12 findings fixed, gate 1040 tests); re-review CP-T12BF dispatched; lands after #4310. T16 redispatched to host=pmoust-mbp (docker) task=a2213a3a
T16 BLOCKED twice (2026-11-16T17:50Z): tasks e82451c4 and a2213a3a both fell back to Linux executors (pimox5 the second time) whose fleet user is not in the docker group; host label is only a preference. Needs owner: docker access for the fleet user on one executor, or permission for one local build on the Mac (Docker Desktop present). Release #4317 waits on it
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=d6db7dc5-65b7-416a-9692-a039737f604a dispatched tier=executor floor=60
CP-T12BF block, round 2 of 3 (2026-11-16T17:55Z): certify censuses only shard 0 of a bucket with no provisioning record (12 earlier fixes confirmed); + certificate downgrade, floors raised before a refused CAS, untested unrecorded-object branch. Fix round T12BG dispatched
landed (2026-11-16T18:24Z): T12A init+activate #4310 1859892 (Refs #4247; #4247 stays open for certify T12B)
- T12 (epic #4233): Certify, init census, activation fence walk, late-write pass #4247 task=a0384103-169a-4dc2-96ec-d7da860d0a82 dispatched tier=script-only floor=10
T12BG task=d6db7dc5 done result=eeed0f714 (2026-11-16T18:45Z); landed by hand on main (lib.rs mod/re-export conflict with T12A, kept both; range diff equal) as PR #4342; round-3 review CP-T12BG dispatched alongside CI (last round)
CP-T12BG round 3 pass, no blocker (2026-11-16T18:53Z): 3 should_fix + 3 nits -> T21 #4344 (before compat certify ships). #4342 merges on CI
- T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243 task=82c4672b-83cc-4b83-a2f0-c33dcc7d1589 dispatched tier=executor floor=60
T16 owner decision delegated to Fable (2026-11-16T19:04Z): option D, no docker: v0.33.0 release binary (sha256 + cosign) + native RustFS 1.0.0 on an amd64 executor; warm path not faked if horizon exceeds the task ceiling. Dispatched T16D
landing (2026-11-16T19:20Z): #4342 (T12B certify) green 45p/0f, CP round 3 pass, enqueued
Wave 4 - landed (2026-11-16T20:05Z): T12 init/activate #4310 1859892, certify #4342 d5baf76; closed #4247. Dispatching T20 #4304 and T21 #4344
- T20 (epic #4233): migrate walk stall on a large held bucket, RLOG/RSPAN re-encode holds, hash-mismatch hold #4304 task=f8e9af75-4b1f-49b7-8750-6cbe3e6fd517 dispatched tier=executor floor=60
- T21 (epic #4233): certify finalization waits for a late-write pass; strict tenant listing; shard name form #4344 task=056aef00-8eca-4eea-9a1d-6a66d29bc8bd dispatched tier=executor floor=60
- T19 (epic #4233): install the ingest and maintenance write policies in ravel-server #4301 task=93153cc9-bad8-43e8-9af9-8f285f3e5e10 dispatched tier=script-only floor=10
T19 #4301 task=1d871ad5 done result=86935b8e8 (2026-11-16T20:19Z); landed as PR #4355 (land/4301-server-policies 125bc3e, range diff equal); CP-T19 dispatched alongside CI. T20 task=f8e9af75, T21 task=056aef00 dispatched from d5baf76
- T19 (epic #4233): install the ingest and maintenance write policies in ravel-server #4301 task=ad2bf5c1-9f2d-4537-8069-739f5e8fc870 dispatched tier=executor floor=60
CP-T19 block, round 1 (2026-11-16T20:29Z): on a never-activated store a 5xx on the first sys/compat read (or an outage > H after a NotFound read) admits writes then drops acknowledged rows at grant (ViewStale) - breaks decision 1. Fix round T19F: admission refuses ViewStale, grant requeues within max_flush_lifetime; + audit pipeline unwired, weak release-bar test, publish-lifetime recheck, docs. RELEASE waits
- T19 (epic #4233): install the ingest and maintenance write policies in ravel-server #4301 task=f6e274df-0ff0-4f9f-9557-642416d9aa24 dispatched tier=executor floor=60
T19F task=ad2bf5c1 below-floor on ci-16gb-fsn1-1 (49 of 60 GB) (2026-11-16T20:32Z); redispatched with host=ravel-fleet-amd64-1
T19F second below-floor (2026-11-16T20:35Z): task f6e274df also fell back to ci-16gb-fsn1-1 (49 of 60 GB); amd64-1 and clickbench-2 are busy with T16D/T20/T21. Holding T19F until one of those hosts frees, then redispatching there (not lowering the floor)
- T19 (epic #4233): install the ingest and maintenance write policies in ravel-server #4301 task=90430ecc-46b0-44e7-966a-255cdab0cf1d dispatched tier=executor floor=60
- T21 (epic #4233): certify finalization waits for a late-write pass; strict tenant listing; shard name form #4344 task=29fc0d60-4622-4859-b013-b01cdac7fcb7 dispatched tier=script-only floor=10
T21 #4344 task=056aef00 done result=6a5659752 (2026-11-16T21:33Z); PR opened (land/4344-certify-finalize); CP-T21 dispatched. T19F third dispatch task=90430ecc with label arch=arm64 (T19 itself passed its gate on a Pi)
CP-T21 pass (2026-11-16T21:37Z): FinalizationPending never clears without a scheduled late-write pass, and background migrate skips floored buckets while contradicted -> T13 #4248 scope. #4358 merges on CI
landing (2026-11-16T22:05Z): #4358 (T21) green 45p/0f, CP-T21 pass, enqueued
T16D DRILL FAIL (2026-11-16T22:08Z), task=82c4672b result=33c23dfec: (1) v0.33.0 admission reconciler rewrites t//{m,l,s}/admission/.snapshot every 10 s into fenced buckets, cold and warm, unbounded; (2) graceful-shutdown teardown bypass published an l0 data object + commit record 62 min after its last ack, past the source-derived horizon (inside ADR G_pre, but tied to process stop, not time). Alerts lease park held; reads identical. ADR-4233 decision 4 kill criterion fired; release #4317 held. Remedy goes to Fable (ADR approval delegated)
- T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243 task=1ae41e8d-7a30-4b94-a91b-83cd84833ffc dispatched tier=executor floor=60
Decision 4 reopened, Fable ruling (2026-11-16T22:13Z): prerequisite (pre-baseline processes terminated, attested, mandatory in compat activate, recorded in activated_by) is the guarantee; fence = defence in depth over admission, folds, queries, alert lease, with a closed exempt list of advisory key patterns. Pre-registered rerun on #4243; T16E dispatched (enumerate v0.33 ungated writes, narrowed verdict, rerun past a scrub rotation). T22 #4362 (main teardown flush obeys policy) filed, release-gating. Reported: #4363 (ack-before-durable inference), #4364 (scrub span false positive). ADR amendment follows T16E's enumeration
- T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243 task=8f0c4cdf-584b-40f8-ab94-c530f94c63b7 dispatched tier=executor floor=35
T16E below-floor on ci-16gb-fsn1-1 (45 of 60) (2026-11-16T22:15Z); floor lowered to 35 GB for the drill (it builds one release binary; the first run finished with 39 GB free); redispatched
- T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243 task=c673e34f-411f-43cd-8203-d20d7b3e800f dispatched tier=executor floor=35
T16E2 landed on rp1 (aarch64 8 GB), preflight refused (2026-11-16T22:18Z); redispatched with label arch=amd64
landed (2026-11-16T22:46Z): T21 #4358 8b512a1, closed #4344
T16E3 task=c673e34f (2026-11-16T22:52Z): executor had no GitHub egress, drill not run; verdict narrowing committed (result 6329c7a, 79 test cases). Enumeration of ungated v0.33.0 writers posted on #4243 (meta, teardown+idem, audit, alert retention without lease, enc, CLI paths). Asked Fable for the rerun rule and exempt list; rerun to go to ci-16gb-fsn1-1 (has egress; first run ran there)
- T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243 task=5cda3a15-e240-4a5a-aed7-ac48d8dbc745 dispatched tier=executor floor=35
- T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243 task=0f1cc36a-d3fa-416f-bac2-83783ea16632 dispatched tier=executor floor=35
Fable ruling 2 (2026-11-16T22:54Z): rerun fails only on the fence's narrowed scope plus unclassified or unflagged data-bearing writes; exempt list = admission snapshots + scrub cursors; prerequisite-only classes recorded; rerun still required once (scrub rotation + graceful stop). Revised pre-registration posted on #4243. T16F dispatched (first try 5cda3a15 cancelled: spec assumed ravel-compat-tests existed), now with that crate created excluded
- T19 (epic #4233): install the ingest and maintenance write policies in ravel-server #4301 task=a7a86e1f-fa29-448e-9e43-b319beb6c84b dispatched tier=script-only floor=10
T19F task=90430ecc done result=eac4f6605 (2026-11-17T00:51Z); appended to #4355 as 39c08e3 (admission refuses stale view; grant and PUT attempts wait within max_flush_lifetime; audit pipeline policy; stronger release-bar test). Re-review CP-T19F (round 2) dispatched alongside CI. T22 #4362 can dispatch once #4355 lands
CP-T19F round 2 pass (2026-11-17T00:57Z): no blocker; should_fix (503 for up to 60 s after sys/compat recovers, refresh only on the ticker; grant-time lifetime re-check drops acknowledged rows when compat init runs against a live fleet) + nits -> T22 #4362. #4355 merges on CI
- T20 (epic #4233): migrate walk stall on a large held bucket, RLOG/RSPAN re-encode holds, hash-mismatch hold #4304 task=8f14f263-dee7-4812-b768-b1a033147e8c dispatched tier=script-only floor=10
T20 #4304 task=f8e9af75 done result=0b2bb5071 (2026-11-17T01:08Z); folded into one commit, PR opened (land/4304-maintain-followups 3739fd49e); CP-T20 dispatched alongside CI
- T20 (epic #4233): migrate walk stall on a large held bucket, RLOG/RSPAN re-encode holds, hash-mismatch hold #4304 task=6f0230b6-7d98-42cf-bdf2-70630e97da68 dispatched tier=executor floor=60
CP-T20 pass with 3 should_fix (2026-11-17T01:12Z): logs/spans re-encode classify a corrupt trailer as out-of-window (no hash check), hash-mismatch hold keeps the claim, changelog overclaims. Fix round T20F appends to #4368 before merge
landing (2026-11-17T01:23Z): #4355 (T19) green 46p/0f, CP-T19F round 2 pass, enqueued
T20F task=6f0230b6 done result=d70e4a5e6 (2026-11-17T01:53Z); appended to #4368 (logs/spans hash check on version refusals; hash-mismatch claim completes; changelog corrected). Merges on CI
landed (2026-11-17T01:56Z): T19 #4355 0b3878f, closed #4301 (release hold item). Dispatching T22 #4362
- T22 (epic #4233): teardown flush obeys the write policy on every signal #4362 task=d53d8484-a3e1-4638-9340-f1dbe386ea82 dispatched tier=executor floor=60
landing (2026-11-17T02:25Z): #4368 (T20) green 45p/0f, CP-T20 pass + fix round, enqueued
T16F drill run 2 (2026-11-17T02:27Z), task=0f1cc36a result=efd638a50: DRILL FAIL on 2 pre-registered rows: census-warm (main certify TimeGate ~6h06m after fence, cannot flag inside a 4h run: spec premise error) and warm-observation (tail 1319 s vs >= 3660 s, stopped early for the ceiling). Every fence-scope row passed warm and cold (0 ingest 2xx after trust limit, 0 queries, 0 alert writes, 0 non-teardown l0/c, 0 maintenance, 0 unclassified; reads identical; teardown data at RSEG 7). Asked Fable whether this supports the go
- Ingest: an acknowledged buffered row may stay unpublished for an hour after the trust view expires #4363 task=f996ee96-cb0f-4001-97bc-2f5bc1a56ddc dispatched tier=script-only floor=10
- T16 (epic #4233): Released-binary fence drill against v0.33.0 (kill criterion) #4243 task=0d529a32-2876-422c-9cac-a5d8d8c5b15d dispatched tier=executor floor=35
Fable ruling 3 (2026-11-17T02:29Z): run 2 supports narrowed decision 4; no third run. Verdict recorded on #4243 as FAIL as pre-registered with fence-scope rows passing; warm tail waived; census row replaced by a deterministic test in ravel-compat-tests plus its CI lane (release-blocking); u/l0,u/c class 'audit: unfenced by design'. Dispatched T16G (finish drill: classes, trust formula, census test, CI lane) and T23 (#4363 read-only: can main ack a row unpublished until shutdown? release-blocking if yes)
T23 #4363 (2026-11-17T02:35Z): MAIN AFFECTED yes: an acked buffered row whose flush opens after the generation trust horizon is held on ScanCheck::Unknown with no bound and written only by the teardown bypass (same on v0.33.0 and main; pre-existing, ADR-1642). Investigation posted on #4363; fix design asked of Fable
(2026-11-17T02:37Z) #4368 (T20) removed from merge queue: failed_checks, tla-models (lifecycle) and tla failed in merge-group run 44002944000 (logs pending run completion). Fable on #4363: drop-and-count bound + admission tightening recommended, not a v0.34.0 blocker if documented; owner confirms drop vs keep (amends ADR-1642). Docs known-issue PR #4371 opened
OWNER (2026-11-17T06:20Z): #4363 fix = drop and count at max_flush_lifetime (+ admission tightening, stacked); fix BEFORE v0.34.0 (added to the #4317 hold). Docs PR #4371 closed as superseded. T24 dispatches after T22 (same ingest files)
(2026-11-17T06:22Z) #4368 queue failure was a CI runner fault: TLC 'Error when writing the disk (StatePoolWriter.run)' in tla-models lifecycle negative window-no-cache-delay (exit 1 not 12; invariant violated as expected). Re-enqueued
T16G task=0d529a32 done (2026-11-17T06:22Z): drill classes, trust formula, deterministic certify test (4 mutations caught), compat-tests CI job, ADR audit-residual amendment; landed as land/4243-fence-drill (7 commits, range equal). PR opened; workflow change, needs green Actions run
- T22 (epic #4233): teardown flush obeys the write policy on every signal #4362 task=ecb8f937-c882-402e-aeea-51890465144f dispatched tier=script-only floor=10
T22 #4362 task=d53d8484 done result=8f5a6b984 (2026-11-17T06:53Z): no teardown path bypassed the policy (tests added on all signals); #4355 round-2 items fixed (refresh wake on view_stale, lifetime check at admission only, final check at deadline). PR opened (land/4362-teardown-policy); CP-T22 alongside CI. T24 (#4363) dispatches after it lands
CP-T22 pass (2026-11-17T07:01Z): should_fix (deadline re-ask counts the drop under the wrong reason) folded into T24 spec; ADR decision 7 abandon-at-P now admission-only -> my ADR-4233 amendment. #4373 merges on CI
landing (2026-11-17T07:01Z): #4372 (drill, certify teardown test, compat-tests CI lane, audit residual amendment) green 48p/0f incl compat-tests job; enqueued
landed (2026-11-17T07:02Z): T20 #4368 2728256 (retry after the TLC disk flake), closed #4304. T13 #4248 now unblocked
ADR-4233 decision-4 narrowing amendment: Fable APPROVE (2026-11-17T07:08Z) after one round of 11 fixes; local commit 159f5fa34 on docs/4233-decision4-narrowed; lands only after #4373 (admission-only lifetime check) and #4372, rebased onto that main
landing (2026-11-17T07:27Z): #4373 (T22) green 45p/0f, CP-T22 pass, enqueued
Activity
docs: scope ADR-4233 existing-store support to v0.33 and later
pmoust commentedon Nov 15, 2026
dispatch-intent nonce=1794754826-56289 ticket=4235 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904
pmoust commentedon Nov 15, 2026
dispatch-record nonce=1794754826-56289 task=ad64d3e1-a6f1-4e23-b9e3-fc92c71a7979
pmoust commentedon Nov 15, 2026
dispatch-intent nonce=1794755068-79448 ticket=4236 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904
pmoust commentedon Nov 15, 2026
dispatch-record nonce=1794755068-79448 task=311b4466-41c8-48ee-b85d-1017945de984
pmoust commentedon Nov 15, 2026
dispatch-intent nonce=1794755291-92508 ticket=4237 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904
pmoust commentedon Nov 15, 2026
dispatch-record nonce=1794755291-92508 task=a5c7741d-9e3d-490b-97f0-fca0eb309e87
pmoust commentedon Nov 15, 2026
dispatch-intent nonce=1794755520-3908 ticket=4238 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904
pmoust commentedon Nov 15, 2026
dispatch-record nonce=1794755520-3908 task=f5590378-f34b-4980-ac62-a691a3ac2f60
pmoust commentedon Nov 15, 2026
dispatch-intent nonce=1794755740-13535 ticket=4239 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904
pmoust commentedon Nov 15, 2026
dispatch-record nonce=1794755740-13535 task=00a328fa-eab6-4557-af15-7f3f7a3e42dd
pmoust commentedon Nov 15, 2026
dispatch-intent nonce=1794755972-32238 ticket=4240 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904
pmoust commentedon Nov 15, 2026
dispatch-record nonce=1794755972-32238 task=5cd8bf52-d063-49ae-9d73-bd60ce9b91c8
wip: migrate hold and strict-listing tests
237 remaining items
wip: teardown flush tests for every signal under the compat policy
wip: refresh wake on view_stale and a last view check at the deadline
test(compat): judge the fence drill by the revised class table
wip: wake the compat refresh loop on view_stale; first touch honours …
pmoust commentedon Nov 17, 2026
dispatch-intent nonce=1794882558-41394 ticket=4363 ref=a8a3be09d5b6492a2b0eefca5f5086c41414cc79
pmoust commentedon Nov 17, 2026
dispatch-record nonce=1794882558-41394 task=f996ee96-cb0f-4001-97bc-2f5bc1a56ddc
pmoust commentedon Nov 17, 2026
dispatch-intent nonce=1794882572-41907 ticket=4243 ref=efd638a50c7b99e184b0f99d0fc3248d8ed20c5b
pmoust commentedon Nov 17, 2026
dispatch-record nonce=1794882572-41907 task=0d529a32-2876-422c-9cac-a5d8d8c5b15d
wip: docs and changelog for the teardown and wake behaviour
wip: test(compat): class audit records and bound trust by the last re…
wip: test(compat): certify flags a v0.33.0 teardown pair
wip: ci: run the compat crate and the fence drill cases as one job
wip: docs(adr): state the audit retention residual in ADR-4233
pmoust commentedon Nov 17, 2026
dispatch-intent nonce=1794898375-38841 ticket=4362 ref=a8a3be09d5b6492a2b0eefca5f5086c41414cc79
pmoust commentedon Nov 17, 2026
dispatch-record nonce=1794898375-38841 task=ecb8f937-c882-402e-aeea-51890465144f