Skip to content

Epic: retained-data compatibility contract (install, activate, migrate, retire) #4233

Open

Description

@pmoust

Make retained Ravel data safely upgradeable from a baseline release onward:
a new binary reads the supported older data and keeps emitting what the
deployed readers read until an operator activates the new write format;
activation is durable and fences processes that cannot read it; migration
converges old objects through the existing commit/catalog protocol and is
resumable under concurrency; and an old reader is retired only on
certification evidence that stays valid while writers and maintenance run.

Scope (owner, 2026-11-15): the contract starts at the release that ships
this epic and covers that release and later. No bridge is built for data
written in formats that earlier releases already retired (RSPAN v4, RLOG v4).

Stage 0

Skipped: this epic makes no performance claim. Its cost figures (requests
per certified object, census pages per shard) are stated in the ADR and
pinned by tests, not measured against a baseline.

ADR

docs/adrs/4233-retained-data-compatibility-contract.md (proposed; number is
this issue's).

Relationship to other issues

Supersedes the open tasks of #16 (T11 reader window flip, T6 rewrite-on-touch
for old parts, T10 mixed-version e2e): this epic replaces the single-version
pre-v1.0 policy of ADR-0531 with a contract that starts at the baseline
release.

Tasks

Ledger

Stage 1 - design (2026-11-15T13:40Z)

base: origin/main 90e7643; worktree store-wt-upgrade
ADR-4233 drafted (uncommitted, worktree); awaiting owner approval
scope (owner 2026-11-15): contract starts at the baseline release; no bridge for RSPAN 4 / RLOG 4
open at gate: adopt v0.24-v0.33 stores in place (recommended) or refuse
approval (2026-11-15T14:35Z): owner delegated the gate to an adversarial Fable review; rounds: 1 BLOCK (5 major, 5 minor), 2 BLOCK (2 major, 5 minor), 3 BLOCK (1 major, 5 minor), 4 APPROVE (3 minor, 2 fixed in text, 1 no change); all findings fixed in the ADR
adoption decision: recommendation (adopt in place via compat init) taken as approved with the ADR
ADR committed a0676df on docs/adr-4233-compat-contract; PR #4234 opened, @claude-fleet review requested

Stage 2 - wave plan (2026-11-15T14:38Z)

W1: T1 #4235, T2 #4236, T3 #4237, T4 #4238, T5 #4239, T6 #4240
W2: T7 #4241
W3: T8 #4242, T16 #4243
W4: T9 #4244
W5: T10 #4245
W6: T11 #4246
W7: T12 #4247
W8: T13 #4248
W9: T14 #4249, T15 #4250
rules: no two tasks in one crate per wave; high-risk tasks solo (W2, W4-W8); W9 pairs T14 (new crate) with T15 (docs only)
T16 runs locally (docker + published images); its result is the kill criterion for ADR-4233 decision 4

Wave 1 - dispatched (2026-11-15T15:20Z, from main 90e7643)

plan: T1 #4235, T2 #4236, T3 #4237, T4 #4238, T5 #4239, T6 #4240

scope (2026-11-15): owner put everything before v0.33 out of scope; ADR second commit 824b3a7; T16 #4243 narrowed to v0.33.0
T1 dispatched with --paths fold.rs:L4100-L6200 after the guard named open PR #4231 (its fold.rs hunks are at 6268+ and 6915-7010)

T3 #4237 task=a5c7741d done result=28109c70e (2026-11-15T15:34Z)

T4 #4238 task=f5590378 done result=514dfeff6 (2026-11-15T15:41Z); open: server compile against MarkerLookupError.source, counter render in ravel-server metrics, server-side no-write test, consistency-model doc; fix at landing

T2 #4236 task=311b4466 done result=ac9990185 (2026-11-15T15:46Z); open: maintenance.md and observability.md doc updates (held buckets, UnaddressableKey, audit hold), ravel-cli MigrateHeld report and audit held count, reencode path still fails on newer part, reported retention.rs bucket_is_empty_but_tombstone unaddressable-key gap

T1 #4235 task=ad64d3e1 below-floor (ci-16gb-fsn1-1, 42 of 60 GB), nothing written (2026-11-15T16:01Z); redispatching once with host=ravel-clickbench-ubuntu-2

T1 #4235 task=7492115e below-floor again (2026-11-15T16:05Z): host=ravel-clickbench-ubuntu-2 preference fell back to ci-16gb-fsn1-1 (42 GB) while that box's slots ran T5/T6; T1 blocked; plan: redispatch with the host label once T5 or T6 frees a slot there; spec floor corrected to 40 (dry-run lists no lane run) but held at the big box for link headroom

T6 #4240 task=5cd8bf52 done result=b9148bf07 (2026-11-15T16:48Z); open: ravel-cli reconstruct.rs test LogFooter literal missing unknown_section_kinds (E0063), fix at landing

ADR PR #4234 review round 1 (2026-11-15T17:21Z): 5 should_fix + 7 nits, all fixed in 008a7ef, replied, review re-requested

T1 #4235 task=a76212d8 done result=1884c7ba7 from c9575d8 (2026-11-15T17:53Z); reported: covering_postings.rs:323 scrubber skips a newer .npost silently; other tenant-config read errors still fold under defaults (existing decision, not reversed); no new CatalogError variants (exhaustive matches outside scope)

ADR PR #4234 review round 2 (2026-11-15T18:21Z): 1 should_fix (fenced prov record unrewritable by baseline) + 3 nits fixed in 1b167cf; T7 #4241 and T14 #4249 updated; review re-requested (round 3 is the last)

wave 1 fix rounds dispatched (2026-11-15T19:03Z): T2F task=a7b784ee from ac99901, T4F task=7aa9cb21 from 514dfef, T6F task=eac4d49a from b9148bf; filed #4260 (retention tombstone over an unaddressable-only bucket)

T6F task=eac4d49a below-floor (ci-16gb-fsn1-1, 36 of 60 GB) (2026-11-15T19:07Z); one-line fixture fix made locally instead (no build), branch land/t6-format-sections 153b914; CI compiles it

T5 #4239 task=00a328fa done result=805134c3a (2026-11-15T19:09Z) on aarch64 8 GB: all-features ravel-server integration binaries not linked (OOM), lib lane 1599 green, clippy all-targets per lane; open: ADR-2708 D6 still describes the skip, amend at landing

re-plan (2026-11-15T19:19Z, owner speed-ups): W1 T1-T6 (landing); W2 T7 (from main, fold part moved to T11), T8 (stacked on land/t6-format-sections); W3 T9, T10, T11, T16; W4 T12 then T13 (depends_on); W5 T14, T15. Each high-risk task gets its own Opus checkpoint; code PRs merge on CI + checkpoint, no fleet review rounds

Wave 2 - dispatched (2026-11-15T19:30Z): T7 #4241 task=e5895fb9 from main 19108bb; T8 #4242 task=f12d88c1 from land/t6-format-sections 153b914

T2F #4236 task=a7b784ee done result=13a42d0 on ac99901 (2026-11-15T19:45Z); gates -p ravel-maintain -p ravel-cli EXIT=0

T4F #4238 task=7aa9cb21 done result=d71f64a9b on 514dfef (2026-11-15T20:04Z); gates passed in pieces, not one gates.sh run; 503/UNAVAILABLE; counter rendered

Wave 1 - review (2026-11-15T20:05Z): checkpoint CP1 task=70a4116b (opus, read-only) over w1 results t1 1884c7b, t2 13a42d0, t3 28109c7, t4 d71f64a, t5 805134c, t6 153b914 against main

T7 #4241 task=e5895fb9 ended mid-gate (2026-11-15T20:17Z) on rp2: backgrounded its test build and ended its turn; result b368812 (3 wip commits, implementation present, no gate result); continuation T7C spec ready, dispatching when a ClickBench slot frees

review CP1 (2026-11-15T20:37Z): t1 block (unread parts still replaced; main 19108bb made kidx v3 known), t3 block (ravel-server test asserts steal; changelog; ADR-1029), t5 block (older unreadable history freezes alerting; retention remedy false); t2, t4, t6 pass with should_fix items routed to T10/T12/T15; ADR-4233 PR #4234 merged fba7e66; T7C task=6d5959ad from b368812

wave 1 landing (2026-11-15T20:42Z): PRs #4264 (T2, 75cae38), #4265 (T4, 4baa6a3), #4266 (T6, d3ac8b6) opened on main fba7e66, merge on CI green; blocked t1/t3/t5 fix rounds T1F, T5F, T3F dispatching to host=ravel-fleet-amd64-1; T6 should_fix filed #4263

T7C #4241 task=6d5959ad below-floor (2026-11-15T20:42Z): fell back to ci-16gb-fsn1-1 (26 GB of 60); held until T8 frees a ClickBench slot; fsn1 free space falling 42 -> 36 -> 26 GB across the session

T3F #4237 task=ed3f398a below-floor on fsn1 (26 GB) (2026-11-15T20:46Z); T1F task=80769a87 and T5F task=9e5ada63 running (ravel-fleet-amd64-1 or ClickBench); queue for next free good slot: T7C then T3F; PR #4266 all-features failed (cargo exit 101), log pending run completion

landing (2026-11-15T21:19Z): #4264 green 45p/0f, enqueued; #4266 closed (landed only b9148bf, the changelog-only head commit of the T6 result), replaced by #4268 (efb333c, full 17-file result + fixture); #4265 rerun pending (promql-difftest deadline test, unrelated)

landing (2026-11-15T21:30Z): #4265 rerun green (flake confirmed, signature 6c3908449567d6e3 not repeated), 46p/0f, enqueued

T8 #4242 task=f12d88c1 done result=cf27ecefe on 153b914 (2026-11-15T21:50Z); gates -p segment,logseg,rspan,cli EXIT=0 (1826/1826), drill lanes EXIT=0, default bytes pinned at e7557ea; T7C redispatched task=07fde44e from b368812

landed (2026-11-15T21:58Z): #4264 merged 5261469 (T2 #4236)

landing (2026-11-15T22:02Z): #4268 green 38p/0f incl. all-features, enqueued

landed (2026-11-15T22:06Z): #4265 merged 5745822 (T4 #4238)

T1F #4235 task=80769a87 done result=c45196af7 on fba7e66 (2026-11-15T22:38Z), gate green; deviation: peek tolerant of store errors (checkpoint to rule); T8 landing branch land/4242-write-versions adc2e35 (+changelog fragment, +format-bump guard fix for format-drill twins)

checkpoints dispatched (2026-11-15T22:39Z): CP-T1F task=74f84a85, CP-T8 task=ac06899f (read-only, opus); T3F redispatched task=7d298f47 from 28109c7; #4268 merged ffe3057 (T6 #4240)

T3F #4237 task=7d298f47 below-floor again (2026-11-15T22:42Z): ci-16gb-fsn1-1 at 18 GB of 60; held until T5F or T7C frees a good slot

CP-T1F (2026-11-15T22:44Z): block, 3 blockers (tolerant peek loses objects: sweep rule 5; read-path GET errors still degrade; rebuilt fold skips leaf peek) + 3 should_fix; second fix round T1F2 dispatching

CP-T8 pass (2026-11-15T22:46Z); guard scope + doc fixed on land/4242-write-versions 3bd23fe; other items routed to #4244/#4245/#4249; PR #4273 opened. T1F2 task=4e1cbdfd dispatched from c45196a

T1F2 #4235 task=4e1cbdfd done result=80385bd on c45196a (2026-11-15T23:26Z), gates green; re-review CP-T1F2 task=291f7ba4; T3F retried task=0da148ee

landing (2026-11-15T23:28Z): #4273 (T8) green 46p/0f, enqueued

T7C #4241 task=07fde44e below-floor (2026-11-15T23:31Z): fell back to ci-16gb-fsn1-1 at 10 GB after the host label's grace; owner notified (fsn1 42 -> 10 GB since 15:00); T7C held for a confirmed-free large slot

CP-T1F2 pass (2026-11-15T23:35Z); doc fixes 4dde443; follow-ups #4275; PR #4276 opened (T1 #4235, land/4235-fold-refuses-newer)

#4276 CI (2026-11-15T23:45Z): ravel-cli fold_human_report_prints_every_fold_report_field failed (newer_part_peek_gets not in text report); fixed in 6b618a8

landed (2026-11-16T00:01Z): #4273 merged be285a7 (T8 #4242)

T3F #4237 task=0da148ee ended mid-gate (2026-11-16T00:07Z): backgrounded its gate (sonnet); result edba847 rebased on main, 3 commits; continuation T3C (opus); T7C redispatched task=de5ef765 to ClickBench

landing (2026-11-16T00:17Z): #4276 (T1) green 45p/0f, enqueued

T5F #4239 task=9e5ada63 failed on final push (2026-11-16T00:46Z); rescue bundle head = checkpoint a80f4f1 (nothing lost); gate killed by signal 9 on rp2; landing branch land/4239-alert-newer-history, PR CI as the gate, checkpoint CP-T5F

CP-T5F pass (2026-11-16T00:53Z); rule description + memo test fixed in beb865e on #4277; H3 wording nit routed to #4250

landed (2026-11-16T00:53Z): #4276 merged 8e41292 (T1 #4235)

T3C #4237 task=17b94c0d done result=470d1558d (2026-11-16T01:01Z), gate green; landing branch land/4237-claims-never-stolen f04e195 (AI trailer stripped from an executor message); re-review CP-T3 dispatched; #4276 merged 8e41292

#4277 CI (2026-11-16T01:03Z): dashboard-coverage test failed (rule description named the refusal counter); fixed in c73e95c

CP-T3 pass (2026-11-16T01:08Z); recheck liveness fix 9af70fa + docs f0cf8c1; nits #4278; T3 PR opened

landing (2026-11-16T01:40Z): #4279 (T3) green 45p/0f, enqueued

landing (2026-11-16T01:41Z): #4277 (T5) green 45p/0f, enqueued

landed (2026-11-16T02:23Z): #4279 merged 05dc749 (T3 #4237)

Wave 1 - landed (2026-11-16T02:23Z): T1 #4276 8e41292, T2 #4264 5261469, T3 #4279 05dc749, T4 #4265 5745822, T5 #4277 42bf6b8, T6 #4268 ffe3057; wave 2: T8 #4273 be285a7 landed, T7 continuation T7C running; closed #4235-#4240 #4242

T7C #4241 task=de5ef765 running ~3h (2026-11-16T03:04Z); checkpoint 57494c1 (00:40Z, one test wip); transcript 401; likely in its ravel-server-closure gate; 4h ceiling ~04:07Z, continue from checkpoint if killed

T7C #4241 task=de5ef765 still running at 04:35Z (~4.5h since dispatch, past the usual 4h ceiling); no change since checkpoint 57494c1 at 00:40Z; one more window, then cancel and continue from the checkpoint

T7C #4241 task=de5ef765 cancelled at 05:05Z (~5h, no change since checkpoint 57494c1); continuation from its result ref if one is pushed, else from the checkpoint

T7 landing (2026-11-16T05:12Z): folded 4 wip commits onto main 09d339d as d752995 (2 textual conflicts resolved: provisioning.rs variants beside T1's, classification comment beside T3's; 0 ProvisioningRecord literals missing fields); checkpoint CP-T7 task=73683d27; PR opened, CI as gate

CP-T7 pass (2026-11-16T05:19Z); adoption fence-time fix + doc fixes in 707b861 on #4283. RELEASE HOLD: no release cut until T9/T10/T11 wire sys/compat (CP-T7 should_fix 4)

re-plan (2026-11-16T05:20Z): T10 split; maintenance hazard follow-ups moved to T17 #4284 (after T10, same crate). Wave 3 (T9, T10, T11) to stack on land/4241-compat-record before T7 merges; T11 gates ravel-server lanes in PR CI (fleet cannot finish them)

Wave 3 - dispatched (2026-11-16T05:38Z), stacked on land/4241-compat-record 707b861: T9 #4244 task=2c3d20ad, T10 #4245 task=ad8072c2, T11 #4246 (--fix: the duplicate guard named #4283, the PR it is stacked on)

landing (2026-11-16T06:02Z): #4283 (T7) green 46p/0f, enqueued

T11 #4246 task=6ef4020d done result=cdb6355bc on 707b861 (2026-11-16T06:30Z); deviations: sys/compat keyed by tenant-hash key (unkeyed store: no view), compat init must record catalog families at current versions (noted on #4247/#4248); checkpoint CP-T11 dispatched

Wave 2 - landed (2026-11-16T06:39Z): T7 #4283 de80c40, T8 #4273 be285a7; closed #4241 #4242. RELEASE HOLD stands until T9/T10/T11 land

CP-T11 block (2026-11-16T06:40Z): IAM templates lack sys/compat GetObject (refresh AccessDenied -> ViewStale forever on an un-inited store); kidx/catalog-head need a ceiling not an exact version; + 3 should_fix; fix round T11F dispatching

T10 #4245 task=ad8072c2 done result=f75968366 on 707b861 (2026-11-16T06:46Z); executor gate 963/963; checkpoint CP-T10 dispatched

CP-T10 block (2026-11-16T06:56Z): floor/rewrite/force2 checks compare against CURRENT not the run version (breaks first activation in a two-version build); force-2 arm aborts on unknown section; before_put copies view freshness; H6 hold mislabelled; query/reshard audit unwired; compaction-record family missing. Fix round T10F dispatching

T11F task=85557574 done result=bc335cf7b (2026-11-16T07:32Z); landed as PR #4286 (land/4246-server-compat f6052ec, tree equal to result); re-review CP-T11F dispatched; PR CI is the gate for feature lanes

CP-T11F pass, no blocker (2026-11-16T07:41Z): should_fix items (blind view unbounded on any store error, provisioning born-fenced check ignores staleness, kidx metrics written version, docs) filed as T18 #4287, after T10F, before compat activate ships. RELEASE HOLD now also covers T18. #4286 merges on CI

T9 #4244 task=2c3d20ad failed (2026-11-16T07:42Z): final push exit 128 mid-gate, no report; rescue bundle unreachable (Tailscale SSH check); recovery continuation T9R dispatched from checkpoint aeaa55d (6 wip commits)

coordination (2026-11-16T07:55Z): #4260 taken standalone by session chatgpt-recommendation (retention.rs bucket_is_empty_but_tombstone only); message it when T10F (#4245) merges

coordination (2026-11-16T08:05Z): #4260 dispatched by chatgpt-recommendation as task 6bf496c4 (retention.rs, server metrics.rs near held_by_lease, one line each in observability.md and maintenance.md); overlaps #4286, T10F and T18 docs files in other regions, rebase on land

landing (2026-11-16T08:15Z): #4286 (T11) green 46p/0f, CP-T11F pass, enqueued

T9R task=60410f91 done result=9bb7df31b on de80c40; T10F task=6289b478 done result=099f06d28 on de80c40 (2026-11-16T08:20Z); T10F conflicts with #4286 in compat.rs check_fresh (needs bucket epoch): resolve in a landing task after #4286 merges; CP-T9 and CP-T10F dispatched

CP-T10F pass (2026-11-16T08:28Z): no blocker; should_fix (force-2 freshness/target, rewrite_and_publish ceiling, floor default-build test, docs metric) + check_fresh(now, bucket_epoch) conflict ruling go to landing round T10L, dispatched once #4286 lands; metric render to T18, drill lane to T14

CP-T9 block (2026-11-16T08:29Z): check_attempt copies view freshness (breaks every flush on a no-record store once #4246 lands); admission/flush-open floor uses compile-time version; logs/spans tests thin. Fix round T9F ready, dispatches once #4286 lands (with T10L)

landed (2026-11-16T08:55Z): T11 #4286 6a5361f, closed #4246. Dispatching T9F and T10L on it

coordination (2026-11-16T09:48Z): #4260 is PR #4293 (chatgpt-recommendation): retention.rs, server metrics.rs counter, docs; no T10F files. #4286 already merged (6a5361f)

T10L task=620e8b56 done result=e3ae1e3db on 6a5361f (2026-11-16T10:08Z); check_fresh(now, bucket_epoch) resolution matches the CP-T10F ruling; landed as land/4245-maintain-compat aaafcfe (range diff equal to result); merges on CI

landing (2026-11-16T10:43Z): #4295 (T10) green 45p/0f, enqueued

landed (2026-11-16T11:30Z): T10 #4295 691c7d6, closed #4245. Dispatching T17 #4284 and T18 #4287 from main

T9F task=86e2c796 done result=950f4a8af on b6646b0 (2026-11-16T11:42Z); landed as land/4244-ingest-compat 3a5c93b (range diff equal); re-review CP-T9F dispatched alongside CI

CP-T9F pass (2026-11-16T11:48Z): no blocker; should_fix (dropped-router teardown flush refused no_bucket_view, latent until a policy is installed) routed to T19 #4301. GAP FOUND: nothing in ravel-server installs the ingest or maintenance policy yet (only the catalog) -> T19 wires both, after T18. RELEASE HOLD now covers T18 and T19. #4300 merges on CI

landing (2026-11-16T12:36Z): #4300 (T9) green 45p/0f, CP-T9F pass, enqueued

T17 #4284 task=843bff0b done result=991b406b1 (2026-11-16T12:54Z); landed as land/4284-maintain-hazards def3d2f; checkpoint CP-T17 dispatched alongside CI

CP-T17 pass (2026-11-16T13:19Z): should_fix (walk stall on a held bucket that exhausts the budget, RLOG/RSPAN re-encode classification, hash-mismatch aborts walk, section-version labels) -> T20 #4304; #4302 merges on CI. T18 #4287 task=a0ccd2a9 done result=1fa7ed7f7; landed as one commit land/4287-blind-view 3cec386, PR #4303; CP-T18 dispatched

CP-T18 block (2026-11-16T13:25Z): sticky non-AccessDenied refresh mark ends the blind exemption for the process lifetime on a never-activated store without a grant (one 5xx = outage); first-touch refusal also hits existing buckets. Fix round T18F dispatched (latest-outcome rule, refuse only on create). Activation must refuse while any writer role cannot read sys/compat -> #4247/#4248

Wave 3 - landed (2026-11-16T13:32Z): T9 #4300 3c7cdbb, T10 #4295 691c7d6, T11 #4286 6a5361f; closed #4244 #4245 #4246. RELEASE HOLD stands: T18 (#4303 + T18F) and T19 (server installs ingest/maintain policies) still open

Wave 4 - dispatched (2026-11-16T13:38Z) from main a63aff6: T12 #4247 split in two: T12A init+activate task=fe10cc4d, T12B certify+late writes+H5 retention probe task=c5f26b92 (--fix, second task on the ticket). T20 #4304 waits for T12 (same crate). T13 #4248 after T12

landing (2026-11-16T13:47Z): #4302 (T17) green 45p/0f, CP-T17 pass, enqueued

T18F task=c2a2e7d5 done result=d8950a749 (2026-11-16T14:10Z); appended to #4303 as b20050b (latest-outcome rule, CheckOnly first touch); re-review CP-T18F dispatched alongside CI

CP-T18F pass (2026-11-16T14:15Z): should_fix (NotFound-then-AccessDenied safety test, outcome ordering ticket, server doc + HELP strings) -> T19 #4301; ADR decision-3 amendment for the exemption -> T15 #4250. #4303 merges on CI

T12A task=fe10cc4d done-without-gate (2026-11-16T14:29Z): backgrounded its gate and ended; result 44d693c (2 wip). Continuation T12AC dispatched from it

landed (2026-11-16T14:31Z): T17 #4302 8d719e7, closed #4284

#4303 CI red (2026-11-16T14:55Z): real semantic conflict with T9 (landed meanwhile): ingest test models a missing grant with FaultStore Permanent, not StoreError::AccessDenied, so T18's latest-outcome rule refuses it. Fix T18G dispatched: rebase on main + test fix; will reopen as a new PR (no force-push of #4303's published head)

T12AC task=9fea8f98 done result=7ae3d3a22 (orphan root commit; applied as a diff against its base a63aff6) (2026-11-16T15:49Z); PR land/4247-compat-activate; CP-T12A dispatched alongside CI

T18G task=14199318 done (2026-11-16T15:51Z): rebased + ingest test fix; #4303 closed, replaced by #4311 (land/4287-blind-view-v2 a2b951a, range diff equal to result). Merges on CI

CP-T12A block (2026-11-16T15:56Z): lease-finding failure cursor skips unparked tenants then sets fence_complete (real); kidx=2 'blocker' is main-without-#4311 (verified: no production fold writes leaf v3; #4311 sets fold_write_version kidx=2), so #4311 must land before #4310. Fix round T12AF dispatched (append to #4310, no rebase)

T12B task=c5f26b92 done result=019c9d6be on a63aff6 (2026-11-16T16:03Z); conflicts with main in alert_retention.rs (T17 added a trailer hold) and will collide with T12A on changelog.d/4247.added.md: land via a rebase task after T12A lands; CP-T12B dispatched on the result now

CP-T12B block (2026-11-16T16:11Z): 4 blockers in certify (replacement CAS keeps max watermark; resume by index skips shifted units and reads fenced=0 as born fenced; cursor not keyed by target/epoch; trailer not checked against stamp) + 8 should_fix (RSPAN H5, legal-hold data objects, P/max_flush_lifetime source, ...). Fix round T12BF dispatched incl. rebase on main (alert_retention merge with T17, fragment 4247-2)

landing (2026-11-16T16:26Z): #4311 (T18) green 45p/0f, CP-T18F pass, enqueued

T12AF task=48683189 done result=0c6fd9da1 (2026-11-16T16:39Z); appended to #4310 as ae752ad; re-review CP-T12AF dispatched alongside CI. #4310 lands after #4311

CP-T12AF pass (2026-11-16T16:45Z): resume paths verified; should_fix activation-stamp race (pre-existing, CAS-side fix) -> T20 #4304 (now ravel-maintain + ravel-catalog). #4310 merges on CI after #4311

release coordination (2026-11-16T16:52Z): v0.34.0 release issue #4317 (store-3e runs it). Posted: #4247 dropped from the hold (unreachable without #4248); go waits on #4311, #4301 (T19), #4243 (T16). T16 drill dispatched task=e82451c4 (fleet, docker preflight)

(2026-11-16T17:45Z) landed T18 #4311 f087752, closed #4287. T19 #4301 dispatched. #4310 (T12A) green 46p/0f, CP-T12AF pass, enqueued. T16 drill task=e82451c4 BLOCKED: no docker access for fleet user on ravel-fleet-amd64-1

(2026-11-16T17:46Z) T12BF task=d2baa25d done result=3fa9b5b23 on ac1c2bf (12 findings fixed, gate 1040 tests); re-review CP-T12BF dispatched; lands after #4310. T16 redispatched to host=pmoust-mbp (docker) task=a2213a3a

T16 BLOCKED twice (2026-11-16T17:50Z): tasks e82451c4 and a2213a3a both fell back to Linux executors (pimox5 the second time) whose fleet user is not in the docker group; host label is only a preference. Needs owner: docker access for the fleet user on one executor, or permission for one local build on the Mac (Docker Desktop present). Release #4317 waits on it

CP-T12BF block, round 2 of 3 (2026-11-16T17:55Z): certify censuses only shard 0 of a bucket with no provisioning record (12 earlier fixes confirmed); + certificate downgrade, floors raised before a refused CAS, untested unrecorded-object branch. Fix round T12BG dispatched

landed (2026-11-16T18:24Z): T12A init+activate #4310 1859892 (Refs #4247; #4247 stays open for certify T12B)

T12BG task=d6db7dc5 done result=eeed0f714 (2026-11-16T18:45Z); landed by hand on main (lib.rs mod/re-export conflict with T12A, kept both; range diff equal) as PR #4342; round-3 review CP-T12BG dispatched alongside CI (last round)

CP-T12BG round 3 pass, no blocker (2026-11-16T18:53Z): 3 should_fix + 3 nits -> T21 #4344 (before compat certify ships). #4342 merges on CI

T16 owner decision delegated to Fable (2026-11-16T19:04Z): option D, no docker: v0.33.0 release binary (sha256 + cosign) + native RustFS 1.0.0 on an amd64 executor; warm path not faked if horizon exceeds the task ceiling. Dispatched T16D

landing (2026-11-16T19:20Z): #4342 (T12B certify) green 45p/0f, CP round 3 pass, enqueued

Wave 4 - landed (2026-11-16T20:05Z): T12 init/activate #4310 1859892, certify #4342 d5baf76; closed #4247. Dispatching T20 #4304 and T21 #4344

T19 #4301 task=1d871ad5 done result=86935b8e8 (2026-11-16T20:19Z); landed as PR #4355 (land/4301-server-policies 125bc3e, range diff equal); CP-T19 dispatched alongside CI. T20 task=f8e9af75, T21 task=056aef00 dispatched from d5baf76

CP-T19 block, round 1 (2026-11-16T20:29Z): on a never-activated store a 5xx on the first sys/compat read (or an outage > H after a NotFound read) admits writes then drops acknowledged rows at grant (ViewStale) - breaks decision 1. Fix round T19F: admission refuses ViewStale, grant requeues within max_flush_lifetime; + audit pipeline unwired, weak release-bar test, publish-lifetime recheck, docs. RELEASE waits

T19F task=ad2bf5c1 below-floor on ci-16gb-fsn1-1 (49 of 60 GB) (2026-11-16T20:32Z); redispatched with host=ravel-fleet-amd64-1

T19F second below-floor (2026-11-16T20:35Z): task f6e274df also fell back to ci-16gb-fsn1-1 (49 of 60 GB); amd64-1 and clickbench-2 are busy with T16D/T20/T21. Holding T19F until one of those hosts frees, then redispatching there (not lowering the floor)

T21 #4344 task=056aef00 done result=6a5659752 (2026-11-16T21:33Z); PR opened (land/4344-certify-finalize); CP-T21 dispatched. T19F third dispatch task=90430ecc with label arch=arm64 (T19 itself passed its gate on a Pi)

CP-T21 pass (2026-11-16T21:37Z): FinalizationPending never clears without a scheduled late-write pass, and background migrate skips floored buckets while contradicted -> T13 #4248 scope. #4358 merges on CI

landing (2026-11-16T22:05Z): #4358 (T21) green 45p/0f, CP-T21 pass, enqueued

T16D DRILL FAIL (2026-11-16T22:08Z), task=82c4672b result=33c23dfec: (1) v0.33.0 admission reconciler rewrites t//{m,l,s}/admission/.snapshot every 10 s into fenced buckets, cold and warm, unbounded; (2) graceful-shutdown teardown bypass published an l0 data object + commit record 62 min after its last ack, past the source-derived horizon (inside ADR G_pre, but tied to process stop, not time). Alerts lease park held; reads identical. ADR-4233 decision 4 kill criterion fired; release #4317 held. Remedy goes to Fable (ADR approval delegated)

Decision 4 reopened, Fable ruling (2026-11-16T22:13Z): prerequisite (pre-baseline processes terminated, attested, mandatory in compat activate, recorded in activated_by) is the guarantee; fence = defence in depth over admission, folds, queries, alert lease, with a closed exempt list of advisory key patterns. Pre-registered rerun on #4243; T16E dispatched (enumerate v0.33 ungated writes, narrowed verdict, rerun past a scrub rotation). T22 #4362 (main teardown flush obeys policy) filed, release-gating. Reported: #4363 (ack-before-durable inference), #4364 (scrub span false positive). ADR amendment follows T16E's enumeration

T16E below-floor on ci-16gb-fsn1-1 (45 of 60) (2026-11-16T22:15Z); floor lowered to 35 GB for the drill (it builds one release binary; the first run finished with 39 GB free); redispatched

T16E2 landed on rp1 (aarch64 8 GB), preflight refused (2026-11-16T22:18Z); redispatched with label arch=amd64

landed (2026-11-16T22:46Z): T21 #4358 8b512a1, closed #4344

T16E3 task=c673e34f (2026-11-16T22:52Z): executor had no GitHub egress, drill not run; verdict narrowing committed (result 6329c7a, 79 test cases). Enumeration of ungated v0.33.0 writers posted on #4243 (meta, teardown+idem, audit, alert retention without lease, enc, CLI paths). Asked Fable for the rerun rule and exempt list; rerun to go to ci-16gb-fsn1-1 (has egress; first run ran there)

Fable ruling 2 (2026-11-16T22:54Z): rerun fails only on the fence's narrowed scope plus unclassified or unflagged data-bearing writes; exempt list = admission snapshots + scrub cursors; prerequisite-only classes recorded; rerun still required once (scrub rotation + graceful stop). Revised pre-registration posted on #4243. T16F dispatched (first try 5cda3a15 cancelled: spec assumed ravel-compat-tests existed), now with that crate created excluded

T19F task=90430ecc done result=eac4f6605 (2026-11-17T00:51Z); appended to #4355 as 39c08e3 (admission refuses stale view; grant and PUT attempts wait within max_flush_lifetime; audit pipeline policy; stronger release-bar test). Re-review CP-T19F (round 2) dispatched alongside CI. T22 #4362 can dispatch once #4355 lands

CP-T19F round 2 pass (2026-11-17T00:57Z): no blocker; should_fix (503 for up to 60 s after sys/compat recovers, refresh only on the ticker; grant-time lifetime re-check drops acknowledged rows when compat init runs against a live fleet) + nits -> T22 #4362. #4355 merges on CI

T20 #4304 task=f8e9af75 done result=0b2bb5071 (2026-11-17T01:08Z); folded into one commit, PR opened (land/4304-maintain-followups 3739fd49e); CP-T20 dispatched alongside CI

CP-T20 pass with 3 should_fix (2026-11-17T01:12Z): logs/spans re-encode classify a corrupt trailer as out-of-window (no hash check), hash-mismatch hold keeps the claim, changelog overclaims. Fix round T20F appends to #4368 before merge

landing (2026-11-17T01:23Z): #4355 (T19) green 46p/0f, CP-T19F round 2 pass, enqueued

T20F task=6f0230b6 done result=d70e4a5e6 (2026-11-17T01:53Z); appended to #4368 (logs/spans hash check on version refusals; hash-mismatch claim completes; changelog corrected). Merges on CI

landed (2026-11-17T01:56Z): T19 #4355 0b3878f, closed #4301 (release hold item). Dispatching T22 #4362

landing (2026-11-17T02:25Z): #4368 (T20) green 45p/0f, CP-T20 pass + fix round, enqueued

T16F drill run 2 (2026-11-17T02:27Z), task=0f1cc36a result=efd638a50: DRILL FAIL on 2 pre-registered rows: census-warm (main certify TimeGate ~6h06m after fence, cannot flag inside a 4h run: spec premise error) and warm-observation (tail 1319 s vs >= 3660 s, stopped early for the ceiling). Every fence-scope row passed warm and cold (0 ingest 2xx after trust limit, 0 queries, 0 alert writes, 0 non-teardown l0/c, 0 maintenance, 0 unclassified; reads identical; teardown data at RSEG 7). Asked Fable whether this supports the go

Fable ruling 3 (2026-11-17T02:29Z): run 2 supports narrowed decision 4; no third run. Verdict recorded on #4243 as FAIL as pre-registered with fence-scope rows passing; warm tail waived; census row replaced by a deterministic test in ravel-compat-tests plus its CI lane (release-blocking); u/l0,u/c class 'audit: unfenced by design'. Dispatched T16G (finish drill: classes, trust formula, census test, CI lane) and T23 (#4363 read-only: can main ack a row unpublished until shutdown? release-blocking if yes)

T23 #4363 (2026-11-17T02:35Z): MAIN AFFECTED yes: an acked buffered row whose flush opens after the generation trust horizon is held on ScanCheck::Unknown with no bound and written only by the teardown bypass (same on v0.33.0 and main; pre-existing, ADR-1642). Investigation posted on #4363; fix design asked of Fable

(2026-11-17T02:37Z) #4368 (T20) removed from merge queue: failed_checks, tla-models (lifecycle) and tla failed in merge-group run 44002944000 (logs pending run completion). Fable on #4363: drop-and-count bound + admission tightening recommended, not a v0.34.0 blocker if documented; owner confirms drop vs keep (amends ADR-1642). Docs known-issue PR #4371 opened

OWNER (2026-11-17T06:20Z): #4363 fix = drop and count at max_flush_lifetime (+ admission tightening, stacked); fix BEFORE v0.34.0 (added to the #4317 hold). Docs PR #4371 closed as superseded. T24 dispatches after T22 (same ingest files)

(2026-11-17T06:22Z) #4368 queue failure was a CI runner fault: TLC 'Error when writing the disk (StatePoolWriter.run)' in tla-models lifecycle negative window-no-cache-delay (exit 1 not 12; invariant violated as expected). Re-enqueued

T16G task=0d529a32 done (2026-11-17T06:22Z): drill classes, trust formula, deterministic certify test (4 mutations caught), compat-tests CI job, ADR audit-residual amendment; landed as land/4243-fence-drill (7 commits, range equal). PR opened; workflow change, needs green Actions run

T22 #4362 task=d53d8484 done result=8f5a6b984 (2026-11-17T06:53Z): no teardown path bypassed the policy (tests added on all signals); #4355 round-2 items fixed (refresh wake on view_stale, lifetime check at admission only, final check at deadline). PR opened (land/4362-teardown-policy); CP-T22 alongside CI. T24 (#4363) dispatches after it lands

CP-T22 pass (2026-11-17T07:01Z): should_fix (deadline re-ask counts the drop under the wrong reason) folded into T24 spec; ADR decision 7 abandon-at-P now admission-only -> my ADR-4233 amendment. #4373 merges on CI

landing (2026-11-17T07:01Z): #4372 (drill, certify teardown test, compat-tests CI lane, audit residual amendment) green 48p/0f incl compat-tests job; enqueued

landed (2026-11-17T07:02Z): T20 #4368 2728256 (retry after the TLC disk flake), closed #4304. T13 #4248 now unblocked

ADR-4233 decision-4 narrowing amendment: Fable APPROVE (2026-11-17T07:08Z) after one round of 11 fixes; local commit 159f5fa34 on docs/4233-decision4-narrowed; lands only after #4373 (admission-only lifetime check) and #4372, rebased onto that main

landing (2026-11-17T07:27Z): #4373 (T22) green 45p/0f, CP-T22 pass, enqueued

Activity

  1. self-assigned this
    on Nov 15, 2026
  2. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-intent nonce=1794754826-56289 ticket=4235 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904

  3. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-record nonce=1794754826-56289 task=ad64d3e1-a6f1-4e23-b9e3-fc92c71a7979

  4. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-intent nonce=1794755068-79448 ticket=4236 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904

  5. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-record nonce=1794755068-79448 task=311b4466-41c8-48ee-b85d-1017945de984

  6. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-intent nonce=1794755291-92508 ticket=4237 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904

  7. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-record nonce=1794755291-92508 task=a5c7741d-9e3d-490b-97f0-fca0eb309e87

  8. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-intent nonce=1794755520-3908 ticket=4238 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904

  9. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-record nonce=1794755520-3908 task=f5590378-f34b-4980-ac62-a691a3ac2f60

  10. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-intent nonce=1794755740-13535 ticket=4239 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904

  11. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-record nonce=1794755740-13535 task=00a328fa-eab6-4557-af15-7f3f7a3e42dd

  12. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-intent nonce=1794755972-32238 ticket=4240 ref=90e7643c98ac952828fbf7573b9250a3ad4c4904

  13. pmoust commented on Nov 15, 2026

    @pmoust
    MemberAuthor

    dispatch-record nonce=1794755972-32238 task=5cd8bf52-d063-49ae-9d73-bd60ce9b91c8

  14. 237 remaining items

  15. pmoust commented on Nov 17, 2026

    @pmoust
    MemberAuthor

    dispatch-intent nonce=1794882558-41394 ticket=4363 ref=a8a3be09d5b6492a2b0eefca5f5086c41414cc79

  16. pmoust commented on Nov 17, 2026

    @pmoust
    MemberAuthor

    dispatch-record nonce=1794882558-41394 task=f996ee96-cb0f-4001-97bc-2f5bc1a56ddc

  17. pmoust commented on Nov 17, 2026

    @pmoust
    MemberAuthor

    dispatch-intent nonce=1794882572-41907 ticket=4243 ref=efd638a50c7b99e184b0f99d0fc3248d8ed20c5b

  18. pmoust commented on Nov 17, 2026

    @pmoust
    MemberAuthor

    dispatch-record nonce=1794882572-41907 task=0d529a32-2876-422c-9cac-a5d8d8c5b15d

  19. pmoust commented on Nov 17, 2026

    @pmoust
    MemberAuthor

    dispatch-intent nonce=1794898375-38841 ticket=4362 ref=a8a3be09d5b6492a2b0eefca5f5086c41414cc79

  20. pmoust commented on Nov 17, 2026

    @pmoust
    MemberAuthor

    dispatch-record nonce=1794898375-38841 task=ecb8f937-c882-402e-aeea-51890465144f

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions