Skip to content

fix(session-store): recover reserved Async roots after majority loss - #910

Merged
VerifiedOrganic merged 11 commits into
mainfrom
fix/908-async-majority-recovery-20260918
Sep 19, 2026
Merged

VerifiedOrganic merged 11 commits into
mainfrom
fix/908-async-majority-recovery-20260918

Conversation

@VerifiedOrganic

@VerifiedOrganic VerifiedOrganic commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Relates to #908. Returning cold Async voters currently require a fresh committed live-quorum barrier before they may vote. After a majority restarts, that barrier cannot be committed. This change supports automatic recovery of reserved OPCNA003 roots when every configured retained voter returns, including all-cold return and a lost acknowledged tail. Normal Async acknowledgements remain independent of disk.

New roots durably reserve a finite authority range before volatile issuance. Recovery drains admitted work, persists a higher reservation on every exact retained owner, and selects a candidate covering all retained committed cuts. Actual Raft election, replication and committed application install a retirement boundary; every voter persists it before activation. Ordinary fresh traffic and application Recovery checks still apply. Orderly shutdown also has a separately validated, one-use participation proof.

Full epoch/root/mode/membership, process incarnation, round, vote and LogId binding remain required. Old leases and cached results cannot supply successor authority; new fences exceed even lost issuance. History and watch retirement preserve immutable request bindings and independently validated accounting. Accepted effects remain owned through cancellation, and delayed completion cannot cross a replacement boundary. Diagnostics expose fixed typed reasons.

Deployment boundary: this does not repair the currently fenced legacy OPCNA001/OPCNA002 installation, and protected-roster retirement is unsupported. Those roots lack the necessary pre-loss authority bound; the existing operator-recovery API itself requires a quorum. A supported migration/repair capability must durably retire the old external scope, finish or revoke accepted effects, and establish an exact successor enforced by every affected consumer. That capability is not implemented here. #908 remains open for that deployment boundary; passing SDK tests is not a repaired ePDG deployment.

Compatibility: recovery without a surviving compatible majority requires all configured retained owners, exact persisted membership and compatible retained history. Missing/corrupt roots, conflicting committed history and exhausted ranges require separate repair. Async members and consumers must upgrade together for OPC-ASYNC-2; this recovery patch leaves Durable session encoding and acknowledgement semantics unchanged. Upstream PR #926 has separate configuration audit schema and wire compatibility requirements. No timeout was extended, validator weakened, storage reset or test ignored to obtain recovery.

Validation on signed/pushed ec0b56043fe926058e9804dd21812928c3c67983, rebased onto main 31bc1a04d7845bfadc6cc0f0f55694105542cec1 through PRs #912, #919, #926, #920, #921, #922 and #924. All eleven signed commit messages and the recovery patch are unchanged. Local gates and actual hosted merge checkouts qualified the same source tree ed5fcbd47b35bf2bc119c0b39c367c9a6614acfc.

  • All 42 required local gate commands passed, including every CI shard plan/precheck, the complete opc-persist command, workspace lint/docs, Go vet/tests and offline manifest rendering. Local and hosted core tests used Rust 1.98.1 and the documented profiles; local TMPDIR was private, disk-backed XFS verified by findmnt.
  • All five hosted workflows and 42 final jobs passed. Separate native, i686, unsupported-platform, Egress and SCTP/N2 profiles were checked against complete workflow logs.
  • The original production-transport baseline RED, volatile-tail RED/GREEN, majority/all-cold abrupt restart, stale-authority controls and fix-removal RED remain preserved. The new full gate runs exercise the rebased recovery tests. Optional ignored latency qualification is not claimed.

On the previous head a854eba8, the initial it-2 command failed durable_readiness_probe_classifies_tls_accept_then_close_as_transport: expected Transport, observed Authentication, with 44 other target cases passing. The fixture, client, TLS classifier and lockfile are unchanged from base. Three unchanged exact controls, the full 45-case target and the complete original it-2 command then passed under the same profile and original deadline. Initial failure log SHA-256: b8e2b85e4937562036ae62edfb207df40ef20c4c105c7f1487a0f344464715ad; full original-command rerun SHA-256: 3e2960741b79329e459b9bce14b2f3719716aa322eda2c9a7410398fd385cb8e. The cause remains unestablished; socket lifetime during client retries is a source-level hypothesis, not a proved cause. No source or assertion changed. This is a documented rerun, not a first-attempt pass.

Earlier failures remain part of the evidence. On 418c9195, the initial full persistence command reached the unchanged survivor-election fixture's 26-second guard (log SHA-256 22e338bdbd36a15bd75cbfc10df9e3e40efe938c5270a9db011fad3602a7304b). Three unchanged exact controls and a complete original-command rerun passed (rerun log SHA-256 5060f9883a8130d1a603608e103611b84a1be683075578eb763e5a36d0f5e3e5). The selected target excludes this PR's modified crates; the precise timeout cause is unestablished.

That head's initial Egress attempt reached the unchanged Durable membership fixture's 15-second convergence guard (log SHA-256 60263d3e51e282126e61267b8e88242a671ab2d0a17d8c3aa149dbe9cc7a399e). The fixture and membership coordinator are byte-identical to baseline, but shared adapters remain on the changed dependency path and the cause is unestablished. Three matching-profile exact controls, the full ten-case target and the complete Egress workflow passed on the later a854eba8 head; the complete current-head Egress workflow passed again as reported above. Superseded cancellations are retained separately and are not passes.

An earlier 64ddeb19 rebase also retained an initial Durable scoped SQL failure under its original one-second guard; its WAL recorded an 844 ms sync. Six unchanged controls and the complete failed-shard rerun passed. Original failure log SHA-256: d4b3bb6f1631d507515b29829b601b44ec3ae4f9f0a3ebd034dad6ead1325e12. No source assertion, deadline, validator or test selector changed to obtain these passes. This is functional SDK qualification, not shared-runner latency qualification.

Current CI: IPsec LB, Egress fence, CI, GTP-U privileged, Security.

Recovery evidence and exact contract, ADR 0022. Complete raw logs, command receipts, source maps and merge-checkout proofs are retained in the task evidence directory. Source-manifest SHA-256: 6feceeb1d790a1b7b5a3a5f347584bbe415ee54b443985f8c3727911f33f9f0e.

No live CRC, Kubernetes, VM, deployment, image, storage-reset or product action was performed. The coordinating investigation must establish the applicable migration/authority contract, qualify consumer retry/history/fencing behavior, and then repeat the retained-majority Durable/Async comparison with the original worker and common service suite. SDK results do not establish product recovery, ongoing-call survival, audio, packet continuity or production HA. #908 remains open for the legacy/protected deployment boundary.

@VerifiedOrganic VerifiedOrganic changed the title test(session-net): reproduce Async majority restart recovery failure (#908) test(session-store): reproduce Async recovery failure and missing authority (#908) Sep 18, 2026
@VerifiedOrganic VerifiedOrganic changed the title test(session-store): reproduce Async recovery failure and missing authority (#908) fix(session-store): recover Async after certified shutdown (#908, partial) Sep 18, 2026
@VerifiedOrganic VerifiedOrganic changed the title fix(session-store): recover Async after certified shutdown (#908, partial) fix(session-store): recover retained Async quorum authority (#908, in progress) Sep 18, 2026
@VerifiedOrganic VerifiedOrganic changed the title fix(session-store): recover retained Async quorum authority (#908, in progress) fix(session-store): recover reserved Async roots after majority loss Sep 18, 2026
@VerifiedOrganic
VerifiedOrganic force-pushed the fix/908-async-majority-recovery-20260918 branch 4 times, most recently from 418c919 to a854eba Compare September 19, 2026 06:01
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Explicitly discard the vote request on hosts without the Linux recovery state. Preserve the Linux vote equality check and the non-Linux fail-closed behavior.

Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
@VerifiedOrganic
VerifiedOrganic force-pushed the fix/908-async-majority-recovery-20260918 branch from a854eba to ec0b560 Compare September 19, 2026 07:34
@VerifiedOrganic
VerifiedOrganic marked this pull request as ready for review September 19, 2026 08:50
@VerifiedOrganic
VerifiedOrganic merged commit 84dd99d into main Sep 19, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant