fix(session-store): recover reserved Async roots after majority loss - #910
Merged
Merged
Conversation
VerifiedOrganic
force-pushed
the
fix/908-async-majority-recovery-20260918
branch
4 times, most recently
from
September 19, 2026 06:01
418c919 to
a854eba
Compare
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Explicitly discard the vote request on hosts without the Linux recovery state. Preserve the Linux vote equality check and the non-Linux fail-closed behavior. Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
VerifiedOrganic
force-pushed
the
fix/908-async-majority-recovery-20260918
branch
from
September 19, 2026 07:34
a854eba to
ec0b560
Compare
VerifiedOrganic
marked this pull request as ready for review
September 19, 2026 08:50
This was referenced Sep 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Relates to #908. Returning cold Async voters currently require a fresh committed live-quorum barrier before they may vote. After a majority restarts, that barrier cannot be committed. This change supports automatic recovery of reserved
OPCNA003roots when every configured retained voter returns, including all-cold return and a lost acknowledged tail. Normal Async acknowledgements remain independent of disk.New roots durably reserve a finite authority range before volatile issuance. Recovery drains admitted work, persists a higher reservation on every exact retained owner, and selects a candidate covering all retained committed cuts. Actual Raft election, replication and committed application install a retirement boundary; every voter persists it before activation. Ordinary fresh traffic and application Recovery checks still apply. Orderly shutdown also has a separately validated, one-use participation proof.
Full epoch/root/mode/membership, process incarnation, round, vote and LogId binding remain required. Old leases and cached results cannot supply successor authority; new fences exceed even lost issuance. History and watch retirement preserve immutable request bindings and independently validated accounting. Accepted effects remain owned through cancellation, and delayed completion cannot cross a replacement boundary. Diagnostics expose fixed typed reasons.
Deployment boundary: this does not repair the currently fenced legacy
OPCNA001/OPCNA002installation, and protected-roster retirement is unsupported. Those roots lack the necessary pre-loss authority bound; the existing operator-recovery API itself requires a quorum. A supported migration/repair capability must durably retire the old external scope, finish or revoke accepted effects, and establish an exact successor enforced by every affected consumer. That capability is not implemented here. #908 remains open for that deployment boundary; passing SDK tests is not a repaired ePDG deployment.Compatibility: recovery without a surviving compatible majority requires all configured retained owners, exact persisted membership and compatible retained history. Missing/corrupt roots, conflicting committed history and exhausted ranges require separate repair. Async members and consumers must upgrade together for
OPC-ASYNC-2; this recovery patch leaves Durable session encoding and acknowledgement semantics unchanged. Upstream PR #926 has separate configuration audit schema and wire compatibility requirements. No timeout was extended, validator weakened, storage reset or test ignored to obtain recovery.Validation on signed/pushed
ec0b56043fe926058e9804dd21812928c3c67983, rebased onto main31bc1a04d7845bfadc6cc0f0f55694105542cec1through PRs #912, #919, #926, #920, #921, #922 and #924. All eleven signed commit messages and the recovery patch are unchanged. Local gates and actual hosted merge checkouts qualified the same source treeed5fcbd47b35bf2bc119c0b39c367c9a6614acfc.On the previous head
a854eba8, the initial it-2 command faileddurable_readiness_probe_classifies_tls_accept_then_close_as_transport: expectedTransport, observedAuthentication, with 44 other target cases passing. The fixture, client, TLS classifier and lockfile are unchanged from base. Three unchanged exact controls, the full 45-case target and the complete original it-2 command then passed under the same profile and original deadline. Initial failure log SHA-256:b8e2b85e4937562036ae62edfb207df40ef20c4c105c7f1487a0f344464715ad; full original-command rerun SHA-256:3e2960741b79329e459b9bce14b2f3719716aa322eda2c9a7410398fd385cb8e. The cause remains unestablished; socket lifetime during client retries is a source-level hypothesis, not a proved cause. No source or assertion changed. This is a documented rerun, not a first-attempt pass.Earlier failures remain part of the evidence. On
418c9195, the initial full persistence command reached the unchanged survivor-election fixture's 26-second guard (log SHA-25622e338bdbd36a15bd75cbfc10df9e3e40efe938c5270a9db011fad3602a7304b). Three unchanged exact controls and a complete original-command rerun passed (rerun log SHA-2565060f9883a8130d1a603608e103611b84a1be683075578eb763e5a36d0f5e3e5). The selected target excludes this PR's modified crates; the precise timeout cause is unestablished.That head's initial Egress attempt reached the unchanged Durable membership fixture's 15-second convergence guard (log SHA-256
60263d3e51e282126e61267b8e88242a671ab2d0a17d8c3aa149dbe9cc7a399e). The fixture and membership coordinator are byte-identical to baseline, but shared adapters remain on the changed dependency path and the cause is unestablished. Three matching-profile exact controls, the full ten-case target and the complete Egress workflow passed on the latera854eba8head; the complete current-head Egress workflow passed again as reported above. Superseded cancellations are retained separately and are not passes.An earlier
64ddeb19rebase also retained an initial Durable scoped SQL failure under its original one-second guard; its WAL recorded an 844 ms sync. Six unchanged controls and the complete failed-shard rerun passed. Original failure log SHA-256:d4b3bb6f1631d507515b29829b601b44ec3ae4f9f0a3ebd034dad6ead1325e12. No source assertion, deadline, validator or test selector changed to obtain these passes. This is functional SDK qualification, not shared-runner latency qualification.Current CI: IPsec LB, Egress fence, CI, GTP-U privileged, Security.
Recovery evidence and exact contract, ADR 0022. Complete raw logs, command receipts, source maps and merge-checkout proofs are retained in the task evidence directory. Source-manifest SHA-256:
6feceeb1d790a1b7b5a3a5f347584bbe415ee54b443985f8c3727911f33f9f0e.No live CRC, Kubernetes, VM, deployment, image, storage-reset or product action was performed. The coordinating investigation must establish the applicable migration/authority contract, qualify consumer retry/history/fencing behavior, and then repeat the retained-majority Durable/Async comparison with the original worker and common service suite. SDK results do not establish product recovery, ongoing-call survival, audio, packet continuity or production HA. #908 remains open for the legacy/protected deployment boundary.