bug(session-store): Async cannot recover a fixed quorum after a majority restarts

[VerifiedOrganic]

Async cannot re-form an existing three-voter quorum after two voters restart, even though their retained storage returns and the third voter remains in the same process. The equivalent Durable CRC exercise recovered automatically. This blocks the required product behavior: temporary majority loss must have a safe recovery path; indefinitely retaining a fence is not successful service recovery.

The pinned SDK explicitly documents this restriction. This issue requests correction of that recovery capability/contract, rather than claiming that the implementation violates its current documentation. Do not solve it by bypassing quarantine, accepting stale authority, or resetting the product's storage.

Component and baseline

• Component: opc-session-store fixed-quorum Async persistence and cold-incarnation admission; associated consensus/transport composition as needed.
• Executed SDK: 89b307767cd3cb497b6cc6eb0a9c70f6b71fe578.
• Product: VerifiedOrganic/epdg commit 02589200a1bd36d03408f40124137f51393bc66c, tree 299b4b73e832d1d9e2460d0d89f28d352538a62d; product PR #270.
• Linux/OpenShift CRC, three dedicated state voters and one unchanged product worker, retained per-voter storage, fixed membership, authenticated replication/consumer transport. This is not production HA or packet-continuity qualification.
• SDK main 76597d3ccd9438d956cb6257733cc7a256c5cc4c has identical async_persistence.rs, persistence_protocol.rs, session-store README and ADR 0022 blobs to the executed pin. That source comparison is not execution of latest main.

Executed reproduction

1. Start a healthy fixed three-voter installation with the same explicit persistence mode on all voters and the consumer. For Async the product uses durability.mode: durable and sessionPersistence: async.
2. Replace each voter sequentially, waiting for it and the original worker to become Ready before replacing the next. This passed for both policies.
3. Prevent the operator from undoing the controlled replica change, remove two voter processes, and retain the third process, all persistent volumes, roster, and Recovery authority. Hold the two-voter absence for 65 seconds, beyond the existing lease bounds. No deadline is raised.
4. Restore the two voters on their original storage and restore the operator. Do not restart the worker or clear authority.
5. Require the original worker to recover authority and service once safe quorum recovery completes.

Policy	Observed result
Durable	All voters returned Ready; the original worker became Active and returned HTTP 200 from liveness, readiness and admission. No worker restart, storage reset or Recovery-authority reset.
Async	The returning voters and original worker remained unready. All 111 worker samples over the 120-second return-observation window were liveness 200, readiness 503 and admission 503. Controller replica counts were restored and there were no cleanup failures.

A later read-only observation at 2026-09-18T13:22:26Z still showed the worker and all three voter containers Running, unready, with zero container restarts. Worker reason: s2b_recovery_lease_reacquisition_pending; all voter readiness reasons: state_quorum_durable_ready_timeout. The product reasons do not expose the SDK's exact internal recovery state, so no per-voter internal quarantine measurement is claimed.

Source mechanism

The pinned README says every existing Async root starts quarantined and requires a surviving live quorum. Durable reopens validated persisted state under its ordinary admission checks.

In recover_async_before, a cold voter requests a fresh ColdQuorumCut from other peers. handle_async_cold_barrier requires an active authoritative leader and a genuinely new committed proposal. The caller cannot vote or acknowledge replication until that cut is certified and applied. The source explicitly requires a live majority without that cold voter.

With two cold voters in a three-voter membership, the surviving process cannot form that majority. Neither returning voter can supply the missing vote before receiving a barrier that the survivor cannot commit. Repeated ordinary initialization does not remove this requirement. This is a source-established obstacle; additional product startup/diagnostic defects are not excluded by the CRC observation.

ADR 0022 explains why local completed Async generations alone do not authorize votes. Preserve that safety argument while providing the missing supported recovery behavior. Async's possible loss of acknowledged but unpersisted results must be distinguished from permanent inability to return to service. No such data loss was measured in this CRC exercise.

Required correction and regression evidence

1. Preserve a focused SDK RED using real fixed-quorum persistence, transport, initialization and authority checks: three healthy Async voters, two stopped and reopened on retained roots, one survivor, no reset or replacement membership. The required assertion is safe recovery, not merely continuing to fence. No new Rust RED is claimed by this report; the executed evidence above is the product CRC run.
2. Define and implement how recovered state and authority are selected when a live majority no longer exists. State the acknowledged-data-loss and lease/fence consequences explicitly. Do not treat stale persisted votes, equal indices, expired timers, or a subscriber key as recovery authority. If ordinary automatic recovery cannot be supported under the current storage contract, the SDK must provide an explicit safe recovery capability and the product must qualify its composition; an indefinite fence alone does not satisfy this requirement.
3. Cover a fully persisted local generation as well as lagging/differing generations and an acknowledged volatile tail. Add a separate all-voters-cold case; the current CRC run did not exercise all-cold recovery. Keep ordinary one-at-a-time rejoin working.
4. Retain negative controls for missing/corrupt storage, foreign epoch/root, mixed persistence mode, stale owner, delayed messages, partial snapshot/catch-up, cancellation, and replacement at an await. No old operation may mutate a recovered successor, and no unsafe quorum may become ready.
5. Assert bounded progress and fixed failure classification. Separate retryable catch-up, unsupported recovery and repair-required authority from generic timeout. Preserve existing deadlines and authentication/ownership validators.
6. After SDK RED/GREEN and required gates, repeat the Durable/Async product exercise with the original worker, exact storage and authority retained; require quorum and worker recovery, followed by the common service suite. CRC is not proof of production HA, audio, or packet continuity.

Retained evidence and classification correction

• Durable result, completed 2026-09-18T12:51:47Z: durable_same_process_active_recovered; receipt SHA-256 3630d4637bb9d34fc216dfcf4064ae62884717a47a992ce21ac7dd74f2e77548.
• Async result, completed 2026-09-18T13:14:46Z: async_cold_majority_remained_fenced; receipt SHA-256 eafcf71a619973835d0d679236fa50f44eadbb3277dae84b2905225aa39b6983.
• Exact product Linux make ci passed before the CRC runs; full log SHA-256 0fa487437f787ec9db447eb4c1869ca1aa43c8f0e7f7ecec49bf490f7718c075.

The Async harness reported passed because it tested the documented negative fence. That artifact is preserved unchanged, but it is not an availability/recovery pass and does not satisfy the operator's acceptance requirement. Current Async service recovery is blocked. No storage, namespace or Recovery authority was reset to turn this outcome green.

This is separate from management-audit issues #796–#798. Closed #814 concerns a post-activation initialization-deadline fixture; it does not provide a barrier without a surviving majority. #823 covers separate recovered-campaign fixture failures and does not establish this capability. No subscriber values, credentials, packet payloads or raw protocol identifiers are included.

[VerifiedOrganic]

The ePDG candidate now consumes merged SDK 84dd99d9e32441561a0ade8e24f4b499e369fb99 (PR #910). The remaining product recovery requirement is the protected-roster profile, which #910 deliberately excludes. This is a source comparison, not a new CRC recovery result.

The product constructs session voters with plan_protected_roster_quorum_consensus_topology (crates/epdg-app/src/serve/wiring.rs) and activates activate_protected_roster_profile_v2 (crates/epdg-app/src/state_quorum.rs). Removing that trust root/profile is not a valid product workaround.

At the exact merged SDK pin, async_recovery_protected_trust_root_needs_authority_even_without_retained_activation in crates/opc-session-store/src/consensus/store/async_persistence/tests/majority_protocol.rs explicitly requires ProtectedAuthorityRequired, failed initialization, and withheld traffic authority after a cold return. docs/async-majority-recovery-908.md likewise excludes protected-roster retirement. Those safety checks must remain until the corresponding exact retirement/recovery authority exists.

Please keep #908 open for this supported product composition. The needed regression is majority and all-cold return of the same protected fixed roster and retained roots, including a retained prepared transition, unavailable member, interrupted recovery, old lease/receipt rejection, and a successful higher-generation operation after exact reconciliation. Both normal Durable controls and Async positive recovery cases must run through the real authenticated transport; no reset, roster bypass, deadline increase, or quarantine-as-recovery pass.

The existing ePDG CRC result still records async_cold_majority_remained_fenced; it is preserved as a successful observation of fencing, not successful recovery. The new pin has not yet been deployed to CRC. A fresh installation for qualification must also remain distinct from recovery of the already-fenced legacy roots.