Skip to content

test(session-net): replace yield_now spin-waits with watch - #622

Draft
VerifiedOrganic wants to merge 3 commits into
mainfrom
test/578-replace-yield-spin
Draft

VerifiedOrganic wants to merge 3 commits into
mainfrom
test/578-replace-yield-spin

Conversation

@VerifiedOrganic

Copy link
Copy Markdown
Contributor

Refs #578

What

Replaces the tokio::task::yield_now() spin-waits in opc-session-net that the issue identifies as dangerous, with level-triggered tokio::sync::watch signalling.

Why

A yield_now spin keeps the waiter runnable for the whole wait. Under #[tokio::test(start_paused = true)] the current-thread runtime then never goes idle, so tokio's virtual clock never auto-advances and the enclosing timeout — measured on that same virtual clock — never fires. The failure mode is a hang until the CI job's wall-clock kill, not a flake. A timeout that cannot fire is not a timeout.

Changes

  • configured_ceiling_also_reserves_post_connect_rpc_time (start_paused): the accounting-settlement spin now parks on a watch generation that ConnectionAttemptTestAccounting bumps on every recorded transition (subscribe_changed).
  • wait_for_cached_lane_to_empty helper (reached from the start_paused cached_consensus_reaper_never_races_an_in_flight_lane): now parks on an emptied latch the cached-connection reaper publishes when it retires the lane. The async connection mutex cannot be polled from wait_for's synchronous predicate, so the reaper publishes the edge directly.
  • historical_cas_is_rejected_after_server_restart_without_redispatch: the backend poll now parks on a cas_effects watch the ReplicationDispatchSpy bumps on each committed CompareAndSetResult::Success, mirroring the file's existing CancellableStallBackend::wait_for_active convention from test(session-net): mtls_backend_deadlines flakes under workspace-level runtime oversubscription #566.

watch::Receiver::wait_for evaluates its predicate against the current value before parking, so an already-reached state resolves immediately with no lost wakeup, and a parked waiter lets the paused clock advance. Per the issue's guidance, this does not raise any deadline and does not weaken any assertion: timeout bounds (1s/1s/2s) and all post-wait assertions are unchanged.

Verification

  • cargo fmt --all --check
  • git diff --check
  • cargo clippy -p opc-session-net --all-targets --all-features -- -D warnings
  • cargo test -p opc-session-net --all-features291 tests, 0 failures ✓ (the two paused tests now finish in 0.00s virtual time instead of hanging)
  • Adversarial review swarm (spec/acceptance, adversarial-runtime, boundary/security): all three SHIP, no BLOCKER/MAJOR findings. The runtime reviewer independently traced the lifecycle math and confirmed every caller reaches the reaper's take(), level-triggered wakeup-safety on all three waits, and a negligible production footprint (a subscriber-less send_replace on the rare retirement path).

Scope note (priority 3)

The issue's priority 1 (the two paused-clock hangs) and priority 2 (the named three_node_quorum test) are fully addressed. Priority 3 ("the remaining instances, as cleanup, only where a test actually contends") is deliberately split out, matching how #566's scope note asked that the broad sweep be a separate decision.

A workspace-wide parse found 73 genuine tokio::task::yield_now spin-loops (68 timeout-wrapped); only the two fixed here are PAUSED-REACHABLE (confirmed hangs). The remaining ~70 span eight other crates (opc-ipsec-lb 17, opc-session-store 15, opc-session-net 11 more, opc-ipsec-xfrm 11, opc-persist 5, opc-diameter-transport 4, opc-runtime 3, opc-session-cache 3, opc-tls 1) and largely need invasive signalling — global METRICS watches crossing into opc-redaction, semaphore-permit waits, or the production opc-ipsec-xfrm/src/observation/linux.rs drain loops (537, 661) — disproportionate to a test-reliability fix. Recommend per-crate follow-up issues for that sweep.

A yield_now spin keeps the waiter runnable for the whole wait. Under
#[tokio::test(start_paused = true)] that stops the current-thread
runtime from ever going idle, so the virtual clock never auto-advances
and the enclosing timeout - measured on that same clock - never fires:
a hang, not a flake.

Replace the two paused-clock spins (the accounting-settlement wait in
configured_ceiling_also_reserves_post_connect_rpc_time, and the
wait_for_cached_lane_to_empty helper reached from the reaper-race
test) and the historical_cas backend poll with level-triggered watch
waits. watch::Receiver::wait_for evaluates its predicate against the
current value before parking, so an already-reached state resolves
immediately with no lost wakeup, and a parked waiter lets the paused
clock advance.

The accounting struct and the dispatch spy grow a watch generation
bumped on each transition; the cached-connection reaper publishes an
emptied latch when it retires the lane (the async connection mutex
cannot be polled from wait_for's synchronous predicate). Timeout
bounds and all post-wait assertions are unchanged.

Refs #578

Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
@VerifiedOrganic

Copy link
Copy Markdown
Contributor Author

Resolved the main merge while preserving the newer consensus regressions, then corrected the test wait in b2273ae35d3beb69bdd04dee953e153f75faa66b. Lane retirement notifications are test-only and cover both lifecycle and idle retirement. The waiter subscribes before checking the current cached connection and rechecks that state after each notification, so an earlier retirement cannot satisfy a wait for a reused lane.

Local verification passed: 89 consensus/lifecycle unit tests with four test threads, the exact historical_cas_is_rejected_after_server_restart_without_redispatch integration regression, affected all-target/all-feature Clippy, formatting, and diff checks. Complete current-head hosted validation remains required before merge.

@VerifiedOrganic

Copy link
Copy Markdown
Contributor Author

Deferring refreshed head b2273ae35d3beb69bdd04dee953e153f75faa66b on a current-head conformance failure. The merge conflict and test-only retirement notification were repaired, preserving newer main tests and rechecking actual connection state after notifications. All 89 focused consensus/lifecycle tests, the historical-CAS restart regression, and affected Clippy pass locally.

The kernel-independent conformance job fails persistent_three_voter_consumer_write_does_not_spend_budget_on_a_read_quorum. The healthy write returns outcome_unknown within the existing operation-budget test, with delayed ReadBarrier=0 and delayed empty AppendEntries=0. This failed conformance requirement remains a merge blocker despite the passing focused wait tests.

Next step: the PR maintainer and session-net/session-store maintainers should reproduce this exact conformance failure against the refreshed branch and main, establish its cause, and land a reviewed correction without increasing the operation budget. Coordinate with #741 only if matching evidence establishes overlap. Then run every required local and current-head hosted gate. Keeping this PR open as a draft; the test failure is preserved and no passing status is inferred from a different head.

@VerifiedOrganic
VerifiedOrganic marked this pull request as draft September 7, 2026 03:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant