test(session-net): replace yield_now spin-waits with watch - #622
VerifiedOrganic wants to merge 3 commits into
Conversation
A yield_now spin keeps the waiter runnable for the whole wait. Under #[tokio::test(start_paused = true)] that stops the current-thread runtime from ever going idle, so the virtual clock never auto-advances and the enclosing timeout - measured on that same clock - never fires: a hang, not a flake. Replace the two paused-clock spins (the accounting-settlement wait in configured_ceiling_also_reserves_post_connect_rpc_time, and the wait_for_cached_lane_to_empty helper reached from the reaper-race test) and the historical_cas backend poll with level-triggered watch waits. watch::Receiver::wait_for evaluates its predicate against the current value before parking, so an already-reached state resolves immediately with no lost wakeup, and a parked waiter lets the paused clock advance. The accounting struct and the dispatch spy grow a watch generation bumped on each transition; the cached-connection reaper publishes an emptied latch when it retires the lane (the async connection mutex cannot be polled from wait_for's synchronous predicate). Timeout bounds and all post-wait assertions are unchanged. Refs #578 Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
Signed-off-by: VerifiedOrganic <verifiedorganic@sent.com>
|
Resolved the main merge while preserving the newer consensus regressions, then corrected the test wait in Local verification passed: 89 consensus/lifecycle unit tests with four test threads, the exact |
|
Deferring refreshed head The kernel-independent conformance job fails Next step: the PR maintainer and session-net/session-store maintainers should reproduce this exact conformance failure against the refreshed branch and main, establish its cause, and land a reviewed correction without increasing the operation budget. Coordinate with #741 only if matching evidence establishes overlap. Then run every required local and current-head hosted gate. Keeping this PR open as a draft; the test failure is preserved and no passing status is inferred from a different head. |
Refs #578
What
Replaces the
tokio::task::yield_now()spin-waits inopc-session-netthat the issue identifies as dangerous, with level-triggeredtokio::sync::watchsignalling.Why
A
yield_nowspin keeps the waiter runnable for the whole wait. Under#[tokio::test(start_paused = true)]the current-thread runtime then never goes idle, so tokio's virtual clock never auto-advances and the enclosingtimeout— measured on that same virtual clock — never fires. The failure mode is a hang until the CI job's wall-clock kill, not a flake. A timeout that cannot fire is not a timeout.Changes
configured_ceiling_also_reserves_post_connect_rpc_time(start_paused): the accounting-settlement spin now parks on a watch generation thatConnectionAttemptTestAccountingbumps on every recorded transition (subscribe_changed).wait_for_cached_lane_to_emptyhelper (reached from thestart_pausedcached_consensus_reaper_never_races_an_in_flight_lane): now parks on anemptiedlatch the cached-connection reaper publishes when it retires the lane. The asyncconnectionmutex cannot be polled fromwait_for's synchronous predicate, so the reaper publishes the edge directly.historical_cas_is_rejected_after_server_restart_without_redispatch: the backend poll now parks on acas_effectswatch theReplicationDispatchSpybumps on each committedCompareAndSetResult::Success, mirroring the file's existingCancellableStallBackend::wait_for_activeconvention from test(session-net): mtls_backend_deadlines flakes under workspace-level runtime oversubscription #566.watch::Receiver::wait_forevaluates its predicate against the current value before parking, so an already-reached state resolves immediately with no lost wakeup, and a parked waiter lets the paused clock advance. Per the issue's guidance, this does not raise any deadline and does not weaken any assertion: timeout bounds (1s/1s/2s) and all post-wait assertions are unchanged.Verification
cargo fmt --all --check✓git diff --check✓cargo clippy -p opc-session-net --all-targets --all-features -- -D warnings✓cargo test -p opc-session-net --all-features— 291 tests, 0 failures ✓ (the two paused tests now finish in 0.00s virtual time instead of hanging)take(), level-triggered wakeup-safety on all three waits, and a negligible production footprint (a subscriber-lesssend_replaceon the rare retirement path).Scope note (priority 3)
The issue's priority 1 (the two paused-clock hangs) and priority 2 (the named
three_node_quorumtest) are fully addressed. Priority 3 ("the remaining instances, as cleanup, only where a test actually contends") is deliberately split out, matching how #566's scope note asked that the broad sweep be a separate decision.A workspace-wide parse found 73 genuine
tokio::task::yield_nowspin-loops (68 timeout-wrapped); only the two fixed here are PAUSED-REACHABLE (confirmed hangs). The remaining ~70 span eight other crates (opc-ipsec-lb17,opc-session-store15,opc-session-net11 more,opc-ipsec-xfrm11,opc-persist5,opc-diameter-transport4,opc-runtime3,opc-session-cache3,opc-tls1) and largely need invasive signalling — globalMETRICSwatches crossing intoopc-redaction, semaphore-permit waits, or the productionopc-ipsec-xfrm/src/observation/linux.rsdrain loops (537, 661) — disproportionate to a test-reliability fix. Recommend per-crate follow-up issues for that sweep.