test(session-net): investigate five-minute fenced-status compaction timeout

[VerifiedOrganic]

A required CI run at commit 594f4e11f55658204dbc5fd7ef6e6194c17b0a05 exceeded the existing five-minute workload guard in persistent_three_voter_fenced_status_converges_after_response_loss_and_compaction. The test reported all three consensus engines as Running, no storage error, and no snapshot/purge progress at failure. The failing run does not establish why progress stopped.

Evidence:

• Failed miscellaneous test shard, attempt 1.
• Workload guard and failure observation.
• The hosted test failed after 311.48 seconds; the guarded workload permits 300 seconds. The same isolated command passed locally at the same commit in 75.36 seconds under the current CI toolchain/profile. This difference is an observation, not proof of a runner-performance cause.
• The opc-session-net, opc-session-store, opc-session-testkit, shard definitions and CI workflow sources are byte-identical to main 427c1811233dc698e3038c3b54e4ea3f7fcef01e in PR feat(persist): bound acknowledged configuration history #847. Source identity alone does not establish independence or cause.

The one unchanged-head retry passed: miscellaneous shard, attempt 2. Its checkout matches the original tested merge tree. All 45 current checks passed, but the original failure remains unexplained. Preserve both attempts; this retry is not a fix or closure criterion. This is distinct from #838's 100 ms protected-transition case and #820's replacement-readiness interruption case.

Acceptance:

1. Reproduce or causally explain the missing workload/snapshot progress, preserving the original failure and exact base/head, test profile and command. Use bounded synthetic diagnostics to distinguish leader/admission state, committed/applied progress, snapshot scheduling and transport behavior.
2. Correct the demonstrated cause at its owning boundary, with a detector that fails on removal of the correction and an adversarial negative case. Do not infer a fix from an unrelated passing retry.
3. Preserve the existing five-minute workload guard, other deadlines, durability/readback assertions and feat(session-store): add native WAL and explicit persistence modes #776 native WAL plus Durable/Async semantics. Do not weaken checks, force a leadership change, add sleeps, or replace the actual recovery path merely to obtain a pass.
4. Qualify the isolated case and the complete current CI profile, including restart/response-loss/compaction behavior, and obtain independent review. State any remaining uncertainty explicitly.

This issue records an observed test failure and does not authorize a performance claim, a waiver of required checks, or a deployment claim.

[VerifiedOrganic]

Investigation update: diagnostic draft #857 is at b957f925172fc5a6fc592befc6bf689c9d256df5. It records bounded snapshot stages, elapsed workload progress, local term/admission and existing transport-attempt counters. The reports send no requests; configured path counts do not prove connectivity, and attempted messages do not prove delivery.

The original five-minute workload timeout has not reproduced and its cause remains unresolved. The historical log contains a final position, not a timeline establishing when progress stopped. Controlled scheduling, storage and publication-delay experiments have not supplied a causal match; temporary fault changes are excluded from the PR.

The current revision passed the exact isolated workspace-feature test and workspace-wide Clippy. Current-head hosted checks and full local CI-plan qualification are running. All 42 hosted checks passed on the preceding diagnostic revision, which does not qualify this successor or establish a fix. The original workload, deadlines, response-loss recovery and durability assertions remain intact. This issue stays open; passing retries are not closure evidence.

[VerifiedOrganic]

The independent exact-head review of diagnostic PR #857 is complete at 3f000b68a50687b536d784dc88b72e46376db8e9 (tree acdd22b3aa4c663993149f5e3602563a1d4e5749). It found no blocking source defects in the four-file diagnostic change. This is source approval only, not a production fix or merge-readiness verdict.

The original timeout remains unreproduced and unexplained. The affected test passed on this tree locally in 86.04 seconds and in hosted CI in 107.14 seconds. The eight preserved original-tree runs also passed. None of these passes establishes a cause or fix.

Full qualification failed elsewhere:

• The local five-process leader-failover test timed out waiting for a readiness reply. The existing test(session-store): qualify networked quorum HA for production #143 owns the matching symptom; the new occurrence and interpretation limits are recorded there. Later local gates were not run.
• Hosted checks finished with 42 passes and one failure in encrypted SQLite retirement replay after restart, tracked in test(ipsec-lb): investigate encrypted SQLite retirement replay failure after adapter restart #863. Its generic ConnectionRefused classification does not prove a network failure.

Both first failing logs and exact source identities are preserved. No rerun, widened deadline, extra exclusion or production correction was introduced. PR #857 remains draft. The source and test runners are unchanged, the local runner has exited, and the prepared snapshot-return experiment has not run.

The review also narrowed one earlier hypothesis: the pinned Openraft read target is the maximum of the current leader's no-op and committed log index, not an arbitrary uncommitted tail. A retained-read explanation therefore requires evidence of the actual selected index and why application cannot reach it. The historical failure log does not supply that evidence.

[VerifiedOrganic]

Diagnostic PR #857 is merged as 75044f43852cd816d9734d03914515395bf20c69. The actual merged tree, f7f9c93c476e18d69093f8fa49b0865f4932dde5, exactly matches the tested merge tree and is on main.

All 42 pre-merge hosted checks passed, including the isolated test for this issue. Independent review approved the full diagnostic change and its documented local qualification boundary. The miscellaneous shard passed after one unchanged-head retry; its original audit-store failure and unresolved cause remain recorded under #868.

The new observations are bounded, passive progress diagnostics. The original 300-second guard, 4,300-command workload and durability assertions remain unchanged. Passing runs do not establish a causal correction. This issue remains open: the original compaction timeout's cause is still unproven, and any recurrence must be evaluated against the new phase/counter evidence before proposing a fix.

[VerifiedOrganic]

The continuation was checked against current main 75044f43852cd816d9734d03914515395bf20c69; the clean investigation branch is already at that commit, so no rebase was required.

The unchanged-source isolated test passed in 85.16 seconds. At an intermediate observation, both surviving voters were stored/applied at 4100 with the last snapshot phase waiting_publication_connection; recovery subsequently completed. That phase and position alone therefore cannot identify the original failure.

One temporary fault experiment paused the current-thread runtime for 20 seconds immediately before the first snapshot metadata publication. An operation deadline expired, then the original recovery path completed all assertions; the selected test passed in 132.09 seconds. The hook was removed after the process exited. This rules out that particular injected schedule only. The original 300-second workload guard, 4,300 commands, election recovery, receipt and durability assertions were preserved.

An independent source review found no verified transport defect. The original log still cannot distinguish a retained core response from waiting for application of the selected committed/no-op index. No causal correction is established, no fix PR has been opened, and this issue remains open. A recurrence with the merged diagnostics or a deterministic source-reachable failure is needed before choosing a correction. These isolated results do not establish full qualification of a fix.

[VerifiedOrganic]

The causal investigation remains open. No production correction has been established.

A separately reviewed, temporary diagnostic branch at b1d8ea853431b27593395daf94d02576e1e62e63 is running a bounded hosted recurrence experiment, based on main 75044f43852cd816d9734d03914515395bf20c69. It is not a merge candidate.

The experiment retains the original 300-second workload guard, 4,300-entry target, sequential commands, response loss, elections, and receipt/durability assertions. It adds publication-phase and periodic aggregate observations, uses the same workspace feature profile, resolves the exact target once, and stops on the first nonzero result. Separate runner limits preserve failure logs if teardown hangs. It runs at most eight samples; a passing sample is not a causal fix or closure criterion.

Local instrumented preflight passed the exact target in 89.98 seconds (one test, 602 filtered). Its log SHA-256 is 69470ef03e3a0841d145abb8b75aa131aab82a558838c3a70312b56919ef26f9. The new observations can affect scheduling; they are diagnostic evidence only. The original failure and unchanged-head retry remain preserved. Dependency updates since that failure are another reason not to treat current-main passes as an explanation.

[VerifiedOrganic]

The diagnostic observer review found that a timer in the workload task can repoll a waiting future. Results from run 35177807656 will therefore be recorded with that limitation; a pass cannot exclude a lost wakeup.

A separate, reviewed comparison is now running at 35178786546. It uses the exact original failing baseline be7b580986b7efaf87b7aa79b13f55d5d32956c1, unchanged Cargo manifests/lockfile, and an independently spawned observer. The original workload is awaited directly under its unchanged 300-second guard. Local preflight passed the exact target in 108.57 seconds (1 passed, 602 filtered); this validates the experiment, not a correction.

Both temporary branches are investigation-only and must not merge. Each hosted experiment runs at most eight sequential samples and stops at its first failure. #856 remains open; no causal correction or weakened acceptance gate has been introduced.

[VerifiedOrganic]

The original-baseline comparison completed: all eight samples passed in
105.67–106.66 seconds, using the original dependencies, the original 300-second
workload guard, and an independently scheduled observer. The artifact log hashes
were verified. Run: https://github.com/openpacketcore/openpacketcore-sdk/actions/runs/35178786546

This establishes nonrecurrence in those samples, not a causal correction. The
earlier current-main samples also passed, but their observer periodically
repolls the workload; that observation limitation remains recorded.

The next bounded experiment restores the original miscellaneous-shard context:
the unchanged precheck, the ordinary workspace lib/bin command, the preceding
isolated contract, and then this target once, with original libtest output
capture. It stops at the first failure and does not retry. A failure before the
target is reported separately. The original workflow is restored only as file
input to its unchanged precheck after the manual diagnostic job starts; the
artifact records the exact workflow bytes, hash and source baseline.

This is a temporary, do-not-merge investigation branch. No production correction
has been made and #856 remains open. The original 4,300-command target, response
loss, normal election/recovery, receipt/readback and durability assertions remain
unchanged.

Temporary head: eb88a454dcd8fbf0384de44261a6be2a5880b791.
Run: https://github.com/openpacketcore/openpacketcore-sdk/actions/runs/35180423590

[VerifiedOrganic]

The bounded original-shard-context experiment completed without reproducing this failure: run 35180423590.

It used the original failing baseline be7b580986b7efaf87b7aa79b13f55d5d32956c1 and unchanged dependency versions, with the previously described diagnostic probes and separate observer. The original miscellaneous precheck and exact three-command prefix ran in order: ordinary library/binary tests, the preceding isolated consumer-write contract, then this target. Every command passed on its first invocation. The target reported one passed, 602 filtered, 105.41 seconds, retaining the original 300-second workload guard and recovery/receipt/readback assertions. No retry or outer timeout occurred.

The artifact records diagnostic head eb88a454dcd8fbf0384de44261a6be2a5880b791, tree c4f00ba1528e8b7659d145c7f6a7512a73cd327c, exact commands, original-workflow substitution and log hashes. These were verified against the downloaded artifact and historical workflow bytes. The temporary diagnostic workflow must not merge. Successful captured libtest output omits its probe timeline, so this run does not supply a sampled trajectory.

Together with the eight original-baseline isolated passes in run 35178786546, this establishes nonrecurrence under these experiments, not a correction. The added observer can still affect scheduling. The earlier current-main experiment retains the separately documented workload-repoll limitation.

Independent source review found no supported permanent SQLite read-worker permit/connection retention path or reciprocal first-snapshot wait. Those findings do not establish that the original failure was caused by runner performance or rule out a timing-sensitive defect. The original failure lacks the timeline needed to distinguish slow progress from a persistent wait.

No causal production correction has been demonstrated; #856 remains open. The original failure and all experiment results are preserved. A correction still needs a demonstrated failing path, fix-removal and adversarial evidence, and the required current-profile qualification. Existing deadlines and durability semantics remain unchanged.

[VerifiedOrganic]

A bounded diagnostic experiment is now running on the existing ubuntu-latest runners: run 35217508373.

This removes the remaining in-process Rust observer used in the previous experiments. Separate clean detached worktrees execute the exact original source be7b580986b7efaf87b7aa79b13f55d5d32956c1 and current main 75044f43852cd816d9734d03914515395bf20c69, with the original command/profile and libtest capture. Each profile stops on its first failure, with at most 16 samples. The 300-second workload guard, 4,300-command workload and all response-loss/receipt/recovery assertions remain unchanged.

The external harness records owned-target CPU/I/O/thread counters and aggregate host pressure/load. Only an outlier may receive one bounded stack-only debugger observation; its intervention is recorded before attachment, so a later pass cannot be described as undisturbed timing evidence. No arguments, locals, memory, environment or payload content is requested. Process ownership/cleanup and false-green checks passed focused local verification and independent review. Diagnostic branch investigate/856-unobserved-soak-20260917 at b5a76bf40b272cddedc3854064897bb15b273184 replaces ordinary CI only for this experiment and must never merge.

A comparison of preserved original logs also shows that the ordinary preceding command finished in about 22 minutes on the failing attempt versus about 25 minutes on the passing unchanged retry, with matching result counts. This does not rule out a transient stall or resource spike during the target, but does not establish a generally slower runner either. No root cause or production correction has been demonstrated; #856 remains open.

[VerifiedOrganic]

The unchanged-source comparison has finished. #856 remains unresolved; no production correction or runner-performance root cause has been established.

Run 35217508373 used standard ubuntu-latest runners, Rust 1.98.1, the original exact Cargo command and the unchanged 300-second workload guard. The harness was b5a76bf40b272cddedc3854064897bb15b273184. It added no Rust observer or source modifications. No outlier debugger probe ran in either profile.

Fixed source	Completed target results	Target duration	Median
Original failing merge be7b580986b7efaf87b7aa79b13f55d5d32956c1	14 passed; zero target failures	157.09–219.69 seconds	178.38 seconds
Main at dispatch, 75044f43852cd816d9734d03914515395bf20c69	16 passed; zero target failures	101.78–114.50 seconds	102.51 seconds

The original-source job and overall run are marked failure because the experiment is incomplete: after 14 passes, the harness refused to start sample 15 without the full 670-second outer-guard/cleanup reserve remaining. Its incomplete.json records experiment_budget. No target failed and no individual sample exhausted its outer guard. This is not a reproduction of the historical timeout and does not count as 16 completed original-source samples.

Verification checked each fixed source head/tree, the harness hash, exact selection and execution commands, every recorded log hash, and exactly one successful target result per sample. Both artifacts (diagnostic-856-original and diagnostic-856-current-main) and full job logs are preserved by the run.

The slower successful job showed substantially higher aggregate host I/O pressure: median cumulative some pressure during its sampled target windows was 45.133 seconds, versus 1.997 seconds in the current-main job. This supports investigating storage delays, but these jobs ran on different machines with different dependency lockfiles. Host-level pressure is not per-operation attribution and does not establish the cause of the historical failure. The original failure still has no progress/resource timeline.

All 30 completed samples passed on standard runners. These results do not justify requiring a higher-performance runner, increasing the original timeout, claiming a dependency upgrade fixed #856, or closing the issue. The diagnostic workflow branch must not merge. A causal correction still requires a preserved recurrence or a demonstrated reachable failure mechanism with a failing detector.

[VerifiedOrganic]

A bounded follow-up is running to remove the different-machine variable from the previous comparison: same-runner comparison, harness 138e9f45d73504393d30007ef1f1952379254052.

It compiles both fixed source trees before sampling, then uses the predetermined order original/current/current/original/current/original/original/current on one standard ubuntu-latest runner and one unchanged storage setup. Rust 1.98.1, the exact Cargo command, clean source trees and the 300-second workload guard remain unchanged. Separate build directories retain both builds. The first target failure stops the comparison; incomplete budgets are reported separately. Thirteen local harness cases plus syntax, actionlint and diff checks passed.

The previous slow successful samples continued writing throughout their sampled workload windows, with I/O pressure spread across the run. That observation cannot locate or explain the historical failure. This comparison tests whether the timing gap follows the source revision on one host; it is not a claimed fix or performance qualification. #856 remains open, and the temporary diagnostic workflow branch must not merge.

[VerifiedOrganic]

The controlled comparison on one standard hosted runner completed: run and preserved artifact. Diagnostic head 138e9f45d73504393d30007ef1f1952379254052 is temporary and must not merge.

Both fixed, clean source revisions compiled before testing, then ran in the predetermined alternating order on the same machine and storage setup. All eight exact target samples passed with the original 300-second workload guard and assertions unchanged. No outlier stack probe was invoked.

Source	Four target durations (seconds)	Median
Original tested merge be7b580986b7efaf87b7aa79b13f55d5d32956c1	107.77, 105.67, 107.19, 109.32	107.48
Current main 75044f43852cd816d9734d03914515395bf20c69	105.01, 103.95, 103.91, 104.57	104.26

Source heads/trees/cleanliness, harness hashes, exact commands and selected target, log hashes, sample order, and complete result count were verified. Median sampled target CPU time was approximately 84.86s versus 84.97s; median aggregate host I/O-pressure time was approximately 1.48s versus 1.58s. These counters are sampled observations, not per-operation attribution.

The large timing gap in the earlier different-runner batches did not persist in this comparison. This supports host variation for those slow successful samples; it does not identify the original timeout's cause, prove a correction, or establish a hardware requirement. The original failure still lacks a progress/resource timeline. #856 remains open.

[VerifiedOrganic]

The bounded current-main campaign completed without reproducing #856:
run 35232749204 and artifacts.
The historical timeout remains unresolved; there is no demonstrated production correction.

Four standard ubuntu-latest runners executed eight samples each against clean,
unchanged source 75044f43852cd816d9734d03914515395bf20c69, tree
f7f9c93c476e18d69093f8fa49b0865f4932dde5. The external harness was
bf2125569a705a619020e11c08426d06d9545478. Rust 1.98.1, the exact isolated CI
command, 4,300-command workload, original 300-second guard, and recovery/receipt
assertions were retained.

Outlier capture was attempted for current-1, sample 1; current-1, sample 2; current-1, sample 3; current-1, sample 5; current-1, sample 6; current-2, sample 6. GDB was unavailable: only kernel thread wait states were recorded, with no debugger attachment or stack trace.

Standard hosted job	Verified target results	Duration	Median
current-1	8 passed, 0 failed	214.74–296.77s	243.90s
current-2	8 passed, 0 failed	192.91–291.52s	218.75s
current-3	8 passed, 0 failed	101.78–105.03s	102.67s
current-4	8 passed, 0 failed	71.80–80.65s	75.94s

All 32 target results, source/tree identity and cleanliness, observer hash,
selection/execution commands, and log hashes were verified. No profile exhausted
its experiment budget and no failure was retried. This is nonrecurrence evidence,
not a fix or performance qualification. The temporary workflow must not merge.

The 291.52-second successful sample in current-2 continued writing in every
10-second observation window. Its sampled target CPU time was about 80.27s and
aggregate host I/O-pressure time about 86.41s; a thread was observed in
folio_wait_bit_common. This is consistent with storage delay in that sample.
It is not a per-operation attribution or proof of the historical timeout's cause.

A separate investigation-only probe exercised the actual Dynamic publication
helper on a current-thread runtime with one blocking worker: queued completion
and registered wakeup, caller cancellation while queued, and cancellation after
completion. All three cases recovered and then completed real append/apply,
readback, and snapshot work. A deliberately stalled helper made the wakeup
detector fail; restoring the exact source returned all three to green. This
narrows those specific scheduling/cancellation hypotheses. It does not reproduce
the historical failure, and is not presented as fix-removal evidence or a fully
qualified production change.

The original failure and its unchanged retry used the same recorded Ubuntu image
version as the completed campaign checks. No runner-image upgrade explains those
passes. The two recent failed normal main CI jobs also passed this exact target;
their failures belong to other tests.

No runner purchase, timeout increase, dependency upgrade, or #856 closure is
justified by these results. A causal correction still needs a failing trace or a
demonstrated reachable failure mechanism with a preserved detector.

[VerifiedOrganic]

A bounded runtime comparison is now running on two ordinary ubuntu-latest runners: run 35239283870.

Each runner builds both fixed sources before four samples in baseline/variant/variant/baseline order. Baseline is main 75044f43852cd816d9734d03914515395bf20c69; variant 64c2ccda4ee5714025f25a458019272349d5e3d1 changes only this target's Tokio attribute from current-thread to multi-thread with one worker. The existing bounded SQLite blocking handoff is available only on the latter runtime. The original workload, 300-second guard, other deadlines and durability/recovery assertions are unchanged. A host stops at its first failed sample; no failure retries are part of this experiment.

A local exploratory pair passed in 81.27s and 62.08s respectively. That comparison does not establish the original failure's cause or a fix. Independent review and a Dynamic-profile real-commit scheduling probe are also in progress. The temporary harness branch is not a merge candidate; #856 remains open.

[VerifiedOrganic]

The fixed same-host runtime comparison completed on two ordinary GitHub-hosted runners: run 35239283870. Each ran main, the runtime-only variant, the variant again, then main after compiling both sources. All eight samples passed; no retry or stack probe was used.

Runner	Main sample seconds	Runtime variant sample seconds
paired-1	105.42, 104.42	89.06, 88.64
paired-2	103.47, 102.54	86.58, 85.50

Artifact checks verified the exact source heads/trees, clean source before and after, original command, one selected target, fixed sample order and log hashes. Main is 75044f43852cd816d9734d03914515395bf20c69; the attribute-only variant is 64c2ccda4ee5714025f25a458019272349d5e3d1. The 4,300 sequential commands, 300-second guard, disk-backed storage and recovery/readback assertions are unchanged.

A separate local Dynamic SQLite experiment exercised a real commit hook and independently spawned runtime task. Progress, cancellation ownership and rollback passed. Switching that probe to current-thread, and separately bypassing the actual handoff, each failed as expected; restoring the original probe restored all three passes. This establishes the scheduling mechanism, not the cause of the historical timeout, and is not a removal detector for the consumer fixture itself.

PR #886 contains the narrowly scoped fixture correction and rationale. It matches the SDK runtime flavor with one async worker and enables the existing bounded blocking handoff. Current-thread execution remains supported with documented inline writes; this scenario changes scheduler coverage. Full local and hosted qualification are running. Neither this comparison nor the probe reproduces the historical failure, so #856 remains open and the original cause remains unresolved.