fix(cluster)

[EnRaiha]

Bug Description

raft readiness timeout is hardcoded to 30s — large replay wedges the node

Steps to Reproduce

1. Build up a raft log >150k entries (long write bursts / big migration).
2. Restart node.
3. await_cluster_ready uses RAFT_READY_TIMEOUT = Duration::from_secs(30) (nodedb/src/bootstrap/cluster_ready.rs:54) while the replay path allows 300s.
4. Replay >30s → raft_gate.fail("raft readiness timeout after 30s") → ReadyGate Failed → process alive, all ports dead (deadlock).

Expected Behavior

Node finishes replay and becomes ready regardless of replay duration; operator can raise the timeout for large logs.

Actual Behavior

Boot fails after 30s with no way to override; node deadlocks (process alive, ports dead) until force-stop + wipe.

Environment

• 2026-08-26: rebuild with 446k-entry replay → 2 boots failed exactly this way. Node unreachable until force-stop + wipe.
• OS: Debian (LXC), nodedb 0.5.0

Proposed Fix

New [server] raft_ready_timeout_ms config key (Option, default 30_000 = unchanged behaviour). Passed through main.rs → await_cluster_ready. Config: raft_ready_timeout_ms = 300000.

Files: section.rs, cluster_ready.rs, main.rs. cargo check clean.

Closes #253

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[farhan-syah]

Correction (later). The phase-split analysis below is wrong. The raft log load runs inside bootstrap_data_plane at main.rs:177, fully awaited, while the 30s timer does not start until await_cluster_ready at main.rs:254 — so the scan was never inside that window and moving the deadline would not have helped. See the correction on #253 for what the 30s actually covers and what the report still needs.

Closing. The bug is real, but the 30s deadline spans two phases with different failure semantics, and a config key asks the operator for a number derived from their log size.

ready_watch flips on the first metadata entry applied — "the leader-election no-op or a replayed entry" (raft_loop/tick/apply_committed.rs:82-91). The 30s therefore covers raft storage open, log and snapshot load, loop start, election, and first apply, all charged together.

Phase	Duration depends on	Correct bound
Storage open, log and snapshot load	Log size, disk speed	None. It completes or it errors.
Election, first apply	Peer reachability	A deadline. This is the race the gate exists for.

The gate's own comment (cluster_ready.rs:47-52) names only the second: "eliminates the restart-window race where the first DDL would observe metadata propose: not leader because election had not yet completed". Local log load is charged against an election budget it has nothing to do with.

The fix

Start the readiness deadline when the raft loop begins ticking, not when await_cluster_ready is entered.

30s then means 30s to elect and apply the first entry. That is correct for a 1k-entry log and a 10M-entry one alike, and it still catches a wedged group in 30s rather than 300s. It needs one more signal out of start_raft — loop-running, alongside the existing ready watch — and no config key.

The worse half of #253 is untouched

The issue reports two symptoms: boot fails at 30s, and the process stays alive with every port dead until force-stop and wipe. This PR addresses the first.

await_cluster_ready(...).await? propagates to server_main, and returning Err from main should exit the process. The reporter observed it does not. A raised timeout hides that in the happy case; every future failure of this gate reproduces the deadlock at 300s instead of 30s.

#253 stays open for both.

Secondary

• The title is fix(cluster) with no subject.
• The default is written twice: Option<u64> defaulting to None, unwrap_or(30_000) at main.rs:246, and a doc comment stating 30_000. Repo style is a serde default fn — default_max_connections and default_memory_limit sit in the same struct.
• No env var, while every neighbouring operator setting has a NODEDB_* equivalent in the same doc. In the reported environment an env var is the reachable lever during an outage.
• docs/getting-started.md reflows three unrelated tables alongside the one added row.