Pipelined Simplex with Stable Leaders (5ms Views)

[BrendanChou]

Summary

Integrates Commonware's pipelined Simplex and adds a stable-leader mode to alto, targeting 5–10 ms views. Validators now run one of two leader modes selected at deploy time, and every certificate consumer (indexer, follower, inspector, explorer) is told which certificate construction the network uses.

• Stable leaders (leader.mode: stable): one round-robin leader per term (term_length), proposals paced by delay_ms and issued up to optimistic_views ahead of certified ancestry, a 12 s stall timeout to evict a stuck leader, and native standard threshold certificates with a single signature per vote and certificate (no VRF seed). All three settings are required in stable mode. The deploy recipes use a 10 ms delay, 1,000-view terms, and 48 optimistic views.
• Rotating leaders (leader.mode: rotating): the previous VRF-seeded scheme, now using commonware's RandomVersion::V1 seed-to-leader mapping, shared by the validator and the explorer WASM through alto_types::ROTATING_ELECTOR.
• The two modes use different certificate wire layouts, so switching is not a rolling upgrade: regenerate the network and move validators, indexer, followers, inspector, and explorer together.

Consensus and validator

• Simplex config maps onto the new SkipPolicy/ForwardPolicy API; forwarding stays disabled and fast skips use the participant-count budget.
• Block timestamps are paced by delay_ms and block_size appends a configurable random payload. Verification waits until a block's timestamp is within 1 s of local time before accepting it and never rejects on the local clock, so the certify verdict stays deterministic across validators.
• A durable finalized-block queue backs the indexer uploader so uploads resume after a restart; concurrency and retry are configurable (backfiller_max_active, backfiller_retry_ms).
• Validators keep the full finalized history in immutable archives (no in-memory key index growth).
• Networking tuned for this workload: 1,500/s for the vote, certificate, and resolver channels, 3,000/s for the block-broadcast and marshal channels, one tracked peer set, a dedicated Rayon signature pool, and configurable storage/network buffer pools and blocking threads.
• Traces export to the monitoring host's Tempo endpoint at a configurable sample rate.

Indexer, explorer, follower, inspector

• The indexer takes --certificate-mode standard|vrf, embeds the explorer build, and serves it with the network identity, certificate mode, participants, and locations injected at runtime (/runtime-config.js). Under the deployer it reads its settings from --config (--hosts is accepted but unused).
• The explorer handles both modes (no seeds in standard mode; leaders resolved from block leaders via PARTICIPANTS), verifies certificates in a web-worker pool, and gains unit tests plus a WASM fixture check that runs in CI and in deploy.sh.
• The follower config gains a required certificate_mode; the inspector gains --certificate-mode and a DEFAULT_CERTIFICATE_MODE next to its default identity.

Deployment

• deploy generate adds --leader-mode, --leader-delay-ms, --leader-term-length, --leader-optimistic-views, --block-size, --traces-sample-rate, buffer-pool knobs, and --indexer to deploy an indexer instance that serves the explorer (or --indexers for external ones). deploy explorer emits CERTIFICATE_MODE and PARTICIPANTS.
• deploy.sh <stable|rotating> runs the full Global deploy end to end in the given leader mode (generate, verify the emitted artifacts exist, test and build the explorer, build the Graviton binaries, deployer aws create, print the explorer URL). A deploy unit test pins that indexer.yaml and the explorer config spell the certificate mode the same way.
• Explicit cross-compilation recipes via Docker Buildx and just: graviton (neoverse-512tvb, Graviton 3/4/5), graviton4 (neoverse-v2), and intel (emeraldrapids), each building validator and indexer with debug-symbol variants.
• Grafana panels for throughput, timeouts, and regional notarization/finalization latency; the deploy README covers both leader modes.

Dependencies

Commonware crates are the published 2026.9.0 release (commonwarexyz/monorepo#4663, tag v2026.9.0), which includes pipelined Simplex (#3416) and the deployer retry fix (#4670). The alto workspace version moves to 2026.9.0 to match.

Hardening (final review pass)

• Block payload bound: Block (and Notarized/Finalized) take a codec configuration for the payload size. Validators decode every block from the network, the broadcast buffer, and their archives with the configured block_size as an upper bound, so an oversized payload fails to decode before it is cached or verified (the exact size is still enforced at verification); the indexer does the same when started with --block-size, and consumers that only trust certificates (follower, client, explorer) use an unbounded configuration.
• Indexer: request bodies are limited to the configured block size plus a fixed overhead (uploads with 2 MiB blocks no longer hit axum's default limit); certificates already held are deduplicated before the BLS verification instead of after it; the WebSocket fan-out skips lagged frames instead of dropping the client; state is bounded to the most recent --max-views (default 200,000; a deployed indexer can set max_views in indexer.yaml); the raw block cache keeps twice that many uploads in insertion order, and blocks carried by retained certificates stay retrievable by digest through an index maintained alongside the certificates, so historical uploads cannot make them unavailable; broadcast payloads are Bytes so subscribers share one buffer.
• Validator config: leader and, in stable mode, optimistic_views are required (no defaults), so a pre-upgrade YAML fails to parse instead of silently booting the wrong certificate mode; the proposal delay must be shorter than the leader timeout (now exported from alto_chain, and the deploy generator rejects --leader-delay-ms values that would fail that check at boot); block payloads are filled from a userspace PRNG seeded once per proposal rather than the OS CSPRNG.
• Explorer: verification workers always reply (errors included) so a failed wasm load or panic cannot wedge in-order release; a worker that crashes or reports a verification error (failed wasm init, panic) is replaced and repeated failures surface the existing banner instead of a worker silently swallowing its share of the feed; the verification queue is bounded (oldest artifacts shed) so slow devices stay at the live head, and shed artifacts are counted in a skipped flag on the next verified artifact so the timeline does not backfill them as timeouts; skipped views are back-filled as unknown/timed-out from notarizations and finalizations too, so stalls are visible in standard mode (gap detection runs outside React state updaters so StrictMode's double invocation cannot skip it); the maintenance page positions itself before first paint, moves via transforms, and re-measures its box on resize.
• Tooling and docs: just recipes resolve paths from the justfile location so they work from any directory; the deploy README states that data directories must be fresh (the storage page layout changed) and the --leader-delay-ms bound, and the indexer README documents --block-size, --max-views, and the memory they imply; the follower example starts from the tip because the indexer only retains recent views.

Cleanup (final pass)

• Removed test-only API from alto_chain (Leader::default and the default delay, term, and optimistic-view constants) and derived Leader's Deserialize with a term_length validator instead of a hand-written mirror of the enum. alto_chain::Leader no longer implements Default.
• CertificateMode carries its own spelling (ALL, as_str, FromStr), used by the indexer, inspector, and deploy generator instead of hand-spelled "standard"/"vrf" lists.
• Indexer: get_block resolves certificate-owned blocks through a digest index instead of scanning every retained certificate. The runtime-config script is a json! literal. --config no longer requires the unused --hosts.
• Explorer: the verification worker protocol is one message in each direction (no initialize handshake, no echoed fields). Shed artifacts surface as a skipped flag on the next verified result. The maintenance page reveal and resize handling share one effect.
• Follower Source uses an associated Scheme type and keeps only the methods the feeder and resolver call (block, notarized, listen). The inspector shares one client constructor across subcommands.
• deploy.sh no longer re-verifies the generator's output (a deploy unit test pins the certificate-mode spelling). Docker bake variables nothing set were inlined, and duplicated computations in the validator and generator were removed.
• Deleted tests that pinned commonware or clap behavior rather than alto code, and restored rotating-mode coverage: the indexer simulation runs in both modes (test_indexer, test_indexer_rotating) and asserts seeds are uploaded only with VRF certificates.
• Behavior notes: an idle backfill consumer now observes shutdown promptly, and the 3,000/s quota constant is named BLOCK_CHANNEL_QUOTA_PER_SECOND for the block-broadcast and marshal channels it applies to (quotas unchanged).
• alto-client: ClientBuilder::new (formerly new_with_scheme) takes an initialized certificate verifier, and the identity-based Client::new, Client::new_standard, and ClientBuilder::new_standard constructors and the VrfScheme default type parameter are gone, so callers name the scheme explicitly.
• Block::genesis() lives in alto_types and is shared by the validator, follower, and tests. optimistic_views has no serde default and must appear in stable-leader YAML (the generator always emits it). engine::Config dropped three unused fetch fields, Scheme::signer and Kind::to_hex are gone, and the indexer serves only exact explorer asset paths since the explorer has no path routes. The worker pool hands verified artifacts to the consumer through drain() instead of a callback, so one bounded queue covers queued, active, and completed work.

Validation

• cargo clippy --all-targets --all-features -- -D warnings, cargo fmt --check, cargo doc --no-deps --document-private-items, cargo udeps --all-targets
• cargo test --workspace including all chain simulations (test_1k), against the published 2026.9.0 crates
• CI=true npm run build (wasm-pack build, check-wasm.mjs, ESLint, production build) and the explorer jest suites
• Generated stable and rotating configurations (local and remote) and ran local networks with an indexer in both modes, debug and release

[cloudflare-workers-and-pages]

Deploying alto with    Cloudflare Pages

Latest commit:	a6be213
Status:	✅  Deploy successful!
Preview URL:	https://e53eacda.alto-8k4.pages.dev
Branch Preview URL:	https://bc-optimistic.alto-8k4.pages.dev

View logs

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit d37625e. Configure here.}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 80.47554% with 427 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.95%. Comparing base (6f7cf02) to head (a6be213).

Files with missing lines	Patch %	Lines
validator/src/main.rs	33.85%	84 Missing ⚠️
indexer/src/main.rs	48.70%	79 Missing ⚠️
follower/src/main.rs	13.11%	53 Missing ⚠️
types/src/wasm.rs	0.00%	51 Missing ⚠️
inspector/src/main.rs	0.00%	40 Missing ⚠️
follower/src/archive.rs	34.09%	29 Missing ⚠️
types/src/consensus.rs	65.82%	27 Missing ⚠️
deploy/src/main.rs	96.12%	25 Missing ⚠️
types/src/block.rs	84.26%	14 Missing ⚠️
indexer/src/lib.rs	96.95%	8 Missing ⚠️
... and 7 more

@@             Coverage Diff             @@
##             main     #190       +/-   ##
===========================================
+ Coverage   67.36%   78.95%   +11.59%     
===========================================
  Files          30       31        +1     
  Lines        5619     7013     +1394     
===========================================
+ Hits         3785     5537     +1752     
+ Misses       1834     1476      -358     

Files with missing lines	Coverage Δ
chain/src/application.rs	98.50% <100.00%> (+0.81%)	⬆️
chain/src/engine.rs	98.95% <100.00%> (+0.09%)	⬆️
chain/src/indexer/backfiller/consumer.rs	88.23% <100.00%> (+2.88%)	⬆️
chain/src/indexer/backfiller/state.rs	100.00% <100.00%> (+0.57%)	⬆️
chain/src/indexer/mocks.rs	96.47% <100.00%> (ø)
chain/src/indexer/pusher.rs	96.52% <100.00%> (+0.09%)	⬆️
client/src/utils.rs	93.33% <ø> (ø)
follower/src/application.rs	58.44% <100.00%> (ø)
follower/src/engine.rs	97.34% <100.00%> (-0.42%)	⬇️
follower/src/feeder.rs	77.08% <100.00%> (ø)
... and 18 more

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6f7cf02...a6be213. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.