QMDB indexer cannot reconnect once a namespace exceeds u32::MAX operation locations

[michaelkaplan13]

Summary

Two related defects prevented the chain indexer's QMDB publisher from reconnecting after a restart, permanently halting ingestion. Both are fixed on branch michael/qmdb-suffix-recovery (commits 2139fa0, 812146e).

1. u32 recovery ceiling. On reconnect, recover_writer_state rebuilt writer state by fetching an operation-range checkpoint over the entire history (from location 0 to the watermark) and converting the location count to u32. Once a namespace exceeds u32::MAX (4,294,967,295) operations, the conversion fails on every attempt and the publisher can never reconnect — it spins in a 1s retry loop and ingestion stops. At ~100k txs/block the state namespace advances ~196k locations/block, crossing the ceiling around height ~21,800.

2. Producer redelivery wedge. A crash between a store commit and the durable-queue ack causes an already-uploaded block to be redelivered on restart. The producer validated writer cursors with no tolerance for the "already uploaded" case and retried forever on WriterOutOfSync.

Environment

• deploy.sh config: c8a.4xlarge validators, external chain store, --spammer-accounts 49152, ~280k TPS, ~100k txs/block.

Reproduction

• Ceiling: run a network under load until a QMDB namespace crosses ~4.29B operations, then restart the indexer. Observed error: cannot initialize QMDB writer from 4572755410 operation locations, repeating indefinitely.
• Wedge: kill the indexer in the window after a store commit but before the queue ack; on restart the redelivered block loops on WriterOutOfSync ... writer is at operation N, but finalized block starts at N-k.

Fix

• Suffix-window recovery: fetch a single-operation checkpoint ending at the watermark instead of the full prefix; reconstruct writer peaks from the pinned nodes. Recovery cost drops from O(history) to O(log n) and the u32 ceiling is removed. Requires a pin-aware reconstruct_peaks in exoware-qmdb (carried as the [patch] exoware git-rev bump in 2139fa0); no commonware API change needed.
• Redelivery skip: when both writer cursors are at or past the block's operation ranges, treat it as already-uploaded and skip capture instead of erroring.
• Validated: read-only recovery against a live store rebuilt a 4.57B-location watermark in seconds; reconnect covered by unit tests.