fix(sync): stop heart-beating a dead owner forever — cures the _Activity/import-* NotFound storm by rbuergi · Pull Request #134 · Systemorph/MeshWeaver

rbuergi · 2026-06-30T19:02:48Z

The bug you keep hitting: `{Partition}/_Activity/import-{fingerprint}` heart-beaten forever

A sync subscription whose owner dies after a successful subscribe kept heart-beating the gone owner every interval forever. JsonSynchronizationStream posted HeartBeatEvent fire-and-forget, with the change feed as the sole recycled-grain detector — which misses the case where the owner dies without a node Created/Deleted change-feed event.

That is exactly a one-shot import-activity lock:

The importer writes {Partition}/_Activity/import-{fingerprint} through a cached GetMeshNodeStream(activityPath).Update handle → opens a heart-beated sync stream.
The import finishes; the dedicated off-router import hub is disposed.
The node persists as history, so there is no change-feed pulse to drive resubscribe.
The cached handle's heartbeat keeps posting to the gone address → [ROUTE] NotFound every interval, per partition, for the life of the silo — the storm that pegs CPU and trips /healthz (local e2e portal → pod killed).

Fix (one place: the heartbeat)

The heartbeat now observes its delivery instead of fire-and-forget:

A healthy owner has no ack → the observe simply times out → ignored (the normal case).
A terminal NotFound means the owner address is gone — a recycled/deactivated grain reactivates on the heartbeat post and acks, so only a permanently-gone owner NotFounds → tear the keep-alive down.

Keep-alive is preserved (the heartbeat is still posted + processed). We only add teardown on a terminal NotFound. Recycled-grain recovery is unchanged — it's change-feed-driven, and a reactivatable owner never NotFounds.

Test — the dedicated repro at this exact shape

HeartbeatStopsWhenOwnerDiesAfterSubscribeTest: the owner serves a real type so the subscribe succeeds and the heartbeat runs, then a flag flips it dead so every further heartbeat NotFounds with no change-feed event (the _Activity/import-* shape). After death the heartbeat count must freeze.

Before the fix: RED — Expected 10 to be ≤ 1 (climbs ~one-per-interval).
After: green.

Complements NonExistentOwnerNoHeartbeatStormTest (owner never existed) — this is the owner-died-after-subscribe half. Regression-checked green: ChangeFeedResubscribeCoalesceTest and ResubscribeOnOwnerDisposeTest (recycled-grain recovery).

Scope

This is the storm cure (the CPU peg / liveness-trip). The related cleanups discussed — bundling the AI static-repo sources (so Skill can't be silently un-imported) and the partition→_Access(public-read)→write-under-System sequencing — are a separate follow-up PR.

🤖 Generated with Claude Code

…ity/import-* NotFound storm A sync subscription whose owner DIES after a successful subscribe kept heart-beating the gone owner every interval forever. JsonSynchronizationStream posted HeartBeatEvent fire-and-forget with the change feed as the SOLE recycled-grain detector — which misses the case where the owner dies WITHOUT a node Created/Deleted change-feed event. That is exactly a one-shot {Partition}/_Activity/import-{fingerprint} lock: the importer writes it through a cached GetMeshNodeStream(activityPath).Update handle, the dedicated off-router import hub is disposed when the import completes, and the node persists as history (no change-feed pulse). Result: [ROUTE] NotFound to that address every interval, per partition, for the life of the silo — the recurring import-activity storm that pegs CPU and trips the liveness probe (local e2e portal /healthz timeout → pod killed). Fix: the heartbeat now OBSERVES its delivery. A healthy owner has no ack, so the observe times out (ignored). A terminal NotFound means the owner ADDRESS is gone — a recycled/deactivated grain REACTIVATES on the post and acks, so only a permanently-gone owner NotFounds — so we tear the keep-alive down. Keep-alive is preserved (the heartbeat is still posted + processed); we only ADD teardown on terminal NotFound. Recycled-grain recovery is unchanged (still change-feed-driven; a reactivatable owner never NotFounds). Pinned by HeartbeatStopsWhenOwnerDiesAfterSubscribeTest (deterministic: owner serves a real type so subscribe succeeds + heartbeat runs, then a flag flips it dead so every heartbeat NotFounds with no change-feed event — the count must FREEZE; RED before the fix at ~one-per-interval). Complements NonExistentOwnerNoHeartbeatStormTest (owner never existed). Regression-checked: ChangeFeedResubscribeCoalesceTest + ResubscribeOnOwnerDisposeTest (recycled-grain recovery) green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR addresses an operational storm where a remote sync subscription continues to heartbeat an owner address that no longer exists (e.g., one-shot {Partition}/_Activity/import-* hubs disposed after a successful subscribe), causing recurring [ROUTE] NotFound traffic and CPU/liveness issues. The fix updates the heartbeat path in JsonSynchronizationStream to observe heartbeat delivery and tear down the keep-alive when a terminal NotFound is detected, and adds a deterministic regression test for the “owner died after subscribe, no change-feed pulse” scenario.

Changes:

Update JsonSynchronizationStream heartbeat logic to observe delivery and dispose keep-alive on terminal NotFound.
Add HeartbeatStopsWhenOwnerDiesAfterSubscribeTest to pin the “dies-after-subscribe” NotFound-storm scenario.
Update SyncStreamOptions.HeartbeatInterval documentation to reflect the new teardown behavior and clarify resubscribe responsibility (change feed vs heartbeat).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
test/MeshWeaver.Data.Test/HeartbeatStopsWhenOwnerDiesAfterSubscribeTest.cs	Adds a deterministic regression test ensuring heartbeats stop once an owner becomes permanently unavailable after a successful subscribe.
src/MeshWeaver.Data/Serialization/SyncStreamOptions.cs	Updates option documentation to explain heartbeat teardown semantics and separate it from change-feed-driven resubscribe.
src/MeshWeaver.Data/Serialization/JsonSynchronizationStream.cs	Observes heartbeat delivery and disposes keep-alive when heartbeats return terminal `NotFound`, preventing endless NotFound loops.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+                        h.Observe(hbDelivery)
+                            .Take(1)
+                            .Timeout(heartbeatInterval)
+                            .Subscribe(
+                                _ => { },   // a healthy owner has no heartbeat ack to send
+                                ex =>
+                                {
+                                    if (ex is DeliveryFailureException { Failure.ErrorType: ErrorType.NotFound })
+                                        keepAlive.Dispose();
+                                    // else: TimeoutException (no ack from a healthy owner) — ignore.
+                                });


… exception-as-control-flow Address Copilot review: the healthy (no-ack) case must not throw a TimeoutException every interval (exceptions-as-control-flow CPU/alloc cost). Switch to the Timeout(dueTime, other) overload that completes via Observable.Empty; a terminal NotFound still arrives fast as OnError → teardown. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rbuergi · 2026-06-30T19:10:12Z

Good catch @Copilot — switched to Timeout(heartbeatInterval, Observable.Empty<IMessageDelivery>()) so the healthy no-ack case completes instead of throwing a TimeoutException every interval. The terminal NotFound still arrives fast as an OnError before the timeout, so teardown is preserved. (3b… pushed.)

github-actions · 2026-06-30T19:28:54Z