You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(release): re-issue an Unavailable release re-cut on the same activation when the store answers again (#7072) - #7087
During a 40-second database restart (#7068), three Hosting NodeTypes settled. For each one, the release create and its same-id re-cut were both answered Unavailable: the store could not be reached, nothing was written, and the log line said a retry with the same id is meaningful. The settle stamped unreleasedBuildPath. After that, nothing on the still-running pod made the create again:
LateReleaseAdoption.Landings only adopts a node that somebody writes.
CompleteInheritedRelease runs only on the NEXT activation.
The release eventually came from unrelated events: another pod's boot after 76 minutes, or the next compile after about 3 hours. The third stamped id was never created.
Root cause:src/MeshWeaver.Compiler.Pipeline/LateReleaseAdoption.cs. Install (around line 271 on main) had no path that re-issues a stamp on the activation that wrote it. The stamp also carried no machine-readable failure class, so it could not tell an Unavailable create (one the store never saw) from a refusal. That class was lost in NodeTypeBuildState.Bounded's catch arm, where ReleaseCreateOutcome.Failed(reason, path) kept only the sentence.
The fix
The failure carries a class.ReleaseCreateOutcome.FailureClass is a ReleaseFailureClass: Unavailable, Expired, Refused or Faulted, an open set of string constants. It is set in NodeTypeBuildState.Classify at the catch arm, from the typed NodeCreationRejectionReason that IMeshService.CreateNode puts on Exception.Data. The message text is never parsed.
The settle and the batch bake stamp it as NodeTypeDefinition.UnreleasedBuildFailureClass. It is masked as operational state and mirrored in NodeTypeCompileState.
Adoption and prebuilt supersession clear it together with the path.
The activation that holds an Unavailable stamp re-issues it. This is LateReleaseAdoption.ReissueWhenTheStoreAnswers. It is gated by UnavailableObligation, which uses the same guards as InheritedObligation (settled, this framework, the stamped id names the recorded bytes) and also requires class Unavailable.
The re-issue goes through the same TryCreateReleaseNode, at the SAME stamped id, as System.
It is idempotent: a late landing at that id is adopted, and the adoption write re-checks the stamp.
Every other class stays a named failure. Refused is never re-issued. An Expired create may still land, and Landings adopts it if it does.
A general primitive for re-issuing on recovery.StoreReachability.StoreAnswersAgain and StoreReachability.ReissueWhenStoreAnswers live in MeshWeaver.Mesh.Contract. ReissueWhenStoreAnswers re-issues the same write, with the same id, once there is evidence the store is back:
Evidence is a declared ChangeFeedGap (the LISTEN connection resuming, which a database restart produces) or a commit announced in the write's partition (the store must have accepted it).
If a re-issue is answered Unavailable again right after a partition commit, that commit has been shown not to count as evidence for this write. Later re-issues then wait only for a feed reconnect. A busy partition therefore cannot turn this into a retry loop, and no count or clock is involved.
Why this signal (no-band-aids)
No timer, poller or widened bound. A timer that retries "in case the store is back" is exactly what StoreReachability already refuses, and it would add load to the bottleneck at the worst moment.
No health probe. A probe would be a second, weaker opinion of facts the feed already reports.
Why the gap and commit feed. They are the facts this process already gets about whether the store is back. Path resolution and the stream cache already use them for the same reason. They stay silent exactly as long as a re-issue would be pointless.
This is the framework-level gap. "Retry this Unavailable write when the store comes back" is the same defect class as the control-instance finding rbuergi/Feedback/unavailable-write-retried-76-min-later. ReissueWhenStoreAnswers is the reusable shape for it, and the release re-cut is its first caller. Other sites can adopt it in their own changes.
Evidence
AnUnavailableReleaseRecutIsReissuedWhenTheStoreAnswersTest in MeshWeaver.Compiler.Pipeline.Test is an isolated fault-injection test on a real monolith mesh. It uses the new shared injector FaultInjectingStorageAdapter.RefuseWritesAsUnreachable, which fails writes with a driver DbException carrying SQLSTATE 08006. The production arm logs [CreateNode] STORE UNREACHABLE and answers Unavailable. The test then checks:
the stamp lands with class Unavailable;
while the store still refuses, the pointer does not move;
after the store comes back and a feed gap is declared, the SAME activation creates the release at the ORIGINAL stamped id and adopts it.
Negative control (run locally): with the re-issue watch replaced by Observable.Never in Install, the main case is red at its final wait. The message reads "Expected the observable to emit a value matching the predicate within 72s … but it did not", and UnreleasedBuildFailureClass = Unavailable is still standing.
Second negative control:ARefusedRecut_IsNeverReissued_HoweverOftenTheStoreAnswers. A Refused stamp is never re-issued under the same signals.
Classification tests:ReleaseRecutReportsWhyItFailedTest checks the classification through the real Bounded chain (Unavailable, Refused, Expired, Faulted).
Local runs: release-related Compiler.Pipeline tests 193 of 193 passed. Graph.Test ownership, mask and mirror guards 19 of 19 passed. Every touched project builds with -c Release -warnaserror and 0 errors (after merging main).
Doc/Architecture/FaultInjectionHarness documents the new injector and case 9.
The NodeTypeCompilation table row mentions the class.
Public surface
This PR only adds: a NodeTypeDefinition property, the ReleaseFailureClass constants, two StoreReachability extension methods, and a test-support injector method. Nothing is removed and no interface member is added, so no pairing lines are needed. No i18n keys are added.
After deploy
No recycle is needed for the fix to take effect. The watch is installed on each new NodeType activation, and the roll creates new activations. No stamp from before the fix carries a class, so this fix never re-issues any of them. The next activation's one-time cut (#6056) still handles them.
…vation when the store answers again (#7072)
A release re-cut refused as Unavailable (the store could not be reached,
nothing written) was stamped as unreleasedBuildPath and never re-issued by
the activation that wrote the stamp; only another pod's boot or the next
compile created a release (76 min to 3 h on memex, Refs #7068).
- ReleaseCreateOutcome carries a ReleaseFailureClass, classified at the
arm from the typed rejection reason; stamped as
NodeTypeDefinition.UnreleasedBuildFailureClass (settle and batch bake).
- StoreReachability.StoreAnswersAgain / ReissueWhenStoreAnswers: re-issue
the same write when the store is evidenced back (a change-feed gap, or a
commit in the partition; after a contradicted commit, gaps only).
- LateReleaseAdoption re-issues an Unavailable stamp at the stamped id on
the live activation and adopts it. Other classes stay named failures.
- Fault injector RefuseWritesAsUnreachable + isolated regression test.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This PR addresses #7072 by enabling a live NodeType activation to reissue an unavailable release creation at its original ID when store recovery is observed.
Changes:
Preserve machine-readable failure classes through release creation, compile state, and batch-bake stamps.
Add recovery-driven reissue logic using change-feed reconnects and partition commits.
Add fault-injection coverage and document recovery behavior and limitations.
…ource, record a re-issue's verdict (review on #7087)
- ReissueWhenStoreAnswers observes evidence for its whole life: evidence
arriving during an in-flight attempt is retained and starts the next one
after an Unavailable answer (gaps are not replayed, so it was lost).
- Each attempt remembers whether a commit or a gap started it; only a
commit-started Unavailable answer disproves commits as evidence.
- A re-issue answered with a verdict writes its class and reason to the
record through the owner stream, only while the same Unavailable
obligation stands, so a rebuilt watch never re-issues a judged create.
- Injector RefuseWritesAsJudged; deterministic regressions for each.
StoreAnswersAgain keeps its public signature.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Coordinate this watcher with the inherited create's outcome. When an activation starts with an Unavailable stamp, CompleteInheritedRelease attempts the same create, but its failure branch only logs and returns (lines 238–245). If that attempt receives Refused, the stored class still says Unavailable, so this new watcher issues the judged create again on the next recovery signal. Persist the inherited attempt's terminal class and reason through the owner stream, and coordinate the handoff so recovery cannot act on its stale class. Add a case starting with an inherited stamp; the new integration tests activate the owner before stamping and do not exercise this path.
Results for commit 30f4e66. ± Comparison against base commit 23a1840.
This pull request removes 44 and adds 39 tests. Note that renamed tests count towards both.
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---)
--- End of inner exception stack trace ---, expected: True)
--- End of inner exception stack trace ---, isDenial: True)
---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
---> (Inner Exception #1) System.InvalidOperationException: boom<---
---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
…
Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "5475925c-e4dc-46ff-a440-7b3f4a4495a6")
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.AggregateException: One or more errors occurred. (Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.)
---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
--- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.PublishStorageFaultIsNotAConflictTest ‑ AStorageFault_IsA500_NamingTheError(fault: System.InvalidOperationException: landing faulted
---> System.UnauthorizedAccessException: Access to the path '/data/modules/activation.d/MeshWeaver.Markdown.Export.json' is denied.
--- End of inner exception stack trace ---)
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
---> System.UnauthorizedAccessException: Access denied
--- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.AnUnavailableReleaseRecutIsReissuedWhenTheStoreAnswersTest ‑ ACommitFromTheStillUnavailableCallback_IsWeighedAfterTheFailure_NotStartedInsideIt
MeshWeaver.Compiler.Pipeline.Test.AnUnavailableReleaseRecutIsReissuedWhenTheStoreAnswersTest ‑ ACommitFromTheStillUnavailableCallback_KeepsTheRetainedReconnect
MeshWeaver.Compiler.Pipeline.Test.AnUnavailableReleaseRecutIsReissuedWhenTheStoreAnswersTest ‑ ACommitStartedFailure_LeavesOnlyReconnectsAsEvidence
MeshWeaver.Compiler.Pipeline.Test.AnUnavailableReleaseRecutIsReissuedWhenTheStoreAnswersTest ‑ AGapDuringAnAttempt_StartsTheNextAttempt_WhenThatOneIsUnavailable
MeshWeaver.Compiler.Pipeline.Test.AnUnavailableReleaseRecutIsReissuedWhenTheStoreAnswersTest ‑ AGapDuringTheHeldEvidenceAttempt_IsRetained_AndStartsTheNextAttempt
MeshWeaver.Compiler.Pipeline.Test.AnUnavailableReleaseRecutIsReissuedWhenTheStoreAnswersTest ‑ AGapStartedFailure_KeepsCommitsAsEvidence
…
♻️ This comment has been updated with latest results.
PR run 39965744393 killed MeshWeaver.Compiler.Pipeline.Test at the 480 s
per-project cap (exit=124 TIMEOUT, recorded as HOST_CRASHED), still progressing
through its classes: no hang, every TEST START before the cut had its TEST END
except the class running at 480 s. The project measured 388 s solo on PR run
39964851393 (81% of the cap, already past the 288 s headroom threshold while
listed at 227 s), and #7072's regression class adds three deliberate 36 s
negative waits (NotEmit(TestTimeouts.Quick) on CI). Per the split rule in
shard-assign.sh the cap is not raised; the project is split instead.
ShardWeightHeadroomGuard fails on the measured weight unsplit (500 s / 1 part)
and passes with three parts.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ure before going idle (review on #7087)
Two re-entrancy defects in ReissueWhenStoreAnswers, both from Copilot review:
- Evidence was the first Merge source, so evidence a feed delivered while
being subscribed ran Start before the hot outcomes subject had a
subscriber; an attempt answering in that window lost its outcome and the
sequence stayed in flight for good. Outcomes are now connected first.
- The attempt went idle before wasUnavailable/onStillUnavailable ran, so a
commit those callbacks published re-entered and started a nested attempt
before a commit-started failure disproved commits, clearing a retained
reconnect. The attempt now stays in flight until its outcome is weighed.
Three deterministic cases; each fails without the fix and passes with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The shared reentrant recovery state machine needs human concurrency review, with unresolved disposal, callback-error, and inherited-attempt coordination issues.
Forward exceptions from wasUnavailable and onStillUnavailable to the returned observable's OnError. Both run inside the terminal Subscribe callback, whose error delegate does not catch exceptions thrown by its own OnNext handler. A callback failure can therefore escape into the producer instead of notifying the recovery subscriber; invalidation-feed delivery catches and logs escaping subscriber exceptions, so the recovery watch can stop without receiving the error it needs to handle. Catch failures from these supplied callbacks, mark the sequence finished, and notify the outer observer. Cover both a throwing predicate and a throwing still-unavailable callback.
…ce; stop on dispose (review on #7087)
- CompleteInheritedRelease leaves an inherited Unavailable stamp to the
recovery watch: two attempts no longer meet the store at one id, and a
verdict is written back rather than only logged.
- The watch counts the stamp itself as evidence (a record the store accepted
after the refused create): a reconnect or commit that preceded the watch is
never replayed to it. A failure of that re-issue disproves nothing.
- onStillUnavailable is told whether commits still count, so the warning no
longer blames a commit for a reconnect-started attempt.
- Disposal stops the state machine first; Start never subscribes a write after.
Each new case fails with its fix removed and passes with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Forward exceptions from wasUnavailable and onStillUnavailable to the returned observable. These delegates run inside the internal Subscribe callback; its error handler does not catch exceptions thrown by that callback. When a pending attempt delivers its outcome later, a delegate failure escapes outcomes.OnNext without notifying the outer observer, so the recovery watch can remain waiting without an outcome or error. Catch each delegate failure and terminate through observer.OnError, or move the delegate calls into fault-forwarding operators. Add regression cases for both throwing delegates with an outcome delivered after subscription setup.
…onnected (review on #7087)
StartWith(true) emitted the caller's held evidence before EvidenceBySource was
subscribed, so a reconnect delivered while that first attempt was being
subscribed was lost to the non-replaying feed; an Unavailable answer then left
an idle partition with nothing to start the next attempt. The held evidence is
now merged in after the live commits and gaps are connected, through the same
serialised pipeline. AGapDuringTheHeldEvidenceAttempt_IsRetained_AndStartsTheNextAttempt
fails without the fix (1 attempt, expected 2) and passes with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The production recovery state machine warrants human validation, particularly the unresolved distinction between optimistic invalidations and durable commit evidence.
…ence (review on #7087)
The invalidation feed also carries optimistic notifications: a compile settle
publishes its own-stream update before the persistence sampler stores it. One
of those starting a re-issue that is answered Unavailable disproved commits
for good, so the durable commit that follows when write connections recover
could never start the next attempt. IMeshInvalidationFeed gains DurableCommits
(default: never), fed by InProcessMeshChangeFeed only from the storage relay
path, and the evidence reads it instead of every invalidation.
AnOptimisticNotification_IsNotEvidence_AndADurableCommitStillStartsTheReissue
fails without the fix and passes with it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The recovery state machine coordinates reentrant callbacks, activation teardown, and durable release writes, warranting final human concurrency review.
…wser step really do; 8 -> 10 shards
Review on #7102:
- the guard summed the weight table, so unlisted suites (scheduled at
DEFAULT_WEIGHT) counted nothing and stale rows counted; it now sums every
test/**/*.csproj shard-assign.sh discovers, at its table weight or the
default;
- it reserved BROWSER_STEP_BUDGET only, while the last attempt's
--kill-after grace, the orphaned-apt reap and the smoke print's timeout run
after that deadline; it now reads all of them (588 s end to end).
Measured that way the suite is 3,832 s against ~462 s of test room per
shard, so eight shards fail the guard (479 > 416). Ten pass (383; ~410 once
#7087 splits Compiler.Pipeline).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…not a second gap signal (#7066)
Consolidates with #7087: the re-run trigger is StoreReachability's one
"the store answers again" evidence (feed reconnect or durable commit),
with its rules (evidence retained while an attempt is in flight; a
commit-started re-run that fails again disproves commits). The duplicate
StoreRecoveryRerun.RecoverySignal is deleted; the bounded early backoff
(StoreRecoveryRerunPolicy) stays as a documented addition. A faulted
evidence feed now surfaces instead of leaving a re-run waiting.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…asure weights, 6 → 8 shards (#7102)
* ci(tests): re-measure shard weights, re-split, 6 -> 8 shards so a stalled browser step no longer cuts a shard
Shards were cancelled at the 20-minute job cap on PR runs 40011143666 and
40011313475 and on main run 40027189653 (attempt 1 each), with every
project still progressing. Memex.Portal.Shared.Test was not hung: part 3/3
entered a new class every few seconds until the cut and measures 315-343 s
on green runs.
Two causes stacked:
- the browser step spent its whole 480 s budget on a stalled apt mirror
(normally 20-70 s);
- shard 4 owed ~880 s of tests against a printed ~454 s (stale weights),
and the re-measured suite is ~3,900 s, i.e. ~650 s per shard at six
even when balanced, while the cap leaves ~570 s after a stalled step.
Changes:
- weights re-measured over three green full main runs (2026-10-23);
- splits chosen from trx per-class durations through the workflow's
sorted every-Nth split (which reproduces the measured parts exactly):
Portal.Shared 3->4, Graph 3->4, Messaging.Hub 2->3 (part 2/2 was
409-416 s, 87% of the per-project cap, invisible to the total/N
guard), Hosting 3->2 (now 223/226);
- CI_SHARD_TOTAL 6->8, matrix and upload steps; SHARD_TOTAL now reads
CI_SHARD_TOTAL;
- new guard ShardWeightHeadroomGuard.TheIdealShardLoadFitsTheShardJobCap
reads the cap, the browser budget and the shard count from the workflow.
Compiler.Pipeline's line is left to #7087.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ci(tests): shard budget guard measures what the scheduler and the browser step really do; 8 -> 10 shards
Review on #7102:
- the guard summed the weight table, so unlisted suites (scheduled at
DEFAULT_WEIGHT) counted nothing and stale rows counted; it now sums every
test/**/*.csproj shard-assign.sh discovers, at its table weight or the
default;
- it reserved BROWSER_STEP_BUDGET only, while the last attempt's
--kill-after grace, the orphaned-apt reap and the smoke print's timeout run
after that deadline; it now reads all of them (588 s end to end).
Measured that way the suite is 3,832 s against ~462 s of test room per
shard, so eight shards fail the guard (479 > 416). Ten pass (383; ~410 once
#7087 splits Compiler.Pipeline).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ci(tests): stage the browser font set once and restore it from a cache; no apt in a shard; 10 -> 8 shards
The shards' browser step ran `playwright-core install --with-deps` for its
apt half, which on ubuntu-24.04 adds only nine font packages. A stalled
Ubuntu mirror spent the step's whole 480 s on them, and that cut shards on
2026-10-23 (PR runs 40011143666 and 40011313475, main run 40027189653).
- New `browser-fonts` job ("Stage the browser font set"), parallel with the
build. On a cache miss it checks .github/chromium-fonts.list against the
pinned Playwright's `install-deps --dry-run` and downloads the .debs with a
private apt state (no system lock, no root). It extracts their font files
without installing them and saves the `chromium-fonts` cache. On a hit it is
a lookup.
- Shards restore that cache (step cap 1 min) and unpack it. The provision
step (step cap 6 min) now only downloads the browser (180 s budget, 90 s per
attempt), installs the fonts and runs the smoke print. No apt, no lock, no
reap. A missing cache is a warning; the smoke print still decides.
- ShardWeightHeadroomGuard reads the browser's cost as the two steps'
`timeout-minutes` (420 s) and checks that the provision step's inner bounds
fit its cap. Eight shards pass it (513 <= 567 s); seven fail (586).
- BrowserProvisioningGateGuard: TheTestShardsNeverReachAPackageMirror forbids
`--with-deps` / `install-deps` / `apt-get` in the test job. A repo with no
`--with-deps` provisioner is no longer a vacuous failure.
- ToImmutableDictionary for the weight lookup (review on #7102).
Verified locally in ubuntu:24.04 as a non-root user: stage fetched 9 packages
(21 MB, 192 font files), install took fontconfig from 13 to 47 faces, and a
list missing fonts-wqy-zenhei made stage fail naming it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ci(fonts): the drift check classifies the dry run's exit status instead of swallowing it
check-workflow-shell flagged `asked=$(… 2>/dev/null | … || :)`: a failed or
stalled `install-deps --dry-run` became an empty list with no trace. The dry
run's output now goes to a file with stderr kept. Exit 0/1 is read (1 is how
playwright-core reports missing deps). 124/137 is a named stall, and anything
else is a named failure, each printing the dry run's output. An empty font
list still refuses the stage, again with that output.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* ci(fonts): a stalled font-cache download ends as a miss, never as the restore step's cap
Review on #7102: the restore step's 1 min cap equalled its 1 min
SEGMENT_DOWNLOAD_TIMEOUT_MINS, whose clock starts after the cache lookup. So
the cap could fire first and fail the shard over an optional input, before
the browser step's warning and the tests. The restore cap is now 2 min, and
the new TheFontRestoreMissesBeforeItsCap guard requires at least 30 s of room
past the abort.
The provision step goes 6 -> 5 min (download budget 180 -> 150 s, font
install 70 -> 40 s; inner bounds 285 of 300 s), so the browser's end-to-end
bound stays 420 s and eight shards still pass the shard budget (513 <= 567 s).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…same pod (#7066) (#7088)
* fix(prewarm): re-run a sweep a transient store fault faulted, on the same pod (#7066)
The boot warm-up sweep that feeds the nodetype_bake readiness gate ran once.
A ~40 s database restart (#7068) refused one pod's Admin/Build write, the gate
recorded Faulted, nothing re-ran the sweep, and the pod refused readiness for
74 minutes after the database was back, until it was replaced.
- MeshNodeError.Transient: the owner classifies a store fault's transience
where the exception still has its type (ClassifyPatchException) and sends it
on the record; StandingWatchRecovery.IsTransient reads it. Before this the
caller held "MeshNode Unknown ... NpgsqlException" with no inner exception
and could not classify it at all.
- StoreRecoveryRerun: a finite job faulted by a transient store fault is
re-run on the store's recovery signal (a ChangeFeedGap — the listener
re-opened its session), raced by a bounded backoff for the first re-runs
only; afterwards only the signal triggers one. Non-transient faults surface
unchanged and stay named.
- DynamicTypePreWarmerHostedService: wires it; the fault is recorded first
(gate Faulted, detail names the pending re-run), the re-run resets the
summary, re-offers the served-build witness and holds the bake barrier
while a timed re-run is pending.
- NodeTypeBakeGateState.MarkRerunning; MarkComplete supersedes an earlier
fault. A pod its fault already let serve is never taken out by a re-run.
Refs #7068
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(prewarm): HubFact bounds, no rooted Rx connection in the re-run trigger (#7066)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* refactor(prewarm): re-run the sweep on StoreReachability's evidence, not a second gap signal (#7066)
Consolidates with #7087: the re-run trigger is StoreReachability's one
"the store answers again" evidence (feed reconnect or durable commit),
with its rules (evidence retained while an attempt is in flight; a
commit-started re-run that fails again disproves commits). The duplicate
StoreRecoveryRerun.RecoverySignal is deleted; the bounded early backoff
(StoreRecoveryRerunPolicy) stays as a documented addition. A faulted
evidence feed now surfaces instead of leaving a re-run waiting.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(prewarm): answer review — aggregate-aware transience, live-only watch dedupe, re-seed on re-run (#7066)
- ClassifyPatchException: an aggregate is transient only when every branch
is (StorageFaults walked only branch 0), so a mixed aggregate never
crosses the wire flagged transient.
- RecoveryWatchSet: de-duplicates only LIVE recovery watches; a watch that
saw its recovery or faulted leaves the set (WatchForRecovery overload with
onTerminated), so a later regression gets a fresh watch.
- SweepAttempt: a re-run re-runs the cold seeding stages before the sweep,
re-offering adoption stamps the refusal discarded.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* test(prewarm): immutable accumulators in the #7066 tests (review)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* fix(prewarm): re-run retracts a regression it measures usable; path-only cycle check; watch set dispose race (#7066 review)
- NodeTypeBakeGateState.MarkOutcome: a usable outcome for a type that
already stands regressed (only a re-run can produce one) retracts it, so
a regression whose recovery watch died in the outage no longer latches
the pod. MarkRerunning forgets the fault (except on a pod the fault let
serve) so a mid-re-run retraction lands on Running.
- IsTransientStoreFault: cycles are detected on the recursion path only; a
repeated reference (Aggregate(f, f), two wrappers over one inner) is no
longer misread as a cycle and sent as non-transient.
- RecoveryWatchSet.Watch re-checks disposal after inserting its slot.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7072. Refs #7068.
The defect
During a 40-second database restart (#7068), three Hosting NodeTypes settled. For each one, the release create and its same-id re-cut were both answered
Unavailable: the store could not be reached, nothing was written, and the log line said a retry with the same id is meaningful. The settle stampedunreleasedBuildPath. After that, nothing on the still-running pod made the create again:LateReleaseAdoption.Landingsonly adopts a node that somebody writes.CompleteInheritedReleaseruns only on the NEXT activation.The release eventually came from unrelated events: another pod's boot after 76 minutes, or the next compile after about 3 hours. The third stamped id was never created.
Root cause:
src/MeshWeaver.Compiler.Pipeline/LateReleaseAdoption.cs.Install(around line 271 on main) had no path that re-issues a stamp on the activation that wrote it. The stamp also carried no machine-readable failure class, so it could not tell anUnavailablecreate (one the store never saw) from a refusal. That class was lost inNodeTypeBuildState.Bounded's catch arm, whereReleaseCreateOutcome.Failed(reason, path)kept only the sentence.The fix
ReleaseCreateOutcome.FailureClassis aReleaseFailureClass:Unavailable,Expired,RefusedorFaulted, an open set of string constants. It is set inNodeTypeBuildState.Classifyat the catch arm, from the typedNodeCreationRejectionReasonthatIMeshService.CreateNodeputs onException.Data. The message text is never parsed.NodeTypeDefinition.UnreleasedBuildFailureClass. It is masked as operational state and mirrored inNodeTypeCompileState.Unavailablestamp re-issues it. This isLateReleaseAdoption.ReissueWhenTheStoreAnswers. It is gated byUnavailableObligation, which uses the same guards asInheritedObligation(settled, this framework, the stamped id names the recorded bytes) and also requires classUnavailable.TryCreateReleaseNode, at the SAME stamped id, as System.Refusedis never re-issued. AnExpiredcreate may still land, andLandingsadopts it if it does.StoreReachability.StoreAnswersAgainandStoreReachability.ReissueWhenStoreAnswerslive inMeshWeaver.Mesh.Contract.ReissueWhenStoreAnswersre-issues the same write, with the same id, once there is evidence the store is back:ChangeFeedGap(the LISTEN connection resuming, which a database restart produces) or a commit announced in the write's partition (the store must have accepted it).Unavailableagain right after a partition commit, that commit has been shown not to count as evidence for this write. Later re-issues then wait only for a feed reconnect. A busy partition therefore cannot turn this into a retry loop, and no count or clock is involved.Why this signal (no-band-aids)
StoreReachabilityalready refuses, and it would add load to the bottleneck at the worst moment.rbuergi/Feedback/unavailable-write-retried-76-min-later.ReissueWhenStoreAnswersis the reusable shape for it, and the release re-cut is its first caller. Other sites can adopt it in their own changes.Evidence
AnUnavailableReleaseRecutIsReissuedWhenTheStoreAnswersTestin MeshWeaver.Compiler.Pipeline.Test is an isolated fault-injection test on a real monolith mesh. It uses the new shared injectorFaultInjectingStorageAdapter.RefuseWritesAsUnreachable, which fails writes with a driverDbExceptioncarrying SQLSTATE08006. The production arm logs[CreateNode] STORE UNREACHABLEand answersUnavailable. The test then checks:Unavailable;Observable.NeverinInstall, the main case is red at its final wait. The message reads "Expected the observable to emit a value matching the predicate within 72s … but it did not", andUnreleasedBuildFailureClass = Unavailableis still standing.ARefusedRecut_IsNeverReissued_HoweverOftenTheStoreAnswers. ARefusedstamp is never re-issued under the same signals.ReleaseRecutReportsWhyItFailedTestchecks the classification through the realBoundedchain (Unavailable,Refused,Expired,Faulted).-c Release -warnaserrorand 0 errors (after merging main).Docs
Doc/Architecture/ReleaseFailureClasseshas a new section: "A re-cut the store never saw is re-issued when the store answers again (A NodeType release re-cut refused as Unavailable is never retried on the live pod — three Hosting types advertised unreleased builds 76 min to 3 h after Postgres recovered #7072)".Doc/Architecture/FaultInjectionHarnessdocuments the new injector and case 9.NodeTypeCompilationtable row mentions the class.Public surface
This PR only adds: a
NodeTypeDefinitionproperty, theReleaseFailureClassconstants, twoStoreReachabilityextension methods, and a test-support injector method. Nothing is removed and no interface member is added, so no pairing lines are needed. No i18n keys are added.After deploy
No recycle is needed for the fix to take effect. The watch is installed on each new NodeType activation, and the roll creates new activations. No stamp from before the fix carries a class, so this fix never re-issues any of them. The next activation's one-time cut (#6056) still handles them.
🤖 Generated with Claude Code