Rebuild a record that names a build the store lacks; keep a type typeable while its generation unloads

[rbuergi]

Refs Systemorph/MeshWeaver.Plugins#2799 (the two residuals in the 2026-10-10 20:45Z comment).

Residual 1: StaleBytes for the same 15 types on every memex boot

Readings (governed Logs on control.systemorph.com, Ops/Actions/w14c-memex-*):

• 13 boots across five consecutive generations logged DynamicContentTypeRegistration: 15 baked dynamic NodeType(s) could NOT be registered on this replica in a 240 min window.
• Two of those lines read in full, 16:53Z and 20:43Z, four generations apart: the same 15 types, each with the same pair of MVIDs (record vs. file). So neither the record nor the store's file moved in between. It is a standing state, not a roll race and not a previous image's bake.
• One of the types logged content for nodeType rsalzmann/Clicker was NEVER resolvable on this replica — 2 read(s) degraded.

Root cause. These records carry a version and a path that resolve to build N and the MVID of a build N+1. That is what the first-write-wins assembly store produced before it became content-addressed. No new record gets into that state, but nothing healed the old ones, because three components asked different questions about the same record:

component	question	answer
activation (bind-time check)	are these the bytes the record names?	no: recompile, and only if an instance activates
registration-only pass / on-demand route	are these the bytes the record names?	no: StaleBytes
boot sweep's bake probe	is there a file under (path, LastCompiledVersion)?	yes: Baked

The sweep is the only pass that rebuilds without an instance, and it takes its work list from the probe.

Fix. NodeTypeBakeStatus.ProbeOne resolves by identity when the record claims a live-framework build and states an MVID (TryGetBuildPath plus an MVID comparison of the file it got). A different build under the record's version is BytesMissing, with both MVIDs in the detail. The sweep then re-fetches the shipped build or rebuilds on the owner, which stamps one coherent (version, path, MVID). The pass and the read route stay non-compiling and non-writing.

Such an entry is never a regression baseline (NodeTypeBakeEntry.RecordNamesABuildTheStoreLacks): the build the record names was available to no replica, so a failed rebuild must not refuse a new replica's readiness and stall the roll.

Residual 2: an outgoing memex-cloud pod read Ops/Status/* untyped during a roll

Readings (Ops/Actions/w14c-cloud-786n6-*), pod 6dc5db759b-786n6, booted 19:51Z:

UTC	event
20:34:49	another replica of its generation stops
20:34:53 – 20:36:03	15 GetStream: Content for Ops/Status/… stayed an untyped JsonElement
20:36:10	Application is shutting down... on this pod
20:36:11	giving up on owner Hosting/DeploymentStatus — 4 DISTINCT owner activations have refused this stream; Hosting/DeploymentStatus was NEVER resolvable on this replica — 15 read(s) degraded

The degraded-read count equals the 15 lines, so the pod typed every earlier read of that type for 45 minutes and lost the type before its own host began to stop.

Root cause. Hub disposal unloads the type's collectible contexts (UnloadNodeContexts). AssemblyLoadContext.Unload() only starts an unload, but MeshContentTypeRegistry dropped the generation's entries the moment Unloading fired, while the generation was still loaded and its standing watches were still being served. No successor registered on that pod, and the on-demand route's kept verdict said Registered.

Fix. The registry demotes an unloading generation's entries to a weak shadow instead of dropping them, the shape TypeRegistry has had since #1169. A demoted type answers while something keeps its generation alive, roots nothing, never answers for a contested discriminator, and is displaced by the first successor registration.

Tests

Each is red without its fix (run with the two source fixes reverted):

• RecordNamesABuildTheStoreLacksTest (MeshWeaver.Updates.Test, scenario 11): Expected value to be BytesMissing … but found Baked.
• ContentTypeRegistryReleasesAnUnloadedGenerationTest, two new facts for the shadow: Expected value to be true because the generation is still loaded and in use … but found False. Two more cover the finalizer path: a context nobody unloads runs the eviction from its finalizer, where a first draft threw and took a test host down.
• AReadAfterTheTypesHubLeftIsStillTypedTest (MeshWeaver.Graph.Test): Did not expect value to be of type JsonElement ….

Added in the review round:

• ATypeRegisteredAfterItsContextBeganUnloading_IsResolvableWhileHeld_AndNeverRooted: red with the late-registration routing disabled (the generation is never collected).
• BakeProbeReadsTheStoreThroughThePoolTest: the store read runs inside the I/O pool for in-process callers (NodeTypeBakeStatus.ProbeThrough); the control is the inline public Probe.

Controls inside the tests: a coherent record probes Baked; an ordinary store miss is a regression baseline; a collected generation is still released (the pre-existing fact).

Not established

• The fifteen records were not read (the MCP connection to that mesh was down). Their fields are inferred from the two MVIDs staying the same across four generations.
• The assembly store's own Information lines are not shipped to the log store, so their absence from a Logs read says nothing.
• What tears the hubs down on an outgoing replica was not traced past the owner-refusal line.
• A registration that ended because its generation was collected is still not re-established by a read (the on-demand verdict is kept for the process). I wrote a release of the lapsed verdict and withdrew it: a test mesh could not be brought to collect the generation, so the state could not be reproduced.

To verify after the roll

• memex, first boot on this build: one DynamicTypePreWarmer: N NodeType(s) claim a usable build … Rebuilding: Warning naming the fifteen with both MVIDs. Every later boot: the could NOT be registered Warning no longer names them. A type named on a second boot was re-broken by something live.
• memex-cloud, during a roll: no GetStream: Content for Ops/Status/… stayed an untyped JsonElement on an outgoing replica between the first new replica's boot and its own shutdown line.

No address needs a recycle: the heal rides the boot sweep.

Docs: Doc/Architecture/DynamicContentTypeRegistration (two new sections), Doc/Architecture/StaleStateUntilRecycle (scenario 11).

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Unresolved unload races can retain or restore incorrect generations, and the probe performs blocking file I/O outside the sanctioned pool.

4 open findings

• Coordinate registration with unload to prevent permanent strong references · New
• Move synchronous assembly metadata reads off the reactive scheduler · New
• Prevent stale generations from overwriting newer shadow entries · New
• Use ImmutableDictionary instead of mutable Dictionary · New

What changed in this PR

Repairs dynamic NodeType recovery and preserves type resolution during collectible-generation unloads.

Changes:

• Detects and rebuilds records whose stored assembly identity is inconsistent.
• Adds weak-shadow type resolution during generation unloading.
• Adds integration, lifecycle, and regression tests plus architecture documentation.

File	Description
test/​MeshWeaver.Updates.Test/​RecordNamesABuildTheStoreLacksTest.cs	Tests stale-build detection and repair.
test/​MeshWeaver.Graph.Test/​AReadAfterTheTypesHubLeftIsStillTypedTest.cs	Tests typed reads during unload.
test/​MeshWeaver.Compiler.Pipeline.Test/​ContentTypeRegistryReleasesAnUnloadedGenerationTest.cs	Tests weak-shadow lifecycle behavior.
src/​MeshWeaver.Messaging.Hub/​Serialization/​IMeshContentTypeRegistry.cs	Adds weak-shadow content-type resolution.
src/​MeshWeaver.Hosting/​MeshWeaver.Hosting.csproj	Exposes internals to the update tests.
src/​MeshWeaver.Hosting/​DynamicTypePreWarmer.cs	Expands stale-build diagnostics.
src/​MeshWeaver.Documentation/​Data/​Architecture/​StaleStateUntilRecycle.md	Documents the new update scenario.
src/​MeshWeaver.Documentation/​Data/​Architecture/​DynamicContentTypeRegistration.md	Documents both root causes and fixes.
src/​MeshWeaver.Compiler.Pipeline/​NodeTypeBakeStatus.cs	Detects build-identity mismatches and adjusts gating.

🧠 Review effort: Balanced

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Unresolved unload-race, scheduler-blocking fallback, and readiness-baseline issues can cause retained generations or unsafe rollout admission.

3 open findings

• Missing build path incorrectly clears the regression baseline · New
• First-registration race can permanently root the generation · New
• Missing pool service falls back to blocking inline I/O · New

4 resolved since last review

• Coordinate registration with unload to prevent permanent strong references
• Prevent stale generations from overwriting newer shadow entries
• Move synchronous assembly metadata reads off the reactive scheduler
• Use ImmutableDictionary instead of mutable Dictionary

🧠 Review effort: Balanced

[github-actions]

Test Results

    20 files   -   4      20 suites   - 4   53m 25s ⏱️ +24s
11 352 tests  - 283  11 352 ✅  - 90  0 💤  - 193  0 ❌ ±0 
11 366 runs   - 283  11 366 ✅  - 90  0 💤  - 193  0 ❌ ±0 

Results for commit e1ef99b. ± Comparison against base commit dc9f15b.

This pull request removes 335 and adds 28 tests. Note that renamed tests count towards both.

   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---, expected: True)
   --- End of inner exception stack trace ---, isDenial: True)
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
 ---> (Inner Exception #1) MeshWeaver.Messaging.Hub.Test.InfrastructureFaultTest+ProviderException (0x80004005): Failed to connect to 10.42.18.4:5432<---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
 ---> (Inner Exception #1) System.InvalidOperationException: boom<---
 ---> (Inner Exception #1) System.InvalidOperationException: source B is misconfigured<---
 ---> (Inner Exception #1) System.Net.Sockets.SocketException (0xFFFDFFFF): Name or service not known<---
…

Memex.Portal.Shared.Test.InstanceIdRulesMatchTheRegistryTest ‑ TheSetupHostAgreesWithTheRegistry(candidate: "1f6649c8-db33-4866-97a5-2a33a5d8ad35")
Memex.Portal.Shared.Test.SessionDenialIsAnAnswerTest ‑ OnlyAVerdictReadsAsADenial(shape: "the same verdict nested, as a late denial dispatch"···, failure: System.InvalidOperationException: write failed
 ---> System.UnauthorizedAccessException: Access denied
   --- End of inner exception stack trace ---, isDenial: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's FIRST member", fault: System.AggregateException: One or more errors occurred. (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall as an aggregate's SECOND member", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---
, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall nested in an aggregate inside a wrapper", fault: System.InvalidOperationException: outer
 ---> System.AggregateException: One or more errors occurred. (Value does not fall within the expected range.) (Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.)
 ---> System.ArgumentException: Value does not fall within the expected range.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.<---

   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "a stall wrapped as InnerException", fault: System.InvalidOperationException: outer
 ---> MeshWeaver.Mesh.QueryProviderStalledException: Query provider(s) [pg] did not emit an Initial within the query fan-in's 16s bound for query 'nodeType:NodeType' (user 'system'). The merged Initial gates on EVERY provider, so this query has NO snapshot to answer with — it is reported as unavailable (retryable) rather than left hanging with no error. This is an availability failure, never a permission verdict: a consumer deciding access must fail CLOSED and say it could not establish the answer. Fix the stalled provider; never bump the consumer's timeout.
   --- End of inner exception stack trace ---, expected: True)
MeshWeaver.Compiler.Pipeline.Test.SourcesWatcherStallBackoffTest ‑ The_production_classifier_recognises_a_stall_through_every_wrapping(because: "an aggregate of unrelated faults", fault: System.AggregateException: One or more errors occurred. (The operation has timed out.) (Value does not fall within the expected range.)
 ---> System.TimeoutException: The operation has timed out.
   --- End of inner exception stack trace ---
 ---> (Inner Exception #1) System.ArgumentException: Value does not fall within the expected range.<---
, expected: False)
MeshWeaver.Graph.Test.AForeignSiblingIsNotAlwaysTheLegacyResidueTest ‑ ARecordWhoseOwnFileIsGone_WithASiblingLeft_NamedAWorkingBuild_AndKeepsItsBaseline
MeshWeaver.Graph.Test.AForeignSiblingIsNotAlwaysTheLegacyResidueTest ‑ ARecordWhoseOwnFileIsThere_IsBaked
MeshWeaver.Graph.Test.AForeignSiblingIsNotAlwaysTheLegacyResidueTest ‑ TheFileAtTheRecordsOwnPath_CarryingAnotherMvid_IsTheResidue_AndNotABaseline
…

♻️ This comment has been updated with latest results.

[Copilot]

🟡 Changes recommended

Unresolved unload races, generation ordering, and probe misclassification can retain or select incorrect dynamic types.

3 open findings

• Unload race leaves collectible contexts with permanent strong type references · New
• Synchronous filesystem reads run on ThreadPool workers · New
• Fallback path selection misidentifies the record's own file status · New

3 resolved since last review

• First-registration race can permanently root the generation
• Missing build path incorrectly clears the regression baseline
• Missing pool service falls back to blocking inline I/O

🧠 Review effort: Balanced

[Copilot]

🟡 Changes recommended

The weak-shadow handoff has lookup gaps and loses discriminator-collision history.

2 open findings

• Publish weak shadow before removing strong registration · New
• Preserve shadow claimants during strong registration · New

3 resolved since last review

• Unload race leaves collectible contexts with permanent strong type references
• Fallback path selection misidentifies the record's own file status
• Synchronous filesystem reads run on ThreadPool workers

🧠 Review effort: Balanced

[rbuergi]

Valid, and deferred rather than pushed. The readers are lock-free, so between TryRemove and the shadow write a lookup can see neither entry; the remedy you give (publish the shadow first under the lock, then remove the strong entry, and still drop a contested claim without one) is the right one.

I am not pushing for it in this round because of its size, not because it is wrong: the window is the two adjacent statements of one sweep, and a read that falls into it degrades once, where before this change every read after the unload began degraded until a successor registered. It is recorded for triage with your ordering as the fix.

[rbuergi]

Valid, and it predates this change, so it is deferred rather than pushed here. Before the shadow existed, declaration A's entry was removed outright when its context began unloading, and a later registration of the same discriminator by declaration B was treated as unique in exactly the same way. The shadow does not widen that; it only keeps A answering until B arrives.

The remedy is the one you describe: carry the declaration into the shadow entry and let it take part in the ambiguity decision, so only a rebuild of the same declaration displaces it. That changes what a strong claim means for a name with a live shadow claimant (it becomes contested), which deserves its own change and tests. Recorded for triage.

The NodeType-keyed route is not affected: content of a node with a NodeType is resolved by that path first, and the path map is per declaration.