Repository navigation
Self-updater never opens a Roll for newer promoted builds — build portal 7 d stale and feature-degraded, client-b 4 d stale #6501
Description
(re-filed 2026-10-11; replaces a deleted issue, formerly number 5757; originally by systemorph-com[bot] on 2026-09-26)
What is failing. FleetWatch's StalePortal roll alarm fired for two fleet instances: build is still running 3.0.0-ci.8411 while 3.0.0-ci.9406 is the newest promoted build (a compatible newer build was first promoted 7 d 11 h ago), and client-b is on 3.0.0-ci.9077 (4 d 2 h behind). In both cases the blocking reason recorded is the same: no self-update Roll was opened by the instance's self-updater for any build newer than the one it runs — the updater has not announced one. On build the alarm additionally carries pending_module_activation: Degraded — 35 modules (the whole MeshWeaver.AI.* family, MeshWeaver.Azure.Blob, …) are landed but not loaded in the running process, plus 1 module REFUSED, so the instance is also feature-degraded until a restart.
The alarm itself is working as designed (FleetWatch — "roll, restart, observe"); the defect is that the self-update loop that should have rolled these portals never announced a Roll.
Probable cause. The self-updater on the affected instances has stopped opening Roll announcements for newer compatible builds. Confidence: medium — the alarm text names the ground ("its self-updater has not announced one"), but the evidence does not distinguish between a stuck self-updater process, an Admin/UpdatePolicy blocked by merge-guard/update-queue failures (the log history around Admin/UpdatePolicy shows repeated UpdateQueue FAILED / MergeGuard refused / could not record the check verdict entries on the memex-cloud pod in the days before), or a deliberately conservative policy. A secondary symptom on build — 35 landed-but-unloaded modules needing a restart — is consistent with a long-lived process that has not restarted in a week, which a working roll cadence (bound: two CD cycles = 6 h) would have forced.
Impact. Both portals are still serving (no outage), but build has been running a 7-day-old build and is feature-degraded (AI modules landed but not active). 2 occurrences on one observing pod, first seen 2026-09-26T12:22Z; the underlying staleness itself goes back to 2026-09-19 (build) and 2026-09-22 (client-b). The CD roll guarantee — a portal no more than ~6 h behind the newest promoted build — is broken for two of the fleet's instances.
Where to look. Start with the self-updater on the affected instances: read each instance's Admin/UpdatePolicy and its self-update log as the alarm instructs, and check the UpdateQueue/MergeGuard/SelfUpdate log categories (repeated FAILED path=Admin/UpdatePolicy lines and stale/reordered cross-hub write refusals appear in Ops/Logs in the preceding days — see the memex-cloud pod entries). Also check whether the refused module on build blocks activation. Related prior work: MeshWeaver#4471 (self-update configuration lambda never ran), #5011, #5384.
Evidence
| Fingerprint | 24851d544b6ccf02 |
| Category | FleetWatch |
| Severity | Error |
| Namespace | memex |
| Pods | memex-portal-deployment-54bf7d958b-8n7r4 |
| Occurrences | 2 |
| First seen | 2026-09-26 12:22:54Z |
| Last seen | 2026-09-26 12:22:54Z |
| Routing | not determined — no configured route matches the category FleetWatch. This repository is the configured fallback, not a finding about who owns the fault; the category names the LOGGER, which may not be the subject. |
Recent log lines
2026-09-26 12:22:54Z memex-portal-deployment-54bf7d958b-8n7r4 fail: FleetWatch[0]
[FleetWatch] roll alarm StalePortal on build: build: Stale portal — still on 3.0.0-ci.8411, newest promoted is 3.0.0-ci.9406. build runs 3.0.0-ci.8411. A newer compatible build was first promoted 2026-09-19 00:28:21Z (7 d 11 h ago); the newest is 3.0.0-ci.9406. Bound: two CD cycles = 6 h 0 m (startup probe 10800 s × 1 replica(s) = 3 h 0 m per roll). Blocking: no self-update Roll was opened for build for any build newer than 3.0.0-ci.8411 — its self-updater has not announced one (read the instance's Admin/UpdatePolicy and its self-update log) · the portal reports pending_module_activation: Degraded — 35 module(s) are landed but not yet loaded in this process — a restart activates them: MeshWeaver.AI, MeshWeaver.AI.Anthropic, MeshWeaver.AI.AppleIntelligence, MeshWeaver.AI.AzureFoundry, MeshWeaver.AI.ClaudeCode, MeshWeaver.AI.Codex, MeshWeaver.AI.Copilot, MeshWeaver.AI.OpenAI, MeshWeaver.AI.WebSearch, MeshWeaver.Azure.Blob, …(+25); 1 module(s) were REFUSED a… Observed running 3.0.0-ci.8411; expected a build no more than 6 h 0 m behind the newest promoted.
2026-09-26 12:22:54Z memex-portal-deployment-54bf7d958b-8n7r4 fail: FleetWatch[0]
[FleetWatch] roll alarm StalePortal on client-b: client-b: Stale portal — still on 3.0.0-ci.9077, newest promoted is 3.0.0-ci.9406. client-b runs 3.0.0-ci.9077. A newer compatible build was first promoted 2026-09-22 10:12:01Z (4 d 2 h ago); the newest is 3.0.0-ci.9406. Bound: two CD cycles = 6 h 0 m (startup probe 10800 s × 1 replica(s) = 3 h 0 m per roll). Blocking: no self-update Roll was opened for client-b for any build newer than 3.0.0-ci.9077 — its self-updater has not announced one (read the instance's Admin/UpdatePolicy and its self-update log) Observed running 3.0.0-ci.9077; expected a build no more than 6 h 0 m behind the newest promoted.
Opened automatically from Admin/_LogIncident/24851d544b6ccf02. Recurrences are folded into this issue rather than opening new ones.
It also stands for the whole log site b164c7a435ab60de: other fingerprints of this site fold in here as comments rather than opening tickets of their own.
Activity
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-26)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.401e97dd2d279572b164c7a435ab60deFleetWatch[FleetWatch] roll alarm StalenessUnobserved on build: build: Staleness not observable — the fleet watch cannot tell whether this portal is current. No self-update Roll in the fleet's action history carries a tag matching pattern {n}.{n}-ci* — whether {n}.{n}-ci.{n} is current is UNKNOWN. This is '{value}', reported as loudly as a fault: nothing here says the roll or the build is fine. Observed no promoted build matching pattern {n}.{n}-ci* is recorded; expected a judgeable observation.memex-cloudmemex-portal-deployment-6d84f5d754-xkfk4Admin/_LogIncident/401e97dd2d279572What triage wrote about this fingerprint
What is failing
The FleetWatch staleness watch on the memex-portal pod cannot determine whether the running portal build is current, and reports this as a
fail-level roll alarm (StalenessUnobserved). The watch looks for a self-updateRollentry in the fleet's action history whose tag matches the CI pattern3.0.0-ci*; none exists, so it cannot judge whether builds3.0.0-ci.8411/3.0.0-ci.9408are current and alarms loudly instead.Probable cause
The portal pod was evidently not deployed through the self-update Roll flow — or its action history was not (yet) recorded — so there is no Roll entry carrying a
3.0.0-ci.*tag for the watch to compare against. The watch treats "no judgeable observation" as a fault-level alarm rather than distinguishing "not deployed via Roll" from "stale". Confidence: moderate — the message itself is explicit about the missing evidence, but I could not locate the FleetWatch source in the readable mesh partitions to confirm whether the pod is expected to carry Roll history at all.Impact
Minimal so far: 2 occurrences within 6 seconds on a single pod (
memex-portal-deployment-6d84f5d754-xkfk4). No user-facing failure is evidenced; the harm is noisy fault-level alarms and a blind spot in staleness monitoring for this pod.Where to look
The FleetWatch staleness watch (log category
FleetWatch): the code path that scans the fleet's action history for a self-update Roll with a tag matching{major}.{minor}-ci*and emitsStalenessUnobservedwhen none is found. Two candidate fixes: have the watch classify "no Roll history" as a distinct, lower-severity state, or ensure portal pods record a self-update Roll on deployment. Note: no existing incident covering FleetWatch/StalenessUnobserved was found, so this appears to be new.Still occurring: 2 more since the last update (2 total).
memex-portal-deployment-6d84f5d754-xkfk4Recent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-26)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.3da53af1c4ad320cb164c7a435ab60deFleetWatchmemexmemex-portal-deployment-6f45b975cf-85qc6Admin/_LogIncident/3da53af1c4ad320cWhat triage wrote about this fingerprint
Likely duplicate of Systemorph/MeshWeaver.Plugins#2411. That issue predicted this false positive from reading the code; this incident looks like the first time it happened in production. A human can probably close this one as a duplicate and link the evidence below to #2411.
What is failing
The fleet watch's stuck-roll alarm (
RollAlarms.StuckRoll) raised a CriticalStuckRollon memex for a roll that the evidence says was still progressing normally:Runningwith 0 restarts. Its readiness checks were fast (24 ms total, slowestpending_module_activation), and it reported a bake in progress (bake-report: Degraded … compiling sweep).Probable cause (medium–high confidence)
RollAlarms.Budgetjudges a not-Ready pod against the startup-probe budget alone (periodSeconds × failureThreshold, 600 s on memex). Since MeshWeaver#5544 the NodeType bake runs on readiness, not startup. The chart pays for that bake inprogressDeadlineSeconds(startup budget + 600 +probes.rollGate.bakeSeconds1800), as #2411 describes. So a pod that has started but is still baking is legitimately not Ready for up to about 50 minutes, and the alarm calls it stuck after 10. The class doc comment says the alarm uses "the same formula" asInstanceActionPlan.RequestedRolloutSeconds. After Plugins#2407 that is no longer true.Not established: whether the pod
memex-portal-deployment-6f45b975cf-s5h96became Ready later. If it never did, this was a real stall and the bake is what to investigate, not the alarm. Check the pod's later readiness before closing.Secondary wording defect. The title reads "Stuck roll — 3.0.0-ci.9408 is not taking over from 3.0.0-ci.9408". The old and new ReplicaSets run the same image, so this was a restart-style roll (a template change without a new tag). The alarm words it as a version takeover. When the tags are equal it should say the new generation is not taking over from the old one, and name the ReplicaSets instead of the tag.
Impact
One occurrence on one instance (memex, the control instance). No user-facing failure is shown. The cost is a false Critical alarm, and its text tells operators "the remedy is the cause the pod names". That can send someone to investigate a healthy roll, and it teaches operators to ignore StuckRoll, which exists because a real three-day stuck roll went unnoticed. Every bake-gated roll on any instance whose bake takes longer than its startup budget will fire the same alarm, so this will recur.
Where to look
Budget.StartupandBudget.Roll, andRecordFactsOf(the budget is read only fromstartupProbe).StuckRoll: thelateStarttrigger (now - StartedAt > budget.Startup) and the headline built fromTagOf(template)/servingTag.InstanceActionPlan.ProgressDeadlineSeconds), and keep a separate, tighter signal for a pod that is crash-looping or has not started at all.RollAlarmsTests(TheBudget_IsTheRecordsOwn,EachTrigger_FiresOnItsOwn) pin the current numbers and will need updating.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6f45b975cf-85qc6Recent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-26)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.3f099caacdba2bf4b164c7a435ab60deFleetWatchmemexmemex-portal-deployment-6f45b975cf-d7ggfAdmin/_LogIncident/3f099caacdba2bf4What triage wrote about this fingerprint
What is failing
The self-update roll of the memex control instance from
3.0.0-ci.9408to3.0.0-ci.9413is stalled: the new portal pod (ReplicaSetmemex-portal-deployment-85f748bdd7) stays phase Running for its whole startup budget without ever becoming Ready, so the ready pods keep servingci.9408and no later build can roll. The pod is not crash-looping (0 restarts) and its health checks are fast — the only degraded finding iscontent-types: one NodeType whose CLR type is not registered in the process, so its pages would render empty.Probable cause
High confidence that readiness fails on the content-types finding: the pod is Running, has zero restarts, every timed check completes in under 100 ms, and the sole non-healthy entry is the unregistered CLR type — there is nothing else for the readiness gate to fail on. The underlying defect sits in build
3.0.0-ci.9413ofmemex-portal-ai: it carries, or its mesh content references, a dynamic NodeType whose backing CLR type is not registered in the portal process — most likely a module the image does not load, or node content that landed before the assembly that types it. Exactly which type and module needs to be identified from the 9413 build.Note the fleet-watch alarm itself is working as designed —
Hosting/Issue/Source/RollAlarmsraised exactly what it should, and the Roll action reporting Done after only setting the image is the documented behaviour. The defect is in what was rolled, not in the watcher.Impact
Judged from the numbers: one roll, one pod, first occurrence minutes into the roll and still inside the 30 m whole-roll budget. Serving traffic is unaffected (the old pods answer). But the control instance's update lane is blocked — this is the lane whose stuck actions previously held memex 149 commits behind for 30 h (see
rbuergi/Feedback/stuck-roll-recurred-20260926T1216Z, a different defect in the same symptom class, fixed by MeshWeaver.Plugins#2407). And if the new pod ever served, pages of the affected NodeType would render empty — silently wrong results.Where to look
memex-portal-ai:3.0.0-ci.9413, against the module inventoryOps/Modules/memex.Ops/Actions/selfupdate-roll-memex-3-0-0-ci-9413-e6a68743and the deployment recordDeployments/memex.content-types: Degradedinto NotReady — the alarm side isHosting/Issue/Source/RollAlarms; the health check in the portal process is the seam to instrument.Not a duplicate of the stuck-observer feedback above (that was a lost action observer); this is the rolled build failing its own readiness.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6f45b975cf-d7ggfRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-26)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.11475f707fef2e3db164c7a435ab60deFleetWatchmemexmemex-portal-deployment-85f748bdd7-nk4wgAdmin/_LogIncident/11475f707fef2e3dWhat triage wrote about this fingerprint
Likely duplicate of Systemorph/MeshWeaver#5757, with the root cause in Systemorph/MeshWeaver.Plugins#2411. This is the third StuckRoll on memex today and the second on the same roll. Incident
3f099caacdba2bf4fired at 17:29Z on the same roll (ReplicaSetmemex-portal-deployment-85f748bdd7, 3.0.0-ci.9408 → 3.0.0-ci.9413). A human can close this as a duplicate and attach the evidence below to #5757 / #2411.What is failing
The fleet watch's stuck-roll rule (
RollAlarms.StuckRoll) raised a CriticalStuckRollon memex through its per-podlateStarttrigger. The third new pod,…-8t2l4, was not Ready 11 m after it started, against a 10 m startup-probe budget. At 17:49Z the roll was converging:qvqf7, the pod the 17:29Z alarm called stuck, did become Ready. That answers the open question on #5757: that alarm was a false positive.8t2l4wasRunningwith 0 restarts, and its 18 readiness checks were fast (25 ms total).The roll had also just passed its 30 m budget (in flight since 17:19:25Z, which is about 30.5 m), so the
overrantrigger was about to fire as well. A surge roll that starts its pods one after another, each gated on a bake, cannot fit a startup × replicas budget.Probable cause (medium–high confidence)
The cause is the one #2411 describes.
Budget.StartupandBudget.Rollcome only from the record'sstartupProbe(600 s here).lateStartandoverranare judged against those numbers.progressDeadlineSecondsallows extra time for that bake. The alarm does not.8b92e657d4cd1295(MeshWeaver#5722). A pod of this same new ReplicaSet (…-nk4wg) reportednodetype_bakeUnhealthy from 17:37Z to 17:38Z, while it was batch-compiling 2 NodeTypes and "waiting on Crm/Client, for up to 00:05:00". So each new pod spends minutes not Ready on the bake, and that is by design.Not established: whether
8t2l4became Ready later. If it did not, look at what gates its readiness, not at the alarm.Secondary defect, again:
Quotecuts the pod's readiness detail atReadinessQuote = 400characters. Almost all of that is the timing preamble and acontent-types: Degradedentry, which is not a readiness gate. The check that actually keeps the pod not Ready (probablynodetype_bake) never appears in the quote, even though the alarm tells operators to act on "the cause the pod names".Impact
There is no user-facing failure: old pods keep serving until new ones are Ready. The cost is repeated false Critical alarms on the control instance, three in about two hours. Each one sends an operator to a healthy roll, and together they teach operators to ignore StuckRoll. That is the silence this alarm was built to end. Every bake-gated roll will keep firing it. This is a monitoring defect with an easy manual check, so
sev:M.Where to look
Budget.Startup/Budget.Roll, andRecordFactsOf, which reads onlystartupProbe.StuckRoll: thelateStartandoverrantriggers.Quote/ReadinessQuote: put the non-Healthy readiness-gating entries (such asnodetype_bake) first, and drop the timing summary.InstanceActionPlan.ProgressDeadlineSeconds), and keep a tighter separate signal for pods that crash-loop or never start.RollAlarmsTests(TheBudget_IsTheRecordsOwn,EachTrigger_FiresOnItsOwn) pin the current numbers.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-85f748bdd7-nk4wgRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-09-26)
2026-09-26 ~19:40Z — build now hands detected releases to the control lane (
apply=control-lane)Governed path on memex.systemorph.com, record from Memex#567 (
Hosting__ControlInbox__Secret← vault objectHosting-PlatformWebhookSecret, the object build already mounts asHosting__PlatformWebhookSecret):Ops/Actions/reconcile-build-20260926-control-inbox-dryrun2c327f29675c, 9 steps, running image keptOps/Actions/reconcile-build-20260926-control-inbox3.0.0-ci.9412df9599fc5-rxrj8(19:31:36Z)Ops/Logs/build-1790451096428006003-599fc5-rxrj8apply=detect-only (… Hosting:ControlInbox:Secret is empty …)Ops/Actions/restart-build-20260926-control-inbox-env56594547b6-4snsf(19:37:14Z)Ops/Logs/build-1790451434155513279-4547b6-4snsfregistry=cr.meshweaver.cloud/memex-portal-ai … (instance-key auth; key validated at memex.meshweaver.cloud), canPatch=False, apply=control-lane (a detected release is handed to https://memex.systemorph.com/api/hooks/Hosting/PlatformBuilds; the control plane rolls)So the registry half (instance key presented to cr.meshweaver.cloud) and the hand-over half are both live on build.
Why a second restart was needed (not measured, filed to triage as
rbuergi/Feedback/reconcile-new-kv-key-needs-second-restart-20260926): the key reaches the pod throughenvFromof the CSI-synced Secretbuild-portal-keyvault; the cluster's provider runs withenableSecretRotation=true, and the likely order is that the pre-existing synced Secret got the new key on a rotation pass after the rolled container had already resolved its env. The Reconcile reported Healthy with the key absent from the process.Still open — the second acceptance: a promoted build newer than
3.0.0-ci.9412(memex already rolled to3.0.0-ci.9419) must open aselfupdate-roll-build-…action on the control instance. The self-updater is event-driven with a 1 h safety net, so that is expected by ~20:37Z at the latest; not yet observed at the time of this comment.rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-27)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.c0b6fd714465a127b164c7a435ab60deFleetWatch[FleetWatch] roll alarm StalePortal on memex: memex: Stale portal — still on {n}.{n}-ci.{n}, newest promoted is {n}.{n}-ci.{n}. memex runs {n}.{n}-ci.{n}. A newer compatible build was first promoted {time} ({n} h {n} m ago); the newest is {n}.{n}-ci.{n}. Bound: two CD cycles = {n} h {n} m (progress deadline {n} m (startup probe {n} s + {n} s + bake window {n} s) × {n} replica(s) = {n} h {n} m per roll). Blocking: Roll {n}.{n}-ci.{n} ({path}) is Running, phase '{value}': ⏳ Step {path} — Verify one generation… Observed running {n}.{n}-ci.{n}; expected a build no more than {n} h {n} m behind the newest promoted.memexmemex-portal-deployment-6d4fbcfddf-nlf8tAdmin/_LogIncident/c0b6fd714465a127What triage wrote about this fingerprint
What is failing
FleetWatch's
StalePortalalarm on memex (the control instance) is reporting a real lag of 5 h. The alarm itself is working. The lag comes from the roll lane. Each self-update Roll on memex loses the process that runs its post-run phases ([6/8] Verify one generationonward). The action then staysRunninguntil a "replacement process" resumes it. BecauseSelfUpdateRouting.Decidetreats aRunningroll as in flight, no newer build can open a roll while it waits.Probable cause (confidence: medium on the mechanism, low on why the resume is slow)
The two memex roll actions show it:
Ops/Actions/selfupdate-roll-memex-3-0-0-ci-9419-d8d5fb6a[6/8] Verify one generationstarted at 18:59:04Z.observedUntilwas 2026-09-26T21:59:34Z, so the action satRunningfor ~2 h 20 m past its own window and ~5 h 20 m in total.Ops/Actions/selfupdate-roll-memex-3-0-0-ci-9435-307acd52Verify one generation, with memex still serving ci.9419.The likely reason the observer is lost every time: this roll replaces the pods of the control instance, and the control instance is what hosts the observer. So every memex self-roll kills its own observer. That is inference and has not been verified in code.
This is a follow-on to the orphaned-
Runningfailure in MeshWeaver.Plugins#2178 (see Why the Fleet Stopped Rolling Itself, section "2026-09-20"). A resume path now exists, so the action no longer staysRunningforever. But the resume fires hours afterobservedUntilinstead of at pod start or when the window lapses, and it can fire twice.Impact
The primary automatic-delivery path is slowed rather than broken, so this is
sev:M. If the same pattern also holds memex-cloud's lane, re-rule it higher.selfupdate-roll-memex-cloud-3-0-0-ci-9435was modified at 00:51Z, which is worth a look.Where to look
observedUntilor on a new pod's startup.SelfUpdateRouting.Decide(Hosting/Deployment/Source/SelfUpdateRouting): it could treat aRunningaction past itsobservedUntilas lapsed, so a newer announcement can supersede it.StalePortalinHosting/Issue/Source/RollAlarms.cs. It reported correctly and needs no change. The live issue node isOps/Issue/memex__StalePortal.Related: MeshWeaver.Plugins#2178 (orphaned Running action blocks the lane). MeshWeaver#5757 is the
buildStalePortal ticket, which has a different cause (no announcement at all) and is not a duplicate.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6d4fbcfddf-nlf8tRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-09-27)
Production trace on 2026-09-27 narrows the cloud recurrence to an orphaned Roll, rather than a silent announcer.
Ops/Actions/verify-5757-cloud-20260927-check-verdictsread and landed 100 lines, 0 failed, truncated. Positive recordOps/Logs/memex-cloud-1790515994320940093-9cd496-r922l, timestamp 13:33:14.320Z, says the ci.9464 handoff was accepted (200), signature verified at the control inbox. This is a positive announcement witness; the truncated collection is not an absence sweep.Ops/Actionssearch forcontent.deployment:Deployments/memex-cloud lastModified:>2026-09-27T12:00:00Zreturned 0, untruncated, coverage["ops"]. The old ci.9443 Roll was still present and Running / Verify one generation:Ops/Actions/selfupdate-roll-memex-cloud-3-0-0-ci-9443-59eeabf9, v22, last modified 06:15:30Z, observation deadline 09:16:00Z. CurrentOps/Status/memex-cloudalready showed five Ready ci.9443 replicas and a converged roster.The receiver code's
SelfUpdateRouting.Decideskips every newer announcement while an existing Roll/Restart remains in an open state. This old action supplies that blocker. The problem remains open: a read-triggered resume is not evidence that the automatic startup resumer is live. Plugins#2403 tracks delivery of that native fix; new ci.9462 control replicas still load the old September 18 SelfUpdate.Aks generation.Build is a separate unresolved branch. Its fresh pod
memex-portal-deployment-6c54474dff-lt8xtstarts ci.9412 at 12:20Z. Governed actionOps/Actions/verify-5757-build-20260927-startupcollected 19 read, 19 landed, 0 failed, untruncated. Startup positively reports OCI instance-key auth andapply=control-lane; startup/safety-net checks successfully list 1305/1308 tags, but select none newer than ci.9412.Deployments/buildand the generated public Helm values declare Continuous /3.0.0-ci*; those values seed only a newly created Admin/UpdatePolicy and do not prove an existing persisted record. Its actual Admin policy is not yet verified: the browser session reaches/setupwith Anonymous denied Read on Admin. No policy or permissions were changed.rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-09-27)
Further production evidence, 2026-09-27 13:59 UTC:
Once the old ci.9443 Roll was resumed to its verified terminal state, the normal self-update route admitted a new cloud Roll without a manual Roll request:
Ops/Actions/selfupdate-roll-memex-cloud-3-0-0-ci-9464-0bcb36f7, created 13:47:38Z, originself-update, target ci.9464. Its operator run https://github.com/Systemorph/Memex/actions/runs/36323630919 completed successfully, including migration version 58 and deployment generation 1591.The control plane is still in
Verify one generation(action v21). The latest fleet sample at 13:57:05Z reports all five new ci.9464 replicas Ready, commit2bb14d8db95fb9dfbacfdfd96111a2f05dabfb63, and two old ci.9443 replicas terminating.convergedremains false: do not read the successful operator callback as full rollout completion.This is positive evidence that the cloud announcement/receiver/routing lane works after the older Running action is completed. It does not prove the startup action-resumer fix has been delivered to the control process, and does not answer the separate build instance's persisted update-policy question. Keep this issue open.
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-29)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.c538e167a7187f52b164c7a435ab60deFleetWatch[FleetWatch] roll alarm StalePortal on control: control: Stale portal — still on {n}.{n}-ci.{n}, newest promoted is {n}.{n}-ci.{n}. control runs {n}.{n}-ci.{n}. A newer compatible build was first promoted {time} ({n} h {n} m ago); the newest is {n}.{n}-ci.{n}. Bound: two CD cycles = {n} h {n} m (progress deadline {n} m (startup probe {n} s + {n} s + bake window {n} s) × {n} replica(s) = {n} h {n} m per roll). Blocking: no self-update Roll was opened for control for any build newer than {n}.{n}-ci.{n} — its self-updater has not announced one (read the instance's {path} and its self-update log) Observed running {n}.{n}-ci.{n}; expected a build no more than {n} h {n} m behind the newest promoted.memexmemex-portal-deployment-79956556d6-qwxbpAdmin/_LogIncident/c538e167a7187f52What triage wrote about this fingerprint
The
controldeployment — the fleet's dedicated control instance, provisioned 2026-09-26 — declaresupdatePolicy: Continuouson3.0.0-ci*, but no self-update Roll has ever been opened for it: every roll since provisioning has been a manualroll-control-*action. It now sits behind the fleet's newest promotion with nothing that will move it. FleetWatch's judgement is correct and its blocker ("no self-update Roll was opened … its self-updater has not announced one") is accurate — this ticket is about why, which the alarm cannot see from the action history.Probable cause — one dead lane, two compounding defects (confidence: medium-high on the first, certain on the second and third).
controlis opened on the acting control plane only when the control instance's own self-updater POSTs a signedself-update-availableevent toHosting__ControlInbox__Url. Every portal record carries that pair — Deployments/memex-cloud declaresHosting__ControlInbox__Url: https://memex.systemorph.com/api/hooks/Hosting/PlatformBuildsplusHosting__ControlInbox__Secretfrom the vault — but Deployments/control carries neither, inextraPortalConfigorkeyVaultSecrets, and its own notes defer exactly this wiring to the cutover ("The inbox target … and the Hosting__ReportTo/ControlInbox wiring … arrive at the cutover, §8 steps 6.1–6.4"). The record renders the pod's config, so the updater has nowhere to hand a build to. I could not read the control instance's ownAdmin/UpdatePolicyor self-update log from this mesh — confirming that is step 1 below.controlis a gated record:rollGate: after [memex-cloud, memex], soakMinutes: 120, approval: required— it rolls LAST, by its own declaration. RollGates.Evaluate holds the gate on any open Critical issue of a followed deployment, and Ops/Issue/memex-cloud__StalePortal is open at Critical (memex-cloud's own updater is silent too — #5757, third recurrence). Control cannot take any build memex-cloud has not converged on.Blockersreads only the action history, so "announced, gate WAITING" is reported with the same words as a dead announce lane.Impact. Control serves (3/3 ready); this is build-delivery lag on the fleet's operational hub, not an outage, and nothing user-facing is affected. But
Continuousis declared and inert, every catch-up is a manual roll (three since 2026-09-26), and the alarm recurs on each promotion wave past the bound. Easy workaround (open the Roll by hand) →sev:M.Where to look.
Admin/UpdatePolicy(lastCheckVerdict,handedOverTag,lastCheckedAt) and its[SelfUpdate]lines since the first newer promotion — did the updater see a newer build, and did the hand-over have a URL to POST to?Hosting__ControlInbox__Url/Secretpair the portals carry (pre-cutover, pointing at the acting control plane), or park the record honestly until the §8 wiring lands.Not a duplicate of #5757 (that is memex-cloud's own dead updater) but coupled to it: memex-cloud's open Critical is what holds control's gate today. Sibling incident
551c35993737731fis the same control lag seen from the memex-cloud sweep asStalenessUnobserved.Still occurring: 2 more since the last update (2 total).
memex-portal-deployment-79956556d6-qwxbpRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-29)Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-6d4fbcfddf-nlf8t,memex-portal-deployment-5dc5995596-d9qj7Recent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-29)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.1fa4db7c70073162b164c7a435ab60deFleetWatch[FleetWatch] roll alarm StalePortal on memex: memex: Stale portal — still on {n}.{n}-ci.{n}, newest promoted is {n}.{n}-ci.{n}. memex runs {n}.{n}-ci.{n}. A newer compatible build was first promoted {time} ({n} h {n} m ago); the newest is {n}.{n}-ci.{n}. Bound: two CD cycles = {n} h {n} m (progress deadline {n} m (startup probe {n} s + {n} s + bake window {n} s) × {n} replica(s) = {n} h {n} m per roll). Blocking: no self-update Roll was opened for memex for any build newer than {n}.{n}-ci.{n} — its self-updater has not announced one (read the instance's {path} and its self-update log) Observed running {n}.{n}-ci.{n}; expected a build no more than {n} h {n} m behind the newest promoted.memexmemex-portal-deployment-54f589555d-xlshbAdmin/_LogIncident/1fa4db7c70073162What triage wrote about this fingerprint
What is failing
memex — the control instance — cannot take any new platform build automatically. Its ci.9606 self-update Roll is orphaned in
state: Runningat phase Verify one generation withobservedUntil/convergeUntil= 2026-09-29T12:51:47Z, andSelfUpdateRouting.Decidetreats that lapsed action as an in-flight rollout, returningOutcome.Skipfor every newer announcement. So memex keeps serving ci.9606 while the fleet promoted ci.9608 hours ago. TheStalePortalalarm fired correctly and on time (the lag crossed its two-CD-cycle bound the same hour) — but its blocker sentence, "its self-updater has not announced one", is false:Admin/UpdatePolicyon memex shows the self-updater announcing continuously, handing ci.9619 to the control lane at 15:10:20Z — four minutes before the alarm (triggerSafetyNet, signature verified). The announcements arrive and are consumed; nothing opens.Probable cause (confidence: high on the mechanism; why the resume died is unexplained)
Recurrence of MeshWeaver.Plugins#2558 (filed 09:50Z today for the identical mechanism on memex-cloud, where an orphaned Restart held it off every routed Roll for 12 h):
Decide'sOpenStatesincludesRunningandExistingActioncarries noObservedUntil/ConvergeUntil, so a lapsed action suppresses announcements indefinitely — and the Skip logs at Information while the announcer keeps readingaccepted (200).The ci.9606 roll itself worked (aks-ops run 36551360441 succeeded 09:51:17Z: migration v59, image set, rollout generation 840), then lost its observer at 10:01:05Z ("a replacement process is resuming the post-run verification phases"). The replacement re-started
[1/3] Verify one generationat 10:01:06Z and then went silent — nothing has been written to the action since, and no reliable resume is coming: theRunningActionResumerfrom Plugins#2407 is still not registered on the instance that ownsOps/Actions(#2558), and the only resumes observed on this lane coincided with investigation reads. Every hourly announcement since (ci.9608 → … → ci.9619) was skipped against the orphan.Corroboration: memex-cloud received its ci.9608 Roll at 10:20Z, minutes after its own orphan went
Done— the control-plane routing lane is healthy; the block is per-deployment. Not read directly: the control pod's Skip log lines for memex's announcements — the mechanism is inferred from the action's state and window, the policy hand-over andDecide's code, and matches #2558's measured account of the same behaviour.Secondary defect worth fixing alongside:
RollAlarms.Blockersinfers "has not announced" from "no Roll node exists for a newer tag" — exactly the false negative the Skip produces — so the alarm sends the operator to instruments (Admin/UpdatePolicy, the self-update log) that look healthy. Naming the deployment's newest open action (state + window vs the clock) would point at the orphan directly.Impact
One instance — memex, the control instance. The portal serves (the ci.9606 roll completed; pods ready). The cost is delivery latency on the lane that opens every fleet roll: memex is 2+ builds behind and cannot catch up until the orphan is cleared, and there is no self-heal. Workaround: close or re-resume
Ops/Actions/selfupdate-roll-memex-3-0-0-ci-9606-4130ebc8— the next hourly SafetyNet announcement then opens the roll automatically. Primary path broken with a workaround →sev:H, consistent with #2558.Where to look
SelfUpdateRouting.Decide—OpenStates/ExistingAction(Hosting/Deployment/Source/SelfUpdateRouting.cs): treat aRunningaction past itsObservedUntil/ConvergeUntilas not-open (it isObserverExpired == Unobservedby the type's own definition) — Modules: a usable registry entry wins over a same-named baseline (#2548) #2558's suggested fix (2).RunningActionResumer(Plugins#2407) is still not delivered to theOps/Actionsowner.RollAlarms.Blockers(Hosting/Issue/Source/RollAlarms.cs): the "has not announced" sentence; name the newest open action and its window instead.Related / duplicate status: MeshWeaver.Plugins#2558 is the same routing defect (sev:H, memex-cloud, filed 09:50Z today) — a human may merge this into it. Also related: Plugins#2403 (first occurrence of the orphaned-Running block), Plugins#2407 (the undelivered resumer), MeshWeaver#5757 (the observer-loss/resume-latency half on this same instance — same failure chain, different emphasis, not a duplicate). This ticket's delta over #2558: the recurrence on the control instance itself, which has no resumer and so will not self-heal, and the alarm's misdirecting blocker sentence.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-54f589555d-xlshbRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-29)Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-79956556d6-qwxbp,memex-portal-deployment-54f589555d-xlshbRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-29)Still occurring: 1 more since the last update (2 total).
memex-portal-deployment-79956556d6-qwxbp,memex-portal-deployment-856458cf8-wcx7dRecent log lines
28 remaining items
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-11)
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-27)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.8cdc5883a533216db164c7a435ab60deFleetWatch[FleetWatch] roll alarm StalePortal on build: build: Stale portal — still on {n}.{n}-ci.{n}, newest promoted is {n}.{n}-ci.{n}. build runs {n}.{n}-ci.{n}. A newer compatible build was first promoted {time} ({n} h {n} m ago); the newest is {n}.{n}-ci.{n}. Bound: two CD cycles = {n} h {n} m (progress deadline {n} h {n} m (startup probe {n} s + {n} s + bake window {n} s) × {n} replica(s) = {n} h {n} m per roll). Blocking: no self-update Roll was opened for build for any build newer than {n}.{n}-ci.{n} — its self-updater has not announced one (read the instance's {path} and its self-update log) Observed running {n}.{n}-ci.{n}; expected a build no more than {n} h {n} m behind the newest promoted.memexmemex-portal-deployment-6d4fbcfddf-nlf8tAdmin/_LogIncident/8cdc5883a533216dbuildis silent again — no Roll announced while newer builds are promoted (recurrence after #5757)What triage wrote about this fingerprint
What is failing
The automatic self-update lane for the
builddeployment is again not delivering builds. Its portal serves ci.9412 while the newest promoted build is ci.9422, and FleetWatch names the blocker plainly: no self-update Roll was opened forbuildfor any newer build — its self-updater has not announced one. This is the same signature as the episode filed as MeshWeaver#5757 (incidentAdmin/_LogIncident/24851d544b6ccf02), wherebuildsat on ci.8411 for 7 d 11 h with its self-updater silent.buildhas since been moved forward (to ci.9412 — likely manually), but the announcing half still never fires, and the lag has now re-crossed the deployment's own bound (7 h 23 m vs 7 h 20 m).Probable cause (confidence: low — evidence points at the announcement lane, not at a specific refusal)
Either
build's own self-updater never emitsself-update-available, or the control instance drops the announcement before a Roll opens. Two concrete things to check, in order:Hosting/SelfUpdateAnnouncementIdentity). That change made a record that declaresannouncementKeySecretrefuse fleet-secret announcements silently (AnnouncementKeys.BindingRefusal— "a refused announcement is a silent stop, not an error page"). Ifbuild'sHosting/Deploymentrecord declares the key but its pod still signs with the fleet secret (Hosting__ControlInbox__Secretnot rendered, or the instance has not rolled the new signing half), its announcements are now refused with no visible action — exactly this alarm. This is the strongest hypothesis precisely because the fix that landed for the previous episode introduced this failure mode.Admin/UpdatePolicyand its self-update log on thebuildinstance, as the alarm text says — whether anycheck (…): update availableline has been emitted at all, and what the control inbox logged when it received (or refused) one.I could not distinguish these from the control instance alone; the per-instance policy and self-update log are not in this mesh.
Impact
One customer-facing instance,
build, is running code ~10 builds behind and drifting; it misses every fix promoted since. The portal still serves (the running pods are Ready), the lag is only just past the bound, and a manual Roll works — so this is not an outage, but the primary automatic-delivery path fails silently for this instance for the second recorded episode. Judged from the numbers: one occurrence, one pod, first alarm of this episode — small now, but the previous episode grew to 7 days because nothing else surfaces it.Where to look
Hosting/Deploymentrecord forbuild: does it declareannouncementKeySecret? If yes, check the control inbox'sAnnouncementKey_*refusal logs and whetherHosting__PlatformWebhookSecret__build/ the pod'sHosting__ControlInbox__Secretrendered correctly. Start from the design inHosting/SelfUpdateAnnouncementIdentity(implemented per record in MeshWeaver#5757) andHosting/Deployment/Source/SelfUpdateRouting.Admin/UpdatePolicyon thebuildinstance.Hosting/Issue/Source/RollAlarms.cs(FleetWatchStalePortal); the durable incident node isOps/Issue/build__StalePortal.Likely duplicate/related: MeshWeaver#5757 covers the previous episode of this exact signature on
buildandclient-band carried the announcement-key fix. If the fix introduced the silence (hypothesis 1), this ticket tracks that regression; if it predates the fix's rollout order, it can be closed as a dup in seconds. Either way, a human should link them.Side note: like the previous incident, this log site puts the deployment name inside the masked message text, so stale-portal alarms for different deployments can fold or split unpredictably — the structured-field suggestion from #5757's ticket still stands.
Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-6d4fbcfddf-nlf8tRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-11)
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-27)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.40fec26b16f8d914b164c7a435ab60deFleetWatch[FleetWatch] roll alarm StalePortal on client-b: client-b: Stale portal — still on {n}.{n}-ci.{n}, newest promoted is {n}.{n}-ci.{n}. client-b runs {n}.{n}-ci.{n}. A newer compatible build was first promoted {time} ({n} h {n} m ago); the newest is {n}.{n}-ci.{n}. Bound: two CD cycles = {n} h {n} m (progress deadline {n} h {n} m (startup probe {n} s + {n} s + bake window {n} s) × {n} replica(s) = {n} h {n} m per roll). Blocking: no self-update Roll was opened for client-b for any build newer than {n}.{n}-ci.{n} — its self-updater has not announced one (read the instance's {path} and its self-update log) Observed running {n}.{n}-ci.{n}; expected a build no more than {n} h {n} m behind the newest promoted.memexmemex-portal-deployment-5db78dfc99-d2cpkAdmin/_LogIncident/40fec26b16f8d914What triage wrote about this fingerprint
What is failing
The client-b portal has no working automatic update path. Its self-updater never sends a
self-update-availablenotice to the control instance, so no Roll is ever opened for client-b. FleetWatch'sStalePortalrule is reporting that correctly: client-b runs ci.9414, and newer compatible builds have been promoted since 18:22Z. The alarm rule is working as intended.Probable cause (confidence: high that this is the known condition; low on the root mechanism)
The last client-b roll,
Ops/Actions/roll-client-b-20260926-9414, states the cause in its own title and reason:That roll was opened by hand (requested and approved by rbuergi at 17:51Z) to bring client-b from ci.9077 to ci.9414. It finished
Doneat 18:25Z, and client-b's health wasDegraded. ci.9414 is exactly the build client-b still runs. Since then no roll action for client-b exists inOps/Actions, which fits a self-updater that is silent rather than one whose announcements are refused.I could not read client-b's own
Admin/UpdatePolicyor its self-update start-up line from this mesh. The start-up line would say whether the instance is inself-patch,control-laneordetect-onlymode; see the control-lane hand-off note. Whether client-b isdetect-only(the chart has not been redeployed with the control-lane config) or has an update policy ofNoneis not confirmed. Those are the two likely readings of Memex#458.Side observation, not the cause of this alarm: the manual roll above also lost its observer twice (at 18:03Z and 18:10Z, "a replacement process is resuming"). That is the same pattern as the memex self-roll ticket (MeshWeaver#5757's sibling incident
c0b6fd714465a127).Impact
Degradedon/health.This is
sev:Mbecause a workaround exists and is in routine use. If client-b is meant to be on unattended delivery for this release, re-rule it tosev:H.Where to look
buildandclient-b, along with the announcement-key work inHosting/SelfUpdateAnnouncementIdentity.Admin/UpdatePolicyand the self-update start-up log line (mode, and the config key named fordetect-only).Hosting/Deployment/Source/SelfUpdateRoutingfor any refused announcement. The alarm rule isStalePortalinHosting/Issue/Source/RollAlarms.cs, and the live issue node isOps/Issue/client-b__StalePortal.StalePortalcould say so ("moves only by manual Roll") instead of pointing at a self-updater log that will never show anything.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-5db78dfc99-d2cpkRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-11)
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-27)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.cdd1d1fbe1d24628b164c7a435ab60deFleetWatch[FleetWatch] roll alarm StalePortal on memex-cloud: memex-cloud: Stale portal — still on {n}.{n}-ci.{n}, newest promoted is {n}.{n}-ci.{n}. memex-cloud runs {n}.{n}-ci.{n}. A newer compatible build was first promoted {time} ({n} h {n} m ago); the newest is {n}.{n}-ci.{n}. Bound: two CD cycles = {n} h {n} m (progress deadline {n} m (startup probe {n} s + {n} s + bake window {n} s) × {n} replica(s) = {n} h {n} m per roll). Blocking: no self-update Roll was opened for memex-cloud for any build newer than {n}.{n}-ci.{n} — its self-updater has not announced one (read the instance's {path} and its self-update log) · the portal reports pending_module_activation: Degraded — {n} module(s) are landed but not yet loaded in this process — a restart activates them: MeshWeaver.Observability, MeshWeaver.Testing · the portal reports pending_module_activation: Degraded — {n} module(s) are landed but not yet loaded in this process — a restart activates them: MeshWeaver.Azure.Blob, MeshWeaver.Blazor.OpenStreetMap, MeshWeaver.Courses, MeshWeaver.Markdown.Export, MeshWeaver.Northwind.Application, MeshWeaver.SelfUpdate.Aks, MeshWeaver.Speech Observed running {n}.{n}-ci.{n}; expected a build no more than {n} h {n} m behind the newest promoted.memexmemex-portal-deployment-7c6dfb8bbb-fhbxfAdmin/_LogIncident/cdd1d1fbe1d24628What triage wrote about this fingerprint
What is failing
memex-cloudis not taking promoted builds on its own. It runs3.0.0-ci.9443while3.0.0-ci.9445is the newest promoted, and no self-update Roll was ever opened for anything newer than ci.9443 — the instance's self-updater has not announced one. FleetWatch itself is working correctly: the alarm fired 5 h 1 m after a compatible build was first promoted, past its 5 h bound (two CD cycles).Probable cause (confidence: medium)
The blocking evidence has changed since #5757: the portal now reports
pending_module_activation: Degraded— 9 modules are landed but not loaded in this process (MeshWeaver.Observability, MeshWeaver.Testing, then Azure.Blob, Blazor.OpenStreetMap, Courses, Markdown.Export, Northwind.Application, SelfUpdate.Aks, Speech), and per the portal's own health text a restart activates them. That is consistent with the fleet's rule that a degraded instance does not roll, but note the apparent deadlock: the modules need a restart, the self-updater that would trigger the roll is silent (and MeshWeaver.SelfUpdate.Aks itself is among the not-yet-loaded modules). I have not confirmed this from the instance's self-update log — readAdmin/UpdatePolicyon memex-cloud and its self-update log as the alarm directs; that is the first thing to check.Note also this is the second day in a row for memex-cloud, and StalePortal alarms fired today on
build,memexandclient-bas well — the pattern from #5757 (3 of 5 instances not self-updating) continues.Impact
The portal serves (its replicas are up; FleetWatch judged it readable). This is build-delivery lag, not an outage — but it is now recurring daily, the lag grew from ~1 h to ~5 h, and the not-yet-loaded modules mean the running process is missing nine modules' worth of shipped functionality until a restart happens. Manual workaround: restart the portal to activate the landed modules, and/or open a Roll by hand. Judged
sev:M; raise if the lag keeps growing across CD cycles.Where to look
memex-cloud:Admin/UpdatePolicyand the self-update log — why no Roll was announced for anything newer than3.0.0-ci.9443.pending_module_activationhealth check: what leaves 9 landed modules unloaded in a running process, and whether a silently-degraded instance should block the very roll that would restart it.Ops/Issue/memex-cloud__StalePortal(already open from yesterday).RollAlarms.StalePortal/FleetWatch.FileAlarmin Hosting/Issue/Source/FleetWatch.Side note (message formatting): the alarm repeats the
pending_module_activation: Degradedclause twice with different module lists (2 modules, then 7) — apparently one per replica — which makes the real evidence harder to read. Deduplicate per-replica health clauses; see the same note on #5757.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-7c6dfb8bbb-fhbxfRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-11)
(re-posted 2026-10-11; originally by @rbuergi on 2026-09-27)
Fresh production evidence separates the two affected installations; this issue remains open.
ClientB: governed Logs action verify-5757-client-b-20260927-selfupdate-latest completed with 6/6 lines landed, zero failed, untruncated. At 2026-09-27 14:32:49Z, pod
memex-portal-deployment-9d68d74dc-shp8ldetected3.0.0-ci.9464but reported detect-and-notify becauseHosting:ReportTonames the control instance whileHosting:ControlInbox:Secretis empty.Deployments/client-bv99 and its declared public Helm overlay both omit the control-inbox secret mapping. This is a named signing-configuration gap, not evidence of a stopped updater. Adding that mapping awaits the maintainer's security decision; no key has been read, copied, or granted by this investigation.Build: governed Logs action verify-5757-build-20260927-selfupdate-latest completed with 17/17 lines landed, zero failed, untruncated. At 14:21:44Z, pod
memex-portal-deployment-6c54474dff-lt8xthanded a pending-module restart to the control lane and receivedaccepted (200), signature verified. The replacementmemex-portal-deployment-557c6767bc-dkc9pstarted its self-updater at 14:23:10Z on3.0.0-ci.9412; its 14:23:26Z startup check listed 1308 tags and reported none newer than its installed version. The signed route works on Build; the reason no newer release is selected is still unproven and requires its effective policy/selection evidence. A successful restart does not prove its roll cadence fixed.The Cloud ci.9464 roll action completed at 14:25:12Z, but that action is not acceptance for Build or ClientB. Later fleet samples show another Cloud generation; each rollout and each named instance needs its own evidence.
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-11)
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-29)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.ae03c2b1c1b948f6b164c7a435ab60deFleetWatch[FleetWatch] roll alarm StalePortal on memex-cloud: memex-cloud: Stale portal — still on {n}.{n}-ci.{n}, newest promoted is {n}.{n}-ci.{n}. memex-cloud runs {n}.{n}-ci.{n}. A newer compatible build was first promoted {time} ({n} h {n} m ago); the newest is {n}.{n}-ci.{n}. Bound: two CD cycles = {n} h {n} m (progress deadline {n} m (startup probe {n} s + {n} s + bake window {n} s) × {n} replica(s) = {n} h {n} m per roll). Blocking: no self-update Roll was opened for memex-cloud for any build newer than {n}.{n}-ci.{n} — its self-updater has not announced one (read the instance's {path} and its self-update log) Observed running {n}.{n}-ci.{n}; expected a build no more than {n} h {n} m behind the newest promoted.memexmemex-portal-deployment-79956556d6-qwxbpAdmin/_LogIncident/ae03c2b1c1b948f6What triage wrote about this fingerprint
What is failing
memex-cloudis not taking promoted builds on its own. It runs3.0.0-ci.9564while3.0.0-ci.9598is the newest promoted compatible build, and no self-update Roll action exists for this deployment for anything newer than what it runs — its self-updater has not announced one. FleetWatch itself is behaving correctly: the alarm fired 4 minutes past its declared bound of two CD cycles.Probable cause (confidence: medium-low)
This occurrence narrows the fault. Each earlier occurrence named a second blocker from the portal's own health — an unsealed publication (Manufacturing, 09-26) and
pending_module_activationwith nine landed-but-unloaded modules, includingMeshWeaver.SelfUpdate.Aksitself (09-27). This one names none: the blocker list carries only the silent self-updater. So the "a degraded instance declines to roll" hypothesis #5757 was chasing no longer explains the silence — with the portal's seal and module activation now reading clean, the self-updater still announces nothing.That points at the announcement path itself: the instance's registry watch / availability and combo gates (which select a candidate), the announcement reaching the control inbox, or the control lane turning an announcement into a Roll action. The record is not holding (a
updatePolicy: Nonerecord does not alarm), not pinning (a pin would be named as the blocker), and the build class matches itsupdatePattern(it is judged compatible), so the policy on paper is willing. I cannot read memex-cloud's self-update log from the control instance; the alarm's own direction — read the instance'sAdmin/UpdatePolicyand its self-update log — is the first check.Two further observations:
build,memex,client-b) have been quiet since 09-27 — this is now the only instance still trippingStalePortal, which argues for an instance-local fault (policy or updater state) over the control plane.ci.9349→ci.9443→ci.9564across the three occurrences and each occurrence's pod carries a different ReplicaSet hash — consistent with manual restarts/rolls clearing it, then falling behind again within about a day.Impact
The portal serves; this is build-delivery lag on one production instance, not an outage, and it has so far always recovered (with help). But delivery to
memex-cloudnow needs manual intervention on roughly a daily cadence, and anything fixed in newer builds — including security fixes — lands late on that instance. Judgedsev:M; raise if the lag grows well past the bound or the siblings relapse.Where to look
memex-cloud:Admin/UpdatePolicyand the self-update log — whether an update for anything newer than3.0.0-ci.9564was ever considered, refused or gated, and by which gate.Ops/Actions, originself-update); distinguishing announced but never became a Roll from never announced splits the remaining suspects in two.Ops/Issue/memex-cloud__StalePortal, open across all three occurrences.Still occurring: 1 more since the last update (1 total).
memex-portal-deployment-79956556d6-qwxbpRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-11)
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-09-30)Folded in — a DIFFERENT fingerprint from the same log site.
This issue tracks the log site
b164c7a435ab60de(FleetWatch), and these occurrences belong to it. They carry their own fingerprint — the masking split one site across several — so before this they would have opened a SEPARATE ticket. One log site fanned out into 46 of those, which is why they fold here instead. Nothing is discarded: the distinguishing evidence is below, and splitting this back out is a person's call.9f0b1d5d6d00bab1b164c7a435ab60deFleetWatch[FleetWatch] roll alarm StalePortal on client-b: client-b: Stale portal — still on {n}.{n}-ci.{n}, newest promoted is {n}.{n}-ci.{n}. client-b runs {n}.{n}-ci.{n}. A newer compatible build was first promoted {time} ({n} h {n} m ago); the newest is {n}.{n}-ci.{n}. Bound: two CD cycles = {n} h {n} m (progress deadline {n} h {n} m (startup probe {n} s + {n} s + bake window {n} s) × {n} replica(s) = {n} h {n} m per roll). Blocking: no self-update Roll was opened for client-b for any build newer than {n}.{n}-ci.{n} — its self-updater has not announced one (read the instance's {path} and its self-update log) (The action history was cut at its limit, so the lag is a lower bound.) Observed running {n}.{n}-ci.{n}; expected a build no more than {n} h {n} m behind the newest promoted.memexmemex-portal-deployment-855468d589-tkvnvAdmin/_LogIncident/9f0b1d5d6d00bab1What triage wrote about this fingerprint
What is failing
The client-b portal instance's self-update path is dead: since it was rolled manually to
3.0.0-ci.9606on 2026-09-29, its self-updater has announced no self-update Roll for any newer build, while the newest promoted compatible build is3.0.0-ci.9672— 22 h behind, past the two-CD-cycle staleness bound. The alarm (RollAlarms.StalePortal) is a correct detection; the defect is that client-b's continuous-delivery loop never opens a Roll, so the instance is kept current only by manual rolls, contrary to itsContinuous/3.0.0-ci*update policy.Probable cause
A recurrence of the defect filed as Systemorph/MeshWeaver#5757 (from incident
Admin/_LogIncident/40fec26b16f8d914, 2026-09-27). That issue's remedy — declaringannouncementKeySecretonDeployments/client-bso client-b announces its own self-updates (MeshWeaver#5757/#5791, record note 2026-09-27) — has never met its stated acceptance: noselfupdate-roll-client-b-*action has ever been opened; every client-b roll inOps/Actionsis manual (2026-09-21, 2026-09-26, 2026-09-29). Meanwhile the fleet-wide pipeline is demonstrably healthy: memex, memex-cloud and build each opened self-update Rolls for ci.9663/9665/9672 within hours of promotion. Confidence medium-high that the fault is client-b-specific and in the announcement/acceptance half of its self-update path — most likely the instance-side key was never pasted and verified on client-b (Settings ▸ Control lane ▸ Test connection, per the record's own acceptance note), or the announcement path still fails on this instance; the evidence here cannot distinguish those, so start with client-b's ownAdmin/UpdatePolicyand self-update log, which the alarm names as the unread sources of truth.Impact
One customer portal (ClientB) runs builds up to 66 revisions behind the newest promoted, delivered only by operator intervention; no serving outage, no data loss. 2 occurrences on one control-instance pod — the low count reflects the alarm's raise-once-per-condition design, not a mild condition. Workaround exists (manual Roll), and it is what is currently being used.
Where to look
Admin/UpdatePolicynode and self-update log — confirm the policy isContinuous/3.0.0-ci*and whether the announcement check runs and fails, or never runs.Deployments/client-b(announcementKeySecretnote): was the key issued, pasted on client-b, and did Test connection succeed?Ops/Actions/verify-5757-client-b-20260927-selfupdate-latest— the verification action run against #5757 on 2026-09-27.Hosting/Issue/Source/FleetWatch,AlarmsOf→RollAlarms.StalePortal).Already known: this is the same defect as Systemorph/MeshWeaver#5757 (same alarm site, same instance, 2026-09-27, at that time 21 builds behind). That ticket's fix did not restore client-b's self-update loop; a maintainer may prefer to reopen/continue #5757 rather than track a new issue.
Still occurring: 2 more since the last update (2 total).
memex-portal-deployment-855468d589-tkvnvRecent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-11)
(re-posted 2026-10-11; originally by
systemorph-com[bot]on 2026-10-04)Still occurring: 1 more since the last update (3 total).
memex-portal-deployment-855468d589-tkvnv,memex-portal-deployment-67c67b456d-s7nt9Recent log lines
rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-11)
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-08)
2026-10-08 ~19:00Z: today's
StalePortal on buildhas a named cause. build hands each new build to the OLD control inbox, which answers 404This covers the newest fold here,
64e7c0118e478776(build on3.0.0-ci.10150, 22 h behind). Readings were taken with governedLogsactions and read-only reads.1. build detects every new build and hands it over, and every hand-over fails with 404.
Ops/Actions/verify-5757-build-20261008-selfupdateon memex.systemorph.com read{namespace="build"} |~ "(?i)(SelfUpdate)|(OciTagLister)|(ControlInbox)|(control-lane)"over 15:49–18:49Z: 42 lines read, 42 landed, 0 failed, not truncated. The podmemex-portal-deployment-584c896459-vln7x(image3.0.0-ci.10150, booted 15:40:37Z) repeats the same cycle for3.0.0-ci.10256,…10257and…10259:[SelfUpdate] handing 3.0.0-ci.10259 to the control lane (https://memex.systemorph.com/…)[SelfUpdate] the hand-over of 3.0.0-ci.10259 to the control lane FAILED; the next check announces it again.SelfUpdateHandoverRejectedException: the control inbox at https://memex.systemorph.com/api/hooks/Hosting/PlatformBuilds answered 404 Not Found(Ops/Logs/build-1791484912563511921-896459-vln7x, 18:41:52Z)So the self-updater is NOT silent here. It announces, and nobody accepts the announcement.
2. memex.systemorph.com no longer accepts
Hosting/PlatformBuilds.Ops/Actions/verify-5757-control-20261008-inboxread{namespace="memex"} |~ "(?i)(PlatformBuilds)|(WebhookInbox)|(ControlInbox)|(SelfUpdateRouting)"over 16:51–18:51Z. It was CUT at 200 lines, so the counts below are a floor:Webhook for unknown/refused target 'Hosting/PlatformBuilds' dropped(WebhookInboxEndpoints, warn).[SelfUpdate] handing 3.0.0-ci.10259 to the control lane (https://control.systemorph.com/api/hooks/Hosting/PlatformBuilds).3. The new control instance rolls the instances that point at it, and build is not among them. On control.systemorph.com,
namespace:Ops/Actions id:selfupdate-* sort:createdDate-descshows a Roll per build formemexandcontrolall day (ci.10245 → ci.10264; the newest isselfupdate-roll-control-3-0-0-ci-10264-38c678b9, 18:52Z).content.deployment:Deployments/buildon control returns 0, untruncated. On memex.systemorph.com, noselfupdate-*action has been created since 2026-10-07 ~17:17Z. That is expected after the move to the new instance, not a stall: routed actions now live on control.systemorph.com.4. The record already has the fix; the pod does not.
Deployments/buildv83 (lastModified2026-10-08T16:30:59Z,system-security) declaresextraPortalConfig.Hosting__ReportTo: https://control.systemorph.com. The serving pod booted 15:40Z, before that change, and still uses the old inbox. Nothing has applied the record since: there is no Reconcile or Roll forDeployments/buildon either control instance today. memex-cloud's record change, by contrast, producedrecord-reconcile-memex-cloud-v1-…(11:32Z) and…-v2-…(17:54Z) on control.systemorph.com.What closes the build shape: apply record v83 to build (a governed Reconcile of
Deployments/build, then confirm the boot line hands tohttps://control.systemorph.com/…and aselfupdate-roll-build-…appears on control.systemorph.com). I did not file that Reconcile. It is a deploy on an instance the record says retires at the operations cutover, and it needs an approval.Not established
RecordChangeReconcile's decision lines.Deployments/client-bv118) was not read.nodetype_bake: … 18 failed with no working build to regress from — Hosting/Backup, Hosting/BugFixPool, … Hosting/InstanceAction, …(Ops/Status/build, 18:47Z). That is unrelated to the hand-over and not investigated.On this issue's scope: it is now the fold target for the FleetWatch roll-alarm log site
b164c7a435ab60de(27 fingerprints: StalePortal on memex-cloud/control/memex/client-b/build, StuckRoll on build/memex, BabysitterDrift, RollUnobserved). These are different causes under one log site. This comment answers only the newest one.rbuergi commentedon Oct 11, 2026
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-11)
(re-posted 2026-10-11; originally by @rbuergi on 2026-10-09)
Verified on production 2026-10-09 ~14:25Z (control instance, control.systemorph.com). Closing for the
buildhalf. Theclient-bhalf was not re-measured here; see the last bullet.What was checked:
Ops/Actions/provision-build-20261009-values-halfisDone, and its operator run concludedsuccess(Memex run 37896239954, 17/17 steps). Step 8 readvault helm-values-build (633 bytes, families only)and applied helm revision 5.Ops/Actions/reconcile-build-20261009-after-values-halfis alsoDone(run 37896741693, 15/15 steps, revision 6). Itshosting-kv-syncedstep reportedall 9 declared key(s) are in their synced Secret(s), includingHosting__ControlInbox__Secret.origin: self-updateandisRouted: true. Each reason reads "build announced … — routed from the control inbox (MeshWeaver#4098)". Denominator: 6 roll actions for build on 2026-10-09 (ci.10300, 10305, 10308, 10310, 10319, 10322) plus 1 Restart.Ops/Actions/selfupdate-roll-build-3-0-0-ci-10319-05d0cc8c. Its reason says "build announced that 3.0.0-ci.10319 is available (running 3.0.0-ci.10310)", trigger SafetyNet. State isDone: migrationVersion: 60, helm revision 10,3/3 ready … running cr.meshweaver.cloud/memex-portal-ai:3.0.0-ci.10319.https://build.meshweaver.cloud/api/versionreturns commit9704e993f50c.ci.10150(07:00Z) toci.10319(14:09Z) by its own announcements.apply=control-lanedirectly (no log query was run). The routed actions above are the evidence instead.selfupdate-roll-build-3-0-0-ci-10322-e2fcb389), failed at mirror.meshweaver.azurecr.io/memex-portal-ai:3.0.0-ci.10322 does not exist, because the announced set has no published portal image. That is a separate defect: an unsealed or unpublished set is admitted. It is not this issue./healthreads Degraded. The ci.10322 announcement cites a failed MeshWeaver.AI live swap (HttpClient factory already has a registered client with the name 'WebSearchPlugin').client-bwas not re-measured: control and client-a-control hold no client-b actions. If client-b is still stale, reopen with its reading.rbuergi commentedon Oct 11, 2026
Re-measured 2026-10-11 ~19:50Z — the original defect is gone on our own instances; two portals remain, and neither needs a code change
The instances in this issue now roll themselves. On the control instance,
Ops/Actionsholds a self-update Roll for every promoted build today, opened byself-updatewith no human request:build:selfupdate-roll-build-3-1-10535,…-10537,…-10548,…-10553and…-10560.memex,memex-cloudandcontrol: the same, up to3.1.10565.build, the 7-day-stale portal in the title, last rolled to3.1.10560at 18:56Z.Two portals are still behind. I separated opened but unapproved (an owner act) from never opened (a defect).
1. A customer portal — opened, awaiting approval (owner act)
Ops/Actions/roll-<customer>-20261011-3-1-10502-bexists: requested 06:05Z, state AwaitingApproval. It needs "a global administrator other than the requester (or the maintainer)" to approve it.Admin/UpdatePolicyexplains why its self-updater never opened one. It runs3.0.0-ci.9887, which predates both of these:PlatformReleaseOrder,3.1.<run>above every3.0.0-ci.*). Underpattern: 3.*it still selects3.0.0-ci.10350as newest.announcementKeySecret, so the route is a per-deployment announcement key, not the fleet inbox secret.PlatformReleaseOrderTest,3.1.10000 > 3.0.0-ci.9999), so there is nothing to fix in code. The portal cannot reach that code without the roll./Admin/Settings/ControlLane) and a control administrator registers it (Doc/Architecture/SelfUpdateAnnouncementKey). Only then can its next promoted build open a Roll by itself.2. An estate's test instance — never opened, by the defect #6480 fixed (owner act to deliver it)
Admin/UpdatePolicyat 18:54Z: "HOLDING 3.1.10558 — the framework identity of release 3.1.10558 is not resolvable — no content bake has been published for it".3.1.10524, built from CD run 10524 at 11:52Z, before the fix. Its own gate therefore holds every release and never announces one, so no Roll is opened.Restartwas filed on its estate control instance at 14:47Z with the reason "bring the instance onto platform 3.1.10533". A Restart pulls no image (the plan says "Restart pulls no image"), so the instance is still on3.1.10524. That Restart is stillRunning / Verify one generation(observed until 22:41Z), and while it is open it also holds the routing lane.3.1.10533, approved on its estate control instance. Its own self-updater takes over from that build on.What I did not establish
Keeping this open until both portals are on a
3.1.xbuild and have each opened one Roll on their own.rbuergi commentedon Oct 13, 2026
Re-measured 2026-10-13 ~15:10Z. The code defect stays gone. One portal is still stale, and only an owner act can move it.
/api/version: build on70362c45ac, memex and memex-cloud onec7763883b, control on3182d81399. Every one is a 3.1.x build.Ops/Status/memexshowsself_update: … update available: 3.1.10729 — handed to the control lane … accepted (200).3.0.0(/api/versioncommit1bafcb2852). The roll opened for it on 2026-10-11,Ops/Actions/roll-<customer>-20261011-3-1-10502-b, was never approved. At 2026-10-12 15:41Z it was retired:state: Refused — stale — … waited for an approval since 2026-10-11T06:05:01Z, longer than 24 h … Re-request it if it is still wanted. Until that portal runs a 3.1.x build, it cannot open a Roll by itself, as the previous comment explains.Owner act:
/Admin/Settings/ControlLane), and a control administrator registers it.This stays open until the portal has opened one Roll on its own.
rbuergi commentedon Oct 16, 2026
State 2026-10-16: the self-updater defect is fixed on
main. What is left is owner acts, not code.Fixed on
main(traced from trigger to hand-over at746e341d4e):be5d157273,9d7b04e473,4459c97e41.3.0.0-ci.*ranked above3.1.x:4a6dfec6fe,02f827d89b.f0a0a35561.79d2cf41d8,187b790ac9.6c587469be.b95b896b48.In production: the platform's own instances roll themselves. On control,
Ops/Actionshas a self-update Roll per promoted build (3.1.11001–3.1.11004) for build, control, memex and memex-cloud, opened byself-updatewith no human request.Still stale: one customer portal. It reads
3.0.0+1bafcb2852. Its record now declaresupdatePattern: 3.*, and the Reconcile that applied it isDone. But a Reconcile pulls no image, so the pod still runs pre-fix code that can't select or announce a 3.1 build. It needs one governed Roll from outside:Closes when:
imageTag, a current3.1.x, and a second global administrator approves it within 24 h./Admin/Settings/ControlLane), and a control administrator registers it.selfupdate-roll-…action opened byself-updatewith no human request, and its/api/versionreads a 3.1 build.rbuergi commentedon Oct 20, 2026
Re-measured 2026-10-20 ~12:40Z. The instances in the title are fixed. One customer instance still cannot announce, and Systemorph/MeshWeaver.Plugins#3858 is the fix. It was a draft and is now ready.
Fixed: the instances in the title.
/api/versionon build, memex (public), control and memex (company) readsee1d4eeb27, which ismain~1.Ops/Actionsholds aselfupdate-roll-<id>-3-1-<run>for every promoted build (3.1.11242…3.1.11282). Each was opened byself-updatefor build, control, memex, memex-cloud and one customer instance. No person requested any of them.Still open: one customer instance (the one this thread called "the customer portal").
3.1.11217on 2026-10-19 (governed Roll, Done). Since then, noselfupdate-roll-<id>-*exists for it (search onOps/Actions: 0 rows).Ops/Status/<id>: 9 of 9 replicas sayself_update: … update available: 3.1.11282 (detect-and-notify — … Hosting:ReportTo names the control instance but Hosting:ControlInbox:Secret is empty, so nothing could be signed).3.*pattern,ContinuousandMinRollInterval 01:00:00are all rendered. Only the hand-over has no key.Root cause.
announcementKeySecretandHosting__ReportTo, but maps noHosting__ControlInbox__Secret. The roll'sactionPlan.renderedValuesconfirm that no such mount was rendered.Doc/Architecture/SelfUpdateAnnouncementKey: a record that declares its own key cannot announce with the fleet secret. So the instance has no route at all.AnnouncementKeyMounts. That derivation existed only in the draft Plugins#3858. Both of its preconditions merged on 2026-10-17: hosting-kv-ensure: --generate / --generate-only — create the self-update announcement key when absent #6917 and Systemorph/Memex#774.What I did.
dotnet build src/MeshWeaver.Fleet.Control -c Release -warnaserrorpasses clean against core main. That project compiles the Hosting sources.AnnouncementKeyMountsTestsandAnnouncementKeyPlanTests. CI runs them, because this macOS host cannot execute the image reference set.After it deploys.
hosting-kv-ensure --generate-onlyfirst.selfupdate-roll-<id>-….Hosting/PlatformBuildInboxhub, and theHosting/InstanceActionandHosting/DeploymentNodeTypes.This closes when that instance has opened one self-update Roll on its own.
Not established:
AdminInstanceSecretsearch returned 0 rows, but that may be RLS.The related issues are separate defects, not this one:
AnnouncementKeysbinding, rule 4. Its remaining acceptance item is removing the fleet-secret path for build and memex-cloud. It explains why the instance has no fallback, but it does not cause the stall.SelfUpdate:RegistryValidationUrlfrom the registry record) is an enhancement. Every affected record declares the URL, and detection works ("update available").rbuergi commentedon Oct 27, 2026
Closing: every live instance now opens its own Roll for each newer promoted build. Re-measured 2026-10-27 ~11:20Z, read-only, on the control instance (https://control.systemorph.com/Ops/Actions) and on the estate control instance.
The closing condition (20 Oct comment) was that the remaining customer instance opens one self-update Roll on its own. It has, since 21 Oct, and every other instance does too:
self-update(newest)build(the 7-day-stale portal in the title)selfupdate-roll-build-3-1-11792-c19fec19, also 11789, 11786, 11784 todaymemex,memex-cloud,controlselfupdate-roll-{memex-3-1-11798, memex-cloud-3-1-11792, control-3-1-11801}-…selfupdate-roll-<customer>-3-1-11792-…:state: Done,origin: self-update,isRouted: true, reason "announced that 3.1.11792 is available (running 3.1.11789), policy Continuous, pattern 3., … routed from the control inbox (MeshWeaver#4098)"*. Its first self-announced Roll was…-3-1-11331-…on 2026-10-21. Before that, every Roll was a manual one (the last on 2026-10-20).selfupdate-roll-<estate>-test-3-1-11789-…, plus one roughly every hour back to 00:28ZThe control instance holds six
Hosting/Deploymentrecords. Five are covered above. The sixth record isstatus: Suspended(teardown decided 2026-10-20), and its self-updater is meant to be off.Not established:
/api/version. TheDonestate on the routed Rolls is the evidence.StalePortalalarm: I did not run aLogsquery for it (this was a read-only pass). If it fires again for a live instance, reopen with that reading.