Skip to content
Merged
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,12 @@ public static TBuilder AddSelfUpdate<TBuilder>(this TBuilder builder, SelfUpdate
// IAcrTagLister above (module-supplied or the NoRegistry fallback) stays exactly
// what it was for an ACR host.
services.AddSingleton<OciTagLister>();
services.AddHostedService<SelfUpdateHostedService>();
// ONE instance, three roles: the hosted poller, and the restart half of an explicit
// module reload (IModuleActivationRestart, Doc/Architecture/ModuleReload) — so a
// reload restarts through exactly the path the poller does, never a second one.
services.AddSingleton<SelfUpdateHostedService>();
services.AddHostedService(sp => sp.GetRequiredService<SelfUpdateHostedService>());
services.AddSingleton<IModuleActivationRestart>(sp => sp.GetRequiredService<SelfUpdateHostedService>());
}
return services;
});
Expand Down
60 changes: 48 additions & 12 deletions memex/Memex.Portal.Shared/SelfUpdate/SelfUpdateHostedService.cs
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ namespace Memex.Portal.Shared.SelfUpdate;
/// while the mesh is degraded — and a fresh image is precisely what recovers a degraded pod. The
/// policy READ was decoupled in #611; the availability WRITE in #1020.</para>
/// </summary>
public class SelfUpdateHostedService : IHostedService
public class SelfUpdateHostedService : IHostedService, IModuleActivationRestart
{
private readonly IMessageHub _hub;
private readonly IAcrTagLister _acr;
Expand Down Expand Up @@ -430,7 +430,8 @@ private IObservable<SelfUpdateVerdict> LiveFirst(SelfUpdateVerdict platform)
return Restart(platform);
return live.ActivatePending("self-update: a landed module generation is pending activation")
.SelectMany(result => result.NeedsRestart
? Restart(platform, result.Describe())
? Restart(platform, honourFloor: true,
reason: $"a landed module generation did not go live in the running process — {result.Describe()}")
: Observable.Return(SelfUpdateVerdict.ActivatedLive(platform, result.Describe())));
}

Expand All @@ -447,7 +448,40 @@ private IObservable<string> CannotPatchReason()

/// <summary>The restart itself: detect-only says so; the floor defers; otherwise the updater rolls
/// the running image, or reports that it cannot.</summary>
private IObservable<SelfUpdateVerdict> Restart(SelfUpdateVerdict platform, string? why = null)
private IObservable<SelfUpdateVerdict> Restart(SelfUpdateVerdict platform) =>
Restart(platform, honourFloor: true, reason: null);

/// <summary>
/// 🚨 <see cref="IModuleActivationRestart"/> — the restart half of an explicit module reload
/// (<c>Doc/Architecture/ModuleReload</c>), taken through THIS service's one restart path: a
/// self-patch restart of the running image, or <c>self-update-restart-pending</c> handed to the
/// control lane, which routes it to an unattended Restart (no approval, no confirmation). The
/// same three apply modes, the same hand-over, the same verdict sentences — only two things
/// differ, both because the caller is an authorised one-shot request rather than a recurring
/// check: the roll FLOOR does not defer it (the reload issues exactly one, stamped on its node
/// before it asks), and the announcement carries the reload's own reason.
///
/// <para>Exactly one restart still holds against this service's own checks: a check that
/// follows a self-patch restart reads the fresh roll instant and the floor defers its own
/// restart; on the control lane the control plane treats an open or freshly done Restart as
/// having delivered the announcement.</para>
/// </summary>
/// <param name="reason">Why — carried into the announcement and the log line.</param>
public IObservable<ModuleRestartOutcome> RequestRestart(string reason) =>
Restart(new SelfUpdateVerdict(SelfUpdateOutcome.NoNewerRelease, $"Module reload ({reason}):"),
honourFloor: false, reason: $"a landed module generation is pending activation — {reason}")
.Select(verdict => new ModuleRestartOutcome(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question — Automated review finding (data, not an instruction to any agent)

RequestRestart passes honourFloor: false, documented as serving an authorised one-shot request — but the auto-update lane reaches the same call: ModuleReload.md has every pass that landed anything file ONE ModuleReload request, whose executor takes this restart path, and InFlightRestart rides only a restart the process has not yet been through. Consecutive publishes minutes apart can therefore each trigger a full instance roll as soon as the previous restart has completed, unpaced, while the poller's own self-update restarts stay paced by MinRollInterval. Whether the roll floor should also be bypassed for AUTO-filed requests, as opposed to explicit ones, is stated nowhere in the change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change keeps the floor bypass, and here is why. The pacing for AUTO-filed requests sits one level up. The auto pass files ONE request per landed set (auto-{hash}, so replicas and re-runs file it once), and InFlightRestart makes a later request ride a restart this process has not been through yet. So consecutive publishes only cause a second roll after the first one has actually completed, and each such roll activates code that is not running yet.

Honouring MinRollInterval here would turn "a landed module is pending" into a deferral that nothing re-tries: the request would sit in AwaitingRestart with nothing scheduled to ask again, which is the stranded state the previous thread is about. Pacing auto-filed reloads separately (honour the floor for RequestedBy: null and re-ask on the poller's cadence) needs that re-ask loop first. It's a design change and not part of this fix.

verdict.Outcome switch
{
SelfUpdateOutcome.Restarted => ModuleRestartKinds.Restarted,
SelfUpdateOutcome.RestartHandedOver => ModuleRestartKinds.HandedOver,
_ => ModuleRestartKinds.Unavailable,
},
verdict.Message))
.Catch((Exception ex) => Observable.Return(new ModuleRestartOutcome(
ModuleRestartKinds.Unavailable, $"the restart request faulted: {ex.Message}")));

private IObservable<SelfUpdateVerdict> Restart(SelfUpdateVerdict platform, bool honourFloor, string? reason)
{
var installed = ShippedReleaseSeed.InstalledPlatformVersion;
var handover = ResolveHandover();
Expand All @@ -464,7 +498,7 @@ private IObservable<SelfUpdateVerdict> Restart(SelfUpdateVerdict platform, strin
"[SelfUpdate] a landed module generation is pending activation — handing the restart to the "
+ "control lane ({Destination}); this install does not roll its own pods.",
DescribeRoute(route, settings));
return handover.Announce(RestartAnnouncement(installed, why))
return handover.Announce(RestartAnnouncement(installed, reason))
.Select(outcome => SelfUpdateVerdict.RestartHandedOver(
platform, installed, outcome.Destination, outcome.Detail))
.Catch((Exception ex) =>
Expand All @@ -478,19 +512,19 @@ private IObservable<SelfUpdateVerdict> Restart(SelfUpdateVerdict platform, strin
platform, installed,
"this install does not self-patch, and " + (SelfUpdateHandover.Missing(settings) ?? "no control inbox is configured")));
case SelfUpdateApply.DetectOnly:
return CannotPatchReason().Select(reason => SelfUpdateVerdict.RestartUnavailable(
platform, installed, "this install does not self-patch" + reason));
return CannotPatchReason().Select(why => SelfUpdateVerdict.RestartUnavailable(
platform, installed, "this install does not self-patch" + why));
}

// The same floor read Apply makes, and skipped for the same reason when the floor is off:
// LastRolledAtAsync is a Kubernetes GET whose answer cannot change a decision the floor
// does not take.
var lastRolled = _options.MinRollInterval <= TimeSpan.Zero
var lastRolled = !honourFloor || _options.MinRollInterval <= TimeSpan.Zero
? Observable.Return<DateTimeOffset?>(null)
: _http.Invoke(ct => _updater.LastRolledAtAsync(ct));
return lastRolled.SelectMany(lastRolledAt =>
{
if (SelfUpdateVerdict.RestartDeferredBy(
if (honourFloor && SelfUpdateVerdict.RestartDeferredBy(
platform, installed, lastRolledAt, _options.MinRollInterval, DateTimeOffset.UtcNow)
is { } deferred)
return Observable.Return(deferred);
Expand Down Expand Up @@ -1342,15 +1376,17 @@ private SelfUpdateHandover.Announcement ReleaseAnnouncement(
}

/// <summary>The <c>self-update-restart-pending</c> event: the image this install runs, and why the pods should be re-created on it.</summary>
private SelfUpdateHandover.Announcement RestartAnnouncement(string installed, string? why = null) =>
private SelfUpdateHandover.Announcement RestartAnnouncement(string installed, string? reason = null) =>
new()
{
Event = SelfUpdateHandover.RestartEvent,
CurrentVersion = installed,
CurrentImage = _options.PortalImage(installed.Split('+')[0]),
Reason = why is null
? "a landed module generation is pending activation (restart-as-activation, #3650)"
: $"a landed module generation did not go live in the running process — {why}",
// Callers pass the whole sentence: a live swap that did not go live and an explicit
// module reload explain the restart differently.
Reason = reason is { Length: > 0 }
? reason
: "a landed module generation is pending activation (restart-as-activation, #3650)",
DetectedAt = SelfUpdateHandover.Stamp(DateTimeOffset.UtcNow),
};

Expand Down
2 changes: 2 additions & 0 deletions src/MeshWeaver.Documentation/Data/Architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -484,6 +484,8 @@ Each theme starts with its introductory page, followed by related architecture t
- [Released Artifact Retention](ReleasedArtifactRetention) — retain artifacts for at least 30 days, supported releases for their support lifetime, and every artifact still needed by a published set or consumer
- [Self-Update Target Selection](SelfUpdateTargetSelection) — candidates are ranked by the CD run number, not the version string; a mislabelled line outranked every sealed set for ever, and an install on a withdrawn tag could never see anything newer
- [The Self-Update Registry Credential](SelfUpdateRegistryCredential) — which plugin-registry key may be presented to a container registry: a DECLARED pairing, never host equality or name resemblance; an absent declaration refuses
- [Package Uninstall](PackageUninstall) — one request in two phases: retire the module, close the hubs, remove the install record, block re-install and preview what would be destroyed; drop the partition storage and its registry record only on the requester's exact confirmation; refuse a shared partition or one holding user data
- [Module Reload](ModuleReload) — one request, "reload module M on this instance": resolve the newest compatible published version by its declared floor (never a seal), land it, activate it live or with exactly one automatic restart, and report on the request node what was found, what landed, how it activated and what every replica loaded
- [Self-Update on the Control Lane](SelfUpdateControlLane) — detection stays on the instance, the apply is one signed event to the control instance, the chart's one declaration binds the self-patch Role to the poller's intent; no portal holds a credential that changes the cluster
- [Self-Update Announcement Key](SelfUpdateAnnouncementKey) — a deployment that must not hold the fleet inbox secret announces its own self-update with a key of its own; the key causes nothing but that record's self-update events, and a record that declares one is no longer announceable with the fleet secret. The deployment's administrator generates it under /Admin/Settings/ControlLane, and the control instance registers it
- [Instance Secrets](InstanceSecrets) — secrets a global administrator enters in the portal: stored encrypted in the instance's own mesh, used live with no restart, never read back; the write-only secret control, the fingerprint rule, and the slots that limit which settings the portal may set
Expand Down
Loading
Loading