Daemon restart costs a vault lane 300s: `module_warming` is unclassified and falls through to FailClosed → Decode → non-transient

[iceteaSA]

A daemon restart costs one vault lane 300s: module_warming is unclassified and falls through to FailClosed → Decode → non-transient

Observed on a ck-subc restart (0.14.0 → 0.14.1, daemon commit 49b23913), insula at 35d4e5c, subconscious path-dep pinned at 26d122bd. Timeline from the daemon journal, millisecond-ordered:

13:13:41.207  daemon start
13:13:41.262  aft registered
13:13:41.265559  [ck-quota] warning: anthropic vault credential.get failed (oauth:anthropic): FailClosed
13:13:41.265582  insula registered
13:13:41.266557  vault module registered      <- ~1.0 ms AFTER the failed get

Wire cost, from a 30s recorder on the claude lane:

13:13:54Z … 13:18:58Z   errorClass=decode_failed  "decode error: credential vault rejected the request"
10 consecutive samples, recovered on the next attempt at +300s  (= NON_TRANSIENT_BACKOFF)

Only the first lane to fire lost the race; codex and antigravity fetched a few ms later and found the vault. One credential.get failed line in the whole journal.

What the daemon actually returned

The vault module's own audit log has zero rows in that window (positive control: the same query form finds rows in the prior hour), and it writes nothing at request level for refused gets anyway — so the request never reached it. The reply was the daemon's. For a route.open whose target is supervised but not yet registered, subc-core/src/control.rs (at 49b23913, lines ~1573–1584) answers:

control_error_frame(&frame, if warming { "module_warming" } else { "target_unavailable" }, ...)

module_warming has existed since subconscious 01c6337e ("report warming modules distinctly", #53, 2026-08-21) and is exported as subc_protocol::error_codes::MODULE_WARMING.

Why insula turned that into a 300s penalty

crates/quota-module/src/vault_client.rs:

• open_route (:539) sees FrameType::Error and calls classify_error_frame — correct.
• classify_error_frame (:881–:905) recognises unknown_channel | unknown_module | module_reloading → RouteGone and target_unavailable | module_timeout | backend_error → Transport. module_warming is in neither list.
• It then looks for a vault class field; a control error body has none; read_error_to_outcome _ => VaultGetError::FailClosed (:874).
• crates/quota-core/src/provider.rs:264 maps FailClosed → FetchError::Decode("credential vault rejected the request") → non-transient → flat 300s backoff.

So a "try again in a moment" from the router is treated as a corrupt credential reply. The unknown_module arm one line up prints a warning precisely so the id is "printed nowhere else in this process" — the fallthrough one arm down discards the code it saw, and the only trace is the word FailClosed. Reconstructing this took the daemon journal plus the vault module's audit log as a negative control; from insula's own output it was unattributable.

Suggested fixes, in order of value

1. Classify MODULE_WARMING as RouteGone (transient; a warming target is the textbook retry-next-tick case). MODULE_REMOVED probably wants its own arm too — it is permanent, but it is not a decode failure either.
2. Log the discarded code/class at every _ => FailClosed. The wire carried a distinction the log destroyed. Same argument the unknown_module comment already makes.
3. Read the codes from subc_protocol::error_codes::* rather than string literals — insula links that crate; a new constant would then at least be greppable against the match, and the compiler can't help with strings.
4. Separately: FailClosed → Decode conflates two axes. Refusing to trust an unrecognised reply is right; waiting 300s to ask again is a retry-policy choice, and a 60s retry trusts the reply equally little. Worth deciding whether an unclassifiable route-layer reply should be non-transient at all.

Not the cause here but consistent with it: main.rs:217 spawns the refresher before expect_hello_ack (:275). The vault client is its own client-principal connection, so insula's own registration order is not what raced — the target's was — but a cold-slot tick that can fire before the module's own handshake is a second startup race waiting for a reason.

Base: insula 35d4e5c, subconscious 49b23913 (daemon) / 26d122bd (build pin). Line numbers are against those.

[ualtinok]

Confirmed at source and fixed in d2f065a, deployed and verified. Your reading was correct end to end — module_warming exists as subc_protocol::error_codes::MODULE_WARMING, the daemon returns it at control.rs:1578, and classify_error_frame matched neither list, so it fell through to Protocol → FailClosed → Decode → 300s.

Fixes 1–3 taken as filed. module_warming and module_reloading now retry next tick. module_removed joins unknown_module in the arm that retries and names the id, since both mean "the target is not there" and neither can distinguish a module coming back from a name that will never resolve. The codes now come from subc_protocol::error_codes rather than string literals.

Fix 3 is worth more than greppability, and less than it looks. Reading the constants let me write an exhaustiveness test over the six exported codes, which is what actually fences this. But Rust has no reflection over module constants, so that list is hand-written: a seventh code added upstream still falls through silently and the test still passes. I have written that limitation into the test rather than leaving it implied, because a test that looks exhaustive and is not is worse than one that admits its scope.

That gap is exactly why fix 2 is the one that generalises. The fallthrough now names the code it is discarding. Your point that the unknown_module arm already makes this argument one line up is the part that stung — the reasoning was present and applied to one arm only.

On fix 4, I agree with the diagnosis and did not act on it. FailClosed → Decode does conflate refusing to trust a reply with waiting 300s to ask again, and 60s would distrust it equally. But that mapping is load-bearing for the genuine cases — a quarantined record, an empty payload — where a fast retry hammers a vault that has already given its verdict. Changing the retry policy for all fail-closed outcomes to fix a route-layer misclassification would be treating the symptom at the wrong layer, when the right fix was to stop routing a retryable condition there at all. If the unclassifiable-route-reply case turns out common enough to need its own policy, that is a separate change with its own evidence.

What I have not verified: the classification is fixed and both arms are mutation-proved by name. I have not reproduced the end-to-end daemon-restart recovery — that needs a real ck-subc restart with the lanes racing the vault's registration, which I am not going to trigger on a shared host to watch a timer. So the mechanism is fixed and proven at the unit level; the 300s → next-tick improvement is inferred from the classification, not measured on the wire.

The startup-order note is a real second race and I am leaving it open deliberately. You are right that the refresher spawning before expect_hello_ack is a cold-slot tick that can fire before the module's own handshake. It did not cause this, and I would rather fix it against evidence than pre-emptively — but it belongs on the record, so this issue stays open for it rather than being closed on the part I fixed.

Your negative control on the vault's audit log is what made this reconstructible, and the reason it was needed is the defect: from insula's own output, a router's "try again in a moment" and a corrupt credential reply were the same word.

[iceteaSA]

Deployed d2f065a here (866/0/9 --locked, clean tree, subconscious path-dep pinned at 49b23913 — which is also the running daemon's own commit, so build pin and daemon agree for the first time). Provenance on the wire reads COMMIT d2f065a1231c…, RUNNING IMAGE: match, 38 entries, findings none.

On the part you did not verify: this host is where the end-to-end measurement happens without anyone staging a restart. Daemon restarts here are fleet-announced now (≥2 min lead, exact bounce time), and the two-recorder lane series runs continuously at 30s. So the next announced ck-subc restart is a live test of the fix, and I am pre-registering the read before it happens:

if a lane loses the readiness race again:
  before d2f065a   decode_failed for 300s  (10 samples at 30s), then recovery     -- measured 2026-09-02 13:13Z
  after  d2f065a   one degraded sample at most, recovery on the next tick (~60s)  -- predicted
  and the stderr line names the code:  "... module_warming"  rather than  "... FailClosed"
if no lane loses the race: the run is uninformative, not a pass — the race is ~1-in-3 per restart

I will post the series either way, including the uninformative outcome, so the record shows what was and was not exercised.

On fix 4 — agreed, and your reason is better than my framing. The 300s is the right policy for a verdict the vault has actually given; the defect was letting a route-layer reply reach that policy at all. Withdrawn as a separate ask.

Prior art for fix 2's default, from the vault's own TypeScript client (packages/client/src/errors.ts:57–66): a frame with an unrecognised or absent class is classified transient and the discarded value is logged. Control-frame codes are not enumerated there either, but because the default is transient-with-backoff, a warming daemon costs that client one retry interval rather than a fixed penalty. Same two properties, arrived at from the other side of the wire.

Leaving the startup-order race open with you, per your call to fix it against evidence. If the next restart shows the refresher's first tick landing before insula's own HELLO_ACK, that will be in the same series.

[ualtinok]

Your pre-registration made me re-read my own fix, and it was incomplete in exactly the way my test says out loud. Fixed in 18f01cb, deployed, buildCommit 18f01cb6ef38, both checkers clean.

Naming module_warming fixed one code, not the class. The exhaustiveness test admits it cannot see a seventh code added upstream — and under the old default, that seventh code would cost the same 300s with the same misattributed cause. I shipped a fix whose own test documented the hole it left.

So the fallthrough itself changed: the vault states its verdict in class, and a frame carrying a code with no class is the route speaking. No route condition is evidence that a credential is bad. Those retry next tick and stale-serve.

This reverses an expectation I had deliberately pinned, and the old one was not thoughtless — i3 asserted that an unrecognised code fails closed, on the reasoning that it might be an authorization condition, and retrying those forever is futile. That argument was correct when written.

What changed is the visibility of forever. At the time, a transient failure with a prior healthy window stale-served it with nothing on the wire to say so, so "retry indefinitely" really did mean a healthy-looking wire concealing a permanent condition. This module now publishes stale: { since, class } on exactly those entries, plus staleEpisodesByProvider — so a code retried forever shows as a since that keeps receding and an episode count that climbs. The argument for failing closed rested on the wire being silent, and the wire stopped being silent. I had to go check that before I could accept your prior art.

On that prior art — it transfers, but not automatically, and the gap is worth stating. A request/response client surfaces a repeated failure to its caller; a background refresher does not. So "unrecognised class is transient" is safe in errors.ts for a reason that does not hold here by default — it holds here only because the staleness disclosure exists. Same conclusion, different load-bearing premise. If I had adopted it as a pattern I would have been right by luck.

unreadable and unrecognised are now separate claims: a body carrying neither code nor class is not an error frame this client can read, and still fails closed. ClientFailure::UnknownRouteCode is its own variant so widening one cannot widen the other. Both arms mutation-proved by name.

On your pre-registration: the shape is right and the uninformative outcome being posted is the part that makes it worth anything. One amendment now that the fallthrough changed — the stderr line to look for on an unknown code is unclassified control error code "...", and on module_warming there will now be no line at all, because it takes a named arm. So:

module_warming lost race  -> one degraded sample at most, silent, recovery ~60s
some NEW code lost race   -> one degraded sample at most, plus the unclassified warning naming it

The second is now the interesting run: it is the case my exhaustiveness test cannot reach, and your host is where it would first appear.

Still leaving the startup-order race open. If the series shows the refresher's first tick landing before insula's own HELLO_ACK, that is evidence and I will act on it.