Mental models: surface when a failed refresh has paused automatic refresh (and let a fix un-pause it) #6077

Use Case

Self-hosted Hindsight (hindsight-api-slim 1.0.0, embedded pg0, local_external mode) sitting behind a long-running agent as its long-term memory. One bank holds four mental models that refresh automatically (bank config mental_model_min_refresh_interval_seconds=900). Operators do not read the model pages by hand — they trust them to be current, the same way a K8s controller trusts its reconciliation loop.

Problem Statement

Since #4532, a failed refresh stamps last_refresh_failed_at, and _automatic_refresh_paused() skips every subsequent scheduler or consolidation trigger until a manual refresh moves last_refreshed_at past that stamp. #5394 correctly carved transient provider errors out of that rule. Deterministic failures still pause forever, and — this is the part I would like to change — nothing tells the operator the pause is in effect. The page goes stale forever while /health is green, the queue is empty and no metric moves.

Observed on 2026-10-25 (local time, single bank):

16:35:04  refresh_mental_model  records a failure for mental model 'active-projects'
16:35:06  Task execution failed: refresh_mental_model,
          error: OutputTooLongError: LLM output exceeded token limits
          (reflect final synthesis hit finish_reason=length)
16:35:21  [CONSOLIDATION] Skipped refresh for mental model active-projects:
          its last refresh failed
16:35 → 18:46  nothing retries it: is_stale=true, zero pending/processing
               operations, /health healthy, queue empty
18:46:42  manual POST /mental-models/active-projects/refresh  → 18:48:18 completed

Facts worth noting:

• The pause was not the problem by itself — replaying the same truncated prompt every tick would have burned tokens for nothing (Unbounded paid-LLM spend: mental-model refresh loop never converges (no cost/rate guard) #4532 is right).
• But by the time we noticed, the failure was already non-repeatable: we had raised HINDSIGHT_API_REFLECT_MAX_COMPLETION_TOKENS to 32000 and lowered HINDSIGHT_API_REFLECT_MAX_CONTEXT_TOKENS to 64000, and the very next manual refresh of the same query succeeded in 96 s. A configuration fix silently has no effect until a human presses the button.
• GET /mental-models exposes is_stale, last_refreshed_at, last_refresh_failed_at but nothing that says "automatic refresh is currently paused". Deriving it requires re-implementing last_refresh_failed_at > last_refreshed_at on the client, which is exactly the kind of duplicated logic that drifts.

Our workaround is a cron job that recomputes that predicate and POSTs /refresh for stale models older than 90 minutes. It works, but it re-derives server-internal state and can race with real in-flight operations.

How This Feature Would Help

Operators (and any self-hosted deployment whose pages are consumed by another agent without a human reading them) could detect a paused refresh the same way they detect any other degraded state — one field, one metric, one alert — instead of noticing days later that a page has been serving stale content. It also lets a configuration fix take effect without manual intervention.

Proposed Solution

Any one of these would close the gap; the first is the smallest:

1. Expose the paused state. Add automatic_refresh_paused: bool to GET /mental-models/{id} (and the list), computed by the same _automatic_refresh_paused() the scheduler uses — no client-side re-derivation. Optionally add a paused_mental_models counter to /stats / /health so a one-line probe catches it.
2. Log the decision. One line when a model enters or leaves the paused state ([MENTAL_MODELS] automatic refresh paused for <id> … / resumed). Today the only trace is [CONSOLIDATION] Skipped refresh …, which is easy to miss and only appears when a consolidation happens to run.
3. Optional: let the pause expire for failure classes that can be configuration-dependent. OutputTooLongError / truncation is not necessarily "the same doomed prompt": a completion-budget or context-budget change resolves it. A cooldown (e.g. HINDSIGHT_API_MENTAL_MODEL_REFRESH_PAUSE_MAX_HOURS, default off) would keep Unbounded paid-LLM spend: mental-model refresh loop never converges (no cost/rate guard) #4532's cost protection while allowing a fixed deployment to recover on its own.

References

• Unbounded paid-LLM spend: mental-model refresh loop never converges (no cost/rate guard) #4532 — the pause itself (cost protection, correct intent)
• Knowledge pages: a temporary LLM failure (rate/session limit) pauses automatic refresh forever #5394 — transient provider failures must not pause (the carve-out my case fell outside of)
• Knowledge pages: a refresh cancelled at the wall-clock limit during a provider backoff still pauses automatic refresh #5561 — a different non-transient path into the same pause (wall-clock cancel during backoff; I confirm its fix is in the v1.1.0 release notes, while this report is from 1.0.0 and is a deterministic failure, not a timeout)