coding-agents: a session write-back that fails on its last submission has no durable cross-session replay

Versions: coding-agents 0.8.1, verified against 0.8.2 (submitWithRetry, cursorFile and planRetain identical between the two releases), Hindsight server 1.1.0 (self-hosted), harnesses codex and workbuddy on Windows 11.

Summary

When a session write-back fails for any reason other than rate limiting, submitWithRetry retries zero times. The failure is then recorded only as a WARN and a retain_failed diagnostic, and the in-flight payload is not handed to anything that will replay it later.

Recovery exists, but only within the same session: the per-session cursor keeps a dirty flag (replace mode) or a pending queue (append mode), and the next Stop for that same sessionId retries. Once the session ends and is not resumed, nothing reads that cursor again — so the defect is the absence of a durable, automatic cross-session replay, not that the payload is never persisted.

Being precise about what is persisted matters, because it narrows the fix:

• replace mode — renderSessionJsonl content is re-derived from the host transcript on every attempt. Only metadata (turns, fingerprint, dirty: true) is written to the cursor before submitting, so the transcript is the real source of truth here.
• append mode — each entry is pushed into queue and written to the cursor with its content and operationId, then dequeued only after that entry submits successfully:

if (plan.mode === "append") {
  const content = turns.slice(plan.fromTurn).map((t) => JSON.stringify(t)).join("\n");
  queue.push({ content, operationId: opId("append", content), at: Date.now() });
}
cursors?.write(sessionId, { ...accepted(), pending: queue });

• Deleting the cursor is not fatal by itself. planRetain returns replace when !cursor || cursor.dirty, so as long as the host transcript is still readable, a subsequent Stop rebuilds a full replace. The cursor is a progress marker, not the only copy of the data.

The two things that actually break

1. No cross-session replay trigger. Both the dirty marker and the append pending queue are keyed by sessionId and consumed only by that session's own Stop hook. SessionStart of a new session does not look at them. If the last Stop of a session fails, that session's remaining turns have no owner unless the user reopens it.

2. Retry excludes the transport errors that actually happen. Only RateLimitedError is retried:

var DEFAULT_RETRY_WINDOW_MS = 6e4;
var RETAIN_RETRY_ATTEMPTS = 2;
var REQUEST_BUDGET_MS = 15e3;
async function submitWithRetry(send, isStillNewest, retryUntil) {
  for (let attempt = 0; ; attempt++) {
    try { return await send(); }
    catch (e) {
      const limited = e instanceof RateLimitedError;
      if (!limited || attempt >= RETAIN_RETRY_ATTEMPTS || !isStillNewest()) throw e;
      const wait = e.retryAfterMs || 1e3;
      if (Date.now() + wait + REQUEST_BUDGET_MS > retryUntil) throw e;
      await sleep(wait);
    }
  }
}

ECONNREFUSED (daemon restarting — a few seconds of backoff would cover it) and request timeouts are rethrown on the first attempt, and the eventual catch in runRetainHook only logs:

catch (e) {
  log.warn(harness, "session write-back failed", { error: describeError(e) });
  diag(harness, "retain_failed", { ms, error: describeError(e), session: sessionId });
}

Evidence (one machine, three weeks)

• retain_ok / retain_failed over 2026-10-05 … 2026-10-25 — codex: 222 ok / 6 failed; workbuddy: 4 ok / 0 failed after being wired up.
• All 6 failures were transport-level, not rate limiting: 5 × connect ECONNREFUSED 127.0.0.1:8888, 1 × The operation was aborted due to timeout.
• The server's failed-operation list was empty (total: 0) at the time of checking, so none of these reached the server as a retryable operation.
• Per-session outcome, from GET /v1/default/banks/<bank>/documents/conversation:<sessionId>:

How to read the 404: it proves only that this document id does not exist now. It does not distinguish “the submission never arrived” from “the submission arrived, extraction failed, and the document was cleaned up” — and a timed-out request may well have been accepted server-side. The row is consistent with the defect above, not proof of it. The durable, unambiguous evidence is the mechanism: no component replays a failed session's payload after that session ends.

How to reproduce the mechanism deterministically

1. Start a session, take at least one turn.
2. Stop the daemon (or block 127.0.0.1:8888) and end the session. Observe WARN session write-back failed + retain_failed; no retry is attempted.
3. Note the cursor at %TEMP%\hindsight-<harness>\<sessionId>.retain.json — it carries dirty: true (or a non-empty pending).
4. Restart the daemon and open a new session. The new session's SessionStart never reads the old cursor; the old session is never written unless it is reopened.

Suggested fixes

1. Retry transport errors, not just 429. ECONNREFUSED, ETIMEDOUT and 5xx are exactly the "not now" cases the existing 60 s window was built for; bounded backoff would already survive a daemon restart.
2. Replay orphaned cursors on SessionStart. Sweep %TEMP%\hindsight-<harness>\* for cursors that are dirty or hold pending, and replay them from the next session of that harness. Submissions are already idempotent via operationId (server ≥ MIN_IDEMPOTENT_RETAIN_VERSION 0.8.6), so replay is safe.
3. Make the failure visible. diag.jsonl already records retain_failed; surfacing it through hindsight_sync_status / hindsight-coding-agents stats, or returning a structured hook result the host can show, would stop this from being silent.
4. Consider moving the cursor out of tmpdir() — #4325 moved the logs to ~/.hindsight/coding-agents-logs but left the cursor in %TEMP%. Not fatal (replace mode recovers from the transcript), but append-mode pending content does disappear with it, and Storage Sense deletes %TEMP% by design.

Relation to existing issues

• fix(coding-agents): drain pending Pi write-backs on shutdown #5168 drains in-flight submissions on Pi shutdown and states explicitly that "flushRetains() waits for existing submissions without resending them; failures and timeouts emit diagnostics and warning logs". This issue is the gap left after that drain — for codex/workbuddy there is no equivalent drain, and even for Pi nothing replays a failed submission.
• Conversation write-back delta supersession silently loses turns #5079 was server-side delta supersession silently dropping turns (fixed) — a different failure point from a submission that never landed.
• coding-agents: a session past MAX_TRANSCRIPT_BYTES permanently loses append mode, replacing the whole transcript every turn #4380 covers sessions past MAX_TRANSCRIPT_BYTES losing append mode; that interacts with this issue, since a replace-mode retry needs a readable transcript.

Happy to test a patch: this machine runs both codex and workbuddy against a self-hosted 1.1.0 server, and the daemon can be killed on demand to reproduce.