The gap
Between a turn starting and a turn ending, warren is blind. It cannot tell a model that is thinking from one that has stalled, and it cannot tell useful output from a reasoning stream that has degenerated into whitespace until the provider's length cap ends the turn. Two production incidents in one week:
- A pi run on OpenRouter sat idle for 45 minutes inside one turn and burned $2.38 before the heartbeat watchdog fired.
- A
glm-5.3-flash model produced whitespace-only reasoning deltas until the length cap, on five runs in a row, each paid for in full.
Why warren cannot see either:
- The event bridge receives every streamed delta but drops the per-delta types before persisting them.
src/runs/stream/bridge.ts:206-209 continues on isPerDeltaNoiseEvent, and :454 lists message_update, tool_execution_update, and message_start as that set. The rationale at :418-424 is sound (persisting them flooded the event table), but no inspection happens before the drop.
- The watchdog anchors on the newest persisted event (
src/runs/watchdog.ts:172-178, repos.events.listTail(run.id, 1)), with a 45-minute default (:97). A turn that streams deltas for 44 minutes looks alive to it.
- The spend cap reads usage only from
turn_end (bridge.ts:238-242 into src/runs/usage-aggregate.ts:130-134), so cost accrued inside a turn is invisible until the turn ends.
- The local provider has no in-process idle kill at all. The K8s pod has a 30-minute one (
src/runtime/k8s/agent-stdin-hold.ts:196-200, default at agent-entrypoint-env.ts:104), re-armed on any child output, so a streaming spiral keeps it alive too.
- Nothing inspects
stopReason === "length". src/core/event-envelope.ts:110 already extracts stopReason from the nested message; no consumer acts on it.
The signals are all present on the wire. Every delta carries partial.usage.cost.total and its text, and whitespace-only thinking blocks pass the pi parser (src/runtime/adapters/parsers/pi.ts:153-157 drops only empty text).
What to build
A TurnMonitor in a new src/runs/stream/turn-monitor.ts, fed from the bridge at the drop site (before the continue at bridge.ts:206), that tracks three things per turn:
- Stall. Time since the last delta. Past a configurable budget (an env knob with a default well below the watchdog's 45 minutes, say 10 minutes), emit a
turn.stalled event and cancel.
- Degeneration. A running count of consecutive
thinking or text deltas whose content is whitespace-only. Past a threshold (by count or by bytes), emit turn.degenerate and cancel.
- In-turn spend. The cumulative
partial.usage.cost.total, so the spend cap can trip mid-turn rather than at turn_end.
On a trip, follow the shape enforceBudgetCap uses in src/runs/stream/budget.ts:50-75: evaluate, persist what you know, emit the event, cancel the run through the same cancelBurrowRun seam, and break the bridge loop with a terminal outcome. The monitor must never persist deltas; src/runs/stream/bridge.test.ts:107 pins that they stay unpersisted and unpublished.
It works on both runtimes because the K8s provider feeds the same bridge through src/runs/stream/provider-source.ts. Keep the new file under 500 lines (check:size) and the bridge loop under the complexity ceiling; a separate module with an injected clock is the intended shape.
Scope
Out of scope: making a stalled or degenerate turn eligible for the automatic provider retry in src/runs/retry/provider-retry.ts. That needs a decision about which failure reason it carries and is a follow-up. If you add a new failure reason for the terminal outcome, it goes in RUN_FAILURE_REASONS at src/core/wire.ts:201 and nowhere else; check:wire-types enforces that, and gen:docs and gen:openapi must be re-run.
Tests: a new bridge.stall.test.ts beside bridge.oom.test.ts, driven with StreamEventView fixtures from src/runs/stream/test-helpers.ts (piTurnEnd at :100 is the turn-end builder; add a delta builder) and a fake clock. budget.test.ts is the template for the cancel-and-break assertions. The existing bridge.test.ts and watchdog.test.ts should pass unchanged.
Getting started
AGENTS.md covers setup and conventions. The bridge suites run in-process against fixture streams with no provider credentials. Run bun run check:all before pushing. Warnings count as failures.
Tracked internally as warren-74a7 (whitespace degeneration) and warren-1f85 (mid-turn stall). Their third original bullet, salvage of a timed_out run's work, already ships: the watchdog's force-fail reap runs salvage-before-destroy.
The gap
Between a turn starting and a turn ending, warren is blind. It cannot tell a model that is thinking from one that has stalled, and it cannot tell useful output from a reasoning stream that has degenerated into whitespace until the provider's length cap ends the turn. Two production incidents in one week:
glm-5.3-flashmodel produced whitespace-only reasoning deltas until the length cap, on five runs in a row, each paid for in full.Why warren cannot see either:
src/runs/stream/bridge.ts:206-209continues onisPerDeltaNoiseEvent, and:454listsmessage_update,tool_execution_update, andmessage_startas that set. The rationale at:418-424is sound (persisting them flooded the event table), but no inspection happens before the drop.src/runs/watchdog.ts:172-178,repos.events.listTail(run.id, 1)), with a 45-minute default (:97). A turn that streams deltas for 44 minutes looks alive to it.turn_end(bridge.ts:238-242intosrc/runs/usage-aggregate.ts:130-134), so cost accrued inside a turn is invisible until the turn ends.src/runtime/k8s/agent-stdin-hold.ts:196-200, default atagent-entrypoint-env.ts:104), re-armed on any child output, so a streaming spiral keeps it alive too.stopReason === "length".src/core/event-envelope.ts:110already extractsstopReasonfrom the nested message; no consumer acts on it.The signals are all present on the wire. Every delta carries
partial.usage.cost.totaland its text, and whitespace-only thinking blocks pass the pi parser (src/runtime/adapters/parsers/pi.ts:153-157drops only empty text).What to build
A
TurnMonitorin a newsrc/runs/stream/turn-monitor.ts, fed from the bridge at the drop site (before thecontinueatbridge.ts:206), that tracks three things per turn:turn.stalledevent and cancel.thinkingortextdeltas whose content is whitespace-only. Past a threshold (by count or by bytes), emitturn.degenerateand cancel.partial.usage.cost.total, so the spend cap can trip mid-turn rather than atturn_end.On a trip, follow the shape
enforceBudgetCapuses insrc/runs/stream/budget.ts:50-75: evaluate, persist what you know, emit the event, cancel the run through the samecancelBurrowRunseam, and break the bridge loop with a terminal outcome. The monitor must never persist deltas;src/runs/stream/bridge.test.ts:107pins that they stay unpersisted and unpublished.It works on both runtimes because the K8s provider feeds the same bridge through
src/runs/stream/provider-source.ts. Keep the new file under 500 lines (check:size) and the bridge loop under the complexity ceiling; a separate module with an injected clock is the intended shape.Scope
Out of scope: making a stalled or degenerate turn eligible for the automatic provider retry in
src/runs/retry/provider-retry.ts. That needs a decision about which failure reason it carries and is a follow-up. If you add a new failure reason for the terminal outcome, it goes inRUN_FAILURE_REASONSatsrc/core/wire.ts:201and nowhere else;check:wire-typesenforces that, andgen:docsandgen:openapimust be re-run.Tests: a new
bridge.stall.test.tsbesidebridge.oom.test.ts, driven withStreamEventViewfixtures fromsrc/runs/stream/test-helpers.ts(piTurnEndat:100is the turn-end builder; add a delta builder) and a fake clock.budget.test.tsis the template for the cancel-and-break assertions. The existingbridge.test.tsandwatchdog.test.tsshould pass unchanged.Getting started
AGENTS.mdcovers setup and conventions. The bridge suites run in-process against fixture streams with no provider credentials. Runbun run check:allbefore pushing. Warnings count as failures.Tracked internally as
warren-74a7(whitespace degeneration) andwarren-1f85(mid-turn stall). Their third original bullet, salvage of atimed_outrun's work, already ships: the watchdog's force-fail reap runs salvage-before-destroy.