Repository navigation
Machine events: a roster larger than the stall grace allows can still stop terminal output #2680
Description
Follow-up to #2672 (machine events on the terminal WebSocket).
A machine event sent onto an empty socket queue gives the connection a stall grace (sideTrafficUntil in server/core/connections.ts) so terminal output queued behind it is paused rather than stopped while the event crosses the link. The grace is capped at OUTPUT_JOIN_MAX_MS (15 s). At the budgeted slowest link (OUTPUT_JOIN_BYTES_PER_SECOND, 32 KiB/s) a roster larger than about 416 KiB needs longer than that, so the output window can return stop and the client is closed with 4008 while the roster is still crossing. A roster carries every PC's SessionSnapshot, so large ones are possible with many PCs or panes. Before #2672 the roster travelled on its own SSE connection.
Reproduced at the model level (real send, machineEventsAhead and OutputWindow, a 614,470-byte roster): verdict stop at 15 s with about 3.7 s of crossing left. Not observed on a real slow network.
Possible fixes:
- add the crossing time beyond the cap, as
OutputWindow.joindoes for a large frame (+ crossingMs(large)), or - end the subscription (
machine-events-ended) for any machine event larger than aboutOUTPUT_LOW_BYTES, so large rosters always go over SSE.
Related, unreproduced: machine-event frames on the WebSocket are not charged to the shared SSE event-stream budget, and a client.send() result of 0 for a machine event closes the socket instead of calling overflow().
Activity
devswha commentedon Nov 3, 2026
Also from omo's review of #2672 at 5558204 (reproduced at model level, MEDIUM): a grace is granted one at a time, a stall's length (2 s) apart, so a small machine event shortly before a large roster leaves the roster without a grace of its own. Repro with the real
send,machineEventsAheadandOutputWindowand simulated buffering: a small event at t=0, a 400 KiB roster at t=1 s, then 256 KiB of terminal output; the client is stopped 2.6 s later while the roster needs 12.5 s at 32 KiB/s.The spacing exists because renewing the grace on every event let a client that never acknowledges hold the shared pty paused indefinitely (fixed in #2672). A fresh grant for every frame sent onto an empty queue would bring that back for a frozen tab whose kernel buffer keeps draining the queue. So the fix needs a bound that covers both, for example a grant sized by the bytes of each machine frame within a total budget per stall, or ending the subscription (
machine-events-ended) for a large frame that cannot get a grace.fix(events): cap machine events on the terminal socket at OUTPUT_LOW_…
devswha commentedon Nov 4, 2026
Resolved in #2672 at d495405 by the maintainer's decision: machine events on the terminal socket are capped at OUTPUT_LOW_BYTES and larger ones move the page to SSE, so no stall grace remains. Closes once #2672 merges.
devswha commentedon Nov 4, 2026
Checked on current
origin/main(7a923ab): this no longer reproduces. #2672 merged as ffde1de with the maintainer's chosen fix, so there is no stall grace left to outgrow.sideTrafficUntilis gone fromserver/andshared/(git grepfinds nothing).send()inserver/core/connections.tscallsoverflow()instead of sending when a machine event would put more thanOUTPUT_LOW_BYTES(64 KiB) on the socket, counting what is already queued.setMachineEventsthen ends the subscription withmachine-events-ended, and the page takes machine events over SSE. A roster of any size therefore never sits in front of terminal output, and it never closes the socket with 4008.client.send()result of 0 for a machine event closes the socket" is fixed too:if (result === 0) { if (overflow) overflow(); else client.close(...) }.Evidence: in a clean worktree at 7a923ab,
bun test server/core/connections.test.ts server/output-window.test.tsgives 30 pass, 0 fail. That includes "ends the machine-event subscription, never the terminal connection, rather than put more than OUTPUT_LOW_BYTES of machine events on the socket". That test sends a 900 KiB roster on an empty queue and gets onlymachine-events-ended, no close. 256 KiB of terminal output then goes out with nothing queued ahead of it. It also covers a live event that would pass the cap, and "when the socket refuses a machine event".No PR needed. This can close.
devswha commentedon Nov 4, 2026
Closing: already fixed on main by #2672 (see the evidence comment above).