Allow executeCode to emit bounded image output

[deregtd]

Summary

• add a host-owned fourth executeCode argument with output.image(bytes, mimeType), where bytes is a Uint8Array
• keep image data binary through native Workers RPC and convert to base64 only when assembling the model-provider message
• return emitted PNG, JPEG, and WebP images as native model content alongside console text
• validate MIME signatures and enforce a shared four-image / 6 MiB limit across every executeCode call in one persisted agent step
• retain at most 12 MiB and 50 of the newest executeCode images in a provider request, leaving image slots for attachments and replacing older pixels with an explicit recapture marker
• scan persisted image results row by row, retaining raw buffers only for selected results so an image-heavy history cannot exhaust isolate memory during replay
• persist raw tool-result image bytes in the server-only model snapshot so compaction reloads and transient retries cannot lose them
• keep the sink write-only over RPC: gatekeepers can emit pixels they already received, while only the host can drain the buffered images

This is intentionally image output rather than arbitrary binary injection into model context. Other binary artifacts should continue to use purpose-built storage or download paths.

Testing

• pnpm exec vp run -F @gadgets/workshop-backend build
• pnpm exec vitest run __tests__/code-mode-output.test.ts __tests__/agent-retry.test.ts (13 tests)
• pnpm exec vp lint

[github-actions]

Preview: pr757-ddr-executeco-ac63286a

https://pr757-ddr-executeco-ac63286a-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[ask-bonk]

Review: 1 findings.

Submitted 1 actionable inline comment.

github run

[ask-bonk]

Review: 2 findings.

Submitted 2 actionable inline comments. The earlier image-replay finding is addressed by the durable snapshot and history-replay changes.

github run

[ask-bonk]

Review: 1 findings.

Submitted 1 actionable inline comment. The three previously reported findings are addressed in the current code.

github run

[ask-bonk]

Review: 2 findings.

Submitted 2 actionable inline comments. The four previously reported issues are addressed in the current code.

github run

[ask-bonk]

Review: 1 findings.

Submitted 1 actionable inline comment. The previously reported replay-memory and image-count issues are addressed in the current code.

github run

[ask-bonk]

Review: 2 findings.

Submitted 1 new actionable inline comment. The existing many-image dimension-limit finding remains unresolved in the current code.

github run

[ask-bonk]

Review: 1 findings.

Submitted 1 new actionable inline comment. The previously reported combined-media and many-image issues are addressed.

Verified the new finding with an in-memory PNG reproduction and provider documentation. Repository tests were not run because pnpm and dependencies are unavailable.

github run

[ask-bonk]

LGTM!

github run

[github-actions]

Eval results

Verdict: ⚪ Unchanged. No task moved beyond what 10 runs can tell apart from noise.

Task	Score	Δ score	Fisher test	Cache hits	Avg min	Avg steps
change-calendar	90% → 100%	+10 pp	p = 1.00	95% 0 pp	3.2 → 3.6	24.4 → 25.5
chess	70% → 90%	+20 pp	p = 0.58	98% 0 pp	7.5 → 9.1	52.5 → 56.7
incident-desk	100%	0 pp	p = 1.00	96% → 97% +1 pp	3.8 → 4.7	32.4 → 39.5
worker-logs	100% → 80%	−20 pp	p = 0.47	95% → 94% −1 pp	4.1 → 3.5	28.2 → 18.9

Failed checks

Task	Check	Failed
change-calendar	t4 names-the-booked-windows-the-new-rules-reject	1/10 → 0/10
chess	t1 agrees-with-the-oracle-on-the-hard-positions	1/10
chess	t1 agrees-with-the-oracle-on-perft-positions	1/10
chess	t1 agrees-with-the-oracle-through-random-games	1/10
chess	t2 imports-known-games-to-the-right-positions	2/9 → 0/9
chess	t2 refuses-invalid-pgn-without-changing-the-game	2/9 → 0/9
chess	t2 exports-pgn-the-oracle-replays-to-the-same-position	2/9 → 0/9
chess	t2 rules-still-agree-with-the-oracle	1/9 → 0/9
worker-logs	t1 ingests-resets-and-summarises-per-worker	0/10 → 2/10
worker-logs	t1 hourly-buckets-cover-every-hour-including-empty-ones	0/10 → 2/10
worker-logs	t1 ranges-are-half-open-and-filter-by-worker	0/10 → 2/10

Run · trajectories and raw results

[ask-bonk]

🔬 Eval runs review

Performance

Pass rates were change-calendar 90% → 100%, chess 70% → 90%, incident-desk 100% → 100%, and worker-logs 100% → 80%; comparison.json judges every difference within noise, as are the cache-hit and cache-break changes. Mean run costs were respectively $0.0202 → $0.0206, $0.0479 → $0.0542, $0.0237 → $0.0279, and $0.0233 → $0.0169, with differences smaller than the within-side run spread. Chess remained the largest cost/time sink at 52.5 → 56.7 steps and 7.5 → 9.1 minutes, with PGN escaping repairs and edit retries on both sides; worker-logs’ lower step count, 28.2 → 18.9, partly reflects two runs stopping after failed ingestion.

⚪ VERDICT: NO REGRESSION FROM THIS PR

No meaningful movement is attributable to the diff: the new output.image path is unexercised, while the failed runs stem from generated SQL/chess code rather than the image sink or replay changes.

Triage

Failure modes

• Oversized SQL batches · worker-logs 2/10 · model error · this PR: no — Turn 1, trials 9 and 10: writeFile(server.js) generated 100-row and 50-row inserts with six parameters per row, exceeding the runtime’s bind-variable limit. Trial 9’s executeCode smoke test used only three events; trial 10 never tested ingestion. The rejected first batch left storage empty, causing the subsequent summary/hourly failures; main’s row-by-row insertion avoided this.
• Missed long-range rook/bishop attacks · chess 1/10 · model error · this PR: no — Turn 1, trial 10: writeFile(server.js) incorrectly required distance === 1 for rook/bishop attacks. Its executeCode castling check used an adjacent rook, missing the defect before the final reply claimed full rules support. Main trial 8 contains the same mistake.

Tool errors

• editFile: No matching text was found in the file. · chess 12 → 6, incident-desk 5 → 0, worker-logs 3 → 2 · model error — Agents quoted incorrect escaping, invented text, or already-replaced snippets. Candidate chess turn 2 retried a PGN regex replacement that had already succeeded; worker-logs turn 3 invented an async modifier.
• editFile: Validation failed · change-calendar 1 → 1, chess 1 → 2, incident-desk 5 → 5, worker-logs 1 → 3 · model error — Calls omitted required filenames or supplied malformed arguments. Both calendar sides misspelled the replacement key; candidate incident-desk turn 2 repeatedly omitted filename, then recovered.
• editFile: Multiple matches were found. The text to match must be unique. · chess 4 → 5, incident-desk 3 → 3 · model error — Snippets matched several functions. Candidate chess trial 2 repeatedly targeted duplicated PGN blocks; incident-desk turn 3 recovered by including the enclosing function name.
• readFile: File does not exist. · change-calendar 0 → 2 · model error — Candidate trial 1, turn 1 read server.js and client.js immediately after creating an empty gadget, despite the prompt explaining that new gadgets have no files.
• executeCode: This step already made code changes · worker-logs 0 → 1 · tool design — Candidate trial 5, turn 3 combined a README edit and execution in one step. The description omits this restriction; the unchanged guard explained recovery, and execution succeeded in the next step.

What to do

• Optional: add packages/workshop-evals/evals/execute-code-images.eval.ts requiring image emission and a later-turn answer from the pixels; the current tasks do not measure this feature’s benefit.
• Optional: add parameter-safe SQL insertion guidance to SYSTEM_PROMPT in packages/workshop-backend/src/agent.ts, recommending row-by-row inserts and realistic ingestion smoke tests.
• Optional: document the same-step edit/execute restriction in EXECUTE_CODE_TOOL_DESCRIPTION in packages/workshop-backend/src/agent.ts.

github run

[kentonv]

What if instead of extending executeCode, we extend the readFile tool call to support reading images?

Then, in executeCode, if the agent receives an image, it can write it into a worktree (using existing APIs), and can read from there.

I also kind of think that when the user attaches files to a message, instead of feeding them straight into the LLM, we should drop them into a worktree. Then the user can upload files for purposes other than just having the agent look at them. E.g. you could upload assets for use in the app, etc.

[kentonv]

Oh you actually added chat attachment worktrees in #770!