Umbrella: internal container integration kernel support (review split PRs)

[deregtd]

Umbrella — review the focused replacements

The user-facing goal is: connect an optional Linux container when an agent needs to run tests/builds, materialize its Git repositories and uploaded files, keep workspace edits synchronized with the DO, and survive resets/approval pauses without silently repeating effects.

That goal originally accumulated in this draft. Review and merge the focused replacements below instead. Do not merge #729 alongside them. This branch remains unchanged as a historical reference at e6eb6431.

User/operator outcome	PR	Review base
Fetch an authorized Git base commit without giving the container credentials	#769	main
Work with uploaded files at normal workspace paths, including binary files	#770	main
Offer an optional integration's Connect button when the agent needs it	#771	main
Let the agent correct an invalid file edit rather than abort its turn	#772	main
Recover turn lifecycle and explicitly deduplicated actions after DO resets	#773	main
Continuously synchronize container/DO worktree edits with durable OT receipts	#774	temporary aggregate prerequisite base
Tell whether an apparently stalled turn is waiting on inference or saving edits	#776	main

Organization and merge order

The six main-targeted PRs are independently reviewable/mergeable. #772 now contains validation only: replica serialization and the shared permission helper belong to #774; generic timing instrumentation is isolated in #776.

#774 targets ddr/split-replica-prerequisites only to hide prerequisite diffs. Do not merge it into that temporary branch as a release step. Reconcile the final prerequisite heads and rebase/retarget onto main after they land. #771's optional connection UI is not an architectural requirement of replication.

That review base pins #770's original text-only attachment contribution, while #770 has since expanded to all uploaded formats and includes its own whole-file binary foundation. Restacking must integrate those newer changes; the current assembled tree is not claimed to contain every latest sibling head.

The organization change preserves #774's production source/configuration exactly relative to its prior head 82186a4d. It relocates the timing test without dropping or duplicating coverage.

Release boundary

The Linux image, Rust bridge, FUSE/native engines, snapshots, and network policy remain in gadgets-internal. The separate #759 binary-replica follow-up and original #729 branch are unchanged. Internal MRs must eventually pin and verify against the merged public revision.

This reorganization does not deploy or enable any integration. Verification and remaining integration notes are maintained on the focused PRs.

---

Original description — historical, not the current merge plan

Summary

Generic kernel support for the internal Linux Container gatekeeper. The Linux image, Rust bridge,
container bindings, network policy, snapshots, and deployment grants stay in gadgets-internal.

• Trusted integration grants, scoped agent/action contexts, durable lifecycle outboxes and
  recoverable contextual actions with original approval authority.
• Revocable replicas: immutable bases, binary streaming, canonical OT catch-up and anchored,
  atomically receipted submissions.
• Repository/checkout provenance and chat attachment worktrees with stable materialization mounts.
• Exact-commit native Git v2 bootstrap through both GitHub and GitLab, sharing gatekeeper-kit
  validation. OAuth and self-hosted GitLab Access credentials stay in the provider.
• Generic owner-opt-in ambient connection card and normal DO-reset recovery discipline.

Current draft status

Rebased and ported onto main at 8cbfb2b6. Storage declarations now live in the extracted
Overseer schema, with version 5 → 6 migration; repository identities are populated lazily.
Preserves main's resource-creation approval flow, blueprint-merge buffering, model retries and
turn notifications. Replica bootstrap now also replays compacted chat overlays.

Keep this a draft. The internal gatekeeper MR must pin the eventual merged public revision and
be verified against it before release. No live deployment or feature enablement was performed
by this rebase; the GitLab provider tests use authenticated transport fixtures, not a live project.
Rev 0 mutations remain text-only, with no exactly-once claim for unjournalled shell effects.

Verification after rebase

• Full backend workerd unit suite: 1,697 passed / 78 files.
• Backend integration suite: 3 passed.
• GitLab: 257 passed (104 Node, 153 workerd).
• GitHub: 151 passed (40 Node, 111 workerd).
• gatekeeper-kit: 786 passed (674 Node, 112 workerd), including native Git against a real
  loopback Git server and bounded exact-commit validation.
• Google workerd fixtures: 538 passed (342 general, 196 Docs).
• Chat approval/ambient connection UI: 23 passed (Node native Web Storage disabled for jsdom).
• Full repository build/type checks, lint and generated Worker config checks passed.
• Real eviction, lost-ACK recovery, bidirectional collision, cross-Worker binary transfer,
  authority revocation, v5 migration and compacted-overlay bootstrap regressions covered.

Review the public API, kernel runtime/storage, verification, and small generic UI commits
separately. Follow-up port and provider commits keep the GitLab adapter apart from kernel storage.

[github-actions]

Preview: pr729-ddr-worktree-77f23069

https://pr729-ddr-worktree-77f23069-router.cloudflare-os-previews.workers.dev

Dashboard · deleted when this PR closes

[ask-bonk]

Review: 8 findings.

Posted all 8 actionable findings inline in a single review. Verified key recovery and capability failures with focused Node reproductions; full repository tests were not run.

github run

[ask-bonk]

Review: 4 findings.

Posted 4 new actionable findings inline in one review: 2 P1 and 2 P2. Existing published findings were not repeated.

Verification was based on code inspection; the test suite was not run.

github run

[ask-bonk]

Review: 3 findings.

Posted 3 new actionable findings inline in one review: 2 P1 and 1 P2. Existing published findings were not repeated.

Verified with focused, source-based Node reproductions. Full repository tests were not run.

github run

[deregtd]

Addressed the newer eval review's request for bounded stall diagnostics in d8d15f5. Debug events now distinguish queued/started/completed/failed persistence barriers from model dispatch, stream creation, and completion; they share an execution ID and carry sequence/timing fields, never prompts, command/file contents, credentials, or provider error text. Stream creation is explicitly not treated as receipt of HTTP response headers.

A deterministic regression blocks the actual persistence barrier, proves that the next model request has not been dispatched, releases it, and verifies the event ordering and absence of sensitive content. Full lint/type checks and the complete backend suite (1,727 unit tests, plus native integration tests) pass; the final focused review suite passes all 41 tests.

This instruments the candidate-only calendar timeout rather than claiming a root cause or adding a speculative runtime fix. A fresh eval must still use these breadcrumbs to attribute the original stall; the unrelated model-output failures and optional prompt changes were left untouched.

[ask-bonk]

Review: 3 findings.

Posted 3 new actionable findings inline in one review: 1 P1 and 2 P2.

Verified with focused, source-based Node reproductions. Full repository tests were not run.

github run

[ask-bonk]

Review: 4 findings.

Posted 3 new findings inline in one review: 1 P1 and 2 P2.

Additional finding — [P2] Close paused integration turns when deleting a chat
packages/workshop-backend/src/overseer.ts:12670

After suspension teardown finishes, the chat has no live context. deleteChat removes its metadata and active-agent record but leaves the new paused continuation, integration journal, and registered participant. destroyLiveChat then returns immediately, so no terminal lifecycle event is queued. A participant with an empty outbox has no retry deadline, leaving integration-owned resources without eventual cleanup.

Journal cancellation before deleting the chat’s workpieces, clear its paused continuation, and retain terminal outbox delivery until acknowledged.

Verification used source inspection and focused, source-based Node reproductions; full test suites were not run.

github run

[ask-bonk]

Review: 3 findings.

Posted 3 new actionable findings inline in one review: 1 P1 and 2 P2.

Verified with focused, source-based Node reproductions. Full repository tests were not run.

github run

[ask-bonk]

Review: 2 findings.

Posted 2 new actionable findings inline in one review: 1 P1 and 1 P2.

Verified with focused, source-based Node reproductions and Workers RPC documentation. Full repository tests were not run.

github run

[ask-bonk]

Review: 3 findings.

Posted 3 new actionable findings inline in one review: 2 P1 and 1 P2. Existing findings were not repeated.

Verified with source inspection and focused Node reproductions. Full test suites were not run; this checkout lacks installed dependencies and pnpm.

github run

[ask-bonk]

[P2] Reconcile canonical appends when handing off between agent passes

The registered access survives until the outer Overseer turn tears down, but runAgent() can reload, compact, or retry several passes inside that turn. After the next pass snapshots loadChatHistory(), its asynchronous binding/pin/blob replay still leaves submissions routed to the preceding pass's access. A replica can commit and receive an ACK in that window, then this registration replaces its updated live view with a replay that omitted the acknowledged transaction. A focused execution of the production pass replay and flush closures reproduced durable a.txt = acknowledged replica text while the newly registered access read a.txt = first. Subsequent correctly anchored replica edits can fail against that stale view, and ordinary edits can be calculated from different text than the canonical file. Fence/drain the old pass and make the handoff consistent with canonical appends, or revalidate and replay the intervening journal tail before publishing the new access. Cover a submission during compaction/retry replay.

[ask-bonk]

[P2] Recheck ordinary first-write pins after concurrent replica appends

These appends can now establish a gadget pin while an ordinary writeFile or editFile is awaiting its first-write readCommitFiles()/anchorKnowledgeToPin(). The ordinary tool keeps its previously prepared pin; when it resumes, appendAgentEdit(..., pin) replaces the entire live gadget map with that old base, discarding the replica's already-ACKed changes from the session. The durable barrier behaves differently: an existing matching pin does not re-root its canonical content, so the live and stored views diverge. I reproduced this with the production writeFile and replica-flush closures: a replica wrote a.txt while the first ordinary write to b.txt awaited its base, and afterward storage retained the replica text while live readFile(a.txt) returned the original base text. Recheck pin ownership after the ordinary tools' awaits (and calculate edits against the resulting current view), or serialize pin establishment, so an already-pinned session cannot be reset by an older snapshot.

[ask-bonk]

Review: 2 findings.

Posted 2 new P2 findings inline in a single review.

Verified both with focused, source-based Node reproductions using production replay, file-tool, and replica-flush code. Full test suites were not run.

github run

[github-actions]

Eval results

Verdict: ⚪ Unchanged. No task moved beyond what 10 runs can tell apart from noise.

Task	Score	Δ score	Fisher test	Cache hits	Avg min	Avg steps
change-calendar	90% → 80%	−10 pp	p = 1.00	95% 0 pp	3.8 → 4.1	25.7 → 26.4
chess	100%	0 pp	p = 1.00	98% 0 pp	9.3 → 9.7	60.2 → 59.3
incident-desk	100%	0 pp	p = 1.00	96% → 97% +1 pp	4.0 → 4.9	29.0 → 39.3
worker-logs	100% → 70%	−30 pp	p = 0.21	95% 0 pp	4.4 → 3.6	26.6 → 20.4

Failed checks

Task	Check	Failed
change-calendar	t3 existing-windows-survive-the-rule-change	0/10 → 1/10
change-calendar	t3 same-service-windows-must-be-a-day-apart	0/10 → 1/10
change-calendar	t3 billing-is-capped-at-four-hours-others-are-not	0/10 → 1/10
change-calendar	t4 earliest-available-applies-every-rule	1/10 → 0/9
change-calendar	t4 earliest-available-matches-the-reference	1/10 → 0/9
change-calendar	t4 earliest-available-slots-are-bookable	1/10 → 0/9
change-calendar	t4 names-the-booked-windows-the-new-rules-reject	0/10 → 1/9
worker-logs	t1 hourly-buckets-cover-every-hour-including-empty-ones	0/10 → 3/10
worker-logs	t1 ranges-are-half-open-and-filter-by-worker	0/10 → 3/10
worker-logs	t1 ingests-resets-and-summarises-per-worker	0/10 → 2/10

Run · trajectories and raw results

[ask-bonk]

🔬 Eval runs review

Performance

Change-calendar passed 9/10 → 8/10 and worker-logs 10/10 → 7/10, while chess and incident-desk stayed at 10/10; comparison.json finds no pass-rate or prompt-cache hit/break movement beyond noise. Incident-desk increased from 29 → 39.3 steps and 4.0 → 4.9 minutes, with mean run cost $0.0225 → $0.0288; its longest candidate run spent substantial work rewriting SQL helpers and replacing subscriptions with polling. Worker-logs averaged $0.0234 → $0.0178 and 4.4 → 3.6 minutes, but three first-turn failures ending after only 5–9 steps explain the apparent saving; calendar/chess costs remained within their run-to-run spread, and exact-text edit retries dominated wasted calls on both sides.

⚪ VERDICT: NO REGRESSION FROM THIS PR

The comparison is unchanged, the failed trajectories show generated-code and arithmetic mistakes rather than consequences of the lifecycle/replica or attachment changes, and no attributable performance improvement or deterioration exceeds noise.

Triage

Failure modes

• Ingestion crashes · worker-logs 2/10 · model error · this PR: no — In turn 1, trial 3’s writeFile(server.js) batched 100 six-column rows into a statement with 600 bindings, causing “too many SQL variables”; trial 5’s subsequent editFile inserted an overescaped timestamp regex and dereferenced its null match. Trial 5 had tested valid ingestion before that edit, but afterwards tested only empty and invalid inputs.
• Hourly totals lose timestamps · worker-logs 1/10 · model error · this PR: no — Trial 6, turn 1’s writeFile(server.js) made readEvents() select only worker, status and duration, while hourly() required event.ts_ms; every bucket therefore stayed zero despite successful ingestion and summaries.
• Calendar stops loading · change-calendar 1/10 · model error · this PR: no — Trial 1, turn 3’s editFile(client.js) added TOO_CLOSE without a trailing comma before UNKNOWN_SERVICE. The agent reported completion without exercising the updated gadget, and verification could not start it.
• Incorrect rejection audit · change-calendar 1/10 · model error · this PR: no — Trial 3, turn 4’s executeCode returned the correct bookings, but the final reply incorrectly added mw-101 TOO_CLOSE and mw-103 TOO_CLOSE: their separation is 45 hours, not under 24. Main’s failed calendar run computed the correct audit using executeCode; its failure instead came from a search implementation skipping same-day 22:00.

Tool errors

• editFile: No matching text was found in the file · change-calendar 3 → 0, chess 13 → 12, incident-desk 1 → 1, worker-logs 2 → 3 · model error — Agents supplied mismatched whitespace, quotes or escaping. Main chess trial 7 repeatedly retried incorrect text; candidate chess trial 5 searched for overescaped PGN expressions before correcting them.
• editFile: Validation failed · change-calendar 1 → 3, chess 2 → 1, incident-desk 2 → 6, worker-logs 0 → 5 · model error — Calls omitted required filename or misspelled replacement; candidate incident-desk trial 2, turn 1 shows both, followed by successful corrected calls. The required schema is unchanged.
• editFile: Multiple matches were found; text must be unique · chess 1 → 4, incident-desk 6 → 1, worker-logs 1 → 0 · model error — Agents attempted replacements using repeated snippets despite the explicit uniqueness requirement. Candidate chess trial 4, turn 1 retried generic helper references, then rewrote the file.
• executeCode: Failed to start Worker · worker-logs 1 → 0 · model error — Main trial 5, turn 1 submitted . as the entire script, received “Unexpected token '.'”, then supplied valid JavaScript and continued.
• grep: Invalid regular expression, Unterminated group · incident-desk 0 → 1 · model error — Candidate trial 3, turn 2 passed an unescaped opening parenthesis in if (incident.status...; it recovered by using readFile.

What to do

• Optional: in packages/workshop-backend/src/agent.ts, add system-prompt guidance to exercise affected RPCs with representative valid and batch inputs after the final code edit, and calculate data-audit answers with executeCode rather than mental arithmetic; these runs indicate no required PR-specific fix.

github run

[deregtd]

Re-reviewed e6eb643123e1bc1d976ea1eb2e90b9b946fe6e2a against 7b0ac75ebcfa53292d56245f288a27f11831aa6d, comparing all 62 real file patches with the completed prior review and checking the newer base interactions. The accepted-directory writability and single-event paging fixes are verified, including three independent native probes.

The two new findings still need fixes: pass handoff leaves the previous access registered while the next pass replays a history snapshot, and first-write pinning reuses a pin prepared before an await even if a replica has since pinned the gadget. I independently traced both production paths; each can replace the live view after a replica ACK. These are source-verified assessments, not additional independent race reproductions.

Backend/browser type checks and 235 focused native tests pass, including the three fix probes. The local run reused installed dependencies and explicitly selected workerd 1.20260926.1, which matches the pinned runtime's SQLite row budget; the initial default-runtime size failures disappear with that selection. No live container/provider end-to-end validation was run. The PR remains draft pending the two races and companion integration validation.