Rewrite grok delegation mechanics around verified behavior - #7
Closed
kelchm wants to merge 1 commit into
Closed
Conversation
The previous guidance (draft PR #6) was built on two claims that don't hold up under test, both traced to experiments run inside /tmp: - "--sandbox does not prevent edits to the working directory" — false. Grok's read-only profile leaves /tmp, /var/tmp, /var/folders and ~/.grok writable by design, so a repo under any of those is unprotected while the sandbox still reports "enforced":true. Outside those paths it blocks the edit tool, shell writes, and new-file creation at the OS level. - "-w gives effective read-only protection for the real checkout" — false, and unsafe. In headless mode (-p / --prompt-file) the flag is silently ignored: no worktree is created, edits land in the real checkout, and --always-approve auto-approves everything. Verified twice, with no stderr warning. Also corrected: there is no ~100 KB input ceiling (151 KB and 204 KB prompts were read in full), and grok does ship a bundled /review skill that works headless — the old skill claimed it had none and hand-rolled a replacement. Changes: - CLAUDE.md: replace the grok Mechanics block with review (/review), custom stance, and implement recipes, plus the verified traps. Notably that grok's --sandbox read-only is NOT codex's -s read-only, and that --tools / --disallowed-tools fail OPEN on an unrecognized name while --deny fails closed. - grok-review: delegate to the bundled /review skill instead of reinventing it; drop the false "physically can't hit the network" claim (untrue on macOS, platform-dependent generally). - grok-implementation: new, mirroring codex-implementation — isolate with plain `git worktree add` + --cwd rather than grok's -w. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #6, whose guidance was built on two claims that don't survive testing.
Why #6 was wrong
Both of its central claims trace back to experiments run inside
/tmp:--sandboxdoesn't prevent edits to the working dir-wworktree gives "effective read-only protection"read-onlyprofile leaves/tmp,/var/tmp,/var/foldersand~/.grokwritable by design, so a repo under any of those is unprotected whilesandbox-events.jsonlstill logs"enforced":true. The event log shows the original experiments ran under/private/tmp/.../scratchpad/. Re-run outside/tmp, grok tried the edit tool,sed -i,echo >>, and creating a new file — all four returnedOperation not permitted.-wclaim. In headless mode (-pand--prompt-file) the flag is silently ignored: no worktree is created,pwdis the real checkout, edits land there, and nothing is printed to stderr. Combined with--always-approve, Update Grok review mechanics #6's recipe was strictly more dangerous than the one it replaced.What else this corrects
/reviewskill that works headless, with--local/--branch/--prmodes — the closest analogue tocodex review. The old skill asserted grok had none and hand-rolled a replacement.--tools/--disallowed-toolsfail OPEN. One unrecognized name anywhere in the list silently restores the entire toolset (exit 0, no warning). The vendor README's tool tables are wrong: it documentsbashandrun_terminal_cmd; the real tool isrun_terminal_command.--denyfails closed with a hard error, so the guard is built on deny rules.grok --sandbox read-onlyis notcodex -s read-only. Verified: codex blocks writes in both$HOMEand/tmp; grok blocks only outside its writable exceptions.grok-reviewwas false —curlreturned 200 under the read-only sandbox on macOS (seccomp is Linux-only).Changes
CLAUDE.md— new grok Mechanics block:/review, custom-stance, and implement recipes, plus the verified traps.grok-review— delegates to the bundled/reviewinstead of reinventing it; keeps a custom-stance path since/reviewtakes no prompt; repo-scoped deny rules (a blanketWrite(**)breaks/review's notes file).grok-implementation— new, mirroringcodex-implementation. Closes a coverage gap: grok-4.5 is the designated default implementer but had no skill, while the fallback (codex) did.Verification
Guard profiles were run against an adversarial canary prompt (modify a tracked file, create a new file, append via shell, write outside the repo, delegate to a subagent), with controls to prove a breach is detectable:
/tmpenforced=true)--toolswith one bad name--always-approveonlyFindings were independently reviewed by gpt-5.6-sol, which corrected two overreaches now reflected above (plan mode does not cancel all tool calls; the prompt-offload mechanism is inferred, not established).
Caveats
/review, which is now the centerpiece.--deny) carries the guarantee and the OS sandbox is defense-in-depth only.🤖 Generated with Claude Code