Repository navigation
Conversation
Current Codex releases invoke shell commands as exec, which neither the documented matcher ^(Bash|apply_patch|Edit|Write)$ nor the Bash-only adapter schema ever gated — every exec call passed through unevaluated. Add an exec schema accepting command (string or argv array) or cmd, refuse exec payloads with no command text instead of silently passing them through, and document the exec matcher.
path.resolve forces the host path style, so on Windows a POSIX cwd like /workspaces/probity gained a drive prefix (C:/workspaces/...) and a Windows file_path under a POSIX cwd was concatenated into a nonexistent path. Resolve in the style of whichever path is absolute and let a relative path inherit cwd's style. Also detect symlinks with lstat before opening: O_NOFOLLOW is undefined on Windows, leaving the symlink guard silently absent in safeReadCapped and readJsonl. Normalize the Copilot session path and the test fixtures' expected paths to POSIX form to match the Action.path contract.
Live verification showed Codex reports exec calls to the hook as tool_name Bash and that the probity.cmd shim in commandWindows fails to spawn on Windows, so the hook silently never ran. Document the direct node invocation and broaden the matcher wording.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and behavior
Devin CLI exposes lifecycle hooks but Probity has no Devin vendor. Native
exec/write/editpayloads cannot safely be treated as Claude/Codex payloads, and the observed Devin hook payloads do not include a native transcript path.This adds
--agent devin, checks native commands and file changes before execution, and supplies real prompt/tool-result history to existing rules through a per-project, per-session lifecycle journal. AI-backed rules require an explicitConfig.aivalidator; there is no speculative Devin SDK, implicit cross-provider authentication, or recursive Devin launch.Closes #90. @nizos, this is the adapter discussed there; feedback on the event-journal approach is welcome.
Implementation
.probity/devin/<session_id>.jsonl; preserve actual output/errors and background output events. PreToolUse attempts are not recorded as successful execution.Validation
15 new tests pass; lint, formatting of tracked changes, typecheck and build pass.
Windows native CLI
3000.11.1 (cc4e349ca55e)and Desktop-bundled CLI3000.10.48 (fcf7ba39), shipped with Desktop product version3.10.48: real hook probes block a marked command and marked file write, allow clean controls, and confirm forbidden files do not exist.Both runtimes: native edit before any test run is blocked;
python -m unittestproduces the actualAssertionError: 0 != 30; the identical edit after red is allowed; the test becomes green. These probes used an explicitly configured Codex validator.Full Windows suite: 31 failures, 533 passes, 26 skips before the last added symlink regression test. An isolated upstream-base run has the identical 31 failing test labels (519 passes, 26 skips), primarily POSIX path/symlink assumptions. New Devin tests pass; the existing failures are not represented as a green full Windows suite.
Staged/history Gitleaks and contribution-range TruffleHog scans pass.
Fork Linux/Node 22 CI passes: 63 test files, 565 tests passed, 26 skipped; lint, format and typecheck also pass. Run: https://github.com/letya999/probity/actions/runs/38053606602 (combined adapter and separate dependency patch).
Fresh global installation, without project hooks: actual red-to-green flow passes. A ModuleNotFoundError does not authorize the production edit; the edit is blocked and the file remains unchanged. Existing global hook entries are preserved.
Limits
Desktop's bundled runtime was exercised headlessly; the Desktop GUI itself and cloud handoff were not driven. Host hook timeouts can prevent enforcement. AI judgments are not a proof of test quality or stability. The event journal is local private session data, not an independent security boundary against code already running as the same OS user.
Follow-up: recording hooks and overhead
Commit 2d4e37c fixes a reproduced issue: PostToolUse/UserPromptSubmit were evaluated as empty commands, so an authorization rule could block or invoke AI after an action had already completed. They now return no actions while preserving history. The regression test failed before the fix and passes after it; all 16 Devin tests pass locally. The setup guide limits PreToolUse to supported native command/file tools and retains all PostToolUse results.
Updated combined-fork CI passes: https://github.com/letya999/probity/actions/runs/38060704717 . Local build, typecheck, lint and tracked-file formatting pass; Gitleaks and contribution-range TruffleHog find no secrets.
Installed the follow-up build globally and rechecked a real native Devin session: a harmless git commit --no-verify --dry-run attempt is blocked before execution; unittest and a read-only file read work. Combined fork CI: 566 passed, 26 skipped. Personal MCP configuration changes are intentionally outside this contribution.