Repository navigation
feat: add harness integration and lifecycle audit skill - #6509
Closed
nikhilachale wants to merge 74 commits into
Closed
nikhilachale wants to merge 74 commits into
nikhilachale wants to merge 74 commits into
Conversation
Contributor
🏆 Review leaderboardOct 3–10, 2026 · UTC
Ranked by distinct external PRs reviewed, then review rounds, then PR comments. Self-activity and bot activity are excluded. Show 6 more reviewers
|
This was referenced Oct 10, 2026
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Collaborator
Author
|
Updated the screenshot evidence requirement in bf6a30d.
Validation on the VPS: |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Checked against upstream/main 0e81e1d, where merged harnesses (Hermes, Memcode, CodeWhale) were reported as unwired: - object keys may be bare (`hermes: hermesLogo`, `memcode: "memcode"`) - auth plans come from documentationPlan/loginMenuPlan/terminalInputPlan - Chat drivers register as `p := pkg.New()` then `add(pkgacp.New(p, ...))` - logos may ship as -dark/-light variants - licence files and docs/harnesses pages are optional (source comments are accepted) The catalog's cross-harness switch now points at provider_switch_tui.go and POST /sessions/{id}/provider-switch, which replaced agent switching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ation-audit-skill
…failure The first live run failed at startup: a cold go build (toolchain and modules download) hit the 120 s readiness timeout, and the reason was 50 'go: downloading' lines. The build now has a 10-minute budget; a timeout says so in one line, other build failures show the real compiler lines. A run-level failure's rerun now repeats the agent selection instead of '--agent null --only startup'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codex debug models prints ~0.5 MB of JSON; the 64 KB command output tail cut it, so local_models_list read 'no models'. The model-list command now keeps up to 16 MB, and unparsable output says so instead of 'no models'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dels to leak them
The first live Codex run showed two audit-prompt flaws, not AO bugs:
- Codex wrote the proof to the AO session artifacts folder, as AO's
using-ao guidance asks for deliverables; the prompt now says the test
file belongs in the worktree (pwd), and a proof found in artifacts is
reported as exactly that instead of "prompt never delivered"
- Codex refused to "disclose private instruction values". Asking a model to
repeat a secret from its system prompt reads as prompt extraction. Project
rules (agentRules) and AGENTS.md now state ordinary conventions ("every
JSON file must include rulesMarker / agentsMarker"), and a rule added
after Kill (restoreRulesMarker, chatRestoreRulesMarker) must show up in
the restored session's next ordinary edit
activity_status no longer depends on the proof file: Codex settled
correctly but the gate failed with the proof.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t-run dialogs in the UI audit Live Codex applied the project rule so faithfully that it refused the cancellation witness command, whose JSON files lacked rulesMarker. The rule now covers only files whose name ends in PROOF.json. The UI audit was blocked by AO's first-run 'Import projects from your earlier AO?' dialog; it now dismisses such dialogs with a defer answer (Skip, Not now) only, never Proceed or Don't Migrate, and records it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he UI audit From the live Codex runs: - instruction_channel_restore checked a stale token: makeScenario copied the scenario, so the restore rule minted after Kill was never the value the extras suite compared. AO did regenerate system.md; scenarios are now one shared object. - Chat approvals wrote inside the worktree, which Codex's workspace-write sandbox (accept-edits) allows without asking. The command now writes outside the worktree (~/.ao/audit-approvals, or AO_AUDIT_APPROVAL_DIR) and the file is removed afterwards. - UI audit: read text with innerText (textContent glued "Account" and "Connected" together), accept "Log in again", dismiss defer-able first-run dialogs anywhere in the stack, route by hash instead of reloading, and start each run from a clean home screen. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g the session A user stops a running TUI agent in AO by archiving the session, not by keystrokes. By default the cancellation gate now starts the witnessed foreground command, archives (kills) the session while it runs, and requires the session to terminate, the command's process to be gone and its delayed file never to appear. That archive doubles as the Kill before restore. The keystroke interrupt over /mux stays available only for a contract that declares `interrupt`; the bundled Codex/Claude contract no longer does. A command that outlives the archived session is classified ao_bug and points at the session manager and process supervisor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live Codex answered correctly (worker / orchestrator) but saved AO_ROLE.txt in the session artifacts folder, as AO's using-ao guidance asks, so the gate saw nothing. The prompt now asks for the working directory, and the gate also reads <data>/artifacts/<session>/AO_ROLE.txt, saying where the answer was. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…into search
The Harness page search filters by display name ("Claude Code"), so typing
the harness ID "claude-code" matched nothing. The UI audit now clears the
search (or types --label) and selects the row by its data-agent ID.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r TUIs - Claude Chat failed with CHAT_DRIVER_UNAVAILABLE: a plain go build does not bundle AO's ACP bridge. The isolated daemon now gets AO_ACP_RUNTIME_DIR from the checkout's frontend/resources/acp-runtime, and host preflight says how to build it when it is missing. - A blocked restored ready cue now keeps the last 1.5 KB of screen text (ANSI-stripped, redacted) so the contract pattern can be fixed from the real frame. - UI audit: wait up to 90 s for a terminal (Claude Code paints later than Codex) and confirm the agent's READY reply when the terminal text is readable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live Codex restore frames failed every ready-cue pattern: Codex spaces words with cursor-forward moves (CSI n C), which ANSI stripping removed, gluing "Conversation interrupted" together, and its spinner window title (OSC 0) leaked after the composer. Terminal text now turns cursor-forward into spaces and drops OSC sequences before matching and in the saved screen tail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n, UI audit picks visible surfaces - Codex's accept-edits sandbox blocked the out-of-worktree write (EPERM) and the agent stopped, as the prompt told it to on denial. Codex only raises an approval when it asks for escalated permissions, so the prompt now asks for that and stops only when the approval itself is denied. - UI audit: AO keeps earlier sessions' terminals mounted but hidden; the surface checks now match visible terminals/chats only, and the default interface is set for every interface tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live Codex orchestrator answered 'orchestrator' but refused to write AO_ROLE.txt into its workspace, as its AO role rules require (non-PR deliverables stay outside the workspace). Both roles now save the answer in their AO session artifacts folder, which every role may use. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude's ACP bridge has no models list; it offers model (15 choices) and effort as Chat config options, as the Chat UI shows. chat_models and chat_settings now use config-options when the models list is empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the "new rule after restore" checks (system_prompt_restore, instruction_channel_restore, chat_system_prompt_restore): a restored session keeps the instructions it started with, which is expected. Move everything harness-specific into the harness contract so the skill works for any integrated harness, with Codex and Claude Code as examples: - capability support is required/optional/manual in the catalog, with per-harness overrides in the contract's `supports` and `surfaces` - adapter directories for failure diagnoses come from `adapters` - instruction channels must be declared; no Codex/Claude defaults - hooks are checked from the contract's `hooks` file, else via the hook-persisted native session id - run_local.sh audits and doctors every harness in the contract file - add assets/contract-template.json; rename the contract file to assets/local-contracts.json; SKILL.md documents the contract step Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Replaced by #7081, a smaller version of this skill. |
This was referenced Nov 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Code changes: +2519 -0
Tests: +1869 -0
Others: +1591 -0
Adds an AO contributor skill for assessing coding harnesses, implementing supported integration surfaces, and auditing native CLI/AO lifecycle behavior. The API-first runner preserves failed, blocked and not-run gates, including fresh instruction delivery, cancellation and exact restore.
Harness-specific audit reports, JSON receipts, screenshots and provenance must be published in comments and GitHub attachments on each harness's integration PR; new evidence files must not be committed. This PR contains reusable skill guidance, the runner, fixtures and CI wiring. Version-specific investigation narratives and the dated session index are excluded from its file diff; general integration and evidence requirements remain.
Historical index and per-harness evidence remain preserved at the pre-cleanup revision and through the respective integration PRs; existing evidence-file migration requires verified attachments before removal.
Operational guidance now includes a compact Prepare/Launch/Doctor/Drive/Evidence/Cleanup checklist and a journey map. It maps actions to proof prerequisites and observed results, adds read-only instance checks, identifies separate UI/Chat and independent-instruction coverage, and requires evidence to survive teardown.
The ACP/shared-Chat-UI guide maps the native binding, registry, typed contracts, daemon events and shared renderer. Its focused Chat checklist covers real UI creation, streaming, history/load, decisions, cancellation, reconnect, daemon recovery and exact restore. Chat Doctor uses controller/snapshot readiness rather than TUI composer cues; this adds guidance, not Chat automation to the runner.
A full integration now starts by reading AO/upstream docs and existing PRs, then planning each native capability and AO gap. Supported TUI, Chat and reviewer surfaces must be qualified independently. AO Account login/setup, both model lists, persisted model enable/disable preferences and applicable picker effects, and provider/account configuration with effective native routing are explicit requirements. Reviewer registration/picker, review execution and correlated result/cancel/rerun have separate evidence checkpoints. Unsupported native behavior needs proof; unfinished AO wiring keeps the full integration incomplete.
Integration handoff now requires a completion matrix covering product registration/UI, model lists/visibility, provider management, setup, OS/config readiness, branding, TUI/Chat, reviewer, permissions/instructions, lifecycle, evidence and current CI/merge state. Each row records supported scope, verification status, and proof or an omission reason. Current-status receipts separate runtime-tested builds from the current CI head, preserve sanitized reproduction inputs and distinct attempts, and reconcile stale claims/counts before handoff. These are contributor review gates; the runner schema and automation are unchanged.
The skill now requires a per-harness, per-attempt screenshot index for every meaningful UI action and result: Agent Harness listing, install offer/progress/result, login/configuration or stored-credential state, Agent Harness models, New Task agent/model selection, running/output, follow-up and active-command cancellation, Kill/archive confirmation, archived entry, exact restore, and the same card back in Kanban with a fresh continuation. Supported Chat also requires real shared-UI streaming, decisions, cancellation, reconnect and recovery evidence. Each checkpoint links actual images and functional receipts with build, identity and capture provenance; failures retain original screenshots or explicit capture-unavailable reasons plus sanitized logs. Images are embedded in journey order with durable, verified links on the harness integration PR.
Already installed/connected does not prove a fresh install/login. The accepted saved-credential route can qualify after fresh verified authorization and a successful provider turn; its unexercised fresh-login alternative remains explicitly labelled without blocking that route. Manual-only plans retain an explicit unexercised in-page step and can qualify only agreed manual scope. A planned/absent Agent Harness model panel remains
NOT_RUN / NOT_IMPLEMENTED; New Task models cannot substitute. Real credentials must not be deleted to stage captures. Later reproductions cannot replace originals. Screenshots prove visible states, while native identity, provider authorization, cancellation and restored continuation require independent functional proof. These are contributor gates; the API-first runner still captures failures only and does not automate successful-step screenshots. Detailed evidence belongs in PR comments/attachments; the description links to them. Existing committed evidence is removed only after its originals and published destinations are verified; source logos/licenses and synthetic fixtures remain source files. Isolated candidate registration enables real Chat qualification before production registration.Validation: whitespace and 53 local documentation links/anchors passed, along with independent baseline/forward capability exercises and focused review. This update changes four Markdown files; runner/tests/workflow are unchanged. Current-head CI is pending. Detailed verification and prior failure history are in the PR comment. No live harness/UI qualification is claimed.
Historical run details and failures remain in the linked integration reports and earlier discussion. This contributor tooling does not add a desktop scheduler, automatic login or automatic PR publisher.
How it works
Repository docs point contributors to the skill. The historical index above links to each agent's detailed evidence outside this PR's file diff. Tests validate the audit tools; a live audit checks the actual agent.