Skip to content

feat: add harness integration and lifecycle audit skill - #6509

Closed
nikhilachale wants to merge 74 commits into
OrchestratorInc:mainfrom
nikhilachale:feat/harness-integration-audit-skill
Closed

nikhilachale wants to merge 74 commits into
OrchestratorInc:mainfrom
nikhilachale:feat/harness-integration-audit-skill

Conversation

@nikhilachale

@nikhilachale nikhilachale commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Code changes: +2519 -0
Tests: +1869 -0
Others: +1591 -0

Adds an AO contributor skill for assessing coding harnesses, implementing supported integration surfaces, and auditing native CLI/AO lifecycle behavior. The API-first runner preserves failed, blocked and not-run gates, including fresh instruction delivery, cancellation and exact restore.

Harness-specific audit reports, JSON receipts, screenshots and provenance must be published in comments and GitHub attachments on each harness's integration PR; new evidence files must not be committed. This PR contains reusable skill guidance, the runner, fixtures and CI wiring. Version-specific investigation narratives and the dated session index are excluded from its file diff; general integration and evidence requirements remain.

Historical index and per-harness evidence remain preserved at the pre-cleanup revision and through the respective integration PRs; existing evidence-file migration requires verified attachments before removal.

Operational guidance now includes a compact Prepare/Launch/Doctor/Drive/Evidence/Cleanup checklist and a journey map. It maps actions to proof prerequisites and observed results, adds read-only instance checks, identifies separate UI/Chat and independent-instruction coverage, and requires evidence to survive teardown.

The ACP/shared-Chat-UI guide maps the native binding, registry, typed contracts, daemon events and shared renderer. Its focused Chat checklist covers real UI creation, streaming, history/load, decisions, cancellation, reconnect, daemon recovery and exact restore. Chat Doctor uses controller/snapshot readiness rather than TUI composer cues; this adds guidance, not Chat automation to the runner.

A full integration now starts by reading AO/upstream docs and existing PRs, then planning each native capability and AO gap. Supported TUI, Chat and reviewer surfaces must be qualified independently. AO Account login/setup, both model lists, persisted model enable/disable preferences and applicable picker effects, and provider/account configuration with effective native routing are explicit requirements. Reviewer registration/picker, review execution and correlated result/cancel/rerun have separate evidence checkpoints. Unsupported native behavior needs proof; unfinished AO wiring keeps the full integration incomplete.

Integration handoff now requires a completion matrix covering product registration/UI, model lists/visibility, provider management, setup, OS/config readiness, branding, TUI/Chat, reviewer, permissions/instructions, lifecycle, evidence and current CI/merge state. Each row records supported scope, verification status, and proof or an omission reason. Current-status receipts separate runtime-tested builds from the current CI head, preserve sanitized reproduction inputs and distinct attempts, and reconcile stale claims/counts before handoff. These are contributor review gates; the runner schema and automation are unchanged.

The skill now requires a per-harness, per-attempt screenshot index for every meaningful UI action and result: Agent Harness listing, install offer/progress/result, login/configuration or stored-credential state, Agent Harness models, New Task agent/model selection, running/output, follow-up and active-command cancellation, Kill/archive confirmation, archived entry, exact restore, and the same card back in Kanban with a fresh continuation. Supported Chat also requires real shared-UI streaming, decisions, cancellation, reconnect and recovery evidence. Each checkpoint links actual images and functional receipts with build, identity and capture provenance; failures retain original screenshots or explicit capture-unavailable reasons plus sanitized logs. Images are embedded in journey order with durable, verified links on the harness integration PR.

Already installed/connected does not prove a fresh install/login. The accepted saved-credential route can qualify after fresh verified authorization and a successful provider turn; its unexercised fresh-login alternative remains explicitly labelled without blocking that route. Manual-only plans retain an explicit unexercised in-page step and can qualify only agreed manual scope. A planned/absent Agent Harness model panel remains NOT_RUN / NOT_IMPLEMENTED; New Task models cannot substitute. Real credentials must not be deleted to stage captures. Later reproductions cannot replace originals. Screenshots prove visible states, while native identity, provider authorization, cancellation and restored continuation require independent functional proof. These are contributor gates; the API-first runner still captures failures only and does not automate successful-step screenshots. Detailed evidence belongs in PR comments/attachments; the description links to them. Existing committed evidence is removed only after its originals and published destinations are verified; source logos/licenses and synthetic fixtures remain source files. Isolated candidate registration enables real Chat qualification before production registration.

Validation: whitespace and 53 local documentation links/anchors passed, along with independent baseline/forward capability exercises and focused review. This update changes four Markdown files; runner/tests/workflow are unchanged. Current-head CI is pending. Detailed verification and prior failure history are in the PR comment. No live harness/UI qualification is claimed.

Historical run details and failures remain in the linked integration reports and earlier discussion. This contributor tooling does not add a desktop scheduler, automatic login or automatic PR publisher.

How it works

flowchart TD
  A["Read the skill<br/>SKILL.md and reference guides"]
  B["Prepare the agent integration<br/>and its audit configuration"]
  C["Run the audit<br/>audit_ao.mjs"]
  D["Check startup, instructions,<br/>cancellation and restore"]
  E["Save results and evidence<br/>audit_ao.mjs + audit_evidence.mjs"]
  F["Review passed, failed<br/>and blocked checks"]
  T["Tests + GitHub Actions<br/>Check that the audit tools work"]

  A --> B --> C --> D --> E --> F
  T -. validates .-> C
  T -. validates .-> E
Loading

Repository docs point contributors to the skill. The historical index above links to each agent's detailed evidence outside this PR's file diff. Tests validate the audit tools; a live audit checks the actual agent.

@github-actions

Copy link
Copy Markdown
Contributor

🏆 Review leaderboard

Oct 3–10, 2026 · UTC

Rank Reviewer PRs reviewed Review rounds PR comments
🥇 @illegalcall @illegalcall 28 47 11
🥈 @ronishrohan @ronishrohan 20 23 17
🥉 @Vaibhaav-Tiwari @Vaibhaav-Tiwari 16 20 11
4 @Prasad-D-Ware @Prasad-D-Ware 10 27 1
5 @nikhilachale @nikhilachale 8 20 3
6 @axisrow @axisrow 8 10 48
7 @Rishet11 @Rishet11 7 16 2
8 @mohakchakraborty2004 @mohakchakraborty2004 7 7 1
9 @Pritom14 @Pritom14 5 5 3
10 @codebanditssss @codebanditssss 4 4 2

Ranked by distinct external PRs reviewed, then review rounds, then PR comments. Self-activity and bot activity are excluded.

Show 6 more reviewers
Rank Reviewer PRs reviewed Review rounds PR comments
11 @Annieeeee11 @Annieeeee11 3 6 1
12 @harshitsinghbhandari @harshitsinghbhandari 2 4 0
13 @neversettle17-101 @neversettle17-101 2 3 0
14 @whoisasx @whoisasx 2 2 0
15 @AgentWrapper @AgentWrapper 1 3 0
16 @ApexYash11 @ApexYash11 1 2 0

@nikhilachale

This comment has been minimized.

@nikhilachale

This comment has been minimized.

@nikhilachale

This comment has been minimized.

@nikhilachale

Copy link
Copy Markdown
Collaborator Author

Updated the screenshot evidence requirement in bf6a30d.

  • Every harness/interface claimed as running inside AO now requires an embedded real AO desktop/web UI screenshot showing that session and its native output, with AO/native identity, native version, tested SHA/binary hash, UTC timestamp, capture method/path/hash, and a durable GitHub-renderable image URL.
  • Failures keep explicit FAIL/BLOCKED/NOT_RUN labels and their actual relevant failure capture when available; unavailable screenshots retain the reason. Missing AO UI captures or attachments leave AO UI evidence: BLOCKED, even if the API runner passes. Functional logs/assertions remain necessary for cancellation and exact restore.
  • Historical screenshots in this PR remain live terminal mirrors, not proof of execution inside the AO UI. They and the original failed attempts are unchanged. This documentation update makes no new live integration claim.

Validation on the VPS: git diff --check, all changed relative links/heading anchors, and the exact three-Markdown-file scope passed. No runner code changed; the previous 98-fixture result was not rerun or represented as a new result. No Mac compute or new provider run. Current-head GitHub CI is in progress; PR category counts were recalculated from GitHub per-file diffs (including 2 unchanged binary screenshot assets in the overall PR).

@nikhilachale

This comment has been minimized.

@OrcInc OrcInc added this to the Agents & orchestration milestone Oct 10, 2026
@OrcInc OrcInc added comp/daemon Go daemon, process lifecycle, and backend control plane. enhancement New feature or request labels Oct 10, 2026
@nikhilachale

This comment has been minimized.

@nikhilachale

This comment has been minimized.

@nikhilachale

This comment has been minimized.

@nikhilachale

This comment has been minimized.

@nikhilachale

This comment has been minimized.

nikhilachale and others added 16 commits October 22, 2026 19:21
Checked against upstream/main 0e81e1d, where merged harnesses (Hermes,
Memcode, CodeWhale) were reported as unwired:
- object keys may be bare (`hermes: hermesLogo`, `memcode: "memcode"`)
- auth plans come from documentationPlan/loginMenuPlan/terminalInputPlan
- Chat drivers register as `p := pkg.New()` then `add(pkgacp.New(p, ...))`
- logos may ship as -dark/-light variants
- licence files and docs/harnesses pages are optional (source comments are
  accepted)
The catalog's cross-harness switch now points at provider_switch_tui.go and
POST /sessions/{id}/provider-switch, which replaced agent switching.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…failure

The first live run failed at startup: a cold go build (toolchain and
modules download) hit the 120 s readiness timeout, and the reason was 50
'go: downloading' lines. The build now has a 10-minute budget; a timeout
says so in one line, other build failures show the real compiler lines.
A run-level failure's rerun now repeats the agent selection instead of
'--agent null --only startup'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
codex debug models prints ~0.5 MB of JSON; the 64 KB command output tail
cut it, so local_models_list read 'no models'. The model-list command now
keeps up to 16 MB, and unparsable output says so instead of 'no models'.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dels to leak them

The first live Codex run showed two audit-prompt flaws, not AO bugs:
- Codex wrote the proof to the AO session artifacts folder, as AO's
  using-ao guidance asks for deliverables; the prompt now says the test
  file belongs in the worktree (pwd), and a proof found in artifacts is
  reported as exactly that instead of "prompt never delivered"
- Codex refused to "disclose private instruction values". Asking a model to
  repeat a secret from its system prompt reads as prompt extraction. Project
  rules (agentRules) and AGENTS.md now state ordinary conventions ("every
  JSON file must include rulesMarker / agentsMarker"), and a rule added
  after Kill (restoreRulesMarker, chatRestoreRulesMarker) must show up in
  the restored session's next ordinary edit
activity_status no longer depends on the proof file: Codex settled
correctly but the gate failed with the proof.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t-run dialogs in the UI audit

Live Codex applied the project rule so faithfully that it refused the
cancellation witness command, whose JSON files lacked rulesMarker. The rule
now covers only files whose name ends in PROOF.json.
The UI audit was blocked by AO's first-run 'Import projects from your
earlier AO?' dialog; it now dismisses such dialogs with a defer answer
(Skip, Not now) only, never Proceed or Don't Migrate, and records it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he UI audit

From the live Codex runs:
- instruction_channel_restore checked a stale token: makeScenario copied the
  scenario, so the restore rule minted after Kill was never the value the
  extras suite compared. AO did regenerate system.md; scenarios are now one
  shared object.
- Chat approvals wrote inside the worktree, which Codex's workspace-write
  sandbox (accept-edits) allows without asking. The command now writes
  outside the worktree (~/.ao/audit-approvals, or AO_AUDIT_APPROVAL_DIR) and
  the file is removed afterwards.
- UI audit: read text with innerText (textContent glued "Account" and
  "Connected" together), accept "Log in again", dismiss defer-able first-run
  dialogs anywhere in the stack, route by hash instead of reloading, and
  start each run from a clean home screen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g the session

A user stops a running TUI agent in AO by archiving the session, not by
keystrokes. By default the cancellation gate now starts the witnessed
foreground command, archives (kills) the session while it runs, and requires
the session to terminate, the command's process to be gone and its delayed
file never to appear. That archive doubles as the Kill before restore.
The keystroke interrupt over /mux stays available only for a contract that
declares `interrupt`; the bundled Codex/Claude contract no longer does.
A command that outlives the archived session is classified ao_bug and points
at the session manager and process supervisor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live Codex answered correctly (worker / orchestrator) but saved AO_ROLE.txt in
the session artifacts folder, as AO's using-ao guidance asks, so the gate saw
nothing. The prompt now asks for the working directory, and the gate also
reads <data>/artifacts/<session>/AO_ROLE.txt, saying where the answer was.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…into search

The Harness page search filters by display name ("Claude Code"), so typing
the harness ID "claude-code" matched nothing. The UI audit now clears the
search (or types --label) and selects the row by its data-agent ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r TUIs

- Claude Chat failed with CHAT_DRIVER_UNAVAILABLE: a plain go build does not
  bundle AO's ACP bridge. The isolated daemon now gets AO_ACP_RUNTIME_DIR
  from the checkout's frontend/resources/acp-runtime, and host preflight
  says how to build it when it is missing.
- A blocked restored ready cue now keeps the last 1.5 KB of screen text
  (ANSI-stripped, redacted) so the contract pattern can be fixed from the
  real frame.
- UI audit: wait up to 90 s for a terminal (Claude Code paints later than
  Codex) and confirm the agent's READY reply when the terminal text is
  readable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Live Codex restore frames failed every ready-cue pattern: Codex spaces words
with cursor-forward moves (CSI n C), which ANSI stripping removed, gluing
"Conversation interrupted" together, and its spinner window title (OSC 0)
leaked after the composer. Terminal text now turns cursor-forward into
spaces and drops OSC sequences before matching and in the saved screen tail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n, UI audit picks visible surfaces

- Codex's accept-edits sandbox blocked the out-of-worktree write (EPERM) and
  the agent stopped, as the prompt told it to on denial. Codex only raises an
  approval when it asks for escalated permissions, so the prompt now asks
  for that and stops only when the approval itself is denied.
- UI audit: AO keeps earlier sessions' terminals mounted but hidden; the
  surface checks now match visible terminals/chats only, and the default
  interface is set for every interface tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The live Codex orchestrator answered 'orchestrator' but refused to write
AO_ROLE.txt into its workspace, as its AO role rules require (non-PR
deliverables stay outside the workspace). Both roles now save the answer
in their AO session artifacts folder, which every role may use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude's ACP bridge has no models list; it offers model (15 choices) and
effort as Chat config options, as the Chat UI shows. chat_models and
chat_settings now use config-options when the models list is empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the "new rule after restore" checks (system_prompt_restore,
instruction_channel_restore, chat_system_prompt_restore): a restored session
keeps the instructions it started with, which is expected.

Move everything harness-specific into the harness contract so the skill
works for any integrated harness, with Codex and Claude Code as examples:
- capability support is required/optional/manual in the catalog, with
  per-harness overrides in the contract's `supports` and `surfaces`
- adapter directories for failure diagnoses come from `adapters`
- instruction channels must be declared; no Codex/Claude defaults
- hooks are checked from the contract's `hooks` file, else via the
  hook-persisted native session id
- run_local.sh audits and doctors every harness in the contract file
- add assets/contract-template.json; rename the contract file to
  assets/local-contracts.json; SKILL.md documents the contract step

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nikhilachale

Copy link
Copy Markdown
Collaborator Author

Replaced by #7081, a smaller version of this skill.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/daemon Go daemon, process lifecycle, and backend control plane. enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants