[claude-code-user-docs-review] 🔍 Claude Code User Documentation Review - 2026-09-23

[github-actions[bot]]

Executive Summary

gh-aw's docs list five engines (Copilot, Claude, Codex, Gemini, Pi) as equals, but Copilot still gets preferential framing, auto-scaffolding, and MCP wiring that Claude/Codex/custom users must replicate by hand. The long-running CLAUDE_CODE_OAUTH_TOKEN silent-failure gap (unresolved 32 consecutive daily runs) remains undocumented in quick-start's Claude section, even though it is now explained in prose elsewhere. No critical blockers this run, but four major obstacles persist — mostly parity gaps for non-Copilot engines.

Severity Findings

Critical Blockers (0) · Major Obstacles (4) · Minor Confusion (3)

Critical Blockers: None this run — the OAuth-token gap is documented in prose (see Auth Gaps below).

Major Obstacles:

1. CLAUDE_CODE_OAUTH_TOKEN (the token from claude login) is silently ignored — cli.md:240 and engines/claude.md:22 explain it, but quick-start.mdx:133-139 (the Claude setup section) never mentions it, so Claude-CLI users following quick-start alone hit a token-less "authentication error" from the Claude CLI with no clue it's a token-type mismatch. Unresolved across 32 consecutive daily runs since 2026-08-22.
2. Example/smoke-test parity gap: 58 Claude workflow declarations vs. 113 Copilot (~1.9x) across .github/workflows/*.md. Copilot has more smoke-test variants (e.g. smoke-copilot-arm, smoke-copilot-aoai-entra) than Claude.
3. gh aw init only auto-scaffolds the custom agent file (.github/agents/agentic-workflows.md) and enables MCP wiring by default for --engine copilot (cli.md:115, cli.md:140); Claude/Codex users get no equivalent scaffolding and must hand-configure their own MCP host.
4. web-search: is disabled by default for Claude and Codex but enabled natively for Copilot (tools.md:128) — the asymmetry is stated as fact with no rationale, so Claude users may not realize the capability exists at all.

Minor Confusion:

1. quick-start.mdx:72 frames Copilot as the path of least resistance ("If you already have GitHub Copilot, start there — it requires no extra account setup"), reading as a default recommendation over Claude.
2. Anthropic WIF keyless setup is link-only in quick-start.mdx:139 (parity with Gemini's link-only WIF at :155, but neither engine gets inline steps).
3. Ambiguous tool scope: web-search and cli-proxy MCP-mounting semantics aren't fully spelled out per engine (tools.md:128, :215).

Engine & Tool Matrix

Engine	Setup	Examples	Auth	Score
Copilot	Auto-scaffolded agent file + MCP wiring by default (cli.md:115,140)	113 workflows, most smoke variants	copilot-requests:write or COPILOT_GITHUB_TOKEN (auth.mdx:25-26)	9/10
Claude	Manual MCP config; dedicated engines/claude.md guide	58 workflows (~51% of Copilot's)	ANTHROPIC_API_KEY or WIF (auth.mdx:31-32); OAuth token silently ignored (cli.md:240)	6/10
Codex	Manual MCP config	75 workflows	OPENAI_API_KEY / CODEX_API_KEY (auth.mdx:36-37)	7/10
Custom	No scaffolding; user-defined harness	1 workflow (shared/genaiscript.md)	Self-managed	4/10

Tool classification (doc-reader): 13 universal tools (edit, github, bash, playwright, cache-memory, etc.), 3 Copilot-only frontmatter fields (max-continuations, engine.agent, engine.harness — claude.md:55), 1 engine-dependent tool (web-search, disabled-by-default for Claude/Codex — tools.md:128).

Parity observations (engine-example-counter): Claude and Copilot both have deep smoke-test/daily-workflow coverage, but Copilot's count is ~2x Claude's; specialized engines (Pi 31, Aider 5, Cursor 3, Gemini 1, custom 1) trail far behind all three majors, suggesting they remain experimental-stage.

Auth Gaps

• CLAUDE_CODE_OAUTH_TOKEN silently ignored; the resulting failure message never names the token (cli.md:240) — no corresponding warning in quick-start's Claude section (quick-start.mdx:133-139).
• Anthropic and Google WIF are both link-only from quick-start (quick-start.mdx:139, :155); full inline steps only live in auth.mdx:226-289 / :354-410.
• gh aw secrets bootstrap (cli.md:244) doesn't detect or warn about a stray CLAUDE_CODE_OAUTH_TOKEN secret before a run fails.
• OAuth user tokens (gho_*) are explicitly rejected at activation for COPILOT_GITHUB_TOKEN/GH_AW_GITHUB_TOKEN (auth.mdx:160) — there's no equivalent explicit rejection-and-remedy note for Claude's OAuth token.

Recommended Actions

Priority 1: Add a one-line inline warning in quick-start.mdx's Claude section (~line 136) that CLAUDE_CODE_OAUTH_TOKEN from claude login is unsupported — don't make users click through to engines/claude.md to discover it.

Priority 2: Give Claude/Codex gh aw init parity with Copilot's auto-scaffolding (cli.md:115), or explicitly document the manual MCP steps for non-Copilot engines inline in cli.md rather than only in a comparison table.

Priority 3: Explain the design rationale for web-search being off-by-default for Claude/Codex (tools.md:128), and consider inlining (not just linking) WIF setup steps for Anthropic in quick-start.

Meta note: For the 9th consecutive day, .claude/agents/doc-reader.md and .claude/agents/engine-example-counter.md lack the name: frontmatter field required to register as invocable agents (only description/model are present), so they fail to appear in the available agent list. An Edit attempt to add name: fields was denied by permissions again this run. Fell back to general-purpose agents carrying the same instructions inline — results should be equivalent, but this registration gap itself is worth fixing at the source.

References

• §35861822798 — this run
• §32573117343 — OAuth-token gap first flagged (2026-08-22)
• §34227198249 — OAuth-token gap downgraded Critical→Major (2026-09-08)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📝 Claude Code User Documentation Review · claude · agent · 175 AIC · ⌖ 5.93 AIC · ⊞ 6.5K · ◷

• expires on Sep 24, 2026, 4:51 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-24T12:51:27.494Z.

Closed by Workflow