[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-23

[github-actions[bot]]

Summary

• Date of test: 2026-09-23
• Pages visited:
  • Home — (localhost/redacted)
  • Quick Start Guide — (localhost/redacted)
  • CLI Commands — (localhost/redacted)
• Overall impression: As a first-time user, I could find my way to "Get Started" quickly and the CLI reference page is genuinely beginner-friendly. The Quick Start guide itself is solid, but the "Configuring authentication" section under Step 2 is overwhelming — five engines' worth of setup instructions are all shown at once with no clear "just pick one" visual cue.

Critical Issues Found

None encountered that would fully block a new user — all code snippets and links on the 3 pages I reviewed appeared complete and internally consistent, and I did not hit any 404s or broken navigation.

Confusing Areas

1. "Configuring authentication" wall of text (Quick Start, Step 2)

   • URL: /gh-aw/setup/quick-start/#configuring-authentication
   • As a noob, when I hit this section I expected a simple decision ("using Copilot? do X") but instead found five stacked engine sections (Copilot, Claude, Codex, Gemini, Pi), each with its own env var names, links, and sub-steps, with no toggle/tabs UI to hide the ones I don't need. It's easy to accidentally follow the wrong engine's steps or feel like you have to read all five before proceeding.
   • Screenshot: see below.

2. Jargon without inline definition: terms like "frontmatter", "lock file", and "safe outputs" are used early in the Quick Start (e.g., "the configuration block between the --- markers") — the frontmatter explanation is present, but a first-time reader may not connect that this is the same concept GitHub Actions/Jekyll users would recognize as YAML frontmatter until they read closely.

3. Multiple near-identical "add a workflow" commands: the CLI Commands page lists gh aw add-wizard, gh aw add, and gh aw new all in the "Day-one commands" table with only a one-line distinction each — a beginner following the Quick Start (which uses add-wizard) may not immediately understand when to reach for add or new instead. A comparison table further down helps, but it's easy to miss if you stop reading after the top table.

What Worked Well

• Home page clearly states what the product does above the fold, and "Get Started with CLI" / "Quick Start" links are impossible to miss (top nav + hero button).
• CLI Commands page puts a "Day-one commands" table at the very top, above the fold, with one-line command + purpose for each — this is exactly what a new user needs first, and there's an explicit tip pointing back to the Quick Start guide for newcomers.
• Quick Start Prerequisites section is a clean checklist (AI engine account, repo write access, Actions enabled, gh CLI version, auth) with links to install each dependency — nothing is assumed silently.
• Step 1 (install extension) includes a fallback "Tip" box with an alternate install script for users hitting auth issues — a thoughtful detail for troubleshooting.
• The guide explicitly estimates "~10 minutes" and lists exactly what tool is required (GitHub CLI only), which sets good expectations up front.

Recommendations

Quick wins

• Convert the "Configuring authentication" section into tabs (one per engine) so a user only sees the engine they picked in the wizard, instead of scrolling past four irrelevant engine setups.
• Add a one-line "if you're not sure, use GitHub Copilot" callout right before the authentication section, since the prerequisites section already suggests starting with Copilot.

Longer-term

• Add a very short glossary tooltip/inline definition the first time "frontmatter" and "lock file" are used in the Quick Start, even though a dedicated Glossary page exists elsewhere — new users are unlikely to leave the current page to look it up.
• Consider merging the "Day-one commands" table and the deeper add-wizard vs add vs new comparison table into a single scannable spot, so the distinction is visible without needing to scroll further down the CLI Commands page.

Screenshots

📎 home.png — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/3795f425ecd14884b6b5c389511b835ca55f26cf3587f018c189006c6c87910e.png?raw=true

📎 quickstart-top.png — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d0e3f2dbe394d8df3abe6a28e2842e527aa8cca39e1d6c877720879a5311b13a.png?raw=true

📎 quickstart-auth-section.png — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/31839850e9d3f95361e88c683ff0583cba217b5628e522df20889004875eb64b.png?raw=true

📎 quickstart-prerequisites.png — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/8f4f78680dc448d1e2ca92379dccefb42abc855916cd8b104c31c5992186a7af.png?raw=true

📎 cli-commands-top.png — https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/c5f732ff7ce25e5d79eb683cf7a3993454da32d9b2b331f136b55e43c54b5f03.png?raw=true

Warning

Firewall blocked 7 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• o205451.ingest.us.sentry.io
• update.googleapis.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "o205451.ingest.us.sentry.io"
    - "update.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 63.4 AIC · ⌖ 7.43 AIC · ⊞ 10.7K · ◷

• expires on Sep 23, 2026, 7:57 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Documentation Noob Tester.

A newer discussion is available at Discussion #63093.