[docs-noob-tester] 📚 Documentation Noob Test Report - 2026-09-20

[github-actions[bot]]

Summary

• Date of test: 2026-09-20
• Pages visited:
  • Home — `(localhost/redacted)
  • Quick Start — `(localhost/redacted)
  • CLI Commands — `(localhost/redacted)
• Overall impression: As a brand-new user, the landing page told me exactly what to click ("Get Started with CLI"), and the Quick Start guide walked me through a complete, runnable example end to end. The main friction point was the sheer size of the left sidebar, which made the site feel overwhelming before I even reached the guide content, and a few CLI/authentication steps assume more background knowledge (PATs, workflow scopes, GitHub Actions settings) than a total beginner would have.

Critical Issues Found

None found that would fully block a new user. No 404s, no broken links, no console errors were observed on any of the 3 pages (confirmed via browser console logs).

Confusing Areas 🟡

1. Sidebar is huge and undifferentiated for a first-time visitor — The left nav on every page (Introduction, Setup, AI Engines, Guides, Gallery, Design Patterns, Practices, Reference...) lists 100+ links with no visual indication of "start here" vs. "advanced/reference." A total beginner landing directly on /setup/quick-start/ (e.g., from a shared link) sees this wall of navigation before any guide content and can lose the thread of "what do I do first?"
   📎 See quickstart-top.png (full sidebar visible alongside guide content).

2. "YAML frontmatter" and "lock file" jargon introduced without a beginner-friendly definition up front — The very first sentence of the Quick Start guide says workflows are defined "in Markdown with YAML frontmatter (the configuration block between the --- markers)." This is a reasonable inline definition, but a first-time reader unfamiliar with static-site generators or Jekyll-style frontmatter may still need a concrete example immediately, not just a parenthetical. Similarly, "lock file" appears in Step 2 with a link, but the concept (a generated .lock.yml companion to the .md source) isn't explained until a "Note" callout much later in the page.

3. Authentication section is dense and branches 5 ways (Copilot org billing vs. COPILOT_GITHUB_TOKEN, Claude, Codex, Gemini, Pi) with PAT-creation steps, gh secret set commands, and links to other pages — all presented as one long undifferentiated block under Step 2. A total beginner following the "just pick Copilot" path still has to visually scan past four irrelevant provider sections to find the two bullets that apply to them.

4. Prerequisites list assumes some GitHub knowledge — "GitHub Actions enabled" and "write access" are listed correctly, but there's no link explaining how to tell if you have write access or what "workflow" OAuth scope means before the gh auth login --scopes repo,workflow command is shown.

What Worked Well 🟢

• Home page clearly states what the tool does in one sentence ("Intelligent automation for GitHub...") and immediately surfaces two clear CTAs: "Get Started with CLI" and "Creating Workflows." No hunting required.
• Quick Start guide structure is genuinely good: it states an "Estimated time: 10 minutes," lists prerequisites up front, then numbers concrete Steps 1–4 with copy-paste-ready commands (gh extension install github/gh-aw, gh aw add-wizard githubnext/agentics/repo-status, gh aw compile, gh aw run repo-status).
• Fallback install path — a "Tip" callout offers a curl | bash script as a backup if gh extension install hits auth issues. This anticipates a real beginner pain point nicely.
• CLI Commands page leads with a compact "Day-one commands" table (10 rows: init, doctor, add-wizard, add, new, compile, list, run, status, logs, audit) with a "Description" and "When to use" column, plus a callout: "New to gh aw? Start with the day-one commands. The advanced and enterprise setup is further down the page." This is exactly the kind of curation a beginner needs and directly mitigates the sidebar-overload issue noted above.
• Concrete success signal — the guide tells you exactly what to expect on success (a new "Daily Repo Report" issue) and even includes a screenshot example, so users know they succeeded without needing to interpret Actions logs.

Recommendations

Quick wins:

1. Add a short "New here? Start with these 3 links" callout at the very top of the sidebar (mirroring the "day-one commands" tip already used on the CLI page) to reduce the wall-of-links overwhelm on first load.
2. Move the "lock file" explanation (currently a late "Note" callout) up to where .lock.yml is first mentioned in Step 2, instead of after Step 4.
3. Collapse the 5-provider authentication block into tabs or a <details>/accordion per engine so a Copilot-only user doesn't have to scroll past Claude/Codex/Gemini/Pi instructions.

Longer-term:

1. Consider a lightweight glossary tooltip or inline link for first-use jargon ("frontmatter", "lock file", "safe outputs") directly in the Quick Start flow rather than relying on the separate Glossary reference page.
2. Add a one-line "how do I know if I have write access?" hint next to the Prerequisites bullet, linking to GitHub's own permissions docs.

Screenshots

📎 [home.png] — Home page, "Get Started with CLI" CTA — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/2d676c2bcea35bcb2a27f97dd535963a63674e7ecd6133a868612587ea5bebc9.png?raw=true

📎 [quickstart-top.png] — Quick Start guide with full sidebar visible (illustrates sidebar-overload issue) — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/d0e3f2dbe394d8df3abe6a28e2842e527aa8cca39e1d6c877720879a5311b13a.png?raw=true

📎 [cli-commands.png] — CLI Commands page with "Day-one commands" table — asset URL: https://github.com/github/gh-aw/blob/assets/Documentation-Noob-Tester/2196c2d73600f9d2d40b76c6192b79d23c4710df8549719684f02dddeda1da02.png?raw=true

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• update.googleapis.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "update.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

Generated by 📝 Documentation Noob Tester · copilot · auto · 56.6 AIC · ⌖ 10.2 AIC · ⊞ 10.7K · ◷

• expires on Sep 20, 2026, 7:54 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-21T03:54:44.855Z.

Closed by Workflow