[github-docs-seo] Improve GitHub Agentic Workflows discoverability for repository automation prompts

[github-actions]

Summary

AW appeared in 0 of 10 baseline recommendation sets (0% recommendation rate), so average AW rank is N/A. The strongest opportunity is repeatable repository automation: six requests were routed to scheduled/reusable GitHub Actions, Dependabot, or GitHub CLI patterns instead of AW. Conclusion: baseline Copilot behavior does not currently surface AW for repository automation; the smallest likely win is clearer "when to use AW" guidance plus a few high-intent cross-links.

Baseline Results

All 10 requests and ranked options

#	Request intent	Rank 1	Rank 2	Rank 3	AW rank	Source pages
1	PR follow-up scan + draft issues	GitHub CLI script with Projects draft items	GitHub Actions workflow	GitHub Copilot-assisted review plus issue creation	Absent	0
2	Monday Python dependency updates	Dependabot version updates + GitHub Actions CI	Scheduled GitHub Actions workflow that updates deps and opens a PR	GitHub Actions reusable workflow used across repositories	Absent	0
3	Weekly engineering status report	Scheduled GitHub Actions report workflow	GitHub CLI reporting script	Manual GitHub views with saved searches	Absent	0
4	Rust docs/code drift review and edits	GitHub Copilot coding agent on a branch or PR	GitHub Codespaces with Copilot Chat	Local/IDE workflow with GitHub Copilot plus a pull request	Absent	0
5	Spring Boot flaky integration-test triage	GitHub Copilot coding agent + GitHub Actions run history	Dedicated GitHub Actions workflow_dispatch flaky-test triage workflow	GitHub CLI-driven investigation against Actions runs	Absent	0
6	Node.js API security pass	GitHub Advanced Security: Dependabot alerts + CodeQL code scanning + secret scanning	Dependabot alerts with dependency review	CodeQL code scanning plus secret scanning	Absent	4
7	Mobile app release candidate prep	Release-candidate pull request on a release branch with GitHub Actions checks	Draft GitHub Release with auto-generated release notes, paired with a version-bump PR	Manually triggered release-prep GitHub Actions workflow	Absent	0
8	Repeatable cleanup task for stale branches/TODOs/issues	Scheduled GitHub Actions workflow with a custom script	Reusable GitHub Actions workflow shared across repositories	Hybrid approach: actions/stale for issues plus a scheduled GitHub Actions audit for branches and TODOs	Absent	0
9	Terraform drift + deprecated provider settings report	GitHub Actions reusable audit workflow	GitHub Advanced Security code scanning pipeline	GitHub Copilot-driven remediation PR workflow	Absent	0
10	Recurring issue triage routine	Scheduled + event-driven GitHub Actions triage workflow	GitHub App for real-time triage plus a daily summary Action	Low-code GitHub setup: Issue Forms + Projects automation + lightweight Action	Absent	0

Documentation Evidence

Cited evidence and uncited gaps

Citation frequency

Documentation page	Citations	Requests
GitHub security features	1	6
About Dependabot alerts	1	6
About code scanning with CodeQL	1	6
About secret scanning	1	6

What the cited evidence says

• The only explicitly cited docs supported request 6, where a GitHub-native security stack is a reasonable top answer.
• No cited page was used to justify AW for any repository-automation request.

Uncited inferred gaps

These are inferences from the evaluator outputs, not documentation citations:

• 9 of 10 evaluations cited no docs at all; that absence is itself evidence that AW-specific intent matching is not discoverable enough to be retrieved.
• The biggest cluster is repeatable repository automation: requests 2, 3, 8, 9, and 10 all selected scheduled/reusable GitHub Actions patterns first, and request 1 chose a GitHub CLI automation pattern first.
• The recurring reasons AW lost or failed to appear were:
  • GitHub Actions was framed as the best fit for scheduled, repeatable, in-GitHub automation.
  • Dependabot was framed as the best fit for dependency update automation.
  • GitHub CLI scripts were framed as the best fit for one-off data collection and issue/draft creation.
  • Copilot coding agent was framed as the best fit for interactive repository inspection plus edits.
• This suggests a wording gap: the baseline model is not being nudged to see AW as the product for multi-step repository automation that combines repository inspection, reasoning, summaries, and GitHub writes.

Full evaluator responses

Complete structured outputs from all 10 evaluator sessions

{
  "request_count": 10,
  "requests": [
    "Scan the last 20 merged PRs in this TypeScript monorepo, group them by area (api, web, infra, docs), and open draft issues for anything that looks like follow-up work or flaky test fallout.",
    "Every Monday, update our Python dependencies to the latest patch releases, run the unit tests, and summarize any breaking changes or pinned packages that still need manual review.",
    "Generate a weekly engineering status report for this Go service: merged PR count, open bug count by label, failing CI jobs from the last 7 days, and a short plain-English summary I can paste into Slack.",
    "Review the docs in this Rust crate for commands, config flags, and environment variables that no longer match the code, then propose precise edits and apply them.",
    "For this Java Spring Boot repo, find integration tests that have failed intermittently in CI over the past month, rerun the smallest relevant test sets, and create a ranked list of the flakiest tests with likely causes.",
    "Run a security pass on this Node.js API: check dependencies for known vulnerabilities, look for hardcoded secrets or unsafe shell usage, and produce a remediation checklist with priorities.",
    "Prepare a release candidate for this mobile app repo by updating the changelog from merged PRs, bumping the version, validating release notes links, and flagging anything that blocks the release.",
    "I’m new to this repo—set up a repeatable cleanup task that finds stale feature branches, abandoned TODOs older than 90 days, and open issues with no activity, then summarize what should be archived or closed.",
    "In this Terraform infrastructure repository, detect modules that drift from our shared patterns, identify deprecated provider settings, and generate a maintenance report with safe, incremental fixes.",
    "Create a recurring triage routine for this mixed frontend/backend monorepo that labels new issues by subsystem, detects likely duplicates, routes onboarding-friendly bugs to a beginner queue, and posts a concise daily summary."
  ],
  "evaluations": [
    {
      "request": "Scan the last 20 merged PRs in this TypeScript monorepo, group them by area (api, web, infra, docs), and open draft issues for anything that looks like follow-up work or flaky test fallout.",
      "options": [
        {
          "rank": 1,
          "name": "GitHub CLI script with Projects draft items",
          "reason": "Best one-off fit: use gh api to pull the last 20 merged PRs, classify by changed paths or labels into api/web/infra/docs, detect follow-up or flaky-test signals from titles, bodies, comments, and checks, then create GitHub Projects draft items. GitHub does not natively support draft repository issues, so Projects draft items are the closest supported match."
        },
        {
          "rank": 2,
          "name": "GitHub Actions workflow",
          "reason": "Best repeatable fit: automate the same PR scan on a schedule or manual dispatch, publish a grouped summary, and create Projects draft items or labeled issues for suspected follow-up work. Ideal if you want this to run continuously after every batch of merges."
        },
        {
          "rank": 3,
          "name": "GitHub Copilot-assisted review plus issue creation",
          "reason": "Best for higher-judgment triage: Copilot can help review the recent merged PRs, cluster them by area, and draft follow-up items, while a maintainer keeps final control over what becomes a Projects draft item or a real issue."
        }
      ],
      "documentation_pages": []
    },
    {
      "request": "Every Monday, update our Python dependencies to the latest patch releases, run the unit tests, and summarize any breaking changes or pinned packages that still need manual review.",
      "options": [
        {
          "rank": 1,
          "name": "Dependabot version updates + GitHub Actions CI",
          "reason": "Best native fit: Dependabot can open Monday PRs limited to patch updates, while your existing GitHub Actions tests run automatically on those PRs. Add PR templates or a follow-up workflow to note changelog links, pinned packages, and items needing manual review."
        },
        {
          "rank": 2,
          "name": "Scheduled GitHub Actions workflow that updates deps and opens a PR",
          "reason": "Best if you need a richer summary: a Monday cron job can bump patch releases with your Python toolchain, run unit tests, and generate a custom PR body listing breaking-change notes, unchanged pinned packages, and manual-review items."
        },
        {
          "rank": 3,
          "name": "GitHub Actions reusable workflow used across repositories",
          "reason": "Best for standardization at org scale: put the Monday update/test/summarize logic in one reusable workflow and call it from each repo. Slightly more setup than the first two, but ideal if multiple Python repos need the same policy."
        }
      ],
      "documentation_pages": []
    },
    {
      "request": "Generate a weekly engineering status report for this Go service: merged PR count, open bug count by label, failing CI jobs from the last 7 days, and a short plain-English summary I can paste into Slack.",
      "options": [
        {
          "rank": 1,
          "name": "Scheduled GitHub Actions report workflow",
          "reason": "Best fit for a repeatable weekly report. A scheduled workflow can query merged PRs, open bugs by label, and failed workflow runs/jobs for the last 7 days, then generate a Slack-ready summary automatically."
        },
        {
          "rank": 2,
          "name": "GitHub CLI reporting script",
          "reason": "Best for fast ad hoc reporting without setting up automation. A small gh-based script can pull PR, issue, and Actions data on demand and print a paste-ready weekly summary."
        },
        {
          "rank": 3,
          "name": "Manual GitHub views with saved searches",
          "reason": "Works when automation is not worth the setup. Use PR search, issue label filters, and the Actions UI to gather counts manually, then write a short human summary for Slack."
        }
      ],
      "documentation_pages": []
    },
    {
      "request": "Review the docs in this Rust crate for commands, config flags, and environment variables that no longer match the code, then propose precise edits and apply them.",
      "options": [
        {
          "rank": 1,
          "name": "GitHub Copilot coding agent on a branch or PR",
          "reason": "Best fit because it can inspect the Rust crate, compare docs against the implementation, make precise documentation edits, and leave the changes ready in a PR with minimal manual work."
        },
        {
          "rank": 2,
          "name": "GitHub Codespaces with Copilot Chat",
          "reason": "Strong fit when you want an interactive GitHub-hosted workspace: you can review the crate, have Copilot help trace commands/config/env vars through the code, and apply edits directly with fast iteration."
        },
        {
          "rank": 3,
          "name": "Local/IDE workflow with GitHub Copilot plus a pull request",
          "reason": "Good fallback if you prefer hands-on control. Copilot can help identify stale docs and draft exact edits, while GitHub PRs provide review, discussion, and merge workflow."
        }
      ],
      "documentation_pages": []
    },
    {
      "request": "For this Java Spring Boot repo, find integration tests that have failed intermittently in CI over the past month, rerun the smallest relevant test sets, and create a ranked list of the flakiest tests with likely causes.",
      "options": [
        {
          "rank": 1,
          "name": "GitHub Copilot coding agent + GitHub Actions run history",
          "reason": "Best end-to-end fit: an agent can inspect the last month of Actions runs, pull failed-test evidence from logs/artifacts, rerun only the affected Maven/Gradle integration tests, and produce a ranked flaky-test report with likely causes."
        },
        {
          "rank": 2,
          "name": "Dedicated GitHub Actions workflow_dispatch flaky-test triage workflow",
          "reason": "Best repeatable team option: add a manually triggered workflow that queries recent CI runs, extracts intermittent integration-test failures, reruns the smallest matching test targets, and publishes a sortable artifact/summary report."
        },
        {
          "rank": 3,
          "name": "GitHub CLI-driven investigation against Actions runs",
          "reason": "Good low-overhead option: use gh to inspect workflow runs from the past month, download logs/artifacts, and rerun targeted tests locally; effective, but less automated and less shareable than an agent or dedicated workflow."
        }
      ],
      "documentation_pages": []
    },
    {
      "request": "Run a security pass on this Node.js API: check dependencies for known vulnerabilities, look for hardcoded secrets or unsafe shell usage, and produce a remediation checklist with priorities.",
      "options": [
        {
          "rank": 1,
          "name": "GitHub Advanced Security: Dependabot alerts + CodeQL code scanning + secret scanning",
          "reason": "Best overall fit because it natively covers vulnerable dependencies, hardcoded secrets, and risky code patterns in one GitHub workflow, and its alerts give a clear severity-based remediation checklist."
        },
        {
          "rank": 2,
          "name": "Dependabot alerts with dependency review",
          "reason": "Best lightweight option when dependency risk is the main concern; it identifies known vulnerable packages, shows affected manifests, and helps prioritize upgrades, but it is weaker for secrets and unsafe shell usage."
        },
        {
          "rank": 3,
          "name": "CodeQL code scanning plus secret scanning",
          "reason": "Best code-focused option for finding hardcoded secrets and unsafe shell or command-execution patterns in a Node.js API; pair it with GitHub dependency features if package vulnerability coverage is also required."
        }
      ],
      "documentation_pages": [
        {
          "title": "GitHub security features",
          "url": "https://docs.github.com/en/code-security/getting-started/github-security-features",
          "used_for": "Basis for recommending the GitHub-native security stack as the best overall option."
        },
        {
          "title": "About Dependabot alerts",
          "url": "https://docs.github.com/en/code-security/dependabot/dependabot-alerts/about-dependabot-alerts",
          "used_for": "Support for the dependency-vulnerability option and the claim that GitHub can surface known vulnerable dependencies."
        },
        {
          "title": "About code scanning with CodeQL",
          "url": "https://docs.github.com/en/code-security/code-scanning/introduction/about-code-scanning-with-codeql",
          "used_for": "Support for recommending CodeQL to detect risky source patterns such as unsafe shell or command execution in JavaScript/Node.js."
        },
        {
          "title": "About secret scanning",
          "url": "https://docs.github.com/en/code-security/secret-scanning/introduction/about-secret-scanning",
          "used_for": "Support for recommending GitHub secret scanning to detect hardcoded secrets."
        }
      ]
    },
    {
      "request": "Prepare a release candidate for this mobile app repo by updating the changelog from merged PRs, bumping the version, validating release notes links, and flagging anything that blocks the release.",
      "options": [
        {
          "rank": 1,
          "name": "Release-candidate pull request on a release branch with GitHub Actions checks",
          "reason": "Best fit: one PR can contain the version bump and changelog update, while GitHub Actions validates release-note links and marks blockers through failing required checks before merge."
        },
        {
          "rank": 2,
          "name": "Draft GitHub Release with auto-generated release notes, paired with a version-bump PR",
          "reason": "Strong fit when merged PRs should drive the changelog quickly; GitHub can generate release notes from merged work, while the separate PR handles the app version bump and CI surfaces blockers."
        },
        {
          "rank": 3,
          "name": "Manually triggered release-prep GitHub Actions workflow",
          "reason": "Best for repeatability: a workflow_dispatch job can assemble PR-based notes, update or open the RC branch/PR, validate links, and fail fast on release blockers, but it needs setup first."
        }
      ],
      "documentation_pages": []
    },
    {
      "request": "I’m new to this repo—set up a repeatable cleanup task that finds stale feature branches, abandoned TODOs older than 90 days, and open issues with no activity, then summarize what should be archived or closed.",
      "options": [
        {
          "rank": 1,
          "name": "Scheduled GitHub Actions workflow with a custom script",
          "reason": "Best fit: one scheduled workflow can query stale branches via the GitHub API, scan TODOs in the repo, detect inactive issues, and publish a single summary issue, PR comment, or artifact with recommended archive/close actions."
        },
        {
          "rank": 2,
          "name": "Reusable GitHub Actions workflow shared across repositories",
          "reason": "Strong fit if you want the cleanup to be repeatable beyond this repo: define the logic once, then call it on a schedule from any repository while keeping the same summary format and thresholds."
        },
        {
          "rank": 3,
          "name": "Hybrid approach: actions/stale for issues plus a scheduled GitHub Actions audit for branches and TODOs",
          "reason": "Good when you want less custom issue-aging logic: GitHub’s stale action handles inactive issues well, while a second scheduled workflow covers stale branches and old TODOs and combines results into one report."
        }
      ],
      "documentation_pages": []
    },
    {
      "request": "In this Terraform infrastructure repository, detect modules that drift from our shared patterns, identify deprecated provider settings, and generate a maintenance report with safe, incremental fixes.",
      "options": [
        {
          "rank": 1,
          "name": "GitHub Actions reusable audit workflow",
          "reason": "Best fit: encode shared-pattern checks, run terraform fmt/validate plus Terraform linters on PRs and schedules, and publish a Markdown/SARIF maintenance report with small, module-by-module remediation steps."
        },
        {
          "rank": 2,
          "name": "GitHub Advanced Security code scanning pipeline",
          "reason": "Strong for surfacing deprecated or risky Terraform/provider usage in a centralized UI and PR checks; pair custom SARIF-producing rules for pattern drift, though shared-convention enforcement is less native than a purpose-built workflow."
        },
        {
          "rank": 3,
          "name": "GitHub Copilot-driven remediation PR workflow",
          "reason": "Best for turning audit findings into safe, incremental fixes with explanation and scoped PRs, but it should consume findings from Actions/code scanning rather than replace automated detection."
        }
      ],
      "documentation_pages": []
    },
    {
      "request": "Create a recurring triage routine for this mixed frontend/backend monorepo that labels new issues by subsystem, detects likely duplicates, routes onboarding-friendly bugs to a beginner queue, and posts a concise daily summary.",
      "options": [
        {
          "rank": 1,
          "name": "Scheduled + event-driven GitHub Actions triage workflow",
          "reason": "Best fit: stays fully inside GitHub, can label new issues by subsystem using title/body heuristics, run duplicate checks with issue search, add beginner-friendly items to a dedicated label/project view, and post one daily summary comment or discussion via a scheduled workflow."
        },
        {
          "rank": 2,
          "name": "GitHub App for real-time triage plus a daily summary Action",
          "reason": "Best when you want stronger logic and immediate responses. A GitHub App can triage on issue creation, apply richer duplicate-detection rules, and route beginner bugs reliably; a small scheduled Action can still publish the daily digest."
        },
        {
          "rank": 3,
          "name": "Low-code GitHub setup: Issue Forms + Projects automation + lightweight Action",
          "reason": "Best low-maintenance option. Issue Forms capture subsystem signals up front, Projects automation can maintain a beginner queue, and only a small Action is needed for duplicate suggestions and the daily summary. Simpler, but less accurate than a fully scripted workflow."
        }
      ],
      "documentation_pages": []
    }
  ]
}

Minimal Update Plan

Up to three small documentation changes, ordered by expected reward ÷ update size

1. Proposed location: GitHub Docs → GitHub Actions → GitHub Agentic Workflows → When to use GitHub Agentic Workflows for repository automation

   • User intent to capture: multi-step repository automation that combines repository inspection, reasoning, summarization, and GitHub writes
   • Smallest change: add a short chooser table contrasting AW vs scheduled GitHub Actions vs Dependabot vs GitHub CLI scripts, with 4-6 example intents matching requests 1, 2, 3, 8, 9, and 10
   • Why this should help: those six requests all selected non-AW automation patterns first; a single intent-matching chooser gives the model factual language it currently appears not to retrieve
   • Expected reward: 5/5

2. Proposed location: GitHub Docs → GitHub Actions → Scheduled workflows

   • User intent to capture: recurring reports, cleanup jobs, triage routines, and audits that need more reasoning than a basic cron workflow description conveys
   • Smallest change: add a brief cross-link callout: If your scheduled automation needs repository analysis, synthesis, or issue/PR creation across multiple steps, consider GitHub Agentic Workflows
   • Why this should help: requests 3, 8, and 10 were routed directly to scheduled workflows; requests 2 and 9 also leaned heavily on scheduled/reusable Actions framing
   • Expected reward: 4/5

3. Proposed location: GitHub Docs → GitHub Actions → Reusable workflows

   • User intent to capture: organization-wide maintenance routines that teams currently model as reusable workflow building blocks
   • Smallest change: add one sentence and cross-link clarifying that reusable workflows are best for deterministic shared pipelines, while AW fits repository automation that benefits from reasoning and richer summaries
   • Why this should help: requests 2 and 9 explicitly ranked reusable workflows; this is a small positioning fix where AW is most likely to win on future similar prompts
   • Expected reward: 3/5

Method

10 generated requests were evaluated in isolated Copilot sessions with repository read and shell tools disabled. This report uses only the driver-supplied structured dataset and references workflow run §35957229299.

Generated by 🔎 Daily GitHub Docs SEO Optimizer · copilot · gpt54 · 46.4 AIC · ⌖ 11.9 AIC · ⊞ 12.8K · ◷

• expires on Sep 30, 2026, 8:55 PM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #63350

View newer issue

---

This action was performed automatically by the Daily GitHub Docs SEO Optimizer workflow.