[AbhitejJohn]

Summary

Why

Official evaluation CI produces detailed results, but it does not provide one recurring view of whether the evaluation suite remains reliable, fast, and able to differentiate treatment from control. This automation is intended to keep evals well groomed and refreshed so they continue to provide useful differentiation quickly, while making systemic problems and grooming priorities visible.

What changes

This adds a Friday 17:00 UTC workflow that:

• deterministically aggregates official retained evaluation data and workflow-run evidence
• separates reliability, treatment pass rate, proven differentiation, credible preference loss, flakiness, and stable model sensitivity
• reports duration and token telemetry without estimating unavailable AI Credits or dollar spend
• compares the current week with the prior window and prior weekly findings
• creates or reuses one bot-owned Weekly Evaluation Health issue and posts one deduplicated report per ISO week
• restricts the agent to evidence synthesis while a separate least-privilege publication job validates and posts the report

Expected outcome

Each week, maintainers should receive a concise executive view, evidence-based recommendations, recurring and new trends, and plugin-by-plugin skill statistics. This should make it easier to identify unreliable, underpowered, slow, expensive, stale, or non-differentiating evals; prioritize the right maintenance work; and verify whether treatment continues to outperform control reliably.

flowchart LR
    A[Official evaluation CI] --> B[Deterministic weekly aggregation]
    B --> C[Evidence-based synthesis]
    C --> D[Weekly Evaluation Health issue]
    D --> E[Eval grooming actions]

Loading

Related issue

N/A

Validation

• node --test eng/evaluation/weekly-eval-insights.test.js — passed, 7/7 tests.
• python eng/agentic-workflows/validate_agentic_workflows.py — passed.
• gh aw compile weekly-eval-insights --strict --validate --schedule-seed dotnet/skills --action-mode action --action-tag 2fbab69bfca02bebd76cd0fc43f2d12acfed994f --approve — passed and regenerated the checked-in lock workflow.
• actionlint 1.7.7 -shellcheck= -pyflakes= against hand-authored workflow YAML — passed.
• git diff --check — passed.
• Live retained-data smoke test — completed successfully across 553 official workflow runs, 1,424 expected evaluation results, and 17 plugins.

Checklist

• I searched existing issues and pull requests to avoid duplicates.
• I kept this pull request focused and avoided unrelated refactors.
• I added or updated tests, evals, or documentation when changing skill or agent behavior.
• I updated CODEOWNERS when adding or moving owned content.
• I updated all marketplace manifests when plugin metadata changed.
• I updated eng/known-domains.txt for any new external domains referenced by skill content.

No CODEOWNERS, marketplace manifest, or known-domain update is required because existing ownership applies and this change adds no plugin metadata or external domain.

Evaluation changes only

For every eval-related change, including a new eval:

This changes evaluation reporting automation, not eval scenarios, graders, routing, or skill behavior. The production data path was exercised through the live retained-data smoke test above.

• Scenarios are necessary, distinct, and use natural prompts.
• Graders cover the full result, accept the golden result, and reject a realistic mutation.
• No-op, dormancy, and statistical power are covered where needed.
• I ran the applicable production evaluation path and recorded the result above.
• If this fixes a failed eval, I classified the failure before editing skill content.
• If this broadly changes routing or behavior, I checked separate model-family evidence.

[Copilot]

🟡 Changes recommended

Unresolved evidence-gating, deduplication, and aggregation errors can produce misleading weekly health reports.

6 open issues

• Restrict metrics to the official evaluation population · New
• Require evaluated and passed activation evidence · New
• Deduplicate judge views using stable execution identity · New
• Calculate plugin signals separately for each skill · New
• Require pass evidence for every metric observation · New
• Exclude unfinished runs from duration averages · New

What changed in this PR

Adds weekly evaluation-health reporting that combines retained CI results with workflow evidence to guide evaluation maintenance.

Changes:

• Aggregates reliability, differentiation, pass rates, runtime, and token telemetry across current and prior weekly windows.
• Adds a Friday reporting workflow with agent synthesis and a separate, validated publication job.
• Adds tests for metric classification, resource deduplication, model sensitivity, and ISO week keys.

File	Description
eng/​evaluation/​weekly-eval-insights.test.js	Tests aggregation behavior and reporting boundaries.
eng/​evaluation/​weekly-eval-insights.js	Builds deterministic weekly evidence and health metrics.
.github/​workflows/​weekly-eval-insights.md	Defines evidence collection, report synthesis, and deduplicated issue publication.
.github/​workflows/​weekly-eval-insights.lock.yml	Provides the compiled GitHub Actions workflow.

_{Review effort: Balanced}

[github-actions]

👋 @AbhitejJohn — this PR has 6 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[Copilot]

🟡 Changes recommended

Unresolved aggregation and activation-gate issues can produce misleading health metrics, and the new regression suite is not wired into CI.

5 open issues

• Dashboard omits native target activation failures · New
• Resource deduplication keeps incomplete judge telemetry · New
• Reliability uses retained results instead of expected evaluations · New
• Resource deduplication drops judge-dependent pass evidence · New
• Weekly aggregation regression tests are not wired into CI · New

6 resolved since last review

• Deduplicate judge views using stable execution identity
• Require evaluated and passed activation evidence
• Restrict metrics to the official evaluation population
• Exclude unfinished runs from duration averages
• Require pass evidence for every metric observation
• Calculate plugin signals separately for each skill

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The new multi-job publication path needs human verification, with unresolved report-validation and verdict-classification issues.

2 open issues

• Use collection-time week metadata for publication validation · New
• Preserve native completion regressions in Skill Value reporting · New

5 resolved since last review

• Resource deduplication drops judge-dependent pass evidence
• Reliability uses retained results instead of expected evaluations
• Resource deduplication keeps incomplete judge telemetry
• Dashboard omits native target activation failures
• Weekly aggregation regression tests are not wired into CI

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Revision comparability remains unresolved, and the new agent-to-publication path still needs end-to-end human validation.

1 open issue

• This grouping ignores the evaluated revision. If a skill changes from no-change on commit A to… · New

2 resolved since last review

• Preserve native completion regressions in Skill Value reporting
• Use collection-time week metadata for publication validation

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Invalid adapter results are mislabeled as unverified activation evidence in skill-detail statuses.

1 open issue

• Invalid adapter results reach the activation-unverified branch before the invalid-measurement… · New

1 resolved since last review

• This grouping ignores the evaluated revision. If a skill changes from no-change on commit A to…

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The new multi-job agentic workflow and automated issue-publication path need final human validation beyond the deterministic aggregation checks.

No open issues 🎉

1 resolved since last review

• Invalid adapter results reach the activation-unverified branch before the invalid-measurement…

_{Review effort: Balanced}

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate babca8a08993f270e507ce90ba9ce0bcdb8fc991 to retry this exact commit.

64 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

❌ Evaluation did not complete successfully (the evaluate job reported failure). Check the workflow run logs, then comment /evaluate babca8a08993f270e507ce90ba9ce0bcdb8fc991 to retry this exact commit.

64 partial result file(s) were preserved for diagnosis but were not consolidated because the full matrix did not complete.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.