[Jiaqi Zhang (JiaqiZhang-Dev)]

Summary

• Keep perf-evaluation.yml as the only one evaluation pipeline with core/full options and remove the offline and online evaluation pipeline.
• Consolidate the basic and perf datasets into one perf/<scenario>.jsonl, corpus with structured testcase value (for example, 001003-version-required-to-optional) and core tags.
• Remove some duplicate or low-value cases.

Current evaluation cases

Category counts use core/full; full includes the core cases.

Scenario	Categories (core/full)	Core	Full
apispec	access 3/5; version 3/6; workflow 3/8; validation 5/7; model 2/3; layout 2/2; review 2/5	20	36
authoring	version 9/9; ARM 10/10; LRO 3/3; decorate 3/3; warning 1/1	26	26
dotnet	generation 3/6; compatibility 2/2; customize 2/2	7	10
general	access 2/4; docs 1/2; release 2/4; design 1/2; validation 1/1; version 3/4; workflow 1/1; logging 1/1	12	19
go	customize 7/7; compatibility 7/7; review 3/3; generation 2/2; validation 3/3; release 6/9; dependencies 1/1; deprecation 2/2	31	34
javascript	dependencies 6/6; testing 4/5; customize 4/5; generation 3/4	17	20
mcpserver	authentication 4/5; resources 4/5; telemetry 1/3; onboarding 5/9; diagnostics 2/5; distribution 3/3; contribution 2/8	21	38
onboarding	release 2/3; workflow 4/4	6	7
python	generation 4/5; customize 1/2; testing 3/6; validation 2/4; runtime 2/4; release 2/3; compliance 2/2	16	26
releasesupport	release 1/2; compliance 1/1	2	3
sdkagent	access 1/1; diagnostics 0/1	1	2
typespec	version 15/40; ARM 7/19; LRO 5/12; decorate 4/17; validation 3/9; model 6/12; example 3/7; sdk 4/8; workflow 1/9; tooling 1/5; migration 0/2	49	140
Total		208	361

Recommended practice

Change type	Recommended evaluation
Scenario-specific	Run full only for the affected scenarios.
Agent-wide or shared behavior	Run core across all scenarios.
High-risk cross-cutting change or major production release	Run full across all scenarios.

Validation

• python -m pytest unit_tests -q — 42 passed
• python -m dataset.validate evaluation_datasets\perf --require-reviewed — 361 cases across 12 files
• git diff --check

Relates to Azure/azure-sdk-pr#3273

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
71 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

⚠️ Skill Eval coverage missing

Grace period: This warning will become a build blocker by November 9, 2026.
For now, missing Eval coverage is warning-only; you may proceed if all other required checks pass.
Add or fix the affected Evals before the deadline.

Coverage/lint build: #42995397363 · Build result: success · Attempt: 1
Skill revision: 6dcd16a0a65a4fea527a1ec8d2718ccdb9af5a15

Skill	Coverage	Associated Eval
sdk-ai-bot-eval-dataset	Missing	—
sdk-ai-bot-run-evaluation	Missing	—

sdk-ai-bot-eval-dataset: No associated Eval with a named, nonempty stimulus was found in the skill folder.

sdk-ai-bot-run-evaluation: No associated Eval with a named, nonempty stimulus was found in the skill folder.

Eval-authoring guide · View Eval dashboard

Dashboard access requires corporate access/sign-in. It shows published historical results, not validation of this PR.

This checks associated Eval presence only. Semantic coverage of the skill's capabilities is not assessed.

Check run

Coverage warnings are nonblocking. This check does not run Evals or prove behavioral correctness. Existing lint failures remain blocking.

[Copilot]

🟡 Changes recommended

Valid staging rows can fail assertion checks, and full-only cases lose required empty tags during serialization, causing valid benchmark rows to be rejected.

2 open issues

• Preserve empty tags when serializing full-only cases · New
• Allow source titles in staging testcase fields · New

What changed in this PR

Unifies the Teams QA bot’s evaluation datasets into a single performance corpus, using structured case identifiers and core/full selection.

Changes:

• Consolidates basic and performance cases under perf/, with core tags identifying representative coverage.
• Adds scope selection to the evaluation runner and manual performance pipeline; removes separate offline and online pipelines.
• Updates dataset tooling, validation tests, documentation, and skills for the unified workflow.

File	Description
tools/​sdk-ai-bots/​perf-evaluation.yml	Adds manual core/full selection and SDK agent coverage.
tools/​sdk-ai-bots/​online-evaluation.yml	Removes the weekly online pipeline.
tools/​sdk-ai-bots/​offline-evaluation.yml	Removes the separate offline pipeline.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​unit_tests/​test_pure_logic.py	Updates schema, selection, promotion, and pipeline tests.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​README.md	Documents unified datasets and evaluation commands.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​perf/​sdkagent.jsonl	Adds migrated SDK agent cases with IDs and tags.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​perf/​releasesupport.jsonl	Assigns release-support case IDs and core tags.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​perf/​onboarding.jsonl	Combines onboarding cases with IDs and tags.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​perf/​javascript.jsonl	Combines JavaScript cases with IDs and tags.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​perf/​dotnet.jsonl	Consolidates .NET cases and marks core coverage.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​basic/​sdkagent.jsonl	Removes the separate SDK agent basic dataset.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​basic/​python.jsonl	Removes the separate Python basic dataset.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​basic/​onboarding.jsonl	Removes the separate onboarding basic dataset.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​basic/​mcpserver.jsonl	Removes the separate MCP server basic dataset.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​basic/​javascript.jsonl	Removes the separate JavaScript basic dataset.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​basic/​go.jsonl	Removes the separate Go basic dataset.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evaluation_datasets/​basic/​dotnet.jsonl	Removes the separate .NET basic dataset.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​evals_run.py	Validates and selects scope before evaluation.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​dataset/​upload.py	Targets the unified performance corpus.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​dataset/​schema.py	Adds structured IDs, tags, and scope validation.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​dataset/​review.py	Validates and sorts promoted performance cases.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​dataset/​online_snapshot.py	Removes online snapshot preparation.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​dataset/​curate.py	Removes basic datasets from curation lookups.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​dataset/​backfill_dedup_key.py	Limits backfill to performance and staging sets.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​dataset/​_storage.py	Updates the module description after snapshot removal.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-evaluation/​dataset/​__init__.py	Updates the documented module inventory.
tools/​sdk-ai-bots/​azure-sdk-qa-bot-agent/​DEVELOP.md	Documents pipeline scope and recommended coverage.
.github/​skills/​sdk-ai-bot-run-evaluation/​SKILL.md	Updates evaluation guidance for core/full runs.
.github/​skills/​sdk-ai-bot-run-evaluation/​references/​running-evaluations.md	Updates scope flags and single-case recipes.
.github/​skills/​sdk-ai-bot-eval-dataset/​SKILL.md	Updates dataset identity and preparation guidance.
.github/​skills/​sdk-ai-bot-eval-dataset/​references/​dataset-schema-and-workflows.md	Documents IDs, tags, and unified dataset workflows.

_{Review effort: Balanced}

[Jiaqi Zhang (JiaqiZhang-Dev)]

⚠️ Skill Eval coverage missing

Grace period: This warning will become a build blocker by November 9, 2026.
For now, missing Eval coverage is warning-only; you may proceed if all other required checks pass.
Add or fix the affected Evals before the deadline.

Coverage/lint build: #42995102034 · Build result: success · Attempt: 1 Skill revision: c2f002549f8aad44e963b37caaf96a251370c2f7

Skill Coverage Associated Eval
sdk-ai-bot-eval-dataset Missing —
sdk-ai-bot-run-evaluation Missing —
sdk-ai-bot-eval-dataset: No associated Eval with a named, nonempty stimulus was found in the skill folder.

sdk-ai-bot-run-evaluation: No associated Eval with a named, nonempty stimulus was found in the skill folder.

Eval-authoring guide · View Eval dashboard

Dashboard access requires corporate access/sign-in. It shows published historical results, not validation of this PR.

This checks associated Eval presence only. Semantic coverage of the skill's capabilities is not assessed.

Check run

Coverage warnings are nonblocking. This check does not run Evals or prove behavioral correctness. Existing lint failures remain blocking.

These are developer-only helper skills for the chatbot evaluation workflow. This PR only updates their guidance for the unified dataset and core/full options; it does not change routing or user-facing behavior. I don’t think dedicated skill evals are necessary, so we can accept this warning.