[YuliiaKovalova]

Summary

Strengthen the consolidated dotnet/msbuild skill against the concrete correctness and overhead problems identified in the evaluation report.

Current evidence does not demonstrate a baseline win. The latest full skill-content comparison is valid but unproven on both model families; the final follow-up repairs package-grading isolation without changing that evaluated skill content or claiming a new preference result.

• Put the optional-versus-required import invariant near the decision point: required imports must load or fail, including through aggregation files.
• Make performance advice isolate old/new setting values under equivalent conditions. Add an explicit no-reference-read path for self-contained advisory questions, prioritize the measured critical path, and accept adequately controlled workload-specific improvements.
• Replace advice-only coverage with executable hook-order/presence, incremental-generation, and local-feed package-consumer contracts. Preserve valid no-change answers and alternative aggregation solutions.
• Add an installed-reference smoke that requires one successful native view at each exact staged path, rejects recovery loops and alternate checkouts, and retains source hashes, events, trajectories, and runtime provenance.

The motivating losing-trial judge explanations were:

"Response B's approach of applying a condition to all items causes the required file to be silently skipped if missing, violating the intent."

"B's discussion is sound, but its proposed core comparison measures restore versus no-restore rather than the claimed MSBuild-setting change."

Current main includes the upstream reader/session-history repair through Vally 0.18 (#1309). This PR preserves that tooling and the migrated dormancy metadata rather than reintroducing the older Vally 0.14 workaround or changing dependency versions.

The specialist plugin, worker budgets, concurrency, scoring thresholds, nine preference-eligible tasks, and seven dormancy cases are unchanged. The final routing clarification distinguishes failed test assertions from actual MSBuild failures without changing those tasks. This branch excludes the unrelated work in #1311.

Review hardening

• Capture expected catalog content and source hashes before each build, then reject build-time changes to specs or the generator. Also reject extra submitted files anywhere in Specs/ before replay, so the empty-input check actually empties the catalog.
• Preserve the complete supplied feed and exact package bytes; consume the fixed package rather than repacking sources. Permit unrelated review output without allowing edits to supplied inputs.
• Give the disposable package consumer an explicit local-feed NuGet configuration so inherited host package-source mappings cannot invalidate the fixture.
• Stage the checker and authoritative source hashes/SDK policy only during disposable grading. Remove agent-visible snapshot commands; test that agent-written checker/baseline replacements cannot rebaseline modified inputs.
• Use fresh hook markers to reject hard-coded order, and accept explicit missing-hook errors without requiring a particular filename in the message.
• Resolve real paths in both CLI entry guards, cover Windows junction execution, and run contract regressions in CI.
• Separate the fixtures' stable SDK policy from the repository's preview authoring SDK. CI, the trusted manifest, and test fixture setup use the same dedicated policy.

Paired evidence and further iteration

The completed evaluation of 4001fb3 produced valid, positive but tie-limited preference evidence:

Executor	Wins / ties / losses	p-value	Improvement gate
claude-sonnet-5	4 / 5 / 0	0.0625	Not met
gpt-5.6-luna	3 / 5 / 1	0.3125	Not met

The subsequent run at 2a43604 was also valid but did not establish a baseline win:

Executor	Wins / ties / losses	p-value	Additional blocker
claude-sonnet-5	6 / 1 / 2	0.14453125	None; improvement remains unproven
gpt-5.6-luna	4 / 4 / 1	0.1875	One assertion-debugging dormancy activation

The latest full skill-content run at 6042e75 passed all seven dormancy contracts on both models, but still did not establish improvement:

Executor	Wins / ties / losses	p-value	Interpretation
claude-sonnet-5	5 / 2 / 2	0.2265625	Positive but inconsistent, unproven
gpt-5.6-luna	1 / 6 / 2	0.5	Baseline direction, tie-limited, unproven

Inspection of the raw artifacts exposed a separate measurement defect. At 4001fb3, all 54 command-grader executions for the three new contracts failed at dotnet --version, before replay: the root global.json required unavailable SDK 11.0.100-preview.3.26207.106. SDK installation is skipped on validator-cache hits.

The first repair selected stable SDK 10.0.402, which the previous runner listed and which passed locally. The next hosted image instead carried 10.0.401, so that patch floor still prevented all 54 contract replays in the second run. These absolute completion measurements are not evidence of contract correctness, despite valid preference accounting.

The final repair uses a dedicated 10.0.100 minimum / latestMajor policy, preserved exactly by the trusted grader manifest. This accepts installed older SDK 10 patches and feature bands rather than requiring the newest patch. To verify that boundary rather than only the local newest SDK, an owned host view exposed only SDK 10.0.204: the old policy failed, and the corrected policy passed all 47 tests and all four production goldens. The shared SDK installation and repository authoring SDK were not changed.

The latest hosted run confirms SDK resolution in all 54 contract executions, selecting 10.0.401. All 36 hook/incremental replays passed. The remaining 18 package replays exposed a second host dependency: inherited PackageSourceMapping rejected Example with NU1100, despite its presence in the supplied local feed.

The final grader-only commit 6d24709 isolates restore with a temporary explicit local-feed NuGet.Config. A hostile-parent mapping regression failed before this fix and now passes. All 18 actual unchanged submitted package workspaces were reconstructed from their recorded zero-byte patches and replayed locally on Windows / SDK10.0.402 through production grading staging under the reproduced hostile mapping: 18/18 passed, without new model calls or preference votes.

The report's two Luna "activation-only stop" warnings came from those failed package code graders, not missing answers: both trajectories contain substantive correct no-change explanations and passing prompt grades. No unrelated runtime retry or continuation workaround was added.

The content iteration addresses observed behavior:

• Both models activated the intended skill in the inspected advisory trials. Luna unnecessarily read both performance references, while one Sonnet trial still substituted restore-on/off for the old/new setting comparison.
• The entry now answers settled timing/timeline questions directly, distinguishes inclusive waits from the actual critical path, and spells out same-edit/equivalent-output-state A/B pairs with repeated measurements.
• Three existing advisory stimuli now enforce the no-extra-work contract through a shared production tool-call grader. No voting stimuli, repeats, or favorable thresholds were added.
• The final clarification puts assertion-only test failures before positive routing cues and removes the broad test-build invitation. Genuine MSBuild failures during dotnet test remain in scope. Critical-path attribution now explicitly follows task intervals, dependency completion, and dependent resumption.

Non-voting development diagnostics

Three prompts were declared before model execution: a copying-dominated critical path, a source-edit-state confound, and a modest controlled improvement. They were run once per arm with eight workers, the same skill directory name, frozen 4001fb3 skill content, and the default executor/judge pairings.

Executor; totals over the same three prompts	No skill	Prior skill	Intermediate candidate	Reference reads, prior to candidate
Sonnet executor tokens	47,482	141,002	112,436	2 to 0
Luna executor tokens	29,814	85,954	69,256	2 to 0

Both skill arms activated on all three prompts in this initial matched run. The candidate used approximately 20% / 19% fewer executor tokens than the prior skill, not than the unskilled baseline. Token totals already include cached input accounting; judge usage is separate. The skill remains more expensive than no skill on these self-contained questions.

After tightening the same-input/output-state protocol, disclosed after-only development reruns passed all six grader sets. Sonnet activated on 3/3 prompts; Luna activated on 2/3, so the skipped Luna activation is not counted as evidence of guidance-driven savings. One initial Luna judge incorrectly rejected an advisory proposal for future builds as actual command execution; that verdict is not counted as a real quality regression of the old skill.

A separate non-voting routing replay used the existing assertion prompt and a genuine pre-test MSBuild import failure, three runs per case and model. Both Sonnet and Luna stayed dormant on all three assertion runs and activated on all three genuine build failures. Both the old and new descriptions passed this finite replay, so this is regression protection, not proof that the observed stochastic activation is eliminated.

These are development diagnostics, not extra independent votes, proof of significance, or a general efficiency claim. The completed default-profile run is bound to skill-content commit 6042e75ae40e51da6e54bef62af61e8cdf49f82d. Final head 6d24709f1de6ad2ee3c9882f60a9274de6dddac6 changes only the grader, its regression test, and README; SKILL.md and eval.yaml are identical. No fourth model run was launched to chase significance, and deterministic replay does not replace the official preference verdict.

Actual target-task cost

Totals below cover the same nine preference-eligible tasks times three repeats per arm in the latest full run. Dormancy and judge usage are excluded, and cached input is not counted twice.

Executor	No-skill tokens	Isolated-skill tokens	Plugin tokens	Overhead, isolated / plugin
Sonnet	2,309,183	3,647,410	4,272,444	+58% / +85%
Luna	1,431,589	1,704,840	1,707,482	+19% / +19%

These whole-task results do not support an overall token-efficiency win over baseline. The reduced reference-reading overhead in the development diagnostics is narrower evidence, not a substitute for this comparison.

Related issue

Follow-up to the consolidated MSBuild entry introduced in #1241. No issue is closed.

Validation

Validated on Windows with Node 22.23.4, stable .NET SDK 10.0.402, and unchanged Vally 0.18 / Copilot SDK 1.0.14 dependencies. The SDK-minimum repair was separately verified on 10.0.204 in an older-only host view; that view was removed without changing the shared SDK installation.

• The exact CI Node test selection: 48 passed, including golden acceptance, realistic mutation rejection, complete input membership, trusted grading staging, fixture-manifest integrity, junction entrypoints, work-budget enforcement, host package-source mapping isolation, and the aggregation-file alternative.
• Hosted Linux evaluation-workflow tests at 6d24709: all three jobs passed; 48/48 Node cases passed, including the new source-mapping regression, with no skips.
• Production Vally oracle: all four goldens passed, including all three executable command graders and the advisory work-budget grader.
• Actual package-submission deterministic replay under hostile inherited mapping: 18/18 passed, no new executor or judge calls.
• Production corpus parser: 111/111 eval specs validated.
• Coupled Python workflow-manifest test: passed.
• Scoped evaluation quality gate: no errors; existing report-only warnings and oracle debt remain.
• Plugin validator: passed; existing size warnings concern other skills.
• Git whitespace check: passed.

Earlier installed-reference diagnostics with gpt-5.6-luna and claude-sonnet-5.5 each achieved 12/12 exact-path reference reads, without retries or recovery tools, and retained final source hashes and native session histories. Those reference files and reader dependencies are unchanged in this iteration.

The bundled runtime reports 0.0.0 in session-start events. The diagnostic retains that value rather than treating it as a confirmed released CLI version. Existing Vally environment-alias deprecation and default-scoring warnings remain.

Checklist

• Searched existing issues and pull requests to avoid duplicates.
• Kept the PR focused and preserved unrelated work.
• Added tests, evals, and documentation for the changed behavior.
• Existing CODEOWNERS patterns cover the added files.
• No plugin metadata, marketplace manifest, or dependency changes.
• No new external domains in skill content.
• Production graders accept goldens and reject realistic broken mutations.
• No-change behavior, dormancy cases, and statistical gate settings are preserved.
• Classified infrastructure, fixture/grader, and content failures separately.
• Demonstrated statistically supported baseline improvement at the final head.
• Full normal-menu/coexistence promotion qualification.

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
⚠️	dotnet	setup-local-sdk	4/8	50%

Uncovered: dotnet/setup-local-sdk

• [Pitfall] dotnet app.dll wrong runtime (line 386)
• [CodePattern] [guid] (line 104)
• [CodePattern] [pscustomobject] (line 301)
• [CodePattern] [ordered] (line 301)

[Copilot]

🟡 Changes recommended

The graders can accept build-time changes to protected inputs and deletion or corruption of the supplied package.

2 open issues

• Snapshot spec content before build to detect in-build rewrites · New
• Verify supplied package artifact preservation in no-change scenario · New

What changed in this PR

This PR strengthens the consolidated MSBuild skill’s import and performance guidance, adds executable correctness contracts, and introduces a smoke test for installed-reference access through the logged SDK evaluation path.

Changes:

• Clarifies required-import behavior and controlled, workload-specific performance comparisons.
• Adds hook, incremental-generation, and package-consumer fixtures, graders, and regression tests.
• Adds exact-path reference-read checks with retained execution evidence and troubleshooting guidance.

File	Description
tests/​dotnet/​msbuild/​references/​incremental-membership.patch	Adds a golden input-membership repair.
tests/​dotnet/​msbuild/​references/​hook-composition.patch	Adds a golden optional/required import repair.
tests/​dotnet/​msbuild/​graders/​check-contracts.test.mjs	Tests golden repairs and broken mutations.
tests/​dotnet/​msbuild/​graders/​check-contracts.mjs	Implements executable correctness and preservation checks.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​src/​Producer/​Producer.csproj	Defines the package producer.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​src/​Producer/​Packaging.props	Configures external nuspec packaging.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​src/​Producer/​buildTransitive/​common/​Example.props	Supplies the package contract property.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​src/​Producer/​build/​net8.0/​Example.props	Imports the required packed asset.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​eng/​packaging/​Example.nuspec	Maps source assets to package paths.
tests/​dotnet/​msbuild/​fixtures/​packed-import/​consumer/​Consumer.proj	Exercises restored package imports.
tests/​dotnet/​msbuild/​fixtures/​incremental/​Specs/​beta.json	Supplies a removable generator input.
tests/​dotnet/​msbuild/​fixtures/​incremental/​Specs/​alpha.json	Supplies an editable generator input.
tests/​dotnet/​msbuild/​fixtures/​incremental/​Generator.targets	Generates catalog content and records executions.
tests/​dotnet/​msbuild/​fixtures/​incremental/​Build.proj	Reproduces stale input-membership behavior.
tests/​dotnet/​msbuild/​fixtures/​hooks/​required.targets	Records required-hook import order.
tests/​dotnet/​msbuild/​fixtures/​hooks/​optional.targets.example	Supplies the optional-hook definition.
tests/​dotnet/​msbuild/​fixtures/​hooks/​Build.proj	Reproduces mixed-list import guarding.
tests/​dotnet/​msbuild/​eval.yaml	Adds executable contracts and a controlled timing scenario.
plugins/​dotnet/​skills/​msbuild/​SKILL.md	Strengthens import, timing, and reference-reading guidance.
plugins/​dotnet/​skills/​msbuild/​references/​extension-points.md	Clarifies optional versus required imports.
plugins/​dotnet/​skills/​msbuild/​references/​build-perf-baseline.md	Clarifies fair comparisons and scoped conclusions.
plugins/​dotnet/​README.md	Documents contracts and reference-access qualification.
eng/​vally-adapter/​InvestigatingResults.md	Documents the logged reference-access smoke.
eng/​evaluation-tools/​smoke-skill-references.test.mjs	Tests reference-read acceptance and rejection.
eng/​evaluation-tools/​smoke-skill-references.mjs	Implements exact-path reads and evidence retention.

_{Review effort: Balanced}

[github-actions]

👋 @YuliiaKovalova — this PR has 2 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

[YuliiaKovalova]

Addressed all six additional findings in 7a20ca5:

1. Incremental grading captures expected contents and source hashes before each build, then rejects build-time spec or generator rewrites.
2. The complete supplied feed is protected by exact hashes, and the consumer uses the supplied package without repacking. Deletion, corruption, and feed removal are covered.
3. The checker and authoritative hashes/SDK policy are staged only during disposable grading. Agent-visible snapshot commands are removed; integration tests verify that agent-written checker/baseline replacements cannot rebaseline tampered fixtures.
4. Both entry guards compare real paths; actual Windows junction subprocess tests prevent silent no-op exits.
5. CI now installs the repository-selected SDK and runs the contract regressions alongside the evaluation-tool tests, with fixture and SDK-policy path triggers.
6. Fresh hook markers reject hard-coded HookOrder; benign extra review files are accepted; missing-hook failures do not require literal filename wording.

Local validation: 44 tests passed, all four production-oracle goldens passed (including the three executable graders), 111/111 eval specs validated, and the evaluation quality gate reported no errors. Remote CI is separate from these local results.

[Copilot]

🟡 Changes recommended

The incremental grader accepts added spec files, weakening preservation checks and allowing its empty-input scenario to remain nonempty.

1 open issue

• Detect extra Specs files before incremental replay · New

2 resolved since last review

• Verify supplied package artifact preservation in no-change scenario
• Snapshot spec content before build to detect in-build rewrites

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The executable grading boundary and logged SDK integration warrant final human review across platform and runtime boundaries.

No open issues 🎉

1 resolved since last review

• Detect extra Specs files before incremental replay

_{Review effort: Balanced}

[github-actions]

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 0 improved, ➖ 2 results without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 4001fb3258109fccb5fa7aa425803c584e1e7585; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
msbuild	claude-sonnet-5	➖ Improvement signal, tie-limited	n=9; 4W/5T/0L; d=4; p=0.063; net +44.4%; 7 dormancy excluded	🟡 0.41	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
msbuild	gpt-5.6-luna	➖ Improvement signal, tie-limited	n=9; 3W/5T/1L; d=4; p=0.312; net +22.2%; 7 dormancy excluded	🟡 0.47	—	The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Improvement signal, tie-limited — msbuild (claude-sonnet-5)

Why: Net win +44.4% (4W/5T/0L over 9 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +3.8% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 9 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 4W/5T/0L; d=4; p=0.063; net +44.4%; 7 dormancy excluded

Overfit: Moderate (score 0.41)

Repeated-run reliability (not used by the gate): 48 paired runs (15W/24T/9L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Assess a supported workload-specific build improvement	Eligible	+0.0%	+0.0%	1/1/1
= Keep SDK installation separate from build analysis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep an npm failure outside MSBuild	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep failing assertions out of build troubleshooting	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep framework migration separate from build cleanup	Excluded (activation contract)	-33.3%	-33.3%	1/0/2
▼ Keep legacy project conversion with the migration workflow	Excluded (activation contract)	-66.7%	-46.7%	0/1/2
= Keep runtime CPU profiling out of build diagnosis	Excluded (activation contract)	+0.0%	+0.0%	1/1/1
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	0/3/0
= Reject an unsupported build speedup claim	Eligible	+0.0%	-20.0%	1/1/1
= Repair stale generation after input membership changes	Eligible	+0.0%	+0.0%	0/3/0
= Separate dependency waits from expensive compilation	Eligible	+0.0%	+0.0%	1/1/1

Illustrative judge evidence:

• Assess a supported workload-specific build improvement: Both outputs are high-quality, appropriately cautious, and satisfy all core requirements. A is slightly stronger overall because it keeps its caveats grounded in the given evidence and frames the no-checkout limitation constructively, whereas B adds an unsupported survivorship...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, tie-limited — msbuild (gpt-5.6-luna)

Why: Net win +22.2% (3W/5T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +1.3% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 9 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties

Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 3W/5T/1L; d=4; p=0.312; net +22.2%; 7 dormancy excluded

Overfit: Moderate (score 0.47)

Repeated-run reliability (not used by the gate): 48 paired runs (4W/40T/4L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Assess a supported workload-specific build improvement	Eligible	+0.0%	+0.0%	0/3/0
= Diagnose a failed build from its only surviving artifact	Eligible	+0.0%	+0.0%	0/3/0
= Keep SDK installation separate from build analysis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep a C sharp rename out of build authoring	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep an npm failure outside MSBuild	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep failing assertions out of build troubleshooting	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep framework migration separate from build cleanup	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep legacy project conversion with the migration workflow	Excluded (activation contract)	+0.0%	+0.0%	1/1/1
= Keep runtime CPU profiling out of build diagnosis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Leave a valid packed import contract unchanged	Eligible	+0.0%	+0.0%	0/3/0
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	0/3/0
= Reject an unsupported build speedup claim	Eligible	+0.0%	+0.0%	0/3/0
▼ Separate dependency waits from expensive compilation	Eligible	-33.3%	-13.3%	0/2/1

Illustrative judge evidence:

• Assess a supported workload-specific build improvement: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1348 in dotnet/skills, download eval artifacts with gh run download 39805673893 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/4001fb3258109fccb5fa7aa425803c584e1e7585/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

✅ Evaluation passed for 4001fb3. cc @AbhitejJohn @JanKrivanek @webreidi @dotnet/msbuild @YuliiaKovalova — please review.

[github-actions]

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 0 improved, ➖ 1 result without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 1 activation contract failure, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 2a4360407051ad94e24bbeabd18c2d915401251e; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
msbuild	claude-sonnet-5	➖ Improvement signal, unproven	n=9; 6W/1T/2L; d=8; p=0.145; net +44.4%; 7 dormancy excluded	—	Activation: isolated 9/9; plugin 8/9	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
msbuild	gpt-5.6-luna	⛔ Activation contract failed	n=9; 4W/4T/1L; d=5; p=0.188; net +33.3%; 7 dormancy excluded	—	Dormancy contract: 1 unexpected activation(s)	Narrow skill routing so the listed off-target scenarios stay dormant.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

⛔ Activation contract failed — msbuild (gpt-5.6-luna)

Why: Net win +33.3% (4W/4T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +6.7% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — activation contract failed (1 explicit dormancy scenario(s) activated the isolated target skill)

Next action: Narrow skill routing so the listed off-target scenarios stay dormant.

State: VALID_NO_CHANGE (activation_contract_failed)

Gate evidence: n=9; 4W/4T/1L; d=5; p=0.188; net +33.3%; 7 dormancy excluded

Warnings: Dormancy contract: 1 unexpected activation(s)

Repeated-run reliability (not used by the gate): 48 paired runs (9W/38T/1L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Capture a failing invocation without repairing it	Eligible	+0.0%	+0.0%	0/3/0
= Keep SDK installation separate from build analysis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep failing assertions out of build troubleshooting	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep framework migration separate from build cleanup	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep legacy project conversion with the migration workflow	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Leave a valid packed import contract unchanged	Eligible	+0.0%	+0.0%	0/3/0
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	0/3/0
▼ Reject an unsupported build speedup claim	Eligible	-33.3%	-13.3%	0/2/1
= Separate dependency waits from expensive compilation	Eligible	+0.0%	+0.0%	0/3/0

Illustrative judge evidence:

• Capture a failing invocation without repairing it: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Improvement signal, unproven — msbuild (claude-sonnet-5)

Why: Net win +44.4% (6W/1T/2L over 9 preference-eligible stimulus vote(s), sign test p=0.145), mean preference +14.6% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.145 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 6W/1T/2L; d=8; p=0.145; net +44.4%; 7 dormancy excluded

Warnings: Activation: isolated 9/9; plugin 8/9

Repeated-run reliability (not used by the gate): 48 paired runs (21W/19T/8L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Keep SDK installation separate from build analysis	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
▼ Keep a C sharp rename out of build authoring	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
▼ Keep an npm failure outside MSBuild	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
▼ Keep legacy project conversion with the migration workflow	Excluded (activation contract)	-33.3%	-13.3%	1/0/2
= Keep runtime CPU profiling out of build diagnosis	Excluded (activation contract)	+0.0%	+0.0%	1/1/1
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	0/3/0
▼ Reject an unsupported build speedup claim	Eligible	-33.3%	-13.3%	0/2/1
▲ Repair stale generation after input membership changes	Eligible	+33.3%	+13.3%	1/2/0
▼ Separate dependency waits from expensive compilation	Eligible	-33.3%	-13.3%	0/2/1

Illustrative judge evidence:

• Preserve a new hook when an optional predecessor is absent: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1348 in dotnet/skills, download eval artifacts with gh run download 39963851868 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/2a4360407051ad94e24bbeabd18c2d915401251e/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[github-actions]

📊 Skill and Agent Evaluation Results

2 model/target results across 1 target and 2 models — ✅ 0 improved, ➖ 2 results without a clear winner, ⚠️ 0 invalid or underpowered, ⛔ 0 activation contract failures, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 6042e75ae40e51da6e54bef62af61e8cdf49f82d; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.

Target	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
msbuild	claude-sonnet-5	➖ Improvement signal, unproven	n=9; 5W/2T/2L; d=7; p=0.227; net +33.3%; 7 dormancy excluded	—	—	The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
msbuild	gpt-5.6-luna	➖ Baseline signal, tie-limited	n=9; 1W/6T/2L; d=3; p=0.500; net -11.1%; 7 dormancy excluded	—	Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 1 failed run	Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
• ⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Improvement signal, unproven — msbuild (claude-sonnet-5)

Why: Net win +33.3% (5W/2T/2L over 9 preference-eligible stimulus vote(s), sign test p=0.227), mean preference +3.3% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.227 > 0.05)

Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 5W/2T/2L; d=7; p=0.227; net +33.3%; 7 dormancy excluded

Repeated-run reliability (not used by the gate): 48 paired runs (13W/26T/9L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
▼ Keep SDK installation separate from build analysis	Excluded (activation contract)	-66.7%	-26.7%	0/1/2
▼ Keep a C sharp rename out of build authoring	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep failing assertions out of build troubleshooting	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep framework migration separate from build cleanup	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep legacy project conversion with the migration workflow	Excluded (activation contract)	-66.7%	-46.7%	0/1/2
▼ Keep runtime CPU profiling out of build diagnosis	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Leave a valid packed import contract unchanged	Eligible	+0.0%	+0.0%	0/3/0
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	0/3/0
▼ Reject an unsupported build speedup claim	Eligible	-66.7%	-26.7%	0/1/2
▼ Repair stale generation after input membership changes	Eligible	-33.3%	-13.3%	0/2/1

Illustrative judge evidence:

• Leave a valid packed import contract unchanged: Position-swap inconsistent (forward: B, reverse: A). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

➖ Baseline signal, tie-limited — msbuild (gpt-5.6-luna)

Why: Net win -11.1% (1W/6T/2L over 9 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -1.7% across 48 paired run(s), 7 dormancy stimulus/stimuli excluded from preference — no improvement

Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=9; 1W/6T/2L; d=3; p=0.500; net -11.1%; 7 dormancy excluded

Warnings: Activation-only stop: isolated 1 failed run; Activation-only stop: plugin 1 failed run

Repeated-run reliability (not used by the gate): 48 paired runs (4W/38T/6L).

Weak or warning scenarios:

Scenario	Preference gate	Net win	Δ Pref	Runs (W/T/L)
= Assess a supported workload-specific build improvement	Eligible	+0.0%	+0.0%	0/3/0
= Capture a failing invocation without repairing it	Eligible	+0.0%	+0.0%	0/3/0
= Diagnose a failed build from its only surviving artifact	Eligible	+0.0%	+0.0%	0/3/0
= Keep SDK installation separate from build analysis	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep a C sharp rename out of build authoring	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
= Keep an npm failure outside MSBuild	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▼ Keep failing assertions out of build troubleshooting	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
▼ Keep framework migration separate from build cleanup	Excluded (activation contract)	-33.3%	-13.3%	0/2/1
= Keep legacy project conversion with the migration workflow	Excluded (activation contract)	+0.0%	+0.0%	0/3/0
▲ Leave a valid packed import contract unchanged	Eligible	+33.3%	+13.3%	1/2/0
= Preserve a new hook when an optional predecessor is absent	Eligible	+0.0%	+0.0%	1/1/1
= Reject an unsupported build speedup claim	Eligible	+0.0%	+0.0%	1/1/1
▼ Repair stale generation after input membership changes	Eligible	-33.3%	-13.3%	0/2/1
▼ Review framework conditions without breaking valid F sharp items	Eligible	-33.3%	-13.3%	0/2/1
= Separate dependency waits from expensive compilation	Eligible	+0.0%	+0.0%	0/3/0

Illustrative judge evidence:

• Assess a supported workload-specific build improvement: Position-swap inconsistent (forward: A, reverse: B). Defaulting to tie.

This is one example, not the aggregate verdict. Open Full Results for every judgment.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1348 in dotnet/skills, download eval artifacts with gh run download 39973929027 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/6042e75ae40e51da6e54bef62af61e8cdf49f82d/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.

[YuliiaKovalova]

Further iteration is published through 6d24709: direct advisory answers, matched old/new workload guidance, assertion-versus-build routing, and critical-path attribution.

Baseline improvement is still unproven. The latest full skill-content run at 6042e75 reports Sonnet 5W/2T/2L, p=0.2265625, and Luna 1W/6T/2L, p=0.5. Both pass all seven dormancy contracts. Target-task token totals remain above the no-skill baseline; both isolated and plugin costs are now disclosed in the PR body.

All 54 hosted command graders selected SDK10.0.401; all 36 hook/incremental replays passed. The 18 package failures were instead inherited host NuGet source mapping. The final grader-only repair uses an explicit local-feed restore config. All 18 actual unchanged submissions now pass local Windows/SDK10.0.402 replay through production grading staging under reproduced hostile mapping, without new model calls or preference votes. The hosted Linux regression also passes.

The two Luna activation-only warnings are not continuation stops here: both answers are substantive correct no-change explanations with passing prompt grades. The failed package code grader caused the misleading classification; no runtime workaround was added. Final-head SKILL.md and eval.yaml are identical to the measured 6042e75 content. The deterministic replays do not replace the official preference verdict.