You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The shipped dnx command inherits the workspace's effective NuGet settings. When those feeds do not provide Microsoft.AITools.BinlogMcp, package resolution fails before the MCP server starts, and the assistant sees only a closed connection rather than the stderr diagnosis.
Add host-specific SessionStart hooks for Copilot CLI and Claude Code. A 5-second exact-match, prerelease-inclusive JSON package search runs in the session workspace, inside an 8-second launcher budget and 10-second host timeout.
Inspect package arrays and both top-level and per-source problems instead of relying on exit codes. Distinguish absence, feed problems, timeouts, unsupported probes, and missing dotnet; successful availability checks remain silent. Serialize only the documented assistant-visible payload, without raw feed details, credentials, SDK banners, or credential-provider warnings.
Isolate the Unix helper in its own Bash process group and terminate the group on timeout/exit. Native Windows Git Bash delegates to PowerShell process-tree cleanup. Before package search launches, the helper joins a process-lifetime kill-on-close Windows Job Object so orphaned providers are terminated on normal/early helper exit and launcher termination. Transfer captured hook input through per-invocation UTF-8 files rather than console-dependent stdin forwarding.
Preserve the existing MCP command and workspace NuGet configuration. Recovery guidance preserves enterprise feed policy. Search is explicitly not proof of MCP startup: dnx also applies source mapping and may use cached tools or unlisted versions.
The shared parser uses dotnet fsi, bundled with the .NET SDK, rather than requiring another runtime or JSON parser. Add focused regression coverage and a Windows/macOS/Linux CI matrix without modifying skill evaluations or validator launch overrides.
python eng/hooks/test_binlog_preflight.py -v: 15 cases on Windows with .NET SDK 10.0.402; 13 passed and the 2 native Unix cases were skipped. Coverage includes inherited <clear />/mirror settings, exact availability, structured problems despite exit code 0, source mapping, timeouts, missing tools, both host schemas, spaces and Unicode in workspace paths, Windows input encoding, the Windows cleanup race, and orphaned stream-holder cleanup after search exit and launcher timeout.
Focused CI matrix passed on Windows, macOS, and Linux. Native Unix timeout/early-exit regressions verify that the helper owns its process group, TERM-ignoring descendants stop, and an unrelated process survives. Windows stream-holder fixtures verify cleanup after the search parent exits, including normal helper exit and forced launcher timeout.
The structured F# runner passed 65 assertions.
Real Copilot CLI 1.0.97 session: the unchanged MCP launch failed against an isolated empty mirror, and the assistant quoted the actionable preflight context without using tools.
Healthy workspace: preflight output was silent; the unchanged shipped dnx command initialized BinlogMcp 3.0.4 and returned 47 MCP tools, including binlog_overview.
Coverage limitations: Claude Code live-session delivery has not been verified because Claude is not installed in the validation environment. Native macOS/Linux hook execution is covered by the passing CI matrix; live assistant delivery was reverified with Copilot CLI 1.0.97 on Windows. Native Windows Claude requires Git Bash and delegates to the Windows PowerShell launcher. Copilot CLI 1.0.68 did not load the plugin hook in the live check and is not covered. No diagnostic support is claimed for Codex, VS Code, or Cursor.
Checklist
I searched existing issues and pull requests to avoid duplicates.
I kept this pull request focused and avoided unrelated refactors.
I added focused tests and documentation for the changed diagnostic behavior.
I reviewed CODEOWNERS; existing plugin, engineering, and workflow rules cover all added files.
I updated only the required host-specific hook declarations; plugin metadata and marketplace MCP declarations are unchanged.
No new external domains were added to skill content; a known-domains update is not applicable.
Evaluation-specific checklist items are not applicable: no skill evaluations or validator evaluation logic changed.
Add bounded assistant-visible NuGet preflight hooks for Copilot CLI and Claude Code without changing the MCP launch or workspace feed policy. Cover structured results, workspace settings, host payloads, and timeout cleanup independently of skill evaluations.
Refs #1312
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
On Windows, if package search exits while a credential-provider descendant keeps stdout or stderr open, this branch returns TimedOut without terminating that descendant. The FSI helper can then exit normally, so session-start.ps1 skips its timeout-only taskkill; disposing the process object does not stop the provider. Establish process-tree ownership before launch (for example, a kill-on-close Windows Job Object), and clean up descendants on normal and early exits as well as timeouts. Add a Windows regression where the search process exits but its descendant keeps a redirected stream open.
Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.
A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.
Target
Model
Verdict
Gate evidence
Overfit
Warnings
Next action
agent.msbuild
claude-sonnet-5
➖ Improvement signal, unproven
n=5; 4W/0T/1L; d=5; p=0.188; net +60.0%
—
—
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
agent.msbuild
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=5; 3W/2T/0L; d=3; p=0.125; net +60.0%
—
—
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
binlog-failure-analysis
claude-sonnet-5
➖ Baseline signal, tie-limited
n=6; 1W/3T/2L; d=3; p=0.500; net -16.7%; 1 dormancy excluded
🟡 0.29
Activation: isolated 5/7; plugin 6/7
Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
binlog-failure-analysis
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded
🟡 0.49
Activation: isolated 6/7; plugin 6/7
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
binlog-generation
claude-sonnet-5
✅ Improved
n=7; 6W/1T/0L; d=6; p=0.016; net +85.7%; 1 dormancy excluded
🔴 0.52
Activation: isolated 5/7; plugin 6/7
Fix activation gaps; Review overfit evidence.
binlog-generation
gpt-5.6-luna
➖ Improvement signal, unproven
n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded
🔴 0.58
Activation: isolated 7/7; plugin 5/7
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
build-parallelism
claude-sonnet-5
➖ Mixed evidence
n=6; 2W/2T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded
✅ 0.11
Activation: isolated 5/6; plugin 5/6
Compare winning and losing scenarios to isolate where the target helps versus hurts.
build-parallelism
gpt-5.6-luna
➖ Baseline signal, unproven
n=6; 2W/0T/4L; d=6; p=0.344; net -33.3%; 1 dormancy excluded
Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
build-perf-baseline
claude-sonnet-5
➖ Improvement signal, unproven
n=6; 3W/1T/2L; d=5; p=0.500; net +16.7%; 1 dormancy excluded
🟡 0.48
Activation: isolated 5/6; plugin 3/6
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
build-perf-baseline
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded
🟡 0.48
Activation: isolated 6/6; plugin 4/6
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
build-perf-diagnostics
claude-sonnet-5
➖ Improvement signal, unproven
n=7; 4W/1T/2L; d=6; p=0.344; net +28.6%; 1 dormancy excluded
🟡 0.44
Activation: isolated 7/7; plugin 3/7
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
build-perf-diagnostics
gpt-5.6-luna
➖ Mixed evidence
n=7; 3W/1T/3L; d=6; p=0.656; net +0.0%; 1 dormancy excluded
🟡 0.37
Activation: isolated 7/7; plugin 3/7
Compare winning and losing scenarios to isolate where the target helps versus hurts.
check-bin-obj-clash
claude-sonnet-5
➖ Improvement signal, tie-limited
n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%
🟡 0.24
—
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
check-bin-obj-clash
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%
🟡 0.33
Activation: isolated 7/7; plugin 6/7
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
directory-build-organization
claude-sonnet-5
➖ Improvement signal, unproven
n=6; 4W/0T/2L; d=6; p=0.344; net +33.3%; 1 dormancy excluded
🟡 0.35
Activation: isolated 4/6; plugin 5/6
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
directory-build-organization
gpt-5.6-luna
➖ Baseline signal, tie-limited
n=6; 1W/2T/3L; d=4; p=0.312; net -33.3%; 1 dormancy excluded
🔴 0.55
Activation: isolated 6/6; plugin 4/6
Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
eval-performance
claude-sonnet-5
➖ Improvement signal, unproven
n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%
🟡 0.40
Activation: isolated 6/8; plugin 5/8
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
eval-performance
gpt-5.6-luna
➖ Improvement signal, unproven
n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%
🟡 0.40
—
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
extension-points
claude-sonnet-5
➖ Improvement signal, unproven
n=7; 4W/2T/1L; d=5; p=0.188; net +42.9%; 2 dormancy excluded
🟡 0.43
Activation: isolated 5/7; plugin 6/7
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
extension-points
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=7; 2W/5T/0L; d=2; p=0.250; net +28.6%; 2 dormancy excluded
🟡 0.36
Activation: isolated 7/7; plugin 6/7
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
including-generated-files
claude-sonnet-5
➖ Improvement signal, unproven
n=7; 4W/1T/2L; d=6; p=0.344; net +28.6%; 1 dormancy excluded
🟡 0.38
Activation: isolated 5/7; plugin 6/7
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
including-generated-files
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%; 1 dormancy excluded
🟡 0.38
Activation: isolated 7/7; plugin 6/7
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
incremental-build
claude-sonnet-5
➖ Improvement signal, unproven
n=9; 5W/3T/1L; d=6; p=0.109; net +44.4%
🟡 0.24
Activation: isolated 4/9; plugin 4/9
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
incremental-build
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=9; 3W/5T/1L; d=4; p=0.312; net +22.2%
🔴 0.57
Activation: isolated 9/9; plugin 6/9
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
item-management
claude-sonnet-5
➖ Improvement signal, tie-limited
n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded
🟡 0.33
Activation: isolated 2/6; plugin 3/6
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
item-management
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 1 dormancy excluded
🔴 0.54
—
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
msbuild-antipatterns
claude-sonnet-5
➖ Improvement signal, tie-limited
n=7; 2W/4T/1L; d=3; p=0.500; net +14.3%; 1 dormancy excluded
🟡 0.25
Activation: isolated 3/7; plugin 3/7
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
msbuild-antipatterns
gpt-5.6-luna
➖ Baseline signal, tie-limited
n=7; 1W/4T/2L; d=3; p=0.500; net -14.3%; 1 dormancy excluded
🔴 0.54
Activation: isolated 5/7; plugin 3/7
Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
msbuild-modernization
claude-sonnet-5
➖ Improvement signal, tie-limited
n=6; 2W/4T/0L; d=2; p=0.250; net +33.3%; 1 dormancy excluded
🟡 0.29
Activation: isolated 6/6; plugin 5/6
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
msbuild-modernization
gpt-5.6-luna
➖ Mixed evidence
n=6; 2W/2T/2L; d=4; p=0.687; net +0.0%; 1 dormancy excluded
🟡 0.21
—
Compare winning and losing scenarios to isolate where the target helps versus hurts.
property-patterns
claude-sonnet-5
➖ Baseline signal, unproven
n=7; 2W/0T/5L; d=7; p=0.227; net -42.9%
🟡 0.21
Activation: isolated 5/7; plugin 6/7
Evidence leans baseline but is not credible; inspect losing scenarios for recurring defects.
property-patterns
gpt-5.6-luna
➖ Mixed evidence
n=7; 2W/3T/2L; d=4; p=0.687; net +0.0%
🟡 0.24
Activation: isolated 6/7; plugin 6/7
Compare winning and losing scenarios to isolate where the target helps versus hurts.
resolve-project-references
claude-sonnet-5
➖ Improvement signal, tie-limited
n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 2 dormancy excluded
🟡 0.36
Activation: isolated 5/6; plugin 4/6
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
resolve-project-references
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 2 dormancy excluded
🟡 0.39
Activation: isolated 6/6; plugin 4/6
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
target-authoring
claude-sonnet-5
➖ Baseline signal, tie-limited
n=6; 0W/4T/2L; d=2; p=0.250; net -33.3%; 1 dormancy excluded
🟡 0.42
Activation: isolated 5/6; plugin 5/6
Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
target-authoring
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=6; 4W/2T/0L; d=4; p=0.063; net +66.7%; 1 dormancy excluded
🟡 0.39
—
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
ℹ️ How to read this report
✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.
Why: Net win +60.0% (4W/0T/1L over 5 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +36.0% across 5 paired run(s) — not credible (sign test p=0.188 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Gate evidence: n=5; 4W/0T/1L; d=5; p=0.188; net +60.0%
Repeated-run reliability (not used by the gate): 5 paired runs (4W/0T/1L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Review a project file for maintainability risks
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Review a project file for maintainability risks: A catches and prioritizes the stronger correctness risk, offers a workable targeted fix, and also covers the incremental-execution issue. B is more concise and avoids optional generic concerns, but misses the directory-ordering defect and includes a technically questionable cl...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +60.0% (3W/2T/0L over 5 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +36.0% across 5 paired run(s) — not credible — 2 of 5 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the agent is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Why: Net win -16.7% (1W/3T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
Repeated-run reliability (not used by the gate): 7 paired runs (1W/4T/2L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Assess a requested binlog investigation when the current build is healthy
Eligible
-100.0%
-40.0%
0/0/1
= Confirm the actual resolved target framework and package version from a binlog
Eligible
+0.0%
+0.0%
0/0/0
= Determine whether a quiet second build actually failed
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose a warning behind a build that actually succeeded
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose build failures from binlog only (no source files)
Eligible
+0.0%
+0.0%
0/1/0
= Stay dormant for a non-MSBuild build failure log
Excluded (activation contract)
+0.0%
+0.0%
0/1/0
▼ Trace why a generated source file is missing at compile time
Eligible
-100.0%
-100.0%
0/0/1
Illustrative judge evidence:
Assess a requested binlog investigation when the current build is healthy: Both reach the correct core conclusion and preserve a binlog. A is modestly stronger because it actually performs additional diagnostic validation and interprets potentially alarming MSBuild resolver and NuGet-code text, yielding a fuller analysis while remaining appropriately...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +66.7% (4W/2T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +37.1% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Why: Net win +57.1% (5W/1T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +50.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Repeated-run reliability (not used by the gate): 8 paired runs (5W/2T/1L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Choose a predictable, non-colliding binlog name for a CI upload step
Eligible
-100.0%
-40.0%
0/0/1
= Decline Maven compiler configuration
Excluded (activation contract)
+0.0%
+0.0%
0/1/0
▲ Fix a CI build script that reuses the same binlog on every run
Eligible
+100.0%
+40.0%
1/0/0
▲ Preserve binlog history while cleaning stale build output
Eligible
+100.0%
+100.0%
1/0/0
= Recognize that a failed build produced no binlog at all
Eligible
+0.0%
+0.0%
0/1/0
Illustrative judge evidence:
Choose a predictable, non-colliding binlog name for a CI upload step: Both responses successfully completed the task and met all rubric requirements with identical output filenames. Response A has a slight edge due to explicit verification of the generated binlogs using ls -lh and file commands, which is a best practice for CI/CD artifacts a...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +0.0% (2W/2T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.687), mean preference +8.6% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.
Repeated-run reliability (not used by the gate): 7 paired runs (2W/3T/2L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Analyze build parallelism bottlenecks
Eligible
-100.0%
-40.0%
0/0/1
= Decline database query tuning request
Excluded (activation contract)
+0.0%
+0.0%
0/1/0
= Decline graph build for runtime-discovered projects
Eligible
+0.0%
+0.0%
0/1/0
= Enable parallel nodes for a wide project graph
Eligible
+0.0%
+0.0%
0/1/0
▼ Reduce CI build scope with a solution filter
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Analyze build parallelism bottlenecks: Both provide the correct dependency, redundancy, and critical-path conclusions. A is stronger because it substantiates the requested binlog-based parallelism analysis with actual observed timing/node evidence, whereas B does not.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win -33.3% (2W/0T/4L over 6 preference-eligible stimulus vote(s), sign test p=0.344), mean preference -17.1% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
Repeated-run reliability (not used by the gate): 7 paired runs (2W/0T/5L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Decline database query tuning request
Excluded (activation contract)
-100.0%
-40.0%
0/0/1
▼ Decline graph build for runtime-discovered projects
Eligible
-100.0%
-40.0%
0/0/1
▼ Enable BuildInParallel on a custom MSBuild task
Eligible
-100.0%
-40.0%
0/0/1
▲ Enable parallel nodes for a wide project graph
Eligible
+100.0%
+40.0%
1/0/0
▼ Preserve valid dependencies and optimize the slow critical-path project
Eligible
-100.0%
-40.0%
0/0/1
▼ Reduce CI build scope with a solution filter
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Decline graph build for runtime-discovered projects: Both responses correctly identify the fundamental problem and explain why /graph is unsafe with runtime project discovery. However, Response A is slightly better because: (1) it more precisely names the artifact (@(Discovered)) and the timing issue; (2) it explicitly acknowled...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +16.7% (3W/1T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -0.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.500 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Repeated-run reliability (not used by the gate): 7 paired runs (3W/1T/3L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▲ Configure deterministic, cache-safe CI builds
Eligible
+100.0%
+40.0%
1/0/0
▼ Decline a non-MSBuild build performance request
Excluded (activation contract)
-100.0%
-100.0%
0/0/1
▼ Leave an already-optimized build unchanged
Eligible
-100.0%
-40.0%
0/0/1
▼ Route a broken no-op rebuild away from generic optimization
Eligible
-100.0%
-40.0%
0/0/1
= Route a restore-bound cold build away from architecture changes
Eligible
+0.0%
+0.0%
0/1/0
Illustrative judge evidence:
Leave an already-optimized build unchanged: A is not ideal because it makes changes where the checklist calls for none, but its assessment of the existing CI-gated settings is sound and its added reference-assembly setting has a plausible incremental-build rationale. B's main finding is backwards for this build-performa...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Why: Net win +28.6% (4W/1T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Why: Net win +0.0% (3W/1T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.656), mean preference +0.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.
Repeated-run reliability (not used by the gate): 8 paired runs (3W/2T/3L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
= Decline a runtime latency request that is not a build performance issue
Excluded (activation contract)
+0.0%
+0.0%
0/1/0
▼ Diagnose NuGet restore running redundantly across CI stages
Eligible
-100.0%
-40.0%
0/0/1
= Diagnose a Copy task dominating build time
Eligible
+0.0%
+0.0%
0/1/0
▲ Diagnose a pathological ResolveAssemblyReference time
Eligible
+100.0%
+40.0%
1/0/0
▲ Diagnose a single custom target dominating one project's build
Eligible
+100.0%
+40.0%
1/0/0
▼ Diagnose evaluation overhead before any target runs
Eligible
-100.0%
-40.0%
0/0/1
▼ Diagnose per-project overhead across many small projects
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Diagnose NuGet restore running redundantly across CI stages: Both responses correctly diagnose the problem and recommend the primary solution (restore-once + --no-restore). The key differentiator is Response A's superior handling of the multi-stage CI scenario: it explicitly warns that NuGet cache alone is insufficient—you must persist/...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +42.9% (3W/4T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +17.1% across 7 paired run(s) — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Gate evidence: n=7; 3W/4T/0L; d=3; p=0.125; net +42.9%
Overfit: Moderate (score 0.24)
Repeated-run reliability (not used by the gate): 7 paired runs (3W/4T/0L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
= Avoid a false clash report when projects share only the top-level artifacts root
Eligible
+0.0%
+0.0%
0/1/0
= Decline output-clash remediation for separate projects using the default SDK layout
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose redundant project reference metadata that forks a same-path build
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose shared output and intermediate path collision
Eligible
+0.0%
+0.0%
0/1/0
Illustrative judge evidence:
Avoid a false clash report when projects share only the top-level artifacts root: Position-swap inconsistent (forward: B, reverse: tie). Defaulting to tie.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +11.4% across 7 paired run(s) — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Gate evidence: n=7; 3W/3T/1L; d=4; p=0.312; net +28.6%
Warnings: Activation: isolated 7/7; plugin 6/7
Overfit: Moderate (score 0.33)
Repeated-run reliability (not used by the gate): 7 paired runs (3W/3T/1L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Audit the mixed solution and separate safe projects from unsafe ones
Eligible
-100.0%
-40.0%
0/0/1
= Avoid a false clash report when projects share only the top-level artifacts root
Eligible
+0.0%
+0.0%
0/1/0
▲ Diagnose multi-targeting outputs that collapse into one path
Eligible
+100.0%
+40.0%
1/0/0
= Diagnose shared output and intermediate path collision
Eligible
+0.0%
+0.0%
0/1/0
= Fix all clash mechanisms in the mixed solution
Eligible
+0.0%
+0.0%
0/1/0
Illustrative judge evidence:
Audit the mixed solution and separate safe projects from unsafe ones: Response A provides clearer and more actionable categorization by correctly placing ToolLib in the "Safe to leave alone" section and precisely identifying the ConsumerApp reference (not ToolLib itself) as the unsafe element. Response B's placement of ToolLib in the "Unsafe" ta...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +33.3% (4W/0T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +20.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Repeated-run reliability (not used by the gate): 7 paired runs (4W/1T/2L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
= Decline adding shared build files to a lone project
Excluded (activation contract)
+0.0%
+0.0%
0/1/0
▼ Diagnose a TargetFramework condition that silently skips in props
Eligible
-100.0%
-40.0%
0/0/1
▲ Diagnose an inner shared-props file that overwrites its own override
Eligible
+100.0%
+40.0%
1/0/0
▼ Preserve an intentional project-specific exception while centralizing
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Diagnose a TargetFramework condition that silently skips in props: Both answers reach the correct practical diagnosis and fix. A is slightly cleaner and more focused; B unnecessarily states that Directory.Build.props is before Sdk.props and adds a muddled, non-actionable alternative about keeping it in props. A's import-order wording is somew...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win -33.3% (1W/2T/3L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference -5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
Repeated-run reliability (not used by the gate): 7 paired runs (2W/2T/3L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Apply repo-level build organization cleanup
Eligible
-100.0%
-40.0%
0/0/1
= Diagnose a TargetFramework condition that silently skips in props
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose a package downgrade chain and reorganize version management
Eligible
+0.0%
+0.0%
0/1/0
▼ Diagnose an inner shared-props file that overwrites its own override
Eligible
-100.0%
-40.0%
0/0/1
▼ Preserve an intentional project-specific exception while centralizing
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Apply repo-level build organization cleanup: Both responses successfully centralized the build layout, but Response A demonstrates superior quality through explicit verification and real problem-solving. A verified the final state through MSBuild property queries confirming duplication removal, and had to debug and resol...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +62.5% (6W/1T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +25.0% across 8 paired run(s) — not credible (sign test p=0.063 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Gate evidence: n=8; 6W/1T/1L; d=7; p=0.063; net +62.5%
Warnings: Activation: isolated 6/8; plugin 5/8
Overfit: Moderate (score 0.40)
Repeated-run reliability (not used by the gate): 8 paired runs (6W/1T/1L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Gather a measurement before proposing evaluation fixes for an unremarkable project
Eligible
-100.0%
-40.0%
0/0/1
▲ Recognize TreatAsLocalProperty overuse versus one justified entry
Eligible
+100.0%
+40.0%
1/0/0
= Redirect an incremental-rebuild complaint mistakenly framed as an evaluation problem
Eligible
+0.0%
+0.0%
0/1/0
▲ Triage which of two property functions actually costs evaluation time
Eligible
+100.0%
+40.0%
1/0/0
Illustrative judge evidence:
Gather a measurement before proposing evaluation fixes for an unremarkable project: Both deliver the correct measurement-first outcome. A is marginally more actionable and precise about how to obtain and interpret evaluation timing, while remaining appropriately non-speculative.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +50.0% (5W/2T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +35.0% across 8 paired run(s) — not credible (sign test p=0.109 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Gate evidence: n=8; 5W/2T/1L; d=6; p=0.109; net +50.0%
Overfit: Moderate (score 0.40)
Repeated-run reliability (not used by the gate): 8 paired runs (5W/2T/1L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Detect a project evaluated twice under different global properties
Eligible
-100.0%
-40.0%
0/0/1
= Redirect a compile-time slowdown mistakenly framed as an evaluation problem
Eligible
+0.0%
+0.0%
0/1/0
= Triage which of two property functions actually costs evaluation time
Eligible
+0.0%
+0.0%
0/1/0
Illustrative judge evidence:
Detect a project evaluated twice under different global properties: Response A provides a more technically precise and complete explanation of the root cause: MSBuild's inability to reuse evaluations when global properties differ. Response A also quantifies the impact (386.5 ms + 32.2 ms ≈ 419 ms of 1.10 s build) and correctly identifies that ...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +42.9% (4W/2T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +31.1% across 9 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Why: Net win +28.6% (2W/5T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +17.8% across 9 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible — 5 of 7 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Why: Net win +28.6% (4W/1T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Repeated-run reliability (not used by the gate): 8 paired runs (4W/2T/2L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
= Decline a Python runtime report-path problem
Excluded (activation contract)
+0.0%
+0.0%
0/1/0
▲ Diagnose missing clean tracking for generated source
Eligible
+100.0%
+40.0%
1/0/0
▼ Diagnose missing output registration for generated non-code file
Eligible
-100.0%
-40.0%
0/0/1
▼ Diagnose project-level glob for generated source
Eligible
-100.0%
-40.0%
0/0/1
▲ Diagnose wrong hook for generated source files
Eligible
+100.0%
+40.0%
1/0/0
= Fix generated source inclusion and clean tracking
Eligible
+0.0%
+0.0%
0/1/0
Illustrative judge evidence:
Diagnose missing output registration for generated non-code file: A delivers a fully applied, clean, verified solution that places the file in the requested bin root. B has sound core diagnosis and timing guidance but leaves its actual applied change with the file nested under obj and asks whether to perform the final correction.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +42.9% (3W/4T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.125), mean preference +20.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Why: Net win +44.4% (5W/3T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +17.8% across 9 paired run(s) — not credible (sign test p=0.109 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Why: Net win +22.2% (3W/5T/1L over 9 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +15.6% across 9 paired run(s) — not credible — 5 of 9 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +17.1% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Repeated-run reliability (not used by the gate): 7 paired runs (4W/2T/1L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▲ Diagnose an ineffective Compile Remove that does not match the glob
Eligible
+100.0%
+40.0%
1/0/0
▼ Diagnose item group and batching issues
Eligible
-100.0%
-40.0%
0/0/1
= Diagnose real and claimed item problems in a code generation pipeline
Eligible
+0.0%
+0.0%
0/1/0
= Fix item management anti-patterns
Eligible
+0.0%
+0.0%
0/1/0
▲ Leave already-correct item management unchanged
Eligible
+100.0%
+40.0%
1/0/0
▲ Leave correct single-list batching unchanged
Eligible
+100.0%
+40.0%
1/0/0
Illustrative judge evidence:
Diagnose item group and batching issues: Both diagnose the three underlying MSBuild item-management problems well, but A is more precise and actionable, especially for defining a reliable data/schema association and using the proper Update operation.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +33.3% (2W/4T/0L over 6 preference-eligible stimulus vote(s), sign test p=0.250), mean preference +28.6% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 6 preference-eligible stimulus vote(s) tied, leaving only 2 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Why: Net win +14.3% (2W/4T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +0.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Repeated-run reliability (not used by the gate): 8 paired runs (2W/4T/2L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
= Add a module to an F# project
Eligible
+0.0%
+0.0%
0/1/0
= Add a signature file to define public API
Eligible
+0.0%
+0.0%
0/1/0
= Fix broken file order causing FS0039
Eligible
+0.0%
+0.0%
0/1/0
= Judge an unguarded import inside a NuGet package build folder
Eligible
+0.0%
+0.0%
0/1/0
▼ Leave a clean project without inventing anti-patterns
Eligible
-100.0%
-40.0%
0/0/1
▼ Non-activation: migrate a legacy project to SDK style
Excluded (activation contract)
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Add a module to an F# project: Both responses fully implement the requested validation module, compile ordering, and pre-processing validation flow, and both verify a successful build and run. The minor differences in validation/check ordering and compile placement relative to Services.fs do not affect comp...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Details for 10 results were omitted to keep this comment under GitHub's limit. Open the workflow summary or Full Results for the complete breakdown.
To investigate non-passing or warning results, paste this to your AI coding agent:
For PR 1315 in dotnet/skills, download eval artifacts with gh run download 38572106478 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/76b96cfdf3a75bc4b3e355d7519f0933ae18777f/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.
⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.
Before starting package search, the FSI helper joins an unnamed kill-on-close Windows Job Object. Its sole non-inheritable handle lives until helper teardown, so Windows terminates orphaned descendants on normal/early helper exit and launcher termination, even after their search parent exits. Failed ownership setup skips the probe and reports unverified availability. Process exit and redirected-stream completion share the same five-second probe budget; the MCP command and NuGet policy are unchanged.
The regression reproduces an exiting search whose provider holds redirected streams open. It verifies provider termination after helper exit and forced eight-second launcher timeout through native PowerShell and both Git Bash host schemas, while an unrelated process survives. Native Windows/macOS/Linux CI passes: Windows 13 passed / 2 Unix-only skips; macOS and Linux 11 passed / 4 Windows-only skips. The structured runner passes 65 assertions. Actual Copilot CLI 1.0.97 assistant delivery and the unchanged healthy MCP launch were reverified on Windows.
Claude Code live delivery remains unverified because Claude is not installed in this environment. Its documented output schema and shell paths have regression coverage, but I am not claiming end-to-end Claude verification; the PR retains that limitation.
👋 @YuliiaKovalova — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)
Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.
A result passes only when preference-eligible distinct-stimulus votes have aggregate net win of at least 20% and an exact one-sided sign-test result of p ≤ 0.05, and every explicit dormancy activation contract passes. Repeated runs measure reliability only.
Target
Model
Verdict
Gate evidence
Overfit
Warnings
Next action
agent.msbuild
claude-sonnet-5
➖ Mixed evidence
n=5; 2W/1T/2L; d=4; p=0.687; net +0.0%
—
—
Compare winning and losing scenarios to isolate where the target helps versus hurts.
agent.msbuild
gpt-5.6-luna
➖ Baseline signal, tie-limited
n=5; 0W/3T/2L; d=2; p=0.250; net -40.0%
—
—
Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
binlog-failure-analysis
claude-sonnet-5
➖ Baseline signal, unproven
n=6; 2W/0T/4L; d=6; p=0.344; net -33.3%; 1 dormancy excluded
🟡 0.24
Activation: isolated 5/7; plugin 6/7
Evidence leans baseline but is not credible; inspect losing scenarios for recurring defects.
binlog-failure-analysis
gpt-5.6-luna
➖ Improvement signal, unproven
n=6; 4W/0T/2L; d=6; p=0.344; net +33.3%; 1 dormancy excluded
🟡 0.30
Activation: isolated 6/7; plugin 6/7
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
binlog-generation
claude-sonnet-5
➖ Improvement signal, unproven
n=7; 4W/1T/2L; d=6; p=0.344; net +28.6%; 1 dormancy excluded
🔴 0.52
Activation: isolated 5/7; plugin 6/7
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
binlog-generation
gpt-5.6-luna
➖ Improvement signal, unproven
n=7; 5W/1T/1L; d=6; p=0.109; net +57.1%; 1 dormancy excluded
🟡 0.39
Activation: isolated 5/7; plugin 6/7
The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
build-parallelism
claude-sonnet-5
➖ Baseline signal, tie-limited
n=6; 0W/2T/4L; d=4; p=0.063; net -66.7%; 1 dormancy excluded
Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
target-authoring
claude-sonnet-5
➖ Baseline signal, tie-limited
n=6; 1W/3T/2L; d=3; p=0.500; net -16.7%; 1 dormancy excluded
🟡 0.46
Activation: isolated 5/6; plugin 3/6
Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
target-authoring
gpt-5.6-luna
➖ Improvement signal, tie-limited
n=6; 3W/2T/1L; d=4; p=0.312; net +33.3%; 1 dormancy excluded
✅ 0.14
—
The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
ℹ️ How to read this report
✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
➖ No clear winner — the result is valid but did not pass both gates. The label distinguishes all ties, mixed evidence, directional but unproven evidence, and credible effects below the practical floor.
⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the target.
⛔ Activation contract failed — the isolated target activated on an explicit dormancy scenario. Dormancy preference is excluded, but this routing failure still blocks a pass.
📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
Gate evidence — n preference-eligible distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, net win, and the count of separately retained dormancy stimuli. The p value applies to one model/target result; no matrix-wide multiple-comparison correction is applied.
Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.
Why: Net win +0.0% (2W/1T/2L over 5 preference-eligible stimulus vote(s), sign test p=0.687), mean preference +12.0% across 5 paired run(s) — no improvement
Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.
Gate evidence: n=5; 2W/1T/2L; d=4; p=0.687; net +0.0%
Repeated-run reliability (not used by the gate): 5 paired runs (2W/1T/2L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Diagnose broken incremental build behavior
Eligible
-100.0%
-40.0%
0/0/1
= Route a slow build to performance analysis
Eligible
+0.0%
+0.0%
0/1/0
▼ Triage a build failure and route to appropriate analysis
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Diagnose broken incremental build behavior: Both reach the correct primary diagnosis and propose a viable Inputs/Outputs fix. A is more useful overall for the user's request because it gives broader, concrete guidance for diagnosing timestamp and input/output mistakes. B is somewhat stronger on explicitly describing a s...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win -40.0% (0W/3T/2L over 5 preference-eligible stimulus vote(s), sign test p=0.250), mean preference -16.0% across 5 paired run(s) — no improvement
Next action: Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
Why: Net win -33.3% (2W/0T/4L over 6 preference-eligible stimulus vote(s), sign test p=0.344), mean preference -20.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Evidence leans baseline but is not credible; inspect losing scenarios for recurring defects.
Repeated-run reliability (not used by the gate): 7 paired runs (2W/1T/4L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Assess a requested binlog investigation when the current build is healthy
Eligible
-100.0%
-40.0%
0/0/1
= Confirm the actual resolved target framework and package version from a binlog
Eligible
+0.0%
+0.0%
0/0/0
▼ Diagnose a warning behind a build that actually succeeded
Eligible
-100.0%
-40.0%
0/0/1
▼ Diagnose build failures from binlog only (no source files)
Eligible
-100.0%
-40.0%
0/0/1
= Stay dormant for a non-MSBuild build failure log
Excluded (activation contract)
+0.0%
+0.0%
0/1/0
▼ Trace why a generated source file is missing at compile time
Eligible
-100.0%
-100.0%
0/0/1
Illustrative judge evidence:
Assess a requested binlog investigation when the current build is healthy: Both results are correct, safe, and satisfy the central request by capturing a binlog and reporting a clean build. A is slightly more useful because its final analysis documents successful restore/output and the benign project context rather than only stating success.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +33.3% (4W/0T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Repeated-run reliability (not used by the gate): 7 paired runs (4W/0T/3L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Assess a requested binlog investigation when the current build is healthy
Eligible
-100.0%
-100.0%
0/0/1
= Confirm the actual resolved target framework and package version from a binlog
Eligible
+0.0%
+0.0%
0/0/0
▼ Stay dormant for a non-MSBuild build failure log
Excluded (activation contract)
-100.0%
-40.0%
0/0/1
▼ Trace why a generated source file is missing at compile time
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Assess a requested binlog investigation when the current build is healthy: Response A correctly identified that InventorySync builds successfully with no code failures, captured appropriate binlogs as requested, and explained why certain incorrect invocations fail. Response B violated the core rubric intent by deliberately using a non-standard build ...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +28.6% (4W/1T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +45.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Why: Net win +57.1% (5W/1T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.109), mean preference +57.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.109 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Why: Net win -66.7% (0W/2T/4L over 6 preference-eligible stimulus vote(s), sign test p=0.063), mean preference -31.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
Repeated-run reliability (not used by the gate): 7 paired runs (0W/3T/4L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Analyze build parallelism bottlenecks
Eligible
-100.0%
-40.0%
0/0/1
= Decline database query tuning request
Excluded (activation contract)
+0.0%
+0.0%
0/1/0
= Decline graph build for runtime-discovered projects
Eligible
+0.0%
+0.0%
0/1/0
▼ Enable BuildInParallel on a custom MSBuild task
Eligible
-100.0%
-100.0%
0/0/1
▼ Enable parallel nodes for a wide project graph
Eligible
-100.0%
-40.0%
0/0/1
= Preserve valid dependencies and optimize the slow critical-path project
Eligible
+0.0%
+0.0%
0/1/0
▼ Reduce CI build scope with a solution filter
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Analyze build parallelism bottlenecks: Both answers are substantively correct and satisfy the task. A is slightly stronger overall because its conclusion about the redundant edge and critical path is more precise and avoids B's unnecessary, potentially misleading hypothetical caveat; B has marginally better stated ...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +33.3% (3W/2T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 2 of 6 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: Inspect activation-only failed runs before rewriting skill content; the model stopped after loading a skill.
Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +28.6% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +20.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Why: Net win +28.6% (3W/3T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.312), mean preference +10.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Why: Net win -14.3% (2W/2T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -5.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Evidence leans baseline but is not credible; inspect losing scenarios for recurring defects.
Repeated-run reliability (not used by the gate): 8 paired runs (2W/3T/3L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
= Decline a runtime latency request that is not a build performance issue
Excluded (activation contract)
+0.0%
+0.0%
0/1/0
▼ Diagnose NuGet restore running redundantly across CI stages
Eligible
-100.0%
-40.0%
0/0/1
= Diagnose a pathological ResolveAssemblyReference time
Eligible
+0.0%
+0.0%
0/1/0
▼ Diagnose a single custom target dominating one project's build
Eligible
-100.0%
-40.0%
0/0/1
▲ Diagnose evaluation overhead before any target runs
Eligible
+100.0%
+40.0%
1/0/0
= Diagnose per-project overhead across many small projects
Eligible
+0.0%
+0.0%
0/1/0
▼ Diagnose slow build for a small project
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Diagnose NuGet restore running redundantly across CI stages: Response A is marginally better overall due to superior technical precision. It includes --locked-mode on the restore command, which is more appropriate for scenarios with packages.lock.json (the implied context from the task). Response A also provides a clearer explanation ...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +0.0% (2W/3T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.687), mean preference +17.1% across 7 paired run(s) — no improvement
Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.
Gate evidence: n=7; 2W/3T/2L; d=4; p=0.687; net +0.0%
Warnings: Activation: isolated 6/7; plugin 5/7
Overfit: Moderate (score 0.22)
Repeated-run reliability (not used by the gate): 7 paired runs (2W/3T/2L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Audit the mixed solution and separate safe projects from unsafe ones
Eligible
-100.0%
-40.0%
0/0/1
▼ Avoid a false clash report when projects share only the top-level artifacts root
Eligible
-100.0%
-40.0%
0/0/1
= Decline output-clash remediation for separate projects using the default SDK layout
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose multi-targeting outputs that collapse into one path
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose redundant project reference metadata that forks a same-path build
Eligible
+0.0%
+0.0%
0/1/0
▲ Fix all clash mechanisms in the mixed solution
Eligible
+100.0%
+100.0%
1/0/0
Illustrative judge evidence:
Audit the mixed solution and separate safe projects from unsafe ones: The substantive findings and fixes are essentially the same, but A gives a more coherent, consistently classified audit. B contains a self-correction that leaves an unnecessary contradictory interim 'safe' assessment.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +0.0% (3W/1T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.656), mean preference +0.0% across 7 paired run(s) — no improvement
Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.
Gate evidence: n=7; 3W/1T/3L; d=6; p=0.656; net +0.0%
Warnings: Activation: isolated 7/7; plugin 5/7
Overfit: Moderate (score 0.20)
Repeated-run reliability (not used by the gate): 7 paired runs (3W/1T/3L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Audit the mixed solution and separate safe projects from unsafe ones
Eligible
-100.0%
-40.0%
0/0/1
▼ Avoid a false clash report when projects share only the top-level artifacts root
Eligible
-100.0%
-40.0%
0/0/1
= Decline output-clash remediation for separate projects using the default SDK layout
Eligible
+0.0%
+0.0%
0/1/0
▲ Diagnose multi-targeting outputs that collapse into one path
Eligible
+100.0%
+40.0%
1/0/0
▲ Diagnose redundant project reference metadata that forks a same-path build
Eligible
+100.0%
+40.0%
1/0/0
▼ Fix all clash mechanisms in the mixed solution
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Audit the mixed solution and separate safe projects from unsafe ones: Response A correctly fulfills all five rubric criteria by explicitly flagging ConsumerApp as unsafe, which is a specific requirement. Response B fails criterion 3 by not flagging ConsumerApp; instead, it flags ToolLib as unsafe and places ConsumerApp in the safe category. Whil...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +0.0% (1W/4T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.750), mean preference -5.7% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.
Repeated-run reliability (not used by the gate): 7 paired runs (1W/4T/2L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Apply repo-level build organization cleanup
Eligible
-100.0%
-40.0%
0/0/1
▼ Decline adding shared build files to a lone project
Excluded (activation contract)
-100.0%
-40.0%
0/0/1
= Diagnose a TargetFramework condition that silently skips in props
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose a package downgrade chain and reorganize version management
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose an inner shared-props file that overwrites its own override
Eligible
+0.0%
+0.0%
0/1/0
= Preserve an intentional project-specific exception while centralizing
Eligible
+0.0%
+0.0%
0/1/0
Illustrative judge evidence:
Apply repo-level build organization cleanup: Both solutions are largely correct and handle central package management and nested props imports properly. A is marginally better aligned with the requested separation: production shared targets are scoped to src and repeated test infrastructure is kept in the test area, yiel...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +50.0% (4W/1T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +11.4% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.188 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Why: Net win +37.5% (4W/3T/1L over 8 preference-eligible stimulus vote(s), sign test p=0.188), mean preference +22.5% across 8 paired run(s) — not credible (sign test p=0.188 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Gate evidence: n=8; 4W/3T/1L; d=5; p=0.188; net +37.5%
Warnings: Activation: isolated 6/8; plugin 5/8
Overfit: Moderate (score 0.24)
Repeated-run reliability (not used by the gate): 8 paired runs (4W/3T/1L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Decline to invent evaluation problems in an already-clean project
Eligible
-100.0%
-40.0%
0/0/1
= Gather a measurement before proposing evaluation fixes for an unremarkable project
Eligible
+0.0%
+0.0%
0/1/0
= Recognize TreatAsLocalProperty overuse versus one justified entry
Eligible
+0.0%
+0.0%
0/1/0
▲ Redirect an incremental-rebuild complaint mistakenly framed as an evaluation problem
Eligible
+100.0%
+100.0%
1/0/0
= Triage which of two property functions actually costs evaluation time
Eligible
+0.0%
+0.0%
0/1/0
Illustrative judge evidence:
Decline to invent evaluation problems in an already-clean project: Both answers are substantively correct and appropriately conclude that the project has no user-introduced evaluation-performance risks. A is marginally more precise about baseline SDK behavior; B is slightly more explicit about property functions but includes an overbroad clai...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +12.5% (4W/1T/3L over 8 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +20.0% across 8 paired run(s) — not credible (sign test p=0.500 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Gate evidence: n=8; 4W/1T/3L; d=7; p=0.500; net +12.5%
Warnings: Activation: isolated 8/8; plugin 7/8
Overfit: Moderate (score 0.29)
Repeated-run reliability (not used by the gate): 8 paired runs (4W/1T/3L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Detect a project evaluated twice under different global properties
Eligible
-100.0%
-40.0%
0/0/1
▼ Recognize TreatAsLocalProperty overuse versus one justified entry
Eligible
-100.0%
-40.0%
0/0/1
▼ Redirect a compile-time slowdown mistakenly framed as an evaluation problem
Eligible
-100.0%
-40.0%
0/0/1
= Triage which of two property functions actually costs evaluation time
Eligible
+0.0%
+0.0%
0/1/0
Illustrative judge evidence:
Detect a project evaluated twice under different global properties: Both responses correctly answer all four rubric criteria. Response A provides stronger evidence by showing the actual XML configuration from Build.proj, making the root cause visually undeniable. It also delivers precise quantification (Build.proj 19ms, Shared 275ms + 24ms, to...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win -14.3% (2W/2T/3L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +6.7% across 9 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Evidence leans baseline but is not credible; inspect losing scenarios for recurring defects.
▼ Diagnose NuGet package and repo extension conflicts
Eligible
-100.0%
-40.0%
0/0/1
▲ Diagnose a broken per-TFM forwarder
Eligible
+100.0%
+40.0%
1/0/0
= Diagnose a package ID and file-name mismatch
Eligible
+0.0%
+0.0%
0/1/0
▼ Diagnose build extension point failures
Eligible
-100.0%
-40.0%
0/0/1
▼ Fix extension point anti-patterns
Eligible
-100.0%
-40.0%
0/0/1
= Review packed layout without a false missing-file bug
Eligible
+0.0%
+0.0%
0/1/0
Illustrative judge evidence:
Diagnose NuGet package and repo extension conflicts: Both miss the important CustomBefore timing issue and the wildcard-properties clarification, and both handle the missing import well. A is overall stronger because its account of the GenerateAssemblyInfo collision is correct and it additionally identifies the fragile internal ...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +57.1% (4W/3T/0L over 7 preference-eligible stimulus vote(s), sign test p=0.063), mean preference +24.4% across 9 paired run(s), 2 dormancy stimulus/stimuli excluded from preference — not credible — 3 of 7 preference-eligible stimulus vote(s) tied, leaving only 4 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Why: Net win +14.3% (2W/4T/1L over 7 preference-eligible stimulus vote(s), sign test p=0.500), mean preference +0.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible — 4 of 7 preference-eligible stimulus vote(s) tied, leaving only 3 discordant preference vote(s). The sign test conditions on non-tie stimulus votes and cannot reach 0.05 below 5, so no record could have passed here — this is not a measured null. Either the skill is inert on these scenarios (make them discriminate) or the eval needs more distinct stimuli to clear the ties
Next action: The signal favors the target, but ties leave too few discordant tasks; inspect ties and predeclare more discriminating breadth.
Repeated-run reliability (not used by the gate): 8 paired runs (2W/4T/2L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Decline a Python runtime report-path problem
Excluded (activation contract)
-100.0%
-40.0%
0/0/1
= Diagnose hardcoded obj path for generated source
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose missing clean tracking for generated source
Eligible
+0.0%
+0.0%
0/1/0
= Diagnose missing generated source inclusion
Eligible
+0.0%
+0.0%
0/1/0
▲ Diagnose project-level glob for generated source
Eligible
+100.0%
+40.0%
1/0/0
= Diagnose wrong hook for generated source files
Eligible
+0.0%
+0.0%
0/1/0
▼ Fix generated source inclusion and clean tracking
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Diagnose hardcoded obj path for generated source: The final answers are substantively equivalent, correct, and complete. B adds a slightly more explicit clean-tracking observation, but A already includes FileWrites and explains the same essential remedy, so there is no meaningful quality difference.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +28.6% (4W/1T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +15.0% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Repeated-run reliability (not used by the gate): 8 paired runs (5W/1T/2L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Diagnose missing clean tracking for generated source
Eligible
-100.0%
-40.0%
0/0/1
= Diagnose missing generated source inclusion
Eligible
+0.0%
+0.0%
0/1/0
▼ Diagnose wrong hook for generated source files
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Diagnose missing clean tracking for generated source: Both responses arrive at the identical, correct solution and explanation. However, Response A demonstrates deeper technical understanding by investigating the actual FileListAbsolute.txt file that records FileWrites entries and explaining the specific mechanism of how the SDK'...
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win +22.2% (4W/3T/2L over 9 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +8.9% across 9 paired run(s) — not credible (sign test p=0.344 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
= Fix broken incremental targets and clean tracking
Eligible
+0.0%
+0.0%
0/1/0
▼ Identify a volatile output path that defeats incrementality
Eligible
-100.0%
-40.0%
0/0/1
▲ Read a diagnostic log to find the stale input
Eligible
+100.0%
+40.0%
1/0/0
Illustrative judge evidence:
Correct the assumption that Outputs alone enables incremental skipping: Both answers correctly resolve the core question and identify the non-file, in-target item issue. A is somewhat more precise about Outputs' caller-return role and frames any future incremental risk in terms of real file paths; both miss an explicit recommendation to use Returns.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Why: Net win -16.7% (1W/3T/2L over 6 preference-eligible stimulus vote(s), sign test p=0.500), mean preference -20.0% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Evidence leans baseline but is not credible; inspect losses and make tied tasks discriminate.
Repeated-run reliability (not used by the gate): 7 paired runs (1W/3T/3L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▼ Decline JavaScript configuration merging
Excluded (activation contract)
-100.0%
-40.0%
0/0/1
▼ Diagnose an ineffective Compile Remove that does not match the glob
Eligible
-100.0%
-100.0%
0/0/1
▲ Diagnose item group and batching issues
Eligible
+100.0%
+40.0%
1/0/0
= Diagnose real and claimed item problems in a code generation pipeline
Eligible
+0.0%
+0.0%
0/1/0
= Fix item management anti-patterns
Eligible
+0.0%
+0.0%
0/1/0
= Leave already-correct item management unchanged
Eligible
+0.0%
+0.0%
0/1/0
▼ Leave correct single-list batching unchanged
Eligible
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Diagnose an ineffective Compile Remove that does not match the glob: A is a complete, self-contained diagnosis and correction. B's final answer is an unsupported terse status claim that omits both the reason for the failure and the concrete fix.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
➖ Mixed evidence — item-management (gpt-5.6-luna)
Why: Net win +0.0% (1W/4T/1L over 6 preference-eligible stimulus vote(s), sign test p=0.750), mean preference +14.3% across 7 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — no improvement
Next action: Compare winning and losing scenarios to isolate where the target helps versus hurts.
Why: Net win +28.6% (4W/1T/2L over 7 preference-eligible stimulus vote(s), sign test p=0.344), mean preference +12.5% across 8 paired run(s), 1 dormancy stimulus/stimuli excluded from preference — not credible (sign test p=0.344 > 0.05)
Next action: The signal favors the target but is inconsistent; inspect tied or lost scenarios and fix weak behavior.
Repeated-run reliability (not used by the gate): 8 paired runs (4W/1T/3L).
Weak or warning scenarios:
Scenario
Preference gate
Net win
Δ Pref
Runs (W/T/L)
▲ Add a module to an F# project
Eligible
+100.0%
+40.0%
1/0/0
= Add a signature file to define public API
Eligible
+0.0%
+0.0%
0/1/0
▲ Fix broken file order causing FS0039
Eligible
+100.0%
+40.0%
1/0/0
▼ Judge an unguarded import inside a NuGet package build folder
Eligible
-100.0%
-40.0%
0/0/1
▼ Leave a clean project without inventing anti-patterns
Eligible
-100.0%
-40.0%
0/0/1
▼ Non-activation: migrate a legacy project to SDK style
Excluded (activation contract)
-100.0%
-40.0%
0/0/1
Illustrative judge evidence:
Add a signature file to define public API: The resulting changes and verification are substantively identical and fully satisfy the task in both responses.
This is one example, not the aggregate verdict. Open Full Results for every judgment.
Details for 10 results were omitted to keep this comment under GitHub's limit. Open the workflow summary or Full Results for the complete breakdown.
To investigate non-passing or warning results, paste this to your AI coding agent:
For PR 1315 in dotnet/skills, download eval artifacts with gh run download 38581149679 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/86f5c1eb7fffc2c8478f2c37ca47b64ea93f15be/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.
⚠️ Session replay telemetry was not published because the auxiliary dotnet/skills-data publisher failed. The evaluation verdicts above remain authoritative; maintainers must repair SKILLS_DATA_TOKEN or the publisher before replay links are available.
The reason will be displayed to describe this comment to others. Learn more.
Could we start dotnet fsi with the payload workspace as its working directory? The PowerShell launcher parses cwd and sets ProcessStartInfo.WorkingDirectory, but this path starts FSI in the inherited hook directory; binlog-preflight.fsx changes directory only after FSI has started. In a repository with global.json, SDK selection can therefore use the wrong directory. The current launcher tests normally make the process directory and payload cwd equal, so a test with different values would cover this case.
The reason will be displayed to describe this comment to others. Learn more.
Could we make the launcher budget account for both FSI startup and the package probe? This loop gives the full helper 8 seconds, while binlog-preflight.fsx can give dotnet package search up to 5 seconds. A cold dotnet fsi startup and script compilation can therefore consume the remaining budget and produce a warning on a healthy host. A shared deadline, or a cold-start regression test with a budget that covers both phases, would make the result reliable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
github-actions
Bot
added
pr-state/ready-for-eval
Summary
The shipped
dnxcommand inherits the workspace's effective NuGet settings. When those feeds do not provideMicrosoft.AITools.BinlogMcp, package resolution fails before the MCP server starts, and the assistant sees only a closed connection rather than the stderr diagnosis.SessionStarthooks for Copilot CLI and Claude Code. A 5-second exact-match, prerelease-inclusive JSON package search runs in the session workspace, inside an 8-second launcher budget and 10-second host timeout.dotnet; successful availability checks remain silent. Serialize only the documented assistant-visible payload, without raw feed details, credentials, SDK banners, or credential-provider warnings.dnxalso applies source mapping and may use cached tools or unlisted versions.The shared parser uses
dotnet fsi, bundled with the .NET SDK, rather than requiring another runtime or JSON parser. Add focused regression coverage and a Windows/macOS/Linux CI matrix without modifying skill evaluations or validator launch overrides.Related issue
Fixes #1312.
Validation
python eng/hooks/test_binlog_preflight.py -v: 15 cases on Windows with .NET SDK 10.0.402; 13 passed and the 2 native Unix cases were skipped. Coverage includes inherited<clear />/mirror settings, exact availability, structured problems despite exit code 0, source mapping, timeouts, missing tools, both host schemas, spaces and Unicode in workspace paths, Windows input encoding, the Windows cleanup race, and orphaned stream-holder cleanup after search exit and launcher timeout.dnxcommand initialized BinlogMcp 3.0.4 and returned 47 MCP tools, includingbinlog_overview.Coverage limitations: Claude Code live-session delivery has not been verified because Claude is not installed in the validation environment. Native macOS/Linux hook execution is covered by the passing CI matrix; live assistant delivery was reverified with Copilot CLI 1.0.97 on Windows. Native Windows Claude requires Git Bash and delegates to the Windows PowerShell launcher. Copilot CLI 1.0.68 did not load the plugin hook in the live check and is not covered. No diagnostic support is claimed for Codex, VS Code, or Cursor.
Checklist
Evaluation-specific checklist items are not applicable: no skill evaluations or validator evaluation logic changed.