Add A/B experiment for Copilot PR NLP prompt style - #63159
Merged
pelikhan merged 2 commits intoSep 24, 2026
Merged
Conversation
8 tasks
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Add experiment campaign for A/B testing prompt_style
Add A/B experiment for Copilot PR NLP prompt style
Sep 24, 2026
pelikhan
marked this pull request as ready for review
September 24, 2026 11:10
pelikhan
deleted the
copilot/ab-advisor-experiment-campaign-prompt-style
branch
September 24, 2026 11:11
Copilot stopped reviewing on behalf of
pelikhan due to an error
September 24, 2026 11:11
Contributor
There was a problem hiding this comment.
Note
Copilot could not run the full agentic suite for this review because it was automatically requested on a bot-authored pull request. Request a review from Copilot under Reviewers to retry with the full agentic suite. Improved support for bot-authored pull requests is coming soon.
Copilot review overview
Review effort: Lite
Findings: 2
Open (3)
continue-on-error: trueand the push step runs withif: always(),… · New{{#if ...}}conditionals typically don’t support==expressions (they usually evaluate… · New
What changed in this PR
Adds an A/B experiment to the Copilot PR Conversation NLP Analysis workflow to compare “concise” vs “structured” prompt wording, and persists experiment state across runs.
Changes:
- Introduces an
nlp_prompt_styleexperiment with metrics/guardrails and adds a prompt variant section to the workflow prompt. - Adds steps to restore/pick variants, propagate the chosen variant via env/outputs, and upload/download an experiment artifact.
- Adds a job to push experiment state back to a dedicated branch and adds an eval to verify the prompt style was applied.
| File | Description |
|---|---|
| .github/workflows/copilot-pr-nlp-analysis.md | Defines the experiment and injects variant-specific prompt instructions into the rendered prompt. |
| .github/workflows/copilot-pr-nlp-analysis.lock.yml | Implements experiment state restore/pick, passes variant values through the run, adds eval question, and pushes experiment state to git. |
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+2434
to
+2436
| - name: Push experiment state to git | ||
| id: push_experiments_state | ||
| if: always() |
Comment on lines
+113
to
+121
| {{#if experiments.nlp_prompt_style == 'concise' }} | ||
| ## Prompt Style Variant: concise | ||
|
|
||
| Use compact instruction execution. Prioritize brevity, top findings, and direct recommendations. Keep narrative commentary short, highlight the 3-5 highest-signal insights, and avoid exhaustive detail unless it is needed to explain a material result or edge case. | ||
| {{else}} | ||
| ## Prompt Style Variant: structured | ||
|
|
||
| Use explicit phase-based execution. Work through data loading, preprocessing, NLP analysis, visualization, report construction, and final verification in order. In the final discussion, preserve the required output sections and verify each success criterion with checklist-style constraints before posting. | ||
| {{/if}} |
| @@ -1,4 +1,4 @@ | |||
| # gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"1ab34f5a7a77581cbce29d8016d0290a140a2092c60cff34797f186ba099e2d8","body_hash":"098697f0b61df6bbd7ded8a00b8138e183b240777d48c8e0396fdb66bcaa53df","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.87","copilot-sdk":"1.0.13"}} | |||
| # gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"5dc0509cf7cc032e3e2756393dd01556de6f00fc11114a1ec42489ecae07d704","body_hash":"faf9f5d4f3771d524b5163b15a330bf1a31430d94913fea4149149e03c682fd1","strict":true,"agent_id":"copilot","engine_versions":{"copilot":"1.0.87","copilot-sdk":"1.0.13"}} | |||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Copilot PR NLP analysis workflow needs an experiment campaign to compare concise vs. structured prompt execution. The campaign measures report usefulness via existing evals while tracking runtime and output guardrails.
Experiment configuration
nlp_prompt_stylewithconciseandstructuredvariants.Evaluation coverage
prompt_style_appliedto verify the assigned prompt style is reflected in output.Prompt variant wiring
{{#if experiments.nlp_prompt_style == 'concise' }} Use compact instruction execution... {{else}} Use explicit phase-based execution... {{/if}}copilot-pr-nlp-analysis.lock.ymlso experiment assignment, eval propagation, and experiment state handling are included.