Skip to content

fix(eval): compare aggregate scores over matching tasks - #3258

Open
lewismosciski wants to merge 1 commit into
ag2ai:mainfrom
lewismosciski:fix/eval-diff-paired-scores
Open

lewismosciski wants to merge 1 commit into
ag2ai:mainfrom
lewismosciski:fix/eval-diff-paired-scores

Conversation

@lewismosciski

Copy link
Copy Markdown

Why are these changes needed?

RunResult.diff() averages each run's available scores independently. If baseline scores are (True, False) and current scores are (True, None), it reports 50% to 100% even though the only task graded in both runs is unchanged.

Compute pass rates and numeric means from paired grades, and omit aggregate deltas when no grades overlap. Regression tests cover missing feedback, ungraded feedback, and incompatible score types.

Validation: 295 eval tests passed, 6 live-model tests deselected. Pre-commit checks passed. All 8 new cases failed before the fix.

Related issue number

None; found while checking eval result comparison.

Checks

  • I've included any doc changes needed for https://docs.ag2.ai/. The aggregate docstring now describes paired grades.
  • I've added tests (if relevant) corresponding to the changes introduced in this PR.
  • I've made sure all auto checks have passed.

AI assistance

  • I understand the changes in this PR and can explain them in my own words.
  • I have verified that the PR description accurately reflects the actual diff.
  • If AI assistance was used, I reviewed, tested, and validated the generated code/text before submitting.

@github-actions github-actions Bot added the area:eval Offline evaluation framework (ag2/eval) label Sep 18, 2026
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@vvlrff

vvlrff commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

@lewismosciski Hello! Please sign the CLA

@lewismosciski

Copy link
Copy Markdown
Author

@lewismosciski Hello! Please sign the CLA

Already signed the CLA. Thanks!

@vvlrff

vvlrff commented Sep 20, 2026

Copy link
Copy Markdown
Collaborator

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:eval Offline evaluation framework (ag2/eval)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants