Skip to content

Add JSONL eval metrics support to trajectory builder - #912

Open
Lai-Kai wants to merge 1 commit into
mlcommons:masterfrom
Lai-Kai:feat/recommendation-jsonl-trajectory
Open

Lai-Kai wants to merge 1 commit into
mlcommons:masterfrom
Lai-Kai:feat/recommendation-jsonl-trajectory

Conversation

@Lai-Kai

@Lai-Kai Lai-Kai commented Oct 24, 2026

Copy link
Copy Markdown

The streaming trainer already writes end-of-pass evaluation metrics to .metrics.jsonl, but the trajectory builder currently only reads text logs. Add JSONL input support so users can export and plot those evaluation metrics with the existing trajectory outputs.

JSONL records use global_step for the evaluation axis and filter metrics by the selected task. The last matching record wins for repeated steps, including resumed runs and final evaluation. Invalid JSONL or a missing task returns a clear error. Existing text-log input remains supported, and no new dependencies are required.

Validation:

  • python3 -m unittest discover -s recommendation/scripts/tests -p 'test_build_ne_auc_trajectory.py' -v — 4 tests passed.
  • CLI checks with small synthetic JSONL and text logs verified JSON/CSV contents without matplotlib.
  • Malformed JSONL returned exit code 2 with a line number and created no output files.

@Lai-Kai
Lai-Kai requested review from a team as code owners October 24, 2026 14:02
@github-actions

Copy link
Copy Markdown

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
❌ @Lai-Kai
You can retrigger this bot by commenting recheck in this Pull Request

@Lai-Kai Lai-Kai closed this Oct 24, 2026
@Lai-Kai Lai-Kai reopened this Oct 24, 2026
@Lai-Kai

Lai-Kai commented Oct 24, 2026

Copy link
Copy Markdown
Author

recheck

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant