Skip to content

Update rcp_checker.py to handle llama31_8b epochs correctly - #434

Closed
gyulaz-htec wants to merge 1 commit into
mlcommons:masterfrom
gyulaz-htec:patch-1
Closed

gyulaz-htec wants to merge 1 commit into
mlcommons:masterfrom
gyulaz-htec:patch-1

Conversation

@gyulaz-htec

Copy link
Copy Markdown
Contributor

Right now the RCP check fails for llama3 using the reference implementation. It throws an error because it trries to get the epch from epoch_nums instead of sample_count:

  File "/home/gyuzakor/actions-runner/_work/_tool/Python/3.12.11/x64/lib/python3.12/site-packages/mlperf_logging/rcp_checker/rcp_checker.py", line 496, in check_directory
    bs, subm_epochs, benchmark = get_submission_epochs(result_files, version, bert_train_samples)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gyuzakor/actions-runner/_work/_tool/Python/3.12.11/x64/lib/python3.12/site-packages/mlperf_logging/rcp_checker/rcp_checker.py", line 136, in get_submission_epochs
    curr_not_converged, curr_subm_epochs, curr_bs, curr_benchmark = read_submission_file(
                                                                    ^^^^^^^^^^^^^^^^^^^^^
  File "/home/gyuzakor/actions-runner/_work/_tool/Python/3.12.11/x64/lib/python3.12/site-packages/mlperf_logging/rcp_checker/rcp_checker.py", line 93, in read_submission_file
    conv_epoch = json.loads(eval_accuracy_str)["metadata"]["epoch_num"]
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^

Processing llama31_8b logs the same way as llama31_405b logs should solve the problem.

Right now the RCP check fails for llama3 using the reference implementation. It throws an error because it trries to get the epch from epoch_nums instead of sample_count:
```
  File "/home/gyuzakor/actions-runner/_work/_tool/Python/3.12.11/x64/lib/python3.12/site-packages/mlperf_logging/rcp_checker/rcp_checker.py", line 496, in check_directory
    bs, subm_epochs, benchmark = get_submission_epochs(result_files, version, bert_train_samples)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/gyuzakor/actions-runner/_work/_tool/Python/3.12.11/x64/lib/python3.12/site-packages/mlperf_logging/rcp_checker/rcp_checker.py", line 136, in get_submission_epochs
    curr_not_converged, curr_subm_epochs, curr_bs, curr_benchmark = read_submission_file(
                                                                    ^^^^^^^^^^^^^^^^^^^^^
  File "/home/gyuzakor/actions-runner/_work/_tool/Python/3.12.11/x64/lib/python3.12/site-packages/mlperf_logging/rcp_checker/rcp_checker.py", line 93, in read_submission_file
    conv_epoch = json.loads(eval_accuracy_str)["metadata"]["epoch_num"]
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
```
Processing llama31_8b logs the same way as llama31_405b logs should solve the problem.
@gyulaz-htec
gyulaz-htec requested review from a team as code owners September 5, 2025 10:19
@github-actions

github-actions Bot commented Sep 5, 2025

Copy link
Copy Markdown

MLCommons CLA bot:
Thank you very much for your submission, we really appreciate it. Before we can accept your contribution, we ask that you sign the MLCommons CLA (Apache 2). Please use this [Google form] (https://forms.gle/Ew1KkBVpyeJDuRw67) to initiate authorization. If you are from an MLCommons member organization, we will request that you be added to the CLA. If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
❌ @gyulaz-htec
You can retrigger this bot by commenting recheck in this Pull Request

@gyulaz-htec

Copy link
Copy Markdown
Contributor Author

recheck

@ShriyaRishab

Copy link
Copy Markdown
Contributor

@gyulaz-htec - can you please fix the CLA error so we can merge this?
Just follow these instructions if it is your time contributing to MLC repositories - #434 (comment)

@ShriyaRishab

Copy link
Copy Markdown
Contributor

Closing as its a duplicate of #435

@github-actions github-actions Bot locked and limited conversation to collaborators Sep 5, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants