Skip to content

Reduce manual work from standardization of accelerator and host processor names #886

Open
Open
Reduce manual work from standardization of accelerator and host processor names#886
Assignees
pavanky
Labels
Fix for v6.1

Description

@ShriyaRishab
ShriyaRishab
opened on Jun 23, 2026

Each round, after submission, we create a mapping from the submitted accelerator names to the ones that hardware providers like AMD/NVIDIA prefer and manually update these to ensure we have standard names.
These should be fixed before submission and should not be left until the review period. For all previously submitted accelerators or host processors, submitters should be forced to use the same names as before for consistency. For new ones, we need this to be decided before the submission deadline.

Activity

  1. ShriyaRishab
    added
    Fix for v6.1
    on Jun 23, 2026

  2. ShriyaRishab commented on Jul 2, 2026

    @ShriyaRishab
    ShriyaRishab
    on Jul 2, 2026
    ContributorAuthor

    WG notes from 7/2/2026:

    1. Should we create a pre-generated list from all recent rounds and include that in the package checker. So if you use a different name, you get a WARNING and recommendations on a new value
    2. Disallow changes to the system.json, especially these two values, post the submission deadline. So force hardware providers to decide and share this with their partners before submission
    3. Have automation post submission that generates a mapping and fixes typos if any
    4. Inference does not standardize actually, but its better to do it.

  3. ShriyaRishab commented on Aug 20, 2026

    @ShriyaRishab
    ShriyaRishab
    on Aug 20, 2026
    ContributorAuthor

    @pavanky any update on the system.json automation which will solve this?

  4. pavanky commented on Aug 27, 2026

    @pavanky
    pavanky
    on Aug 27, 2026

    https://github.com/mlcommons/mlperf-automations/blob/main/script/get-mlperf-multi-node-system-info/info.md

  5. ShriyaRishab commented on Aug 27, 2026

    @ShriyaRishab
    ShriyaRishab
    on Aug 27, 2026
    ContributorAuthor

    WG meeting 8/27/26:

    1. Pavan to test without venv and share simple instructions on how to run this in isolation. Write-up will be ready by 8/27/26 end of the day
    2. Currently supports NVIDIA and AMD hardware. Pavan to work with Ran Ran from Google to add Google support
    3. amd-smi names aren't looking right. Pavan to share with Miro offline with the full logs over email to figure this out
    4. Should this be part of the submission run or separately generate system.json? This cannot be easily run with the submission since it runs an additional binary so its not as lightweight as MLLOGGER.
    5. We can make this optional for v6.1 and required going forward.
  6. ShriyaRishab
    assigned
    pavanky
    on Aug 27, 2026

  7. pavanky commented on Aug 27, 2026

    @pavanky
    pavanky
    on Aug 27, 2026

    Here are the instructions I followed to install and test on a B200 system:
    https://gist.github.com/pavanky/908bf975995482c46da9b138fcd9fef6

  8. ShriyaRishab commented on Sep 3, 2026

    @ShriyaRishab
    ShriyaRishab
    on Sep 3, 2026
    ContributorAuthor

    WG meeting discussion 9/3/26:

    1. One concern is that the full MLPerf automation repo and dependencies
    2. What about Google hardware support because the tool currently doesn't support it? Google to drive this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Fix for v6.1

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions