Skip to content

nvproxy: allow configuring the nvidia-container-cli path #15052

Description

@kvnloo

Problem

nvproxy currently locates the NVIDIA container CLI with:

exec.LookPath("nvidia-container-cli")

in runsc/container/container.go.

This assumes nvidia-container-cli is on the PATH inherited by runsc.

That is not always true for NVIDIA Container Toolkit installations. The toolkit can install its binaries under paths such as:

/usr/local/nvidia/toolkit

or another configured installDir, while runsc intentionally does not parse /etc/nvidia-container-runtime/config.toml.

The result is that an otherwise valid nvproxy configuration can fail before container creation with:

failed to locate nvidia-container-cli in PATH

Related NVIDIA issue:

NVIDIA/nvidia-container-toolkit#1880

Proposed behavior

Allow nvproxy to receive an explicit path to nvidia-container-cli, for example:

--nvproxy-cli-path=/usr/local/nvidia/toolkit/nvidia-container-cli

Suggested compatibility behavior:

  • if the option is unset, preserve the current exec.LookPath("nvidia-container-cli") behavior;
  • if set, use the configured executable directly;
  • return an actionable error if the configured path does not exist or cannot be executed.

The option could be exposed through the existing runsc_config configuration path alongside the other nvproxy settings.

Why here instead of modifying containerd PATH

Changing the parent runtime's PATH has a much larger scope:

  • it changes the environment inherited by every shim/process;
  • systemd Environment=PATH=... replaces rather than extends the existing value;
  • the NVIDIA toolkit would need to manage host service configuration and cleanup;
  • arbitrary install paths require escaping and lifecycle handling.

An explicit nvproxy dependency path keeps ownership at the component that invokes the binary.

Test shape

A small no-GPU test should be sufficient for path resolution:

  1. no configured path -> existing PATH lookup behavior;
  2. configured executable -> exact path is used;
  3. invalid configured path -> clear error.

I can work on the implementation if this direction fits the nvproxy configuration model.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions