Skip to content

feat(runpod): allow requesting a minimum host vCPU / RAM for pods - #76

Open
vohonen wants to merge 1 commit into
longtermrisk:mainfrom
vohonen:feat/min-vcpu-count
Open

feat(runpod): allow requesting a minimum host vCPU / RAM for pods#76
vohonen wants to merge 1 commit into
longtermrisk:mainfrom
vohonen:feat/min-vcpu-count

Conversation

@vohonen

@vohonen vohonen commented Aug 17, 2026

Copy link
Copy Markdown

What

Lets a pod request a minimum host vCPU count and RAM, via OW_RUNPOD_MIN_VCPU_COUNT /
OW_RUNPOD_MIN_MEMORY_GB, start_worker(min_vcpu_count=..., min_memory_in_gb=...), and
ow ssh --min-vcpu / --min-memory-gb.

Why

Not every workload is VRAM-bound. The case that prompted this is an RL environment that executes
model-generated Python for every rollout, so throughput depends on how many grader subprocesses
can run concurrently. A multi-GPU pod that lands on a thin host runs materially slower, and there
is currently no way to express that preference — requires_vram_gb and allowed_hardware both
only speak to GPUs.

RunPod's deploy mutation already accepts minVcpuCount and minMemoryInGb, and the runpod SDK
exposes them on create_pod; OpenWeights just never passed them through.

Semantics

Both values are absolute per-pod (not per-GPU), matching RunPod, and act as host filters rather
than allocations
: they exclude hosts that would give the pod less, but never grant more than the
host offers, and a pod may well receive considerably more than it asked for.

They are opt-in and unset by default. When neither is provided the fields are omitted from the
mutation entirely and behaviour is byte-identical to today.

The tradeoff worth documenting: a tighter filter shrinks the eligible host pool. On the inventory
I checked, 4x H200 went from Medium to Low stock when filtered, so provisioning can take
longer or fall into the existing failure-cooldown ladder.

Design notes

Implemented as environment defaults plus optional arguments, following the existing
OW_RUNPOD_MIN_DOWNLOAD / OW_RUNPOD_MIN_UPLOAD / OW_RUNPOD_DATA_CENTER_ID pattern in
start_runpod.py, so it needs no database migration and no change to the job schema. Explicit
arguments take precedence over the environment defaults.

A per-job version (alongside requires_vram_gb) would arguably be nicer, but that needs a new
column on jobs and a Supabase migration. Happy to follow up with that if you'd prefer it — this
seemed like the smaller first step.

Verification

Unit tests in tests/test_min_vcpu_passthrough.py cover forwarding, the unset default, the
environment-variable path, and argument precedence. Existing tests/test_runpod_hardware_registry.py
still passes.

Checked against the live RunPod API as well:

  • A deliberately impossible request (512 vCPU on a 1-GPU pod) is rejected with "no longer any
    instances available with the requested specifications"
    — a scheduling error, not a schema
    error, so the constraint genuinely reaches placement rather than being silently dropped.
  • A satisfiable request provisions normally. A pod asked for >=8 vCPU came back with 42,
    confirming the floor-not-allocation semantics described above.
  • The pricing API shows the filter changing which host class is offered: 4x H200 returns 96 vCPU
    unfiltered and 128 vCPU at minVcpuCount>=97, at the same $14.36/hr.

Not covered

No change to the cluster manager's job-to-hardware matching; workers still advertise and match on
hardware_type exactly as before. This only affects pod provisioning.

Some workloads are CPU-bound rather than VRAM-bound. RL environments that
execute model-generated code for every rollout are the motivating case: a
4-GPU pod that lands on a thin host runs materially slower, and today there is
no way to express that preference.

RunPod's deploy mutation already accepts `minVcpuCount` and `minMemoryInGb`,
and the runpod SDK exposes them on create_pod, but OpenWeights never passed
them through.

Adds:
  - OW_RUNPOD_MIN_VCPU_COUNT / OW_RUNPOD_MIN_MEMORY_GB env defaults, following
    the existing OW_RUNPOD_MIN_DOWNLOAD / OW_RUNPOD_DATA_CENTER_ID pattern
  - min_vcpu_count / min_memory_in_gb args on start_worker (also exposed as
    fire CLI flags) and on the RunpodProvider
  - --min-vcpu / --min-memory-gb flags on `ow ssh`
  - tests/test_min_vcpu_passthrough.py

Semantics: both are absolute per-pod values, matching RunPod, and act as host
*filters* rather than allocations. They exclude hosts that would give the pod
less, but never grant more than the host offers, and the pod may well receive
more than requested. Setting them too high shrinks the eligible host pool and
makes provisioning failures more likely, so they are opt-in and unset by
default - when neither is provided the fields are omitted entirely and
behaviour is unchanged.

Verified against the live RunPod API:
  - a deliberately impossible request (512 vCPU on a 1-GPU pod) is rejected
    with "no longer any instances available with the requested specifications",
    confirming the constraint reaches the scheduler rather than being dropped
  - a satisfiable request provisions normally; a pod asked for >=8 vCPU came
    back with 42, confirming floor-not-allocation semantics
  - the pricing API shows the filter changing which host class is offered
    (4x H200: 96 vCPU unfiltered vs 128 vCPU at minVcpuCount>=97, same price)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant