feat(runpod): allow requesting a minimum host vCPU / RAM for pods - #76
Open
vohonen wants to merge 1 commit into
Open
feat(runpod): allow requesting a minimum host vCPU / RAM for pods#76vohonen wants to merge 1 commit into
vohonen wants to merge 1 commit into
Conversation
Some workloads are CPU-bound rather than VRAM-bound. RL environments that
execute model-generated code for every rollout are the motivating case: a
4-GPU pod that lands on a thin host runs materially slower, and today there is
no way to express that preference.
RunPod's deploy mutation already accepts `minVcpuCount` and `minMemoryInGb`,
and the runpod SDK exposes them on create_pod, but OpenWeights never passed
them through.
Adds:
- OW_RUNPOD_MIN_VCPU_COUNT / OW_RUNPOD_MIN_MEMORY_GB env defaults, following
the existing OW_RUNPOD_MIN_DOWNLOAD / OW_RUNPOD_DATA_CENTER_ID pattern
- min_vcpu_count / min_memory_in_gb args on start_worker (also exposed as
fire CLI flags) and on the RunpodProvider
- --min-vcpu / --min-memory-gb flags on `ow ssh`
- tests/test_min_vcpu_passthrough.py
Semantics: both are absolute per-pod values, matching RunPod, and act as host
*filters* rather than allocations. They exclude hosts that would give the pod
less, but never grant more than the host offers, and the pod may well receive
more than requested. Setting them too high shrinks the eligible host pool and
makes provisioning failures more likely, so they are opt-in and unset by
default - when neither is provided the fields are omitted entirely and
behaviour is unchanged.
Verified against the live RunPod API:
- a deliberately impossible request (512 vCPU on a 1-GPU pod) is rejected
with "no longer any instances available with the requested specifications",
confirming the constraint reaches the scheduler rather than being dropped
- a satisfiable request provisions normally; a pod asked for >=8 vCPU came
back with 42, confirming floor-not-allocation semantics
- the pricing API shows the filter changing which host class is offered
(4x H200: 96 vCPU unfiltered vs 128 vCPU at minVcpuCount>=97, same price)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Lets a pod request a minimum host vCPU count and RAM, via
OW_RUNPOD_MIN_VCPU_COUNT/OW_RUNPOD_MIN_MEMORY_GB,start_worker(min_vcpu_count=..., min_memory_in_gb=...), andow ssh --min-vcpu / --min-memory-gb.Why
Not every workload is VRAM-bound. The case that prompted this is an RL environment that executes
model-generated Python for every rollout, so throughput depends on how many grader subprocesses
can run concurrently. A multi-GPU pod that lands on a thin host runs materially slower, and there
is currently no way to express that preference —
requires_vram_gbandallowed_hardwarebothonly speak to GPUs.
RunPod's deploy mutation already accepts
minVcpuCountandminMemoryInGb, and therunpodSDKexposes them on
create_pod; OpenWeights just never passed them through.Semantics
Both values are absolute per-pod (not per-GPU), matching RunPod, and act as host filters rather
than allocations: they exclude hosts that would give the pod less, but never grant more than the
host offers, and a pod may well receive considerably more than it asked for.
They are opt-in and unset by default. When neither is provided the fields are omitted from the
mutation entirely and behaviour is byte-identical to today.
The tradeoff worth documenting: a tighter filter shrinks the eligible host pool. On the inventory
I checked, 4x H200 went from
MediumtoLowstock when filtered, so provisioning can takelonger or fall into the existing failure-cooldown ladder.
Design notes
Implemented as environment defaults plus optional arguments, following the existing
OW_RUNPOD_MIN_DOWNLOAD/OW_RUNPOD_MIN_UPLOAD/OW_RUNPOD_DATA_CENTER_IDpattern instart_runpod.py, so it needs no database migration and no change to the job schema. Explicitarguments take precedence over the environment defaults.
A per-job version (alongside
requires_vram_gb) would arguably be nicer, but that needs a newcolumn on
jobsand a Supabase migration. Happy to follow up with that if you'd prefer it — thisseemed like the smaller first step.
Verification
Unit tests in
tests/test_min_vcpu_passthrough.pycover forwarding, the unset default, theenvironment-variable path, and argument precedence. Existing
tests/test_runpod_hardware_registry.pystill passes.
Checked against the live RunPod API as well:
instances available with the requested specifications" — a scheduling error, not a schema
error, so the constraint genuinely reaches placement rather than being silently dropped.
>=8vCPU came back with 42,confirming the floor-not-allocation semantics described above.
unfiltered and 128 vCPU at
minVcpuCount>=97, at the same $14.36/hr.Not covered
No change to the cluster manager's job-to-hardware matching; workers still advertise and match on
hardware_typeexactly as before. This only affects pod provisioning.