Repository navigation
[Doc] Recommend demand and replicaCapacity, with vLLM queries - #34
Merged
Merged
Conversation
The AWS comparison ran Plumb without demand and replicaCapacity, though the sample policy sets them: without them a member's need comes only from unschedulable replicas and a step at a time while saturated, and traffic weights follow ready replicas alone. The signals reference now says so, with vLLM queries for requests in flight and a worked example, and the operations checklist asks for both. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Jungwoo Song <bluayer@gmail.com>
5 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The AWS comparison ran Plumb without
signals.demandandcapacity.replicaCapacity, even though the sample policy sets both. Without them:capacity.stepat a time while saturated or over its latency SLO;The docs did not say this anywhere.
docs/configuration.md(spec.signals): a paragraph on what goes missing without the two fields, vLLM queries (requests in flight as demand, queued per replica as saturation and pressure,replicaCapacityas the requests one replica serves within the SLO), and a worked example: 35 in flight with 1 replica gives 3 more.docs/operations.md: a checklist item asking for both, linking to the example.Not a duplicate
This came from the AWS comparison's configuration. No issue or PR covers it.
How it was tested
Docs only. Checked against the code:
max(pending, ceil(demand / replicaCapacity) − desired), or at leaststepwhile saturated (core.Needed, memberobserve);capacity.stepper step;capacityOf).The vLLM metric names are the ones the docs already cite from
vllm/v1/metrics/loggers.py.Checklist
make test lint verify-generate verify-modpasses (no code changes)git commit -s, DCO)Co-authored-by: Claudetrailer.shadowmode first, and hub decisions are recorded in the decision log (n/a)docs/, chart values, CRD field comments)🤖 Generated with Claude Code