Skip to content

[Doc] Recommend demand and replicaCapacity, with vLLM queries - #34

Merged
bluayer merged 1 commit into
mainfrom
docs/recommend-demand-signal
Oct 5, 2026
Merged

bluayer merged 1 commit into
mainfrom
docs/recommend-demand-signal

Conversation

@bluayer

@bluayer bluayer commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

What and why

The AWS comparison ran Plumb without signals.demand and capacity.replicaCapacity, even though the sample policy sets both. Without them:

  • a member's need comes only from unschedulable replicas, plus one capacity.step at a time while saturated or over its latency SLO;
  • how many replicas the load needs is never computed;
  • traffic weights follow ready replicas alone.

The docs did not say this anywhere.

  • docs/configuration.md (spec.signals): a paragraph on what goes missing without the two fields, vLLM queries (requests in flight as demand, queued per replica as saturation and pressure, replicaCapacity as the requests one replica serves within the SLO), and a worked example: 35 in flight with 1 replica gives 3 more.
  • docs/operations.md: a checklist item asking for both, linking to the example.

Not a duplicate

This came from the AWS comparison's configuration. No issue or PR covers it.

How it was tested

Docs only. Checked against the code:

  • the need is max(pending, ceil(demand / replicaCapacity) − desired), or at least step while saturated (core.Needed, member observe);
  • the hub adds at most capacity.step per step;
  • weights fall back to a capacity of 1 per replica (capacityOf).

The vLLM metric names are the ones the docs already cite from vllm/v1/metrics/loggers.py.

Checklist

  • make test lint verify-generate verify-mod passes (no code changes)
  • Every commit is signed off (git commit -s, DCO)
  • AI assistance, if any, is stated here and in commit trailers, and I reviewed every changed line. Written with Claude Code; the commit carries a Co-authored-by: Claude trailer.
  • New behavior works in shadow mode first, and hub decisions are recorded in the decision log (n/a)
  • Strings from other projects (Karpenter, KEDA, Gateway API, model hosts, …) are copied from source with a comment naming it (vLLM metric names, as already cited)
  • Docs updated (docs/, chart values, CRD field comments)

🤖 Generated with Claude Code

The AWS comparison ran Plumb without demand and replicaCapacity, though the
sample policy sets them: without them a member's need comes only from
unschedulable replicas and a step at a time while saturated, and traffic
weights follow ready replicas alone. The signals reference now says so, with
vLLM queries for requests in flight and a worked example, and the
operations checklist asks for both.

Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Jungwoo Song <bluayer@gmail.com>
@bluayer
bluayer merged commit 4510704 into main Oct 5, 2026
5 of 6 checks passed
@bluayer
bluayer deleted the docs/recommend-demand-signal branch October 5, 2026 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant