DEP: Revision-aware DGD rollout and release groups

[AllenXu93]

Summary

Add revision-aware rollout to DynamoGraphDeployment (DGD). The proposal adds per-component paused and partition controls, then extends the rollout model with release groups so a candidate router routes only to candidate workers while the stable router continues to route only to stable workers. The Dynamo Operator manages lifecycle and calls an authenticated router management API to install the revision-routing policy. Gateway API selects a router revision through ordinary revision-specific InferencePool targets; Gateway itself does not interpret DGD revision IDs.

Motivation

The current managed worker rollout identifies worker generations with a content hash, but it does not provide a durable DGD rollout contract for pausing a component, retaining a stable partition, assigning a graph-wide revision identity, or preserving router-to-worker affinity during a simultaneous update.

A Kubernetes Service can briefly contain old and new Pods, but it cannot route a header-selected request to one exact router revision. It also cannot ensure that a new router selects a compatible new worker, prefill, and decode generation. These gaps make an in-place canary unsafe for protocol, model, engine, or KV-transfer changes that require a coherent graph revision.

Proposal

Scope and lifecycle

Introduce a revision-aware DGD rollout for single-node, Deployment-backed components. The first supported lifecycle retains at most two serving revisions: stable and candidate. It excludes Grove and other multinode workload providers until their lifecycle coordination can satisfy the same invariants.

Every candidate snapshot receives an Operator-generated, content-addressed DGD revision ID. The Operator labels every participating DCD and Pod with the revision ID and the release-group name. The existing nvidia.com/dynamo-worker-hash remains a worker implementation detail; it is not the DGD revision identity used by Gateway, router policy, or user-visible status.

Per-component pause and partition

Add a stable DGD spec API. The initial annotation prototype is nvidia.com/dgd-component-rolling-update, but it is experimental and must not become the long-term public contract.

spec:
  rollout:
    components:
      - name: router
        paused: true
        partition: 7
      - name: prefill
        paused: true
        partition: 7
      - name: decode
        partition: 3

name matches spec.components[].name and is valid only for a revision-managed component.

• paused: true freezes that component at its observed stable/candidate replica split. It does not remove a candidate that already exists.
• partition is the minimum number of desired replicas retained on the stable revision for that component. It is a non-negative integer no larger than the component's desired replica count.
• partition controls a replica count, not an individual Pod. Deployment Pods have no stable ordinal suitable for selecting a particular replica.
• A component without a rollout entry follows the default rollout policy. partition is rejected with Recreate strategy.

While paused: false, the Operator may reconcile toward the partition boundary but must not move past it. Operators pause a component before a manual verification gate, then unpause or adjust partition to proceed.

Release groups and revision affinity

Add a DGD release group that declares components which must use one coherent revision for a request:

spec:
  rollout:
    releaseGroups:
      - name: serving
        components: [router, prefill, decode]
        revisionAffinity: required

For a simultaneous router and worker update, the Operator creates stable and candidate instances of all changed members. The invariant is:

router stable    -> worker/prefill/decode stable
router candidate -> worker/prefill/decode candidate

For a router-only update, the candidate router initially uses the stable worker revision. This exception requires an explicit compatibility declaration. In a disaggregated prefill/decode graph, the router selects prefill and decode as one policy decision; mixed prefill/decode revisions are rejected unless the DGD declares the engine protocol and KV-transfer contract compatible.

The Operator may create candidate worker capacity before candidate router capacity, but it must not direct a router to a candidate worker until that worker is Dynamo-routable. It must not send Gateway traffic to a candidate router until the router acknowledges the active policy and is routable.

Router management API

Add an authenticated internal router management API. It is called by the Operator, not exposed as a public inference header:

PUT /v1/control/revision-routing-policy

{
  "policyVersion": "42",
  "releaseGroup": "serving",
  "default": {
    "workerRevision": "r-stable"
  },
  "routerRevisionBindings": [
    {
      "routerRevision": "r-candidate",
      "prefillRevision": "r-candidate",
      "decodeRevision": "r-candidate"
    }
  ]
}

The API is idempotent by policyVersion, validates that each referenced revision is registered and routable, applies the policy atomically to new requests, and returns an acknowledgement containing the accepted policy version. In-flight requests retain their selected revision through normal completion, cancellation, or retry boundaries. Authentication and authorization use the Operator's Kubernetes identity or a workload-scoped mTLS identity.

The Operator records the acknowledged policy version in DGD status. It removes a revision from a router policy before scaling that revision to zero or deleting its DCDs.

Gateway integration

The Operator publishes router revision targets in DGD status:

status:
  rollout:
    stableRevision: r-stable
    candidateRevision: r-candidate
    revisions:
      - id: r-stable
        router:
          routableReplicas: 7
          inferencePool: example-router-r-stable
      - id: r-candidate
        router:
          routableReplicas: 1
          inferencePool: example-router-r-candidate

Gateway API remains generic. An HTTPRoute references the published stable or candidate InferencePool; a header rule such as X-Dynamo-EPP-Canary may select the candidate target, and a default rule may use backend weights. The header has no special meaning inside Dynamo.

An optional DynamoRevisionRollout controller may own a dedicated HTTPRoute and translate desired traffic policy into Gateway API resources. It must not patch a user-owned route. Manual Gateway users can consume DGD status directly.

Gateway route acceptance is part of the lifecycle: before deleting a revision, set its route weight to zero or remove its header match, wait for Accepted=True and ResolvedRefs=True, then remove the router policy binding, drain in-flight requests, and finally scale down the revision.

Status and safety invariants

DGD status reports revision IDs, each component's desired and routable replica counts, release-group policy acknowledgement, and revision-specific Gateway targets. The Operator uses Dynamo routability rather than Kubernetes Pod Ready alone for promotion and drain decisions.

The rollout enforces these invariants:

1. A stable revision remains available at or above every active component partition.
2. A candidate router is not a Gateway target until it is routable and has acknowledged a valid worker-revision policy.
3. A router may not select a worker revision outside its release group unless a compatibility declaration explicitly permits it.
4. A revision remains present while it receives Gateway traffic, is selected by a router policy, or has in-flight requests that require it.
5. Rollback first removes candidate traffic and policy bindings, then drains and deletes candidate capacity.

Delivery phases

1. Land the component-level paused and partition API, validation, status, and Deployment-backed worker reconciliation. Keep the annotation prototype only while the spec API is introduced.
2. Add DGD revision snapshots, revision labels, revision-specific router Services and InferencePool targets, and routability gates.
3. Add release groups and the router management API, including policy acknowledgement and stable/candidate affinity.
4. Add optional DynamoRevisionRollout Gateway automation and explicit attach/detach deletion handoff.
5. Define compatibility contracts and evaluate Grove and multinode providers.

Alternate Solutions

Use a separate DGD for every revision

This works with today's Gateway canary mechanism and remains a valid blue/green strategy. It duplicates the graph and cannot provide in-place component partitions or Operator-enforced router-to-worker affinity.

Use only Kubernetes rolling updates

This permits old and new Pods behind one Service but cannot select a precise router revision by request header or guarantee a coherent router/worker revision group.

Let a public request header select workers

This exposes internal topology, permits unsafe prefill/decode combinations, and bypasses Operator readiness checks. Worker revision selection belongs to the authenticated router control plane.

Requirements

• Add unit and envtest coverage for component pause, partition bounds, resume, rollback, and rejection of invalid component names and strategies.
• Add router API conformance tests for idempotency, atomic policy replacement, validation, and in-flight request behavior.
• Add end-to-end tests that verify header-selected candidate router traffic reaches only candidate workers and stable traffic reaches only stable workers.
• Publish metrics and status conditions for revision routability, Gateway attachment, router policy acknowledgement, and drain progress.

References

• Internal design record: https://zhishu.zhipin.com/wiki/abVf5ZEnkGO
• Grove rolling-update tracking issue
• Routability gap tracking issue