RFC: Prefix-Aware Delta KV Cache Transfer for Disaggregated Prefill/Decode Serving

[liuhuijiayou]

Hi @nvda-mesharma @tedzhouhk and the Dynamo community,

We are the Kuaishou (快手) infrastructure team working on large-scale LLM serving. We've been following the Dynamo disagg roadmap (#2436) closely and would like to implement Enhancement 1 from that roadmap:

"D can tell P how much KV block prefix cache hit it has in its local KV cache so that only the rest of the blocks need to be transferred."

We've done a detailed design analysis and would love to contribute this to the project. This RFC describes the problem, root causes, and a phased implementation plan.

---

Summary

In disaggregated prefill/decode serving, when a Decode worker already has part of the request's prefix KV cache, the Prefill worker should transfer only the missing KV blocks — not all of them. This significantly reduces KV transfer latency and bandwidth, especially for long-context and shared-prefix workloads.

---

Motivation

The Problem

Currently, Dynamo's disagg mode transfers all KV cache for every request — even if the Decode worker already has an identical prefix cached.

A typical example:

System Prompt (2048 tokens) + User Input (100 tokens) = 2148 tokens

Decode worker D1 already has the 2048-token System Prompt KV cached
(from a previous request with the same system prompt routed to D1)

Current behavior:
  P computes KV for all 2148 tokens → transfers all 2148 tokens to D1
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                       2048 tokens are redundantly transferred!

Expected behavior:
  P computes KV for all 2148 tokens → transfers only the last 100 tokens to D1
                                       D1 uses local cache + transferred data

Scale of Impact

Using Llama-70B (GQA, 8 KV heads, 128 head_dim, FP16) as an example:

Scenario	Prompt Length	D Cached	To Transfer	Full Transfer	Delta Transfer	Savings
Shared System Prompt	4096 tokens	3072	1024	640 MB	160 MB	75%
Multi-turn dialogue	8192 tokens	6144	2048	1.28 GB	320 MB	75%
RAG (shared docs)	32768 tokens	28672	4096	5.12 GB	640 MB	87.5%
Long Context	131072 tokens	114688	16384	20.48 GB	2.56 GB	87.5%

On a 100Gbps RDMA network:

• Full transfer for 131K context: ~1.6 seconds
• Delta transfer (only 16K new tokens): ~0.2 seconds
• TTFT reduced by ~1.4 seconds

---

Root Cause Analysis: Three Broken Links

There are 3 broken links in the current codebase that prevent delta transfer:

Broken Link 1: Decode Router has no awareness of D worker's cache

In lib/llm/src/kv_router/prefill_router/types.rs, build_decode_router_override sets:

RouterConfigOverride {
    overlap_score_weight: Some(0.0),    // ← forced to 0, no KV cache lookup
    assume_kv_reuse: Some(false),       // ← uses random hash, no tracking
    track_prefill_tokens: Some(false),
    ...
}

overlap_score_weight = 0 causes the Decode Router's Indexer to be set to None (lib/llm/src/kv_router/indexer.rs):

if kv_router_config.overlap_score_weight == 0.0 {
    return Ok(Self::None);  // no tracking of D worker cache state at all
}

KV event subscription is also skipped (lib/kv-router/src/scheduling/config.rs):

pub fn should_subscribe_to_kv_events(&self) -> bool {
    self.use_kv_events && self.overlap_score_weight > 0.0
    // overlap_score_weight == 0 → false → no subscription
}

Result: The Decode Router is completely unaware of D worker KV cache state.

Broken Link 2: KV events don't distinguish "locally computed" vs "received via transfer"

In lib/kv-router/src/zmq_wire.rs, KV events only have 3 types:

• BlockStored { block_hashes, token_ids, ... }
• BlockRemoved { block_hashes }
• AllBlocksCleared

There's no field indicating whether a block was computed locally by prefill or received via transfer from a P worker.

Broken Link 3: PrefillRouter doesn't pass D's cache info to P

In lib/llm/src/kv_router/prefill_router/mod.rs, the generate() flow:

1. Select P worker
2. Select D worker
3. Build prefill request → send to P
4. Build decode request → send to D

In step 3, the prefill request contains no information about "which blocks D already has cached." P has no knowledge of D's cache state and can only perform full transfer.

---

Proposed Design

Core Idea

1. Select D first, then P (or at least determine D's cache state first)
2. Enable the Decode Router's Indexer to track D worker KV cache
3. Pass D's cache hit info to the P worker
4. P worker only transfers the blocks D is missing

Phase 1: Enable Decode Router Indexer (fix Broken Links 1 & 2)

Change 1a: Don't force overlap_score_weight=0 for Decode Router

// lib/llm/src/kv_router/prefill_router/types.rs
pub(super) fn build_decode_router_override(...) -> RouterConfigOverride {
    RouterConfigOverride {
        overlap_score_weight: Some(decode_overlap_weight),  // new param, default > 0
        assume_kv_reuse: Some(true),   // transferred KV should also be tracked
        track_prefill_tokens: Some(false),
        ...
    }
}

Change 1b: Ensure D worker's transferred KV triggers BlockStored events

• vLLM: when prefix_caching is enabled, transferred blocks enter prefix cache → events fired ✓
• SGLang: need to verify if RadixAttention tracks transferred blocks
• TRT-LLM: needs verification

Phase 2: Reorder scheduling — D before P (fix Broken Link 3)

Change 2a: PrefillRouter queries Decode Router overlap before selecting P

// lib/llm/src/kv_router/prefill_router/mod.rs — generate() method
let decode_overlap = next.query_overlap(&request)?;
// decode_overlap = {
//     best_decode_worker: D1,
//     overlap_blocks: 50,        // D1 has 50 blocks cached
//     overlap_block_hashes: [...] // which blocks are cached
// }

Change 2b: Attach D's cache info to the prefill request

prefill_req.decode_cache_info = Some(DecodeCacheInfo {
    decode_worker_id: D1,
    cached_prefix_blocks: 50,
});

Phase 3: P Worker Implements Delta Transfer

# In vLLM disagg backend
def transfer_kv_to_decode(self, request, kv_cache):
    cached_blocks = request.decode_cache_info.cached_prefix_blocks
    delta_kv = kv_cache[cached_blocks:]  # only transfer missing blocks
    self.nixl_transfer(delta_kv, request.bootstrap_info)

D worker stitches together local cache + transferred data:

def receive_kv(self, request, delta_kv):
    local_prefix = self.prefix_cache.get(request.prefix_hash)
    full_kv = concat(local_prefix, delta_kv)
    self.schedule_decode(request, full_kv)

Phase 4 (Optional): P reads from D's cache

This is Enhancement 2 from the roadmap: when D has more cached blocks than P, P can read from D to reduce its own compute. We propose this as a follow-up RFC.

---

Implementation Plan

Step	Files	Change
1	lib/llm/src/kv_router/prefill_router/types.rs	Don't force overlap_score_weight=0 for Decode Router
1	lib/kv-router/src/scheduling/config.rs	Add decode_overlap_score_weight config param
2	Backend KV event publisher	Ensure transferred blocks also fire BlockStored
2	lib/kv-router/src/zmq_wire.rs	Optional: add source: KvSource field (Local/Transferred)
3	lib/llm/src/kv_router/prefill_router/mod.rs	Query D overlap before selecting P
3	lib/llm/src/protocols/common/preprocessor.rs	Add DecodeCacheInfo struct
3	lib/llm/src/kv_router/push_router.rs	Expose query-only overlap interface
4	components/src/dynamo/backends/vllm/	Parse decode_cache_info, execute delta transfer
4	components/src/dynamo/backends/sglang/	Same as vLLM
4	NIXL layer	Verify offset-based transfer support

---

Open Questions

1. Scheduling order latency: "D first, then P" adds one overlap query latency (~10-50μs). Can this be parallelized with P selection in the bootstrap path?

2. Cache eviction during transfer: D may evict blocks under memory pressure while P is transferring. We need a "lock" mechanism or at least detection.

3. Bootstrap path compatibility: Currently P and D start almost simultaneously. Delta transfer requires knowing D's cache state before P starts transferring — this may require adjusting bootstrap timing.

4. Multiple D worker candidates: If D0 has 50 blocks cached and D1 has 80, should we prefer D1 (more cache but possibly higher load)? With overlap_score_weight > 0, the existing cost function already handles this balance.

5. Block hash consistency: Are block hashes computed by P and D consistent for the same token sequence? In Dynamo, block hashes are computed by the Router side via XXH3 (independent of the backend engine), so consistency is guaranteed.

---

Expected Impact

Scenario	Expected TTFT Improvement	Expected Bandwidth Savings
Shared System Prompt (2-4K tokens)	20-40%	50-75%
Multi-turn dialogue (8-16K context)	30-50%	60-80%
RAG shared docs (32K+ context)	50-70%	75-90%
Long Context (128K tokens)	60-80%	80-90%

Prerequisite: D worker prefix cache hit rate must be high enough. This depends on the Decode Router routing similar requests to the same D worker — which Phase 1 (enabling the Decode Router Indexer) directly addresses.

---

We are planning to start with Phase 1 (enabling the Decode Router Indexer) and would appreciate feedback on:

• Whether this design aligns with the team's direction
• Any concerns about the scheduling reorder in Phase 2
• Which backend (vLLM/SGLang) would be preferred to prototype Phase 3 first

Happy to open individual issues for each phase once we get alignment on the overall direction. Looking forward to the discussion!