Let the KV cache arrive at the next GPU in advance

The Trouble With Traffic Spikes Is Not Just Insufficient GPUs

Recently, an inference optimization approach called “Predictive Speculative KV Replication” has sparked discussion among developers. Rather than focusing on how fast a single request can run, it targets a problem closer to production reality: when traffic suddenly surges and the GPU holding a session’s KV Cache is already fully occupied, should new requests be scheduled to another GPU?

Moving them may take advantage of idle compute capacity, but the cache is not there. The server can only rerun prefill, passing thousands or even tens of thousands of historical tokens through the model again. Keeping them on the original node means continuing to wait in the queue. Both options increase time to first token (TTFT); the only difference is whether the time is spent on recomputation or waiting.

Predictive speculative replication offers a straightforward answer: do not wait until the request arrives before moving the cache. When the system determines that a prefix, session, or type of agent task may soon become hot, it asynchronously replicates the corresponding KV Cache to one or more candidate workers in advance. If the predicted surge does occur, the scheduler can route requests directly to nodes that already have the cache.

This is somewhat like CPU branch prediction: guess which path execution will take next and prepare for it in advance. If the guess is correct, the pipeline keeps moving; if it is wrong, some bandwidth and cache space are wasted.

As of July 31, 2026, this still looks more like a system design direction worth validating than a default feature that every inference cluster should enable immediately. It addresses tail latency and scheduling elasticity; it does not create VRAM or throughput out of thin air. In fact, replication increases the total amount of KV data in the cluster. If predictions are inaccurate, the optimization can easily turn into an expensive exercise in moving caches around.

Why KV Cache “Binds” the Scheduler to One GPU

During autoregressive decoding in a Transformer, generating each new token requires reading the Keys and Values corresponding to previous tokens. KV Cache stores these intermediate results, avoiding the need to recompute the entire context at every step.

This reduces computation, but it introduces state into distributed scheduling. Traditional web service instances can generally be treated as approximately stateless: a request can be sent to any machine. LLM requests are different. In multi-turn conversations, long-context RAG, coding agents, and tree-search tasks in particular, the KV associated with the historical prefix often exists on only one worker.

The scheduling system must therefore balance three conflicting goals:

• Cache affinity: Continue sending subsequent requests to the original worker to maximize prefix-cache hit rates.
• Load balancing: Prevent one GPU from developing a long queue while other GPUs remain idle.
• VRAM efficiency: Avoid keeping complete copies of every active session on every GPU.

Sticky session routing solves the first problem but may worsen the second. Random load balancing helps spread requests evenly but destroys cache locality. Migrating KV on demand is more sensible than rerunning prefill, but the migration still occurs on the request’s critical path, so the user must wait for the data transfer to finish.

What predictive replication really changes is the timing: it moves cache migration off the critical path and into the idle interval between two requests. The time during which a user reads the previous response, an agent calls an external tool, or a RAG system queries a database—from tens of milliseconds to several seconds—can all serve as replication windows.

How Large Is a KV Cache?

Whether replication pays off depends first on the size of the KV Cache. For a model using GQA, the KV size of a single sequence can be roughly estimated as:

KV bytes ≈ 2 × number of layers × number of KV heads × dimension per head × number of tokens × bytes per element

The leading 2 represents Key and Value. Consider a BF16 model with 32 layers, 8 KV heads, a head dimension of 128, and a 32K context:

2 × 32 × 8 × 128 × 32768 × 2 bytes ≈ 4 GiB

That is for only one sequence. If the model does not use GQA and instead has more KV heads, the size increases substantially. KV quantization can reduce the amount of data transferred, but it also introduces quantization, dequantization, and kernel adaptation costs.

Therefore, “replicating it in advance” is not as easy as copying an object of a few megabytes in an ordinary caching system. With long contexts, a single replication may move several gigabytes of data while consuming target-GPU VRAM, PCIe or NVLink bandwidth, and cross-node network bandwidth.

This is the key consideration behind the approach: it is not simply trading memory for computation. It is repricing the trade-offs among queueing time, prefill computation, network transfer, and replica VRAM.

If rerunning prefill takes 500 milliseconds, while proactively replicating over idle NVLink takes only 100 milliseconds and does not block online requests, the benefit is clear. Conversely, if the context is short, the network is already congested, or the next request does not arrive for a long time, replication is pure overhead.

“Prediction” Must Answer at Least Three Questions

A practical predictor cannot merely output “this session may return.” It must answer at least three questions.

1. Which KV Cache Is Worth Replicating?

High-value objects usually share several characteristics: long prefixes, high recomputation costs, frequent recent access, and a high probability of subsequent requests. Multi-turn coding agents are a typical use case: the system prompt, repository context, and tool-call history continue to accumulate, while each tool invocation provides a time window for background replication.

By contrast, even if a one-off short Q&A has a high cache-hit rate, replication offers limited benefit because rerunning prefill is already inexpensive.

2. Which Worker Should Receive the Replica?

The target node must have the same model version, a compatible parallelism layout, sufficient VRAM, and a short expected queue. Looking only at current idleness is not enough: by the time replication finishes, the GPU may already be occupied by other requests.

A more practical approach is for the predictor to score candidate workers using queue length, token generation rate, estimated request completion time, and tenant traffic trends. In tensor-parallel deployments, the replication target may not be “one GPU” but an entire parallel group.

3. How Much Should Be Replicated?

Full replication is the simplest option, but also the most expensive. The system can instead replicate only a high-value prefix, allowing the new worker to resume prefill from a checkpoint; alternatively, it can first replicate the initial layers or a subset of KV blocks, then fill in the rest based on popularity.

Tiered caching offers a useful model here: keep the hottest KV data on GPUs, place warm data in CPU memory or a global KV store, and evict cold data entirely. The predictor decides not only whether to replicate, but also the replica’s tier and lifetime.

A simplified scheduling loop might look like this:

for prefix in active_prefixes:
    reuse_prob = predictor(prefix, traffic_window)
    saved_prefill = estimate_prefill_cost(prefix)
    copy_cost = estimate_transfer_cost(prefix, candidate_workers)
    memory_cost = estimate_eviction_cost(prefix)

    expected_gain = reuse_prob * saved_prefill - copy_cost - memory_cost

    if expected_gain > replication_threshold:
        target = choose_compatible_worker(prefix)
        replicate_async(prefix.kv_blocks, target)

The truly difficult part is not writing this policy, but ensuring that every cost estimate remains reliable as traffic patterns change.

It Does Not Replace Existing KV Optimizations

Predictive replication can easily be confused with Prefix Cache, PagedAttention, or stateful routing, but they solve different problems.

| Technology | Primary Problem Addressed | Limitation | |---|---|---| | PagedAttention | Manages KV in pages to reduce internal and external fragmentation | Does not determine which worker should hold the cache | | Prefix Cache / RadixAttention | Reuses existing KV for shared prefixes | The cache may not exist on the target node | | Stateful routing | Continually routes the same session to its original instance | Not flexible enough when a hot instance becomes overloaded | | KV Offload / global storage | Places KV in CPU memory, SSDs, or remote storage | Retrieval may enter the request’s critical path | | Predictive speculative replication | Sends KV to potential target nodes before requests arrive | Incorrect predictions increase bandwidth and VRAM consumption |

A more sensible combination is to use PagedAttention for block-level memory management, Prefix Cache to identify reusable prefixes, a global directory to track KV locations, a predictor to decide whether to create additional replicas, and a scheduler to choose between cache hits and queueing time.

In other words, predictive replication is not a new attention algorithm. It adds another layer of “betting” between the caching system and cluster scheduler.

Its Best Use Case Is Not the Ordinary Chatbot

This approach is more attractive in the following scenarios:

1. Long-context, multi-turn agents: A single session’s KV is large, while tool calls create long idle intervals that can hide replication latency.
2. Enterprise prompt templates: Many requests share system prompts, few-shot examples, or fixed knowledge prefixes, making hot spots relatively predictable.
3. Event-driven traffic spikes: Livestreams, product launches, trading sessions, or batch workflows create pronounced peaks over short periods.
4. Multi-candidate generation and tree search: Multiple branches share a long prefix and can be expanded in parallel on different workers after proactive replication.
5. Disaggregated prefill and decode architectures: The system already has channels for cross-node KV transfer, reducing the engineering effort required to add speculative replicas.

Ordinary short conversations may not be a good fit. When the context is only a few hundred tokens, rerunning prefill may be cheaper than cross-node replication and directory maintenance. Clusters with completely unpredictable requests, frequent model switching, or GPUs that remain close to full VRAM utilization are also unlikely to benefit.

Four Common Engineering Pitfalls

Prediction Hit Rate Is Not Enough on Its Own

A 90% hit rate is not necessarily good. If the predictor replicates many inexpensive short prefixes, even those that are eventually accessed may not save much computation. A more meaningful metric is the ratio between “prefill time effectively saved” and “total bytes replicated.”

At a minimum, the system should monitor:

• Changes in P50, P95, and P99 TTFT;
• Request hit rate and byte hit rate for speculative replicas;
• Network bytes consumed per second of prefill saved;
• The number of normal KV entries displaced by replicas;
• Cross-node link utilization and its impact on online inference;
• The replication amplification factor caused by incorrect predictions.

Replicas Compete With Normal Caches for VRAM

KV replicas are not free shadow data. Space reserved on the target GPU for speculative objects may cause active requests to be preempted or other high-value prefixes to be evicted. Replicas should have short TTLs and be reclaimed first when VRAM pressure rises.

KV Is Not a Universal Format Independent of the Model

A KV Cache is typically tied to the model-weight version, RoPE configuration, KV precision, tensor-parallel sharding, adapters, and even the inference engine’s data layout. Before reusing KV across workers, the system must validate a compatibility fingerprint. Otherwise, the best-case outcome is that the cache cannot be read; the worst is subtly incorrect output.

Multi-Tenant Isolation Cannot Be Added as an Afterthought

KV contains intermediate representations computed from user input. It must not be mistakenly shared across tenants or leaked through a global directory, remote memory, or incomplete replica reclamation. Production systems must include the tenant ID, model version, and session permissions in cache keys, while enforcing isolation across transport links and storage tiers.

Our Assessment: Worth Pursuing, but Not a Free Lunch

The greatest value of predictive speculative KV replication is that it recognizes LLM inference serving is no longer merely a matrix-computation problem. It is a distributed-systems problem involving enormous, dynamic, location-dependent state.

Past optimization efforts focused on maximizing the utilization of a single GPU: continuous batching, FlashAttention, PagedAttention, and quantization. The next and more difficult challenge is ensuring that requests, computation, and state meet at the right time when caches are distributed across dozens or even thousands of workers.

Predictive replication offers a pragmatic path forward: since traffic spikes cannot be eliminated, use the time gaps between requests to move expensive data in advance. It may be particularly effective at improving P99 TTFT, because tail latency is often caused by the combination of queues on hot workers and cache mismatches.

But it does not reduce the theoretical size of KV, nor does it automatically increase each GPU’s decoding speed. More aggressive prediction creates more replicas; more replicas put greater pressure on VRAM and the network. The ultimate benefit depends on whether workload traffic follows predictable patterns and whether the system can accurately determine which is more expensive: recomputation or data movement.

For teams preparing to validate this approach, the safest starting point is not to train a complex predictive model immediately, but to establish a rule-based baseline: replicate only prefixes that exceed a certain token length, have been hit repeatedly in the recent past, and reside on an original worker whose queue is about to exceed a threshold. Use shadow mode to record “what would have happened if replication were enabled,” then gradually introduce a small number of real replicas.

If simple rules cannot improve P99, a machine-learning predictor is unlikely to rescue the scenario. Conversely, if traffic patterns are sufficiently stable, even a sliding-window popularity score combined with queue estimates may capture most of the potential benefit.

References

• vLLM project: Implementations of PagedAttention, continuous batching, and distributed inference that provide a foundation for understanding KV block management and scheduling.
• SGLang project: Includes implementations related to RadixAttention, prefix caching, and high-performance LLM serving.
• Hugging Face Transformers: KV Cache: Introduces different KV Cache strategies, memory usage, and offloading methods in Transformers.
• TensorRT-LLM project: Provides engineering references for multi-GPU inference, KV Cache management, and high-performance serving.

Note: This article draws on the recent original article “Predictive Speculative KV Replication for Bursty LLM Inference.” Due to link-domain restrictions, its external URL is not included in the references.