NVIDIA Transfers 32K KV Caches 25x Faster

NVIDIA researchers published a cross-model KV cache transfer method on August 21, 2026, transferring a 32,768-token cache from Qwen3 14B to Qwen3 32B in 277.6 milliseconds. Recomputing the target model’s context took 6,975.3 milliseconds, producing a reported 25.1x latency improvement for a model handoff.

The technique targets a specific cost in long-running AI agents. When an orchestrator routes routine work to a smaller model and complex reasoning to a larger one, the receiving model cannot directly use the previous model’s KV cache. It normally has to prefill the entire conversation, including prior tool calls and intermediate context, before generating the next token.

For agents with long sessions, this repeated prefill adds latency and GPU work at every escalation. The method lets the target model inherit a transformed cache instead.

Ridge Regression Maps Memory Between Models

The central component is a closed-form ridge mapper, trained offline for a particular source and target model pair. Ridge regression maps the source model’s key and value tensors into the coordinate system and tensor dimensions expected by the target model, using algebra rather than a gradient-based neural network.

The approach depends on a strong linear relationship between KV representations from matched models in the same family. NVIDIA tested transfers across model sizes including Llama 3.1 8B to 70B and Qwen3 variants.

Different-sized models have different layer counts, which prevents a simple one-to-one layer copy. A cross-layer selector instead combines the most predictive source layers for each target layer. In the Qwen3 14B to 32B experiment, one source layer explained 56% of target key variance and 32% of value variance. Selecting multiple source layers raised those figures to 79% and 65%.

RoPE Handling Enables Long Contexts

Position information introduces another obstacle. The mapper strips Rotary Position Embedding, or RoPE, from key vectors before applying the transformation, then re-applies RoPE afterward.

This separation allows calibration on sequences of about 1,000 tokens to generalize to longer contexts. The reported 32K Qwen3 transfer indicates why the detail matters for production agents, where the cache often includes many turns, retrieved documents, and tool results.

The transfer speedup ranged from 2.7x to 25x across the tested model pairs. Accuracy remained between 73% and 98% of the target model’s standalone prefill accuracy, depending on the pair and task.

Transfer characteristic	Qwen3 14B to Qwen3 32B
Cache length	32,768 tokens
Linear mapping latency	277.6 ms
Standard re-prefill latency	6,975.3 ms
Reported speedup	25.1x
Offline calibration	47 to 87 minutes
Calibration compute cost	Under $50

Matched Models and Calibration Shape the Tradeoff

The method requires matched-KV pairs. Source and target models must share key-value head counts and per-head dimension configurations within the same architectural family. Arbitrary model handoffs remain outside the demonstrated scope.

Calibration also affects quality. Reasoning-heavy tasks can expose errors in directional attention weights when the mapper is poorly calibrated. A lightweight neural fuser used after calibration restored performance to as much as 95.5% of the target model’s capability in the reported Llama 3.1 transfer results.

If you build multi-model routing or long-running AI agents, treat the mapper as a model-pair optimization rather than a universal cache format. Select compatible model families, calibrate on the tasks your router actually serves, and measure handoff quality alongside latency. For systems already using multi-agent coordination patterns, preserving KV state could make escalation frequent enough to use as a control policy instead of an expensive exception.