AWS described bidirectional KV cache transfer in vLLM for multi-turn agent tasks
Bidirectional KV cache transfer in vLLM allows earlier computations to be reused in subsequent conversation turns. In its own measurements, AWS reports time-to-first-token speedups of up to 1.6-fold for Qwen3-32B and up to 3.3-fold for zai-org/GLM-5-FP8.
AWS described an architecture for bidirectional KV cache transfer in vLLM and said it had contributed this feature to the project. When input processing and response generation are handled separately, it allows the prefill node to retrieve previously computed representations of earlier context from the decode node. According to the company, this reduces redundant computation in multi-turn conversations and agent workflows.
The solution extends the NIXL connector and requires a routing component that retains metadata about the location of KV cache blocks between turns. This component does not store the data blocks themselves. After completing a response, the decode node retains the relevant blocks; on the next turn, the prefill node retrieves them if the number of reusable tokens meets the configured threshold, and computes the remaining portion of the input. The vLLM project provides a sample proxy to coordinate this process.
AWS reports measurements using one prefill node and one decode node on Amazon EC2 P5en instances connected via AWS EFA networking. According to the company, this configuration delivered time-to-first-token speedups of up to 1.6-fold for Qwen3-32B and up to 3.3-fold for zai-org/GLM-5-FP8. Results depend on workload and configuration. Details are available in the source article.
Why it matters
For teams running conversational assistants and agents with separate prefill and decode nodes, this provides a concrete way to reduce repeated processing of a growing conversation history. Deployment also requires coordination between turns through a routing component; transfer support in vLLM alone is not sufficient.
Relevant practical impact
What this means
For a business
Companies running their own agents can evaluate this architecture as an option for reducing latency and redundant computation. The reported speedup factors come from measurements by AWS and cannot automatically be applied to a different deployment configuration.
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.