Amazon SageMaker AI gains KV cache-aware routing for real-time LLM inference
AWS has introduced a new KV cache-aware routing strategy in Amazon SageMaker AI for real-time endpoints, which routes LLM requests to instances with actually available KV cache, aiming to improve the efficiency of multi-turn conversations and long shared prompts.
Amazon Web Services has introduced a new routing strategy, KV cache-aware routing (KV-aware routing), for real-time endpoints of the Amazon SageMaker AI service. It uses information that the model container reports about the state of its KV cache in GPU memory, and sends each request to an instance that already has a long match in its cache for the given prompt and is not overly loaded. The strategy is intended for deployments serving large language models (LLMs) with multi-turn conversations, long shared prompts, or high load.
According to AWS, KV-aware routing addresses the shortcomings of the previously introduced prefix-aware routing, which estimates cache placement only from the content of the request without knowledge of the actual state of the instance. Prefix-aware routing therefore cannot detect when an instance has meanwhile evicted the cache, and in short conversations each subsequent turn may get shuffled to a different instance than the one holding the history. KV-aware routing, by contrast, maintains a managed index based on actual store and eviction events from the container, with the index containing only one-way truncated hashes, not the prompts themselves or tensor data.
Activation is done by setting the SM_VLLM_KV_EVENTS_CONFIG environment variable on the model side and the RoutingStrategy parameter to KV_AWARE in the endpoint configuration; clients call the endpoint the same way as before. The feature requires at least two instances or two copies of the inference component, otherwise it has no effect. If no instance has a useful match or all are loaded, the request is redirected using load-aware routing.
See the source article for details.
Why it matters
For teams running chatbots, RAG systems, or agents with long system prompts on SageMaker AI, this is a way to reduce unnecessary recomputation of context on repeated queries within the same conversation. Unlike the earlier prefix-aware routing, it reacts to the actual state of the GPU cache, so it should work better even with short conversations and during cache eviction, which according to AWS translates into latency and instance utilization efficiency under higher load.
Relevant practical impact
What this means
For a business
Companies running LLM inference on Amazon SageMaker AI can reduce repeated computation in multi-turn conversations and long shared prompts, which according to AWS improves GPU memory usage efficiency and can reduce inference latency and cost in multi-instance deployments.
DevelopmentCheck the original
Event sources
clearly official source · 1 publisher, 0 independent. We count feeds from the same owner only once.