Skip to content
worth noting Productivity

Amazon SageMaker AI gains KV cache-aware routing for real-time LLM inference

clearly official source

AWS has introduced a new KV cache-aware routing strategy in Amazon SageMaker AI for real-time endpoints, which routes LLM requests to instances with actually available KV cache, aiming to improve the efficiency of multi-turn conversations and long shared prompts.

Amazon Web Services has introduced a new routing strategy, KV cache-aware routing (KV-aware routing), for real-time endpoints of the Amazon SageMaker AI service. It uses information that the model container reports about the state of its KV cache in GPU memory, and sends each request to an instance that already has a long match in its cache for the given prompt and is not overly loaded. The strategy is intended for deployments serving large language models (LLMs) with multi-turn conversations, long shared prompts, or high load.

According to AWS, KV-aware routing addresses the shortcomings of the previously introduced prefix-aware routing, which estimates cache placement only from the content of the request without knowledge of the actual state of the instance. Prefix-aware routing therefore cannot detect when an instance has meanwhile evicted the cache, and in short conversations each subsequent turn may get shuffled to a different instance than the one holding the history. KV-aware routing, by contrast, maintains a managed index based on actual store and eviction events from the container, with the index containing only one-way truncated hashes, not the prompts themselves or tensor data.

Activation is done by setting the SM_VLLM_KV_EVENTS_CONFIG environment variable on the model side and the RoutingStrategy parameter to KV_AWARE in the endpoint configuration; clients call the endpoint the same way as before. The feature requires at least two instances or two copies of the inference component, otherwise it has no effect. If no instance has a useful match or all are loaded, the request is redirected using load-aware routing.

See the source article for details.

What changed

Why it matters

For teams running chatbots, RAG systems, or agents with long system prompts on SageMaker AI, this is a way to reduce unnecessary recomputation of context on repeated queries within the same conversation. Unlike the earlier prefix-aware routing, it reacts to the actual state of the GPU cache, so it should work better even with short conversations and during cache eviction, which according to AWS translates into latency and instance utilization efficiency under higher load.

Relevant practical impact

What this means

01

For a business

Companies running LLM inference on Amazon SageMaker AI can reduce repeated computation in multi-turn conversations and long shared prompts, which according to AWS improves GPU memory usage efficiency and can reduce inference latency and cost in multi-instance deployments.

Development
What to decide If a company runs LLMs on Amazon SageMaker AI with multi-turn conversations or long shared prompts, verify the benefit of enabling KV-aware routing in a test deployment.
More business impacts →
Amazon SageMaker AWS GPU memory KV-cache LLM inference routing optimization

Check the original

Event sources

clearly official source · 1 publisher, 0 independent. We count feeds from the same owner only once.

1
AWS Machine Learning Blog primary source · first detected Amazon SageMaker AI real-time endpoints now support KV cache aware routing