AWS integrates NVIDIA's Dynamo framework into Amazon SageMaker HyperPod for LLM inference
Amazon SageMaker HyperPod Inference now supports the open-source Dynamo framework from NVIDIA for distributed, multi-node inference of large language models with separation of prefill and decode phases.
Amazon Web Services announced support for the open-source Dynamo framework from NVIDIA in the Amazon SageMaker HyperPod Inference product. Dynamo is a framework for distributed inference that enables multi-node deployment, separated processing of prompts (prefill) and token generation (decode) on different nodes, and KV-aware routing with support for the vLLM, SGLang, and TensorRT-LLM engines. HyperPod provides the infrastructure for this — a high-speed Elastic Fabric Adapter (EFA) network, coordinated GPU scheduling, and node health monitoring with automatic recovery.
As a reason for needing such a solution, AWS states that the FP8 weights of the GLM-5.3 model exceed 640 GB of GPU memory on a single ml.p5.48xlarge instance, and that is without even counting the KV cache and operational load. Separating prefill and decode into separate pools allows them to be scaled and tuned independently; both groups of workers simultaneously load the full model weights, and the KV cache is transferred between them over the EFA network.
The integration offers two deployment methods: either a custom DynamoGraphDeployment manifest for teams that want direct control over the serving graph, or an InferenceEndpointConfig configuration with dynamoSpec for users who want endpoint management handled by the HyperPod Inference Operator, including automatic model weight downloads, TLS management, and autoscaling via KEDA. In the article, AWS describes configuration examples using ml.p5.48xlarge instances and mentions benchmarks comparing aggregated and disaggregated serving of the Qwen3-235B-A22B-FP8 model using the Dynamo framework with TensorRT-LLM — however, detailed benchmark results are missing from the source. See the source article for details.
Why it matters
For companies running their own large language models, this is a concrete tool for using GPU infrastructure more efficiently for models that exceed the memory of a single instance due to their size — separating prefill and decode allows each phase to be scaled individually according to load, which, according to AWS, improves time to first token thanks to reuse of the prefix KV cache and balanced load distribution among workers.
Relevant practical impact
What this means
For a business
Companies running their own large language models on AWS infrastructure can deploy distributed inference with separated prefill and decode phases, which reduces GPU memory requirements on a single instance and allows independent scaling according to load.
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.