AWS expands Amazon SageMaker Inference with Inference Gateway, managed NVIDIA Dynamo, and cross-region routing
The company AWS has released new features for Amazon SageMaker Inference: Inference Gateway with managed NVIDIA Dynamo, cross-region and cross-account routing of endpoints, and expanded configuration recommendations including support for LoRA adapters.
The company Amazon Web Services (AWS) announced a set of new features for Amazon SageMaker Inference. HyperPod Inference gains additional routing strategies, broader support for serving frameworks, and expanded multi-region networking. Routing is now handled by the Inference Gateway, to which managed NVIDIA Dynamo has been added as a new serving framework. SageMaker AI endpoints can now route requests across both AWS regions and accounts.
The Inference recommendations feature, previously available only as an API, is now also accessible in SageMaker AI Studio as a guided low-code workflow (Jobs → Inference optimization section) and in SageMaker Python SDK version 3.17.0 and higher, so it fits into notebooks, CI pipelines, and MLOps processes. According to the company, benchmarking runs on real GPU infrastructure using the NVIDIA AIPerf tool and selects techniques based on the optimization goal, for example speculative decoding for throughput or kernel tuning for latency.
It is now also possible to include LoRA adapters in recommendations, specified as a Model Package ARN or S3 URI, and the system benchmarks combinations of the number of adapters and concurrency; the recommended deployment uses SageMaker Inference Components with a separate component for the base model and for each adapter. Another new feature is search_recipe max-concurrency-under-sla, which uses AIPerf to find the highest concurrency at which the endpoint still meets specified targets for TTFT, time per output token, overall latency, or error rate, and returns the boundary value (feasible_max or infeasible_min).
The source article is part of a broader overview of 2026 updates, and some technical details on other topics in the overview are not available.
Why it matters
For companies running models in production on SageMaker AI, these features automate tasks that previously required manual monitoring: how many LoRA adapters an endpoint can support, what concurrency it can handle without violating the latency SLA, and how to distribute traffic across regions and accounts during an outage or uneven load. This directly affects GPU capacity utilization, costs, and the resilience of the deployment against regional issues.
Two audiences, two different impacts
What this means
For individuals
For ML engineers working with SageMaker, benchmarking and configuration recommendations are moving from the console and direct API calls into the SageMaker Python SDK and coding agents, so they can be incorporated directly into notebooks and CI pipelines without switching tools.
For a business
Companies running inference on SageMaker AI gain tools for more accurate GPU capacity planning (number of LoRA adapters, maximum concurrency under SLA) and for cross-region/cross-account routing, which reduces the risk of downtime and helps keep GPU costs under control as traffic grows.
DevelopmentCheck the original
Event sources
clearly official source · 1 publisher, 0 independent. We count feeds from the same owner only once.