Skip to content
context Productivity

AWS expands Amazon SageMaker Inference with Inference Gateway, managed NVIDIA Dynamo, and cross-region routing

clearly official source

The company AWS has released new features for Amazon SageMaker Inference: Inference Gateway with managed NVIDIA Dynamo, cross-region and cross-account routing of endpoints, and expanded configuration recommendations including support for LoRA adapters.

The company Amazon Web Services (AWS) announced a set of new features for Amazon SageMaker Inference. HyperPod Inference gains additional routing strategies, broader support for serving frameworks, and expanded multi-region networking. Routing is now handled by the Inference Gateway, to which managed NVIDIA Dynamo has been added as a new serving framework. SageMaker AI endpoints can now route requests across both AWS regions and accounts.

The Inference recommendations feature, previously available only as an API, is now also accessible in SageMaker AI Studio as a guided low-code workflow (Jobs → Inference optimization section) and in SageMaker Python SDK version 3.17.0 and higher, so it fits into notebooks, CI pipelines, and MLOps processes. According to the company, benchmarking runs on real GPU infrastructure using the NVIDIA AIPerf tool and selects techniques based on the optimization goal, for example speculative decoding for throughput or kernel tuning for latency.

It is now also possible to include LoRA adapters in recommendations, specified as a Model Package ARN or S3 URI, and the system benchmarks combinations of the number of adapters and concurrency; the recommended deployment uses SageMaker Inference Components with a separate component for the base model and for each adapter. Another new feature is search_recipe max-concurrency-under-sla, which uses AIPerf to find the highest concurrency at which the endpoint still meets specified targets for TTFT, time per output token, overall latency, or error rate, and returns the boundary value (feasible_max or infeasible_min).

The source article is part of a broader overview of 2026 updates, and some technical details on other topics in the overview are not available.

What changed

Why it matters

For companies running models in production on SageMaker AI, these features automate tasks that previously required manual monitoring: how many LoRA adapters an endpoint can support, what concurrency it can handle without violating the latency SLA, and how to distribute traffic across regions and accounts during an outage or uneven load. This directly affects GPU capacity utilization, costs, and the resilience of the deployment against regional issues.

Two audiences, two different impacts

What this means

01

For individuals

For ML engineers working with SageMaker, benchmarking and configuration recommendations are moving from the console and direct API calls into the SageMaker Python SDK and coding agents, so they can be incorporated directly into notebooks and CI pipelines without switching tools.

What to do Try out the new configuration recommendations in SageMaker Python SDK version 3.17.0 and higher on an existing endpoint.
More practical updates →
02

For a business

Companies running inference on SageMaker AI gain tools for more accurate GPU capacity planning (number of LoRA adapters, maximum concurrency under SLA) and for cross-region/cross-account routing, which reduces the risk of downtime and helps keep GPU costs under control as traffic grows.

Development
What to decide Investigate whether cross-region and cross-account routing of endpoints can reduce the risk of outages and optimize GPU costs in production deployment.
More business impacts →
Amazon SageMaker Inference Inference Gateway Latency optimization Multi-region routing NVIDIA Dynamo SageMaker AI Studio

Check the original

Event sources

clearly official source · 1 publisher, 0 independent. We count feeds from the same owner only once.

1
AWS Machine Learning Blog primary source · first detected Amazon SageMaker Inference: 2026 launches in review, part 2