AWS SageMaker Inference supports GPU partitioning using NVIDIA MIG
AWS SageMaker Inference now allows splitting a physical GPU into up to 7 hardware-isolated partitions using NVIDIA MIG and deploying multiple models on a single card without them affecting each other.
Amazon Web Services has extended the SageMaker Inference service with support for GPU partitioning using NVIDIA Multi-Instance GPU (MIG) technology. A physical GPU can be divided into up to seven independent instances (the exact number depends on the accelerator type), with each partition having its own compute units, cache, and memory, with isolation enforced at the hardware level. This allows multiple models to be deployed on a single card such that the load of one model does not affect the performance of the others.
The feature works only with so-called inference components, a construct for deploying multiple models to a single endpoint with a set allocation of compute power for each of them. It does not work with single-model endpoints, serverless endpoints, or heterogeneous instance pools. According to AWS, until now using MIG required running your own Kubernetes stack with driver version management, partition configuration, and scheduling; SageMaker Inference has taken over this work and automatically handles GPU configuration, placement, and scaling.
Technically, these are two new optional fields in already existing API calls (AcceleratorPartitionLayouts for CreateEndpointConfig and AcceleratorPartitionType for the inference component), without any new API operations, resource types, or IAM actions. According to the article, partition layout can only be changed while the partitions are idle, which on SageMaker AI means choosing the layout already at endpoint creation time; a model requiring more GPUs still needs whole cards.
According to AWS, Salesforce participated in the development of the feature, taking part in defining requirements and validating the initial implementation. A company representative states in the article that MIG will make it possible to consolidate workloads that don't need a full GPU, increase its utilization, and reduce inference costs while maintaining performance isolation. The rest of the article, with measured results on the price impact, was not available in the source.
Why it matters
Companies running multiple AI models on GPU infrastructure can now allocate just a fraction of performance according to the model's actual needs instead of a whole card, without risking performance interference between models. This has a direct impact on the efficiency of utilizing expensive GPU hardware and on inference costs, especially for portfolios of smaller or per-tenant models that previously had to occupy an entire accelerator. At the same time, there is no longer a need to build a custom Kubernetes layer for managing MIG, which reduces operational burden for MLOps teams.
Two audiences, two different impacts
What this means
For individuals
For ML engineers and developers working with SageMaker Inference, the way GPU endpoints are configured is changing — it's now possible to choose a fraction of an accelerator instead of a whole card via two new optional fields in the existing API.
For a business
Companies running multiple models on SageMaker Inference can reduce GPU infrastructure costs through better utilization of their cards, and no longer need to build their own Kubernetes solution for MIG.
ProcessesCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.