AWS Batch adds compute capacity sharing between teams for Amazon SageMaker Training
AWS Batch now allows capacity for Amazon SageMaker Training to be divided among teams and unused resources to be lent out automatically. If the original team needs them, the service interrupts jobs using the borrowed capacity.
AWS Batch is adding quota management for training jobs in Amazon SageMaker Training. Administrators can divide compute capacity among teams, projects or jobs and configure sharing rules for each allocation. According to AWS, this capability is intended to improve GPU utilization while preserving the capacity allocated to individual teams.
The RESERVE strategy keeps unused capacity reserved, LEND allows it to be lent out, and LEND_AND_BORROW allows both lending and borrowing. When the original team needs the lent resources, AWS Batch automatically interrupts jobs using that capacity. Administrators can also optionally enable interruption of lower-priority jobs within a single allocation so that higher-priority jobs can start sooner.
Configuration requires a Scheduling Policy resource with a quotaSharePolicy entry, a Service Environment resource with capacity limits, and a linked training job queue. The source describes an example using SageMaker Python SDK v3; for jobs that may be interrupted, it uses checkpointing to save progress. Details are available in the source article.
Why it matters
Teams can use spare capacity from others, but must account for it being reclaimed. The choice of strategy therefore determines the practical trade-off between utilizing available GPUs and the need for training jobs to start quickly or experience fewer interruptions.
Two audiences, two different impacts
What this means
For individuals
Anyone creating a training job that uses borrowed capacity must account for automatic interruption as soon as the team allocated that capacity needs it.
For a business
Administrators of shared training infrastructure can configure allocations and capacity lending rules for individual teams. Reclaiming lent resources then requires no manual tracking down and stopping of other teams' jobs.
ProcessesCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.