Skip to content
context Other

AssemblyAI describes an architecture for cheaper GPU inference on AWS using EC2 Spot Instances

only one source so far

According to an AWS case study, AssemblyAI runs thousands of GPU instances for speech-to-text through a combination of EC2 Spot and On-Demand instances, ECS, and SQS, reducing costs while maintaining SLA and data residency in European regions.

According to a case study published on the AWS Machine Learning Blog, AssemblyAI, a provider of an API for speech transcription, speaker detection, sentiment analysis, and PII redaction available through AWS Marketplace, runs its inference platform on thousands of GPU instances within Amazon ECS. The architecture combines EC2 Spot Instances and On-Demand instances across three GPU instance families, managed via ECS capacity providers, with On-Demand capacity serving as the baseline and Spot capacity covering demand spikes.

For resilience against Spot instance interruptions (EC2 gives a 2-minute warning before reclaiming capacity), the company separates job submission from job processing using Amazon SQS. When a Spot instance is interrupted, Amazon EventBridge triggers a Lambda function that returns the unfinished job to the main SQS queue, from where it is picked up by another available ECS task; jobs that exhaust their number of retries are routed by the company to a dead letter queue for investigation. For faster recovery, the company uses a custom AMI with model artifacts that builds in approximately 2 minutes. According to the company, the impact of a Spot instance outage on the end customer shows up only as a slight increase in processing time, but the transcription always completes successfully.

For customers in regulated industries (healthcare, financial, and legal services) who must keep audio data within specific geographic boundaries, AssemblyAI runs a fully independent copy of the architecture in each AWS region, with its own SQS queues, ECS clusters, capacity providers, and GPU fleets.

The source article was only partially available and is cut off mid-sentence at the end; see the source article for details.

What changed

Why it matters

The case study shows a concrete design for how companies running GPU inference at high volume can reduce infrastructure costs without losing availability or latency — relevant for teams planning or operating AI services with high GPU demands. It also documents a concrete approach to meeting data residency requirements in regulated industries through independent regional deployments.

Relevant practical impact

What this means

01

For a business

According to the described architecture, companies running GPU inference at large scale can reduce costs by using Spot Instances instead of purely On-Demand capacity, while maintaining SLA thanks to the retry mechanism in SQS and diversification across GPU instance families.

Development
What to decide When planning GPU inference infrastructure, consider combining EC2 Spot and On-Demand instances across multiple GPU instance families with a retry mechanism via SQS, in order to reduce costs without losing SLA.
More business impacts →
AssemblyAI AWS cost optimization EC2 Spot ECS GPU inference

Check the original

Event sources

only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.

1
AWS Machine Learning Blog primary source · first detected Cost-optimizing AssemblyAI speech-to-text inference on AWS with Amazon EC2 Spot