AssemblyAI describes an architecture for cheaper GPU inference on AWS using EC2 Spot Instances
According to an AWS case study, AssemblyAI runs thousands of GPU instances for speech-to-text through a combination of EC2 Spot and On-Demand instances, ECS, and SQS, reducing costs while maintaining SLA and data residency in European regions.
According to the described architecture, companies running GPU inference at large scale can reduce costs by using Spot Instances instead of purely On-Demand capacity, while maintaining SLA thanks to the retry mechanism in SQS and diversification across GPU instance families.