Deepgram and AWS optimized batch transcription with Nova-3 for AWS Trainium2 chips
Deepgram and AWS ported the Nova-3 model to AWS Trainium2 chips. According to their measurements, a configuration with eight replicas on a single instance processed 11–70 % more audio per dollar than the compute options previously used. The results apply to batch transcription.
Deepgram and AWS described porting and optimizing the Nova-3 model for batch speech transcription on AWS Trainium2 chips. The resulting configuration runs eight replicas of the model on a single Amazon EC2 trn2.3xlarge instance. According to internal measurements, it processed 11–70 % more audio per dollar than the compute options previously used by Deepgram. The comparison used instance prices in effect at the time of the evaluation and covered batches processed outside real time.
The original implementation used the PyTorch framework. The team first verified that it could run using the TorchNeuron integration, then separately compiled the encoder and decoder into graphs with fixed dimensions using torch.compile. According to the authors, all model operations then ran on AWS Trainium2 chips without offloading computation to the CPU.
Further optimizations used prebuilt compute functions from the NKILib library as well as custom implementations created using Neuron Kernel Interface. Profiling determined which parts to modify; the team also tested batch sizes from 1 to 128. Throughout the changes, it checked numerical agreement with CPU computations and word error rate (WER) against reference values from Deepgram. Details are available in the source article.
Why it matters
The case study shows a concrete procedure for increasing batch transcription volume within a given infrastructure budget, for example for analyzing contact center calls or transcribing medical records. The reported range of improvements in cost efficiency is based on internal workloads at Deepgram and instance prices at the time; it does not, by itself, demonstrate the same benefit for other deployments.
Two audiences, two different impacts
What this means
For individuals
Developers working on a model port get a concrete procedure: compile parts of the model separately, use profiling to identify bottlenecks, and check accuracy throughout the changes.
For a business
For companies with large volumes of batch transcription, the case study provides a basis for assessing infrastructure costs. The demonstrated benefit concerns the amount of audio processed per dollar, rather than an announced reduction in service prices for customers.
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.