Exa increased embedding infrastructure performance by 50% through optimization on AWS Inferentia2 and Trainium2 chips
The company Exa, which operates a web index for semantic search, reports increasing embedding document throughput by 50% through optimizations on AWS Inferentia2 and Trainium2 chips. This is related to re-indexing 30 billion documents with a new model.
The company Exa operates its own vector database for semantic search, into which it embeds data from the web as well as private and synthetic sources. According to the company, the cost economics of embedding a single document, at a volume of tens to hundreds of billions of documents, is critical to the whole business, so even a small improvement in performance per document has a significant effect at that scale.
The company describes that the impetus for optimization was a move to a new model, which required re-embedding 30 billion priority-selected documents from the existing corpus — this temporarily doubled the load on the embedding pipeline, since both the live indexing and the new indexing were running concurrently. This prompted a reassessment of the hardware. Through a successive series of optimizations — adjusting the weight layout, enabling autocast to reduce computation precision, deploying prebuilt kernels from the Neuron Kernel Interface (NKI) library, a custom fused SwiGLU MLP kernel addressing memory issues (HBM spill), and a rewritten attention kernel adapted to the model's specific dimensions — the company reports achieving an approximately 1.5x (50%) increase in throughput in documents per second. The company verified the correctness of the changes using a threshold of 0.999 cosine similarity against the reference output.
The source article, in the available portion, cuts off in the middle of the final summary of results, so some details (e.g., exact figures for individual optimization steps or the conclusion regarding the agentic development workflow) are not available. See the source article for details.
Why it matters
For companies operating search or RAG systems based on embeddings at large scale, this case shows an alternative to GPUs — specialized AWS Inferentia2/Trainium2 chips with the Neuron SDK and NKI tools, which according to the company Exa make it possible to noticeably reduce cost per document while maintaining accuracy. For ML engineers, the case is a concrete example of kernel-level optimization (weight layout, autocast, fused kernels) addressing real bottlenecks such as memory spill and kernel contention.
Two audiences, two different impacts
What this means
For individuals
For ML/infra engineers, this is a concrete set of techniques (weight layout, autocast, custom fused kernels, adapting the attention layer to the model's dimensions) applicable when optimizing your own embedding or inference workloads on specialized hardware.
For a business
Companies with high-volume embedding or semantic search pipelines may consider AWS Inferentia2/Trainium2 as a more cost-effective alternative to GPUs, since according to the company Exa it delivers a noticeable improvement in performance per document while keeping output quality unchanged (a threshold of 0.999 cosine similarity).
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.