Skip to content
context Hardware

Exa increased embedding infrastructure performance by 50% through optimization on AWS Inferentia2 and Trainium2 chips

only one source so far

The company Exa, which operates a web index for semantic search, reports increasing embedding document throughput by 50% through optimizations on AWS Inferentia2 and Trainium2 chips. This is related to re-indexing 30 billion documents with a new model.

The company Exa operates its own vector database for semantic search, into which it embeds data from the web as well as private and synthetic sources. According to the company, the cost economics of embedding a single document, at a volume of tens to hundreds of billions of documents, is critical to the whole business, so even a small improvement in performance per document has a significant effect at that scale.

The company describes that the impetus for optimization was a move to a new model, which required re-embedding 30 billion priority-selected documents from the existing corpus — this temporarily doubled the load on the embedding pipeline, since both the live indexing and the new indexing were running concurrently. This prompted a reassessment of the hardware. Through a successive series of optimizations — adjusting the weight layout, enabling autocast to reduce computation precision, deploying prebuilt kernels from the Neuron Kernel Interface (NKI) library, a custom fused SwiGLU MLP kernel addressing memory issues (HBM spill), and a rewritten attention kernel adapted to the model's specific dimensions — the company reports achieving an approximately 1.5x (50%) increase in throughput in documents per second. The company verified the correctness of the changes using a threshold of 0.999 cosine similarity against the reference output.

The source article, in the available portion, cuts off in the middle of the final summary of results, so some details (e.g., exact figures for individual optimization steps or the conclusion regarding the agentic development workflow) are not available. See the source article for details.

What changed

Why it matters

For companies operating search or RAG systems based on embeddings at large scale, this case shows an alternative to GPUs — specialized AWS Inferentia2/Trainium2 chips with the Neuron SDK and NKI tools, which according to the company Exa make it possible to noticeably reduce cost per document while maintaining accuracy. For ML engineers, the case is a concrete example of kernel-level optimization (weight layout, autocast, fused kernels) addressing real bottlenecks such as memory spill and kernel contention.

Two audiences, two different impacts

What this means

01

For individuals

For ML/infra engineers, this is a concrete set of techniques (weight layout, autocast, custom fused kernels, adapting the attention layer to the model's dimensions) applicable when optimizing your own embedding or inference workloads on specialized hardware.

What to do Review the kernel-level optimization techniques described in the source article for potential use in your own embedding/inference workloads.
More practical updates →
02

For a business

Companies with high-volume embedding or semantic search pipelines may consider AWS Inferentia2/Trainium2 as a more cost-effective alternative to GPUs, since according to the company Exa it delivers a noticeable improvement in performance per document while keeping output quality unchanged (a threshold of 0.999 cosine similarity).

Development
What to decide When planning infrastructure for high-volume embedding/semantic search, consider evaluating AWS Inferentia2/Trainium2 as an alternative to GPUs in terms of cost per document.
More business impacts →
AWS Inferentia2 AWS Trainium2 embedding Exa inference optimization semantic search

Check the original

Event sources

only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.

1
AWS Machine Learning Blog primary source · first detected Exa boosts embedding throughput by 50% to index billions of documents on AWS AI chips