AWS releases SGLang-Neuron plugin for serving LLM models on Trainium chips
The SGLang-Neuron plugin is now publicly available via the AWS Deep Learning Container. It allows LLM models to be served using the SGLang interface on AWS Trainium chips using native PyTorch. It was developed by RadixArk in collaboration with Annapurna Labs.
AWS has announced the public availability of the SGLang-Neuron plugin, which connects the SGLang serving framework with AWS Trainium AI chips. The plugin was developed by RadixArk in collaboration with Annapurna Labs and is available via the AWS Deep Learning Container. It allows models to be served and rollouts generated for reinforcement learning using the same SGLang interface that users already use today on other hardware.
It is a so-called out-of-tree plugin: the SGLang core continues to provide the API server, request lifecycle, RadixAttention, and prefix-cache management, while the plugin implements the models, attention, and mixture of experts specific to Trainium. Technically, SGLang-Neuron runs on native PyTorch via TorchNeuron, which registers Trainium as a PyTorch device, and for performance it uses kernels written in the Neuron Kernel Interface. Operations are compiled using torch.compile.
The plugin also supports so-called PD disaggregation, where a prefill server computes the prompt's KV cache and transfers it to a decode server via NIXL over the Elastic Fabric Adapter; the decode server then generates the remaining tokens. According to AWS, you can get started with the plugin by running the SGLang-Neuron container on a Trainium3 EC2 instance and following the quickstart guide for supported models.
Why it matters
For companies running large language model inference, the plugin offers the ability to deploy the same SGLang workflow on Trainium chips without rewriting serving code, expanding hardware choice beyond Nvidia GPUs. For developers and infrastructure engineers, it means they can try out an alternative inference infrastructure while keeping the familiar API and tools for RadixAttention and prefix-cache.
Two audiences, two different impacts
What this means
For individuals
Developers and infrastructure engineers who currently serve models via SGLang can deploy the same workflow on AWS Trainium chips without changing the API and without rewriting serving code.
For a business
Companies running LLM inference on AWS may consider Trainium chips as an alternative to GPUs for serving models, thanks to compatibility with existing SGLang tooling without the need to migrate to a different interface.
DevelopmentCheck the original
Event sources
clearly official source · 1 publisher, 0 independent. We count feeds from the same owner only once.