AWS describes deploying an LLM on Amazon EKS with separate prefill and decode phases
The guide from AWS shows how to deploy a single LLM on Amazon EKS across two GPU nodes: one processes the input, and the other generates the response. Ray LLM Deep Learning Containers connect both phases behind a shared interface compatible with OpenAI API.
AWS has published a guide to deploying a single LLM using Ray LLM Deep Learning Containers on Amazon EKS. Two GPU nodes handle separate inference phases: prefill processes the input prompt and creates the KV cache, while decode generates the response incrementally. Both nodes serve a shared interface compatible with OpenAI API. According to AWS, this separation is intended to prevent the processing of long prompts from slowing down the generation of responses already in progress.
The example architecture uses two g6.8xlarge GPU nodes and one m7i.xlarge CPU node for system tasks, KubeRay operator and Ray head. KubeRay connects the head node to the worker nodes according to the RayService configuration. The GPU nodes are located in a single Availability Zone because Elastic Fabric Adapter (EFA) does not support communication between these zones. KV cache transfer between the phases must be fast enough to avoid negating the benefit of separating them.
The container includes Ray Serve LLM, the vLLM inference engine, NVIDIA Inference Transfer Library (NIXL) for KV cache transfer, and the EFA communication stack. The application can be defined in YAML; the build_pd_openai_app function connects the prefill and decode configurations behind a shared interface without custom application code. The article links to an accompanying repository with the complete code. See the source article for details.
Why it matters
The guide provides a concrete architecture for RAG, document summarization and agent applications with long inputs. According to AWS, separating the phases allows prompt processing and response generation to scale independently. The practical benefit, however, also depends on the speed of KV cache transfer between nodes.
Two audiences, two different impacts
What this means
For individuals
A developer can configure the connection between the prefill and decode phases in YAML using the build_pd_openai_app function, without writing custom request-handling code or building a custom container image.
For a business
The example deployment requires two GPU nodes and a separate CPU node. Infrastructure planning is further constrained by the requirement to place both GPU nodes in a single Availability Zone for communication over EFA.
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.