AWS described three deployment options for NVIDIA Nemotron 3.5 Super VL
AWS published a reference procedure for deploying NVIDIA Nemotron 3.5 Super VL on Amazon EKS, Amazon EKS Auto Mode and Amazon SageMaker HyperPod EKS. The aws-do-eks project covers cluster creation, deployment, testing, performance measurement and resource cleanup.
AWS described a reference deployment of the NVIDIA Nemotron 3.5 Super VL multimodal model on Amazon EKS, Amazon EKS Auto Mode and Amazon SageMaker HyperPod EKS. All three options use the same Kubernetes manifests. The aws-do-eks project covers the process from preparing the environment and creating a cluster through downloading model weights and deployment to testing, performance measurement and resource cleanup.
CPU nodes run system and testing tasks, while GPU nodes serve the model. Amazon ECR stores container images, shared Amazon FSx for Lustre storage holds the model weights, and Elastic Fabric Adapter connects the GPU nodes. NVIDIA Dynamo supports both running on a single node and separating the prefill and decode phases across different nodes. Image and video input processing can also run separately.
The model has 120 billion parameters, of which it activates 12 billion for each token; however, GPU memory must hold all the weights. The requirements table for vLLM lists a minimum of two B200/GB200 GPUs, one B300/GB300 GPU, four H200 GPUs or eight H100 GPUs with NVFP4. With BF16, the corresponding minimums are eight, four, eight and sixteen GPUs.
According to AWS, the architecture was validated on p6-b200.48xlarge instances with eight NVIDIA B200 GPUs and p5en.48xlarge instances with eight NVIDIA H200 GPUs. The procedure includes verifying communication using the NCCL all-reduce test and taking measurements using NVIDIA aiperf. It tracks time to first token, inter-token latency and throughput at different numbers of concurrent requests; these measurements inform deployment configuration and capacity planning.
Why it matters
The procedure combines deployment with network verification and latency measurement. Teams can use it to assess the load a specific configuration can handle within the required latency limits and determine the capacity needed based on the results.
Two audiences, two different impacts
What this means
For individuals
Developers get a complete procedure they can follow to try deploying a multimodal model in their own account and verify that the cluster works.
For a business
The operations team can use measurements of concurrent requests to determine how many replicas are needed and decide whether to increase capacity for prefill or decode. This provides a basis for sizing GPU infrastructure.
DevelopmentCheck the original
Event sources
only one source so far · 1 publisher, 0 independent. We count feeds from the same owner only once.