For the documented NVIDIA GPU path, deploy vLLM 0.28.0 with Dynamo 1.5.1 on Kubernetes 1.30 or later, using NVIDIA GPU nodes and kubectl 1.30 or later. Treat that pairing as a pinned starting point, not a permanent compatibility guarantee: NVIDIA’s version selector and manifests can change, so verify the matrix immediately before installation.
The production path is a Kubernetes DynamoGraphDeployment (DGD) managed by the Dynamo operator. Begin with aggregated serving unless your traffic benefits from independent prefill and decode capacity. Add RDMA, scheduling, metrics, and shared-storage components only when the selected topology and model size require them. Finish by checking health endpoints, sending an OpenAI-compatible request, and load-testing your own prompt, generation, concurrency, and latency profile.
As an Amazon Associate I earn from qualifying purchases.
What this deployment covers
Dynamo is an open-source inference framework that can run vLLM, SGLang, or TensorRT-LLM backends. NVIDIA documents Kubernetes, Slurm, and local execution across NVIDIA and AMD GPUs and Intel XPUs. This guide is deliberately narrower: it covers vLLM on NVIDIA GPU nodes in Kubernetes, where the Dynamo operator creates and manages the workloads and services described by a DGD.
The version and requirement statements below reflect NVIDIA documentation checked on 2026-10-13. They are not a throughput promise, a universal GPU-sizing formula, or evidence that one topology is best for every model. Your model, GPU inventory, prompt-to-generation ratio, service-level objective, and region determine the final design.
Pin the Dynamo and vLLM versions first
Do not start by copying an old manifest and changing only the model name. Pin the operator, runtime image, templates, and vLLM version as one tested set.
| Dynamo release | vLLM pairing shown in NVIDIA’s selector | How to treat it |
|---|---|---|
| Dynamo 1.5.1 | vLLM 0.28.0 | Current stable NVIDIA GPU Kubernetes pairing shown on 2026-10-13 |
| Dynamo 1.4.2 | vLLM 0.26.0 | Older documented pairing; use only when your environment is intentionally pinned to that release line |
Record the selected versions in your deployment repository and use matching image tags everywhere. A runtime image from one release combined with an operator or generated template from another can fail at admission, startup, or runtime even when each component works independently. Before applying anything, recheck NVIDIA’s current compatibility selector, API versions, image tags, and installation requirements; the documentation contains both stable versioned pages and development-path pages.
Prepare the Kubernetes and GPU prerequisites
The documented NVIDIA GPU quickstart requires all of the following:
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →- Kubernetes 1.30 or later.
kubectl1.30 or later.- At least one Kubernetes node with an NVIDIA GPU.
- Helm 3 or later for the installation path that uses Helm.
- Working accelerator support so Kubernetes advertises GPU resources to scheduled pods.
Verify the control plane and client versions
Check the client and server versions before installing the operator. A newer client cannot compensate for an unsupported cluster API level, and a cluster upgrade can change admission behavior or available CRD versions.
kubectl version --client
kubectl get nodesConfirm that the server is Kubernetes 1.30 or later and that the node list includes the GPU workers intended for inference. Keep control-plane and node maintenance procedures separate from the Dynamo rollout; a DGD cannot repair a cluster that does not schedule ordinary GPU workloads.
Install accelerator support once
Install the accelerator components and the Dynamo Platform in the order specified by the matching NVIDIA release instructions. NVIDIA notes that the NVIDIA GPU Operator can install NVIDIA drivers. Do not enable it alongside provider-managed drivers unless you have followed the instructions for that specific cluster; two driver-management systems can conflict and leave nodes unschedulable or devices undiscoverable.
After accelerator installation, verify that a node advertises GPU capacity and that a test pod can request it. The exact resource key and device-plugin behavior come from the installed NVIDIA stack, so use the values exposed by your cluster rather than copying limits from an unrelated example.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #2
- Part number 900-53651-2500-000 and model: P3651
- This is the 2 slot version for when there is no empty slots between 2 slot cards. If you have one or more empty slots between the cards or the cards are 3 slot this NVLink will not work. See the attached images showing the card layout.
- NVLink 3.0 for any brand of RTX Ampere model graphics cards: 3090, A30, A40, A100 / H100 (Requires three NVLinks), A800, A4500, A5000, A5500, A6000
- This is the same as PNY part number: NVLAMP-2SLOT-BSP and RTXA6000NVLINK-KIT
- This is the same as Dell part number: 0RWJ7Y
Understand the DynamoGraphDeployment resource
“A DynamoGraphDeployment (DGD) is the Kubernetes entry point for serving a model with Dynamo.”
— NVIDIA Model Deployment documentation
A DGD is the deployment artifact whether you copied a tuned recipe, generated a backend template, or authored the manifest yourself. It describes the serving graph and gives the operator enough information to create the corresponding Kubernetes workloads and services.
What the DGD describes
- Frontend: the network-facing component that accepts inference requests and routes them into the graph.
- Workers: one or more vLLM runtime processes that load the model and execute inference.
- Runtime image: the Dynamo vLLM image tag matching the pinned release.
- Model arguments: the model identifier, serving flags, parallelism settings, and any backend-specific options required by the chosen recipe.
- Resources: CPU, memory, and GPU requests and limits for each component.
- Credentials: a Kubernetes Secret containing a Hugging Face token when the model requires authenticated access.
- Request flow: the graph connections that determine how the frontend reaches workers and, for disaggregated serving, how prefill and decode pools exchange state.
Example model names and GPU values in NVIDIA templates are starting points, not production capacity guidance. Replace them only after checking the model’s architecture, weight size, context length, quantization, parallelism requirements, and available device memory.
Choose aggregated or disaggregated serving
Topology is the first material production decision after compatibility. Compare the options against your workload rather than assuming that a more distributed graph is automatically faster.
| Decision axis | Aggregated serving | Disaggregated serving |
|---|---|---|
| Execution model | One worker handles prompt processing (prefill) and token generation (decode). | Separate worker pools handle prefill and decode. |
| Best starting point | A model and traffic pattern that fit on one worker and do not need independent phase scaling. | A workload where prompt processing and generation need different capacity or scaling policies. |
| Scaling control | Scale the combined worker unit. | Scale prefill and decode pools independently. |
| Network sensitivity | Lower graph-level transfer complexity. | KV-cache movement and inter-worker networking become critical. |
| Operations | Simpler scheduling, deployment, and troubleshooting. | More components, scheduler coordination, observability, and failure modes. |
When aggregated serving is the right baseline
Use aggregated serving when a single worker can keep the model resident and meet the target prompt and generation behavior. It minimizes moving parts, avoids a separate KV-cache transfer path, and gives you a straightforward baseline for health checks and load tests. This is a sensible first production topology for a single-node or modest replica deployment, provided measured results meet your SLO.
When to consider disaggregation
Disaggregation is useful when prompt-heavy and generation-heavy traffic compete for the same resources, or when independent prefill and decode pools let you match capacity to changing demand. Evaluate it with the model’s parallelism, replica count, GPU availability, and prompt-to-output distribution in mind.
Disaggregation introduces a hard networking requirement. NVIDIA identifies Network Operator and RDMA for production disaggregated inference and warns that a TCP fallback can cause approximately 200–500× degradation in KV-cache transfer performance relative to the expected production networking path. That range is NVIDIA’s published estimate, not an independently measured guarantee for your cluster. If RDMA is unavailable, treat the fallback as an explicit risk to validate rather than an invisible implementation detail.
Rank #3
- Video/Sound Cards
- Passive Cooling
A practical selection process
- Confirm whether the complete model and its required parallelism fit on the worker shape you can actually schedule.
- Characterize the traffic mix: prompt-token length, generated-token length, streaming behavior, concurrency, and burstiness.
- Set the latency and throughput objectives for both phases, not just an average request target.
- Estimate the operational cost of extra pools, scheduling, networking, and autoscaling.
- Benchmark both viable designs with representative requests before committing to replica counts or capacity claims.
Add only the production components your topology needs
NVIDIA’s installation guidance makes several dependencies conditional. Installing all of them on every cluster increases operational surface area without improving a simple aggregated deployment.
Recommended Free Tools
| Component or capability | Use it when | Why it matters |
|---|---|---|
| Grove plus a gang scheduler | Running multinode or disaggregated inference. | Coordinates placement of related workers so a graph does not start with only part of its required capacity. |
| Network Operator and RDMA | Running production disaggregated inference. | Provides the networking path needed for efficient KV-cache transfer; TCP fallback carries NVIDIA’s stated 200–500× degradation risk. |
| kube-prometheus-stack | Using planner SLA mode or KEDA/HPA autoscaling. | Supplies the Prometheus-based metrics path those planning and autoscaling modes depend on. |
| Shared model storage | Serving models larger than 70B parameters or operating many replicas. | NVIDIA recommends shared storage in these cases to avoid repeated downloads and Hugging Face rate limits. |
The “over 70B” guidance is a documented recommendation, not an immutable storage boundary. Smaller models can still benefit from shared storage when replicas restart frequently or nodes have slow local disks; large models may require additional storage and image-pull planning beyond this recommendation.
Create a deployment that matches the model
Start with a tuned recipe when it matches your model, backend, GPU type, and topology. Otherwise adapt a vLLM backend template or write the DGD directly. In every case, review the generated resource before applying it.
Set the model and credentials
Use the exact model identifier expected by the runtime. If the repository requires authentication, create a Kubernetes Secret and reference it from the worker configuration. Keep the token out of Git, container arguments visible in ordinary logs, and copied shell history. Test that the worker’s service account can read the Secret in the namespace where the DGD runs.
Set resource requests and limits deliberately
Give each worker enough GPU memory for weights, runtime overhead, KV cache, and the maximum context you intend to support. CPU and memory requests must cover tokenization, request handling, model loading, and metrics. Do not treat a template’s GPU count as a recommendation for your model; it is an example that must be validated against actual memory and parallelism requirements.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallKeep frontend and worker concerns separate
The frontend needs stable service discovery and enough CPU to handle connection management and request routing. Workers need GPU placement, model storage access, and restart behavior that will not repeatedly overload the model repository. In a disaggregated graph, define prefill and decode worker resources independently so each pool can be scaled and observed on its own.
Apply the DGD through the operator
After installing the pinned platform components and any conditional dependencies, apply the reviewed resource:
Rank #4
- CUDA Cores: 4608 / NVIDIA Tensor Cores: 576 / NVIDIA RT Cores: 72
- GPU Memory: 24 GB GDDR6 with ECC / Bandwidth: 624 GB/Sec
- System Interface: PCI Express 3.0 x16
- Four DisplayPort 1.4 Connectors
- 3D Stereo Support with Stereo Connector
kubectl apply -f dgd.yaml
kubectl get pods,services
kubectl describe dynamographdeployment <deployment-name>Use the resource names and namespace from your manifest. Wait for the operator to create the frontend and worker workloads, then inspect pod events before changing model flags. A pod that is Pending because of GPU capacity, a missing Secret, or an unavailable volume is an infrastructure problem, not a vLLM tuning problem.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Install in a controlled order
- Freeze the matrix: record Dynamo 1.5.1 and vLLM 0.28.0, or another pair confirmed by the current selector.
- Prepare Kubernetes: confirm Kubernetes 1.30 or later, kubectl 1.30 or later, Helm 3 or later, and GPU nodes.
- Install accelerator support: ensure drivers and the NVIDIA device integration are managed by one supported mechanism.
- Install the Dynamo Platform and operator: use the release-matched NVIDIA installation procedure.
- Add conditional dependencies: Grove and gang scheduling for multinode or disaggregated graphs, RDMA networking for production disaggregation, Prometheus components for planner SLA or KEDA/HPA, and shared storage where the model or replica count warrants it.
- Apply the DGD: create or reference required Secrets, then submit the reviewed resource.
- Observe rollout: inspect events, pod readiness, service endpoints, logs, and operator status before sending traffic.
Verify health before testing capacity
Dynamo exposes /live and /health endpoints used by Kubernetes probes. Check the frontend’s live and ready behavior through the service address exposed by your deployment. A live process can still be unable to load a model, reach a worker, or serve an inference request, so use both endpoint checks and a representative request.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsSend an OpenAI-compatible request
Once the frontend service is reachable, send a small chat completion using the endpoint and authentication policy configured in your environment:
curl -sS http://<frontend-address>/v1/chat/completions
-H 'Content-Type: application/json'
-d '{
"model": "<model-id>",
"messages": [{"role": "user", "content": "Reply with one short sentence."}],
"max_tokens": 32,
"temperature": 0
}'Use a model identifier accepted by the deployed runtime and add the authorization header required by your frontend. Check that the response contains an assistant message, then correlate the request with worker logs and metrics. This smoke test proves connectivity and basic serving only; it does not establish sustainable throughput, tail latency, autoscaling behavior, or an SLO.
Instrument the serving graph
Production readiness requires more than a green pod. Dynamo’s documented operational signals include request, latency, routing, worker, cache, and operator metrics; component logs; distributed traces exported through OTLP; and the /live and /health endpoints used by probes.
Metrics to watch
- Request rate, error rate, and status-code distribution at the frontend.
- Queueing and end-to-end latency, including tail percentiles rather than only averages.
- Worker utilization, restarts, and model-load duration.
- KV-cache usage and cache-hit behavior where exposed by the runtime.
- Routing decisions and imbalance between replicas or between prefill and decode pools.
- Operator reconciliation errors and rollout progress.
Logs and traces
Centralize frontend, worker, and operator logs with request correlation where your security policy permits it. Export distributed traces through OTLP to the tracing system selected for your cluster. In a disaggregated deployment, trace spans should let you distinguish frontend routing, prefill execution, KV-cache transfer, decode execution, and response streaming; otherwise a slow request can be misdiagnosed as a generic GPU problem.
Free tools Windows power users keep installed
One-click scans. No signup required.
Load-test the workload you will actually serve
Do not publish capacity numbers from the smoke request. Build a test matrix that reflects the intended application:
- Short, medium, and long prompts.
- Short and long generated responses.
- Streaming and non-streaming requests.
- Low concurrency, steady concurrency, and burst traffic.
- Warm-cache and cold-cache behavior.
- Successive requests that exercise the maximum context and any batching policy.
Measure time to first token, inter-token latency, end-to-end latency, achieved output-token rate, error rate, queue depth, GPU memory, and worker restarts. For disaggregated serving, measure prefill and decode phases separately and observe KV-cache transfer performance under the same concurrency. Repeat tests after changing model quantization, parallelism, batch limits, or replica counts; each change can alter the balance between phases.
Use the results to choose replica counts and autoscaling thresholds. The available sources do not specify your model, target traffic, latency objective, GPU inventory, or cloud pricing, so no exact GPU count, replica number, cost, or universal throughput claim can be calculated from this guide.
Quick Recap
Troubleshoot by symptom
Pods remain Pending
- Check node labels, taints, and available GPU resources.
- Compare worker requests with the actual free capacity on eligible nodes.
- For multinode or disaggregated graphs, verify that the required gang scheduler and Grove components are installed and healthy.
- Inspect pod events for unsatisfied volumes, Secrets, or affinity rules.
Workers restart while loading the model
- Read the worker logs for authentication failures, missing files, or out-of-memory errors.
- Verify that the referenced Secret is present in the DGD namespace and that the pod can read it.
- Check model storage bandwidth and free space. Repeated downloads across many replicas are a reason to adopt shared storage.
- Revisit context length, quantization, tensor or pipeline parallelism, and GPU memory reservations before increasing replica count.
The frontend is healthy but requests fail
- Confirm that worker services are ready and that the frontend is routing to the expected graph.
- Check the request’s model identifier, authorization header, content type, and endpoint path.
- Correlate the request ID across frontend and worker logs, then inspect operator reconciliation errors.
Disaggregated latency is unexpectedly high
- Verify that the intended RDMA path is active rather than a TCP fallback.
- Inspect KV-cache transfer metrics and traces for retransmits, saturation, or queueing.
- Check that prefill and decode pools are balanced for the actual prompt and output mix.
- Compare the result with an aggregated baseline under identical requests before changing model flags.
Production readiness checklist
- The Dynamo and vLLM versions are pinned to a currently supported pair.
- Kubernetes, kubectl, Helm, GPU nodes, drivers, and device integration meet the documented prerequisites.
- Only one supported component manages NVIDIA drivers on the nodes.
- The DGD has been reviewed for model arguments, image tags, resources, Secrets, storage, and topology.
- Aggregated versus disaggregated serving was chosen from measured workload requirements.
- Grove, gang scheduling, RDMA, Prometheus components, and shared storage were installed only where required.
- Frontend and worker health endpoints pass, and a representative OpenAI-compatible request succeeds.
- Metrics, logs, traces, and probe results are retained and alertable.
- Load tests cover realistic prompt lengths, output lengths, concurrency, streaming, cache state, and failure behavior.
- Capacity and latency statements are based on those tests rather than on the smoke request or template defaults.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →
