When a Kubernetes Pod remains Pending while waiting for a GPU, start with its scheduling events—not the container image or the physical GPU purchase. Run kubectl describe pod and read the latest FailedScheduling message. It usually identifies whether the scheduler cannot find a node with the requested extended resource, or whether labels, affinity, taints, topology, scheduling gates, or resource-allocation mechanisms have filtered every GPU node out.
Why is my Kubernetes pod stuck in Pending waiting for a GPU?
A GPU Pod can remain pending for several different reasons that look similar from the workload’s point of view:
As an Amazon Associate I earn from qualifying purchases.
- The Pod requests a resource name that no node advertises, such as using the wrong vendor key.
- The requested quantity does not fit the node’s currently allocatable GPU count.
- GPU nodes exist, but selectors, required node affinity, taints, or other scheduler filters exclude them.
- A device plugin is absent, unregistered, unhealthy, or unable to report the vendor’s devices.
- A scheduling gate intentionally prevents the Pod from entering normal scheduling.
- Topology policy, Dynamic Resource Allocation (DRA), or device preparation prevents allocation after ordinary resource checks.
The first job is to determine which category the cluster is reporting. Hardware availability alone does not make a GPU schedulable: Kubernetes needs a node advertisement, a matching Pod request, and a placement that satisfies every applicable constraint.
1. Read the Pod’s state and scheduling events first
Get the node view and full description
- Set the actual namespace and Pod name, then run
kubectl get pod <pod> -n <namespace> -o wide. A Pod with no assigned node is still in the scheduler’s placement phase. - Run
kubectl describe pod <pod> -n <namespace>. - Scroll to the Events section and read the newest
FailedSchedulingentries. Record the exact resource key, quantity, and any placement reason.
Kubernetes keeps an unschedulable Pod in Pending until a suitable placement is found and emits events when scheduling fails. The wording matters: Insufficient nvidia.com/gpu, a node-affinity failure, and an untolerated taint require different fixes. Do not change the image, restart the Pod, or buy hardware until the event identifies a cluster-side cause.
#1 Best Overall
- [Ultra-Lightweight Ear Buds Designed for Small Ears] Each earbud weighs only 3.7g and features a compact, ergonomic in-ear design made especially for small ears. Secure, low-profile, and comfortable for workouts, all-day wear or overnight sleeping.The ultra-slim profile sits flush in your ear with no hard edges poking or rubbing, so you can even sleep on your side without discomfort.
- [Immersive Stereo Sound with TOZO OrigX Technology] TOZO OrigX tuning delivers clear vocals, balanced mids, and natural stereo sound for music, podcasts, and videos.
- [Long Battery Life for Daily Use] Get up to 7 hours of playtime on a single charge, with up to 32 hours total using the charging case—ideal for workdays, commuting, and extended listening sessions.
- [Bluetooth 5.3 & Stable Connection] Bluetooth 5.3 provides fast pairing, stable wireless performance, and reduced dropouts as you move around home or office.
- [Deep Bass with Clear Vocals] High-performance drivers produce punchy bass while keeping vocals clean and detailed for everyday listening.
Separate scheduling from later container failures
If kubectl get pod shows a node in the NODE column, the scheduler has already assigned the Pod. A subsequent container error—such as a runtime, driver, CUDA, or application failure—is not the original “stuck waiting for a GPU” problem. Check container state and kubelet or runtime logs separately. An unhealthy device can also affect an already-assigned Pod differently from the allocatable count used to place new Pods.
2. Verify the GPU resource name and request
Use the exact extended-resource key
In the traditional device-plugin model, vendors commonly advertise keys such as nvidia.com/gpu or amd.com/gpu. These names are not interchangeable. The Pod must request the exact key shown by the target nodes. A request for nvidia.com/gpu cannot consume a node that advertises only another resource name.
Kubernetes documents GPU requests in the container’s limits section. The official Schedule GPUs documentation states: “GPUs are only supposed to be specified in the limits section.” A GPU limit can be used without a separate request; if both are set, they must be equal. A request without a limit is unsupported. This rule does not, by itself, solve a placement failure: the matching resource must still be advertised and available.
Free tools Windows power users keep installed
One-click scans. No signup required.
resources:
limits:
nvidia.com/gpu: 1Extended device resources are integer resources and cannot be overcommitted. A value of 1 requests one whole advertised device; fractional values and ordinary CPU-style overcommit do not apply to these resources. Confirm that the quantity in the event and manifest is the quantity you intended.
Compare the manifest with what nodes advertise
Inspect node status rather than guessing the key:
kubectl get nodes -o json
kubectl describe node <gpu-node>In the node description, look under Capacity and Allocatable for the extended resource. The resource name and integer count must match the Pod’s limit. If the node lists no GPU resource at all, fixing the Pod YAML will not create one; follow the driver and device-plugin checks in the next section.
Rank #2
- 【IP68 Waterproof Multifunctinal Smart Watch】Torjalph fitness activity tracker smart watch is IP68 waterproof technology ,you can wear it when swimming,handwashing,it's rain-proof which means that you perform multiple exercise modes outside without worrying about sweat or rain corroding your fitness smart watch,and this watch equipped with a 230 mAh large capacity battery magnetic charging ,and this sports activity tracker watch can be used 5-7 days If fully charged.
- 【Larger 1.83'' Full Screen Watch & Multi-dial Selection】1.83 inch full HD color touch screen smart watch for men / women provides excellent 240*284 resolution image quality ,This Bluetooth smart watch offers a better touch experience and remarkable immersion and ,more than 200 watch faces for you to choose from App"Da Fit ", you can change the dial style according to your mood to wear keeping the world on your wrist
- 【Accurate Activity Tracker Smart Watch 】sports watch activity tracker uses sweat and dust resistant breathable silicone strap, our Fitness Tracker Smart Watch for Android iOS Phones supports 12 sports modes,walking,running,bicycle,swimming,badminton,basketball,football,rope skipping. This bluetooth calls smart watch for men women is lightweight and stylish, bringing more convenient and relaxing sports experience while also providing more protection.
- 【Your 24 Hour Personal Daily Health Monitor】Fitness watch for women men full touch screen IP67 waterproof bluetooth smartwatch for Android iOS Phones with heart rate blood pressure monitor multifunction Andriod smart watch with steps counter sleep monitor sports fitness watch, monitoring blood oxygen saturation allows you to understand your physical condition in real time to track fitness progress ,to know your health status in time.
- 【More Functions Bluetooth Call Smart Watch】This fitness activity smart watch full screen for iPhone and Android phones includes a bunch of new features including:The bluetooth smart watch can hang up the phone, receive and display WhatsAPP /Text /Facebook /Twitter /Messenger reminder.Support sedentary reminder, alarm clock, stopwatch, timer,weather information,music control, remote camera control compatible with Android 5.0 and above or iOS 9.0 and above smartphones.
3. Check capacity, allocatable devices, and device health
Understand capacity versus allocatable
A registered device plugin reports devices to kubelet, and kubelet publishes them in node status. Capacity is the total count reported for the resource. Allocatable is the amount currently available for scheduling after reservations and device health are considered. Compare both values on every node that could otherwise host the Pod.
When devices become unhealthy, the allocatable count can fall while capacity remains unchanged. That produces a particularly confusing symptom: the node appears to have GPUs, but new Pods receive an insufficient-resource event. Treat a lower allocatable count as a health or registration signal, not proof that the physical cards have disappeared.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Check every eligible node, not just one favorite node
A cluster may have GPU capacity elsewhere, while none of the nodes that satisfy the Pod’s other constraints has a free device. This is an inference from the scheduler’s resource and placement filters, so confirm it by comparing node status with the Pod’s selectors, affinity, taints, and topology settings. A cluster-wide GPU total is not the same as capacity on eligible nodes.
4. Confirm drivers and the device-plugin deployment
What the classic GPU path requires
For the traditional extended-resource path, administrators install the vendor’s GPU driver on each appropriate node and run the corresponding vendor device plugin. The plugin registers with kubelet through Kubernetes’ device-plugin framework, reports devices, and monitors their health. Only after successful registration does the node advertise the vendor resource to the scheduler.
Inspect the plugin Pods and logs
Find the DaemonSet or Pods used by your GPU integration in their namespace, then inspect status and logs. The exact namespace and labels are cluster-specific, so first list workloads rather than assuming a name:
Rank #3
- [Low Noise High Fidelity] New bee guitar cord with 22AWG oxygen-free-copper center conductors provides enhanced signal transfer, the PVC insulation material and the high-density stranded copper double shield provide noise rejection for quiet operation
- [Ideal for Choice] New bee 1/4 straight and right angle metal connectors, ensures a secure connection in any instrument, pedal, or amp jack. L shape great fro on stage guitar stands
- [Wide Compatible] Suitable for mixer, power amplifier, microphone, effector, decoder, equalizer, electoronic organ, electric guitar, electric bass and other electronic equipments. Widely used in the installation of smart home, karaoke speaker, professional stage audio, etc
- [Easy Disassemble] Making incidental repairs quick and easy! Very nice, flexible cable
- [Dont Hesite to Order] 24 hours customer services and professional technology team are standing by
kubectl get daemonsets,pods -A
kubectl get pods -A -o wide
kubectl logs -n <plugin-namespace> <plugin-pod>Look for registration failures, inability to access the driver, device-health errors, crash loops, or a plugin Pod missing from a GPU node. Also inspect the node’s conditions and kubelet logs when registration appears to succeed but the resource is absent. Verify the node resource list again after correcting the plugin; the scheduler cannot use a device until the node status reflects it.
Recommended Free Tools
Interpret a missing resource correctly
If a GPU node has no vendor resource in its node status, the leading possibilities are driver installation, plugin placement, registration, or health reporting. A container image problem by itself does not explain a missing node resource. Confirm node status and plugin health before changing the application image or its command.
5. Check labels, affinity, selectors, taints, and tolerations
Compare required labels with real node labels
Print labels on candidate nodes and compare them character-for-character with the Pod’s nodeSelector and required node-affinity rules:
kubectl get nodes --show-labels
kubectl get node <gpu-node> --show-labels
kubectl describe pod <pod> -n <namespace>A selector for a nonexistent key, a wrong value, or an affinity expression that excludes every GPU node leaves the Pod pending even when devices are free. Required affinity is a hard filter; preferred affinity influences ranking but does not by itself make a node ineligible.
Inspect taints and tolerations
GPU nodes are often tainted so that only GPU workloads land on them. Check taints directly:
Rank #4
- Digital Family Calendar: BSIMB digital calendar wall touch screen puts an end to your chaotic life. Designed for busy families, the digital calendar keeps you organised and in step with your family. Assign different colours and tasks to each family member to make family scheduling easier and more intuitive.
- Easy Setup and Auto-Sync: Plug in, connect to WLAN and auto-syn your electric calendar in minutes. Digital wall calendar compatible with Google calendar, iCloud calendar, Outlook calendar, Cozi calendar and Yahoo calendar. Add events directly on the device or through the ecalendar app. Electronic calendar stays current whether you're at work or traveling—no manual updates needed.
- All-In-One Home Assistant: Smart calendar builds habits and encourage independence with personalised chores. Display daily meal plan to reduce the daily ‘What's for dinner?’ stress. To-do lists keep track of everything you want. Turning chores into valuable achievements with stars and rewards, making life easier and more enjoyable.
- Manage Anytime Anywhere: BSIMB electronic calendar for family stay connected with the powerful ecalendar app. Access and manage your digital wall calendar anytime, anywhere with free features like stars & rewards, photo screensaver, chore charts, meals plans, to-do lists etc. Electronic planner paid features like Magic Import tailor advanced tools for users with special needs.
- Stay in Touch with Loved Ones: Share photos or videos from your phone to electronic wall calendar with app, instantly update your life moment, and keep in touch with your family, especially those who are far away; you can also surprise your smart calendar digital wall by sending him a wish card; it's also great for keeping tech-illiterate elders in touch with the youngsters.
kubectl describe node <gpu-node>Then compare each taint’s key, value, and effect with the Pod’s tolerations. An untolerated NoSchedule taint filters the node. Adding a broad toleration just to make the Pod schedule can place it on nodes that the cluster intentionally reserved, so match the specific taint and verify the workload’s placement policy.
Know which scheduler filters are involved
Kubernetes’ NodeResourcesFit, NodeAffinity, and TaintToleration plugins represent three common filters: requested resources must fit, node-selection rules must match, and taints must be tolerated. A valid GPU limit can therefore coexist with zero eligible nodes. The event message, node labels, taints, and allocatable values together show which filter removed the candidates.
6. Investigate scheduling gates and Pod Scheduling Readiness
A scheduling gate deliberately keeps a Pod from being considered by the scheduler until another controller removes the gate. In that case, ordinary GPU capacity can be healthy and no amount of changing the resource limit will help. Inspect the Pod’s specification and status for scheduling gates, then identify the controller responsible for removing them. Check its logs and the condition that it is waiting for.
Pod Scheduling Readiness is stable starting in Kubernetes v1.30. Clusters running older versions, or clusters whose controllers do not support the relevant behavior, may expose different fields or semantics. Confirm the actual cluster version before relying on newer status fields. Remove a gate only after the prerequisite it represents is genuinely complete; bypassing it can create an unsafe or incomplete allocation.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →7. Check topology constraints and TopologyAffinityError
A Pod can fail after ordinary resource matching when locality requirements cannot be met. For example, the node may have a GPU, but not enough resources in the topology domain required by the device or by the Pod’s other constraints. Kubernetes also reports TopologyAffinityError when a topology-manager policy rejects the request.
Diagnostic path
- Run
kubectl describe pod <pod> -n <namespace>and capture the exact event or status condition. - Inspect labels used for topology, the Pod’s affinity and spread rules, and any device or resource-locality requirements.
- Check kubelet and node logs for topology-manager decisions.
- Compare the requested GPU count with what is available in the required locality, not merely the node or cluster total.
Do not “fix” a topology error by removing locality rules without understanding why they were required. Relaxing them may allow scheduling but can change performance, NUMA behavior, or device placement.
8. Investigate Dynamic Resource Allocation (DRA) when the cluster uses it
DRA is a different allocation path from a simple extended-resource limit. A Pod can refer to a ResourceClaim, while a DRA driver handles allocation and device preparation. Inspect the claim, its status, and the driver that owns it when ordinary node-resource checks do not explain the wait.
kubectl get resourceclaim -A
kubectl describe resourceclaim <claim> -n <namespace>
kubectl get pods -A | grep -i <dra-driver-name>Look for an unfulfilled claim, a driver that is not running on the required nodes, a binding condition that has not completed, or device preparation that failed. DRA device-binding conditions are described as beta in Kubernetes v1.36 documentation, so field names and behavior are version-sensitive. Confirm the cluster version and the DRA driver’s supported feature set before applying instructions that depend on those conditions.
Do not mix troubleshooting assumptions from the classic device-plugin path with DRA. In the classic path, the node’s extended resource and device-plugin registration are central. In DRA, the claim and driver state may be the decisive evidence even when a node advertises other GPU-related resources.
Common event messages and the next check
| Observed event or state | Most useful interpretation | Next checks |
|---|---|---|
Insufficient nvidia.com/gpu or another vendor key |
The requested integer resource does not fit on an eligible node, or the key/quantity does not match what is available. | Compare the exact key and limit with node Capacity/Allocatable; then check selectors, affinity, taints, and tolerations. |
| No GPU resource listed on a supposed GPU node | The driver, device plugin, registration, or health report is not producing a schedulable resource. | Inspect the plugin DaemonSet and logs, node conditions, kubelet state, and driver access. |
| Node selector or affinity failure | GPU nodes are being filtered by a label expression or required affinity rule. | Print actual labels and compare every required key and value with the Pod manifest. |
| Untolerated taint | A node reservation policy excludes the Pod. | Read the node taint and add only the matching, intentional toleration. |
| Scheduling gate present | A controller has deliberately withheld the Pod from scheduling. | Identify the gate owner, inspect its condition and logs, and wait for or repair gate removal. |
TopologyAffinityError |
Required topology locality or topology-manager policy rejected the request. | Inspect Pod events, topology labels, kubelet logs, and resources in the required locality. |
| DRA claim or preparation not ready | The claim or DRA driver has not completed binding or device preparation. | Describe the ResourceClaim and inspect the driver’s status and logs. |
A repeatable end-to-end workflow
- Capture the evidence. Run
kubectl get pod <pod> -n <namespace> -o wideandkubectl describe pod <pod> -n <namespace>. Save the latest event text before making changes. - Classify the phase. No assigned node means scheduling is unresolved. An assigned node means investigate kubelet, runtime, driver, or application behavior instead.
- Validate the request. Confirm the exact extended-resource key, integer quantity, and GPU limit syntax. If using DRA, identify the ResourceClaim instead.
- Validate node advertisement. Compare GPU
CapacityandAllocatableon candidate nodes. A missing or reduced value directs you to device health and plugin registration. - Validate placement. Compare node selectors, required affinity, taints, tolerations, and topology rules against the actual nodes.
- Check gates and claims. Inspect scheduling gates, ResourceClaims, DRA conditions, and device-preparation status when the ordinary filters pass.
- Repair the narrowest confirmed cause. Correct the resource key, free or repair the device, fix a label or toleration, restore the plugin, or address the responsible gate/driver. Avoid broad changes that hide the original constraint.
- Recheck events and node status. After the correction, verify that the node advertises the resource and that a new scheduling attempt produces a different event or assigns the Pod.
What not to conclude from a pending GPU Pod
- “The cluster needs more GPUs.” It may instead have free GPUs on nodes excluded by affinity, taints, topology, or a selector.
- “The image is broken.” An image cannot explain a Pod that has not been assigned to a node; inspect the scheduler event first.
- “Capacity equals availability.” Unhealthy devices can leave capacity unchanged while reducing allocatable devices.
- “A correct GPU limit guarantees scheduling.” The scheduler still needs a matching node advertisement and a placement that passes every filter.
- “A newer status field exists everywhere.” Pod Scheduling Readiness, DRA conditions, and device-health details have version-sensitive support; verify the cluster version and driver integration.
Version and implementation boundaries
The core GPU scheduling and device-plugin framework is documented as stable since Kubernetes v1.26. Pod Scheduling Readiness is stable since v1.30. Device-resource health status and DRA device-binding conditions are described as beta in v1.36 documentation. These labels describe Kubernetes feature maturity, not a guarantee that every vendor plugin, driver, or distribution exposes identical fields. Record the control-plane version, kubelet version, driver version, and plugin version when escalating a failure.
Use the classic extended-resource procedure only when the workload actually uses a device plugin. Use claim and driver diagnostics for DRA workloads. If the event does not clearly identify a cause, preserve the Pod description, node descriptions, plugin logs, and relevant kubelet events; changing several scheduling rules at once makes the next event harder to interpret.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →
