When a Kubernetes node stops reporting, the control plane marks it unreachable, taints it, and eventually may evict its pods. A replacement can be created by the workload’s controller, but it is a new pod—not the same pod moved elsewhere. If the node is only cut off by a network partition, its original processes may still be running even after the API shows those pods deleted.
What Kubernetes does when a node stops reporting
Kubernetes tracks node health using status updates and Lease objects. If the control plane does not hear from a node within the configured node-monitor-grace-period, the node controller changes its Ready condition to Unknown and applies the node.kubernetes.io/unreachable taint with NoExecute behavior. The documented default grace period is 50 seconds; cluster operators can configure a different value. See the Kubernetes node status documentation and node reference.
As an Amazon Associate I earn from qualifying purchases.
That 50-second default is a detection threshold, not an end-to-end recovery promise. After the taint is applied, pod tolerations determine whether and when existing pods become eligible for eviction. Scheduling constraints, capacity, storage, and controller behavior then affect whether and when replacements run.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →How long before Kubernetes evicts a pod?
For ordinary pods, Kubernetes automatically adds a 300-second toleration for the unreachable and not-ready taints unless the pod or a controller supplies different tolerations. The five-minute window begins when the taint is applied, not necessarily at the moment the node first fails. The documented defaults and behavior are described in Kubernetes taints and tolerations.
#1 Best Overall
- No matching
NoExecutetoleration: the pod is eligible for eviction once the taint is applied. - Finite
tolerationSeconds: the pod remains bound for the specified period after the taint, then becomes eligible for eviction. - No
tolerationSeconds: a matching toleration can keep the pod bound indefinitely. - DaemonSet pod: DaemonSet pods tolerate unreachable and not-ready taints indefinitely, so those taints do not evict them.
Since Kubernetes 1.29, a separate taint-eviction-controller handles taint-based eviction. Cluster administrators can disable it through kube-controller-manager configuration, so the release and configuration in a particular cluster matter.
Does Kubernetes restart the pod on another node?
No. Kubernetes does not transfer a pod’s binding to another node. The failed-node pod may eventually be deleted from the API, and an owning controller can create a replacement pod to restore the desired state. The replacement has a different UID and is scheduled as a new pod; it may land on another suitable node, but placement depends on available capacity, affinity and topology rules, storage, and other scheduling constraints. As the Pod Lifecycle documentation puts it, a pod is not rescheduled as the same object; it can be replaced by a near-identical pod.
What happens next depends in part on the workload type:
- Deployment or ReplicaSet: the controller can create a replacement to meet the desired replica count.
- StatefulSet: the controller manages replacement according to StatefulSet identity and storage behavior; safe recovery can depend on whether the previous instance is truly stopped and its storage is available.
- Job or another controller: recovery follows that controller’s own desired-state and retry behavior.
- Standalone pod: there may be no higher-level controller to create a replacement.
A Pod Disruption Budget is not a guarantee against node-failure eviction. PDBs govern voluntary disruptions through the eviction API; hardware failure and network partition are involuntary disruptions. See Kubernetes Pod disruptions.
Rank #3
Why an unreachable pod may still be running
Missing heartbeats do not tell Kubernetes with certainty whether a machine is powered off or merely disconnected from the control plane. In a network partition, the API server can record a pod’s deletion without being able to deliver that deletion to the kubelet. The original process may keep running on the isolated node while a replacement starts elsewhere. Kubernetes documents this possibility in its taints and tolerations guidance.
That creates a risk of two active instances doing work at once, especially for stateful services or workloads that assume exclusive ownership. An API-level deletion is not proof that the process stopped. Before forcing recovery, consider whether the old node is fenced, whether the application uses reliable leadership or leases, and whether storage can safely be attached to a replacement.
Check node, pod, and toleration state
- Run
kubectl describe node <node-name>and inspect theReadycondition, recent events, and taints. The node reference documents this command for viewing node conditions. - Run
kubectl get pods -o wideto see which pods are associated with the node and whether replacement pods have been created. - Inspect the affected pod’s tolerations and owner references to establish its eviction window and whether a controller should create a replacement.
- If a replacement is
Pending, inspect scheduler events and constraints such as available capacity, affinity, topology, and volume requirements. - For stateful workloads, verify the node and storage state before attempting force deletion or volume detachment. Deleting a pod object alone does not establish that an unreachable process has stopped.
When force-detaching storage may be appropriate
If a node has truly shut down non-gracefully and normal storage recovery is blocked, Kubernetes documents an node.kubernetes.io/out-of-service taint workflow that can force deletion of pods and immediate volume detach. This is an administrator procedure, not a routine response to missing heartbeats: first verify the machine is shut down and will not restart while the workload may still be active. Kubernetes warns that force-detaching a volume while the old node is still running the workload can violate storage ordering expectations and risk data corruption.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The Kubernetes documentation describes forced volume detach after a six-minute deletion timeout as an optional, configuration-dependent behavior; it is not a universal recovery timer. After the node recovers and migrated pods have been checked, remove the out-of-service taint. See the node shutdown documentation.
Why recovery time varies
The documented defaults provide useful reference points, but not a fixed service-level timeline. Actual recovery depends on the configured node-monitor grace period, pod tolerations, whether taint eviction is enabled, the workload controller, scheduler capacity and constraints, cloud-provider behavior, and storage integration. Treat the 50-second detection default and 300-second ordinary-pod toleration as separate intervals: neither alone tells you when an application will be serving again.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
