Kubernetes does not use one timer for node failure and pod eviction. A node’s heartbeat determines when the control plane recognizes a problem; node conditions and taints then interact with each pod’s tolerations, while eviction rate limits and control-plane connectivity affect what happens next. The documented defaults include a 10-second Lease update interval, a 50-second node-monitor grace period, and automatic 300-second tolerations for the not-ready and unreachable taints—but those values do not guarantee a fixed time to rescheduling.
How the failure-to-eviction sequence works
Think of the process as distinct stages rather than a single “node failure timeout.” The Kubernetes documentation gives defaults for several stages, but actual behavior also depends on cluster version, controller configuration, pod tolerations, and whether the control plane can reach the node.
As an Amazon Associate I earn from qualifying purchases.
- Heartbeat: The kubelet reports node health through Node status updates and Lease objects in the
kube-node-leasenamespace. Kubernetes documents a default Lease update interval of 10 seconds. Node status has a separate update cadence; its documented default interval is five minutes, though updates can also be triggered by changes. See Node Status. - Failure recognition: The node controller waits for the configured
--node-monitor-grace-period. The Node Status reference lists 50 seconds as the documented default. This is a heartbeat grace period, not a promise that every pod will be rescheduled 50 seconds after a machine stops responding. - Condition and taint: A node reporting
Ready=Falseis unhealthy;Ready=Unknownmeans the controller has not heard from it within the grace period. The corresponding failure taints arenode.kubernetes.io/not-readyandnode.kubernetes.io/unreachable.Unknownindicates lost communication;Falseindicates the node reports itself as not ready. - Pod eviction eligibility: These taints have the
NoExecuteeffect. A pod without a matching toleration is eligible for immediate eviction by the taint-based mechanism. A matching toleration withouttolerationSecondspermits the pod to remain bound indefinitely. WithtolerationSeconds, it may remain bound for that duration after the taint is added, unless the taint is removed first. - Eviction request and recovery: Controller behavior and rate limits affect when eviction requests are submitted and processed. Even after a deletion request, a partitioned node may continue running its local process if the control plane cannot reach it; eviction does not itself fence a machine.
The Kubernetes Nodes documentation describes the node controller waiting five minutes after marking a node Unknown before submitting its first eviction request. This is a separate controller description from the automatic five-minute pod toleration discussed below; do not treat the two descriptions as one universal additive timer. Version and controller configuration influence the execution path. See Nodes.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsWhat the documented defaults mean
| Setting or behavior | Documented value | How to interpret it |
|---|---|---|
| Kubelet Lease update interval | 10 seconds | Default heartbeat cadence for Lease objects, as documented in the Kubernetes Node Status reference; it is not the failure grace period. |
--node-monitor-grace-period |
50 seconds | Documented default period without hearing from a node before the controller can set Ready=Unknown. |
Automatic tolerations for not-ready and unreachable |
300 seconds | Default tolerationSeconds added to pods unless the pod or its controller sets those tolerations explicitly. DaemonSet pods have indefinite tolerations for these two taints. |
Node controller initial wait after marking Unknown |
5 minutes | Nodes documentation describes this wait before the first eviction request; it is not the same setting as the automatic pod toleration. |
--node-eviction-rate |
0.1 node per second | Documented default rate, equivalent to one node per 10 seconds, subject to zone and cluster-health behavior. |
These are Kubernetes documentation defaults, not guaranteed timings for every distribution or managed service. The 50-second and 10-second values appear in the Node Status reference; the automatic tolerations and controller behavior are described in Taints and Tolerations; node-controller waiting and eviction-rate caveats are in Nodes.
#1 Best Overall
Set a per-pod eviction delay
For an ordinary pod, configure matching NoExecute tolerations in its PodSpec. This example allows the pod to remain bound for 600 seconds after either failure taint is applied:
tolerations:
- key: "node.kubernetes.io/unreachable"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 600
- key: "node.kubernetes.io/not-ready"
operator: "Exists"
effect: "NoExecute"
tolerationSeconds: 600The 600-second value is illustrative, not a Kubernetes recommendation. Apply the tolerations to the relevant pod template when a controller such as a Deployment or StatefulSet creates the pods; editing only an existing pod may not provide a durable workload-level configuration.
To change the delay, adjust tolerationSeconds on the matching toleration. A longer value can avoid eviction during brief communication loss but delays recovery after genuine node failure. A shorter value can make recovery eligible sooner, while increasing the risk of acting on a transient partition. Omitting tolerationSeconds from a matching toleration means it tolerates the taint indefinitely.
Kubernetes automatically adds 300-second tolerations for these two taints unless the pod or its controller explicitly specifies them. The project documentation explains that the automatic tolerations keep pods bound to nodes for five minutes after one of these problems is detected. DaemonSet pods use indefinite tolerations for both taints. Details are in Taints and Tolerations.
Rank #3
Configure cluster-level failure detection
For cluster-level detection behavior, the relevant kube-controller-manager flags documented here are --node-monitor-grace-period and --node-monitor-period. The grace-period flag controls how long the controller waits without node communication; the monitor-period flag is also relevant to node monitoring. Changing heartbeat reporting cadence alone does not change the grace-period threshold.
- Inspect the control-plane configuration for the Kubernetes version and distribution you actually run before changing either flag.
- Check whether your provider exposes these controller-manager settings. Managed Kubernetes services may expose only a subset of control-plane configuration.
- Review eviction-rate behavior as well as detection timing. The documented default
--node-eviction-rateis 0.1 node per second, with behavior affected by zone and cluster health; mass failures may therefore be handled differently from a single-node failure.
Since Kubernetes 1.29, taint-based eviction is performed by the separate taint-eviction-controller. The Kubernetes documentation says it can be disabled through --controllers=-taint-eviction-controller in kube-controller-manager. Check the actual controller configuration and manifests rather than assuming every cluster uses identical defaults. See Taints and Tolerations.
Choose a delay for the workload, not just the timer
There is no universally safe toleration duration. Decide based on how costly a false eviction is compared with delayed recovery, and whether the old process could continue doing work during a partition.
- Stateful workloads: Check storage attachment and detach behavior before shortening the delay. A replacement pod must not assume the prior node has stopped using data simply because the control plane requested eviction.
- Replicated services: Confirm replica placement and application-level safeguards. A pod that keeps running on an isolated node while a replacement starts elsewhere can create duplicate work or conflicting writes.
- Transient network interruptions: A longer delay may reduce unnecessary churn, but means a truly failed node can take longer to yield an eligible replacement.
- Hard failure or partition scenarios: Plan for fencing or another workload-specific mechanism if duplicate execution is unsafe. Eviction timing alone does not guarantee that the old process has stopped.
For an operational estimate, treat the heartbeat interval, grace period, taint application, toleration duration, controller wait and rate limit as separate contributors. Do not simply add the documented defaults and promise that total as a rescheduling time: controller paths differ, rate limiting can apply, and API-server reachability affects whether deletion reaches the kubelet.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
