The 3-second detection and 13-second traffic window are incident-specific observations, not Kubernetes-wide defaults. Kubernetes uses several separate clocks: node heartbeats and health checks, the grace period before a node is considered unhealthy, taints and Pod tolerations, eviction requests, EndpointSlice updates, and the time your proxy or load balancer takes to apply those updates. Without the cluster version, controller settings, networking implementation and timestamped events, the two intervals cannot be reconstructed from Kubernetes defaults alone.
What the two timings actually tell you
A node can be recognized as unhealthy before traffic stops reaching applications that were running on it. Detection is a control-plane decision; removing a backend is a separate process. Kubernetes documentation describes node heartbeats as the signal used to determine node availability and take action when failures are detected.
As an Amazon Associate I earn from qualifying purchases.
If the node is partitioned rather than powered off, the control plane may be unable to tell the kubelet to stop or delete anything. A process on the isolated host can therefore continue serving, even while Kubernetes marks the node unhealthy.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Kubernetes’ published timing defaults
These are upstream documentation defaults, not measurements of the incident in the title. A managed Kubernetes service or distribution may change controller flags and other components.
#1 Best Overall
| Stage | Published value | What it controls |
|---|---|---|
| Node-controller state check | 5 seconds | How often the node controller checks node state by default. |
| Unknown-to-eviction delay | 5 minutes | Default delay after a node is marked Unknown before the first eviction request is submitted. |
| Automatic not-ready/unreachable tolerations | 300 seconds | How long Pods normally tolerate node.kubernetes.io/not-ready or node.kubernetes.io/unreachable taints unless their tolerations are changed. |
| Node eviction rate | 0.1 nodes per second in most cases | The default rate at which node evictions are initiated, subject to cluster-scale and failure-zone behavior. |
The 5-second check is not a 5-second failure detector. Heartbeat loss must pass the configured health grace period before the node condition changes, and later stages still have their own delays.
Why a Pod can keep running after the node is unreachable
Control-plane communication is broken
During a network partition, the API server may update the Node object and schedule Pod deletion without being able to deliver that deletion to the kubelet. The workload process on the partitioned machine can remain alive and continue accepting connections.
Taints and tolerations postpone replacement
Kubernetes adds 300-second tolerations for the standard not-ready and unreachable taints unless a Pod or its controller specifies different tolerations. This affects when a Pod is eligible for eviction; it does not itself remove an old process from an isolated host.
Eviction is not endpoint withdrawal
Eviction requests, Pod termination, EndpointSlice changes and data-plane programming are related but distinct events. A short observed interval may result from custom health checks or aggressive settings, while a longer interval may include toleration, eviction or backend-refresh delays.
Why traffic can continue after Kubernetes starts terminating a Pod
EndpointSlice readiness changes
For ordinary service traffic, a terminating EndpointSlice endpoint has ready=false, so conforming load balancers should stop selecting it for new requests. The serving condition can be used to drain existing connections, so established sessions may outlive the point at which new selections stop.
Data-plane refresh time varies
kube-proxy, an ingress controller, a service mesh, cloud load balancers and other proxies each have their own watch, programming and refresh behavior. The control plane can publish an endpoint update before every traffic-handling component has applied it. Consequently, “Pod is terminating” and “no new request reaches that backend” are not the same timestamp.
External load balancers add another clock
A cloud or hardware load balancer may run an independent health check and backend-removal process. If the 13-second interval was measured at the client, include connection reuse, retries and load-balancer health-check settings before attributing it to Kubernetes.
Recommended Free Tools
How to investigate the 3-second and 13-second intervals
- Identify the implementation. Record the Kubernetes version, distribution, cloud provider, kube-proxy mode, CNI, ingress or service-mesh components, and the external load balancer.
- Build one timestamped timeline. Compare the last node heartbeat or lease update, Node condition transition, taint creation, Pod termination timestamp, EndpointSlice condition changes, and the first point at which the actual proxy or load balancer removed the backend.
- Inspect node state. Use
kubectl get node <node> -o yamlandkubectl describe node <node>to examine conditions, taints, leases and events. Do not treat the time of an alert as the time Kubernetes changed the Node object. - Inspect the affected Pod. Use
kubectl get pod <pod> -n <namespace> -o yamland checkdeletionTimestamp, tolerations, readiness gates, termination settings and container state. - Inspect EndpointSlices. Run
kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service> -o yaml. Compare the endpoint’sready,servingandterminatingconditions with proxy and load-balancer logs. - Verify the data path. Capture the backend list and health status from the ingress, mesh or external load balancer at the same timestamps. Check whether the observed requests were new connections, reused connections or retries.
- Check controller configuration. Compare effective node-monitor, eviction and taint-related settings with upstream defaults. Managed services may expose different values or prevent direct changes.
What would explain a genuinely three-second failure declaration?
A three-second declaration is more plausibly a custom node health check, an external monitoring system, a cloud load-balancer probe or a locally configured controller than the general upstream sequence. The node-controller source comments on the moving main branch say the health grace period should allow multiple health-signal intervals and exceed the HTTP/2 health-check ping plus read-idle timeout (30 seconds plus 15 seconds in those comments). Those comments are not a version-independent setting; use the release branch and effective configuration for a precise explanation.
Best Value
Practical interpretation
If traffic stopped 13 seconds after an observed failure, that number could represent endpoint propagation and data-plane programming rather than node detection. If traffic continued for 13 seconds after a node alert, the alert may have preceded the Node condition, or the client may have been observing connection reuse. Only correlated control-plane and data-plane timestamps can distinguish those cases.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
