Driver FixRecommendedSound, Wi-Fi or graphics acting up? Check drivers firstFind missing or outdated drivers fast.Check DriversHalloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsClean PCRecommendedOne scan can reveal what keeps slowing WindowsLook for cleanup and repair opportunities.Run Scan×
Skip to content
World desk10 min

Which Hop Is Broken? Diagnosing Kubernetes Incidents Along the Request Path

A boundary-by-boundary method for locating Kubernetes request failures, from the entry point through Services, DNS, NetworkPolicy and the proxy, with commands and decision tables.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

The fastest way to find a failing or slow Kubernetes request is to stop asking whether “the network” or “Kubernetes” is at fault and walk the request path one boundary at a time, starting where the request enters. At each boundary, compare expected behavior with observed behavior. The first boundary where they diverge is where to dig.

The starting point matters. A request from outside the cluster, a request from another Pod, and a request from inside the same Pod exercise different hops. Ingress is only relevant when traffic enters through an HTTP or HTTPS routing rule, and kube-proxy is only relevant when it is the component implementing Services in your cluster. The sequence below treats both as conditional.

As an Amazon Associate I earn from qualifying purchases.

Start from where the request begins

The path depends on the caller. Use this table to choose the first components to check.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Request origin Hops it exercises Check first
Outside the cluster (browser, external client, partner system) External entry point, then Ingress, Gateway API or a LoadBalancer Service, then the Service, its endpoints and the Pod Host and path routing, controller or load balancer health, TLS configuration
Another Pod in the cluster DNS, then the Service IP, then the service proxy or Pod networking, then the destination Pod Name resolution, Service selector and endpoints, NetworkPolicy
The same Pod calling its own Service Service IP routed back to a backend that may be the caller itself Hairpin behavior, which depends on node and network configuration

Turn the symptom into a testable statement

Write the incident down before you touch the cluster. A useful record contains:

  • The caller, the destination, the protocol and, for HTTP, the hostname and path.
  • The expected result and the observed result, including status code, error text and latency.
  • The timestamp, with its time zone.
  • The scope: do all requests fail, or only requests from one namespace, one node or one set of backends?
  • A known-good comparison, such as the same request from a different caller, or a healthy backend behind the same Service.

These scoping questions are a method for isolating the incident, not a Kubernetes rule. Each answer removes a group of hops from consideration. If only Pods on one node fail, node-level networking and the proxy on that node move up the list. If every caller fails for one Service, the Service object and its backends move up.

When traffic enters from outside, check the external routing hop first

For HTTP and HTTPS, Ingress is the Kubernetes API for routing inbound traffic. Kubernetes’ Ingress API reference defines it this way: “Ingress is a collection of rules that allow inbound connections to reach the endpoints defined by a backend.” An Ingress object records routing intent. It does nothing unless an Ingress controller implements it, so a correct-looking Ingress with no running controller behaves like a missing route.

Ingress rules and the controller

  1. Confirm the Ingress exists and that its host and path match the failing request: kubectl get ingress -n <namespace>, then kubectl describe ingress <name> -n <namespace>.
  2. Confirm the backend Service named in the rule exists in the same namespace as the Ingress.
  3. Confirm the Ingress references an ingress class that a running controller serves. Check the controller’s Pods and logs in the namespace where it was installed; that namespace and its labels depend on the controller you use.

Gateway API and LoadBalancer Services

Some clusters route external traffic through Gateway API resources or through a Service of type LoadBalancer. The implementation differs by cluster and provider, but the logic is the same. Confirm which resource the external address points to, confirm the provider’s load balancer is healthy, and check whether its health checks are failing against the node or Pod ports it expects. Consult your provider’s documentation for how its load balancer is provisioned and health-checked.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Use an internal comparison

If the backend answers from inside the cluster but not from outside, the fault is most likely at the entry point. This is an inference from comparing boundaries, not a claim that Kubernetes prescribes a single route. If internal access also fails, the external hop is no longer the only suspect, and you should continue to the Service hop below.

Check the Service: selector, endpoints and ports

Kubernetes’ Service documentation describes the object this way: “The Service API lets you provide a stable (long lived) IP address or hostname for a service implemented by one or more backend pods.” The address is stable, but the Pods behind it change as they are rescheduled, scaled or replaced. EndpointSlices show which Pods currently back the Service. A Service object can be valid and still select no Pods, the wrong Pods, or Pods that do not listen on the port the Service targets. Kubernetes’ official Pod debugging guidance lists selector and target-port mismatches as first checks when endpoints or traffic are missing.

  1. Read the Service definition, including its selector, port and targetPort: kubectl get svc <name> -n <namespace> -o yaml.
  2. List the Pods that match the selector from step 1: kubectl get pods -n <namespace> -l <key>=<value> -o wide. If nothing is returned, the selector and the Pod labels disagree.
  3. Check the EndpointSlices for the Service: kubectl get endpointslices -n <namespace> -l kubernetes.io/service-name=<name>. The expected Pod IPs and port should appear. An endpoint marked not ready is normally excluded from receiving traffic, so check that Pod’s readiness probe next.
  4. Compare targetPort with the port the application actually listens on. If the image includes ss, run kubectl exec -it <pod> -n <namespace> -- ss -ltn inside the Pod. If it does not, check the application’s configuration and its container definition.

Separate DNS from Service routing

From the caller’s perspective, two different faults look the same: the name does not resolve, or the name resolves but the address does not route. Test them separately and in this order.

Test name resolution from a Pod

  1. Read the resolver configuration inside a Pod in the caller’s namespace: kubectl exec -it <pod> -n <namespace> -- cat /etc/resolv.conf. The nameserver should be the cluster DNS Service address, and the search path should match the Pod’s namespace and the cluster domain. Kubernetes’ official DNS debugging guidance notes that search paths and resolver behavior can vary by provider and environment.
  2. Query the Service by its short name and by its fully qualified name: kubectl exec -it <pod> -n <namespace> -- nslookup <service>.<namespace>.svc.cluster.local. Use an image whose nslookup output you understand, because resolver tool behavior differs between images.
  3. If cluster DNS itself is suspect, check the kube-dns Service and its EndpointSlices in the kube-system namespace: kubectl get svc kube-dns -n kube-system and kubectl get endpointslices -n kube-system -l kubernetes.io/service-name=kube-dns. Clusters that run CoreDNS may expose it through a Service with a different name, so confirm the name in your cluster first.

Test the Service IP directly

From the same Pod, request the Service’s ClusterIP and port with a short timeout, so that a silent failure does not hang the session: kubectl exec -it <pod> -n <namespace> -- wget -qO- -T 5 http://<cluster-ip>:<port>/. Use curl if the image provides it. Kubernetes’ Service guidance separates these DNS and IP checks in exactly this order.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Name resolves Service IP responds Investigate first
No Yes DNS: resolv.conf, search path, cluster DNS Pods and the kube-dns Service
Yes No Service definition, EndpointSlices, NetworkPolicy and the service proxy
Yes Yes, but the caller still fails Application-level behavior: protocol, port, TLS, timeouts or the caller’s own configuration
No No Continue through Service configuration, endpoints, policy and proxy implementation; the IP test shows whether DNS is the only failing part

Check policy and the service proxy that actually runs

Only after the Service and its endpoints check out should you move to the data plane. Two questions come first: is a NetworkPolicy blocking the traffic, and which component implements Services in this cluster?

NetworkPolicy

When a NetworkPolicy selects a Pod for ingress, traffic to that Pod that no policy allows is dropped. Review the policies in the destination namespace with kubectl get networkpolicy -n <namespace> -o yaml. Look for a policy whose podSelector matches the destination Pods and whose ingress rules admit the caller’s namespace or labels on the destination port. Enforcement depends on the network plugin. A cluster whose plugin does not enforce NetworkPolicy will not show a policy-based block at all, so a policy that looks correct on paper may have no effect.

Confirm the Service proxy implementation

kube-proxy is Kubernetes’ common default implementation for Services, but it is not the only option. Some Pod networking implementations supply their own service proxy. Before you run kube-proxy commands, establish which component is in use from your installation method or your network add-on’s documentation. Running kube-proxy checks in a cluster where another component handles Services wastes time and can point you at the wrong layer.

If kube-proxy is the implementation

  1. Confirm kube-proxy is running on the node where the caller is scheduled: kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide. This label is common in kubeadm-style installations; confirm it for yours.
  2. Read the kube-proxy logs for that node: kubectl logs -n kube-system <kube-proxy-pod>.
  3. Verify that the expected Service endpoints are programmed on that node. Kubernetes’ Service guidance gives examples for iptables and IPVS modes. The exact commands depend on the mode and the operating system, so confirm both before you run anything on the node.

Hairpin: a Pod calling its own Service

A Pod that reaches its own Service IP can fail while other callers succeed, because the traffic is routed back to the caller. Whether this works depends on node and network configuration, and Kubernetes’ Service documentation covers it as an edge case. If only self-calls fail, test from a different Pod before changing the Service.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Correlate logs, metrics and traces

Logs, metrics and traces answer different questions, and Kubernetes’ observability documentation treats them as complementary. Use them in that order of specificity, not interchangeably.

Best Value
Kubernetes Software - Powerful Container Orchestration Tools T-Shirt
  • Kubernetes is an open platform that automates container orchestration, enabling seamless deployment, automatic scaling, self-healing, and efficient management of applications across servers or clouds with high availability and optimal resource use
  • Kubernetes is perfect for development operations engineers, cloud architects, site reliability engineers, platform engineering teams and infrastructure specialists who build, operate and maintain modern containerized applications in production environments
  • Lightweight, Classic fit, Double-needle sleeve and bottom hem
Signal Answers Use it when
Logs What ran, and what it reported, at a given time You need the error text or the sequence of events on one component
Metrics How a resource or service behaves over time You need to know whether a failure is isolated or part of a pattern, such as rising error rates or saturation on one node
Traces How one request moved across operations and how long each step took You need the end-to-end flow or the step where latency appears, across components and applications

Control plane and kubelet traces

Kubernetes components can export spans in OTLP format, either to an OpenTelemetry Collector or directly to a backend, in supported configurations. According to the Kubernetes system tracing documentation, kube-apiserver spans cover incoming HTTP requests and calls such as webhooks and etcd, while kubelet spans cover CRI and authenticated HTTP operations. That documentation states that kubelet tracing has been stable since v1.34. Confirm the feature status for your own Kubernetes version before relying on it.

Trace export adds CPU and network overhead, and the amount depends on configuration. Sampling and deployment choices matter, so measure the effect in your own cluster rather than assuming it is negligible.

Record the environment before naming a cause

The same symptom can have different causes on different platforms, and advice that ignores the platform can send you to the wrong layer. Record these facts with the incident:

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Kubernetes version
  • Cloud provider, or on-premises environment
  • Operating system distribution of the nodes
  • Network configuration, including the Pod network plugin and whether it enforces NetworkPolicy
  • Container runtime and its version
  • The Service proxy implementation and mode, the DNS setup, and the Ingress or Gateway controller in use
  • A minimal reproduction: the smallest manifest or command that still fails

Kubernetes’ debugging overview asks for most of these details when an issue is reported, and the same list makes an incident record reproducible.

Choosing a tracing backend

Kubernetes’ observability documentation names several tracing projects, including Grafana Tempo, Jaeger, the OpenTelemetry Collector and Zipkin. The list is not a ranking. The OpenTelemetry Collector is vendor-agnostic and can receive, process and export Kubernetes telemetry, so it can sit in front of whichever backend you choose. Compare backends on these criteria:

  • Hosting: whether your team needs a self-managed or a hosted service.
  • OTLP ingestion and Kubernetes fit: how the backend receives OTLP and how well it fits your cluster setup.
  • Storage and query needs: retention, data volume and the query patterns your engineers use.
  • Correlation: how traces sit alongside your existing metrics and logs.
  • Operational effort: upgrades, scaling and on-call burden.
  • Cost and data handling: pricing model, and where trace data is stored, since spans can carry request metadata.

These criteria are decision points for your team. The Kubernetes and OpenTelemetry documentation establishes that these tools exist and how OTLP export flows to a collector or backend; it does not rank them.

What this framework can and cannot tell you

  • It narrows the search. A failing boundary shows where to look, but it is not proof of root cause until configuration, logs or trace evidence confirms it.
  • It is not a runbook for every managed distribution. Ingress controllers, network plugins, DNS resolvers, operating systems and Kubernetes versions all change commands and expected behavior.
  • The commands above are general forms built on standard Kubernetes resources. Check their output against your own cluster before drawing conclusions from it.

“

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Leave a Reply

Your email address will not be published. Required fields are marked *

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Windows Errors? Fix Them Before They SpreadFree repair scan
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.