Halloween DealsAmazon USHalloween week deals: useful picks before month-endAmazon US: current deals, useful picks and tech finds.Check DealsPC HealthRecommendedCrashes, freezes, slowdowns? Check your PC nowSpot repairable issues before they interrupt work.Check PCHalloween DealsAmazon USOctober picks are moving - compare before checkoutAmazon US: current deals, useful picks and tech finds.See Picks×
FREEDOM251

Freedom251 › Blog › Docker

A Failing Docker Swarm Service and Healthy Containers Losing Overlay Connectivity

By Freedom251 Team · · 4 min read

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

A failing Swarm service does not, by itself, explain why other healthy containers lost overlay connectivity. Diagnose the service and its tasks separately from network attachment and the inter-node data path; without task history, network inspection, node and firewall details, and logs, the cause of this incident cannot be identified.

Why can healthy containers no longer communicate over a Docker Swarm overlay network?

Swarm services describe desired state, including which networks a service should use; tasks are the running or attempted instances that implement that state. A task may be restarting, rejected, or pending while a container elsewhere reports healthy. Neither service status nor a container health check establishes that peers can communicate across an overlay.

As an Amazon Associate I earn from qualifying purchases.

Start by separating the control state (what Swarm wants and where tasks are placed) from actual network reachability. Treat task failure, missing network membership, and a broken cross-node data path as distinct diagnostic branches, not as proof of a single cause.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

How do I check whether a Swarm service task is attached to an overlay network?

  1. On a manager, run docker service ls, then docker service ps <service-name>. Record desired and current states, node placement, task errors, and task history. Docker documents docker service ps as a way to list tasks and their networks. Identify whether the affected containers are tasks of the failing service or belong to other services.

    #1 Best Overall
    Sale
    NETGEAR 5-Port Gigabit Ethernet Unmanaged Network Switch (GS305) | Home Network Hub, Office Ethernet Splitter, Plug-and-Play, Silent Operation, Desktop or Wall Mount, Essentials Model
    • GIGABIT ETHERNET PORTS: Features 5 x 1.0Gbps Ethernet ports for high-speed connectivity. Auto-negotiating ports detect the optimal speed for connected devices and work with existing Cat5e or Cat6 Ethernet cables.
    • PLUG-AND-PLAY UNMANAGED NETWORK SWITCH: Simple plug-and-play setup with no software to install or configuration required.
    • FLEXIBLE MOUNTING OPTIONS: Compact metal design supports desktop or wall-mount placement for versatile installation.
    • SILENT & ENERGY-EFFICIENT OPERATION: Fanless design ensures silent performance, while IEEE 802.3az Energy Efficient Ethernet reduces power consumption without compromising high-speed network performance.
    • REGIONAL COMPATIBILITY: Made for use in U.S. & CA only
  2. Inspect the service definition and its network list, then run docker network inspect <network-name> for the named overlay. Compare the failing service’s tasks with the peer services and the containers or tasks shown on that network. Check whether every participant is attached to the same intended overlay.

  3. Compare the task timeline with the first reported connectivity failure. Swarm managers try to reconcile actual tasks with desired service state, so a current service summary alone may not show when tasks were replaced or where they ran.

Network membership is configurable: Docker supports adding an overlay to a new or existing service and removing one during an update. That makes an unintended removal or mismatched attachment worth checking, but neither is established as the explanation here. Do not change the service until inspection indicates what is wrong; Docker documents docker service update --network-add <network> <service> and --network-rm for configuration changes.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Rank #2
Sale
TP-Link TL-SG105, 5 Port Gigabit Unmanaged Ethernet Switch | Network Hub, Ethernet Splitter, Plug & Play, Fanless Metal Design, Shielded Ports, Traffic Optimization
  • 𝗢𝗻𝗲 𝗦𝘄𝗶𝘁𝗰𝗵 𝗠𝗮𝗱𝗲 𝘁𝗼 𝗘𝘅𝗽𝗮𝗻𝗱 𝗡𝗲𝘁𝘄𝗼𝗿𝗸: 5× 10/100/1000Mbps RJ45 Ports supporting Auto Negotiation and Auto MDI/MDIX.
  • 𝗚𝗶𝗴𝗮𝗯𝗶𝘁 𝘁𝗵𝗮𝘁 𝗦𝗮𝘃𝗲𝘀 𝗘𝗻𝗲𝗿𝗴𝘆: Latest innovative energy-efficient technology greatly expands your network capacity with much less power consumption and helps save money.
  • 𝗥𝗲𝗹𝗶𝗮𝗯𝗹𝗲 𝗮𝗻𝗱 𝗤𝘂𝗶𝗲𝘁: IEEE 802.3X flow control provides reliable data transfer and Fanless design ensures quiet operation.
  • 𝗣𝗹𝘂𝗴 𝗮𝗻𝗱 𝗣𝗹𝗮𝘆: Easy setup with no software installation or configuration needed.
  • 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀: Prioritize your traffic and guarantee high quality of video or voice data transmission with Port-based 802.1p/DSCP QoS and IGMP Snooping.

Does the failure happen only between nodes?

Test communication between affected tasks or containers on the same host, then across different hosts. A failure limited to cross-node traffic points toward the inter-node path; same-host failures mean you should not assume that opening an inter-node port will address the problem.

For multi-node Swarms, Docker documents these inter-node requirements:

  • TCP and UDP 7946: used for network discovery.

  • UDP 4789: used for the overlay data path by default. If the cluster is configured with an alternate data-path port, verify that port instead.

    Rank #3
    Sale
    NETGEAR 8-Port Gigabit Ethernet Unmanaged Network Switch (GS308) | Home Network Hub, Office Ethernet Splitter, Plug-and-Play, Silent Operation, Desktop or Wall Mount, Essentials Model
    • GIGABIT ETHERNET PORTS: Features 8 x 1.0Gbps Ethernet ports for high-speed connectivity. Auto-negotiating ports detect the optimal speed for connected devices and work with existing Cat5e or Cat6 Ethernet cables.
    • PLUG-AND-PLAY UNMANAGED NETWORK SWITCH: Simple plug-and-play setup with no software to install or configuration required.
    • FLEXIBLE MOUNTING OPTIONS: Compact metal design supports desktop or wall-mount placement for versatile installation.
    • SILENT & ENERGY-EFFICIENT OPERATION: Fanless design ensures silent performance, while IEEE 802.3az Energy Efficient Ethernet reduces power consumption without compromising high-speed network performance.
    • REGIONAL COMPATIBILITY: Made for use in U.S. & CA only

Discovery and overlay data transport are different functions, so a block affecting one is not interchangeable with a block affecting the other. Check routing and firewall reachability between Swarm nodes under the cluster’s security policy; these requirements are not a reason to expose ports broadly to the public internet.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Did the connectivity failure coincide with an update or task replacement?

Compare the first failure time with service updates, node availability changes, and task replacements. If symptoms began during a deployment, inspect the update status and policy as well as task history. Docker documents a default --update-monitor period of 30 seconds: a task failure in that period counts toward the service update failure threshold, while a failure after it does not. This timing helps interpret update outcomes, but it does not prove that an update caused overlay connectivity to fail.

Could the number of containers on a host matter?

Check scale only if the deployment approaches Docker’s documented condition. Docker Docs’ “Overlay network driver” warns that Linux kernel limitations can make overlay networks unstable and inter-container communication may break when 1,000 containers are co-located on one host. This is a specific co-location condition, not a universal container limit or an explanation to apply to a smaller deployment without matching evidence.

Rank #4
Sale
TP-Link LS1005G, LiteWave 5 Port Gigabit Ethernet Unmanaged Switch | Desktop Ethernet Splitter, Unshielded Network Switch, Plug & Play, Fanless Quiet
  • 【One Switch Made to Expand Network】Features 5 RJ45 ports with 10/100/1000Mbps speeds, supporting Auto-Negotiation and Auto MDI/MDIX for hassle-free setup. Ideal for expanding your network, with 1 uplink (input) port and 4 output ports to split your Ethernet connection to multiple devices.
  • 【Gigabit that Saves Energy】Latest innovative energy-efficient technology greatly expands your network capacity with much less power consumption and helps save money
  • 【Reliable and Quiet】IEEE 802.3X flow control provides reliable data transfer and Fanless design ensures quiet operation
  • 【Plug and Play】Easy setup with no software installation or configuration needed
  • 【Ethernet Splitter】Connect to your router or modem for additional wired connections (laptop, gaming console, printer, etc)
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

What evidence is needed to identify this incident’s cause?

The available incident description does not include the details needed to distinguish these branches or name a root cause. Gather the following for the affected time window:

With those artifacts, an operator can determine whether the evidence points to task state, network membership, cross-node transport, deployment timing, or the documented high-density condition. Without them, attributing the loss of reachability to the failing service would go beyond what is known.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

DockerDocker SwarmNetwork Troubleshootingoverlay networking

Leave a Reply

Your email address will not be published. Required fields are marked *

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Recommended PC Tool
Recommended PC Tool
Outdated Drivers Are Slowing You DownFree scan - exact matches
Windows Errors? Fix Them Before They SpreadFree repair scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.