Hardware FixRecommendedDevice not working? Your driver may be the problemCheck updates for common hardware issues.Fix DriversOctober DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsWindows FixRecommendedWindows errors stealing your time? Find the fix fastScan stability, cleanup and performance issues.Fix Now×
Skip to content
World desk5 min

Network Partitions: Four Ways to Keep a Distributed System Safe

A network split does not have to mean competing writes. Learn how quorum, zone placement, client behavior, and recovery plans keep a distributed system safe.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

A distributed system can survive a network split without keeping every part writable. The key is to define which side may make authoritative changes, keep replicas and access paths resilient to the failures you expect, make clients handle interruptions safely, and prepare to restore quorum and catch up afterward. In quorum-based systems such as etcd, the majority side can continue while the minority stops accepting consensus-dependent writes; that trade-off protects a single authoritative history at the cost of partial availability.

1. Let quorum decide which side can commit

A network partition divides cluster members into groups that cannot communicate. In etcd, configured membership determines the quorum: the majority side remains available, while the minority is unavailable. If the leader is isolated in the minority, it steps down and the majority elects a new leader. When connectivity returns, the minority recognizes the majority’s leader and recovers its state. etcd’s failure guide describes this behavior.

As an Amazon Associate I earn from qualifying purchases.

This is a deliberate consistency-versus-availability choice. Restricting authoritative writes to the quorum side avoids allowing disconnected groups to create competing histories. It also means users whose requests reach the minority may see failures or delays. A cluster cannot promise both unrestricted writes on every isolated side and one authoritative history without some reconciliation or consistency cost.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Quorum is measured against the configured cluster membership, not simply the number of nodes that happen to see one another. If a majority of members is unavailable, operations requiring consensus—including writes—cannot proceed until quorum returns or operators recover the cluster. Raft-based behavior and its availability trade-off are system-specific; do not assume every distributed database or service uses identical rules. RabbitMQ’s partition guide documents the behavior of its quorum queues and related Raft-based features.

#1 Best Overall
FortiGate-40F Firewall Appliance - 5 Gigabit Ethernet RJ45 Ports, Ideal for Small Businesses (Appliance Only, No Subscription) (FG-40F)
  • Compact and Efficient Design: The FortiGate 40F is designed for small to mid-sized businesses and enterprise branch offices, featuring a compact, fanless desktop form factor that ensures quiet operation and minimizes space usage.
  • Robust Connectivity Options: Equipped with 5 GE RJ45 ports, including 1 WAN port and 4 internal ports, this model provides essential connectivity and flexibility for various network configurations in a small-scale environment.
  • High-Performance Security: Offers up to 1 Gbps IPS throughput and 600 Mbps threat protection throughput, using Fortinet’s purpose-built security processor technology to deliver industry-leading performance and protection for SSL encrypted traffic.
  • Advanced Threat Protection: Integrated with Fortinet’s AI-powered FortiGuard Labs, the FortiGate 40F offers comprehensive cybersecurity, identifying and mitigating both known and unknown threats to maintain robust security across your network.
  • Simplified Management and Deployment: Features a user-friendly management console that provides comprehensive network automation and visibility, coupled with Zero Touch Integration with Fortinet’s Security Fabric for easy deployment.

2. Put replicas across failure zones—and protect the endpoint too

Replicas spread across independent failure zones can reduce exposure to the loss of a single zone. Kubernetes recommends choosing at least three zones and replicating each control-plane component across at least three zones when availability is important. These are deployment recommendations, not measured guarantees of uptime. Kubernetes also supports Pod placement with topology-spread constraints. The Kubernetes multi-zone guidance explains the approach.

Replica placement alone does not make the service reachable. Kubernetes states: “Kubernetes does not provide cross-zone resilience for the API server endpoints.” Endpoint resilience needs its own design; the documentation gives DNS round-robin, SRV records, or a third-party load balancer with health checks as examples. Zone placement also does not automatically make a network plugin zone-aware.

Rank #2
FortiGate-60F Network Security Appliance Plus 1 Year FortiGuard Unified Threat Protection (UTP) and FortiCare Premium (FG-60F-BDL-950-12)
  • HARDWARE PLUS SECURITY SERVICES: FortiGate-60F Firewall Appliance bundled with 1 year of FortiCare Premium and FortiGuard Unified Threat Protection.
  • UNIFIED THREAT PROTECTION (UTP): Secures against advanced online threats with comprehensive web filtering and anti-botnet technologies.
  • OPTIMIZED FOR MEDIUM-SIZED BUSINESSES: Tailored for businesses needing robust security without the infrastructure of larger enterprises.
  • RELIABLE CUSTOMER SUPPORT: FortiCare Premium ensures high-quality support and service continuity.
  • EFFECTIVE PROTECTION: Employs advanced filtering technologies to safeguard against sophisticated threats.
  • Check whether the replicas and the communication paths between them remain reachable under the specific failure your design is meant to withstand.
  • Check each client-facing endpoint independently, including how clients discover a healthy endpoint if one zone or path is unavailable.
  • Review storage, networking, and control-plane dependencies for correlated failures; replicas in different zones do not by themselves remove those risks.

Actual resilience depends on the cloud provider, network plugin, storage design, and endpoint configuration in use. Validate those components against their documentation rather than treating “multi-zone” as a complete availability plan.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

3. Design clients for elections, timeouts, and ambiguous outcomes

A partition or leader change can interrupt requests even when the cluster ultimately recovers. RabbitMQ documents that, during quorum-queue leader changes, publisher confirms can arrive late or be rejected in some scenarios. Publishing applications may need to publish again. Consumer registration and polling require a reachable leader and can block until an election completes or time out; some operations can be buffered and replayed against the new leader. See RabbitMQ’s partition documentation for details.

Rank #3
GL.iNet GL-MT5000 Brume 3 Wired VPN Security Gateway NO Wi-Fi
  • 【Up to 1100 Mbps VPN Speed 】 Hardware-accelerated WireGuard and OpenVPN-DCO deliver up to 1100 Mbps VPN throughput, over 3× faster than Brume 2 for smooth remote access and file transfers.
  • 【Three 2.5G Ports & Multi-WAN】Tri-port 2.5GbE design with flexible WAN LAN configuration supports multi-gigabit wired setups, dual-ISP Multi-WAN and failover to keep home and SOHO networks online.
  • 【Stealth VPN Obfuscation】VPN obfuscation disguises VPN traffic as regular HTTPS, helping you evade blocking, bypass restrictive networks and maintain stable, private connections.
  • 【DPI protection】Deep Packet Inspection with visual dashboards blocks adult/gambling/malicious sites, while SQM and QoS prioritize gaming, calls, and video when bandwidth is tight
  • 【OpenWrt & USB 3.0 Expansion】OpenWrt with 1GB DDR4 and 8GB eMMC lets you install plugins and build VPN, ad-blocking or NAS, while USB 3.0 Type‑C connects high-speed storage or 4G/5G dongles

A timeout does not necessarily tell an application whether an operation took effect. The request might have been committed while its acknowledgement was delayed or lost, or it might not have completed. Retrying blindly can therefore duplicate an effect. Retry only when the operation is safe to repeat, or when the application uses an appropriate mechanism to prevent duplicate effects. RabbitMQ’s documented client outcomes are not a blanket guarantee that arbitrary application retries are safe.

  • Set timeouts that let callers fail or recover instead of waiting indefinitely for a leader or endpoint.
  • Handle delayed acknowledgements and rejected operations as distinct outcomes where the client API allows it.
  • Make retry behavior explicit, and ensure repeated requests cannot cause unintended duplicate effects.
  • Expect temporary errors during leader election and catch-up; avoid turning a brief interruption into uncontrolled retry traffic.

Read semantics matter too. The etcd Raft library documents quorum checks for linearizable reads and notes that lease-based linearizable reads rely on the clocks of machines in the Raft group. The etcd-io Raft documentation describes these considerations. Applications should choose read guarantees and retry behavior based on the system’s actual contract.

Rank #4
Ubiquiti Cloud Gateway Ultra (UCG-Ultra)
  • Runs UniFi Network for full-stack network management
  • Manages 30+ UniFi Network devices and 300+ clients
  • 1 Gbps routing with IDS/IPS
  • Multi-WAN load balancing
  • 0.96" LCM status display
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

4. Plan for reconnection, catch-up, and loss of quorum

Healing a network does not mean every replica is immediately ready to serve. In etcd, the minority recognizes the majority leader and recovers after connectivity returns. RabbitMQ says a reconnected Raft member discovers the elected leader and receives missing log entries. After a long interruption, that catch-up may involve substantial data, so the returning member should be treated as temporarily unavailable. The two products document their own recovery behavior; timing depends on the system and the amount of missing state. RabbitMQ’s partition guide and etcd’s failure guide provide the relevant details.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

For Kubernetes-backed etcd, the official operations guide recommends periodic backups and calls a five-member cluster a production recommendation. Confirm guidance for the Kubernetes and etcd versions you operate before making changes. The same guide warns that if a majority of etcd members has permanently failed, Kubernetes cannot change the currently stored cluster state until the cluster is recovered. The Kubernetes etcd operations guide covers backups and recovery.

Before an incident, make sure the recovery plan answers these operational questions:

  • Where are periodic backups stored, and how will their integrity be checked?
  • Who decides whether a lost quorum is recoverable or requires disaster recovery?
  • How will operators restore service without allowing competing cluster histories?
  • How will returning members catch up, and how will operators know when they are safe to count on?
  • Which deployed-version-specific procedures apply to the cluster?

How the four approaches fit together

Quorum rules decide which partition can make authoritative progress. Failure-zone placement reduces exposure to some infrastructure failures, while endpoint and network design determine whether clients can reach the available side. Client handling makes elections and ambiguous requests survivable at the application boundary. Backups and recovery procedures address cases where quorum does not return. None of these measures makes every partition independently writable without a consistency cost.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Leave a Reply

Your email address will not be published. Required fields are marked *

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Windows Errors? Fix Them Before They SpreadFree repair scan
Crashes, No Sound, or Screen Glitches?Free driver scan

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.