October DealsAmazon USOctober deal check: compare before you payAmazon US: current deals, useful picks and tech finds.Check DealsClean PCRecommendedOne scan can reveal what keeps slowing WindowsLook for cleanup and repair opportunities.Run ScanOctober DealsAmazon USDeal season is back - check today's better picksAmazon US: current deals, useful picks and tech finds.See Picks×
Skip to content
World desk5 min

Failover Testing: How to Avoid Stopping the Wrong Node

A safe failover test verifies cluster identity, controls promotion, fences the old primary, and checks that replicas and applications recover.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

A safe failover test begins by proving which node is the current leader and which one is meant to take over. Then verify that the old primary will be fenced from writes, observe promotion through supported cluster and health interfaces, and confirm the former primary rejoins safely. The headline does not identify a particular incident or platform; Patroni with PostgreSQL offers a documented example of how to make those checks concrete.

Why a failover test can stop the wrong node

“The wrong node” can mean different failures: an operator targets the wrong machine, an independent service manager restarts an old primary, or a standby is promoted while the original primary remains writable. The common risk is losing a trustworthy view of cluster identity and control. In a PostgreSQL cluster managed by Patroni, the leader lock coordinates leadership, and Patroni attempts to stop PostgreSQL if it cannot renew that lock. That safety model depends on Patroni retaining control of the database process; an outside restart path can undermine it. The Patroni FAQ says, “Only Patroni should be able to start, stop and promote Postgres instances in the cluster.” Patroni FAQ.

As an Amazon Associate I earn from qualifying purchases.

Identify the failure you are testing

Do not treat every exercise as the same generic failover. A planned transfer, an unplanned primary loss, loss of access to the distributed configuration store (DCS), a network partition, and a multi-site disaster-recovery promotion have different safety conditions. Decide which case applies and define what success means before you act.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Planned switchover: Transfer leadership deliberately while the cluster is available.
  • Primary loss: Check whether the surviving cluster can safely promote a candidate after the primary is unavailable.
  • DCS or network failure: Test the consequences of losing coordination or communication, not just the database host.
  • Multi-site recovery: Confirm the source site is isolated before promoting a separate standby cluster.

Set an acceptable recovery point objective (RPO): how much recent data, if any, the organization can tolerate losing. Also decide how the exercise will detect missing or divergent writes. In Patroni, asynchronous replication is the default, and a promoted replica may not contain every recent write; the README describes a configurable maximum-lag threshold. Patroni README.

#1 Best Overall
Sale
NETGEAR 5-Port Gigabit Ethernet Unmanaged Network Switch (GS305)
  • GIGABIT ETHERNET PORTS: Features 5 x 1.0Gbps Ethernet ports for high-speed connectivity. Auto-negotiating ports detect the optimal speed for connected devices and work with existing Cat5e or Cat6 Ethernet cables.
  • PLUG-AND-PLAY UNMANAGED NETWORK SWITCH: Simple plug-and-play setup with no software to install or configuration required.
  • FLEXIBLE MOUNTING OPTIONS: Compact metal design supports desktop or wall-mount placement for versatile installation.
  • SILENT & ENERGY-EFFICIENT OPERATION: Fanless design ensures silent performance, while IEEE 802.3az Energy Efficient Ethernet reduces power consumption without compromising high-speed network performance.
  • REGIONAL COMPATIBILITY: Made for use in U.S. & CA only

Verify identity and candidate health before the test

Use the cluster’s supported status interface to establish a baseline. Patroni’s REST API exposes member roles and states; record the names, roles, states, and leader identity. Check that the proposed candidate is the intended machine and is healthy enough to promote. A manual failover request names a candidate, and the API warns that manual failover can cause data loss. It can be used even when a leader exists, so do not mistake the endpoint for a planned switchover procedure. Patroni REST API.

  1. Capture cluster status and write down the current leader and each member’s role and state.
  2. Confirm the candidate’s identity against the machine or instance you intend to promote, not just an assumed ordering or hostname pattern.
  3. Check candidate health and replication state, then compare observed lag with the RPO you set.
  4. Choose the operation that matches the scenario; do not use an emergency/manual failover request as a substitute for a planned transfer.

Make sure one control plane owns PostgreSQL

Confirm that Patroni alone controls PostgreSQL start, stop, and promotion. Disable independent service-manager restart behavior for the database process in a Patroni deployment. Otherwise, a service manager may bring an old primary back after Patroni has relinquished leadership, creating two writable primaries.

Rank #2
Sale
TP-Link TL-SG105, 5 Port Gigabit Unmanaged Ethernet Switch, Network Hub, Ethernet Splitter, Plug & Play, Fanless Metal Design, Shielded Ports, Traffic Optimization
  • 𝗢𝗻𝗲 𝗦𝘄𝗶𝘁𝗰𝗵 𝗠𝗮𝗱𝗲 𝘁𝗼 𝗘𝘅𝗽𝗮𝗻𝗱 𝗡𝗲𝘁𝘄𝗼𝗿𝗸: 5× 10/100/1000Mbps RJ45 Ports supporting Auto Negotiation and Auto MDI/MDIX.
  • 𝗚𝗶𝗴𝗮𝗯𝗶𝘁 𝘁𝗵𝗮𝘁 𝗦𝗮𝘃𝗲𝘀 𝗘𝗻𝗲𝗿𝗴𝘆: Latest innovative energy-efficient technology greatly expands your network capacity with much less power consumption and helps save money.
  • 𝗥𝗲𝗹𝗶𝗮𝗯𝗹𝗲 𝗮𝗻𝗱 𝗤𝘂𝗶𝗲𝘁: IEEE 802.3X flow control provides reliable data transfer and Fanless design ensures quiet operation.
  • 𝗣𝗹𝘂𝗴 𝗮𝗻𝗱 𝗣𝗹𝗮𝘆: Easy setup with no software installation or configuration needed.
  • 𝗔𝗱𝘃𝗮𝗻𝗰𝗲𝗱 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗙𝗲𝗮𝘁𝘂𝗿𝗲𝘀: Prioritize your traffic and guarantee high quality of video or voice data transmission with Port-based 802.1p/DSCP QoS and IGMP Snooping.

Before a live exercise, verify the fencing action and what happens if it fails. Patroni’s pre_promote hook runs after the candidate acquires the leader lock and before it promotes PostgreSQL. If the hook exits nonzero, Patroni blocks promotion and removes the leader key. This is a useful gate, but it is not proof that every old-primary path is fenced; validate the actual script and its effect in the environment. Patroni watchdog and fencing documentation.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Use layered protection against two writable primaries

Fencing is the action that prevents a node believed to be unavailable or demoted from continuing to serve writes. Patroni’s leader lock and attempt to stop PostgreSQL when lock renewal fails are important protections, but a process that Patroni cannot stop—or that another manager restarts—can still defeat the intended boundary.

Rank #3
Sale
NETGEAR 8-Port Gigabit Ethernet Unmanaged Network Switch (GS308)
  • GIGABIT ETHERNET PORTS: Features 8 x 1.0Gbps Ethernet ports for high-speed connectivity. Auto-negotiating ports detect the optimal speed for connected devices and work with existing Cat5e or Cat6 Ethernet cables.
  • PLUG-AND-PLAY UNMANAGED NETWORK SWITCH: Simple plug-and-play setup with no software to install or configuration required.
  • FLEXIBLE MOUNTING OPTIONS: Compact metal design supports desktop or wall-mount placement for versatile installation.
  • SILENT & ENERGY-EFFICIENT OPERATION: Fanless design ensures silent performance, while IEEE 802.3az Energy Efficient Ethernet reduces power consumption without compromising high-speed network performance.
  • REGIONAL COMPATIBILITY: Made for use in U.S. & CA only

A watchdog provides another layer if Patroni crashes, is killed, runs too slowly, or cannot execute because a virtual machine is paused or heavily loaded. Patroni coordinates watchdog expiry with the DCS leader-lock time-to-live (TTL); timing margins matter. The documentation gives configuration examples, including a 30-second default TTL and five-second default safety margin, but these are defaults, not universal recommendations. Check the deployed values for loop_wait, retry_timeout, and ttl rather than copying example timings. Patroni watchdog documentation.

Run the exercise while watching the right signals

Observe the cluster using the same supported status and health interfaces used by operations and monitoring. Patroni’s REST API includes endpoints that help distinguish primary status from replica readiness. During the test, record whether the leader lock is valid, which node is writable, whether replicas catch up, and when application connections recover. Patroni REST API.

Rank #4
Sale
TP-Link LS1005G, Litewave 5 Port Gigabit Ethernet Unmanaged Switch
  • 【One Switch Made to Expand Network】Features 5 RJ45 ports with 10/100/1000Mbps speeds, supporting Auto-Negotiation and Auto MDI/MDIX for hassle-free setup. Ideal for expanding your network, with 1 uplink (input) port and 4 output ports to split your Ethernet connection to multiple devices.
  • 【Gigabit that Saves Energy】Latest innovative energy-efficient technology greatly expands your network capacity with much less power consumption and helps save money
  • 【Reliable and Quiet】IEEE 802.3X flow control provides reliable data transfer and Fanless design ensures quiet operation
  • 【Plug and Play】Easy setup with no software installation or configuration needed
  • 【Ethernet Splitter】Connect to your router or modem for additional wired connections (laptop, gaming console, printer, etc)
  • Check that the intended candidate, not another member, becomes leader.
  • Verify that only the new leader accepts writes.
  • Compare recovered data with the RPO and inspect for missing or divergent writes.
  • Measure client recovery using the application’s actual connection path and health checks.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

Apply stricter rules to a two-site standby

A standby site in a two-site asynchronous design cannot infer the state of the source site merely from a communication failure. In the documented Patroni arrangement, automatic promotion is not possible for that reason. Confirm that the old source is down or otherwise fenced—STONITH—before promoting the standby. Patroni’s multi-datacenter guide warns that promoting while the source cluster is still running creates split brain. After source recovery, reconcile the topology before returning it to service. Patroni multi-datacenter guide.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

Close the test by proving safe rejoin

Promotion is not the end of the exercise. Verify that the old node cannot accept writes, that replicas follow the new leader, and that the former primary rejoins safely. Patroni’s README notes that redundancy is temporarily reduced until the failed member returns, so include restoration of redundancy in the test’s completion criteria. Patroni README.

Best Value
Sale
TP-Link TL-SG108S-M2, 8-Port Multi-Gigabit 2.5G Unmanaged Ethernet Switch
  • 𝗘𝗶𝗴𝗵𝘁 𝟮.𝟱 𝗚𝗯𝗽𝘀 𝗣𝗼𝗿𝘁𝘀 𝗳𝗼𝗿 𝗦𝘂𝗽𝗲𝗿-𝗙𝗮𝘀𝘁 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻𝘀: 8× 2.5-Gigabit ports unlock the highest performance of your Multi-Gig bandwidth and devices, and provide up to 40 Gbps of switching capacity.
  • 𝗔𝘂𝘁𝗼-𝗡𝗲𝗴𝗼𝘁𝗶𝗮𝘁𝗶𝗼𝗻: Auto-negotiation intelligently senses the link speeds and adjusts between 3-speeds (100Mb/1G/2.5G) for compatibility and optimal performance for all your devices, including 2.5G WiFi 6 AP, 2.5G NAS, 2.5G PCIe Adapter, 2.5G Server, gaming computer, 4K video, and more.
  • 𝗜𝗱𝗲𝗮𝗹 𝗳𝗼𝗿 𝗩𝗮𝗿𝗶𝗼𝘂𝘀 𝗦𝗰𝗲𝗻𝗮𝗿𝗶𝗼𝘀: Built for LAN parties, home entertainment, small and home offices, and instant transfer for workstations.
  • 𝗛𝗮𝘀𝘀𝗹𝗲-𝗙𝗿𝗲𝗲 𝗖𝗮𝗯𝗹𝗶𝗻𝗴: Instantly upgrade to 2.5 Gbps without the need to upgrade to Cat6 wiring, reducing wiring costs and hassle. *
  • 𝗦𝗶𝗹𝗲𝗻𝘁 𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗼𝗻: Industry-leading fanless design ensures silent operation, ideal for any home or business.

Keep instructions matched to the deployed release

These examples describe Patroni and PostgreSQL, not an identified incident behind the headline. The cited Patroni documentation is on the project’s mutable documentation and master pages; check the documentation for the release actually deployed before applying endpoint behavior, hook semantics, or configuration examples.

Quick Recap

SaleBestseller No. 1
NETGEAR 5-Port Gigabit Ethernet Unmanaged Network Switch (GS305)
NETGEAR 5-Port Gigabit Ethernet Unmanaged Network Switch (GS305)
REGIONAL COMPATIBILITY: Made for use in U.S. & CA only
$13.49
SaleBestseller No. 3
NETGEAR 8-Port Gigabit Ethernet Unmanaged Network Switch (GS308)
NETGEAR 8-Port Gigabit Ethernet Unmanaged Network Switch (GS308)
REGIONAL COMPATIBILITY: Made for use in U.S. & CA only
$18.99
SaleBestseller No. 4
TP-Link LS1005G, Litewave 5 Port Gigabit Ethernet Unmanaged Switch
TP-Link LS1005G, Litewave 5 Port Gigabit Ethernet Unmanaged Switch
【Plug and Play】Easy setup with no software installation or configuration needed
$9.99

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

Leave a Reply

Your email address will not be published. Required fields are marked *

Free tools Windows power users keep installed

One-click scans. No signup required.

Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

More from the Wire

  1. World desk4 min
    How to Spot an AI Voice Scam Before Sending MoneyDon’t rely on how a caller sounds. Pause, call back through a known number, and verify the emergency with another trusted person before sending money.
  2. Mountain View desk4 min
    Google’s SynthID Detector: How to Check AI-Generated Images, Video and AudioGoogle’s SynthID Detector looks for an embedded watermark in supported images, video and audio. Here is what its results do—and do not—show.
  3. Redmond desk20 min
    How to create a link to File or Folder in Windows 11Windows 11 gives you several ways to point to a file or folder without moving or duplicating it. You can create a desktop shortcut,…
Recommended PC Tool
Recommended PC Tool
Crashes, No Sound, or Screen Glitches?Free driver scan
PC Slower Than It Used to Be?Free scan - under a minute

Two free Windows tools

One Free Minute Could Fix That PC

Before you go - each of these free tools takes about a minute and tackles what quietly slows a Windows PC down.

Special offer. View Outbyte info, uninstall instructions, EULA, and Privacy Policy.