Adding a fourth Consul server does not, by itself, make a cluster more tolerant of failures. In Sergey Shinder’s account of a maintenance incident, four servers were split two and two after a switch rebooted; neither partition could reach the three-vote majority required for writes. The lesson is not that four servers are inherently unsafe, but that quorum math and failure-domain placement matter more than the raw server count.
What happened when the fourth server was added?
Shinder describes a cluster migration in which a switch between two data halls rebooted during planned maintenance. The four Consul servers wound up divided evenly between the halls. Since neither side had enough votes to form a quorum, the cluster could not commit new log entries.
As an Amazon Associate I earn from qualifying purchases.
Shinder reports that writes stopped for 34 minutes. In his account, clients configured to allow stale reads continued working. Those outage and workload details are the author’s report, not an independently measured reliability study.
Why can four voters be less resilient than expected?
Consul server nodes form the Raft peer set; client agents do not vote in it. HashiCorp defines quorum as a majority of that peer set, at least (N/2)+1 members. A log entry must be stored durably on a quorum to be committed. Without quorum, new log entries cannot be committed, so writes requiring them stop.
#1 Best Overall
- HP ProLiant DL360 G7 Business Server, the perfect enterprise server or small business server!
- Processors: Dual (2) Xeon X5675 6-Core 3.06 GHz 12MB CPUs Max Turbo 3.46 GHz
- Memory: 72GB (4 x 16GB) DDR3 PC3-10600R Memory; Storage: 3.6TB (4 x 900GB) 10K 12Gb/s SAS 2.5" HDDs
- Power: Redundant Power Supplies; RAID: HP Smart Array P410i-a 12Gb/s with 4×GigaBit NIC
- Hard drives and memory upgrades included separately NOT installed, installation required.
| Consul servers | Votes needed for quorum | Server failures tolerated while retaining quorum |
|---|---|---|
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
The thresholds and failure-tolerance figures are from HashiCorp’s current Consul consensus documentation. Moving from three servers to four raises the vote threshold from two to three but does not increase the number of individual server failures the cluster can withstand: both sizes tolerate one. If a shared failure splits four voters two and two, neither side has the required three votes.
Why placement matters as much as the count
Quorum arithmetic counts voters; it does not make those voters independent. A rack, hall, power feed or network link that takes out several servers together can remove a majority even when the nominal server count looks ample. Conversely, voters spread across independent failure domains are less exposed to one local failure taking out the votes needed to operate.
Rank #2
- Dell PowerEdge R730xd 24B SFF 2U Server
- 2x Intel Xeon E5-2690 v4 2.6Ghz 14-Core (28-cores Total)
- 128GB DDR4 RAM – 4x 1.2TB 10K SAS 2.5” 12Gb/s
- Dell H730P mini 2GB 12Gb/s RAID
- 2x 750W PSU - 2x 10Gb SFP+ 2x 1Gb (RJ45) NIC
That is why “odd is always safe” is too crude. With three voters, one failure is tolerable; with four, one failure is also tolerable; with five, two failures are tolerable. But those figures assume the remaining servers can communicate and that failures are counted individually. Placement and shared infrastructure determine whether a single event can affect several voters at once.
What the team changed after the incident
Shinder says the team completed its migration using paired add/remove operations, documented which failure each intermediate membership state could survive, and now runs five servers across three halls with an automated placement check. These are practices from that team’s account, not a universal Consul requirement.
Rank #3
- High-Density, High-Speed Storage Platform: Hosts eight 12Gbps hot-swap drive bays in a compact 2U form, delivering exceptional storage density and bandwidth for data-intensive tasks like video editing, virtualization, or as a primary storage server.
- Flagship E-ATX Compatibility for Demanding Workloads: Supports the largest E-ATX server motherboards, enabling builds with maximum CPU core count, vast RAM capacity, and extensive PCIe expansion for the most demanding computational workloads.
- Enterprise-Grade, Serviceable Cooling System: The 3 Hot-Swap 80x38mm fans delivers high-static pressure to cool components effectively. The hot-swap capability guarantees that cooling integrity is never compromised, even during fan maintenance.
- Accelerate External Workflows with 10Gbps Type-C: The integrated front Type-C port provides ultra-fast connectivity for modern peripherals, significantly cutting down time spent on large file transfers.
- Support Full length CRPS PSU: The max depth of PSU is 280mm
The useful operational principle is to evaluate each intermediate state, not only the intended final cluster. During a membership change, ask which servers are voting, how many votes are needed, and whether a plausible failure could leave fewer reachable voters than quorum. A migration plan should make those answers explicit before each change.
What quorum loss means for reads and other workloads
Quorum loss prevents new log entries from being committed; it does not establish that every read mode behaves identically. In Shinder’s incident, clients that allowed stale reads continued to work, according to his account. Read behavior depends on the client request and configuration, so it should be tested separately from write availability.
Rank #4
- Spacious Chassis: This massive 4U server case has 8 internal 3.5" HDD bays plus room for 3 additional 5.25" devices
- Expandable & ATX/CEB Compatible: 7 PCI expansion slots and ATX and CEB motherboard compatibility give you growth options for all of your needs
- Quiet Cooling: 4 pre-installed cooling fans provide excellent airflow and heat protection at reduced noise. 2 front 120mm PWM fans and 2 rear 80mm fans ensure your drives and chassis avoid overheating
- Desired Features: Front panel LED indicators for power, HDD, and LAN status monitoring allow quick, easy visual assessment. Additional utility with 2 x USB 3.0 port and built-in front panel lock provides extra security for your server case
- Rackmount Design: Standard 4U rackmount form factor allows easy installation in server racks and data center environments with included mounting hardware for professional deployment
For a real failure review, distinguish writes from stale reads, health updates, and operations that depend on a leader or session. The incident account supports a specific report about writes and stale reads, not a general guarantee that all reads or other Consul functions remain available whenever quorum is lost.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →How to assess a Consul server layout
- Calculate quorum: determine the current server peer-set size and the majority it requires.
- Map failure domains: record which voters share halls, racks, power, switches and network links.
- Check correlated failures: consider whether one credible event could make a majority unreachable together.
- Review every migration state: identify the voters and failure tolerance after each membership change, not just after completion.
- Test workload behavior: establish which writes, stale reads, health updates, leader-dependent operations and sessions continue or stop during quorum loss.
HashiCorp’s current Consul documentation recommends three or five servers for production deployments. That recommendation aligns with the fault-tolerance arithmetic above, but a server count cannot compensate for voters sharing a failure domain.
Quick Recap
Best Value
- [CPU] Intel Core Ultra 7 265 Processor (20 Cores, 20 Threads, 3.9 GHz Base Clock Speed up to 5.5 GHz Max Boost Clock Speed) for Elite Gaming and Content Creation | [STORAGE] 2TB PCIe NVMe M.2 SSD - Experience Hyper-Fast Bootup and Data Transfer thats up to 30x Faster Performance than a Traditional Hard Drive.
- [GPU] Integrated Intel UHD Graphics: Get All the Power You Need for Fast, Smooth, Power-Efficient Performance | [RAM] 24GB DDR5 RAM 5600 Gaming Memory for Seamless Multitasking from Multiple Web Pages to Playing Games Online Simultaneously | [OS] Windows 11 Pro x64
- 2x 3.5" Drive Bays | 4x Expansion Slots | mATX Motherboard | ATX PSU
- [BUY WITH CONFIDENCE] Empowered PCs are Assembled in the USA, Rigorously Stress-Tested Before Shipping, and Supported with Lifetime Technical and Diagnostic Support and 3-Year Limited Hardware Warranty.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
