Check cluster health after an upgrade in layers: confirm each node has rejoined, verify the leader and voting peers, check version and replicated-state convergence, then validate services or workloads. Repeat those checks after each node in a rolling upgrade—not just once at the end. A running process or successful health endpoint alone does not establish that the cluster is fully caught up or that applications are healthy.
What to verify after each node restarts
HashiCorp’s Nomad upgrade guidance recommends making changes incrementally and verifying cluster health at each step. The same layered approach is useful for the other products, while their exact procedures and expected states depend on the product, version pair, topology, and—in Vault’s case—storage backend.
As an Amazon Associate I earn from qualifying purchases.
- Membership: Is the restarted agent visible to its peers?
- Consensus and roles: Is there an expected leader, and are the expected voters present?
- Convergence: Have versions and replicated state caught up?
- Service or workload behavior: Are the workloads, service checks, or application-facing services healthy?
Membership and consensus are separate signals: a node can appear in a membership list while the peer set, leader, or replicated state still needs attention.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Use a rolling verification sequence
- Record the baseline. Before maintenance, note the expected node count, roles, versions, voters, and relevant service or workload status.
- Upgrade one node at a time when the product’s documented procedure calls for a rolling sequence. Follow the source-to-target version guidance for each product; this health checklist does not replace compatibility planning.
- After each restart, check membership and consensus. Confirm the node has rejoined and that the leader and voter set match the intended topology.
- Check convergence. Compare the product’s available version and replication indicators, rather than treating process startup as proof that state is synchronized.
- Validate the workload or service layer. Check the deployments, allocations, service checks, or application-facing services that matter to users.
- Proceed only when the expected state is restored. If a check fails, inspect logs and the upgrade guidance for the specific versions and configuration before continuing.
- Capture the final state. Record membership, versions, and service or workload health after the upgrade is complete.
Check Nomad servers, clients, and workloads
Verify servers and replicated state
During a server upgrade, inspect the new server’s logs and run nomad agent-info. Compare the last_log_index reported for that server with the other servers; a mismatch can indicate that replicated changes have not yet caught up. Use nomad server members to inspect server membership.
The Nomad upgrade guidance calls for checking cluster health as servers are added or upgraded and again after old servers are removed. Upgrade clients after the servers have succeeded, then use nomad node status to confirm that clients are ready.
Check deployment and allocation health
For a deployment, run nomad deployment status <deployment-id>. Review the desired and applied changes, deployment state, and healthy or unhealthy allocation counts. A deployment that is still running, unhealthy, awaiting canary promotion, or recovering is not equivalent to a completed healthy rollout.
For a specific allocation, run nomad alloc checks <allocation-id> to see the latest service health-check status. The required ACL capabilities depend on the query and namespace; read-job or list-jobs permissions may be needed.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsCheck Consul membership, Raft peers, and services
Confirm agents and the Raft peer set
Run consul members -detailed to inspect agent membership, then run consul operator raft list-peers to inspect the Raft peer set. Check that there is one leader, the expected voters are present, and peer states fit the intended topology. Depending on Consul version, the output can include Raft protocol and commit-index information.
Wait for a restarted server to rejoin and synchronize before upgrading another server. The status API offers GET /v1/status/leader and GET /v1/status/peers; HashiCorp describes the peer list as strongly consistent and useful for determining whether a server has joined.
Check the services applications depend on
Use the Consul UI or query service health through an endpoint such as /v1/health/service/<service>?passing. This checks service instances, not just whether Consul agents are visible. Consul’s troubleshooting guidance notes that unhealthy services are not returned through standard DNS discovery and some HTTP API calls, so query the specific services the application relies on rather than treating their absence from discovery as proof that no unhealthy instances exist.
Interpret Vault health in the node’s role
Read the health response by status code
Call GET /v1/sys/health, or use vault status for local CLI status. The documented default API codes describe different states:
Free tools Windows power users keep installed
One-click scans. No signup required.
| HTTP status | Documented meaning | How to interpret it after an upgrade |
|---|---|---|
200 |
Initialized, unsealed, active | Expected for the active node. |
429 |
Unsealed standby | Can be normal for a standby; it is not, by itself, an upgrade failure. |
472 |
Disaster-recovery secondary | Interpret in light of the node’s configured role. |
473 |
Performance standby | Interpret in light of the node’s configured role. |
474 |
Standby cannot connect to active | Investigate the standby’s connection to the active node. |
501 |
Not initialized | Investigate before treating the node as ready. |
503 |
Sealed | The node is not in the unsealed state. |
530 |
Removed | The node is reported as removed. |
HashiCorp notes that in rare cluster-instability cases a node may return 429 even when it belongs to a DR-secondary or performance-standby group. Consider the node’s role and cluster context instead of interpreting the code in isolation.
Check integrated Raft storage
If Vault uses integrated Raft storage, run vault operator raft list-peers to verify the expected nodes, leader, and voter status. Where Autopilot is available, run vault operator raft autopilot state and inspect Healthy, Status, Last Index, Version, Node Type, and Last Contact. Compare follower indexes with the leader, and confirm that each server reports the expected post-upgrade version.
vault operator members provides active-node and peer visibility, including version and upgrade-version fields. These Raft commands are specific to integrated storage. For another Vault storage backend, use the health and status information appropriate to that backend; there is no single backend-independent consensus command established here.
How the health signals differ by product
| Product | Membership and consensus | Catch-up or version signal | Workload or service signal |
|---|---|---|---|
| Nomad | nomad server members; inspect server membership. |
nomad agent-info; compare last_log_index across servers. |
nomad node status, nomad deployment status, and nomad alloc checks. |
| Consul | consul members -detailed and consul operator raft list-peers. |
Raft output may include commit-index information, depending on version; wait for a restarted server to synchronize. | Query service health, for example /v1/health/service/<service>?passing. |
| Vault | /v1/sys/health; for integrated Raft, vault operator raft list-peers. |
For integrated Raft, Autopilot can show per-server Last Index, Last Contact, and Version; vault operator members also shows version fields. |
The health endpoint reports Vault node state and role; pair it with the application checks relevant to the deployment. |
The signals complement one another. Nomad’s checks extend from server and client readiness to deployment and allocation health; Consul separates agent membership, Raft peers, and service health; Vault’s status must be interpreted by node role and storage configuration.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
