My Homelab Had a Dead Container for 64 Days: Why Host and Docker Monitoring Missed It

A container can be down for two months while every dashboard you own stays green. The cause is usually scope, not a broken tool. A host graph answers “is the machine alive?” A Docker daemon scrape answers “is Docker alive?” Neither answers “is the service I care about doing its job?”

The 64-day figure and the “popular monitoring tools” in the title are the author’s own account of a homelab incident. No logs are published, and no specific product is named here. This article doesn’t claim any named tool would or wouldn’t have alerted. It explains the gap that lets a dead container go unnoticed, and how to close it.

#	Preview	Product	Price
1		Omada OC220, Hardware Controller		Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

Why a healthy server can hide a dead app

Docker’s own documentation, “Collect Docker metrics with Prometheus,” puts the limit plainly: “Currently, you can only monitor Docker itself. You can’t currently monitor your application using the Docker target.” The daemon’s metrics endpoint tells you about the engine. It is not a probe of the services running on it. Docker also warns that the metric names are under active development and may change, so check any dashboard or alert you build against your Docker version.

The same trap applies one level up. Prometheus’s /-/healthy and /-/ready endpoints check Prometheus itself. A green monitoring server says nothing about each container it is supposed to watch.

#1 Best Overall

Omada OC220, Hardware Controller

• Centralized hardware controller for managing Omada network devices
• Supports up to 200 Omada access points, switches, and gateways
• Cloud access and local management for flexible network administration
• Real-time monitoring and alerts for network performance and security
• Easy setup with intuitive web interface and mobile app support

Buy on Amazon

The five layers that get confused

A setup can be excellent at the first row and have nothing at the others. That is how a stopped container goes unnoticed: a missing workload often doesn’t look like a failure. It looks like a line that is no longer on the dashboard.

Why does Docker say my container is stopped?

Start with the state Docker reports. By default, docker ps lists only running containers, so a stopped one is invisible there. Add -a:

docker ps -a
docker ps -a --filter status=exited
docker inspect --format '{{.State.Status}} exit={{.State.ExitCode}} finished={{.State.FinishedAt}}' NAME

Docker distinguishes several states, including exited and dead. The FinishedAt timestamp tells you how long it has been down. That is the number to compare with the reported outage.

• Exited: the container stopped. Check docker logs NAME for the cause, then docker start NAME if it should be running.
• Dead: Docker documents this as a defunct state that cannot be restarted. Don’t try docker start. Remove the container and recreate it from its image or compose file.

How do I know if a container is unhealthy?

A Docker health check runs a command inside the container at an interval and reports a health status. You configure the command, interval, timeout, retries and start period. A compose example:

services:
  app:
    image: example/app:1.2
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s

Read the result with docker inspect --format '{{.State.Health.Status}}' NAME. It reports starting, healthy or unhealthy.

The check is only as good as its command. A process that exists, or a port that accepts connections, doesn’t prove the application is doing useful work. Point the check at something that exercises the real function, such as a response from a route that touches the database. Also note that a health check does not help if the container isn’t running at all. A stopped container runs no checks and reports no failures.

How do I monitor whether a container is running?

You need a signal that fires on absence, not just on bad values. There are three common approaches, and they differ in what they catch.

Docker events

Docker emits lifecycle events including start, stop, die and health_status. You can watch them live with docker events --filter event=die. Treat this as a trigger, not a record. Docker’s documentation says only the last 256 events are returned, so the stream can’t reconstruct a long outage on its own. If you want history, something else has to store the events.

Metrics scraping

Prometheus can discover Docker targets through service discovery with configurable refresh behavior. Discovery builds the target list, but it doesn’t alert. You still need a rule that fires when an expected target or series is missing. Without one, a container that vanishes simply stops appearing in graphs.

External probes

The strongest check is a request from outside the container that behaves like a user and expects a meaningful reply. Docker’s app-monitoring guide describes Prometheus scraping application metrics endpoints, with Grafana for visualization. A probe that checks the service’s actual address, with an alert on failure, covers the gap the daemon target leaves open.

Will Docker restart a container if it stops?

Only if a restart policy says so, and the policy has limits. Docker documents that a container you stop manually isn’t restarted by the policy until the daemon restarts or you start it yourself. Policies also apply only after the container has started successfully. Two consequences follow:

• A restart policy is recovery, not detection. A container that crashes and is silently restarted ten times a day looks fine from outside.
• A deliberately stopped or permanently failed container stays down, quietly.

Use restart policies for transient failures, and keep a separate alert so you learn when they are being used.

Comparing monitoring setups honestly

Whatever you run, evaluate it on these axes rather than on a feature list. Verify each one against your own configuration and current documentation, because none of them can be assumed from a product’s name.

A test that exposes the blind spot in ten minutes

1. Run docker ps -a and list every container that should be running. Write the list down; that is your expected inventory.
2. Stop one non-critical container with docker stop NAME.
3. Wait through your normal alert interval, then see whether any message arrives and whether any dashboard marks the service as down.
4. If nothing fires, add an absence-based alert or an external probe. Repeat the test.
5. Start the container again and confirm the alert resolves.

Run the same test periodically. Monitoring that has never been seen to fail isn’t verified.

Quick Recap

Bestseller No. 1

Omada OC220, Hardware Controller

Centralized hardware controller for managing Omada network devices; Supports up to 200 Omada access points, switches, and gateways

What to put in place

• An explicit inventory of containers that should be running, checked against docker ps -a output so stopped ones are not invisible.
• A health check on each service whose command exercises real behavior.
• An external probe of each user-facing service from the network path users use.
• An alert on missing targets and on unhealthy or exited states, delivered somewhere you will actually see.
• Restart policies for transient faults, with the restarts themselves visible.
• Durable storage for events or state changes, since Docker’s own event output is capped.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.

DockerHealthchecksHomelabMonitoringPrometheusself-hosting