DockerSelf-Hosting

Docker Health Checks Explained: The Probes I Actually Trust in Compose

Learn how Docker health checks really work, which probes to trust, and how to use service_healthy in Compose without fooling yourself.

AU

Author

Marcus Chen

FTC disclosure: This article contains affiliate links. If you buy through them, HomelabAddiction may earn a commission at no extra cost to you.

After running Docker on little fanless boxes, proper servers, and one deeply unwise mini PC tucked behind a UPS that was louder than the machine itself, I learned a simple lesson: a container that says Up 3 hours can still be completely useless.

I have had containers stay alive while the app behind them was dead, jammed, disconnected from its database, or serving nothing but disappointment. Docker health checks fix that blind spot - if you configure them like you mean it.

Key Takeaways

  • A running container is not the same thing as a healthy service.
  • Docker HEALTHCHECK marks state as starting, healthy, or unhealthy, but it does not restart a failed container by itself.
  • The best health checks test the thing that matters - not just whether a process exists.
  • docker compose health checks become much more useful when paired with depends_on: condition: service_healthy.
  • Lazy probes are worse than no probes. If your check always passes, you are just cosplaying observability.
  • docker inspect gives you the health log you need when a container flips to unhealthy.

Why health checks matter more in a homelab than people admit

In production, everybody talks about resilience. In homelabs, people often settle for vibes.

That is usually fine right up until your reverse proxy boots faster than your database, your app starts before migrations finish, or your monitoring stack tells you everything is green because the container process still exists. That is how you end up debugging a service that is technically running and practically broken.

A good health check gives Docker one honest answer to one useful question: is this service actually doing its job?

That matters even more when you run multiple self-hosted services on one box. If you already care about Docker Compose best practices, Docker security hardening, and clean logging, health checks are the next obvious step. They make the rest of your stack less dumb.

What Docker health checks actually do

Docker runs a command inside the container on a schedule.

If the command exits with 0, the container is healthy. If it fails enough times in a row, Docker marks the container unhealthy.

The health states are simple:

  • starting - Docker is still in the early startup grace window
  • healthy - the latest checks passed
  • unhealthy - the check failed retries times in a row

That state appears in docker ps, docker inspect, and Compose tooling.

What Docker does not do is just as important.

A health check does not automatically restart the container. That part still belongs to restart policies, Compose behavior, or external automation. This is the mistake I see constantly, and yes, I made it too. I assumed unhealthy meant Docker would heroically save me from myself. It did not.

Official docs if you want the exact syntax:

The mistake I made: checking the wrong thing

My early health checks were garbage.

I used commands that proved almost nothing:

HEALTHCHECK CMD pgrep nginx || exit 1

That only tells me the process exists. A process can exist while the app is wedged, misconfigured, or unable to serve requests.

Then I swung too far the other direction and wrote checks that were too clever. They tested every dependency, every route, and half my application logic. Those checks flapped constantly and turned normal startup into chaos.

The sweet spot is boring and specific.

A good health check should answer one practical question:

  • for a web app: can it answer a lightweight local request?
  • for Postgres: is the database ready to accept connections?
  • for Redis: does PING work?
  • for a worker: can the process respond to a minimal built-in status command?

If you can keep the probe fast, local, and meaningful, you are in good shape.

The timing knobs that matter

Docker gives you four controls that determine how health checks behave:

  • interval - how often to run the check
  • timeout - how long the check can take before it counts as failed
  • retries - how many consecutive failures before Docker marks it unhealthy
  • start_period - grace time after startup before failures count fully

A sane default for many self-hosted web apps looks like this:

HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
  CMD curl -fsS http://localhost:3000/health || exit 1

That setup checks every 30 seconds, allows five seconds per check, waits through a 20-second startup period, and only declares failure after three misses.

Why does start_period matter? Because some services are slow to initialize, especially when they run migrations, warm caches, or wait for storage. Without a grace period, Docker starts screaming before your app has put its shoes on.

Health checks in a Dockerfile vs Compose

You can define a health check in the Dockerfile:

FROM node:22-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN apk add --no-cache curl

HEALTHCHECK --interval=30s --timeout=5s --start-period=20s --retries=3 \
  CMD curl -fsS http://localhost:3000/health || exit 1

CMD ["node", "server.js"]

That bakes the check into the image, which is great if the image always runs the same service with the same expectation.

You can also define it in Compose, which is what I do more often in a homelab because it keeps deployment behavior close to the stack config:

services:
  app:
    image: ghcr.io/example/myapp:latest
    ports:
      - "3000:3000"
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:3000/health || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s

If the image already ships with a good health check, I leave it alone. If the image ships with a lazy one, I override it without guilt.

The probes I actually trust

Here is the short version.

I trust probes that test the service boundary users or dependent services care about. I do not trust probes that only prove a PID exists.

1. HTTP app probe

Best for web apps, dashboards, APIs, reverse proxies, and anything with a clean local endpoint.

healthcheck:
  test: ["CMD-SHELL", "curl -fsS http://localhost:8080/healthz || exit 1"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 20s

Why it works:

  • tests the actual listener
  • fails on HTTP 4xx and 5xx with -f
  • stays fast and local

What can go wrong:

  • if /healthz does expensive dependency checks, it may flap
  • if the image lacks curl, the probe fails forever

Alpine-based images often work better with wget if you want to avoid adding curl:

healthcheck:
  test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:8080/healthz || exit 1"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 20s

2. Postgres probe

For Postgres, use the tool built for the job.

services:
  postgres:
    image: postgres:17
    environment:
      POSTGRES_DB: appdb
      POSTGRES_USER: appuser
      POSTGRES_PASSWORD: supersecret
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U appuser -d appdb -h localhost || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 20s

This is better than testing whether port 5432 is open. A port can be open while the database is still starting or replaying WAL.

3. Redis probe

Redis is easy.

healthcheck:
  test: ["CMD", "redis-cli", "ping"]
  interval: 15s
  timeout: 3s
  retries: 5
  start_period: 10s

You want PONG, not wishful thinking.

4. Nginx or reverse proxy probe

A reverse proxy should prove it can answer an HTTP request.

healthcheck:
  test: ["CMD-SHELL", "curl -fsS http://localhost/ || exit 1"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 10s

This is also where it helps to pair health checks with a solid proxy setup. If you are already running Nginx Proxy Manager on Docker or debugging daemon proxy issues, this is one more place where local HTTP truth beats assumptions.

5. Worker probe

Workers are trickier because they may not expose HTTP.

I prefer a command that proves the worker runtime is alive and can reach the system it depends on. That might be a queue ping, a lightweight application command, or a process-specific status endpoint.

Example pattern:

healthcheck:
  test: ["CMD-SHELL", "python /app/bin/worker_healthcheck.py || exit 1"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 20s

Keep that script narrow. If your health check becomes a full integration test, you built a monitoring system with worse ergonomics.

A Compose example I would actually run

Here is a realistic stack with an app, Postgres, and Redis. This is the kind of setup where health checks stop being optional and start saving you time.

services:
  postgres:
    image: postgres:17
    container_name: demo-postgres
    environment:
      POSTGRES_DB: appdb
      POSTGRES_USER: appuser
      POSTGRES_PASSWORD: change-me
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U appuser -d appdb -h localhost || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 20s
    restart: unless-stopped

  redis:
    image: redis:7-alpine
    container_name: demo-redis
    command: ["redis-server", "--appendonly", "yes"]
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 10s
    restart: unless-stopped

  app:
    image: ghcr.io/example/myapp:latest
    container_name: demo-app
    environment:
      DATABASE_URL: postgres://appuser:change-me@postgres:5432/appdb
      REDIS_URL: redis://redis:6379/0
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    ports:
      - "3000:3000"
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://localhost:3000/healthz || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 20s
    restart: unless-stopped

volumes:
  postgres_data:
  redis_data:

This does two useful things.

First, each service reports whether it is actually usable. Second, the app waits for the database and cache to become healthy before startup dependencies resolve. That alone kills a surprising amount of flaky boot behavior.

Commands I use to verify health checks are real

After you define the check, do not just admire the YAML and move on. Test it.

Start the stack:

docker compose up -d

Check status:

docker compose ps

Inspect health details for one container:

docker inspect --format '{{json .State.Health}}' demo-app | jq

Or if jq is not installed:

docker inspect demo-app --format '{{range .State.Health.Log}}{{println .ExitCode .Output}}{{end}}'

Watch logs while you test:

docker compose logs -f app postgres redis

That log view is one reason I keep recommending solid container logging habits. If you need a refresher, this Docker logs guide covers the basics cleanly.

How to test a failure on purpose

You should break the service deliberately once.

Yes, really.

If you never test the failure mode, you do not know whether your check is honest.

A few examples:

Break the HTTP endpoint

If your app exposes /healthz, stop the internal service or temporarily point the endpoint at something invalid.

Break database connectivity

Change the database host in a test stack or stop Postgres briefly:

docker compose stop postgres

Then watch the app health transition.

Break Redis

docker compose stop redis

Then confirm whether your worker or app reports failure the way you expect.

What you are looking for is simple:

  • does the container move from healthy to unhealthy when the real function breaks?
  • does it stay healthy during normal startup?
  • does it avoid flapping on harmless transient delays?

If the answer to those is yes, the probe is probably good.

Common mistakes

1. Using a port-open check as proof the app is healthy

A listening port only proves something is listening.

It does not prove the app can serve a real request, talk to its database, or return anything besides an error page.

2. Forgetting that the check runs inside the container

localhost means the container itself, not the host.

I still see people write health checks that assume host networking behavior. That road leads to confusion and muttering.

3. Not installing the probe tool

If your image does not include curl, wget, pg_isready, or redis-cli, the health check fails because the command does not exist. Docker is not being unfair here. It is being extremely literal.

4. Making the check too expensive

A health check should be cheap.

Do not run a heavy SQL query, hit five upstream APIs, or perform a mini synthetic transaction every ten seconds unless you enjoy creating your own incidents.

5. Expecting unhealthy to trigger restarts automatically

This one deserves repetition.

Health status is a signal. Restart behavior is separate. If you want automatic recovery, pair health checks with restart policy, external monitoring, or orchestration logic.

6. Skipping start_period

Without a startup grace period, slower apps can fail a few checks during boot and look sick before they are actually ready.

How health checks fit into the bigger reliability picture

Health checks are not full monitoring.

They are one of the cheapest local truth signals you can add to a stack, and they become more useful when combined with alerting and telemetry. In my own setups, they sit alongside uptime monitoring, logs, and metrics. If you are already running Prometheus and Grafana, health state becomes another useful clue when a service behaves strangely.

They also work nicely with sane backup and recovery practices. A health check will not save your data, but it will tell you faster when a restored service did not come back cleanly. That matters whether you use Restic for offsite copies or a more storage-heavy plan.

Recommended gear for a reliable Docker host

You do not need expensive hardware for health checks, but stable hosts make debugging much less irritating.

  • Fanless N100 mini PC with 2.5GbE - a solid low-power Docker box for small self-hosted stacks
    https://www.amazon.com/s?k=fanless+n100+mini+pc+2.5gbe&tag=homelabaddiction-20
  • APC Back-UPS 1500VA - because clean shutdowns are cheaper than filesystem regret
    https://www.amazon.com/s?k=apc+back-ups+1500va&tag=homelabaddiction-20
  • Samsung T7 Shield SSD - handy for quick config backups, compose repos, and emergency copies
    https://www.amazon.com/s?k=samsung+t7+shield+ssd&tag=homelabaddiction-20

What to learn next

If you have health checks working, the next steps that actually compound are:

  1. standardize your Compose files
  2. add basic metrics and uptime checks
  3. harden exposed services
  4. back up the stack metadata and data volumes properly

These guides are a good next pass:

FAQ

Does Docker restart an unhealthy container automatically?

No. A health check changes health status. It does not, by itself, restart the container. Automatic recovery depends on restart policy, Compose behavior, or external tooling.

Should every container have a health check?

Not necessarily. Stateless web apps, databases, caches, and important workers usually benefit. Tiny one-shot helpers or containers with no meaningful runtime service boundary often do not need one.

What is the best command for a Docker health check?

The best command is the cheapest command that proves the real service is working. For web apps, that is usually a local HTTP request. For Postgres, pg_isready is better than a raw port check. For Redis, redis-cli ping is usually enough.

Why does my container stay unhealthy even though it is running?

Because running only means the main process still exists. Your health check may be failing due to bad config, dependency failures, missing probe tools, slow startup, or an endpoint that returns an error.

Is depends_on: condition: service_healthy worth using?

Yes, especially in self-hosted multi-service stacks. It reduces race conditions during startup and makes Compose bring services up in a more useful order.

Final opinion

Docker health checks are one of those features people ignore until they spend an evening debugging a container that is technically alive and functionally dead.

Use simple checks. Use honest checks. Test failure on purpose once. Then let Docker tell you something useful for a change.