Health Checks and Graceful Degradation

A health check is an endpoint or probe that reports whether a service instance can do its job. Orchestrators and load balancers use health checks to decide whether to send traffic to an instance, restart it, or remove it. Graceful degradation is the design goal that a service keeps providing reduced functionality when parts of it or its dependencies fail. Graceful shutdown lets an instance finish in-flight work before it exits.

Kinds of probes

Kubernetes defines three probe types. Other platforms use the same distinctions under different names.

Probes can be an HTTP GET (success is a status from 200 to 399), a TCP connection, a gRPC health check, or a command executed inside the container. Each probe has a period, a timeout, and a failure threshold that sets how many consecutive failures count as failed. In Kubernetes the defaults are a period of 10 s, a timeout of 1 s, and a failure threshold of 3.

startupProbe:            # allows up to 30 x 10 s = 300 s for startup
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 30
livenessProbe:           # restarts after 3 failed checks, 10 s apart
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:          # removes the pod from Service endpoints after 2 failures
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2

stateDiagram-v2
    [*] --> Starting
    Starting --> Ready: startup and readiness pass
    Ready --> NotReady: readiness fails
    NotReady --> Ready: readiness passes
    Starting --> Restarting: startup probe never passes
    Ready --> Restarting: liveness fails
    NotReady --> Restarting: liveness fails
    Restarting --> Starting

Liveness

A liveness probe detects states the process cannot recover from itself: deadlock, stuck event loop, corrupted internal state. It should check only the instance itself. It must not check dependencies. If a liveness probe fails when the database is down, the orchestrator restarts every instance at once, which does not fix the database and adds load from restarts and cold caches.

Readiness

A readiness probe reports temporary inability to serve: still warming a cache, loading data, overloaded, or a required dependency unreachable. Failure removes the instance from the set receiving traffic. The instance keeps running and is added back when the probe passes again.

Readiness and dependencies need care. If readiness fails whenever a shared dependency is down, all instances become unready together, and the service returns errors for everything, including requests that could have been served from cache or without that dependency. Include only dependencies without which the instance cannot serve any useful request, and prefer degraded responses over declaring the instance unready.

Startup

Slow-starting applications (large JVM services, services loading big models) can be killed by a liveness probe before they are ready. A startup probe with a generous failure threshold holds liveness and readiness checks back until the application has started, so liveness thresholds can stay strict for normal operation.

Shallow versus deep checks

Guidelines:

• Use shallow checks for liveness.
• For readiness, check the local resources the instance needs (initialized, connection pool created) and limit dependency checks to critical ones.
• Run deep checks in the background and cache the result for a short time, instead of executing them on every probe, so probe traffic does not overload dependencies.
• Give each check its own short timeout, and return quickly.
• Keep detailed diagnostic checks on a separate endpoint used by operators and monitoring, not by the orchestrator.

Health checks at load balancers

Load balancers run their own checks, and the same rules apply. Active checks send periodic probes. Passive checks observe real traffic and remove instances that produce errors or timeouts (outlier detection). Use a threshold of several consecutive failures to avoid flapping on a single lost probe, and a separate threshold to add an instance back. See load balancing.

Graceful degradation

When a dependency or resource fails, a service can reduce functionality instead of failing entirely. It requires deciding in advance which features are essential.

Implementation points:

• Classify dependencies as critical (the request cannot succeed without it) or optional (a response without it is acceptable). Timeouts, circuit breakers, and fallbacks are set according to the class.
• Feature flags allow operators to turn off expensive or failing features quickly (see feature flags).
• Signal degradation to clients where useful, for example with a response field or header, and record it in metrics.
• Test degraded modes. They run rarely, so they break silently. Fault injection verifies them (see chaos engineering).

Graceful shutdown

Instances stop during deployments, scale-in, and node maintenance. If a process exits abruptly, in-flight requests fail and clients see errors. A graceful shutdown sequence:

1. The platform sends a termination signal (SIGTERM).
2. The instance stops reporting ready, so the load balancer or Service stops sending it new requests.
3. The instance stops accepting new connections and finishes in-flight requests, within a time limit.
4. It closes resources: flushes buffers, commits or releases queue messages, closes database connections.
5. The process exits.

sequenceDiagram
    participant K as Orchestrator
    participant LB as Load balancer
    participant P as Pod
    K->>P: SIGTERM
    K->>LB: remove endpoint
    Note over P: readiness fails, stop accepting new connections
    LB->>P: no new requests
    P->>P: finish in-flight requests
    P-->>K: exit
    K->>P: SIGKILL if grace period exceeded

Kubernetes details:

• The pod is marked terminating, sends SIGTERM to containers, and removes the pod from Service endpoints. These happen concurrently, and endpoint removal takes time to reach all proxies and load balancers. Requests can still arrive for a short period after SIGTERM, so applications often keep serving for a few seconds before closing the listener. A preStop hook with a short delay is a common technique for this.
• terminationGracePeriodSeconds sets the time before SIGKILL. Its default is 30 seconds. The time spent in a preStop hook counts against this period, so the period must cover the hook delay plus the longest in-flight request.
• The application must handle SIGTERM. Applications running as a shell wrapper that does not forward signals never receive it.
• Long-lived connections (WebSockets, streaming) need a protocol-level signal to reconnect elsewhere, such as HTTP/2 GOAWAY.
• Queue consumers stop fetching new messages, finish or return the current ones, and commit offsets.

Trade-offs

• Aggressive probes detect failure quickly but cause restarts on transient slowness. Lenient probes tolerate blips but keep bad instances in rotation longer.
• Deep checks find more problems but couple instance health to dependency health.
• Degradation modes add code paths and test surface.

Interview questions

• What is the difference between liveness and readiness probes? Liveness failure restarts the container because the process is stuck. Readiness failure only removes the instance from load balancing until it recovers.
• Why should a liveness probe not check the database? A database outage would fail liveness on all instances, and the orchestrator would restart them all, adding load and losing warm state without fixing the cause.
• What is a startup probe for? It delays liveness and readiness checks for slow-starting applications so they are not killed before initialization completes.
• How do you shut down a service without dropping requests? On SIGTERM, fail readiness, stop accepting new connections, finish in-flight requests within a time limit, release resources, and exit. Allow for the delay before load balancers stop routing to the instance.
• What is graceful degradation? Continuing to provide reduced functionality, such as serving stale data or disabling optional features, when a dependency fails, based on a prior classification of critical and optional functions.