Health Checks, Graceful Shutdown, and Configuration

An orchestrator such as Kubernetes starts, stops, and routes traffic to service instances. This lesson builds one Axum service with health endpoints, a drain-then-stop shutdown, typed configuration, and percentage feature flags.

Term	Meaning
Liveness	The process runs and is not stuck. If it fails, the orchestrator restarts it
Readiness	The instance can serve traffic now. If it fails, the orchestrator stops routing to it and does not restart it
Graceful shutdown	Finishing accepted requests before the process exits
Feature flag	A setting that turns behavior on or off, or on for a percentage of users, without a deployment

Project setup

Cargo.toml

TOML

[package]
name = "storefront"
version = "0.1.0"
edition = "2021"

[dependencies]
axum = "0.8.4"
tokio = { version = "1.47", features = ["macros", "rt-multi-thread", "signal", "net", "time"] }
sqlx = { version = "0.8.6", features = ["runtime-tokio", "postgres"] }
thiserror = "2"
tracing = "0.1"
tracing-subscriber = "0.3"

The readiness check needs Postgres:

Terminal

docker run -d --name storefront-pg -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:16

File layout: src/main.rs, src/config.rs, src/flags.rs, src/state.rs, src/health.rs.

Example 1: Typed configuration from the environment

Configuration is everything that differs between deployments. Code stays the same and environment variables supply the values. A typed struct parsed once at startup fails on the first bad value instead of at the first request that needs it.

src/config.rs

Rust

use std::{collections::HashMap, fmt, net::SocketAddr, str::FromStr, time::Duration};

#[derive(Clone)]
pub struct Secret(String);

impl Secret {
    pub fn expose(&self) -> &str {
        &self.0
    }
}

impl fmt::Debug for Secret {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str("Secret(***)")
    }
}

#[derive(Debug, Clone)]
pub struct Config {
    pub listen_addr: SocketAddr,
    pub database_url: Secret,
    pub db_max_connections: u32,
    pub drain_delay: Duration,
    pub feature_flags: HashMap<String, u8>,
}

#[derive(Debug, thiserror::Error)]
pub enum ConfigError {
    #[error("missing environment variable {0}")]
    Missing(&'static str),
    #[error("invalid value for {name}: {reason}")]
    Invalid { name: &'static str, reason: String },
}

fn required(name: &'static str) -> Result<String, ConfigError> {
    std::env::var(name).map_err(|_| ConfigError::Missing(name))
}

fn parsed<T>(name: &'static str, default: T) -> Result<T, ConfigError>
where
    T: FromStr,
    T::Err: fmt::Display,
{
    match std::env::var(name) {
        Ok(text) => text.parse::<T>().map_err(|e| ConfigError::Invalid {
            name,
            reason: e.to_string(),
        }),
        Err(_) => Ok(default),
    }
}

// Format: "new_checkout=25,recommendations=0"
fn parse_flags(text: &str) -> Result<HashMap<String, u8>, ConfigError> {
    let mut flags = HashMap::new();
    for entry in text.split(',').filter(|e| !e.is_empty()) {
        let invalid = |reason: &str| ConfigError::Invalid {
            name: "FEATURE_FLAGS",
            reason: format!("{entry}: {reason}"),
        };
        let (name, percent) = entry.split_once('=').ok_or_else(|| invalid("expected name=percent"))?;
        let percent: u8 = percent.parse().map_err(|_| invalid("percent is not a number"))?;
        if percent > 100 {
            return Err(invalid("percent above 100"));
        }
        flags.insert(name.to_owned(), percent);
    }
    Ok(flags)
}

impl Config {
    pub fn from_env() -> Result<Self, ConfigError> {
        let db_max_connections: u32 = parsed("DB_MAX_CONNECTIONS", 10)?;
        if db_max_connections == 0 {
            return Err(ConfigError::Invalid {
                name: "DB_MAX_CONNECTIONS",
                reason: "must be at least 1".to_owned(),
            });
        }
        let flags_text = std::env::var("FEATURE_FLAGS").unwrap_or_default();
        Ok(Self {
            listen_addr: parsed("LISTEN_ADDR", "0.0.0.0:8080".parse().unwrap())?,
            database_url: Secret(required("DATABASE_URL")?),
            db_max_connections,
            drain_delay: Duration::from_millis(parsed("DRAIN_DELAY_MS", 5_000)?),
            feature_flags: parse_flags(&flags_text)?,
        })
    }
}

Secret prints as Secret(***), so logging the whole Config never leaks the database URL.
required fails when a variable is absent. parsed uses a default when it is absent and fails when it is present but invalid.
Every setting has one name. Secrets belong only in environment variables or mounted files, never in files committed to the repository.
Files and command-line flags are further layers (defaults, then a base file, then an environment file, then variables, then flags, with later layers overriding earlier ones). The config, figment, and clap crates implement them, and their builder APIs differ by version.

Example 2: Feature flags with stable buckets

A flag stores a rollout percentage. Hashing the flag name and user id gives each user a fixed bucket from 0 to 99, so one customer does not flip between old and new behavior on each request. The hash is written out here (FNV-1a) because the standard library does not promise a stable algorithm across Rust releases.

src/flags.rs

Rust

use std::collections::HashMap;

#[derive(Debug, Clone, Default)]
pub struct Flags {
    rollout: HashMap<String, u8>,
}

impl Flags {
    pub fn new(rollout: HashMap<String, u8>) -> Self {
        Self { rollout }
    }

    /// Unknown flags are off. 0 is off for everyone, 100 is on for everyone.
    pub fn is_on(&self, name: &str, user_id: i64) -> bool {
        match self.rollout.get(name) {
            Some(&percent) => bucket(name, user_id) < u64::from(percent),
            None => false,
        }
    }
}

fn bucket(name: &str, user_id: i64) -> u64 {
    let mut hash: u64 = 0xcbf2_9ce4_8422_2325;
    let bytes = name
        .bytes()
        .chain(std::iter::once(0xff))
        .chain(user_id.to_le_bytes());
    for byte in bytes {
        hash ^= u64::from(byte);
        hash = hash.wrapping_mul(0x0000_0100_0000_01b3);
    }
    hash % 100
}

Uses of flags: gradual rollout (5%, then 50%, then 100%), kill switch (set to 0), migration control (read from the new table), and beta access. Each flag needs an owner and a removal date, and the flag and old code path are deleted after the rollout.

Example 3: Health endpoints

The two probes answer different questions, so they are different endpoints.

/livez checks nothing external. If liveness depended on the database, a database outage would restart every instance, which does not repair the database.
/readyz returns 200 only when the instance can serve most requests. Here that means the ready flag is set and the database answers within one second.

src/state.rs

Rust

use std::sync::{atomic::AtomicBool, Arc};

use sqlx::PgPool;

use crate::{config::Config, flags::Flags};

#[derive(Clone)]
pub struct AppState {
    pub pool: PgPool,
    pub ready: Arc<AtomicBool>,
    pub config: Arc<Config>,
    pub flags: Arc<Flags>,
}

src/health.rs

Rust

use std::{sync::atomic::Ordering, time::Duration};

use axum::{extract::State, http::StatusCode};

use crate::state::AppState;

pub async fn livez() -> &'static str {
    "alive"
}

pub async fn readyz(State(state): State<AppState>) -> (StatusCode, &'static str) {
    if !state.ready.load(Ordering::Relaxed) {
        return (StatusCode::SERVICE_UNAVAILABLE, "not ready");
    }
    let query = sqlx::query("SELECT 1").execute(&state.pool);
    match tokio::time::timeout(Duration::from_secs(1), query).await {
        Ok(Ok(_)) => (StatusCode::OK, "ready"),
        _ => (StatusCode::SERVICE_UNAVAILABLE, "not ready"),
    }
}

Startup order: parse configuration, initialize logging, connect the pool, build state, bind the listener, set ready, serve. Migrations run before ready is set, or in a separate job. Rolling updates run old and new versions together, so migrations must be backward compatible with the previous version.
with_graceful_shutdown takes a future. When it completes, the server stops accepting connections and waits for in-flight requests.
Kubernetes sends SIGTERM, waits the grace period, then sends SIGKILL. Idle keep-alive connections and long streams can delay the drain until SIGKILL.

Example 4: Graceful shutdown with a drain delay

Load balancers learn that an instance is unready with a delay. An instance that stops accepting connections at once still receives requests for a few seconds and fails them. The correct order is: mark unready, keep serving during a delay, then stop.

src/main.rs

Rust

mod config;
mod flags;
mod health;
mod state;

use std::{
    sync::{atomic::{AtomicBool, Ordering}, Arc},
    time::Duration,
};

use axum::{extract::{Path, State}, routing::get, Router};
use sqlx::postgres::PgPoolOptions;
use tokio::{net::TcpListener, signal};

use crate::{config::Config, flags::Flags, state::AppState};

async fn checkout(State(state): State<AppState>, Path(user_id): Path<i64>) -> String {
    if state.flags.is_on("new_checkout", user_id) {
        "checkout: new".to_owned()
    } else {
        "checkout: old".to_owned()
    }
}

async fn shutdown_with_drain(ready: Arc<AtomicBool>, drain_delay: Duration) {
    let mut terminate = signal::unix::signal(signal::unix::SignalKind::terminate())
        .expect("install SIGTERM handler");
    tokio::select! {
        _ = signal::ctrl_c() => {}
        _ = terminate.recv() => {}
    }
    tracing::info!("shutdown signal received, marking unready");
    ready.store(false, Ordering::Relaxed);
    tokio::time::sleep(drain_delay).await;
    tracing::info!("drain delay over, stopping server");
}

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt::init();

    let config = match Config::from_env() {
        Ok(config) => Arc::new(config),
        Err(error) => {
            eprintln!("configuration error: {error}");
            std::process::exit(1);
        }
    };
    tracing::info!(?config, "configuration loaded");

    let pool = PgPoolOptions::new()
        .max_connections(config.db_max_connections)
        .acquire_timeout(Duration::from_secs(3))
        .connect(config.database_url.expose())
        .await
        .expect("connect to database");

    let ready = Arc::new(AtomicBool::new(false));
    let state = AppState {
        pool: pool.clone(),
        ready: ready.clone(),
        config: config.clone(),
        flags: Arc::new(Flags::new(config.feature_flags.clone())),
    };

    let app = Router::new()
        .route("/livez", get(health::livez))
        .route("/readyz", get(health::readyz))
        .route("/checkout/{user_id}", get(checkout))
        .with_state(state);

    let listener = TcpListener::bind(config.listen_addr).await.expect("bind listener");
    ready.store(true, Ordering::Relaxed);

    axum::serve(listener, app)
        .with_graceful_shutdown(shutdown_with_drain(ready, config.drain_delay))
        .await
        .expect("serve");

    pool.close().await;
    tracing::info!("server stopped");
}

The drain delay plus the request drain time plus cleanup must be shorter than the pod's termination grace period (30 seconds by default), or the process is killed mid-cleanup. The Kubernetes side is in Docker, Kubernetes, and CI/CD. Background tasks and message consumers stop through a CancellationToken as covered in Tokio graceful shutdown.

Complete example

Prerequisites: Rust 1.85 or newer, Docker. File tree:

Text

storefront/
├── Cargo.toml
└── src/
    ├── config.rs
    ├── flags.rs
    ├── health.rs
    ├── main.rs
    └── state.rs

Start Postgres, then create each file below.

Terminal

docker run -d --name storefront-pg -e POSTGRES_PASSWORD=postgres -p 5432:5432 postgres:16

Cargo.toml

TOML

[package]
name = "storefront"
version = "0.1.0"
edition = "2021"

[dependencies]
axum = "0.8.4"
tokio = { version = "1.47", features = ["macros", "rt-multi-thread", "signal", "net", "time"] }
sqlx = { version = "0.8.6", features = ["runtime-tokio", "postgres"] }
thiserror = "2"
tracing = "0.1"
tracing-subscriber = "0.3"

src/config.rs

Rust

use std::{collections::HashMap, fmt, net::SocketAddr, str::FromStr, time::Duration};

#[derive(Clone)]
pub struct Secret(String);

impl Secret {
    pub fn expose(&self) -> &str {
        &self.0
    }
}

impl fmt::Debug for Secret {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        f.write_str("Secret(***)")
    }
}

#[derive(Debug, Clone)]
pub struct Config {
    pub listen_addr: SocketAddr,
    pub database_url: Secret,
    pub db_max_connections: u32,
    pub drain_delay: Duration,
    pub feature_flags: HashMap<String, u8>,
}

#[derive(Debug, thiserror::Error)]
pub enum ConfigError {
    #[error("missing environment variable {0}")]
    Missing(&'static str),
    #[error("invalid value for {name}: {reason}")]
    Invalid { name: &'static str, reason: String },
}

fn required(name: &'static str) -> Result<String, ConfigError> {
    std::env::var(name).map_err(|_| ConfigError::Missing(name))
}

fn parsed<T>(name: &'static str, default: T) -> Result<T, ConfigError>
where
    T: FromStr,
    T::Err: fmt::Display,
{
    match std::env::var(name) {
        Ok(text) => text.parse::<T>().map_err(|e| ConfigError::Invalid {
            name,
            reason: e.to_string(),
        }),
        Err(_) => Ok(default),
    }
}

// Format: "new_checkout=25,recommendations=0"
fn parse_flags(text: &str) -> Result<HashMap<String, u8>, ConfigError> {
    let mut flags = HashMap::new();
    for entry in text.split(',').filter(|e| !e.is_empty()) {
        let invalid = |reason: &str| ConfigError::Invalid {
            name: "FEATURE_FLAGS",
            reason: format!("{entry}: {reason}"),
        };
        let (name, percent) = entry.split_once('=').ok_or_else(|| invalid("expected name=percent"))?;
        let percent: u8 = percent.parse().map_err(|_| invalid("percent is not a number"))?;
        if percent > 100 {
            return Err(invalid("percent above 100"));
        }
        flags.insert(name.to_owned(), percent);
    }
    Ok(flags)
}

impl Config {
    pub fn from_env() -> Result<Self, ConfigError> {
        let db_max_connections: u32 = parsed("DB_MAX_CONNECTIONS", 10)?;
        if db_max_connections == 0 {
            return Err(ConfigError::Invalid {
                name: "DB_MAX_CONNECTIONS",
                reason: "must be at least 1".to_owned(),
            });
        }
        let flags_text = std::env::var("FEATURE_FLAGS").unwrap_or_default();
        Ok(Self {
            listen_addr: parsed("LISTEN_ADDR", "0.0.0.0:8080".parse().unwrap())?,
            database_url: Secret(required("DATABASE_URL")?),
            db_max_connections,
            drain_delay: Duration::from_millis(parsed("DRAIN_DELAY_MS", 5_000)?),
            feature_flags: parse_flags(&flags_text)?,
        })
    }
}

src/flags.rs

Rust

use std::collections::HashMap;

#[derive(Debug, Clone, Default)]
pub struct Flags {
    rollout: HashMap<String, u8>,
}

impl Flags {
    pub fn new(rollout: HashMap<String, u8>) -> Self {
        Self { rollout }
    }

    /// Unknown flags are off. 0 is off for everyone, 100 is on for everyone.
    pub fn is_on(&self, name: &str, user_id: i64) -> bool {
        match self.rollout.get(name) {
            Some(&percent) => bucket(name, user_id) < u64::from(percent),
            None => false,
        }
    }
}

fn bucket(name: &str, user_id: i64) -> u64 {
    let mut hash: u64 = 0xcbf2_9ce4_8422_2325;
    let bytes = name
        .bytes()
        .chain(std::iter::once(0xff))
        .chain(user_id.to_le_bytes());
    for byte in bytes {
        hash ^= u64::from(byte);
        hash = hash.wrapping_mul(0x0000_0100_0000_01b3);
    }
    hash % 100
}

src/state.rs

Rust

use std::sync::{atomic::AtomicBool, Arc};

use sqlx::PgPool;

use crate::{config::Config, flags::Flags};

#[derive(Clone)]
pub struct AppState {
    pub pool: PgPool,
    pub ready: Arc<AtomicBool>,
    pub config: Arc<Config>,
    pub flags: Arc<Flags>,
}

src/health.rs

Rust

use std::{sync::atomic::Ordering, time::Duration};

use axum::{extract::State, http::StatusCode};

use crate::state::AppState;

pub async fn livez() -> &'static str {
    "alive"
}

pub async fn readyz(State(state): State<AppState>) -> (StatusCode, &'static str) {
    if !state.ready.load(Ordering::Relaxed) {
        return (StatusCode::SERVICE_UNAVAILABLE, "not ready");
    }
    let query = sqlx::query("SELECT 1").execute(&state.pool);
    match tokio::time::timeout(Duration::from_secs(1), query).await {
        Ok(Ok(_)) => (StatusCode::OK, "ready"),
        _ => (StatusCode::SERVICE_UNAVAILABLE, "not ready"),
    }
}

src/main.rs

Rust

mod config;
mod flags;
mod health;
mod state;

use std::{
    sync::{atomic::{AtomicBool, Ordering}, Arc},
    time::Duration,
};

use axum::{extract::{Path, State}, routing::get, Router};
use sqlx::postgres::PgPoolOptions;
use tokio::{net::TcpListener, signal};

use crate::{config::Config, flags::Flags, state::AppState};

async fn checkout(State(state): State<AppState>, Path(user_id): Path<i64>) -> String {
    if state.flags.is_on("new_checkout", user_id) {
        "checkout: new".to_owned()
    } else {
        "checkout: old".to_owned()
    }
}

async fn shutdown_with_drain(ready: Arc<AtomicBool>, drain_delay: Duration) {
    let mut terminate = signal::unix::signal(signal::unix::SignalKind::terminate())
        .expect("install SIGTERM handler");
    tokio::select! {
        _ = signal::ctrl_c() => {}
        _ = terminate.recv() => {}
    }
    tracing::info!("shutdown signal received, marking unready");
    ready.store(false, Ordering::Relaxed);
    tokio::time::sleep(drain_delay).await;
    tracing::info!("drain delay over, stopping server");
}

#[tokio::main]
async fn main() {
    tracing_subscriber::fmt::init();

    let config = match Config::from_env() {
        Ok(config) => Arc::new(config),
        Err(error) => {
            eprintln!("configuration error: {error}");
            std::process::exit(1);
        }
    };
    tracing::info!(?config, "configuration loaded");

    let pool = PgPoolOptions::new()
        .max_connections(config.db_max_connections)
        .acquire_timeout(Duration::from_secs(3))
        .connect(config.database_url.expose())
        .await
        .expect("connect to database");

    let ready = Arc::new(AtomicBool::new(false));
    let state = AppState {
        pool: pool.clone(),
        ready: ready.clone(),
        config: config.clone(),
        flags: Arc::new(Flags::new(config.feature_flags.clone())),
    };

    let app = Router::new()
        .route("/livez", get(health::livez))
        .route("/readyz", get(health::readyz))
        .route("/checkout/{user_id}", get(checkout))
        .with_state(state);

    let listener = TcpListener::bind(config.listen_addr).await.expect("bind listener");
    ready.store(true, Ordering::Relaxed);

    axum::serve(listener, app)
        .with_graceful_shutdown(shutdown_with_drain(ready, config.drain_delay))
        .await
        .expect("serve");

    pool.close().await;
    tracing::info!("server stopped");
}

Run and try

Start the service with a 5 second drain, one flag on for everyone, and one off:

Terminal

DATABASE_URL=postgres://postgres:postgres@127.0.0.1:5432/postgres \
FEATURE_FLAGS="new_checkout=100,recommendations=0" \
DRAIN_DELAY_MS=5000 \
cargo run

Requests and responses:

Terminal

curl -s -w ' [%{http_code}]\n' localhost:8080/livez
curl -s -w ' [%{http_code}]\n' localhost:8080/readyz
curl -s -w ' [%{http_code}]\n' localhost:8080/checkout/7

Output

alive [200]
ready [200]
checkout: new [200]

Stop the database and probe again. Liveness stays 200 and readiness fails after the one second timeout:

Terminal

docker stop storefront-pg
curl -s -w ' [%{http_code}]\n' localhost:8080/livez
curl -s -w ' [%{http_code}]\n' localhost:8080/readyz

Output

alive [200]
not ready [503]

Send SIGTERM and probe within 5 seconds. The server still answers and reports unready. After the delay the log shows server stopped and the process exits with code 0:

Terminal

kill -TERM "$(pgrep -f target/debug/storefront)"
curl -s -w ' [%{http_code}]\n' localhost:8080/readyz

Output

not ready [503]

Startup with a bad value exits with status 1 before any connection opens. DB_MAX_CONNECTIONS=abc cargo run prints:

Output

configuration error: invalid value for DB_MAX_CONNECTIONS: invalid digit found in string

Summary

Serve /livez and /readyz separately. Liveness checks nothing external. Readiness checks what the instance needs for most requests, with a timeout.
Keep health endpoints internal and unauthenticated, and return no dependency detail.
Shutdown order: mark unready, keep serving for a drain delay, stop accepting connections, finish in-flight requests, close pools, exit.
The total shutdown time must fit within the termination grace period.
Parse configuration once into a typed struct. Required values fail at startup with the setting's name.
Wrap secrets so that Debug output redacts them, and log the effective configuration once.
Total pool size across instances must fit under the database's max_connections.
Feature flags need stable per-user buckets, an owner, and a removal plan.