codeloopback
Guide contents

Lesson 24 of 44

Graceful Shutdown

A graceful shutdown stops a service from taking new work, lets the work already started finish, and then exits. The alternative, an abrupt stop, drops in-flight work at whatever .await it had reached. This lesson shows the steps, the tools that implement them, and what happens to the work in progress at every step.

Tool Role in shutdown
tokio::signal receives the request to stop (ctrl_c, SIGTERM)
CancellationToken (tokio-util) tells many tasks to stop: cancel, cancelled, child_token, drop_guard, run_until_cancelled
TaskTracker (tokio-util) counts running tasks and waits for all of them: spawn, close, wait
dropped Senders close a channel so receivers finish after the queue is empty
watch and broadcast carry a shutdown flag or phase
timeout puts a deadline on draining
Runtime::shutdown_timeout limits how long the runtime waits for its own tasks and blocking threads

Add the crate with cargo add tokio-util --features rt. Time-based examples run on the paused clock (start_paused = true, explained in Time: sleep, timeout, and interval), so the output does not depend on real time.

The shutdown sequence

Step Action State of in-flight work
1. Signal a signal, a message, or a failed component starts shutdown running normally
2. Stop accepting close listeners and stop reading new work from queues running normally; no new work arrives
3. Drain wait for tasks that already started running until they finish
4. Deadline if draining takes too long, cancel what remains unfinished work is dropped at its next .await
5. Exit flush logs and metrics, close connections, return from main nothing left

The examples below implement one step at a time, then combine them.

Example 1: Aborting and draining compared

The smallest case. A job has three steps, each taking 10 milliseconds. In the first run the main task aborts the job after 15 milliseconds. In the second run it waits for the job (drains it).

Rust
use std::time::Duration;
use tokio::time::sleep;

struct Guard(&'static str);

impl Drop for Guard {
    fn drop(&mut self) {
        println!("  [{}: future dropped]", self.0);
    }
}

async fn job(name: &'static str) {
    let _guard = Guard(name);
    for step in 1..=3 {
        sleep(Duration::from_millis(10)).await;
        println!("  {name}: step {step} done");
    }
}

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    println!("abort after 15 ms:");
    let handle = tokio::spawn(job("aborted"));
    sleep(Duration::from_millis(15)).await;
    handle.abort();
    let _ = handle.await;

    println!("drain (wait for the job):");
    let handle = tokio::spawn(job("drained"));
    sleep(Duration::from_millis(15)).await;
    handle.await.unwrap();
}
Output
abort after 15 ms:
  aborted: step 1 done
  [aborted: future dropped]
drain (wait for the job):
  drained: step 1 done
  drained: step 2 done
  drained: step 3 done
  [drained: future dropped]

The aborted job completed step 1 and lost steps 2 and 3. The Drop implementation ran, but no code after the .await did. The drained job completed all three steps. Every later example is a way to choose between these two outcomes for many tasks at once.

Example 2: Waiting for a stop signal

The request to stop usually arrives as a process signal. tokio::signal::ctrl_c completes when the process receives Ctrl-C (SIGINT). On Unix, signal(SignalKind::terminate()) completes on SIGTERM, the signal that container runtimes and process managers send first (Files, Processes, and Signals covers signals). The function below completes on whichever arrives first. The code is no_run because the checker cannot deliver a signal to it.

Rust
use tokio::signal;
use tokio::signal::unix::{signal, SignalKind};

async fn shutdown_signal() {
    let ctrl_c = async {
        signal::ctrl_c().await.expect("failed to install Ctrl-C handler");
    };
    let mut terminate = signal(SignalKind::terminate()).expect("failed to install SIGTERM handler");

    tokio::select! {
        _ = ctrl_c => println!("received Ctrl-C"),
        _ = terminate.recv() => println!("received SIGTERM"),
    }
}

#[tokio::main]
async fn main() {
    println!("running; press Ctrl-C to stop");
    shutdown_signal().await;
    println!("shutting down");
}

The rest of this lesson replaces the signal with a timer or a message, so the examples can run and print the same lines every time. In a real service, shutdown_signal().await sits where those examples call token.cancel().

Example 3: CancellationToken

A CancellationToken is a shared flag that starts as not cancelled. cancel sets it. cancelled() returns a future that completes once the token is cancelled, and it completes immediately if the token was cancelled earlier. Clones share the same state.

Rust
use tokio::time::{sleep, Duration};
use tokio_util::sync::CancellationToken;

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let token = CancellationToken::new();
    println!("cancelled at start: {}", token.is_cancelled());

    let waiter = {
        let token = token.clone();
        tokio::spawn(async move {
            token.cancelled().await;
            println!("waiter: token was cancelled");
        })
    };

    sleep(Duration::from_millis(10)).await;
    token.cancel();
    waiter.await.unwrap();
    println!("cancelled now: {}", token.is_cancelled());

    // A token that is already cancelled completes immediately.
    token.cancelled().await;
    println!("second wait returned at once");
}
Output
cancelled at start: false
waiter: token was cancelled
cancelled now: true
second wait returned at once

cancelled() is cancel safe, so it can be a branch of select! in a loop. A token cannot be un-cancelled: once cancel is called the token stays cancelled.

Example 4: A worker that finishes its current job

A worker loop waits for either a job or the token. When the token fires, the worker exits the loop. A job that was already taken is not part of the select!, so it runs to the end. biased; makes select! check branches in order, so a cancelled token wins over a waiting job (select!: Racing Futures).

Rust
use std::time::Duration;
use tokio::sync::mpsc;
use tokio::time::sleep;
use tokio_util::sync::CancellationToken;

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let token = CancellationToken::new();
    let (tx, mut rx) = mpsc::channel::<u32>(10);
    for job in 1..=5 {
        tx.send(job).await.unwrap();
    }

    let worker = {
        let token = token.clone();
        tokio::spawn(async move {
            let mut finished = 0;
            loop {
                let job = tokio::select! {
                    biased;
                    _ = token.cancelled() => break,
                    job = rx.recv() => match job {
                        Some(job) => job,
                        None => break,
                    },
                };
                println!("job {job} started");
                sleep(Duration::from_millis(10)).await;
                println!("job {job} finished");
                finished += 1;
            }
            (finished, rx.len())
        })
    };

    sleep(Duration::from_millis(15)).await;
    println!("cancelling");
    token.cancel();

    let (finished, left) = worker.await.unwrap();
    println!("finished {finished} jobs, {left} jobs left in the queue");
}
Output
job 1 started
job 1 finished
job 2 started
cancelling
job 2 finished
finished 2 jobs, 3 jobs left in the queue

The cancel arrived while job 2 was running. Job 2 finished, the loop saw the cancelled token, and jobs 3 to 5 stayed in the channel. What happens to those queued jobs is a design decision: drop them, persist them, or drain them (Example 12).

Example 5: Child tokens

child_token creates a token that is cancelled when its parent is cancelled. Cancelling a child does not cancel the parent or the siblings. A tree of tokens matches a tree of components: cancel the root to stop everything, or cancel one branch to stop one component.

Rust
use tokio_util::sync::CancellationToken;

#[tokio::main]
async fn main() {
    let root = CancellationToken::new();
    let database = root.child_token();
    let network = root.child_token();
    let connection = network.child_token();

    network.cancel();
    println!("root:       {}", root.is_cancelled());
    println!("database:   {}", database.is_cancelled());
    println!("network:    {}", network.is_cancelled());
    println!("connection: {}", connection.is_cancelled());

    root.cancel();
    println!("after root.cancel(), database: {}", database.is_cancelled());
}
Output
root:       false
database:   false
network:    true
connection: true
after root.cancel(), database: true

Cancelling network cancelled its child connection and nothing else. Cancelling root then reached database.

Example 6: DropGuard cancels when it is dropped

drop_guard consumes a token and returns a DropGuard. Dropping the guard cancels the token. A function that owns the guard cancels its helpers on every exit path: normal return, early return with ?, and panic unwinding. disarm turns the guard back into a plain token without cancelling.

Rust
use tokio::time::{sleep, Duration};
use tokio_util::sync::CancellationToken;

async fn helper(token: CancellationToken) {
    token.cancelled().await;
    println!("helper: told to stop");
}

async fn with_guard() {
    let token = CancellationToken::new();
    let _guard = token.clone().drop_guard();
    tokio::spawn(helper(token));
    sleep(Duration::from_millis(1)).await;
    println!("with_guard: returning");
}

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    with_guard().await;
    sleep(Duration::from_millis(1)).await;

    let token = CancellationToken::new();
    let guard = token.clone().drop_guard();
    let token_again = guard.disarm();
    println!("after disarm and drop, cancelled = {}", token.is_cancelled());
    drop(token_again);
}
Output
with_guard: returning
helper: told to stop
after disarm and drop, cancelled = false

with_guard never called cancel. Leaving the function dropped _guard, which cancelled the token and stopped helper.

Example 7: run_until_cancelled

token.run_until_cancelled(future) runs a future and returns Some(output) if it completes, or None if the token is cancelled first. In the second case the future is dropped at its current .await. This is the abort style of Example 1 driven by a token.

Rust
use tokio::time::{sleep, Duration};
use tokio_util::sync::CancellationToken;

async fn work(ms: u64) -> &'static str {
    sleep(Duration::from_millis(ms)).await;
    "finished"
}

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let token = CancellationToken::new();

    let quick = token.run_until_cancelled(work(10)).await;
    println!("before cancel: {quick:?}");

    let canceller = {
        let token = token.clone();
        tokio::spawn(async move {
            sleep(Duration::from_millis(20)).await;
            token.cancel();
        })
    };
    let slow = token.run_until_cancelled(work(1000)).await;
    println!("cancelled during work: {slow:?}");

    let after = token.run_until_cancelled(work(10)).await;
    println!("already cancelled: {after:?}");
    canceller.await.unwrap();
}
Output
before cancel: Some("finished")
cancelled during work: None
already cancelled: None

run_until_cancelled is cancel safe only if the wrapped future is cancel safe. If the future is dropped while it is halfway through a multi-step write, the write stays half done (Example 11).

Example 8: TaskTracker waits for tasks to exit

Cancelling a token does not wait for anything. To know that the tasks have really stopped, count them with a TaskTracker. spawn adds a task, close says no more tasks are planned, and wait completes when the tracker is closed and all tasks have exited.

Rust
use std::time::Duration;
use tokio::time::sleep;
use tokio_util::sync::CancellationToken;
use tokio_util::task::TaskTracker;

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let token = CancellationToken::new();
    let tracker = TaskTracker::new();

    for id in 0..3u64 {
        let token = token.clone();
        tracker.spawn(async move {
            loop {
                tokio::select! {
                    biased;
                    _ = token.cancelled() => break,
                    _ = sleep(Duration::from_millis(10)) => {}
                }
            }
            // Cleanup takes time that is different for each task.
            sleep(Duration::from_millis(5 * (id + 1))).await;
            println!("task {id} cleaned up");
        });
    }

    sleep(Duration::from_millis(25)).await;
    println!("cancel; running tasks: {}", tracker.len());
    token.cancel();
    tracker.close();
    tracker.wait().await;
    println!("wait returned; running tasks: {}", tracker.len());
}
Output
cancel; running tasks: 3
task 0 cleaned up
task 1 cleaned up
task 2 cleaned up
wait returned; running tasks: 0

token.cancel() returned at once. Only tracker.wait() blocked until the last task finished its cleanup. Calling wait on a tracker that was never closed would not return; JoinSet and Structured Concurrency explains the tracker in more detail.

Example 9: A complete sequence for a small server

This program follows the five steps. The "server" reads requests from a channel instead of a socket so the example is deterministic. Requests 1, 2, and 3 arrive at 0, 10, and 20 ms and each takes 50 ms to handle. Shutdown starts at 25 ms. Request 4 arrives at 30 ms, after the server stopped accepting.

Rust
use std::time::Duration;
use tokio::sync::mpsc;
use tokio::time::{sleep, timeout};
use tokio_util::sync::CancellationToken;
use tokio_util::task::TaskTracker;

async fn handle(id: u32) {
    sleep(Duration::from_millis(50)).await;
    println!("  request {id} finished");
}

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let token = CancellationToken::new();
    let tracker = TaskTracker::new();
    let (tx, mut incoming) = mpsc::channel::<u32>(16);

    // Client: one request every 10 ms.
    let client = tokio::spawn(async move {
        for id in 1..=4 {
            match tx.send(id).await {
                Ok(()) => println!("client: request {id} sent"),
                Err(_) => println!("client: request {id} refused (server stopped accepting)"),
            }
            sleep(Duration::from_millis(10)).await;
        }
    });

    // Accept loop: one task per request, tracked.
    let accept = {
        let token = token.clone();
        let tracker = tracker.clone();
        tokio::spawn(async move {
            loop {
                tokio::select! {
                    biased;
                    _ = token.cancelled() => break,
                    request = incoming.recv() => match request {
                        Some(id) => { tracker.spawn(handle(id)); }
                        None => break,
                    },
                }
            }
            // `incoming` is dropped here: senders now get an error.
        })
    };

    sleep(Duration::from_millis(25)).await;
    println!("1. signal received");
    token.cancel();

    accept.await.unwrap();
    println!("2. stopped accepting; in flight: {}", tracker.len());

    tracker.close();
    println!("3. draining");
    match timeout(Duration::from_millis(200), tracker.wait()).await {
        Ok(()) => println!("4. drained; in flight: {}", tracker.len()),
        Err(_) => println!("4. deadline passed; in flight: {}", tracker.len()),
    }

    client.await.unwrap();
    println!("5. exit");
}
Output
client: request 1 sent
client: request 2 sent
client: request 3 sent
1. signal received
2. stopped accepting; in flight: 3
3. draining
client: request 4 refused (server stopped accepting)
  request 1 finished
  request 2 finished
  request 3 finished
4. drained; in flight: 0
5. exit

Reading the output as a timeline:

Time What happened In-flight requests
0, 10, 20 ms requests 1 to 3 were accepted and spawned 1, 2, 3
25 ms step 1: the token was cancelled; step 2: the accept loop exited and dropped its receiver 3
30 ms request 4 was refused by the closed channel 3
50, 60, 70 ms step 3: the handlers finished one by one 2, 1, 0
after 70 ms step 4: wait returned before the 200 ms deadline; step 5: exit 0

Example 10: A drain deadline followed by a hard cancel

A drain that never ends blocks the exit. The pattern is two tokens. The soft phase lets in-flight work finish. When the deadline passes, a second token cancels the handlers that remain. Each handler wraps its work in run_until_cancelled on the hard token.

Rust
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Duration;
use tokio::time::{sleep, timeout};
use tokio_util::sync::CancellationToken;
use tokio_util::task::TaskTracker;

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let hard = CancellationToken::new();
    let tracker = TaskTracker::new();
    let completed = Arc::new(AtomicUsize::new(0));
    let cancelled = Arc::new(AtomicUsize::new(0));

    for (id, ms) in [(1u32, 20u64), (2, 500), (3, 30)] {
        let hard = hard.clone();
        let completed = completed.clone();
        let cancelled = cancelled.clone();
        tracker.spawn(async move {
            match hard.run_until_cancelled(sleep(Duration::from_millis(ms))).await {
                Some(()) => {
                    completed.fetch_add(1, Ordering::SeqCst);
                    println!("request {id} completed");
                }
                None => {
                    cancelled.fetch_add(1, Ordering::SeqCst);
                    println!("request {id} cancelled at the deadline");
                }
            }
        });
    }

    tracker.close();
    println!("draining with a 100 ms deadline");
    if timeout(Duration::from_millis(100), tracker.wait()).await.is_err() {
        println!("deadline passed; in flight: {}", tracker.len());
        hard.cancel();
        tracker.wait().await;
    }
    println!(
        "completed = {}, cancelled = {}, in flight = {}",
        completed.load(Ordering::SeqCst),
        cancelled.load(Ordering::SeqCst),
        tracker.len()
    );
}
Output
draining with a 100 ms deadline
request 1 completed
request 3 completed
deadline passed; in flight: 1
request 2 cancelled at the deadline
completed = 2, cancelled = 1, in flight = 0

Requests 1 and 3 drained normally. Request 2 needed 500 ms, so the deadline cancelled it. The second tracker.wait() finishes quickly because the cancelled future is dropped at once.

Example 11: What abort does to half-finished work

Aborting drops a task at an .await, which can be between two related writes. This transfer debits one account, waits, and then credits the other. The program aborts one transfer and drains another, then prints the total money in the system.

Rust
use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio::time::sleep;

async fn transfer(accounts: Arc<Mutex<[i64; 2]>>, amount: i64) {
    accounts.lock().unwrap()[0] -= amount;
    sleep(Duration::from_millis(10)).await;
    accounts.lock().unwrap()[1] += amount;
}

fn report(label: &str, accounts: &Arc<Mutex<[i64; 2]>>) {
    let balances = *accounts.lock().unwrap();
    println!("{label}: {balances:?}, total {}", balances.iter().sum::<i64>());
}

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let accounts = Arc::new(Mutex::new([1000, 1000]));
    let handle = tokio::spawn(transfer(accounts.clone(), 100));
    sleep(Duration::from_millis(5)).await;
    handle.abort();
    let _ = handle.await;
    report("after abort", &accounts);

    let accounts = Arc::new(Mutex::new([1000, 1000]));
    let handle = tokio::spawn(transfer(accounts.clone(), 100));
    sleep(Duration::from_millis(5)).await;
    handle.await.unwrap();
    report("after drain", &accounts);
}
Output
after abort: [900, 1000], total 1900
after drain: [900, 1100], total 2000

The abort left the debit in place and skipped the credit, so 100 units vanished. Draining kept the total at 2000. Work with several steps that must stay together needs a drain, or steps that can be repeated safely after a restart (a transaction, or an operation that is safe to retry).

Abort Drain
Work that has not started never starts is either skipped (queue left unread) or run (queue read to the end)
Work in progress dropped at its next .await; code after it never runs finishes
Drop implementations run run
Time to exit short up to the longest in-flight job
Risk half-finished side effects never exiting, if a job hangs
Fits stateless work, work that can be repeated, the deadline phase work with side effects that must stay consistent

Example 12: Shutdown by dropping senders

A channel closes when all its Senders are dropped. recv then returns None, but only after every message already in the queue has been delivered. A consumer that loops on recv therefore drains the queue and exits without any extra signal.

Rust
use tokio::sync::mpsc;

#[tokio::main]
async fn main() {
    let (tx, mut rx) = mpsc::channel::<u32>(10);
    for job in 1..=3 {
        tx.send(job).await.unwrap();
    }
    let extra = tx.clone();
    extra.send(4).await.unwrap();

    drop(tx);
    println!("original sender dropped; is_closed = {}", extra.is_closed());
    drop(extra);

    let consumer = tokio::spawn(async move {
        let mut processed = Vec::new();
        while let Some(job) = rx.recv().await {
            processed.push(job);
        }
        processed
    });
    println!("consumer drained: {:?}", consumer.await.unwrap());
}
Output
original sender dropped; is_closed = false
consumer drained: [1, 2, 3, 4]

The channel closed only when the last sender was dropped. If any sender is kept alive, for example in a long-lived struct, recv waits forever, and the consumer never exits. When this design is used, find every owner of a sender before relying on it for shutdown. mpsc Channels covers the closing rules.

Example 13: A watch channel as a shutdown flag

A watch channel holds one value and lets any number of receivers wait for it to change. A watch::Sender<bool> set to true works as a shutdown flag with one difference from a token: a receiver created after the change still sees the current value. wait_for returns as soon as the predicate is true, including when the value is already true.

Rust
use tokio::sync::watch;
use tokio::time::{sleep, Duration};

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let (tx, rx) = watch::channel(false);

    let mut tasks = Vec::new();
    for id in 0..2 {
        let mut rx = rx.clone();
        tasks.push(tokio::spawn(async move {
            rx.wait_for(|stop| *stop).await.unwrap();
            id
        }));
    }

    sleep(Duration::from_millis(10)).await;
    tx.send(true).unwrap();
    for task in tasks {
        println!("task {}: shutdown seen", task.await.unwrap());
    }

    let mut late = tx.subscribe();
    late.wait_for(|stop| *stop).await.unwrap();
    println!("late subscriber: shutdown seen immediately");
}
Output
task 0: shutdown seen
task 1: shutdown seen
late subscriber: shutdown seen immediately

A watch value can also carry a phase such as Running, Draining, or Stopped, so tasks can react differently at each step. oneshot, broadcast, and watch covers the channel.

Example 14: A broadcast channel as a shutdown signal

A broadcast channel delivers each message to every receiver that exists when the message is sent. A receiver created afterwards does not get earlier messages. This suits a signal that only running tasks need to see.

Rust
use tokio::sync::broadcast;

#[tokio::main]
async fn main() {
    let (tx, _keep) = broadcast::channel::<()>(1);
    let mut early_a = tx.subscribe();
    let mut early_b = tx.subscribe();

    println!("receivers before send: {}", tx.receiver_count());
    tx.send(()).unwrap();

    println!("early_a got: {:?}", early_a.recv().await);
    println!("early_b got: {:?}", early_b.recv().await);

    let mut late = tx.subscribe();
    println!("late receiver: {:?}", late.try_recv());
}
Output
receivers before send: 3
early_a got: Ok(())
early_b got: Ok(())
late receiver: Err(Empty)

The late receiver got nothing, so a task that starts after the signal would never learn about it. The token (Example 3) and the watch flag (Example 13) do not have this gap. They are the usual choices when tasks can start at any time.

Mechanism Late listener sees the signal Carries a value Notes
CancellationToken yes no cannot be reset; supports child tokens
watch<bool> or watch<Phase> yes yes can move through several phases
broadcast<()> no yes receivers must exist before the send
dropped Sender yes (channel stays closed) no shuts down consumers after the queue is empty

Example 15: What happens to tasks when the Runtime is dropped

Dropping a Runtime stops it. Tasks that are still alive are dropped when they next yield, so their Drop code runs but their remaining .await code does not. The drop blocks until the tasks have been cleaned up. The program uses the runtime by hand (Your First Tokio Program introduces Runtime and block_on).

Rust
use tokio::runtime::Runtime;
use tokio::sync::oneshot;

struct Guard;

impl Drop for Guard {
    fn drop(&mut self) {
        println!("task: future dropped by runtime shutdown");
    }
}

fn main() {
    let rt = Runtime::new().unwrap();
    rt.block_on(async {
        let (started_tx, started_rx) = oneshot::channel();
        tokio::spawn(async move {
            let _guard = Guard;
            started_tx.send(()).unwrap();
            std::future::pending::<()>().await;
            println!("task: never printed");
        });
        started_rx.await.unwrap();
        println!("block_on: main future returns");
    });

    println!("dropping the runtime");
    drop(rt);
    println!("runtime dropped");
}
Output
block_on: main future returns
dropping the runtime
task: future dropped by runtime shutdown
runtime dropped

The spawned task never finished. Returning from block_on (or from #[tokio::main]'s main) does not wait for spawned tasks. Without a tracker or a join, the last steps of shutdown are left to the runtime drop, and that is an abort.

Example 16: shutdown_timeout and blocking threads

Runtime::shutdown_timeout(duration) stops the runtime and waits at most duration for the blocking pool. A closure started with spawn_blocking cannot be cancelled, so a long one delays a plain drop, which waits without a limit. shutdown_timeout returns after the duration and leaves the closure running on its thread. The example holds a blocking closure until the main thread releases it.

Rust
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{mpsc, Arc};
use std::thread;
use std::time::Duration;
use tokio::runtime::Runtime;

fn main() {
    let rt = Runtime::new().unwrap();
    let (release_tx, release_rx) = mpsc::channel::<()>();
    let (started_tx, started_rx) = mpsc::channel::<()>();
    let finished = Arc::new(AtomicBool::new(false));

    {
        let finished = finished.clone();
        rt.spawn_blocking(move || {
            started_tx.send(()).unwrap();
            release_rx.recv().unwrap();
            finished.store(true, Ordering::SeqCst);
        });
    }
    started_rx.recv().unwrap();

    rt.shutdown_timeout(Duration::from_millis(50));
    println!("shutdown_timeout returned");
    println!("blocking closure finished: {}", finished.load(Ordering::SeqCst));

    release_tx.send(()).unwrap();
    while !finished.load(Ordering::SeqCst) {
        thread::sleep(Duration::from_millis(5));
    }
    println!("blocking closure finished after release: true");
}
Output
shutdown_timeout returned
blocking closure finished: false
blocking closure finished after release: true

shutdown_timeout returned while the closure was still blocked. Runtime::shutdown_background does the same without any wait. Both leave the blocking thread running until the closure returns or the process exits, so a blocking closure that must stop needs its own stop flag or channel.

Way to end the runtime Waits for async tasks Waits for blocking closures
drop the Runtime drops them when they yield without a limit
shutdown_timeout(d) drops them when they yield up to d
shutdown_background() drops them when they yield not at all

Example 17: All steps together, with a hard limit

The last program combines the parts of a production shutdown: a soft token that ends accepting, a tracker that drains, a deadline, and a hard token that cancels what remains. Requests 1, 2, and 3 take 30, 60, and 400 ms and are queued before the server starts. Request 4 takes 5 ms and arrives at 10 ms. The signal arrives at 20 ms and the drain deadline is 100 ms after it.

Rust
use std::time::Duration;
use tokio::sync::mpsc;
use tokio::time::{sleep, timeout};
use tokio_util::sync::CancellationToken;
use tokio_util::task::TaskTracker;

async fn handle(id: u32, ms: u64, hard: CancellationToken) {
    let finished = hard.run_until_cancelled(sleep(Duration::from_millis(ms))).await.is_some();
    if finished {
        println!("  request {id}: finished");
    } else {
        println!("  request {id}: cancelled before it finished");
    }
}

#[tokio::main(flavor = "current_thread", start_paused = true)]
async fn main() {
    let soft = CancellationToken::new();
    let hard = CancellationToken::new();
    let tracker = TaskTracker::new();
    let (tx, mut incoming) = mpsc::channel::<(u32, u64)>(8);

    for request in [(1, 30), (2, 60), (3, 400)] {
        tx.send(request).await.unwrap();
    }

    let accept = {
        let (soft, hard, tracker) = (soft.clone(), hard.clone(), tracker.clone());
        tokio::spawn(async move {
            loop {
                tokio::select! {
                    biased;
                    _ = soft.cancelled() => break,
                    request = incoming.recv() => match request {
                        Some((id, ms)) => { tracker.spawn(handle(id, ms, hard.clone())); }
                        None => break,
                    },
                }
            }
            incoming.len()
        })
    };

    sleep(Duration::from_millis(10)).await;
    tx.send((4, 5)).await.unwrap();
    sleep(Duration::from_millis(10)).await;

    println!("signal: stop accepting");
    soft.cancel();
    let unread = accept.await.unwrap();
    println!("accept loop exited; requests never read: {unread}");
    println!("in flight: {}", tracker.len());

    tracker.close();
    let drained = timeout(Duration::from_millis(100), tracker.wait()).await.is_ok();
    println!("drained within 100 ms: {drained}");
    if !drained {
        println!("hard cancel; in flight: {}", tracker.len());
        hard.cancel();
        tracker.wait().await;
    }
    println!("exit; in flight: {}", tracker.len());
}
Output
  request 4: finished
signal: stop accepting
accept loop exited; requests never read: 0
in flight: 3
  request 1: finished
  request 2: finished
drained within 100 ms: false
hard cancel; in flight: 1
  request 3: cancelled before it finished
exit; in flight: 0

The output shows each class of work: requests that completed normally (1, 2, and 4), a request cancelled at the deadline (3), and the number of requests never read from the queue. In this program all queued requests were read before the signal, so the count is zero. A real server reports or persists that number, because those requests are lost unless the client retries.

Interview points

  • A graceful shutdown has five steps: signal, stop accepting, drain, deadline, exit.
  • A CancellationToken signals many tasks; it does not wait for them. A TaskTracker waits; it does not signal.
  • TaskTracker::wait completes only after close and when the tracker is empty. Dropping a tracker does not abort its tasks.
  • child_token cancels with the parent; cancelling the child does not cancel the parent.
  • DropGuard cancels its token when dropped, so every exit path of a function cancels its helpers.
  • run_until_cancelled drops the wrapped future at its current .await, and is cancel safe only if that future is.
  • Abort drops a task at an .await: Drop runs, later code does not. Work with several dependent steps can be left half done.
  • A channel closed by dropping all Senders still delivers queued messages before recv returns None. A forgotten sender keeps the consumer alive.
  • watch and CancellationToken are seen by tasks that start late. broadcast is not.
  • #[tokio::main] returning does not wait for spawned tasks. Dropping the runtime drops them at their next yield.
  • Blocking closures cannot be cancelled. shutdown_timeout bounds the wait and leaves them running.
  • A drain needs a deadline, because one hung job blocks the exit indefinitely.

Summary

  • Shutdown order: signal, stop accepting, drain, deadline, exit.
  • signal::ctrl_c and signal::unix::signal(SignalKind::terminate()) deliver the stop request.
  • CancellationToken: cancel, cancelled, is_cancelled, child_token, drop_guard, run_until_cancelled.
  • TaskTracker: spawn, close, wait, len; wait returns for a closed, empty tracker.
  • Two-phase pattern: soft token for accepting, timeout on wait, hard token for the tasks that remain.
  • Abort loses the code after the current .await; drain finishes it.
  • Dropping senders shuts consumers down after the queue is read to the end.
  • Runtime drop waits without limit for blocking closures; shutdown_timeout waits up to a limit; shutdown_background does not wait.