Debugging & Profiling in Production

A service is slow, stuck, crashing, or using too much memory, and it's happening in production. Every experienced engineer has a toolbox for this moment.

A Rust program is a native binary, so its toolbox is the native one: debuggers, core dumps, the Linux perf profiler, and a few Rust-specific tools for async code. They work on any native program, and are effective once the binary is built to support them. All output in this lesson comes from real runs on Linux with Rust 1.94.

Part 1: Building a debuggable binary

1. Debug info in release builds

All of the tools below need debug info to turn addresses into function names, files, and line numbers. The default release profile has none. Add at least line tables; the cost is binary size, not speed.

[profile.release]
debug = "line-tables-only"   # file:line in backtraces, profiles, and debuggers
# debug = true               # also variables and types, for debugging core dumps

Optimized code still loses information: variables live in registers and get reused, and inlined functions have no frame of their own. Debuggers show such variables as <optimized out> (example 6).

2. Ship small binaries, keep the symbols

Debug info makes binaries large. For the log-parsing program used later in this lesson, the release binary with line tables was 11.2 MB. You don't have to ship that: split the debug info into a separate file, deploy the stripped binary, and keep the debug file in your artifact storage, keyed by the build ID.

objcopy --only-keep-debug target/release/app app.debug
objcopy --strip-debug --add-gnu-debuglink=app.debug target/release/app app.stripped
ls -l app.debug app.stripped

-rw-r--r-- 1 root root 8995992 Sep 27 22:58 app.debug
-rwxr-xr-x 1 root root 2722408 Sep 27 22:58 app.stripped

The deployed binary is 2.7 MB. When you need to debug a core dump from production, gdb finds app.debug through the debug link (or you point it there), and you get full file and line information. Cargo can also do the split for you with split-debuginfo = "packed" in the profile, and symbol servers such as debuginfod automate the lookup.

3. Backtraces from panics

For a panic, the cheapest investigation tool is the backtrace printed by the default panic hook. Set RUST_BACKTRACE=1 in the service's environment, so the first crash in production comes with a stack trace instead of just a message. With debug = true, a release build panicking on an out-of-bounds index prints:

thread 'main' (1026) panicked at src/main.rs:7:16:
index out of bounds: the len is 1 but the index is 1
stack backtrace:
   0: __rustc::rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::panicking::panic_bounds_check
   ...
   6: core1::apply_discount
             at ./src/main.rs:7:16
   7: core1::main::{{closure}}
             at ./src/main.rs:13:41
   ...
  13: core1::main
             at ./src/main.rs:13:67
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

The frames above your own code are the panic machinery; read from the first frame in your crate downward. RUST_BACKTRACE=full also shows the frames the standard library normally hides, which is occasionally needed for panics inside library code.

Part 2: Debuggers

4. rust-gdb and rust-lldb: stop at a panic and look around

rust-gdb and rust-lldb are wrappers installed with the toolchain that load pretty-printers for Rust types, so a Vec, String, HashMap, or Option prints as its contents rather than as raw pointers. A breakpoint on core::panicking::panic_fmt stops the program at the moment any panic starts, with the whole stack intact:

cargo build
rust-gdb -batch \
  -ex "break core::panicking::panic_fmt" -ex run \
  -ex "frame function core1::apply_discount" -ex "info args" -ex "print *order" \
  -ex "frame function core1::main" -ex "info locals" \
  target/debug/core1

Breakpoint 1, core::panicking::panic_fmt () at library/core/src/panicking.rs:72
#5  0x000055555556a1fa in core1::apply_discount (order=0x5555555afd60, position=1) at src/main.rs:7
7	    order.items[position] / 10 * 9
order = 0x5555555afd60
position = 1
$1 = core1::Order {id: 1, items: Vec(size=1) = {100}}
#12 0x000055555556a35c in core1::main () at src/main.rs:13
13	        let total: u32 = (0..2).map(|p| apply_discount(order, p)).sum();
order = 0x5555555afd60
orders = Vec(size=3) = {core1::Order {id: 1, items: Vec(size=1) = {100}}, core1::Order {id: 2, items: Vec(size=2) = {200, 200}}, core1::Order {id: 3, items: Vec(size=3) = {300, 300, 300}}}

The program being debugged:

struct Order {
    id: u64,
    items: Vec<u32>,
}

fn apply_discount(order: &Order, position: usize) -> u32 {
    order.items[position] / 10 * 9
}

fn main() {
    let orders: Vec<Order> = (1..=3).map(|id| Order { id, items: vec![100 * id as u32; id as usize] }).collect();
    for order in &orders {
        let total: u32 = (0..2).map(|p| apply_discount(order, p)).sum();
        println!("order {} total {}", order.id, total);
    }
}

thread 'main' panicked at src/main.rs:7:16:
index out of bounds: the len is 1 but the index is 1

The debugger shows what the panic message can't: which order (id 1, with a single item) and which caller asked for position 1. Interactive debugging in an IDE works the same way: VS Code with the CodeLLDB extension and RustRover both use these debuggers and pretty-printers underneath.

5. Attaching to a stuck process

When a service stops responding, attach a debugger to the running process and print every thread's stack. It's the fastest way to diagnose a deadlock. The service below has one: two transfers lock the same two accounts in opposite order.

use std::sync::{Arc, Mutex};
use std::thread;
use std::time::Duration;

fn transfer(from: &Mutex<i64>, to: &Mutex<i64>, amount: i64) {
    let mut a = from.lock().unwrap();
    thread::sleep(Duration::from_millis(50)); // widen the window
    let mut b = to.lock().unwrap();
    *a -= amount;
    *b += amount;
}

fn main() {
    let alice = Arc::new(Mutex::new(100));
    let bob = Arc::new(Mutex::new(100));
    let (a1, b1) = (Arc::clone(&alice), Arc::clone(&bob));
    let t1 = thread::Builder::new().name("pay-alice-to-bob".into()).spawn(move || transfer(&a1, &b1, 10)).unwrap();
    let (a2, b2) = (Arc::clone(&alice), Arc::clone(&bob));
    let t2 = thread::Builder::new().name("pay-bob-to-alice".into()).spawn(move || transfer(&b2, &a2, 5)).unwrap();
    t1.join().unwrap();
    t2.join().unwrap();
    println!("done");
}

The process hangs. Attaching gdb to it (release build with line tables; long standard-library frames trimmed):

gdb -p $(pidof app) -batch -ex "thread apply all bt"

Thread 3 (Thread 0x7fbbaddff6c0 (LWP 955) "pay-alice-to-bo"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000557ba6497486 in std::sys::pal::unix::futex::futex_wait () at library/std/src/sys/pal/unix/futex.rs:73
#2  std::sys::sync::mutex::futex::Mutex::lock_contended () at library/std/src/sys/sync/mutex/futex.rs:61
#3  0x0000557ba64856b7 in lock () at .../library/std/src/sys/sync/mutex/futex.rs:33
#4  lock<i64> () at .../library/std/src/sync/poison/mutex.rs:492
#5  transfer () at src/main.rs:8
#6  0x0000557ba64859c9 in {closure#0} () at src/main.rs:17
...

Thread 2 (Thread 0x7fbbadbfe6c0 (LWP 956) "pay-bob-to-alic"):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x0000557ba6497486 in std::sys::pal::unix::futex::futex_wait () at library/std/src/sys/pal/unix/futex.rs:73
#2  std::sys::sync::mutex::futex::Mutex::lock_contended () at library/std/src/sys/sync/mutex/futex.rs:61
#3  0x0000557ba64856b7 in lock () at .../library/std/src/sys/sync/mutex/futex.rs:33
#4  lock<i64> () at .../library/std/src/sync/poison/mutex.rs:492
#5  transfer () at src/main.rs:8
#6  0x0000557ba6485a39 in {closure#1} () at src/main.rs:19
...

Thread 1 (Thread 0x7fbbae1a7780 (LWP 953) "dbg"):
...
#4  0x0000557ba6497c81 in std::sys::thread::unix::Thread::join () at library/std/src/sys/thread/unix.rs:138
...
#7  main () at src/main.rs:20

Both worker threads are in Mutex::lock_contended at src/main.rs:8, the second lock(), while holding the first: a classic lock-order deadlock. main is waiting in join. Two details help in real incidents: the thread names from thread::Builder::name appear in the output (truncated to 15 characters by Linux), so name your threads; and gdb pauses the process only while it's attached, so this is safe to do on a live but already-stuck service.

The fix is a consistent lock order, for example by account id:

use std::sync::{Arc, Mutex};
use std::thread;

struct Account {
    id: u32,
    balance: Mutex<i64>,
}

fn transfer(from: &Account, to: &Account, amount: i64) {
    // Always lock the lower id first, whatever the direction of the transfer.
    let (first, second) = if from.id < to.id { (from, to) } else { (to, from) };
    let mut a = first.balance.lock().unwrap();
    let mut b = second.balance.lock().unwrap();
    let (from_bal, to_bal) = if from.id < to.id { (&mut *a, &mut *b) } else { (&mut *b, &mut *a) };
    *from_bal -= amount;
    *to_bal += amount;
}

fn main() {
    let alice = Arc::new(Account { id: 1, balance: Mutex::new(100) });
    let bob = Arc::new(Account { id: 2, balance: Mutex::new(100) });
    let handles: Vec<_> = (0..100)
        .map(|i| {
            let (alice, bob) = (Arc::clone(&alice), Arc::clone(&bob));
            thread::spawn(move || if i % 2 == 0 { transfer(&alice, &bob, 10) } else { transfer(&bob, &alice, 5) })
        })
        .collect();
    for h in handles {
        h.join().unwrap();
    }
    println!("alice {} bob {}", alice.balance.lock().unwrap(), bob.balance.lock().unwrap());
}

alice -150 bob 350

6. Core dumps: debugging a crash after the fact

A core dump is a snapshot of the process's memory at the moment it died: the native equivalent of a heap dump plus thread dump. Rust processes produce one when they abort: with panic = "abort", on a panic during a panic, on stack overflow, or on a segfault in unsafe or foreign code. Enable them where the service runs:

ulimit -c unlimited                         # per shell or service unit (LimitCORE=infinity in systemd)
cat /proc/sys/kernel/core_pattern           # where cores go; systemd-coredump, a path, or a pipe

The same order program, built with panic = "abort" and debug = true, crashes and dumps core:

thread 'main' (1026) panicked at src/main.rs:7:16:
index out of bounds: the len is 1 but the index is 1
...
Aborted                 (core dumped) ./target/release/core1

Load the binary and the core into gdb:

gdb -batch -ex bt target/release/core1 core

Core was generated by `./target/release/core1'.
Program terminated with signal SIGABRT, Aborted.
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
...
#4  0x00007fb4caa288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x000055b3f263125a in std::sys::pal::unix::abort_internal () at library/std/src/sys/pal/unix/mod.rs:305
...
#15 0x000055b3f2660abc in core::panicking::panic_bounds_check () at library/core/src/panicking.rs:271
...
#19 core1::apply_discount (order=<optimized out>, position=<optimized out>) at src/main.rs:7
#20 core1::main::{closure#1} (p=<optimized out>) at src/main.rs:13
...
#26 core1::main () at src/main.rs:13

Notice order=<optimized out>: in an optimized build, the arguments were kept in registers that were reused by the time of the crash. That's the practical limit of post-mortem debugging of release builds. The stack and line numbers are reliable; individual variables often aren't. When you need values, log them (next lesson) or reproduce with a debug build.

On systems with systemd-coredump, coredumpctl list and coredumpctl gdb PID find and open cores. In Kubernetes, cores are written inside the container unless the node's core_pattern pipes them elsewhere, so set this up before you need it.

Part 3: Profiling

7. CPU profiling with perf

perf samples the program's stack thousands of times per second and counts where it was. It works on any native program, needs no code changes, and can attach to a running production process. This log-processing program takes 27 seconds for 200,000 lines, far too long:

use regex::Regex;
use std::collections::HashMap;

fn status_of(line: &str) -> Option<u16> {
    let re = Regex::new(r#"" (\d{3}) "#).unwrap(); // compiled on every call
    re.captures(line)?.get(1)?.as_str().parse().ok()
}

fn main() {
    let lines: Vec<String> = (0..2_000)
        .map(|i| format!(r#"10.0.0.{} - - [27/Sep/2026] "GET /api/{} HTTP/1.1" {} 512"#, i % 255, i % 97, [200, 404, 500][i % 3]))
        .collect();
    let mut counts: HashMap<u16, usize> = HashMap::new();
    for line in &lines {
        if let Some(status) = status_of(line) {
            *counts.entry(status).or_default() += 1;
        }
    }
    let mut v: Vec<_> = counts.into_iter().collect();
    v.sort();
    println!("{v:?}");
}

[(200, 667), (404, 667), (500, 666)]

(The example above processes 2,000 lines so it runs quickly here; the profile below is from the same code with 200,000 lines, built as a binary named slow.) Record a profile and look at where the time is spent, including time in callees (--children):

RUSTFLAGS="-C force-frame-pointers=yes" cargo build --release   # see example 8
perf record -g ./target/release/slow
perf report --children --sort symbol

100.00%  0.00%  [.] main
 99.98%  0.07%  [.] slow::main
 88.53%  0.01%  [.] regex::regex::string::Regex::new
 88.42%  0.05%  [.] regex::builders::string::RegexBuilder::build
 88.28%  0.05%  [.] regex_automata::meta::regex::Builder::build
 84.39%  0.17%  [.] regex_automata::meta::strategy::new
  ...
  5.69%  0.03%  [.] regex_automata::hybrid::regex::Regex::try_search

The first column is inclusive time, the second is time in the function itself. 88.5% of the run is spent in Regex::new, compiling the same pattern 200,000 times; the actual matching (try_search) takes under 6%. Compiling the regex once fixes it:

use regex::Regex;
use std::collections::HashMap;
use std::sync::LazyLock;

static STATUS: LazyLock<Regex> = LazyLock::new(|| Regex::new(r#"" (\d{3}) "#).unwrap());

fn status_of(line: &str) -> Option<u16> {
    STATUS.captures(line)?.get(1)?.as_str().parse().ok()
}

fn main() {
    let lines: Vec<String> = (0..2_000)
        .map(|i| format!(r#"10.0.0.{} - - [27/Sep/2026] "GET /api/{} HTTP/1.1" {} 512"#, i % 255, i % 97, [200, 404, 500][i % 3]))
        .collect();
    let mut counts: HashMap<u16, usize> = HashMap::new();
    for line in &lines {
        if let Some(status) = status_of(line) {
            *counts.entry(status).or_default() += 1;
        }
    }
    let mut v: Vec<_> = counts.into_iter().collect();
    v.sort();
    println!("{v:?}");
}

[(200, 667), (404, 667), (500, 666)]

With 200,000 lines, the run time dropped from 27.0 seconds to 0.07 seconds. The profile made the cause obvious in one command; guessing would probably have focused on the HashMap or the string formatting.

8. Frame pointers make profiles trustworthy

To attribute a sample to its callers, the profiler has to walk the stack. Optimized Rust code omits frame pointers by default, so perf must either reconstruct stacks from DWARF debug info (--call-graph dwarf) or get them wrong. For the program above, the DWARF method copied only part of each stack, and 83% of samples ended in an unknown frame (0xffffffffffffffff), which made the caller view useless. Rebuilding with frame pointers produced the clean profile shown in example 7:

RUSTFLAGS="-C force-frame-pointers=yes" cargo build --release
perf record -g ./target/release/app        # plain -g: walks frame pointers, cheap and complete

Frame pointers cost roughly 1–2% of performance, which many companies (and several Linux distributions) now accept for always-profilable production builds. Set it for release builds in .cargo/config.toml with rustflags = ["-C", "force-frame-pointers=yes"].

9. Flamegraphs and friendlier profilers

A flamegraph turns thousands of stacks into one picture: each box is a function, its width is the share of samples it appears in, and callees sit on top of callers. Wide boxes near the top are where the time goes.

cargo install flamegraph
cargo flamegraph --bin app            # runs perf and writes flamegraph.svg

cargo install samply
samply record ./target/release/app    # opens the Firefox Profiler UI in the browser

Profiling a live production process with perf for 30 seconds typically costs a few percent of CPU while it runs, and nothing before or after. Continuous profilers make this the default.

Part 4: Memory

10. Finding leaks with valgrind

valgrind --leak-check=full runs the program on a virtual CPU and reports every heap block that's unreachable at exit, with the stack of the allocation. It's slow (often 20–50× slower), so use it on tests or a reproduction, not in production. This program leaks through an Rc cycle (Memory Without a Garbage Collector):

use std::cell::RefCell;
use std::rc::Rc;

struct Session {
    user: String,
    peer: RefCell<Option<Rc<Session>>>,
}

fn pair_sessions(a: &str, b: &str) {
    let first = Rc::new(Session { user: a.to_string(), peer: RefCell::new(None) });
    let second = Rc::new(Session { user: b.to_string(), peer: RefCell::new(Some(Rc::clone(&first))) });
    *first.peer.borrow_mut() = Some(Rc::clone(&second)); // a cycle: neither is ever freed
    println!("paired {} with {}", first.user, second.user);
}

fn main() {
    for i in 0..2 {
        pair_sessions(&format!("user-{i}"), &format!("agent-{i}"));
    }
}

paired user-0 with agent-0
paired user-1 with agent-1

With 100 pairs instead of 2, valgrind --leak-check=full ./target/debug/app reports (stack trimmed):

HEAP SUMMARY:
    in use at exit: 13,224 bytes in 401 blocks
  total heap usage: 610 allocs, 209 frees, 18,524 bytes allocated
12,680 (5,600 direct, 7,080 indirect) bytes in 100 blocks are definitely lost in loss record 5 of 5
   at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
   ...
   by 0x11EE80: alloc::rc::Rc<T>::new (library/alloc/src/rc.rs:420)
   by 0x11E698: leak::pair_sessions (main.rs:10)
   by 0x11EC2C: leak::main (main.rs:18)
   ...
LEAK SUMMARY:
   definitely lost: 5,600 bytes in 100 blocks
   indirectly lost: 7,080 bytes in 300 blocks
     possibly lost: 0 bytes in 0 blocks
   still reachable: 544 bytes in 1 blocks

"Definitely lost" points at pair_sessions, line 10, the Rc::new of the first session; the "indirectly lost" blocks are the second session and the strings reachable only through the cycle. "Still reachable" is memory the standard library keeps for the whole program (the stdout buffer) and isn't a leak.

11. Heap profiling: where is the memory?

A leak detector finds memory that's lost. More often the question is "why is the service using 3 GB?", where the memory is reachable but larger than expected. Heap profilers record every allocation with its stack and show which call sites hold the most memory over time.

For a production service, jemalloc's sampling profiler (or a continuous profiler with memory support) is the closest thing to taking a heap dump: you can trigger a profile dump from a running process and compare two dumps to see what grew.

Part 5: Async and I/O

12. Tokio task dumps: what is every task waiting for?

A thread dump of a stuck async service is unhelpful: the worker threads are idle, parked in epoll_wait, while the tasks are stuck. Tokio can dump the async stack of every task instead. It requires unstable flags and the taskdump feature, and works on Linux (x86-64 and AArch64):

[dependencies]
tokio = { version = "1", features = ["full", "taskdump"] }

use std::time::Duration;
use tokio::sync::{Mutex, oneshot};

async fn wait_for_reply(rx: oneshot::Receiver<u32>) -> u32 {
    rx.await.unwrap_or(0) // nobody will ever send
}

async fn update_cache(lock: &Mutex<Vec<u32>>) {
    let _guard = lock.lock().await; // waits for a lock that is never released
}

#[tokio::main]
async fn main() {
    let (_tx, rx) = oneshot::channel::<u32>();
    tokio::spawn(wait_for_reply(rx));

    let lock: &'static Mutex<Vec<u32>> = Box::leak(Box::new(Mutex::new(Vec::new())));
    let held = lock.lock().await;
    tokio::spawn(update_cache(lock));

    tokio::time::sleep(Duration::from_millis(100)).await;
    // In a service: trigger this from an admin endpoint or a signal handler.
    let dump = tokio::runtime::Handle::current().dump().await;
    for (i, task) in dump.tasks().iter().enumerate() {
        println!("---- task {i} ----\n{}", task.trace());
    }
    drop(held);
}

RUSTFLAGS="--cfg tokio_unstable --cfg tokio_taskdump" cargo run

---- task 0 ----
╼ td::wait_for_reply at src/main.rs:5:8
  └╼ <tokio::sync::oneshot::Receiver<T> as core::future::future::Future> at .../tokio-1.53.1/src/sync/oneshot.rs:1287:36
     └╼ tokio::sync::oneshot::Inner<T> at .../tokio-1.53.1/src/sync/oneshot.rs:1318:16
---- task 1 ----
╼ td::update_cache at src/main.rs:9:30
  └╼ tokio::sync::mutex::Mutex<T>::lock at .../tokio-1.53.1/src/sync/mutex.rs:455:33
     └╼ tokio::sync::mutex::Mutex<T>::lock::{{closure}} at .../tokio-1.53.1/src/sync/mutex.rs:436:28
        └╼ tokio::sync::mutex::Mutex<T>::acquire at .../tokio-1.53.1/src/sync/mutex.rs:658:27
           └╼ <tokio::sync::batch_semaphore::Acquire as core::future::future::Future> at .../tokio-1.53.1/src/sync/batch_semaphore.rs:579:16

Each task shows the .await it's suspended at: task 0 waits on a oneshot receiver at line 5, task 1 on a mutex at line 9. That's the async equivalent of the deadlock thread dump in example 5.

13. tokio-console: a live view of tasks

tokio-console is an interactive terminal UI that connects to a running service and lists every task with its state, total busy time, idle time, number of polls, and wakeups. It flags tasks that have been busy for a long time in a single poll (blocking the runtime, see Async Pitfalls in Production) and tasks that were never woken again (lost wakeups).

[dependencies]
console-subscriber = "0.5"
tokio = { version = "1", features = ["full", "tracing"] }

#[tokio::main]
async fn main() {
    console_subscriber::init(); // listens on 127.0.0.1:6669 by default
    run_service().await;
}

RUSTFLAGS="--cfg tokio_unstable" cargo run
cargo install --locked tokio-console
tokio-console                      # connects to localhost:6669

The instrumentation has a cost, so it's usually enabled in staging, or behind a feature flag and a configuration switch in production.

14. strace: what the program asks the operating system

Many production problems are about the environment rather than the code: a config file in the wrong place, a DNS lookup that times out, a permission error hidden behind a generic message. strace prints the program's system calls. This program throws away the reason for its failure:

use std::fs;

fn load_settings() -> Option<String> {
    ["/etc/myapp/settings.toml", "settings.toml", "config/settings.toml"]
        .iter()
        .find_map(|p| fs::read_to_string(p).ok()) // the error details are thrown away
}

fn main() {
    match load_settings() {
        Some(s) => println!("loaded {} bytes", s.len()),
        None => println!("error: no settings found"),
    }
}

error: no settings found

strace shows which paths it tried, relative to which working directory, and why each failed:

strace -f -e trace=openat ./target/debug/app

openat(AT_FDCWD, "/etc/myapp/settings.toml", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "settings.toml", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "config/settings.toml", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Other useful filters: -e trace=network (connections and DNS), -c (a summary of time per system call, which exposes surprising futex or write counts), and -p PID to attach to a running process. And, of course, the better fix is an error message that includes what strace had to reveal:

use std::fs;
use std::io;

fn load_settings() -> Result<String, String> {
    let candidates = ["/etc/myapp/settings.toml", "settings.toml", "config/settings.toml"];
    let mut tried = Vec::new();
    for path in candidates {
        match fs::read_to_string(path) {
            Ok(s) => return Ok(s),
            Err(e) if e.kind() == io::ErrorKind::NotFound => tried.push(path),
            Err(e) => return Err(format!("{path}: {e}")), // e.g. permission denied: report it
        }
    }
    Err(format!("no settings file found; tried {tried:?}"))
}

fn main() {
    match load_settings() {
        Ok(s) => println!("loaded {} bytes", s.len()),
        Err(e) => println!("error: {e}"),
    }
}

error: no settings file found; tried ["/etc/myapp/settings.toml", "settings.toml", "config/settings.toml"]

For relative paths, including std::env::current_dir() in the message as well removes the last bit of guesswork.

15. A debugging playbook

Summary

• Build release binaries with at least debug = "line-tables-only"; ship stripped binaries and keep the debug files.
• Panic backtraces (RUST_BACKTRACE=1) solve most crashes. rust-gdb and rust-lldb add Rust pretty-printers; a breakpoint on core::panicking::panic_fmt stops at any panic.
• gdb -p PID -batch -ex "thread apply all bt" finds deadlocks in a live process. Name your threads.
• Core dumps capture aborting processes for later analysis; expect <optimized out> variables in release builds.
• perf, flamegraphs, and samply show where CPU time goes; build with frame pointers for reliable stacks.
• valgrind finds leaks in reproductions; heaptrack, bytehound, dhat, and jemalloc profiling explain memory growth.
• For async services, Tokio task dumps and tokio-console show what each task is waiting on. strace shows what the process asks the operating system.

Next: Observability: structured logs, spans, metrics, and distributed tracing with tracing and OpenTelemetry.