Lesson 89 of 94
Testing, Fuzzing & Benchmarking
Testing covered Rust's built-in test framework: #[test], assertions, fakes, integration tests, and doc tests. That's enough for most code. Senior engineers also own the testing strategy: which bugs example-based tests miss, how to find inputs nobody thought of, how to prove a change made something faster, and how to keep a large test suite fast and trustworthy.
This lesson covers the tools for that:
| Goal | Rust |
|---|---|
| Generate inputs and check properties | proptest |
| Find crashes with random or mutated inputs | cargo fuzz (libFuzzer), arbitrary |
| Microbenchmarks | criterion, divan |
| Snapshot tests | insta |
| Mocks | mockall (or hand-written fakes) |
| Coverage | cargo llvm-cov |
| Mutation testing | cargo mutants |
| Faster, isolated test runs | cargo nextest |
Part 1: Property-based testing
1. A property instead of examples
An example-based test checks a handful of inputs you chose. A property says something that must hold for every input, and the framework generates hundreds of inputs to try to break it. proptest is the standard Rust library for this.
Here the property is that decoding an encoded string gives back the original:
/// Run-length encoding: "aaab" -> [('a', 3), ('b', 1)].
pub fn encode(s: &str) -> Vec<(char, usize)> {
let mut out: Vec<(char, usize)> = Vec::new();
for c in s.chars() {
match out.last_mut() {
Some((last, n)) if *last == c => *n += 1,
_ => out.push((c, 1)),
}
}
out
}
pub fn decode(runs: &[(char, usize)]) -> String {
runs.iter().flat_map(|&(c, n)| std::iter::repeat_n(c, n)).collect()
}
fn main() {}
#[cfg(test)]
mod tests {
use super::*;
use proptest::prelude::*;
#[test]
fn example() {
assert_eq!(encode("aaab"), vec![('a', 3), ('b', 1)]);
}
proptest! {
#[test]
fn decode_inverts_encode(s in ".*") {
prop_assert_eq!(decode(&encode(&s)), s);
}
#[test]
fn runs_never_repeat_a_char(s in "[abc]{0,50}") {
let runs = encode(&s);
for pair in runs.windows(2) {
prop_assert_ne!(pair[0].0, pair[1].0);
}
}
}
}running 3 tests
test tests::decode_inverts_encode ... ok
test tests::example ... ok
test tests::runs_never_repeat_a_char ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01sEach proptest! test runs 256 generated cases by default. ".*" is a strategy: a regular expression that generates arbitrary strings, including empty ones and non-ASCII text. The second property uses a narrow alphabet on purpose, so that runs of repeated characters are common.
2. Shrinking: from a random failure to a minimal one
The real power of property testing is what happens when a property fails. proptest doesn't just report the random input that broke it: it shrinks that input step by step until it finds the simplest one that still fails. This helper, meant to shorten strings for log lines, has a bug that example tests with ASCII input never find:
/// Shortens `s` to at most `max` bytes for a log line.
pub fn truncate(s: &str, max: usize) -> &str {
if s.len() <= max { s } else { &s[..max] } // BUG: can cut a UTF-8 character in half
}
fn main() {}
#[cfg(test)]
mod tests {
use super::*;
use proptest::prelude::*;
#[test]
fn ascii_examples_pass() {
assert_eq!(truncate("hello", 3), "hel");
assert_eq!(truncate("hi", 10), "hi");
}
proptest! {
#![proptest_config(ProptestConfig { failure_persistence: None, ..ProptestConfig::default() })]
#[test]
fn truncate_is_a_prefix_within_max(s in any::<String>(), max in 0usize..20) {
let t = truncate(&s, max);
prop_assert!(t.len() <= max);
prop_assert!(s.starts_with(t));
}
}
}The generated inputs are random, so the exact report differs between runs, but it always ends like this:
test tests::truncate_is_a_prefix_within_max ... FAILED
...
Test failed: byte index 1 is not a char boundary; it is inside 'Σ' (bytes 0..2) of `Σ`.
minimal failing input: s = "Σ", max = 1Proptest first found some long random string that panicked, then shrunk it to a single two-byte character and max = 1. That minimal case explains the bug by itself. The fix backs up to the nearest character boundary:
pub fn truncate(s: &str, max: usize) -> &str {
if s.len() <= max {
return s;
}
let mut end = max;
while !s.is_char_boundary(end) {
end -= 1; // back up to the start of the character
}
&s[..end]
}
fn main() {}
#[cfg(test)]
mod tests {
use super::*;
use proptest::prelude::*;
proptest! {
#[test]
fn truncate_is_a_prefix_within_max(s in any::<String>(), max in 0usize..20) {
let t = truncate(&s, max);
prop_assert!(t.len() <= max);
prop_assert!(s.starts_with(t));
}
}
}running 1 test
test tests::truncate_is_a_prefix_within_max ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02sBy default, proptest saves failing inputs in a proptest-regressions/ directory next to the source file and replays them first on every run. Commit that directory: it turns every bug proptest ever found into a permanent regression test. (The example above turns persistence off only because it's a demonstration.)
3. Strategies: generating the inputs you need
A strategy describes a space of values. Proptest has strategies for primitives, ranges, regexes, collections, and combinators to build domain types from them.
use proptest::prelude::*;
#[derive(Debug, Clone, PartialEq)]
enum Currency {
Eur,
Usd,
Inr,
}
#[derive(Debug, Clone)]
struct Order {
sku: String,
quantity: u32,
unit_price_cents: u64,
currency: Currency,
}
impl Order {
fn total_cents(&self) -> Option<u64> {
self.unit_price_cents.checked_mul(self.quantity as u64)
}
}
fn currency() -> impl Strategy<Value = Currency> {
prop_oneof![Just(Currency::Eur), Just(Currency::Usd), Just(Currency::Inr)]
}
fn order() -> impl Strategy<Value = Order> {
("[A-Z]{3}-[0-9]{4}", 1u32..1_000, 0u64..1_000_000, currency())
.prop_map(|(sku, quantity, unit_price_cents, currency)| Order { sku, quantity, unit_price_cents, currency })
}
fn main() {}
proptest! {
#[test]
fn totals_never_overflow_for_realistic_orders(o in order()) {
prop_assert!(o.total_cents().is_some());
prop_assert_eq!(o.sku.len(), 8);
}
#[test]
fn a_basket_total_is_the_sum_of_its_orders(orders in prop::collection::vec(order(), 0..20)) {
let total: u64 = orders.iter().map(|o| o.total_cents().unwrap()).sum();
let by_currency: u64 = [Currency::Eur, Currency::Usd, Currency::Inr]
.iter()
.map(|c| orders.iter().filter(|o| &o.currency == c).map(|o| o.total_cents().unwrap()).sum::<u64>())
.sum();
prop_assert_eq!(total, by_currency);
}
}running 2 tests
test a_basket_total_is_the_sum_of_its_orders ... ok
test totals_never_overflow_for_realistic_orders ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s| Strategy | Generates |
|---|---|
any::<T>() |
any value of T (for types implementing Arbitrary) |
0..100u32, 1.0..2.0f64 |
numbers in a range |
"[a-z]{1,8}" |
strings matching a regex |
prop::collection::vec(s, 0..20) |
vectors whose elements come from s (also hash_map, btree_set, ...) |
(s1, s2, s3) |
tuples |
s.prop_map(f) |
values transformed by f |
s.prop_filter("why", pred) |
values satisfying pred (use sparingly: rejected values are wasted) |
prop_oneof![a, b, c] |
a choice between strategies |
Just(v) |
always v |
With the proptest-derive crate, #[derive(Arbitrary)] generates a strategy for your own structs and enums.
4. Kinds of properties worth testing
Coming up with properties is the skill. A handful of patterns cover most code:
| Pattern | Property | Example |
|---|---|---|
| Round trip | decode(encode(x)) == x |
serialization, compression, parsing and printing |
| Invariant | output always satisfies a rule | sorted output, balanced tree, total never negative |
| Oracle / model | fast implementation agrees with a simple one | a custom cache vs HashMap, SIMD vs scalar code |
| Idempotence | f(f(x)) == f(x) |
normalization, formatting, deduplication |
| Commutativity / order independence | result doesn't depend on input order | merging, aggregation, set operations |
| No crash | the function returns for every input | parsers, validators given untrusted input |
| Metamorphic | a known change to the input causes a known change to the output | adding an item raises the total by its price |
5. Model-based testing: an oracle for a data structure
For data structures and stateful components, generate a sequence of operations, apply it both to your implementation and to a simple model known to be correct, and check they agree after every step. Here a small sorted-vector set is checked against BTreeSet:
use proptest::prelude::*;
use std::collections::BTreeSet;
/// A set stored as a sorted Vec: fast iteration, compact memory.
#[derive(Default)]
pub struct SortedVecSet {
items: Vec<i32>,
}
impl SortedVecSet {
pub fn insert(&mut self, x: i32) -> bool {
match self.items.binary_search(&x) {
Ok(_) => false,
Err(pos) => {
self.items.insert(pos, x);
true
}
}
}
pub fn remove(&mut self, x: i32) -> bool {
match self.items.binary_search(&x) {
Ok(pos) => {
self.items.remove(pos);
true
}
Err(_) => false,
}
}
pub fn contains(&self, x: i32) -> bool {
self.items.binary_search(&x).is_ok()
}
}
#[derive(Debug, Clone)]
enum Op {
Insert(i32),
Remove(i32),
Contains(i32),
}
fn op() -> impl Strategy<Value = Op> {
// Small values, so that inserts, removes, and lookups hit the same elements often.
let v = -10i32..10;
prop_oneof![v.clone().prop_map(Op::Insert), v.clone().prop_map(Op::Remove), v.prop_map(Op::Contains)]
}
fn main() {}
proptest! {
#[test]
fn behaves_like_btreeset(ops in prop::collection::vec(op(), 0..100)) {
let mut real = SortedVecSet::default();
let mut model = BTreeSet::new();
for op in ops {
match op {
Op::Insert(x) => prop_assert_eq!(real.insert(x), model.insert(x)),
Op::Remove(x) => prop_assert_eq!(real.remove(x), model.remove(&x)),
Op::Contains(x) => prop_assert_eq!(real.contains(x), model.contains(&x)),
}
prop_assert!(real.items.iter().copied().eq(model.iter().copied()));
}
}
}running 1 test
test behaves_like_btreeset ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04sWhen such a test fails, shrinking reduces the operation list to the shortest sequence that reproduces the bug, which reads like a minimal bug report. The same technique scales up to caches, connection pools, and state machines.
6. Configuring runs
use proptest::prelude::*;
fn slugify(title: &str) -> String {
let mut slug = String::new();
for word in title.split(|c: char| !c.is_ascii_alphanumeric()).filter(|w| !w.is_empty()) {
if !slug.is_empty() {
slug.push('-');
}
slug.push_str(&word.to_ascii_lowercase());
}
slug
}
fn main() {}
proptest! {
// More cases for an important property; the default is 256.
#![proptest_config(ProptestConfig::with_cases(2_000))]
#[test]
fn slugify_is_idempotent(title in "\\PC{0,40}") {
let once = slugify(&title);
prop_assert_eq!(slugify(&once), once.clone());
prop_assert!(once.chars().all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-'));
}
}running 1 test
test slugify_is_idempotent ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s\PC means "any printable Unicode character". The PROPTEST_CASES environment variable overrides the number of cases without changing code, which is handy for a nightly CI job that runs 100,000 cases.
Part 2: Fuzzing
7. Fuzzing: coverage-guided input search
A property test generates inputs from a strategy you wrote. A fuzzer starts from a few inputs (or none) and mutates bytes, keeping mutations that reach new code paths. Given minutes or hours, it finds inputs no human would write: malformed length fields, deeply nested structures, invalid UTF-8 in exactly the wrong place. Fuzzing is standard practice for anything that parses untrusted data, and essential for unsafe code.
cargo fuzz runs LLVM's libFuzzer. It needs a nightly toolchain because it compiles with sanitizer instrumentation.
cargo install cargo-fuzz
cargo fuzz init # creates fuzz/ with its own Cargo.toml
cargo fuzz add parse_header # creates fuzz/fuzz_targets/parse_header.rsA fuzz target is a function that receives the fuzzer's bytes:
// fuzz/fuzz_targets/parse_header.rs
#![no_main]
use libfuzzer_sys::fuzz_target;
fuzz_target!(|data: &[u8]| {
// Any panic, crash, or sanitizer error is a finding.
let _ = my_crate::parse_header(data);
});The function under test, a parser for a tiny binary format (a 1-byte version, a 2-byte length, then that many bytes of name), with a bug:
pub fn parse_header(data: &[u8]) -> Option<(u8, &str)> {
let version = *data.first()?;
let len = u16::from_be_bytes([*data.get(1)?, *data.get(2)?]) as usize;
let name = &data[3..3 + len]; // BUG: trusts the length field
std::str::from_utf8(name).ok().map(|n| (version, n))
}
fn main() {
println!("{:?}", parse_header(&[1, 0, 3, b'a', b'b', b'c']));
}Some((1, "abc"))8. Running the fuzzer and reproducing a crash
cargo +nightly fuzz run parse_header -- -max_total_time=60The fuzzer found the bug within its first few executions (the log lines are shortened here):
#2 INITED cov: 24 ft: 25 corp: 1/1b exec/s: 0 rss: 32Mb
#4 NEW cov: 25 ft: 26 corp: 2/3b lim: 4 exec/s: 0 rss: 32Mb L: 2/2 MS: 2 ChangeBit-CopyPart-
thread '<unnamed>' (11939) panicked at src/lib.rs:4:21:
range end index 2607 out of range for slice of length 3
...
SUMMARY: libFuzzer: deadly signal
────────────────────────────────────────────────────────────────────────────────
Failing input:
fuzz/artifacts/parse_header/crash-9bf838bfcbb8144b5f8b9b93e1719bc35cbd789b
Output of `std::fmt::Debug`:
[176, 10, 44]
Reproduce with:
cargo fuzz run parse_header fuzz/artifacts/parse_header/crash-9bf838bfcbb8144b5f8b9b93e1719bc35cbd789b
Minimize test case with:
cargo fuzz tmin parse_header fuzz/artifacts/parse_header/crash-9bf838bfcbb8144b5f8b9b93e1719bc35cbd789b
────────────────────────────────────────────────────────────────────────────────The input [176, 10, 44] is three bytes whose length field claims 2,604 bytes of name (0x0A2C). cov is the number of code edges reached so far; NEW lines are inputs that reached new code and were added to the corpus. The workflow after a finding:
- Reproduce it with the saved artifact (the command above), and minimize it with
cargo fuzz tmin. - Turn it into an ordinary unit test, so the bug stays fixed without the fuzzer:
pub fn parse_header(data: &[u8]) -> Option<(u8, &str)> {
let version = *data.first()?;
let len = u16::from_be_bytes([*data.get(1)?, *data.get(2)?]) as usize;
let name = data.get(3..3 + len)?; // fixed: a short buffer is None, not a panic
std::str::from_utf8(name).ok().map(|n| (version, n))
}
fn main() {}
#[test]
fn fuzz_crash_length_field_longer_than_data() {
assert_eq!(parse_header(&[176, 10, 44]), None);
}
#[test]
fn valid_header_still_parses() {
assert_eq!(parse_header(&[1, 0, 3, b'a', b'b', b'c']), Some((1, "abc")));
}running 2 tests
test fuzz_crash_length_field_longer_than_data ... ok
test valid_header_still_parses ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s- Run the fuzzer again until it stops finding crashes, and keep the
fuzz/corpus/directory: it's the fuzzer's memory of interesting inputs.
cargo fuzz builds with AddressSanitizer by default, so in unsafe code it also reports out-of-bounds reads and use-after-free that wouldn't panic in a normal build.
9. Structured fuzzing with arbitrary
Raw bytes are ideal for parsers. For APIs that take structured values, the arbitrary crate turns fuzzer bytes into typed values, so the fuzzer explores meaningful inputs instead of mostly rejected ones:
#![no_main]
use arbitrary::Arbitrary;
use libfuzzer_sys::fuzz_target;
#[derive(Debug, Arbitrary)]
enum Op {
Insert(i32),
Remove(i32),
Contains(i32),
}
fuzz_target!(|ops: Vec<Op>| {
let mut real = my_crate::SortedVecSet::default();
let mut model = std::collections::BTreeSet::new();
for op in ops {
match op {
Op::Insert(x) => assert_eq!(real.insert(x), model.insert(x)),
Op::Remove(x) => assert_eq!(real.remove(x), model.remove(&x)),
Op::Contains(x) => assert_eq!(real.contains(x), model.contains(&x)),
}
}
});This is the model-based test from example 5, driven by a coverage-guided fuzzer instead of random generation. The two approaches complement each other:
Property testing (proptest) |
Fuzzing (cargo fuzz) |
|
|---|---|---|
| Runs in | every cargo test |
dedicated jobs (minutes to hours) |
| Input generation | random from strategies you write | mutation guided by code coverage |
| Finds | logic bugs in stated properties | crashes, panics, hangs, memory errors, sanitizer findings |
| Minimal failing case | shrinking | cargo fuzz tmin |
| Toolchain | stable | nightly |
| Best for | business logic, data structures | parsers, decoders, unsafe code, anything facing untrusted input |
In CI, run each fuzz target for a bounded time (cargo fuzz run parse_header -- -max_total_time=300) on a schedule, and keep the corpus between runs so the fuzzer builds on previous coverage. Services such as OSS-Fuzz run fuzzers continuously for open-source projects.
Part 3: Benchmarking
10. Why timing loops lie
A benchmark written as "call it a million times in a loop and time it" is wrong more often than right. The optimizer may delete or precompute the work (How the Rust Compiler Works), the first iterations run with cold caches, CPU frequency changes during the run, and one measurement can't tell a real 5% improvement from noise. criterion warms up, runs many samples, applies statistics, and compares against earlier runs.
11. A criterion benchmark
Benchmarks live in benches/ and use their own harness:
[dev-dependencies]
criterion = "0.7"
[[bench]]
name = "parsing"
harness = falseThe code under test, in src/lib.rs:
/// Sums the numbers in a comma-separated line, allocating a String per field first.
pub fn sum_collect(line: &str) -> u64 {
let parts: Vec<String> = line.split(',').map(|s| s.trim().to_string()).collect();
parts.iter().map(|p| p.parse::<u64>().unwrap_or(0)).sum()
}
/// Same result, no allocation.
pub fn sum_streaming(line: &str) -> u64 {
line.split(',').map(|s| s.trim().parse::<u64>().unwrap_or(0)).sum()
}
fn main() {
let line = "1, 2, 3, 4";
println!("{} {}", sum_collect(line), sum_streaming(line));
}10 10The benchmark compares both at two input sizes:
// benches/parsing.rs
use criterion::{BenchmarkId, Criterion, criterion_group, criterion_main};
use my_crate::{sum_collect, sum_streaming};
use std::hint::black_box;
fn bench_parsing(c: &mut Criterion) {
let mut group = c.benchmark_group("sum_line");
for n in [10, 1_000] {
let line: String = (0..n).map(|i| i.to_string()).collect::<Vec<_>>().join(", ");
group.bench_with_input(BenchmarkId::new("collect", n), &line, |b, line| b.iter(|| sum_collect(black_box(line))));
group.bench_with_input(BenchmarkId::new("streaming", n), &line, |b, line| b.iter(|| sum_streaming(black_box(line))));
}
group.finish();
}
criterion_group!(benches, bench_parsing);
criterion_main!(benches);cargo bench --bench parsing prints (numbers are from one run on a cloud VM; yours will differ):
sum_line/collect/10 time: [449.63 ns 450.90 ns 452.19 ns]
Found 4 outliers among 100 measurements (4.00%)
1 (1.00%) low mild
1 (1.00%) high mild
2 (2.00%) high severe
sum_line/streaming/10 time: [160.48 ns 161.16 ns 161.87 ns]
sum_line/collect/1000 time: [55.493 µs 57.030 µs 58.848 µs]
sum_line/streaming/1000 time: [21.141 µs 21.240 µs 21.343 µs]The three numbers are a confidence interval: the lower bound, the best estimate, and the upper bound of the mean time per call. The streaming version is about 2.7 times faster at both sizes, which points at the per-field String allocations. HTML reports with plots are written to target/criterion/.
12. Baselines: proving a change helped
Criterion remembers the last run and reports the change automatically. For a before/after comparison across branches, save a named baseline:
git switch main
cargo bench --bench parsing -- --save-baseline main
git switch my-optimization
cargo bench --bench parsing -- --baseline mainAfter changing sum_collect to collect &str slices instead of Strings, the comparison reports:
sum_line/collect/1000 time: [21.053 µs 21.096 µs 21.139 µs]
change: [−62.156% −61.975% −61.805%] (p = 0.00 < 0.05)
Performance has improved.Now the other side of the story. Here is the same benchmark run twice against its own baseline without any code change, on a shared cloud machine:
sum_line/collect/1000 time: [54.968 µs 55.440 µs 56.025 µs]
change: [+6.8835% +12.137% +17.157%] (p = 0.00 < 0.05)
Performance has regressed.
Found 22 outliers among 100 measurements (22.00%)
22 (22.00%) high severeA "statistically significant 12% regression" from noise: another tenant on the same host was busy. The 22% severe outliers are the warning sign. Lessons:
- Trust large, repeatable differences; re-run before believing small ones.
- Run comparisons on a quiet, dedicated machine, with CPU frequency scaling fixed if you can.
- In shared CI, prefer benchmarks that count instructions instead of time (Callgrind-based harnesses such as
iai-callgrind), which are stable across noisy machines, and keep wall-clock benchmarks for manual investigation.
13. Benchmark design
| Do | Why |
|---|---|
Wrap inputs and outputs in black_box |
stops constant folding and dead-code elimination |
Build inputs outside b.iter(...), or use iter_batched |
measure the operation, not the setup |
| Benchmark several input sizes | exposes accidental O(n²) behavior |
| Use realistic data (real log lines, real payloads) | branch prediction and caches depend on data |
Benchmark release builds only (cargo bench does) |
debug builds are 10–100× slower and distort ratios |
| Keep benchmarks in the repository and run them before merging performance work | regressions creep in one "harmless" change at a time |
For async code, criterion can drive a Tokio runtime with b.to_async(&runtime).iter(|| async { ... }) (the async_tokio feature). divan is a newer harness with a simpler API and built-in allocation counting.
Part 4: More tools for the test suite
14. Snapshot tests with insta
Some outputs are too big to assert field by field: a rendered report, a pretty-printed AST, a JSON response, an error message. A snapshot test stores the expected output and fails when it changes, showing a diff. With inline snapshots, the expected value lives in the test itself:
#[derive(Debug)]
struct Invoice {
customer: String,
lines: Vec<(String, u32, u64)>,
}
fn render(inv: &Invoice) -> String {
let mut out = format!("INVOICE for {}\n", inv.customer);
let mut total = 0;
for (item, qty, cents) in &inv.lines {
let line_total = *qty as u64 * cents;
total += line_total;
out.push_str(&format!("{item:<10} {qty:>3} x {:>7.2} = {:>8.2}\n", *cents as f64 / 100.0, line_total as f64 / 100.0));
}
out.push_str(&format!("TOTAL {:>29.2}\n", total as f64 / 100.0));
out
}
fn main() {}
#[cfg(test)]
mod tests {
use super::*;
fn sample() -> Invoice {
Invoice { customer: "ACME".into(), lines: vec![("widget".into(), 3, 250), ("gadget".into(), 1, 1999)] }
}
#[test]
fn invoice_text() {
insta::assert_snapshot!(render(&sample()), @r"
INVOICE for ACME
widget 3 x 2.50 = 7.50
gadget 1 x 19.99 = 19.99
TOTAL 27.49
");
}
#[test]
fn invoice_debug() {
insta::assert_debug_snapshot!(sample().lines[0], @r#"
(
"widget",
3,
250,
)
"#);
}
}running 2 tests
test tests::invoice_debug ... ok
test tests::invoice_text ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00sThe workflow is what makes it productive: when output changes on purpose, run cargo insta review (from cargo install cargo-insta), look at each diff, and accept it; insta rewrites the snapshot in the source file. Snapshots without @ are stored in snapshots/*.snap files next to the tests. Use assert_json_snapshot! or assert_yaml_snapshot! (with insta's json/yaml features) for API responses, and redactions for timestamps and ids that change every run.
15. Mocks with mockall
Rust code usually tests collaborators with hand-written fakes behind a trait (Testing shows this). When you need expectations (called exactly once, with these arguments, returning this), mockall generates a mock from the trait:
use mockall::automock;
#[derive(Debug, PartialEq)]
pub struct Rate(pub f64);
#[automock]
pub trait RateProvider {
fn rate(&self, from: &str, to: &str) -> Option<Rate>;
}
pub fn convert(provider: &dyn RateProvider, cents: u64, from: &str, to: &str) -> Result<u64, String> {
if from == to {
return Ok(cents); // must not call the provider
}
let Rate(r) = provider.rate(from, to).ok_or_else(|| format!("no rate {from}->{to}"))?;
Ok((cents as f64 * r).round() as u64)
}
fn main() {}
#[cfg(test)]
mod tests {
use super::*;
use mockall::predicate::eq;
#[test]
fn converts_with_the_provider_rate() {
let mut provider = MockRateProvider::new();
provider.expect_rate().with(eq("EUR"), eq("USD")).times(1).returning(|_, _| Some(Rate(1.1)));
assert_eq!(convert(&provider, 1000, "EUR", "USD"), Ok(1100));
}
#[test]
fn same_currency_skips_the_provider() {
let mut provider = MockRateProvider::new();
provider.expect_rate().never();
assert_eq!(convert(&provider, 1000, "EUR", "EUR"), Ok(1000));
}
#[test]
fn missing_rate_is_an_error() {
let mut provider = MockRateProvider::new();
provider.expect_rate().returning(|_, _| None);
assert_eq!(convert(&provider, 1000, "EUR", "JPY"), Err("no rate EUR->JPY".to_string()));
}
}running 3 tests
test tests::converts_with_the_provider_rate ... ok
test tests::missing_rate_is_an_error ... ok
test tests::same_currency_skips_the_provider ... ok
test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00sExpectations are verified when the mock is dropped at the end of the test. Keep mocks at real boundaries (HTTP clients, payment providers, clocks): mocking everything couples tests to implementation details.
16. Coverage with cargo llvm-cov
cargo llvm-cov uses the compiler's own instrumentation (source-based coverage), so line and region coverage are exact, including for generics and macros.
cargo install cargo-llvm-cov
cargo llvm-cov # summary table in the terminal
cargo llvm-cov --html --open # browsable report
cargo llvm-cov --lcov --output-path lcov.info # for Codecov, Coveralls, or SonarQube
cargo llvm-cov nextest # with cargo-nextest as the runnerCoverage shows what's not tested; a high number doesn't show that the tests check anything. That's what the next tool is for.
17. Mutation testing with cargo mutants
cargo mutants changes your code in small ways (replaces a function body with Default::default(), flips < to <=, deletes a !), runs the tests for each change, and reports mutants that survived: changes no test noticed.
cargo install cargo-mutants
cargo mutants # all mutants (slow on big crates)
cargo mutants --file src/pricing.rs # focus on one module
cargo mutants --in-diff pr.diff # only code changed in a pull requestA surviving mutant such as replace apply_discount -> u64 with 0 means no test checks the discount's effect on a total. It's the best available measure of test quality rather than quantity.
18. cargo nextest: a faster, stricter test runner
cargo nextest runs each test in its own process, in parallel across all test binaries. That's faster for large workspaces, isolates tests that touch global state, and adds features the built-in runner lacks.
cargo install cargo-nextest --locked
cargo nextest run # instead of cargo test
cargo nextest run --retries 2 # retry, and report tests as flaky if they pass on retry
cargo nextest run --partition count:1/4 # split the suite across 4 CI machinesTimeouts per test (slow-timeout in .config/nextest.toml) catch hanging tests, and XML test reports plug into CI dashboards. Doc tests still need cargo test --doc.
19. A test strategy for a Rust service
| Layer | Tool | Runs | Catches |
|---|---|---|---|
| Unit tests | #[test], fakes, mockall |
every build | logic errors in functions and types |
| Property tests | proptest |
every build | edge cases in parsers, algorithms, data structures |
| Snapshot tests | insta |
every build | unintended changes to rendered output, errors, API responses |
| Integration tests | tests/, testcontainers for real databases |
every PR | wiring, SQL, serialization, configuration |
| Concurrency tests | loom, Miri, tokio::test(start_paused) |
every PR (Miri: nightly job) | races, orderings, UB, timeouts |
| Fuzzing | cargo fuzz |
scheduled jobs | crashes on untrusted input |
| Benchmarks | criterion, instruction-count harnesses |
performance PRs, nightly | regressions in hot paths |
| Coverage and mutants | cargo llvm-cov, cargo mutants |
weekly or on demand | untested code, weak assertions |
| Load tests | oha, k6, goose |
before releases | throughput, tail latency, memory under load |
Summary
- Property-based tests state what must hold for all inputs;
proptestgenerates inputs from strategies and shrinks failures to minimal cases. Commitproptest-regressions/. - Useful properties: round trips, invariants, model/oracle comparisons, idempotence, order independence, and "never crashes".
- Fuzzing with
cargo fuzzsearches for crashing inputs guided by coverage; use it for parsers, untrusted input, andunsafecode, witharbitraryfor structured inputs. - Benchmark with
criterion:black_box, several input sizes, saved baselines. Distrust small differences on noisy machines; prefer instruction counts in CI. instasnapshot tests cover large outputs with a fast review workflow;mockallprovides expectations when hand-written fakes aren't enough.cargo llvm-covshows what isn't tested,cargo mutantsshows what isn't checked, andcargo nextestmakes big suites faster and flaky tests visible.
Next: Compile Times & Build Engineering: why Rust builds are slow, how to measure them, and how to make them fast.
