Chaos Engineering

Chaos engineering is the practice of running controlled experiments on a system by injecting failures, in order to find weaknesses before they cause incidents. Resilience mechanisms such as timeouts and retries, circuit breakers, and fallbacks are configured from assumptions about how the system fails. Those assumptions are often wrong, and mechanisms that are rarely exercised often do not work. Chaos experiments test them under real conditions.

The experiment method

The Principles of Chaos Engineering (published by Netflix engineers) describe experiments built on a steady-state hypothesis, real-world fault events, comparison of a control and an experimental group, and minimal blast radius. The following cycle follows those principles:

1. Define steady state. Choose measurable outputs that indicate normal behavior, such as request success rate, latency percentiles, or orders per minute. Business-level metrics are preferable to low-level ones such as CPU.
2. Form a hypothesis. State that steady state will continue in both the control group and the experimental group, for example "if one payment service instance is terminated, the checkout success rate stays above 99.9%".
3. Introduce real-world events. Inject faults that mirror real failures: instance crashes, network delay, dependency errors.
4. Compare. Measure the difference between control and experiment. A deviation disproves the hypothesis and reveals a weakness.
5. Fix and repeat. Correct the weakness, then rerun the experiment. Automate it to catch regressions.

flowchart LR
    S["Define steady state metrics"] --> H["Form hypothesis"]
    H --> I["Inject fault with limited blast radius"]
    I --> O["Observe and compare with control"]
    O --> D{"Steady state held?"}
    D -->|Yes| E["Increase scope or add new fault"]
    D -->|No| F["Fix weakness"]
    F --> S
    E --> H

Example experiment: steady state is a checkout success rate of at least 99.9% and a p99 latency below 800 ms. The hypothesis is that adding 500 ms of latency to the inventory service for 5% of traffic keeps both within bounds, because the caller has a 300 ms timeout and a cached fallback. The experiment injects the delay for 10 minutes with an abort condition of success rate below 99.5%. If success rate drops, the timeout or fallback is not working as assumed, and that becomes a fix.

A chaos experiment is not random breakage. It has a hypothesis, defined metrics, an abort condition, and a bounded scope.

Fault types

Latency injection is often the most revealing, since slow dependencies are more dangerous than dead ones (a dead one fails fast). It exposes missing timeouts and retry amplification.

Blast radius and safety

The blast radius is the set of users and components affected by an experiment. Experiments start small and expand only after they succeed.

• Start in a test or staging environment, then move to production, where real traffic, data, and configuration exist. Staging rarely reproduces production load and dependencies.
• Limit scope: one instance, one percent of traffic, one tenant, or internal users only.
• Define abort conditions: if error rate or latency crosses a threshold, stop the experiment automatically and restore normal operation (the "stop button").
• Verify monitoring and alerting first. An experiment cannot be observed if the system is not instrumented (see observability).
• Announce experiments to affected teams, and run them when engineers are available to respond, not during peak business events.
• Prefer experiments that are reversible, and never target data in a way that risks permanent loss.

Game days

A game day is a scheduled, team-wide exercise in which failures are injected, either automatically or manually, and participants practice detecting, diagnosing, and resolving them.

Typical structure:

1. Choose scenarios, for example loss of a database primary, an availability zone outage, or an expired certificate.
2. Write down hypotheses and expected system behavior beforehand.
3. Run the scenario with observers recording detection time, response actions, and unexpected effects.
4. Review afterwards: whether alerts fired, whether runbooks were correct, and whether the system behaved as predicted.
5. Record action items, and assign owners.

Game days test people and processes, including alerts, dashboards, escalation paths, and runbooks, as well as software.

Tools

Fault injection in a service mesh works at the proxy layer, so it needs no application changes and can target specific routes or percentages of traffic (see service mesh and sidecars).

What experiments commonly find

• Missing or excessive timeouts on calls to dependencies.
• Retries without backoff or budgets that amplify failures.
• Fallbacks that were never exercised and fail themselves.
• Health checks that keep unhealthy instances in rotation, or that restart all instances when a dependency fails.
• Single points of failure that were assumed redundant, such as failover that has never been triggered.
• Autoscaling that reacts too slowly to instance loss.
• Alerts and dashboards that do not show the failure.

Adopting chaos engineering

• Have basic resilience mechanisms and observability in place first. Injecting faults into a system with no monitoring produces outages, not knowledge.
• Begin with simple experiments: terminate one instance, add latency to one dependency.
• Automate experiments in continuous delivery pipelines or run them on a schedule, so regressions are detected.
• Treat findings as engineering work items with owners.

Interview questions

• What is a steady-state hypothesis? A prediction that a measurable indicator of normal behavior (such as success rate or latency) will stay within bounds during an injected fault. If it does not, the experiment has found a weakness.
• How do you limit the blast radius of an experiment? Start with small scope (one instance, a small share of traffic, or internal users), define automatic abort conditions, ensure monitoring is in place, and keep the fault reversible.
• Why run chaos experiments in production? Production has real traffic patterns, data volumes, configuration, and dependencies that staging does not reproduce. The risk is managed through limited scope and abort conditions.
• Why is injecting latency often more informative than killing an instance? A dead dependency fails fast, and clients see errors immediately. A slow dependency ties up threads and connections and exposes missing timeouts, bulkheads, and retry amplification.
• What is a game day? A planned exercise that injects failures to test both the system and the team's detection, response, and runbooks, ending with a review and action items.