Chaos Engineering

A practitioner's framework for running controlled failure experiments in production systems. This skill covers how to design, execute, and learn from chaos experiments - from simple latency injections to full game days - with an emphasis on safety, minimal blast radius, and translating findings into durable resilience improvements.

When to use this skill

Trigger this skill when the user:

• Wants to design a chaos experiment or fault injection scenario
• Is setting up a chaos engineering program from scratch
• Needs to implement network latency, packet loss, or service dependency failures
• Is planning or facilitating a game day exercise
• Needs to validate circuit breakers, retries, or failover logic under real failure conditions
• Wants to measure and improve MTTR (Mean Time to Recovery)
• Is evaluating chaos tooling (Chaos Monkey, Litmus, Gremlin, AWS Fault Injection Simulator)

Do NOT trigger this skill for:

• Writing standard retry or circuit breaker code without the intent to test it under chaos (use backend-engineering skill)
• Load testing or performance benchmarking that does not involve failure injection (use performance-engineering skill)

Key principles

1. Define steady state before breaking anything - You cannot detect a deviation without a baseline. Before every experiment, define the precise metric (p99 latency, error rate, success count) that proves the system is healthy. If the system is already degraded, stop and fix it first.

2. Start small in staging, graduate to production slowly - Every experiment starts in a non-production environment. Only move to production after the hypothesis is proven correct in staging and blast radius is understood. Even in production, target a small traffic percentage or a single availability zone first.

3. Minimize blast radius - The experiment scope must be as small as possible. Isolate the failure to one service, one host, or one region. Have a kill switch ready before starting. The goal is learning, not causing an incident.

4. Build the hypothesis before turning on the failure - A hypothesis has three parts: "When X happens, the system will Y, as evidenced by Z metric." Without a pre-written hypothesis you cannot distinguish a passing experiment from an outage.

5. Automate experiments and run them continuously - A chaos experiment run once is a one-time curiosity. Automated experiments that run on every deploy catch regressions before production. The goal is a resilience gate in CI/CD, not a quarterly fire drill.

Core concepts

Steady State Hypothesis

The foundation of every experiment. A steady state is a measurable, normal behavior of the system:

Hypothesis template:
  "Under normal conditions, [service] processes [metric] at [baseline value].
   When [failure condition] is introduced, [metric] will remain within [acceptable range]
   because [resilience mechanism] will compensate."

Example:
  "Under normal conditions, the checkout service processes 95% of requests in <500ms.
   When the inventory service has 500ms of added latency, checkout p99 will remain
   <800ms because the circuit breaker will open and return cached availability data."

Metrics for steady state (RED method):

• Rate - requests per second
• Errors - error rate (%)
• Duration - latency percentiles (p50, p95, p99)

Blast Radius

The maximum potential impact of the experiment if something goes wrong. Always quantify before starting:

Experiment Lifecycle

1. DEFINE    -> Write steady state hypothesis + success/failure criteria
2. SCOPE     -> Identify target environment, blast radius, and rollback mechanism
3. INSTRUMENT -> Confirm observability is in place to measure the hypothesis metric
4. RUN       -> Inject failure; observe metric in real time
5. ANALYZE   -> Did steady state hold? If not, why? What was the real failure mode?
6. IMPROVE   -> Fix the gap. Update runbooks. Automate the experiment.
7. REPEAT    -> Re-run to confirm the fix. Graduate to broader scope.

Failure Modes Taxonomy

Common tasks

Design a chaos experiment

Use this template to structure every experiment:

## Chaos Experiment: [Short Name]

**Date:** YYYY-MM-DD
**Hypothesis:**
  When [failure condition], [service] will [expected behavior]
  as evidenced by [metric staying within range].

**Steady State (before):**
  - Metric: checkout.success_rate
  - Baseline: >= 99.5%
  - Measured via: Datadog SLO dashboard / Prometheus query

**Failure injection:**
  - Tool: Toxiproxy / Litmus / AWS FIS
  - Target: inventory-service, 1 of 5 pods
  - Type: HTTP 503 response, 100% of requests to /api/stock
  - Duration: 10 minutes

**Blast radius:**
  - Scope: Single pod in staging environment
  - Traffic affected: ~20% of inventory requests
  - Kill switch: `kubectl delete chaosexperiment inventory-latency`

**Success criteria:**
  - checkout.success_rate remains >= 99.5% during injection
  - Circuit breaker opens within 30s
  - Fallback (cached stock) is served to users

**Failure criteria:**
  - checkout.success_rate drops below 99% for > 2 minutes
  - Any user-visible 500 errors during injection

**Result:** [PASS / FAIL]
**Finding:** [What actually happened]
**Action:** [Ticket number + fix description]

Implement network latency injection

Inject latency at the network level using Linux Traffic Control (tc) or Toxiproxy (application-level proxy). Prefer Toxiproxy for service-specific targeting; prefer tc for host-level experiments.

Using Toxiproxy (service-level, recommended for staging):

# Install and start Toxiproxy
toxiproxy-server &

# Create a proxy for the downstream service
toxiproxy-cli create --listen 0.0.0.0:8474 --upstream inventory-svc:8080 inventory_proxy

# Add 200ms of latency with 50ms jitter to 100% of connections
toxiproxy-cli toxic add inventory_proxy \
  --type latency \
  --attribute latency=200 \
  --attribute jitter=50 \
  --toxicity 1.0

# Point your service at localhost:8474 instead of inventory-svc:8080
# ... run the experiment, observe metrics ...

# Remove the toxic (kill switch)
toxiproxy-cli toxic remove inventory_proxy --toxicName latency_downstream

Using tc netem (host-level, for infrastructure experiments):

# Add 300ms latency + 30ms jitter to all outbound traffic on eth0
sudo tc qdisc add dev eth0 root netem delay 300ms 30ms

# Add 10% packet loss
sudo tc qdisc change dev eth0 root netem loss 10%

# Remove (kill switch)
sudo tc qdisc del dev eth0 root

Always test the kill switch before starting the experiment. A failed kill switch turns a chaos experiment into a real incident.

Simulate service dependency failure

Test what happens when a downstream service becomes unavailable. Use Wiremock or a simple mock server to return error responses:

// Using Wiremock (Java/Docker) - stub 100% 503s for /api/stock
{
  "request": { "method": "GET", "urlPattern": "/api/stock/.*" },
  "response": {
    "status": 503,
    "headers": { "Content-Type": "application/json" },
    "body": "{\"error\": \"Service Unavailable\"}",
    "fixedDelayMilliseconds": 5000
  }
}

// Verify your circuit breaker opened:
//   - Log line: "Circuit breaker OPEN for inventory-service"
//   - Metric: circuit_breaker_state{service="inventory"} == 1
//   - Fallback response served to callers

Checklist for dependency failure experiments:

• Circuit breaker opens within the configured threshold
• Fallback value or cached response is served (not a 500)
• Downstream errors do not propagate to user-facing error rate
• Circuit breaker closes when dependency recovers
• Alerting fires within SLO window, not after it

Run a game day - facilitation guide

A game day is a structured, cross-team exercise that rehearses failure scenarios. It combines chaos experiments with human coordination practice.

Preparation (2 weeks before):

1. Choose a realistic scenario (e.g., "Primary database AZ goes down")
2. Define the experiment scope and blast radius in writing
3. Confirm on-call rotation and escalation paths are documented
4. Brief all participants: on-call engineers, product owner, leadership observer
5. Set up a dedicated incident Slack channel and shared dashboard link

Day-of agenda (3-hour format):

00:00 - 00:15  Kickoff: review scenario, confirm kill switches, assign roles
               Roles: Incident Commander, Chaos Operator, Scribe, Observer
00:15 - 00:30  Baseline check: confirm steady state metrics look healthy
00:30 - 01:30  Inject failure; team responds as if it were a real incident
               Scribe records every action and timestamp
01:30 - 01:45  Halt injection; confirm system recovers to steady state
01:45 - 02:30  Hot debrief: timeline walkthrough while memory is fresh
               Key questions: What surprised you? Where were the gaps?
02:30 - 03:00  Action items: each gap gets a ticket, owner, and due date

Post-game day outputs:

• Updated runbook with gaps filled
• Action items tracked in a backlog with SLO-aligned due dates
• Recorded MTTR for the scenario (use as a benchmark for next game day)
• Decision on whether to automate the experiment in CI

Test database failover

Verify that your application correctly handles a primary database failover without data loss or extended downtime:

# 1. Confirm replication lag is near zero before starting
#    psql -h replica -c "SELECT now() - pg_last_xact_replay_timestamp() AS replication_lag;"

# 2. Start continuous writes to the primary (background process)
while true; do
  psql -h primary -c "INSERT INTO chaos_probe (ts) VALUES (now());" 2>&1
  sleep 0.5
done &
PROBE_PID=$!

# 3. Inject: promote the replica (or use your cloud provider's failover API)
#    AWS RDS: aws rds failover-db-cluster --db-cluster-identifier my-cluster
#    Manual:  pg_ctl promote -D /var/lib/postgresql/data

# 4. Observe:
#    - How long until the application reconnects?
#    - Were any writes lost? (check probe table row count)
#    - Did health checks detect the failover promptly?
#    - Did connection pool recover without restart?

# 5. Kill the probe writer
kill $PROBE_PID

# 6. Measure:
#    - Connection downtime: seconds between last successful write and first write to new primary
#    - Data loss: rows missing from probe table
#    - Recovery time: time until application traffic normalizes

Success criteria: Connection re-established within 30s, zero data loss, no application restart required.

Implement circuit breaker validation

After implementing a circuit breaker, verify it actually works under failure conditions. This is the most commonly skipped verification step.

# Validation test: assert circuit breaker opens under failure threshold
import pytest
import time
from unittest.mock import patch

def test_circuit_breaker_opens_on_failure_threshold():
    cb = CircuitBreaker(threshold=5, reset_ms=30000)
    failures = 0

    def failing_op():
        raise ConnectionError("downstream unavailable")

    # Exhaust the threshold
    for _ in range(5):
        with pytest.raises((ConnectionError, CircuitOpenError)):
            cb.call(failing_op)

    # Next call must fast-fail without calling the dependency
    call_count = 0
    def counting_op():
        nonlocal call_count
        call_count += 1
        return "ok"

    with pytest.raises(CircuitOpenError):
        cb.call(counting_op)

    assert call_count == 0, "Circuit breaker must NOT call the dependency when OPEN"
    assert cb.state == OPEN

def test_circuit_breaker_recovers_after_reset_timeout():
    cb = CircuitBreaker(threshold=5, reset_ms=100)  # 100ms for test speed
    # ... trip the breaker ...
    time.sleep(0.15)
    # Should transition to HALF-OPEN and allow one trial call
    result = cb.call(lambda: "ok")
    assert cb.state == CLOSED

Experiment to run in staging:

1. Deploy with circuit breaker configured
2. Use Toxiproxy to make the dependency return 503
3. Verify: breaker opens within threshold, fallback activates, logs confirm state transitions
4. Remove the toxic, verify: breaker moves to half-open, trial succeeds, breaker closes

Measure and improve MTTR

MTTR (Mean Time to Recovery) is the primary output metric of a chaos engineering program. Improve it by reducing each phase:

Incident timeline phases:
  Detection  - time from failure start to alert firing
  Triage     - time from alert to understanding root cause
  Response   - time from diagnosis to fix applied
  Recovery   - time from fix applied to steady state restored

MTTR = Detection + Triage + Response + Recovery

Measurement query (Prometheus example):

# Time from incident start (SLO breach) to recovery (SLO restored)
# Track this per incident type in a spreadsheet; compute rolling mean

# Alert on SLO burn rate (detection proxy):
(
  rate(http_requests_total{status=~"5.."}[5m]) /
  rate(http_requests_total[5m])
) > 0.01  # >1% error rate

Improvement levers by phase:

Track MTTR per failure category. A single average hides that your database failovers recover in 2 min but your certificate expiry incidents take 45 min.

Anti-patterns

Gotchas

1. Kill switch first, experiment second - The most common mistake is discovering the kill switch doesn't work only after the experiment has started causing damage. Always test the kill switch (e.g., kubectl delete chaosexperiment, toxiproxy-cli toxic remove) before injecting any failure.

2. Observability blind spots - If your metrics pipeline routes through the same service you're injecting failure into, you'll lose visibility at exactly the moment you need it most. Confirm that dashboards and alerting are independent of the experiment target before starting.

3. Staging ≠ production behavior - A hypothesis that holds in staging often fails in production due to traffic volume differences, connection pool sizing, or infrastructure configurations that only exist in prod. Graduate scope incrementally - don't treat a staging pass as proof production will hold.

4. Circuit breaker misconfiguration in tests - Unit tests often use a timeout of 0 or 1ms for the circuit breaker reset window to speed tests up. The production timeout may be 30 seconds. Validate circuit breaker behavior with production-realistic timeouts in at least one integration test.

5. Experiment automation without human review - Fully automated chaos experiments that run on every deploy are the goal, but skipping the review step when a new experiment type is added risks running untested blast-radius assumptions in production. Treat new experiment types as requiring manual approval for the first 2-3 runs.

References

For experiment catalogs, failure injection recipes, and advanced tooling guidance:

• references/experiment-catalog.md - ready-to-use experiments organized by failure type

Only load the references file if the current task requires a specific experiment recipe.