Post-mortem Template

Document header

Title:           [SEV level] [Brief description of the incident]
Date:            [Date of incident]
Duration:        [Start time - End time, including timezone]
Authors:         [Post-mortem owner]
Status:          Draft | In Review | Final
Severity:        SEV1 | SEV2 | SEV3
Services affected: [List of affected services]
Customer impact: [Brief description of user-facing impact]

1. Summary

Write 3-5 sentences covering: what broke, who was affected, how long it lasted, and how it was resolved. This should be readable by a non-engineer.

Example:

On 2026-03-10 between 14:22 and 15:47 UTC, the checkout service returned 500 errors for approximately 30% of payment requests. An estimated 2,400 customers were unable to complete purchases during this window. The root cause was a connection pool exhaustion triggered by a configuration change deployed at 14:15 UTC. The incident was resolved by rolling back the configuration change and increasing the connection pool size.

2. Timeline

Use UTC timestamps. Include both automated events (alerts, deploys) and human actions (who did what).

14:15 UTC  - Deploy #4521 pushed to production (config change to DB pool settings)
14:22 UTC  - Checkout-api error rate alert fires (threshold: 1%, observed: 8%)
14:24 UTC  - On-call engineer @alice acks the page
14:27 UTC  - @alice opens war room, assigns IC role to @bob
14:30 UTC  - Status page updated: "Investigating increased errors on checkout"
14:35 UTC  - @alice identifies connection pool exhaustion in service metrics
14:40 UTC  - @alice correlates with deploy #4521 timeline
14:45 UTC  - Decision: rollback deploy #4521
14:48 UTC  - Rollback initiated
14:55 UTC  - Rollback complete. Error rate dropping.
15:00 UTC  - Status page updated: "Fix deployed, monitoring"
15:30 UTC  - Error rate back to baseline (0.05%)
15:47 UTC  - IC @bob declares incident resolved
15:47 UTC  - Status page updated: "Resolved"

3. Root cause analysis

What happened

Describe the technical chain of events. Be specific about the failure mode.

Five whys

Why 1: Why did checkout fail?
  -> Connection pool was exhausted; new requests could not get a DB connection.

Why 2: Why was the connection pool exhausted?
  -> Deploy #4521 reduced max_connections from 100 to 10.

Why 3: Why was that configuration change deployed?
  -> An engineer was tuning connection settings for a staging environment
     and accidentally included the production config file.

Why 4: Why did the production config get included?
  -> The staging and production configs are in the same directory with
     similar names (db-config-staging.yaml, db-config-prod.yaml).

Why 5: Why was there no safeguard?
  -> No automated validation checks connection pool size against a minimum
     threshold before deploy.

Contributing factors

List factors that did not cause the incident but made it worse or slower to resolve:

• No deployment diff review required for config-only changes
• Connection pool metric was not on the checkout service dashboard
• Runbook for this alert did not mention checking recent deploys

4. Detection analysis

5. Response analysis

6. Impact assessment

7. Action items

Every action item must have: owner, due date, priority, and definition of done.

8. Lessons learned

What went well

• Alert fired quickly (7 minutes from cause)
• War room was organized and focused
• Status page was updated within 8 minutes of the page
• Rollback was clean and effective

What did not go well

• Config change had no validation gate
• It took 13 minutes to identify the deploy as the cause
• The runbook did not mention checking recent deployments

Where we got lucky

• The config change was easily rollbackable. A schema migration with the same type of error would have been much harder to reverse.

Facilitation guide

Before the meeting

• Post-mortem owner drafts sections 1-3 before the meeting
• Share the draft with all participants 2 hours before the meeting
• Remind everyone: this is a blameless review. We discuss systems, not individuals.

During the meeting (60-90 minutes)

1. (5 min) IC reads the summary and timeline aloud
2. (20 min) Walk through the root cause analysis. Ask: "Is there anything missing?"
3. (15 min) Review detection and response. Ask: "Where could we have been faster?"
4. (20 min) Draft action items as a group. For each: agree on owner and priority.
5. (10 min) Capture lessons learned. Ask: "What went well? What did not?"
6. (5 min) Agree on review date for action items (usually 30 days)

After the meeting

• Owner finalizes the document within 24 hours
• Publish to the team's incident archive
• Enter all action items in the issue tracker with the agreed due dates
• Schedule a 30-day follow-up to verify action item completion

Runbook Template

Standard runbook structure

Every runbook follows the same six-section format. Consistency across runbooks means on-call engineers can find information in the same place every time, even for services they have never seen before.

Template

# [Alert Name] - [Service Name] Runbook

**Last updated:** [YYYY-MM-DD]
**Owner:** [Team or individual responsible for this runbook]
**Alert source:** [Monitoring tool and alert ID/link]
**Related services:** [Upstream and downstream dependencies]

---

## 1. SYMPTOM

[Quote the alert condition verbatim. Include the metric, threshold, and window.]

Example:
> Alert: checkout-api-error-rate
> Condition: HTTP 5xx rate > 1% for 5 minutes
> Dashboard: [link to dashboard]

---

## 2. IMPACT

**Who is affected:** [Customer segment or internal users]
**How they are affected:** [Cannot checkout, see errors, experience slowness]
**Severity:** [SEV1/SEV2/SEV3 - reference the severity classification table]
**Business impact:** [Revenue, data integrity, compliance, reputation]

---

## 3. INVESTIGATION STEPS

Follow these steps in order. At each step, the result tells you where to go next.

### Step 1: Check the dashboard
- Open: [dashboard link]
- Normal: Error rate < 0.1%, latency p99 < 300ms
- Abnormal: If error rate is elevated, proceed to Step 2

### Step 2: Check recent deployments
- Run: `kubectl rollout history deployment/checkout-api -n production`
- Or check: [deploy tool link]
- If a deploy happened in the last 30 minutes, this is likely the cause.
  Go to Mitigation Step A (Rollback).

### Step 3: Check downstream dependencies
- Open: [dependency dashboard link]
- Check: database connection pool, payment gateway status, cache hit rate
- If a dependency is degraded, the issue is upstream. Escalate to that
  service's on-call (see Escalation section).

### Step 4: Check resource utilization
- Run: `kubectl top pods -n production -l app=checkout-api`
- Normal: CPU < 70%, Memory < 80%
- If resources are exhausted, go to Mitigation Step B (Scale).

### Step 5: Check application logs
- Run: `kubectl logs -l app=checkout-api -n production --tail=200 | grep ERROR`
- Or query: [log aggregator link with pre-built query]
- Look for: stack traces, connection refused, timeout errors
- If you see a new error pattern, document it and escalate.

---

## 4. MITIGATION STEPS

### Step A: Rollback the last deployment
```bash
kubectl rollout undo deployment/checkout-api -n production

Monitor error rate for 10 minutes. If it returns to baseline, the deploy was the cause. Document and proceed to post-mortem.

Step B: Scale the service

kubectl scale deployment/checkout-api -n production --replicas=10

Monitor for 5 minutes. If error rate drops, the issue is capacity-related. Investigate the traffic spike source.

Step C: Restart pods (last resort)

kubectl rollout restart deployment/checkout-api -n production

Use only if Steps A and B did not help and you suspect a memory leak or stuck process. This causes brief service disruption during rolling restart.

Step D: Toggle feature flag (if applicable)

• Open: [feature flag tool link]
• Disable: [flag name] for production environment
• This removes the most recent feature change without a full rollback.

5. ESCALATION

If the above steps do not resolve the issue within 30 minutes, escalate:

6. CONTEXT

• Architecture doc: [link]
• Service dashboard: [link]
• Dependency map: [link]
• Past incidents:
  • [INC-1234] - Similar error spike caused by config change (2026-01)
  • [INC-1189] - Database failover caused checkout errors (2025-11)
• On-call schedule: [link]
• Deployment pipeline: [link]

---

## Runbook quality checklist

Before publishing a runbook, verify:

- [ ] Alert condition is quoted verbatim with metric name and threshold
- [ ] Impact section states who is affected in plain language
- [ ] Every investigation step has a "normal" and "abnormal" result
- [ ] Mitigation steps include actual commands or tool links, not just descriptions
- [ ] Escalation contacts are current (review quarterly)
- [ ] Context links are not broken
- [ ] A new team member can follow this without prior service knowledge

## Runbook maintenance

- **Review cadence:** Every runbook must be reviewed every 90 days
- **Update triggers:** Any incident where the runbook was incomplete or wrong
- **Ownership:** The team that owns the service owns the runbook
- **Testing:** During on-call onboarding, have new engineers walk through runbooks
  for the top 5 most-paged alerts as a tabletop exercise

## Common investigation commands reference

```bash
# Kubernetes - check pod status
kubectl get pods -n production -l app=SERVICE_NAME

# Kubernetes - check recent events
kubectl get events -n production --sort-by='.lastTimestamp' | head -20

# Kubernetes - check resource usage
kubectl top pods -n production -l app=SERVICE_NAME

# Kubernetes - check rollout status
kubectl rollout status deployment/SERVICE_NAME -n production

# Kubernetes - view recent logs
kubectl logs -l app=SERVICE_NAME -n production --tail=100 --since=10m

# Database - check active connections (PostgreSQL)
SELECT count(*) FROM pg_stat_activity WHERE state = 'active';

# Database - check long-running queries (PostgreSQL)
SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity
WHERE state != 'idle' AND now() - pg_stat_activity.query_start > interval '30 seconds'
ORDER BY duration DESC;

Status Page Guide

Status page structure

Components

Organize by user-facing service, not internal architecture. Customers do not care which microservice is down - they care what they cannot do.

Good component names:

• Checkout & Payments
• User Dashboard
• API (v2)
• Mobile App
• Webhooks & Notifications
• Data Exports

Bad component names (internal jargon):

• payment-gateway-service
• redis-cache-cluster
• kafka-consumer-group-3

Component statuses

Incident update templates

Investigating

Title: Elevated error rates on [Component]

Update (HH:MM UTC):
We are investigating reports of [symptom in plain language]. Some customers
may experience [specific impact - e.g., "errors when attempting to check out"
or "slower page load times on the dashboard"].

We will provide an update within 30 minutes.

Identified

Update (HH:MM UTC):
We have identified the cause of [symptom]. [One sentence about the cause in
plain language - e.g., "A configuration change is causing connection issues
with our payment processor."]

Our engineering team is working on a fix. We expect to have this resolved
within [estimated time if known, or "the next 1-2 hours"].

We will provide another update within 30 minutes.

Monitoring

Update (HH:MM UTC):
We have deployed a fix for [symptom]. Our systems are recovering and we are
monitoring to confirm the issue is fully resolved.

[If applicable: "Some customers may continue to see intermittent errors for
the next 10-15 minutes as the fix propagates."]

We will provide a final update once we confirm full recovery.

Resolved

Update (HH:MM UTC):
This incident has been resolved. [Component] is operating normally.

Summary: Between [start time] and [end time] UTC, [brief description of what
happened and who was affected]. [One sentence on what was done to fix it.]

[If applicable: "No customer action is required." or "If you experienced
[specific issue], please [specific action - e.g., retry your request, contact
support at support@example.com]."]

We will be conducting a thorough post-incident review to prevent recurrence.
We apologize for the disruption.

Maintenance notification templates

Scheduled maintenance (advance notice)

Title: Scheduled maintenance for [Component]

We will be performing scheduled maintenance on [Component] on [date] from
[start time] to [end time] UTC ([convert to major customer timezones]).

During this window:
- [Specific impact - e.g., "The API will return 503 errors"]
- [What will still work - e.g., "The dashboard will remain accessible in
  read-only mode"]
- [Estimated duration of actual downtime within the window]

[If applicable: "We recommend scheduling any critical operations before or
after this maintenance window."]

We will update this notice when maintenance begins and when it is complete.

Communication cadence

Rules:

• Never go more than 30 minutes without an update during an active incident
• If there is no new information, say so: "We are continuing to investigate. No new information at this time. Next update in 30 minutes."
• All timestamps in UTC with local timezone equivalents for major customer regions
• The IC reviews the resolved update before publishing

Writing guidelines

Do

• State the customer impact first, then what you are doing
• Use plain language a non-technical person can understand
• Be honest about what you know and do not know
• Include specific times and durations
• Acknowledge the disruption: "We apologize for the inconvenience"

Do not

• Use internal service names, error codes, or technical jargon
• Say "no impact" if customers are reporting problems
• Blame third parties without confirmation ("our cloud provider caused...")
• Promise specific resolution times unless you are confident
• Use passive voice to hide accountability ("errors were experienced")

Tone

• Professional but human
• Direct and factual
• Empathetic without being overly apologetic
• Confident about what you know, transparent about what you do not

Status page tool setup checklist

When setting up a new status page:

• Define components based on user-facing services (not internal architecture)
• Set up subscriber notifications (email, SMS, webhook, RSS)
• Configure automated status updates from monitoring (operational/degraded)
• Pre-draft incident templates (copy from this guide)
• Assign ownership: who can publish updates (on-call + Communications Lead)
• Test the notification flow end-to-end before going live
• Add the status page link to your application's footer and support docs
• Configure maintenance window scheduling
• Set up uptime metrics display (90-day rolling window per component)
• Review and update component list quarterly as services evolve

War Room Checklist

Activation criteria

Open a war room when any of these conditions are met:

• SEV1 incident declared
• SEV2 not mitigated within 30 minutes
• Incident affects multiple services or teams
• Incident has customer-visible impact and no clear cause after 15 minutes
• Incident commander requests a war room for any reason

War room activation checklist (first 5 minutes)

The person who activates the war room (usually the on-call engineer or IC) runs through this checklist:

• Open a dedicated video call (use the team's standing war room link)
• Create or identify the incident channel (e.g., #inc-2026-03-14-checkout)
• Post the incident summary in the channel:

  INCIDENT ACTIVE
  Severity: [SEV1/SEV2]
  Summary: [One sentence - what is broken]
  Impact: [Who is affected and how]
  Started: [HH:MM UTC]
  IC: [@name]
  War room: [video call link]

• Assign roles (see Role Cards below)
• Confirm all role holders have joined the war room
• Scribe creates the incident timeline document

Role cards

Incident Commander (IC)

Primary job: Coordinate the response. You are NOT debugging.

Checklist:

• Confirm severity classification
• Assign all roles (Communications Lead, Technical Lead, Scribe)
• Run 15-minute checkpoints (see Checkpoint Script)
• Make escalation decisions
• Approve rollback or mitigation actions
• Decide when to declare resolution
• Assign post-mortem owner before closing the war room

Rules:

• Do not investigate the issue yourself. Delegate.
• If you are also the most qualified person to debug, hand off IC to someone else.
• If the war room exceeds 2 hours, rotate IC or bring in a fresh one.

Communications Lead

Primary job: Keep customers, stakeholders, and support informed.

Checklist:

• Post first status page update within 10 minutes of war room opening
• Update status page every 30 minutes (or sooner if there is new information)
• Notify internal stakeholders (support team, account managers for affected customers)
• Draft the resolution update for IC review before publishing
• Compile a list of customer reports or support tickets for the post-mortem

Rules:

• Use the templates from references/status-page-guide.md
• Every update must be reviewed by IC before publishing (exception: "still investigating" updates)
• Never share internal details, blame, or unconfirmed root causes externally

Technical Lead

Primary job: Drive the investigation and fix.

Checklist:

• Follow the relevant runbook (if one exists)
• Announce investigation steps before executing them
• Report findings to IC at each checkpoint
• Propose mitigation options with trade-offs
• Execute the approved fix
• Confirm metrics are recovering after the fix

Rules:

• Announce all production commands before running them
• If the runbook does not cover this scenario, say so immediately
• If you need help, tell the IC. Do not silently struggle.

Scribe

Primary job: Maintain a real-time timeline of the incident.

Checklist:

• Create the timeline document (use the team's incident template)
• Log every significant action with a UTC timestamp
• Log who did what (not just what happened)
• Log decisions and the reasoning behind them
• Log things that were tried but did not work
• At resolution, hand the timeline to the post-mortem owner

Rules:

• Capture facts, not interpretations
• If something is unclear, ask for clarification and log the answer
• The timeline is the primary input for the post-mortem - completeness matters

Checkpoint script (every 15 minutes)

The IC runs this script at each checkpoint. Read it aloud:

CHECKPOINT - [HH:MM UTC]

1. STATUS CHECK
   "Technical Lead: What do we know now that we didn't know 15 minutes ago?"

2. NEXT STEPS
   "What are we trying next? Who is doing it?"

3. ESCALATION CHECK
   "Do we need to bring in anyone else? Any dependent teams?"

4. COMMUNICATIONS CHECK
   "Communications Lead: Is the status page current? When is the next update due?"

5. TIMELINE CHECK
   "Scribe: Are we capturing everything? Anything to add?"

6. SEVERITY CHECK
   "Has the severity changed? Should we escalate or de-escalate?"

War room rules

Post these rules in the incident channel at the start of every war room:

WAR ROOM RULES
1. One conversation at a time. IC moderates.
2. Announce all production commands BEFORE running them.
3. No side investigations without telling the IC.
4. If you join late, read the timeline first. Do not ask "what happened?"
5. Mute when not speaking (video call).
6. Keep the channel for incident discussion only. Use threads for tangents.
7. If you do not have a role, observe silently unless asked.

War room closure checklist

When the IC declares the incident resolved:

• Confirm metrics have returned to baseline for at least 15 minutes
• Communications Lead posts the resolution update on the status page
• IC assigns a post-mortem owner and sets a deadline (within 48 hours)
• Scribe finalizes the timeline and shares it with the post-mortem owner
• IC posts a summary in the incident channel:

  INCIDENT RESOLVED
  Duration: [X hours Y minutes]
  Root cause: [One sentence]
  Fix applied: [One sentence]
  Post-mortem owner: [@name]
  Post-mortem deadline: [date]

• IC thanks everyone who participated
• Close the war room video call
• Archive the incident channel (do not delete - it is a historical record)

War room anti-patterns