Knowledge Management

The documentation quadrant

Adapt the Divio documentation framework to organize internal knowledge into four distinct categories, each with its own purpose, style, and audience:

Why this matters

The most common documentation problem is mixing categories. A tutorial that stops to explain architectural history loses the reader. A reference page that tries to teach concepts becomes unusable for quick lookups. Keep categories separate and link between them.

Information architecture

Organizing by system, not by team

Teams change. Systems persist. Organize documentation around systems and domains, not org chart structure.

Bad structure:

/docs
  /backend-team
  /frontend-team
  /platform-team

Good structure:

/docs
  /services
    /checkout-service
    /user-service
    /payment-gateway
  /infrastructure
    /kubernetes
    /databases
    /ci-cd
  /guides
    /onboarding
    /deployment
    /incident-response
  /decisions
    /rfcs
    /adrs

Naming conventions

• Use lowercase with hyphens: checkout-service-runbook.md not CheckoutServiceRunbook.md
• Prefix with document type when browsing matters: rfc-002-new-auth-flow.md
• Include dates for time-sensitive docs: 2024-03-postmortem-checkout-outage.md
• Use index files for directories: every folder gets a README.md or index.md

Search and discoverability

Documentation that can't be found doesn't exist. Improve discoverability with:

1. Tags/labels - Consistent tagging taxonomy across all docs (system, team, type)
2. Cross-links - Every doc links to related docs. Post-mortems link to runbooks. RFCs link to ADRs. How-to guides link to reference pages.
3. A single entry point - One "documentation home" page that links to all categories with brief descriptions
4. Search optimization - Use descriptive titles, include synonyms in the first paragraph, use standard terminology

Documentation culture

The documentation-as-code approach

Treat docs like code:

• Version controlled - Store in git alongside the code they describe
• Reviewed - Documentation changes go through PR review
• Tested - Links are checked, code examples are validated
• Deployed - Published automatically via CI/CD to a docs site

Making documentation a habit

Documentation doesn't happen by default. Build it into workflows:

Ownership model

Every document needs an owner. Without ownership, docs rot.

Reducing friction

The biggest enemy of documentation is friction. Reduce it with:

• Templates - Pre-built templates for every doc type (RFC, post-mortem, runbook, ADR)
• Automation - Auto-generate reference docs from code (API specs, config schemas)
• Low ceremony - A rough doc today is better than a perfect doc never written
• Visible wins - Celebrate when documentation prevents an incident or speeds up onboarding

Onboarding documentation

The 30-60-90 day guide

Structure onboarding docs around what a new hire needs at each stage:

Week 1 (Day 1-5): Setup and orientation

• Local dev environment setup (step-by-step, tested monthly)
• Key tools and access: list every tool with how to request access
• Team norms: communication channels, meeting schedule, PR conventions
• Architecture overview: one-page system diagram with brief descriptions

Month 1 (Day 6-30): First contributions

• "Good first issues" labeled in the issue tracker
• Code walkthrough of the main service the team owns
• Key contacts: who to ask about what
• Common workflows: how to deploy, how to run tests, how to debug

Month 2-3 (Day 31-90): Independence

• On-call training and runbook orientation
• Deep dives into complex subsystems
• Cross-team collaboration guides
• Contributing to documentation themselves (closing the loop)

Managing documentation debt

Documentation audit checklist

Run quarterly:

• Identify docs with "Last updated" > 6 months ago
• Check all links for 404s (automate this)
• Verify code examples still compile/run
• Remove docs for decommissioned systems
• Merge duplicate docs covering the same topic
• Update ownership for docs owned by people who left

The archive decision

Not everything needs to be kept current. Use this framework:

Tooling recommendations

Choose tools that reduce friction and support the documentation-as-code approach:

Post-Mortems

Blameless culture

Blameless post-mortems are the foundation of a learning organization. The principle is simple: people make mistakes because systems allow those mistakes to happen. Blaming individuals creates fear, suppresses reporting, and prevents the organization from fixing systemic issues.

Blameless does NOT mean:

• Nobody is accountable
• No changes are needed
• The incident was acceptable

Blameless DOES mean:

• We focus on system and process failures, not individual errors
• We assume everyone acted with the best information available at the time
• We create an environment where people report honestly without fear

Language guide

Severity framework

Use a consistent severity scale so post-mortems are prioritized appropriately:

The 5 Whys technique

The 5 Whys is a root cause analysis method. Start with the symptom and ask "why" repeatedly until you reach a systemic cause.

Example:

1. Why did users see 500 errors? Because the checkout service was returning errors.
2. Why was the checkout service returning errors? Because it couldn't connect to the database.
3. Why couldn't it connect? Because the connection pool was exhausted.
4. Why was the pool exhausted? Because a long-running query was holding connections.
5. Why was there a long-running query? Because the new report feature had no query timeout configured, and the ORM generated an unindexed full-table scan.

Root cause: Missing query timeout configuration and no index on the report query.

5 Whys pitfalls

• Stopping too early - "The config was wrong" is a symptom, not a root cause. Ask why the wrong config was possible.
• Single-threading - Most incidents have multiple contributing causes. Branch your "whys" when you hit a fork.
• Going too deep - "Why do humans make mistakes?" is too philosophical. Stop when you reach something you can fix with a concrete action.

Timeline writing

The timeline is the backbone of a post-mortem. Write it in UTC with this format:

## Timeline (all times UTC)

| Time | Event |
|---|---|
| 14:00 | Deploy of checkout-service v2.4.1 begins via CI/CD pipeline |
| 14:03 | Deploy completes. No alerts triggered |
| 14:15 | Monitoring shows p99 latency increase from 200ms to 1200ms |
| 14:18 | PagerDuty alert fires: "checkout-service latency > 1000ms" |
| 14:20 | On-call engineer acknowledges alert, begins investigation |
| 14:25 | Engineer identifies database connection pool exhaustion in metrics |
| 14:32 | Decision to rollback deploy. Rollback initiated |
| 14:38 | Rollback complete. Latency returns to normal within 2 minutes |
| 14:40 | Incident declared resolved |

Timeline tips

• Include detection time, investigation steps, decision points, and resolution
• Note who was involved at each step (by role, not name, in a blameless doc)
• Include things that didn't work ("Attempted restart, no improvement")
• Be precise about timing - round to the nearest minute

Action items that stick

The most common failure of post-mortems is action items that never get completed. Make them stick with this framework:

The SMART action item

Priority categories

• P0 - Immediate (within 1 week): Prevents recurrence of this exact incident
• P1 - Soon (within 1 month): Reduces blast radius or improves detection
• P2 - Planned (within 1 quarter): Systemic improvements and tech debt

Tracking action items

• Create tickets in the team's issue tracker immediately after the post-mortem meeting
• Link tickets back to the post-mortem document
• Review completion status in weekly team standup until all P0/P1 items are done
• Include action item completion rate in quarterly engineering metrics

Post-mortem review meeting

Agenda (45-60 minutes)

1. Summary and timeline walkthrough (10 min) - Author presents the incident
2. Root cause discussion (15 min) - Group validates or challenges the analysis
3. Action item review (15 min) - Assign owners and priorities for each item
4. Process improvements (5 min) - Meta-discussion on the incident response itself
5. Closing (5 min) - Confirm action items and document owner

Facilitation rules

• The facilitator is NOT the author - get a neutral party
• Redirect any blame to systems: "Let's focus on what the system allowed to happen"
• Time-box tangents: "Great point, let's capture that as an action item"
• End with appreciation: thank the responders and the author

RFCs and Design Docs

RFC vs Design Doc vs ADR

These terms are often used interchangeably but serve different purposes:

Rule of thumb: If the change affects more than one team's codebase or introduces a new technology, it's an RFC. If it's a complex feature within one team, it's a design doc. If it's a focused architectural choice, it's an ADR.

RFC lifecycle

Draft -> In Review -> Approved / Rejected / Withdrawn
                  \-> Needs Revision -> In Review (loop)

Draft phase

• Author writes the initial proposal using the RFC template
• Share early with 1-2 trusted reviewers for a "pre-review" before formal circulation
• Include a decision deadline (typically 1-2 weeks from circulation)

Review phase

• Circulate to all stakeholders via the team's standard channel (email, Slack, doc comments)
• Reviewers leave inline comments on specific sections
• Author responds to every comment - either incorporate feedback or explain why not
• Schedule a synchronous review meeting only if async comments reveal fundamental disagreements

Decision phase

• The designated decision-maker (tech lead, architect, or committee) makes the call
• Document the decision and reasoning at the top of the RFC
• If rejected, explain why clearly - rejected RFCs are valuable institutional knowledge

Writing effective motivation sections

The motivation section is the most important part of an RFC. It must answer three questions:

1. What problem exists today? Describe the pain concretely with data if possible. "API latency p99 has increased from 200ms to 800ms over the last quarter due to N+1 queries in the order service" is better than "performance is degrading."

2. Why does it matter? Connect the problem to business or engineering outcomes. "This latency increase has caused a 12% drop in checkout completion rate."

3. Why now? Explain the urgency. Is there a deadline, a scaling cliff, or a dependency that makes this the right time?

Alternatives section best practices

The alternatives section proves you've done your homework. For each alternative:

• Name it clearly - "Alternative A: Migrate to GraphQL" not "Another option"
• Give it a fair shot - Describe it as if you were proposing it
• List honest pros and cons - If an alternative has no pros, you haven't thought hard enough
• Explain why you didn't choose it - Be specific about the deciding factor

Minimum: 2 alternatives. If you can only think of one alternative ("do nothing"), you haven't explored the solution space enough.

Design doc specifics

Design docs are lighter than RFCs. Key differences:

• Shorter review cycle - 2-5 days instead of 1-2 weeks
• Narrower audience - Team members and direct stakeholders
• More implementation detail - Include API schemas, data models, sequence diagrams
• Less process - No formal approval committee, team lead signs off

Design doc template additions (beyond RFC template)

## API design
<Endpoint definitions, request/response schemas>

## Data model
<Schema changes, new tables/collections, migration plan>

## Sequence diagram
<Key flows showing component interactions>

## Testing strategy
<How will this be tested? Unit, integration, E2E coverage plan>

## Observability
<What metrics, logs, and alerts will be added?>

Review etiquette

For reviewers

• Be specific - "This doesn't handle the case where X" is useful. "I don't like this" is not.
• Distinguish blocking vs non-blocking - Prefix with "Blocking:" or "Nit:" or "Question:"
• Suggest, don't prescribe - "Have you considered X?" not "You should do X"
• Focus on the proposal, not the person - "This approach has a scalability risk" not "You didn't think about scale"
• Respond within the deadline - No response is implicit approval in most RFC processes

For authors

• Respond to every comment - Even if just "Acknowledged, updated section 3"
• Don't be defensive - Reviewers are improving the proposal, not attacking you
• Update the doc, not just the comment thread - The doc is the source of truth
• Call out material changes - If review feedback significantly changes the proposal, re-notify reviewers

Common RFC anti-patterns

Runbooks

What makes a good runbook

A runbook is a set of step-by-step instructions for performing an operational task. The golden test: can an engineer who has never seen this system follow the runbook at 3 AM and resolve the issue without calling anyone?

The three rules

1. No ambiguity - Every step has one interpretation. "Check the logs" fails. "Run kubectl logs -n production deployment/checkout --tail=100" succeeds.

2. No assumed knowledge - Don't assume the reader knows where things are, what tools to use, or what "normal" looks like. Show expected output.

3. No dead ends - Every step has a "what if this doesn't work" path. The reader should never be stuck without a next action.

Runbook anatomy

# Runbook: <Descriptive title>

**Owner:** <team name>
**Last verified:** <date someone actually ran through this>
**Estimated time:** <realistic duration>
**Risk level:** Low | Medium | High
**Related alerts:** <list PagerDuty/Datadog alert names that lead here>

## When to use
<Specific trigger conditions. What alert fires? What symptom appears? What request
comes in? Be explicit about when this runbook applies AND when it does not.>

## Prerequisites
- [ ] Access to <specific system> via <specific tool>
- [ ] Permissions: <specific IAM role, kubectl context, or credentials>
- [ ] Tools installed: <specific CLI tools with version requirements>

## Steps

### Step 1: <Verb phrase describing the action>

<Exact command or UI navigation path>

```bash
<command to run>

Expected output:

<what the output should look like when things are working>

If this fails:

• If you see <error message>:
• If the command hangs:
• If output differs from expected:

Step 2: ...

Verification

<How to confirm the procedure worked. Specific checks, not "verify it's working.">

Rollback

## Writing effective steps

### Command formatting

Always provide complete, copy-pasteable commands:

**Bad:**

**Good:**
```bash
# Check pod status in the production namespace
kubectl get pods -n production -l app=checkout-service

# Expected: All pods should show STATUS=Running, READY=1/1
# If any pod shows CrashLoopBackOff, proceed to Step 3

### Step 3: Assess memory usage

```bash
kubectl top pods -n production -l app=checkout-service

### Variable substitution

When commands need environment-specific values, use clear placeholders:

```bash
# Replace <ENVIRONMENT> with: production | staging
# Replace <POD_NAME> with the output of Step 1
kubectl exec -it <POD_NAME> -n <ENVIRONMENT> -- /bin/sh

Runbook created -> Author verifies -> Peer review -> Published
     ^                                                    |
     |                                                    v
     +-- Update needed <-- Quarterly review <-- In use <--+

# PagerDuty / Datadog / Grafana alert annotation
annotations:
  runbook_url: "https://wiki.internal/runbooks/checkout-high-latency"
  summary: "Checkout service p99 latency > 1000ms for 5 minutes"