⬡ Senior Interview Prep
⬡ Hub

Real-World Incident Postmortems — Learning from Major Outages

Target Level: Staff / Principal SRE & DevOps — 15+ Yrs Experience

Interviewers at the senior level often ask: "Tell me about a complex incident you've led." They want to see your incident command, root cause analysis, and systematic thinking. Studying major industry outages gives you a shared vocabulary and demonstrates depth. Below are the most instructive postmortems with engineering and cultural takeaways.

1. AWS Kinesis (November 2020) — 18-hour outage

Timeline

Root Cause

A single minor Kinesis service was deployed with insufficiently tested changes. The deployment mechanism itself didn't have a tested rollback plan for the scenario where the control plane is unavailable. The incident exposed a meta-failure: the tool used to fix the system depended on the system being fixed.

Lessons for Senior Engineers

  1. Always have an "out-of-band" path to administer your systems — a separate control plane that doesn't depend on the primary system.
  2. Test rollbacks in production. If you can't roll back through the normal pipeline, you don't have a rollback.
  3. Defense in depth for control planes: The deployment orchestrator should be a separate system with its own lifecycle.
  4. Incident communication: AWS's transparent, hourly updates during the outage were a model for how to communicate during a crisis.

Interview Discussion Point

"How would you design a deployment system that can recover even when the primary control plane is down?" This tests understanding of: separate management networks, out-of-band access, static binary deployments, and runbook-driven manual recovery paths.

2. GitHub (October 2021) — Secret key loss causes 24+ hour code freeze

Timeline

Root Cause

A single secret key was stored in one engineer's memory and a running process — not in a secret management system. The key was inadvertently destroyed during a routine hardware replacement because its existence wasn't documented.

Lessons for Senior Engineers

  1. Secrets must never exist in only one place. Use HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault for all secrets — including PKI root keys.
  2. Document key metadata. Every secret should have: owner, rotation schedule, which services depend on it, recovery procedure.
  3. Practice disaster recovery for PKI. Run a game day where you deliberately lose the PKI root key and practice rebuilding it.
  4. Short-lived certificates are safer. With short TTLs (24h), even if keys leak, the blast radius is limited. But short TTLs require automated renewal — no manual cert management.

Interview Discussion Point

"Design a secrets management and PKI strategy that prevents this class of failure." Expected answer: Vault PKI with intermediate CAs per service, short-lived certs (24h), automated renewal via cert-manager, root CA offline in a safe with M-of-N quorum to access.

3. Fastly CDN (June 2021) — Global CDN outage

Timeline

Root Cause

A configuration change was deployed to all edge nodes simultaneously with no canary or staged rollout. The change validation in the CI pipeline didn't catch the bug because it tested against a single customer configuration, not the global multi-tenant configuration matrix.

Lessons for Senior Engineers

  1. Canary every configuration change. Even "trivial" config changes should roll out incrementally (1% → 5% → 25% → 100%).
  2. Test the intersection, not just the union. The bug was in how the config template interacted with a specific customer — testing each config in isolation wouldn't catch it. Use pairwise testing or property-based testing for config templates.
  3. Feature flags for customer configuration. If a specific customer's config causes issues, a kill switch should be available to disable that config without redeploying.
  4. Graceful degradation. Edge nodes should have a "stale config" fallback — serve the last-known-good configuration instead of entering a no-traffic state.

Interview Discussion Point

"Design a CDN or edge-config deployment system that prevents global outages." Expected answer: staged rollout, canary POP, automated health checks before promotion, last-known-good fallback, per-customer kill switch, separate control plane from data plane.

4. Google Cloud (August 2022) — Multi-region DNS failure

Timeline

Root Cause

A single DNS system was a single point of failure across multiple regions. The maintenance procedure had no canary or staged rollout. DNS had "hidden" dependencies — services assumed DNS was always available and had no fallback for DNS failure (no cached DNS entries, no static IP references).

Lessons for Senior Engineers

  1. DNS is the most critical infrastructure — design for DNS failures. Implement: client-side DNS caching (HOSTALIASES, /etc/hosts fallback), retry with jitter, use of IP addresses where possible in internal configurations.
  2. Never assume DNS is always available. Every service should have a local DNS cache with adequate TTL, and should handle DNS failures gracefully (circuit breakers, stale cache serving).
  3. Canary for infrastructure procedures too. Not just application code — DNS changes, load balancer configs, firewall rules should all be canary-deployed.
  4. Cross-region architectures should not have a single DNS dependency. Use multiple DNS providers, client-side service discovery (Consul), or DNS resolvers per region.

5. CrowdStrike (July 2024) — Global Windows system crash

Timeline

Root Cause

The content update contained a malformed input that crashed the Windows kernel driver. The validation pipeline tested the update format but didn't test it against all Windows configurations. The update was released simultaneously to all endpoints with no canary rings, no gradual rollout, and no automated quality gates.

Lessons for Senior Engineers

  1. Rapid-response updates need the same safety as regular updates. Even "emergency" antivirus signature updates should have staged rollout (1% → 10% → 100% with monitoring between stages).
  2. Kernel-level software requires extreme validation. Any code that runs in kernel space should pass multiple validation stages: static analysis, fuzz testing, integration testing on reference hardware, and staged rollout.
  3. Design for degraded operation. If the security agent crashes, the system should survive (with reduced security posture) — not crash entirely. The Falcon driver should have had a failsafe mode.
  4. Real user monitoring for deployment safety. CrowdStrike's first indication of a problem was millions of crashed machines — there was no pre-deployment canary or synthetic monitoring.

Interview Discussion Point

"Design a deployment system for kernel-level agents that prevents a repeat of CrowdStrike 2024." Expected answer: staged rollout rings (canary → internal → 1% → 10% → 100%), automated rollback on crash detection, synthetic test environments that mirror customer configurations, circuit breaker on agent install (auto-disable if system metrics degrade), and dual-boot with last-known-good agent.

6. Postmortem Culture — What Senior Engineers Should Know

Building a Blameless Postmortem Culture

  1. Focus on systems, not people. The question is always "what in the system allowed this to happen?" not "who caused it?"
  2. Action items must have owners and deadlines. A postmortem without concrete, assigned, tracked actions is theater. Classify items: P0 (must fix within 7 days), P1 (30 days), P2 (90 days).
  3. Track postmortem completion rates. If action items from previous postmortems aren't being completed, your reliability culture has a hole.
  4. Postmortems should be searchable and accessible. Every postmortem is a learning resource. Tag by: service, severity, root cause type, team. Run quarterly trend analysis: what types of failures are repeating?
  5. Public postmortems build trust. Writing public postmortems (as AWS, Google, GitHub, Fastly, and CrowdStrike all did) demonstrates transparency and builds customer trust — and holds the team to a higher standard.

Postmortem Template

## Summary
[1–2 line description of what happened]

## Impact
- Services affected:
- User impact:
- Duration:
- Cost/revenue impact:

## Timeline
[Key events with timestamps]

## Root Cause
[Deep technical analysis]

## Contributing Factors
[Systemic issues that allowed the root cause]

## Detection
How was this discovered? How long between incident start and detection?

## Resolution
How was it fixed? How long did the fix take?

## Lessons Learned
[What did we learn?]

## Action Items
| Item | Owner | Due | Type (P0/P1/P2) |
|------|-------|-----|------------------|
| ...  | ...   | ... | ...              |

7. Interview Questions on Incident Management

Q1: Walk me through how you would run an incident bridge for a P0 outage.

Answer: 1) Declare the incident — engage on-call via PagerDuty, open a Zoom/Teams bridge, create a Slack channel. 2) Assign roles: Incident Commander (IC — runs the bridge, makes decisions, tracks timeline), Scribe (documents events and actions), Subject Matter Experts (individuals diagnosing specific systems), Comms Lead (handles internal/external status updates). 3) Establish the severity — SEV1 = critical customer impact, SEV2 = degraded functionality. 4) Start the timeline — document every action taken, every change made, every observation. 5) Parallelize investigation — SMEs diagnose independently while the IC tracks the big picture. 6) Attempt mitigation first — rollback, scale up, route around — find the root cause after the bleeding stops. 7) Communicate — status page updates every 15 minutes. 8) When service is restored — monitor for 30 minutes before closing the incident. 9) Postmortem — schedule within 48 hours.

Q2: Your postmortem finds the same root cause pattern for the third time this quarter. What do you do?

Answer: This signals a systemic weakness that individual action items aren't solving. Escalate: (1) Trace the action items from the previous two postmortems — were they completed? If not, the process is broken. (2) If they were completed but didn't prevent recurrence, the mitigation was insufficient — propose a more fundamental solution. (3) Consider a "lock" — temporarily prevent any changes to the affected system until a permanent fix is in place. (4) Re-evaluate if this is a training, tooling, or architecture problem. A pattern of repeat incidents in database connection pool exhaustion suggests you need a connection pool middleware or auto-scaling connection pools, not just more runbook documentation. (5) Propose an error budget policy change — if a service has 3+ repeat incidents, its error budget is reduced, or a mandatory architectural review is triggered.

Q3: You're joining a company with no incident management process. How do you build one from scratch?

Answer: Phase 1 (Week 1): Document the current state — how do people currently handle incidents? What's painful? Find 1–2 willing early adopters. Phase 2 (Month 1): Define severity levels (SEV1–5). Create a simple runbook for SEV1: "Who to page, where to meet, how to communicate." Use existing tools (Slack, etc.) — don't buy new tools yet. Phase 3 (Month 2–3): Introduce the IC role. Run a tabletop exercise. Add a simple postmortem template. Phase 4 (Month 4–6): Standardize on-call rotations. Add monitoring and paging. Introduce error budgets. Phase 5 (Month 7+): Automate — auto-create incident channels, auto-document timeline, integrate with monitoring. Key: don't over-engineer at first — a simple, consistently followed process beats a complex but ignored one.

← Back to Interview Hub