Browse
On-Call & Alerting Design
Designing alerts that page for what actually needs a human, and structuring on-call sustainably.
What it is
Alerting design decides what actually pages a human being versus what just gets logged or dashboarded — getting this wrong in either direction (too noisy or too quiet) undermines the whole on-call system.
Key points
- Alert on symptoms, not causes: page on "error rate is elevated" (a symptom users actually feel), not on every possible underlying cause individually — this keeps the alert count manageable as the system grows.
- Actionability: every page should have something a human can actually do about it right now — if an alert never leads to action, it should be a dashboard metric, not a page.
- Alert fatigue from noisy or low-signal alerts is the primary failure mode — it trains on-call engineers to ignore or delay reacting to alerts, including the real ones.
- Sustainable on-call design also includes rotation length, escalation policies (who gets paged next if the first responder doesn't ack), and time-of-day awareness — this is a people-process problem as much as a technical one.