r/rootly•
jim_at_rootly

Alert engineering reduces MTTR faster than adding more monitoring tools

MTTR comes down to detection and diagnosis time, roughly half of total recovery on most teams. The fix is not more monitoring tools. It is making every alert actionable so engineers stop guessing after the page fires.

Start by listing every active alert across PagerDuty, Opsgenie, status pages, and Slack channels. Most teams run three to five times more alerts than needed because each tool adds its own defaults. Export everything into one place, then for each alert record how often it fired in the past 30 days and whether it led to a real incident or was noise. Alerts firing more than once daily without causing incidents go straight to tuning.

Categorize what remains. Detection alerts tell you something broke. Diagnostic alerts narrow down what broke after detection fires. Status alerts confirm recovery. Most teams underinvest in diagnostics, which means responders spend half their time figuring out scope before they can start fixing anything. Rootly groups correlated events under shared incidents so response stays scoped to one problem instead of scattered across twelve individual pages.

Every alert needs to pass one question: does this lead to a specific action? If the answer is no, remove it regardless of how alarming it sounds. Error rate increase is a symptom. Checkout latency over 500ms maps directly to lost revenue. Impact based alerts are shorter, louder, and always point to someone who should act right now.

TLDR: Cut MTTR by auditing the full alert inventory, removing symptom only monitors, keeping only customer impact alerts, and grouping correlated events under shared incidents while targeting three or fewer alert pages per incident with weekly cleanup of stale thresholds.

Posted Sep 16, 2026, 2:04 PM UTC

13
Loading comments…