How I Automated the First Hour of Incident Investigation Across 550 Microservices

You cannot brute-force a microservices outage. I learned this the hard way.

We monitor 550+ services. When something breaks, the telemetry volume is genuinely impossible to correlate manually. By the time an engineer opens their laptop, tabs into five dashboards, and forms a hypothesis, the first hour is already gone.

So I spent the last 2 weeks building a system that owns that first hour entirely.

Here's how it actually works.

Every 5 minutes, a Lambda fires and kicks off detection. Instead of querying each service individually, the engine fires 9 bulk PromQL queries and gets back error rates, latency, RPM, and Apdex for all 550 services at once — each with 1-hour and 24-hour baselines already baked in. The whole detection pass takes about 3 seconds.

Six signal analyzers then run across that data. Not just "error rate is high." More like — are 50% of ALB errors coming back with no backend response at all? That means pods are completely gone, not just slow. Is latency above 1 second AND above 2x both baselines simultaneously? That's not a fluke. Is RPM near zero compared to baseline? The service is probably down.

Before anything escalates, three filters run to kill the noise. Has this service been hitting this threshold regularly over the last 7 days? Did we already alert on this exact signal in the last 6 hours? Is the error count less than 0.1% of total request volume? Five errors on a service handling 10,000 requests in 5 minutes is just noise.

Whatever survives that goes to an LLM validation gate. The model gets 7 days of anomaly history for the service plus a 168-point hourly metric time series. It classifies each anomaly as real, recurring, or noise. Recurring means the same pattern fires at the same hour across multiple days — usually a batch job or cron — and it gets suppressed automatically.

I tested this on 20 production anomalies. 12 real, 8 suppressed, 0 wrong calls. 60% noise reduction before a single human sees anything.

For anomalies that make it through, enrichment fires asynchronously — 8 parallel queries with 60-second timeouts. Deployments in the last 60 minutes. Error traces with pod names, exceptions, and AWS resource tags. Dependency graph with upstream callers and downstream services. Slow DB query counts against 24-hour baseline. ALB logs by domain. AWS infra health across 12 services. And CloudTrail infrastructure changes in the 2-hour window before the anomaly.

For high and critical anomalies, a full RCA investigation kicks off. Source control commits, application logs, WAF data — all cross-correlated by the LLM, which returns a root cause, remediation steps, and a confidence score. A ticket only gets created if confidence is 70% or above. No ticket for a hunch.

If the RCA points to a bad deployment, the system proposes a rollback. If it finds OOMKilled or CrashLoopBackOff, it proposes a pod restart. Every proposed action requires admin approval, expires in an hour, and is fully audited — who approved it, when, and what happened.

The whole thing — from Lambda trigger to a threaded RCA notification in our engineering chat — runs without anyone touching anything.

The best platform engineers aren't going to be the ones who debug the fastest. They're going to be the ones who build the systems that do it for them.

If you're working on something similar, I'd genuinely like to compare notes.