Adding custom status message to Prometheus alerts
Hello,
I'm moving towards Prometheus for data collection and alerting needs, and I was wondering what would be the best way to enrich alerts with some custom status messages.
Let's say I have a service doing some operation periodically, which can fail for one or more reasons. I was thinking of exporting a 0/1 gauge with a "status" label, which would either have the value of "ok", or the reason the last loop failed. Is this the preferred way to do it?
Thanks!
Posted Feb 8, 2019, 8:57 AM UTC
Typically, you don't do this kind of thing in Prometheus. Rather, you have separate counters for each type of failure. The metric name is your "status label". It's best to avoid gauges, because the status output can be missed if a scrape is fails.
You then create alerts for each of these. It's done this way to reduce the instrumentation overhead. By moving the description and calculation to the alert, you can make more intelligent decisions.
Hey, thanks for the reply.
The disadvantage I see with this is that if the operation can fail for let's say 5 different reasons, then I have to create/register/pass in 5 different Counter instances in the application, which seems clunkier than using 1 counter and changing the labels.
Yes, it seems that way, but in order to know that there are no errors, you still have to init all label possibilities.
While I don't recommend it, you could do something like this:
It has the advantage that you can aggregate all failure types if you want, but you have to be careful to initialize all of the error types at startup. It also allows you to create a simple error rate ratio like this: