r/devops•
ethCore7

Adding custom status message to Prometheus alerts

Hello,

I'm moving towards Prometheus for data collection and alerting needs, and I was wondering what would be the best way to enrich alerts with some custom status messages.

Let's say I have a service doing some operation periodically, which can fail for one or more reasons. I was thinking of exporting a 0/1 gauge with a "status" label, which would either have the value of "ok", or the reason the last loop failed. Is this the preferred way to do it?

Thanks!

Posted Feb 8, 2019, 8:57 AM UTC

59
SuperQue•

Typically, you don't do this kind of thing in Prometheus. Rather, you have separate counters for each type of failure. The metric name is your "status label". It's best to avoid gauges, because the status output can be missed if a scrape is fails.

app_foo_failures_total 0
app_bar_failures_total 0

You then create alerts for each of these. It's done this way to reduce the instrumentation overhead. By moving the description and calculation to the alert, you can make more intelligent decisions.

2
ethCore7OP•

Hey, thanks for the reply.

The disadvantage I see with this is that if the operation can fail for let's say 5 different reasons, then I have to create/register/pass in 5 different Counter instances in the application, which seems clunkier than using 1 counter and changing the labels.

1
SuperQue•

Yes, it seems that way, but in order to know that there are no errors, you still have to init all label possibilities.

While I don't recommend it, you could do something like this:

app_completed_total 0
app_failures_total{status="foo"} 0
app_failures_total{status="bar"} 0

It has the advantage that you can aggregate all failure types if you want, but you have to be careful to initialize all of the error types at startup. It also allows you to create a simple error rate ratio like this:

rate(app_completed_total[5m]) / sum without (status) (rate(app_failures_total[5m]))
2
pymag09•

You definitely can do this. Let's assume we have created metric service_status. Its value is 0 or 1 (down or up). This metric can have lable status, for example "unknown error", "no such file or directory", "no free space" and so on. To access labels in alert template you can use .CommonLabels variable. Example:

​

.CommonLabels.status

or list all lables

{{ range .CommonLabels.SortedPairs }} • *{{ .Name }}:* \{{ .Value }}``

​

or {{ $labels.status }} in prometheus config file.

annotations:

summary: 'bla-bla {{ $labels.status }}'

2
mariusvoila•

Sorry, no answer for you but want to follow the discussion because I'm interested too

1
clevarius•

Me too 😉

1
ethCore7OP•

You might want to check the thread again, as there have been some new replies ;)

1
ethCore7OP•

You might want to check the thread again, as there have been some new replies ;)

1
mariusvoila•

Thanks a lot!

1