How do you guys monitor a bunch of services running in a cluster?

Let's take this example, we have service deployed that scales from 1 to N, this can be very dynamic. Can be a part of auto scaling. We use Nagios as your primary alerting tool. Some are system checks like Load, Mem, Disk etc and some are application checks. Code emits metrics to Graphite and we have Nagios Checks that polls the Graphite and alerts when the threshold crosses certain value.

One of the problems that we are facing right now is if one the checks fail, it fails in all the boxes. For example, recently we had a DB related issue and because of that, we got tons of alerts from all the boxes. We have a cross region service running and at that time we had like 20 Boxes running and we got around 20 notifications both for alert and after that, once the service was restored we got recovery emails. The problem with this approach is that we might miss some critical notification because of that. One possible approach is that, rather than monitoring N boxes, you just monitor a just couple of boxes rather than N. Another approach would be measure overall metrics of the service, like overall requests per second etc.

I am aware of monitoring tools like Prometheus / Sensu not sure if that might help or not.

End Goal is less spam and actionable alerts. Also, we extensively use Spot Boxes. This service is deployed on spot fleet right now. We have a simple approach any node comes up it emits messages and your monitoring services take care of adding it and removing it when the spot is about to terminate. Please suggest some books, articles, solutions monitor application in a distributed systems.