Help with Nagios Loads
Hi Guys,
I know there are quite a few Nagios users on r/sysadmins and I am hoping someone can help out with some issues we have.
I stepped in to the environment here only a few months ago but Nagios has been pretty consistent with generating false alerts. I'm pretty sure its a performance issue and that we are monitoring too many things but so far I haven't gotten very far with the guy responsible for taking care of it.
Here is a sample of the system health
We're using NagiosXI and we setup the VM appliance which they provide. The other admin monitors the CPU usage in vCenter, see's it is at 70% and concludes that Nagios is fine.
I am not convinced. What are those load values? Why are they red? Service check max value of 52s - that can't be good.
The event log is full of services timing out, or having insane response times of 3000+ms. I can assure you our network is healthy, we've got 10Mbps links minimum out to all locations. I'm looking at one event:
2011-07-06 08:26:33HOST ALERT: BEEPBEEPBOOP3;DOWN;SOFT;1;PING CRITICAL - Packet loss = 0%, RTA = 5521.26 ms
That host is in the same Datacenter as the Nagios box.
Last night a few of us damn near had a heart attack because it started reporting all the SAN volumes were offline - which thank Odin they were not.
I tossed out a few ideas like reduce the number of services we check, move to a distributed environment - all which were met with "70% CPU in vCenter" sigh
Problem Resolved* We moved our Nagios VM to a shiny new ESX host and also added a seconded vCPU. The average load has dropped from 10+ down to -2.
Thanks for everyone's help!
Posted Jul 6, 2011, 3:24 PM UTC · edited
Consider distributing the monitoring across multiple Nagios instances and deliver the results to the central host as passive checks.
Where can I read about implementing this? Our system could use a similar upgrade.
NSCA, DNX, etc.
http://nagios.sourceforge.net/docs/3_0/distributed.html
http://exchange.nagios.org/directory/Addons/Distributed-Monitoring