Help with Nagios Loads

Hi Guys,

I know there are quite a few Nagios users on r/sysadmins and I am hoping someone can help out with some issues we have.

I stepped in to the environment here only a few months ago but Nagios has been pretty consistent with generating false alerts. I'm pretty sure its a performance issue and that we are monitoring too many things but so far I haven't gotten very far with the guy responsible for taking care of it.

Here is a sample of the system health

We're using NagiosXI and we setup the VM appliance which they provide. The other admin monitors the CPU usage in vCenter, see's it is at 70% and concludes that Nagios is fine.

I am not convinced. What are those load values? Why are they red? Service check max value of 52s - that can't be good.

The event log is full of services timing out, or having insane response times of 3000+ms. I can assure you our network is healthy, we've got 10Mbps links minimum out to all locations. I'm looking at one event:

2011-07-06 08:26:33HOST ALERT: BEEPBEEPBOOP3;DOWN;SOFT;1;PING CRITICAL - Packet loss = 0%, RTA = 5521.26 ms

That host is in the same Datacenter as the Nagios box.

Last night a few of us damn near had a heart attack because it started reporting all the SAN volumes were offline - which thank Odin they were not.

I tossed out a few ideas like reduce the number of services we check, move to a distributed environment - all which were met with "70% CPU in vCenter" sigh

Problem Resolved* We moved our Nagios VM to a shiny new ESX host and also added a seconded vCPU. The average load has dropped from 10+ down to -2.

Thanks for everyone's help!

Posted Jul 6, 2011, 3:24 PM UTC · edited

1328
RemyJeAKA Raszh•

Consider distributing the monitoring across multiple Nagios instances and deliver the results to the central host as passive checks.

5
[deleted]•

Where can I read about implementing this? Our system could use a similar upgrade.

1
[deleted]•

No help here, just came here to say BEEPBEEPBOOP3 is an awesome hostname.

2
fidotasDevOp Evangalist•

To make sense of the load values, how many vCPUs does the machine have assigned to it? How many physical CPU's in the ESX host?

1
simulated_conscienceOPIT Manager•

It has 1 vCPU - pretty sure it was downloaded like that. The host it is on right now has 2x Intel E5450 or 8 cores total. There are 20 running VMs which is probably too many. Step #1 - I'll move it to a box with 24 cpus.

1
fidotasDevOp Evangalist•

Hmm. After moving it to less congested hardware I'd consider a second (or more) vCPU for the box, an average load of 10 on a single CPU is high. Too high. That load indicates that there is a backlog of processor work waiting to execute. I'm fairly sure Nagios forks to execute the plugins so they should benefit from additional vCPUs to enable concurrency.

You should also check the "CPU Ready" metric in your vCenter performance graph for the Nagios VM. If this is above 100 consistently you need to take it to your VM admin; it indicates the VM is struggling to find available CPU cycles to execute because of competition with other VMs.

2
simulated_conscienceOPIT Manager•

We don't have a dedicated VM admin sadly, just a team of guys that do everything.

I took a look - sure enough it is consistently over 100.

1
fidotasDevOp Evangalist•

Yeah, you have CPU contention issues on that physical box then. Move VMs to other physical boxes (or this one).

You should see the load drop almost immediately. Watch it and consider a second vCPU if the load stays consistently above 5.

As a general rule Nagios can handle thousands of service checks on a single host. It's all down to the hardware capabilities and the type of checks you're performing.

1
mattelmoreSysadmin•

I keep my Nagios box separate from my virtual infrastructure for a few reasons. One is for the USB hookup to my cellular modem, and also so if the VMware host it is on takes a dive, I still have something to monitor it.

I am monitoring 780 services on 109 hosts on a quad-core Xeon 2.8GHz with 4G RAM. It's surprising how much load Nagios can generate.

11:14:46 up 19 days, 18:23, 1 user, load average: 0.71, 0.69, 0.84

So I wouldn't be surprised to see a VM struggle. Give it some more CPU and see if that helps.

1
rsfkykillerSystems Engineer•

Completely serious question, how do you monitor the nagios box in case IT goes down?

2
TheSojournerSr. Systems Engineer •

A second Nagios box, or even just a script on another host that checks if it's alive.

1
mattelmoreSysadmin•

At one point, I did have a VM setup running Nagios that monitored only the main Nagios box and did email alerts. I've since taken it down but should probably bring it back up again.

1
Apologetic_Jerk•

Which method are you using for checks? Are you using NRPE?

1
simulated_conscienceOPIT Manager•

That I do not know. I wasn't here when they set it up, and usually when it needs to be changed it falls on the other guy.

1
Apologetic_Jerk•

This is the very possible reason your load is climbing so high. Are you swapping when the CPU load increases?

1
simulated_conscienceOPIT Manager•

Doesn't look like it, whenever I check swap file usage is 0.

1
cjdanielLinux Admin•

If you're getting alerts about SOFT states, you might have Nagios misconfigured.

Nagios has a bit of built-in false alarm prevention. It has two types of states -- SOFT and HARD. SOFT states are designed to be reached when, for example, a single check fails, and a service goes from "ok" to "critical". With the proper configuration, you won't get an alert until a few more checks fail, and the service reaches a HARD "critical" state.

An example configuration would be to set "max_check_attempts" for a service to 5 -- meaning, the check of this service must fail 5 times in a row for a HARD critical state to be reached, and therefore a notification sent.

That said, on my Nagios system, monitoring 63 hosts and 254 services all on the same LAN, I rarely get any even SOFT state changes in the log.

1
simulated_conscienceOPIT Manager•

Yes this is part of it. For majority of the false alerts we get the max_check_attempts is set to 1 - I just discovered that this morning.

Its set on 1 only for critical things like SAN volumes, and I agree that it should be set higher. But I do still see a tonne of soft state changes and those need to be addressed. Thanks to some of the other responses I've identified that CPU load is WAAAAAY to high, 10+ and it should be under 1. This looks like a symptom of CPU contention on the ESX host, and also possibly needing another vCPU.

1
choffman•

You may also want to start up "top", sort by CPU utilization, and see if one particular script/check is causing a problem.

I've also seen weird things like timeouts happen when IPv6 wasn't working correctly.

1
zbfw•

Load average is way too high. You may want to go through this

http://www.linuxjournal.com/magazine/hack-and-linux-troubleshooting-part-i-high-load

It's possible that disk is not keeping up with Nagios. You can get low CPU util and high load average when it's waiting on something else all the time.

1
bandman614Standalone SysAdmin - LOPSA•

If you're using NagiosXI, you should really talk to them about it first. That's a supported product. Take advantage of the money you're paying for it.

1
simulated_conscienceOPIT Manager•

If it was me being tasked to solve this problem that is what I would be doing, and it is has been suggested to my colleague on multiple occasions.

He's one of those guys that just needs to figure it out himself I guess.

1
[deleted]•

You might want to have a look at check_mk its alternative to NRPE (and more) that scales a bit better. We are currently using it monitoring >500 hosts and >20000 services.

We do however have one instance of nagios for every datacenter (3 of them) with one central taking care of aggregating all the data and displaying it.

1
brain_juice•

+1 for check_mk and omd.

1
spychipperSr. Sysadmin•

If your using a high number of snmp checks then what I found to cut my load avg by about 80% was to avoid check plugins that call snmpget from the host system and to favor ones that use the perl snmp libs instead, saves a good chunk of startup costs for the plugin, esp. on ones that are written to do a single check on a single host, those are bad, bad, bad.

A remote plugin that reports in will still be needed sooner or later, but this can help as a stop gap or to lighten the load if you require active checks from the nagios host for some things.

1