Comments on: Debugging Nagios performance problems http://www.barkingseal.com/2009/04/debugging-nagios-performance-problems/ Applied Trust off-leash: IT infrastructure, security, and performance Thu, 29 Jul 2010 20:30:05 +0000 hourly 1 http://wordpress.org/?v=3.0 By: Mike Kniaziewicz http://www.barkingseal.com/2009/04/debugging-nagios-performance-problems/comment-page-1/#comment-67 Mike Kniaziewicz Fri, 19 Jun 2009 00:14:52 +0000 http://www.barkingseal.com/?p=725#comment-67 Great article Ben! I like the way you draw the picture then present solutions. We noticed an increase with our load averages over a period of time. My enterprise monitors over 1700 hosts and 3100 services and is growing. We developed the same type of solutions you propose in your article. Performing a "lsof" we were able to identify directories being hit the hardest, so we placed those directories on SAN disks, which reduced the load dramatically. Directories included /usr/lib/mysql and /var/opt/nagios. To help reduce the load even further we developed time periods for monitoring certain aspects of the enterprise. We only monitor development and test servers during business hours. We also only monitor services that impact production during the night time hours. We found that receiving a notification at 2 AM about a high CPU was not the answer, so we track these problems then work on solutions the following morning. You may also want to turn-up monitoing on systems that the end users are reporting problems. You can create custom templates and even call them problem_child.cfg. You would add hosts to this template only when a need occurs to step up the monitoring. Thanks Ben for the article and I found it quite insightful. Keep up the fine work. Great article Ben! I like the way you draw the picture then present solutions.

We noticed an increase with our load averages over a period of time. My enterprise monitors over 1700 hosts and 3100 services and is growing. We developed the same type of solutions you propose in your article.

Performing a “lsof” we were able to identify directories being hit the hardest, so we placed those directories on SAN disks, which reduced the load dramatically. Directories included /usr/lib/mysql and /var/opt/nagios.

To help reduce the load even further we developed time periods for monitoring certain aspects of the enterprise. We only monitor development and test servers during business hours. We also only monitor services that impact production during the night time hours. We found that receiving a notification at 2 AM about a high CPU was not the answer, so we track these problems then work on solutions the following morning.

You may also want to turn-up monitoing on systems that the end users are reporting problems. You can create custom templates and even call them problem_child.cfg. You would add hosts to this template only when a need occurs to step up the monitoring.

Thanks Ben for the article and I found it quite insightful. Keep up the fine work.

]]> By: ben http://www.barkingseal.com/2009/04/debugging-nagios-performance-problems/comment-page-1/#comment-63 ben Mon, 01 Jun 2009 13:59:30 +0000 http://www.barkingseal.com/?p=725#comment-63 @Kumaran - Yes, I absolutely agree. Some service checks need to be more granular (2-5 minutes) and others can be monitored on the 10-15 minute scale. It really depends on the service's purpose and the amount of downtime that can be tolerated. Ultimately, I'd say the interval should be up to the service owner. @Kumaran – Yes, I absolutely agree. Some service checks need to be more granular (2-5 minutes) and others can be monitored on the 10-15 minute scale. It really depends on the service’s purpose and the amount of downtime that can be tolerated. Ultimately, I’d say the interval should be up to the service owner.

]]> By: Kumaran http://www.barkingseal.com/2009/04/debugging-nagios-performance-problems/comment-page-1/#comment-62 Kumaran Mon, 01 Jun 2009 10:34:48 +0000 http://www.barkingseal.com/?p=725#comment-62 I think moving the service check timings to an interval of 10-15mins would have had an impact in improved performance. 10-15 mins can be ideal for a few devices but not something that I would recommend in a more dynamic environment. For example, for the WAN infras that we configure nagios 5mins checks themselves are too big an interval as a network/interface flap for a couple of mins can cause serious issues as a lot of HA systems would kickin. So, sooner the engineers know is better. And sometimes you may miss a whole flap completely in these 10-15mins interval CPU 30-50% isn't an impact again. I think moving the service check timings to an interval of 10-15mins would have had an impact in improved performance. 10-15 mins can be ideal for a few devices but not something that I would recommend in a more dynamic environment. For example, for the WAN infras that we configure nagios 5mins checks themselves are too big an interval as a network/interface flap for a couple of mins can cause serious issues as a lot of HA systems would kickin. So, sooner the engineers know is better. And sometimes you may miss a whole flap completely in these 10-15mins interval

CPU 30-50% isn’t an impact again.

]]> By: Hendrik http://www.barkingseal.com/2009/04/debugging-nagios-performance-problems/comment-page-1/#comment-57 Hendrik Thu, 07 May 2009 08:34:04 +0000 http://www.barkingseal.com/?p=725#comment-57 I found something new for me! Thanks! I found something new for me! Thanks!

]]>