DevOps NY

Tag: monitoring

  • Icinga Monitoring System

    After many installations of Icinga, I finally decided to cobble together the documentation that I use and add it to a central location. Links to the original documents are found throughout this post. This document provides instructions on setting up the Icinga monitoring system. It will go over the following:

    • Icinga and IDOUtils
    • Icinga Web
    • PNP4Nagios
    • Icinga Mobile

    Pre-Requisites

    • Ubuntu Server 14.04 LTS 64 Bit
    • 2GB RAM
    • 2 CPU
    • 60 GB HDD

    Icinga and IDOUtils

    The installation of the following software will be covered in this section:

    • Icinga Core
    • Icinga Classic UI (the “CGIs”)
    • Icinga IDOUtils
      • Mysql or Postgresql as database
    • Icinga Docs

    Icinga Core

    This document describes the procedure for installing Icinga core using the official packages. There is also documentation for installing the package maintainer version (PPA) which may be more up to date.

    The following command will install all required packages:

    sudo apt-get install icinga icinga-doc icinga-idoutils mysql-server libdbd-mysql mysql-client

    This installation will use MySQL as the database server. Postgress can also be used, but will not be covered in this document.

    Plugins

    The following command will install the Nagios plugins.

    sudo apt-get install nagios-plugins

    Configuration

    Enable IDO2DB

    Edit the following file:

    /etc/default/icinga

    change the following line:

    IDO2DB=no

    to

    IDO2DB=yes

    Then restart the ido2db service:

    sudo service ido2db start

    check to see if it is enabled on startup:

    sh -x /etc/init.d/ido2db start | grep IDO2DB

     

     

    Enable IDOMOD

    Copy the example configuration file into the Icinga configuration directory:

    sudo cp /usr/share/doc/icinga-idoutils/examples/idoutils.cfg-sample /etc/icinga/modules/idoutils.cfg

    Restart Icinga with:

    sudo service icinga restart

    Check the logfile (/var/log/icinga/icinga.log) to see if the module has been loaded and connected to IDO2DB:

    [1399482608] idomod: IDOMOD 1.10.3 (02-11-2014) Copyright(c) 2005-2008 Ethan Galstad, Copyright(c) 2009-2013 Icinga Development Team (https://www.icinga.org)
[1399482608] idomod: Successfully connected to data sink. 0 queued items to flush.
[1399482608] Event broker module 'IDOMOD' version '1.10.3' from '/usr/lib/icinga/idomod.so' initialized successfully.

    Enable External Commands

    This may have been enabled during the package installation. To check, do the following:

    See if:

    check_external_commands=1

    exists in /etc/icinga/icinga.cfg

    If it is set to 0, change it to 1 as above and do the following:

    sudo service icinga stop
    sudo dpkg-statoverride --update --add nagios www-data 2710 /var/lib/icinga/rw
    sudo dpkg-statoverride --update --add nagios nagios 751 /var/lib/icinga
    sudo service icinga start

    Classic UI Authentication

    Users can be added to access the portal with the following command:

    sudo htpasswd /etc/icinga/htpasswd.users <username>

    Adapted from: https://wiki.icinga.org/display/howtos/Setting+up+Icinga+with+IDOUtils+on+Ubuntu

    PNP4Nagios

    “PNP is an addon for the Nagios Network Monitoring System. PNP provides easy to use, easy to configure RRDTools based performance charts feeded by the performance data output of the Nagios Plugins.” – http://docs.pnp4nagios.org/pnp-0.6/start

    This document covers installing PNP for Icinga.

    Requirements

    • Icinga running
    • Icinga Classic UI or Icinga Web running

    Setup

    First, ensure the system is up to date with:

    sudo apt-get update

    Then install the pnp4nagios package:

    sudo apt-get install --no-install-recommends pnp4nagios

    without –no-install-recommends, icinga will be installed as a dependancy if not already installed. Icinga installation was covered in the previous section.

    Enable RRDCached in PNP4Nagios

    “rrdcached is a daemon that receives updates to existing RRD files, accumulates them and, if enough have been received or a defined time has passed, writes the updates to the RRD file. A flush command may be used to force writing of values to disk, so that graphing facilities and similar can work with up-to-date data.” – http://oss.oetiker.ch/rrdtool/doc/rrdcached.en.html

    General purpose – decrease I/O, increase performance.

    Installation

    Install the software with the following command:

    sudo apt-get install rrdcached

    Now, add it to system init:

    sudo update-rc.d rrdcached defaults

    RRDCACHED Configuration

    Create the cache directory:

    sudo mkdir -p /var/cache/rrdcached

    Add the nagios user to the www-data group with:

    sudo usermod -aG www-data nagios

    Now the nagios user account can write to web directories.

    Edit the configuration file, /etc/default/rrcached and add the following line:

    OPTS="-w 1800 -z 1800 -j /var/cache/rrdcached -s nagios -m 0660 -l unix:/var/run/rrdcached.sock"

    Now, restart the service:

    sudo service rrdcached restart

    PNP4Nagios Configuration for RRDCACHED

    Add the following line to /etc/pnp4nagios/process_perfdata.cfg

    RRD_DAEMON_OPTS = unix:/var/run/rrdcached.sock

    Then modify the following line in /etc/pnp4nagios/config.php

    $conf['RRD_DAEMON_OPTS'] = 'unix:/var/run/rrdcached.sock';

    Also, change the nagios_base  in config.php to the following:

    $conf[‘nagios_base’] = “/cgi-bin/icinga”;

    Restart Services

    sudo service rrdcached restart

    and

    sudo service apache2 restart

    Adapted from: https://wiki.icinga.org/display/howtos/Enable+RRDCached+in+PNP

    Enable PNP4Nagios in Icinga

    Configuration

    Add performance data in icinga.cfg. Edit /etc/icinga/icinga.cfg and add the following lines:

    # performance data
process_performance_data=1

service_perfdata_file_template=DATATYPE::SERVICEPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tSERVICEDESC::$SERVICEDESC$\tSERVICEPERFDATA::$SERVICEPERFDATA$\tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$\tSERVICESTATE::$SERVICESTATE$\tSERVICESTATETYPE::$SERVICESTATETYPE$

host_perfdata_file_template=DATATYPE::HOSTPERFDATA\tTIMET::$TIMET$\tHOSTNAME::$HOSTNAME$\tHOSTPERFDATA::$HOSTPERFDATA$\tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$\tHOSTSTATE::$HOSTSTATE$\tHOSTSTATETYPE::$HOSTSTATETYPE$

service_perfdata_file_mode=a
host_perfdata_file_mode=a

service_perfdata_file_processing_interval=30
host_perfdata_file_processing_interval=30

service_perfdata_file=/var/spool/pnp4nagios/nagios/service-perfdata
host_perfdata_file=/var/spool/pnp4nagios/nagios/host-perfdata

service_perfdata_file_processing_command=pnp-bulknpcd-service
host_perfdata_file_processing_command=pnp-bulknpcd-host

    Add commands to /etc/icinga/commands.cfg:

    # pnp4nagios
define command{
        command_name    pnp-bulknpcd-service
        command_line    /bin/mv /var/spool/pnp4nagios/nagios/service-perfdata /var/spool/pnp4nagios/npcd/service-perfdata.$TIMET$
}

define command{
        command_name    pnp-bulknpcd-host
        command_line    /bin/mv /var/spool/pnp4nagios/nagios/host-perfdata /var/spool/pnp4nagios/npcd/host-perfdata.$TIMET$
}

    Web Configuration

    Edit the /etc/apache2/conf-available/pnp4nagios.conf file and change to the following:

            AuthName "Icinga Access"
        AuthType Basic
        AuthUserFile /etc/icinga/htpasswd.users

    Then restart apache with service apache2 reload.

    Enable NPCD

    Edit /etc/default/npcd and edit the fllowin gline to look like:

    Run="yes"

    Then restart the service with:

    sudo service npcd start

    Integration Into Icinga

    Create the file /etc/icinga/objects/pnptemplates.cfg

    define host {
        name       pnp-hst
        register   0
        action_url /pnp4nagios/graph?host=$HOSTNAME$' class='tips' rel='/pnp4nagios/popup?host=$HOSTNAME$&srv=_HOST_
}

define service {
        name       pnp-svc
        register   0
        action_url /pnp4nagios/graph?host=$HOSTNAME$&srv=$SERVICEDESC$' class='tips' rel='/pnp4nagios/popup?host=$HOSTNAME$&srv=$SERVICEDESC$
}

    Then use them in host and service configurations:

    define host{
        name                            generic-host
        use                             pnp-hst
...

define service{
        name                            generic-service
        use                             pnp-svc
...

    Copy the ssi file to the web UI:

    sudo cp /usr/share/doc/pnp4nagios/examples/ssi/status-header.ssi /usr/share/icinga/htdocs/ssi/status-header.ssi
    sudo chmod 644 /usr/share/icinga/htdocs/ssi/status-header.ssi

    then reload icinga for the changes to take effect:

    sudo service icinga reload

    Adapted from: https://wiki.icinga.org/display/howtos/Setting+up+PNP+with+Icinga+on+Debian

  • WMI and JMX with Icinga

    Over the past week I have been flexing my monitoring skills again. Of course the landscape has changed since I worked on this a few years ago. Today, I am struggling with a commercial product, unnamed. Bloated, expensive, horrible GUI, and time consuming. I am sure it is a good product given that you have an entire staff dedicated to managing it.

    So, I went back to my roots and set up an Icinga instance. I have it fully working on a Linux box monitoring Windows hosts. Agentless. No more nsclient installs. I am using check_wmi_plus plugin for everything Windows. You can supply a wmi query string to the plugin for anything WMI. Here are some URLs:

    https://wiki.icinga.org/display/howtos/check_wmi_plus

    some service definitions:

    http://www.edcint.co.nz/checkwmiplus/?q=nagiosservicedefinitions

    I am also monitoring out Java infrastructure. You need to make sure the java app is set up for remote jmx monitoring before this will work.

    http://exchange.nagios.org/directory/Plugins/Java-Applications-and-Servers/check_jmx/details

    and some nice service defs and commands:

    https://kb.op5.com/display/HOWTOs/Jboss+Monitoring+with+check_jmx

    Maybe I will get some time to write up a comprehensive faq.

  • graphdat

    Trying out a new tool today. http://www.graphdat.com/

  • Zenoss

    I think I might try Zenoss again. It has been a year or two since I last visited this product. I am looking for a product that is all encompassing. I will probably use CentoOS as the OS.

    http://wiki.zenoss.org/Install_Zenoss