How to monitor your monitoring platform – part 2

This post is also available in: French

This summer, we’ve had a look at the theory: How to monitor your monitoring platform – Part 1, it’s time to practice! Now the holidays are over, it is time to take over the control of our monitoring platform.

Some reading

Before starting to practice, you should read some previous articles:

Theory

The monitoring of the monitoring platform must be “crossed”.
In the case of a distributed architecture, one of the distant pollers has to monitor the processes of the central server, whereas the central server will monitor all the processes of the remote pollers.

Note: For those who would like to have the safest monitoring platform, the must would be to have a moved monitoring platform that monitors the monitoring platform. This system would ensure the proper functioning of the monitoring platform.

The ultimate (therefore above the must;)), would be to have a 2nd duplicated server in order to ensure a good availability of monitored data.

For those who only have a single central sever, we advise you to monitor the following:

  • A stream multiplexer (Centreon Broker, NDOutil, …);
  • A Web presentation server (Apache…);
  • A poller presentation and configuration interface (Centreon Web)
  • A MySQL database server (remote or not);
  • Databases : « centreon », « centreon_storage »;
  • RRD databases for the generation of performance graphics;
  • Centreon Processes (workflow)
  • A exploitation system state (CPU, RAM, disk free space)

For those who have a distributed environment, these are all the elements that you have to monitor:

  • For the central server from the poller ::
  • A monitoring engine (Centreon Engine, Nagios, …);
  • A stream multiplexer (Centreon Broker, NDOutil, …);
  • A Web presentation server (Apache…);
  • A poller presentation and configuration interface (Centreon Web) ;
  • A MySQL database server (remote or not);
  • Databases : « centreon », « centreon_storage » ;
  • RRD databases for the generation of performance graphics;
  • Centreon Processes (workflow) ;
  • Event logs (logs) ;
  • Notification processes;
  • Operating system state (CPU, RAM, disk space).

For the pollers from central server:

  • A monitoring engine (Centreon Engine, Nagios …);
  • A stream multiplexer (Centreon Broker, NDOutil …);
  • A Web presentation server (Apache…);
  • A poller presentation and configuration interface (Centreon Web);
  • Event logs (logs);
  • Notification processes;
  • Operating system state (CPU, RAM, disk space).

In practice

In practice, it is a little more complex. Therefore, we are not going to set up a monitoring agent on the pollers in order to control the event logs and all the processes that need a local command execution. For this, we should use simple protocol such as SNMP, HTTP and SQL.

Control access to the Centreon Web interface

We will control the Apache server state via the “mod_status” module. The load configuration of the module is available on the web and we won’t spend to much time on this subject. That will be the same with the set up of “check_apache_status” control that are available on the Monitoring Exchange website.

Create the following command:

  • Command name : APP-Apache-Status
  • Command line : $USER1$/check_apache_status.pl -H $HOSTADDRESS$

commandapache_uec1

Then create the associated services to the host that represents your Centreon server (if it is possible , this host should be monitored by a remote server):

  • Service name: Apache-status
  • Service model: generic-service
  • Control command: APP-Apache-Status

serviceapache_uec2

The result will be the following:

resultatapache_uec3

Comment: You can also control the access to the Centreon interface via the plugin “check_http” and create a service dependence on the previous created one and thus, you can generate an availability report.

Control of MySQL database server state

We are going to use the community plugin “check_mysql” that is available on Nagios© standard plugins.
Create the following commands:

  • Command name : DB-MySQL
  • Command line : $USER1$/check_mysql -H $HOSTADDRESS$ -u $_HOSTMYSQLUSER$ -p $_HOSTMYSQLPASSWD$

commanddemysql_uec4

Comment: this command uses custom macros.

Then create the service associated to host that represents your Centreon server (if it is possible, this host should be monitored by a remote server):

  • Service name: MySQL-status
  • Service model: generic-service
  • Control command: DB-MySQL

servicemysql_uec5

Comment: Add the following macros and the associated value to your host:

  • MYSQLUSER
  • MYSQLPASSWD

The result will be the following:

resultatsql_uec6

Comment: it is quite possible to monitor in more depth the DBMS and all databases hosted on this one. The community plugin ‘check_mysql_health’ allows to test:

  • The buffers and MySQL/InnoDB caches;
  • The slow queries ;
  • The index usage ;
  • The time of SGBD connection ;

Comment: Don’t forget to add all remote reading rights on SGBD to the poller realizing the test.

Control of server systems

We are here using the Nagios© and Centreon plugins.

Create the following commands:

  • Command name: OS-Linux-SNMP-CPU
  • Command: $USER1$/check_centreon_snmp_cpu -H $HOSTADDRESS$ -C $_HOSTSNMPCOMMUNITY$ -v $_HOSTSNMPVERSION$ -w $_SERVICEWARNING$ -c $_SERVICECRITICAL$

 

commandserveurcpu_uec7

  • Command name: OS-Linux-SNMP-Load
  • Command line: $USER1$/check_centreon_snmp_loadaverage -H $HOSTADDRESS$ -C $_HOSTSNMPCOMMUNITY$ -v $_HOSTSNMPVERSION$ -w $_SERVICEWARNING$ -c $_SERVICECRITICAL$

commandserveurload_uec8

  • Command name: OS-Linux-SNMP-Memory
  • Command line: $USER1$/check_centreon_snmp_memory -H $HOSTADDRESS$ -C $_HOSTSNMPCOMMUNITY$ -v $_HOSTSNMPVERSION$ -w $_SERVICEWARNING$ -c $_SERVICECRITICAL$

commandserveurmemory_uec9

  • Command name: OS-Linux-SNMP-FS
  • Command line: $USER1$/check_centreon_snmp_remote_storage -H $HOSTADDRESS$ -C $_HOSTSNMPCOMMUNITY$ -v $_HOSTSNMPVERSION$ -d $_SERVICEFSNAME$ -n -w $_SERVICESNMPWARNING$ -c $_SERVICECRITICAL$

commandserveurfs_uec10

  • Command name: OS-Linux-SNMP-Traffic
  • Command line: $USER1$/check_centreon_snmp_traffic -H $HOSTADDRESS$ -C $_HOSTSNMPCOMMUNITY$ -v $_HOSTSNMPVERSION$ -i $_SERVICEINTERFACENAME$ -n -w $_SERVICESNMPWARNING$ -c $_SERVICECRITICAL$

commandserveurtraffic_uec11

Then create the associated services:

  • CPU:

servicecpu_uec12

  • Load:

serviceload_uec13

  • RAM:

servicememory_uec14

  • Traffic:

servicetraffic_uec15

  • File System:

servicefs_uec16

Comment: Don’t forget to fill in the “SNMP Community & Version” on the definition of your host fields.

The result will be the following:

resultatserveur_uec17

Processes monitoring

The processes monitoring will be done through the Centreon plugin ‘check_centreon_snmp_process’, available in the standard Centreon version.

Create the following commands:

  • Command name: OS-Linux-SNMP-Process
  • Command line: $USER1$/check_centreon_snmp_process -H $HOSTADDRESS$ -C $_HOSTSNMPCOMMUNITY$ -v $_HOSTSNMPVERSION$ -p $_SERVICEPROCESSNAME$

commandprocess_uec18

Then create the following service:

serviceprocess_uec19

Do the same with all processes:

  • Apache process : httpd
  • Centcore process : centcore
  • Centreon Broker process : cbd
  • CRON process: crond
  • MySQL process : mysqld
  • NTP process: ntpd
  • Rsyslog process : rsyslogd
  • SNMP process : snmptrapd

The result is the following:

resultatprocess_uec20

Control of remote pollers

For remote pollers, take the previous created controls:

  • OS:
    • CPU ;
    • Load ;
    • Memory ;
    • FS-<partition name> ;
    • Traffic-<interface name> ;
  • Process:
    • Centreon Engine ;
    • Cron ;
    • NTP ;
    • Rsyslog ;
    • Traps SNMP ;

Additional controls

Many additional controls could be added to check deeper into the Centreon server:

  • Check each server partition (/var/log, /var/lib/mysql, /var/lib/centreon, …) ;
  • Check the errors in the event log of Centreon Engine and Centreon Broker, Centreon processes (/var/log/centreon/*.log) ;
  • Check pack error server of network interfaces
  • Check access to mail server (POP, IMAP, SMTP, …) ;

The results

After setting up, you should have a monitoring that looks like those for Centreon server:

resultatserveurcentral_21

For pollers:

resultatspollers_22

Incoming search terms:

  • CENTREON controls
  • centreon change free disk

Leave a Reply