High Availability Zabbix and Zabbix Proxy Servers

Introduction

This article explains how to install high availability (fail over) Zabbix and Zabbix Proxy servers.

High-availability clustering is designed to increase systems availability.  For this article, we will be using:

• Distributed Replicated Block Device (DRBD)
• Corosync - Pacemaker

These technologies present two servers as one to the network; one server is active and the other is waiting to take over if the first fails or is taken off line.

The cluster is composed of elements that differ from a single-server deployment.  

• Resources unique to each node
• Data shared between nodes on the cluster
• Services shared between the nodes on the cluster

Copyright © 2008, 2009 LINBIT Information Technologies GmbH

Copyright © 2009, 2010, 2011, 2012 LINBIT HA Solutions GmbH

Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA")

at http://www.drbd.org/uploads/pics/overview_02.gif

Resources Unique to Each Node

Each node has is a server with its own operating system and hardware.  The processor, memory, disk and IO subsystems (including network interfaces) are controlled by the operating system installed on the boot partition.

Data Shared Between Nodes on the Cluster

For a MySQL cluster, there are two types of shared data:  configuration files and databases.  The configuration files are those located in the /etc/mysql/ directory.  When shared between the two nodes, the MySQL server will have an identical configuration regardless of the node that is active.  However, there are circumstances in which the MySQL configuration files may be unique to each server.  The databases are kept in the /var/lib/mysql/ directory and include the log files.

MySQL Clustering Caveats

Although the two nodes share the same MySQL databases, UNDER NO CIRCUMSTANCES SHALL THE TWO NODES SIMULTANEOUSLY ACCESS THE DATABASES.  That is, only one node may run the mysqld daemon at any given time.  If two MySQL daemons access the same database, there will eventually be corruption.  The clustering software controls which node accesses the data.

DRBD - Corosync - Pacemaker Overview

The illustration below depicts a high-availability cluster design.  Each server has four network interfaces:

• eth0 -- the publicly addressable interfaces
• eth1 -- the DRBD data replication and control interfaces
• eth2 and eth3 -- the Corosync - Pacemaker control interfaces

The first interface -- eth0 -- is the publicly addressable interface that provides MySQL database and Apache web server (for PHPMyAdmin) access.  Two IP addresses (unique to server) are assigned at boot time and a third is assigned by the Corosync - Pacemaker portion of the clustering software.

The second interface -- eth1 -- is controlled by the DRBD daemon.  This daemon is configured with two or more files: /etc/drbd.d/global_common.conf and /etc/drbd.d/ro.res, r1.res, r2.res... assigned for each shared block device.  For this example, only r0.res is installed.  System-wide settings are defined in the global_common.conf file.  Settings specific to each pair of shared block devices are defined in the r0.res (and other) resource files.  DRBD defines an entire block device (or hard drive) as shared and replicated between two nodes.  In this example, block device /dev/sdb (a SCSI drive in each server) is shared between the two nodes as /dev/drbd0.  Once configured, both servers see /dev/sdb as a new block device /dev/drbd0 and only ONE may mount it at any given time.  The server with the mounted partition replicates any changes to the failover node.  If the first server fails or is taken off line, the other server immediately mounts its /dev/drbd0 block device and assumes control of replication.

The third and fourth interfaces -- eth2 and eth3 -- are controlled by the Corosync - Pacemaker software.  These interfaces provide communication links defining the status of each defined resource and control how and where shared services (such as shared DRBD block devices, IP addresses, and the MySQL / Apache2 daemons) run.  In failover clustering, only one node may actively be in control at any time.

Installing and Configuring the Zabbix - MYSQL - Apache - Postfix Failover Cluster

Begin by installing and MySQL, Apache and Postfix.  Leave MySQL listening on the default loopback interface 127.0.0.1.  Then, start the Linux Cluster Management Console (LCMC) -- a Java application that will install and configure everything required for clustering.  Select the two nodes by name or IP address, install Pacemaker (NOT Heartbeat) and DRBD.

Once both nodes have the required software, configure the cluster.  LCMC will prompt you for two interfaces to use in the cluster (select eth2 and eth3) and the two-node system will be recognized as a Cluster.  Configure global options and then select device /dev/sdb on one node and mirror it to /dev/sdb on the other to configure DRBD device /dev/drbd0.  Format it with an ext4 file system and make sure to perform an initial full synchronization.

When the DRBD device finishes synchronization, create a shared mount point -- /mnt/sdb -- and IP address (10.195.0.100 shared between eth0 on the nodes).  The cluster will now recognize the DRBD device as a file system on the Active node.

The shared data must then be moved to the DRBD device.  Stop MySQL on each node.  On the Active node, move the directories /etc/mysql, /var/lib/mysql and /etc/zabbix/ to /mnt/sdb/etc/mysql, /mnt/sdb/var/lib/mysql and /mnt/sdb/etc/zabbix, respectively, and create symlinks back to their original locations.  On the Inactive node, delete the mysql and zabbix directories and replace them with symlinks to the same locations -- even though those locations are not yet visible.

The LCMC console is then used to finalize the shared services.  Add Primitive LSB Init Script resources (that is, only running on one server at a time) for MySQL, Apache2, Postfix, Zabbix Server and Zabbix-Agent.

Fail the servers back and forth several times to check that the system performs as expected.  Finally, install a database.  For this article, I install the Zabbix Monitoring database.

The video below illustrates the entire installation and configuration process.

Installing the Zabbix Proxy - MySQL Failover Cluster

Zabbix Proxy only requires the zabbix-agent and zabbix-proxy-mysql packages.  The proxy is linked to the server by configuration file settings.  It is then configured and controlled through the server console.

The cluster configuration -- unlike the system set up -- is very much like configuring Zabbix Server failover clustering.  Similar hardware and identical clustering software and symlinked shared directories are used.

The video below demonstrates the Zabbix Proxy failover cluster installation.

Zabbix Proxy Performance

Zabbix Proxy is not designed to distribute load.  While it has its own database, is configured from and subordinate to a Zabbix server, all the results are still processed by the Zabbix server.  There is a small performace improvement at the server, but not much.

Zabbix Proxy is intended to provide fault-tolerance, particularly across unreliable links.  The brief video below demonstrates the process loads on an active Zabbix Server and Proxy.

SAN-Backed High Availability MySQL Servers

High-Availability Clustering Technologies

High-availability clustering is designed to increase systems availability.  In a previous article, we looked at DRBD-based file storage -- analogous to network disk mirroring.  For this article, we will be using:

• SAN-Backed Storage
• Corosync - Pacemaker

DRBD is useful in situations that lack a SAN.  However a good storage network design is preferable for reliability and performance.  DRBD runs as a daemon on each node, replicating block devices.  There is increased overhead, not only for disk IO, but also for processor iowait states and memory usage.  This overhead is relieved by a SAN, particularly valuable when deploying resource-intensive database servers.

The cluster is composed of elements that differ from a single-server deployment.  

• Resources unique to each node
• Data shared between nodes on the cluster
• Services shared between the nodes on the cluster

Resources Unique to Each Node

Each node has is a server with its own operating system and hardware.  The processor, memory, disk and IO subsystems (including network interfaces) are controlled by the operating system installed on the boot partition.

Data Shared Between Nodes on the Cluster

For a MySQL cluster, there are two types of shared data:  configuration files and databases.  The configuration files are those located in the /etc/mysql/ directory.  When shared between the two nodes, the MySQL server will have an identical configuration regardless of the node that is active.  However, there are circumstances in which the MySQL configuration files may be unique to each server.  The databases are kept in the /var/lib/mysql/ directory and include the log files.  These will be copied to a shared ext4 partition.

MySQL Clustering Caveats

Although the two nodes share the same MySQL databases, UNDER NO CIRCUMSTANCES SHALL THE TWO NODES SIMULTANEOUSLY ACCESS THE DATABASES.  That is, only one node may run the mysqld daemon at any given time.  If two MySQL daemons access the same database, there will eventually be corruption.  The clustering software controls which node accesses the data.

Corosync - Pacemaker Overview

The illustration below depicts a high-availability cluster design.  Each server has four network interfaces:

• eth0 -- the publicly addressable interfaces
• eth1 -- the iSCSI interfaces
• eth2 and eth3 -- the Corosync - Pacemaker control interfaces

The first interface -- eth0 -- is the publicly addressable interface that provides MySQL database and Apache web server (for PHPMyAdmin) access.  Two IP addresses (unique to server) are assigned at boot time and a third is assigned by the Corosync - Pacemaker portion of the clustering software.

The second interface -- eth1 -- is controlled by the open-iscsi daemon that connects to iSCSI SAN storage.

The third and fourth interfaces -- eth2 and eth3 -- are controlled by the Corosync - Pacemaker software.  These interfaces provide communication links defining the status of each defined resource and control how and where shared services (such as shared DRBD block devices, IP addresses, and the MySQL / Apache2 daemons) run.  In failover clustering, only one node may actively be in control at any time.

Installing and Configuring the MYSQL - Apache Failover Cluster

Begin by installing and MySQL and Apache.  Leave MySQL listening on the default loopback interface 127.0.0.1.  Then, start the Linux Cluster Management Console (LCMC) -- a Java application that will install and configure everything required for clustering.  Select the two nodes by name or IP address, install Pacemaker (NOT Heartbeat).

Once both nodes have the required software, configure the cluster.  LCMC will prompt you for two interfaces to use in the cluster (select eth2 and eth3) and the two-node system will be recognized as a Cluster.  Configure global options for the cluster and begin by adding a the shared storage device.  In this case, the SAN partition is formatted with ext4 and recognized as device /dev/sdc1 on each node.  Mount it at /mnt/dev/sdc using the LCMC Filesystem resource.

The shared data must then be moved to the DRBD device.  Stop MySQL on each node.  On the Active node, move the directories /etc/mysql and /var/lib/mysql to /mnt/sdc/etc/mysql and /mnt/sdc/var/lib/mysql, respectively, and create symlinks back to their original locations.  On the Inactive node, delete the mysql directories and replace them with symlinks to the same locations -- even though those locations are not yet visible.

The LCMC console is then used to finalize the shared services.  Add a shared IP Address 10.195.0.110/32 and Primitive LSB Init Script resources (that is, only running on one server at a time) for MySQL and Apache2.

Fail the servers back and forth several times to check that the system performs as expected.  Finally, install a database and modify the /etc/mysql/my.cnf configuration file, removing the loopback 127.0.0.1 address and subtituting the shared 10.195.0.110 address.  For this article, I install two Zabbix Monitoring databases (zabbix_01 and zabbix_02) for use in distributed monitoring.

The video below illustrates the entire installation and configuration process.

Scaling High Performance and Availability Zabbix Systems Monitoring

Background

Systems Engineers frequently comment about heavy loads, performance and scalability problems with Zabbix installations.  Indeed, Zabbix places a heavy load on resources; processes service check information, writes to a database, uses a PHP web application to present data and caches history information.  The database alone can be a heavy processor - memory and disk IO load.  Adding the other processor and memory intensive processes simply adds to the system load.

However, Zabbix was designed to scale up to very large demands.  Those only familiar with the default all-in-one web-application-database installation may consider implementing the available modular architecture to distribute loads.

Scalable Zabbix Architecture

A Scalable Zabbix implementation distributes loads among three classes of servers:

1. Database Back End Servers
2. Zabbix Application Servers
3. Zabbix GUI Web Front End Servers

This article will explain how to implement these classes of servers in a highly scalable deployment.  A High-Availability SAN-Backed MySQL-PHPMyAdmin server will provide two databases, zabbix_01 and zabbix_02.  The child node database will send information to the Master Node, from which data and configurations for all child servers are available.  There are two Zabbix Application Servers dedicated to processing host checks.  There is one GUI Web Front End server dedicated to processing and presenting information (graphs, etc.).

This is a relatively simple implementation of a Scalable Zabbix architecture.  Additional descriptions of High-Availability systems are recommended reading:

• DRBD High Availability MySQL Servers
• SAN-Backed High Availability MySQL Servers
• High Availability Stand Alone Zabbix
• High Availability Zabbix and Zabbix Proxy Servers

The two Zabbix Application and one Zabbix GUI Front End servers may all be deployed as high availability clusters.

Deploying Scalable Zabbix Servers

Create and Populate the Databases

The first task is deploying the two required databases -- zabbix_01 and zabbix_02 -- on the database server.  This is depicted at the end of the video in SAN-Backed High Availability MySQL Servers.

Deploy the Zabbix Application Servers

The two Zabbix Application servers may then be deployed.  Although they only require the zabbix-server-mysql package, you may wish to deploy the zabbix-frontend-php package for initial configuration.  The Apache and MySQL servers installed may be disabled or removed once they are defined and Master and Child servers.  I typically configure the servers entirely from the /etc/zabbix/zabbix_server.conf file, specifying the desired database server (in this case hbg-db.harrisburg.mydomain.com) and database credentials.  It is necessary to also specify a unique NodeID for each server, ranging up to 999.

Once the server is installed, configured and connected to its own database, you must then upgrade the databases from each server for distributed monitoring.  ONLY UPGRADE THE DATABASE ONCE.  Running the upgrade more than once will result in database corruption. To perform the upgrade:

• Stop the Zabbix and Apache services
• Issue the command: zabbix_server -n <NodeID> -c /etc/zabbix/zabbix_server.conf

Deploy the GUI Web Front End Server

The front end only requires the zabbix-frontend-php package.  Once installed, browse to the /zabbix directory and perform the web-based installation.  This server must connect to the backend database cluster (hbg-db.harrisburg.mydomain.com) and the Master Zabbix server (monitor01.harrisburg.mydomain.com).

Once installed, update the default Zabbix Server information to reflect the local nomenclature.  Then, set it as a Local Node, ID 1 and add the child Zabbix Server as Node 2.  Once completed, move to the graphical front-end on the child node (this is the only time it will be needed).  Update its default information and then specify it as a local Node ID 2 and add the Master server Node ID 1.  Then move back to the Master Server GUI and check that the architecture is updated (i.e. the child database is feeding information to the master).

From this point on, other than selecting the desired node, all configuration procedures are like a stand alone Zabbix installation.  The process is illustrated in the video below.

High Availability Stand Alone Zabbix (Failover Zabbix, MySQL, Apache and Postfix with DRBD-Pacemaker)

This article describes Distributed Replicated Block Device (DRBD) fault tolerance, analogous to network disk mirroring.

High-Availability Clustering Technologies

High-availability clustering is designed to increase systems availability.  For this article, we will be using:

• Distributed Replicated Block Device (DRBD)
• Corosync - Pacemaker

These technologies present two servers as one to the network; one server is active and the other is waiting to take over if the first fails or is taken off line.

The cluster is composed of elements that differ from a single-server deployment.  

• Resources unique to each node
• Data shared between nodes on the cluster
• Services shared between the nodes on the cluster

Resources Unique to Each Node

Each node has is a server with its own operating system and hardware.  The processor, memory, disk and IO subsystems (including network interfaces) are controlled by the operating system installed on the boot partition.

Data Shared Between Nodes on the Cluster

Zabbix server includes serveral components:

• Apache2 Web Server
• MySQL Database Server
• Postfix Mail Server
• Zabbix Server
• Zabbix Agent
• Zabbix PHP Frontend

Once configured,  Apache and Postfix do not require additional modifications and may remain unique to each server; theie configuration files do not need to be shared.

For a MySQL cluster, there are two types of shared data:  configuration files and databases.  The configuration files are those located in the /etc/mysql/ directory.  When shared between the two nodes, the MySQL server will have an identical configuration regardless of the node that is active.  However, there are circumstances in which the MySQL configuration files may be unique to each server.  The databases are kept in the /var/lib/mysql/ directory and include the log files.

The Zabbix configuration files (of the server, client and front-end web server) are stored in the /etc/zabbix directory.  The PHP Frontend files (used by Apache) are stored in the /usr/share/zabbix/directory; these may remain on each server.

MySQL Clustering Caveats

Although the two nodes share the same MySQL databases, UNDER NO CIRCUMSTANCES SHALL THE TWO NODES SIMULTANEOUSLY ACCESS THE DATABASES.  That is, only one node may run the mysqld daemon at any given time.  If two MySQL daemons access the same database, there will eventually be corruption.  The clustering software controls which node accesses the data.

DRBD - Corosync - Pacemaker Overview

The illustration below depicts a high-availability cluster design.  Each server has four network interfaces:

• eth0 -- the publicly addressable interfaces
• eth1 -- the DRBD data replication and control interfaces
• eth2 and eth3 -- the Corosync - Pacemaker control interfaces

The first interface -- eth0 -- is the publicly addressable interface that provides MySQL database and Apache web server (for PHPMyAdmin) access.  Two IP addresses (unique to server) are assigned at boot time and a third is assigned by the Corosync - Pacemaker portion of the clustering software.

The second interface -- eth1 -- is controlled by the DRBD daemon.  This daemon is configured with two or more files: /etc/drbd.d/global_common.conf and /etc/drbd.d/ro.res, r1.res, r2.res... assigned for each shared block device.  For this example, only r0.res is installed.  System-wide settings are defined in the global_common.conf file.  Settings specific to each pair of shared block devices are defined in the r0.res (and other) resource files.  DRBD defines an entire block device (or hard drive) as shared and replicated between two nodes.  In this example, block device /dev/sdb (a SCSI drive in each server) is shared between the two nodes as /dev/drbd0.  Once configured, both servers see /dev/sdb as a new block device /dev/drbd0 and only ONE may mount it at any given time.  The server with the mounted partition replicates any changes to the failover node.  If the first server fails or is taken off line, the other server immediately mounts its /dev/drbd0 block device and assumes control of replication.

The third and fourth interfaces -- eth2 and eth3 -- are controlled by the Corosync - Pacemaker software.  These interfaces provide communication links defining the status of each defined resource and control how and where shared services (such as shared DRBD block devices, IP addresses, and the MySQL / Apache2 daemons) run.  In failover clustering, only one node may actively be in control at any time.

Installing and Configuring the MYSQL - Apache - Postfix - Zabbix Failover Cluster

It is assumed the reader knows how to set up a Zabbix server.  If not, please read and understand this article:  Installing and Configuring Basic Zabbix Functionality on Debian Wheezy. 

Leave MySQL listening on the default loopback interface 127.0.0.1.  Then, start the Linux Cluster Management Console (LCMC) -- a Java application that will install and configure everything required for clustering.  Select the two nodes by name or IP address, install Pacemaker (NOT Heartbeat) and DRBD.

Once both nodes have the required software, configure the cluster.  LCMC will prompt you for two interfaces to use in the cluster (select eth2 and eth3) and the two-node system will be recognized as a Cluster.  Configure global options and then select device /dev/sdb on one node and mirror it to /dev/sdb on the other to configure DRBD device /dev/drbd0.  Format it with an ext4 file system and make sure to perform an initial full synchronization.

When the DRBD device finishes synchronization, create a shared mount point -- /mnt/sdb -- and IP address (10.195.0.100 shared between eth0 on the nodes).  The cluster will now recognize the DRBD device as a file system on the Active node.

The shared data must then be moved to the DRBD device.  Stop MySQL on each node.  On the Active node, move the directories /etc/mysql, /var/lib/mysql and /etc/zabbix/ to /mnt/sdb/etc/mysql, /mnt/sdb/var/lib/mysql and /etc/zabbix/, respectively, and create symlinks back to their original locations.  On the Inactive node, delete the mysql and zabbix directories and replace them with symlinks to the same locations -- even though those locations are not yet visible.

On the Active Node:

• mv /etc/mysql /mnt/sdb/etc/mysql
• mv /etc/zabbix /mnt/sdb/etc/zabbic
• mv /var/lib/mysql /mnt/sdb/var/lib/mysql
• ln -s /mnt/sdb/etc/mysql /etc/mysql
• ln -s /mnt/sdb/etc/zabbix /etc/zabbix
• ln -s /mnt/sdb/var/lib/mysql /var/lib/mysql

 On the Inactive Node:

• rm /etc/mysql /mnt/sdb/etc/mysql -r
• rm /etc/zabbix /mnt/sdb/etc/zabbic  -r
• rm /var/lib/mysql /mnt/sdb/var/lib/mysql -r
• ln -s /mnt/sdb/etc/mysql /etc/mysql
• ln -s /mnt/sdb/etc/zabbix /etc/zabbix
• ln -s /mnt/sdb/var/lib/mysql /var/lib/mysql

The LCMC console is then used to finalize the shared services.  Add Primitive LSB Init Script resources (that is, only running on one server at a time) for MySQL and Apache2.  Once the services are installed, change the listener address in /etc/mysql/my.cnf to the shared IP Address of the cluster (10.195.0.100 for this installation).

Fail the servers back and forth several times to check that the system performs as expected.  Finally, install a database.  For this article, I install the Zabbix Monitoring database.

The video below illustrates the entire installation and configuration process.

DRBD High Availability MySQL Servers

High-Availability Clustering Technologies

High-availability clustering is designed to increase systems availability.  For this article, we will be using:

• Distributed Replicated Block Device (DRBD)
• Corosync - Pacemaker

These technologies present two servers as one to the network; one server is active and the other is waiting to take over if the first fails or is taken off line.

The cluster is composed of elements that differ from a single-server deployment.  

• Resources unique to each node
• Data shared between nodes on the cluster
• Services shared between the nodes on the cluster

Resources Unique to Each Node

Each node has is a server with its own operating system and hardware.  The processor, memory, disk and IO subsystems (including network interfaces) are controlled by the operating system installed on the boot partition.

Data Shared Between Nodes on the Cluster

For a MySQL cluster, there are two types of shared data:  configuration files and databases.  The configuration files are those located in the /etc/mysql/ directory.  When shared between the two nodes, the MySQL server will have an identical configuration regardless of the node that is active.  However, there are circumstances in which the MySQL configuration files may be unique to each server.  The databases are kept in the /var/lib/mysql/ directory and include the log files.

MySQL Clustering Caveats

Although the two nodes share the same MySQL databases, UNDER NO CIRCUMSTANCES SHALL THE TWO NODES SIMULTANEOUSLY ACCESS THE DATABASES.  That is, only one node may run the mysqld daemon at any given time.  If two MySQL daemons access the same database, there will eventually be corruption.  The clustering software controls which node accesses the data.

DRBD - Corosync - Pacemaker Overview

The illustration below depicts a high-availability cluster design.  Each server has four network interfaces:

• eth0 -- the publicly addressable interfaces
• eth1 -- the DRBD data replication and control interfaces
• eth2 and eth3 -- the Corosync - Pacemaker control interfaces

The first interface -- eth0 -- is the publicly addressable interface that provides MySQL database and Apache web server (for PHPMyAdmin) access.  Two IP addresses (unique to server) are assigned at boot time and a third is assigned by the Corosync - Pacemaker portion of the clustering software.

The second interface -- eth1 -- is controlled by the DRBD daemon.  This daemon is configured with two or more files: /etc/drbd.d/global_common.conf and /etc/drbd.d/ro.res, r1.res, r2.res... assigned for each shared block device.  For this example, only r0.res is installed.  System-wide settings are defined in the global_common.conf file.  Settings specific to each pair of shared block devices are defined in the r0.res (and other) resource files.  DRBD defines an entire block device (or hard drive) as shared and replicated between two nodes.  In this example, block device /dev/sdb (a SCSI drive in each server) is shared between the two nodes as /dev/drbd0.  Once configured, both servers see /dev/sdb as a new block device /dev/drbd0 and only ONE may mount it at any given time.  The server with the mounted partition replicates any changes to the failover node.  If the first server fails or is taken off line, the other server immediately mounts its /dev/drbd0 block device and assumes control of replication.

The third and fourth interfaces -- eth2 and eth3 -- are controlled by the Corosync - Pacemaker software.  These interfaces provide communication links defining the status of each defined resource and control how and where shared services (such as shared DRBD block devices, IP addresses, and the MySQL / Apache2 daemons) run.  In failover clustering, only one node may actively be in control at any time.

Installing and Configuring the MYSQL - Apache Failover Cluster

Begin by installing and MySQL and Apache.  Leave MySQL listening on the default loopback interface 127.0.0.1.  Then, start the Linux Cluster Management Console (LCMC) -- a Java application that will install and configure everything required for clustering.  Select the two nodes by name or IP address, install Pacemaker (NOT Heartbeat) and DRBD.

Once both nodes have the required software, configure the cluster.  LCMC will prompt you for two interfaces to use in the cluster (select eth2 and eth3) and the two-node system will be recognized as a Cluster.  Configure global options and then select device /dev/sdb on one node and mirror it to /dev/sdb on the other to configure DRBD device /dev/drbd0.  Format it with an ext4 file system and make sure to perform an initial full synchronization.

When the DRBD device finishes synchronization, create a shared mount point -- /mnt/sdb -- and IP address (10.195.0.100 shared between eth0 on the nodes).  The cluster will now recognize the DRBD device as a file system on the Active node.

The shared data must then be moved to the DRBD device.  Stop MySQL on each node.  On the Active node, move the directories /etc/mysql and /var/lib/mysql to /mnt/sdb/etc/mysql and /mnt/sdb/var/lib/mysql, respectively, and create symlinks back to their original locations.  On the Inactive node, delete the mysql directories and replace them with symlinks to the same locations -- even though those locations are not yet visible.

The LCMC console is then used to finalize the shared services.  Add Primitive LSB Init Script resources (that is, only running on one server at a time) for MySQL and Apache2.

Fail the servers back and forth several times to check that the system performs as expected.  Finally, install a database.  For this article, I install the Zabbix Monitoring database.

The video below illustrates the entire installation and configuration process.

High-Availability Nagios / Icinga on a DRBD - Corosync - Pacemaker Failover Cluster

Failover clustering is relatively uncomplicated and provides high availability.  It consists of two servers -- preferably almost identical -- in which one is active and the other passive.  In the event the active server fails or is manually taken offline, the passive server assumes the active role.

This may sound like the two nodes are independent, but they share resources.  There is data and shared services that must be available to both servers (but not simultaneously). Shared data is stored on a shared drive controlled by the operating system with the help of clustering software or clustering-aware file systems on a SAN; shared services are controlled by clustering software.

Installing and Configuring DRBD, Corosync and Pacemaker with LCMC

The first step is to provide the servers with a shared data drive.  In this case, the shared drive will be two physical drives -- one on each server -- replicated by the Distributed Replicated Block Device (DRBD) software package.  DRBD is analogous to drive mirroring, however the drives reside on different servers and are mirrored over a network connection.  The / partition of the servers is installed on drive /dev/sda and a blank drive -- /dev/sdb -- will be controlled by DRBD.  Once configured, /dev/sdb will no longer be accessible and will be addressed as /dev/drbd0.

The second step is to share the drive with the servers, control which one is active and assign a shared IP address by which the cluster will be available.  This is controlled by the Pacemaker - Corosync clustering software packages.  Pacemaker and Corosync use network interfaces (preferable at least two) to maintain a "ring" over which the servers provide staus updates.  The cluster will move resource control from one server to the other, either manually or automatically.

For this setup, the servers each have four network controllers:

• eth0 -- publicly-available
• eth1 -- dedicated to DRBD data control and replication
• eth2 and eth3 -- dedicated to Pacemaker - Corosync services control

There are several files that control these applications and they may be configured manually. DRBD is set up with the files /etc/drbd.d/global_common.conf and /etc/drbd.d/r0.res (or more if there are multiple drive resources).  Pacemaker - Corosync is set up with the /etc/corosync/corosync.conf file.

There is another application -- the Linux Cluster Management Console (LCMC) -- that automates setting up such a cluster.  LCMC is Java-based and will install / configure all of the software necessary to operate a failover cluster.

The video below demonstrates adding the two unconfigured servers to LCMC, installing and configuring DRBD, Pacemaker and Corosync, setting up and replicating a shared ext4 partition on the shared data drive, adding a Pacemaker-controlled drive partition, adding a shared IP address and testing Active-Passive operations and failover.

Shared Apache, Postfix, MySQL and Icinga Resources

Once the basic shared resources are configured and tested, we can add applications that are controlled by the cluster.  There are a number of other posts that describe setting up Nagios and Icinga in this blog; refer to them for details.

Begin by installing and configuring the Apache2 web server, Postfix mail server and MySQL database server.  Then install and configure all of the Icinga monitoring, web and database packages.  These must be operating correctly on both servers and have identical configurations before clustering may proceed.

Now important distinctions must be identified:  data and services that are controlled by the operating system versus those controlled by the cluster.  For instance, the clustering software runs independently on each server and is controlled by the operating system, available at boot time.  The web, mail and database servers share data and resources and are controlled by the cluster; they must be disabled at boot time and started by the clustering software.  This is very important.  For instance, MySQL will be configured so that the configuration files (/etc/mysql/*.*) and data drive and symlinked back to their original location. Thus, since only one server has access to the configuration and data, the MySQL daemon must be disabled at boot and the DRBD - Pacemaker - Corosync clustering software decides upon which server has access to the files and starts the servers.  I use the Webmin interface to disable all shared services at boot time.

This process is illustrated in the video below.  Upon completion, the cluster will be in control of all shared services (web, mail, database, Icinga, etc.).  Failover is illustrated.

Testing High-Availability Nagios / Icinga on a DRBD - Corosync - Pacemaker Failover Cluster

Once failover is demonstrated, it is time to allow the servers to collect, process and display data.  Which shared data files to move to the replicated DRBD drive depend upon what services are installed.  For instance, PNP4Nagios -- the RRDTool graphing add-on -- stores shared data in /var/lib/pnp4nagios; this file is moved to the DRBD drive and symlinked back to each node.

Also be aware that some directories do not move well.  There are numerous symlinks in the base installations that end up pointing to nothing if moved and symlinked back to the node file systems.  The /etc/icinga directory is a case in point.  If this directory is moved to the shared drive, Icinga's operation becomes, at best, unstable.  Thus, updates to the Icinga configuration must be manually installed on EACH node.


However, as the video below demonstrates, shared MySQL databases and moving selected PNP4Nagios and NagVis directories to the shared drive provides high-availability performance.