Scaling High Performance and Availability Zabbix Systems Monitoring

Background

Systems Engineers frequently comment about heavy loads, performance and scalability problems with Zabbix installations.  Indeed, Zabbix places a heavy load on resources; processes service check information, writes to a database, uses a PHP web application to present data and caches history information.  The database alone can be a heavy processor - memory and disk IO load.  Adding the other processor and memory intensive processes simply adds to the system load.

However, Zabbix was designed to scale up to very large demands.  Those only familiar with the default all-in-one web-application-database installation may consider implementing the available modular architecture to distribute loads.

Scalable Zabbix Architecture

A Scalable Zabbix implementation distributes loads among three classes of servers:

1. Database Back End Servers
2. Zabbix Application Servers
3. Zabbix GUI Web Front End Servers

This article will explain how to implement these classes of servers in a highly scalable deployment.  A High-Availability SAN-Backed MySQL-PHPMyAdmin server will provide two databases, zabbix_01 and zabbix_02.  The child node database will send information to the Master Node, from which data and configurations for all child servers are available.  There are two Zabbix Application Servers dedicated to processing host checks.  There is one GUI Web Front End server dedicated to processing and presenting information (graphs, etc.).

This is a relatively simple implementation of a Scalable Zabbix architecture.  Additional descriptions of High-Availability systems are recommended reading:

• DRBD High Availability MySQL Servers
• SAN-Backed High Availability MySQL Servers
• High Availability Stand Alone Zabbix
• High Availability Zabbix and Zabbix Proxy Servers

The two Zabbix Application and one Zabbix GUI Front End servers may all be deployed as high availability clusters.

Deploying Scalable Zabbix Servers

Create and Populate the Databases

The first task is deploying the two required databases -- zabbix_01 and zabbix_02 -- on the database server.  This is depicted at the end of the video in SAN-Backed High Availability MySQL Servers.

Deploy the Zabbix Application Servers

The two Zabbix Application servers may then be deployed.  Although they only require the zabbix-server-mysql package, you may wish to deploy the zabbix-frontend-php package for initial configuration.  The Apache and MySQL servers installed may be disabled or removed once they are defined and Master and Child servers.  I typically configure the servers entirely from the /etc/zabbix/zabbix_server.conf file, specifying the desired database server (in this case hbg-db.harrisburg.mydomain.com) and database credentials.  It is necessary to also specify a unique NodeID for each server, ranging up to 999.

Once the server is installed, configured and connected to its own database, you must then upgrade the databases from each server for distributed monitoring.  ONLY UPGRADE THE DATABASE ONCE.  Running the upgrade more than once will result in database corruption. To perform the upgrade:

• Stop the Zabbix and Apache services
• Issue the command: zabbix_server -n <NodeID> -c /etc/zabbix/zabbix_server.conf

Deploy the GUI Web Front End Server

The front end only requires the zabbix-frontend-php package.  Once installed, browse to the /zabbix directory and perform the web-based installation.  This server must connect to the backend database cluster (hbg-db.harrisburg.mydomain.com) and the Master Zabbix server (monitor01.harrisburg.mydomain.com).

Once installed, update the default Zabbix Server information to reflect the local nomenclature.  Then, set it as a Local Node, ID 1 and add the child Zabbix Server as Node 2.  Once completed, move to the graphical front-end on the child node (this is the only time it will be needed).  Update its default information and then specify it as a local Node ID 2 and add the Master server Node ID 1.  Then move back to the Master Server GUI and check that the architecture is updated (i.e. the child database is feeding information to the master).

From this point on, other than selecting the desired node, all configuration procedures are like a stand alone Zabbix installation.  The process is illustrated in the video below.