Xymon Host Monitoring

View Stephen Fritz's profile

Xymon, like rrdtool, has a long history.  Its oldest antecedent is Big Brother, a project that has become commercial and well worth checking out.  The original Big Brother project lay dormant as the developers concentrated on the Professional edition.  Two notable forks picked up the original project:  Big Sister and Hobbit.  Forks are confusing enough, but the Hobbit team eventually decided to drop the Tolkien-themed names (preferring to avoid copyrighted and trademarked nomenclature) and adopted Xymon.  Working with Xymon can be a bit of a dinosaur dig, with old names sprinkled throughout the software's configuration files.

Xymon is gathers information with an installed client application and monitors TCP services.  The original Big Brother project did not provide much more than that, but the Xymon team has added more functionality.  With a bit of extra work, specific processes may also be monitored.

As with most Debian software, installation is easy:  apt-get install xymon xymon-client apache2.

Also like many Debian web-based application installations, the default installation is locked down and needs a bit of configuration file editing.  

Below you see the first indication of the project's history -- the /etc/apache/conf.d directory's Xymon configuration file is named hobbit.

Simply change the line "Allow from localhost  ::1/128" to "Allow from all."

"Allow from all" by itself is insecure and the web server should be protected with authentication -- the .htaccess file.  The Xymon team has provided documentation of how to use the files that implement their version of Linux/Apache .htaccess authentication -- hobbitpasswd and hobbitgroups located in the /etc/hobbit directory.  Yes, another bit of inherited nomenclature.

 

Once the web server is configured, browse to http://<servername/hobbit.  The monitoring host is already configured.

The configuration files are kept in the /etc/hobbit directory.  Lo and behold, we are greeted with even older heritage -- the bb-hosts and bb-services directories.  Ahhh, that brings back some fond memories.

Most of the editing is in the bb-hosts file.  The format is well-explained in the comments, but the general idea is the hosts lines consist of an IP address, host name, # character and list of services.  These services are in addition to those provided by the xymon client; more on that later.

A few quick edits brings up more hosts.

Unfortunately, it is not uncommon for the client to insert the wrong host name in the monitored hosts /etc/default/hobbit-client file.  Often (as in this case), the host name is pulled from an old /etc/hosts entry that was never updated (my bad).

The host names on the clients must match those in the /etc/hobbit/bb-hosts file.  Once they do, the default client services (CPU, disk, files, memory and msgs, ports and procs) begin to show up.

There is also a monitoring agent for Windows.  The 32-bit application runs as a service and is configured from a file in the C:\Program Files (x86)\BBWin\etc directory.

The configuration file is fairly well documented,but not as simple as the Linux client.  However, it is easy to add server-side TCP service monitoring as described below.

The information and presentation has improved significantly under the Xymon team.  Although no port checks are defined, clicking on a ports icon brings up a netstat output.

Clicking on a procs icon brings up ps output.

Adding addition services to monitor is very easy.  The bb-services file is well documented and defining a service is as simple as naming it and adding a TCP port number to monitor.   This definition will only return a green (up) or red (down) state.  Adding SEND, EXPECT and OPTIONS lines can distinguish a service running in a degraded state and return yellow (degraded) as long as you know how the service operates.  Below is an illustration of the mysql service running on port 3306.

One host returns green and one red.  The explanation is simple:  the down host is newly installed and has the default listen on address 127.0.0.1 configuration in the /etc/mysql/my.cnf file.  Commenting out that line and restarting the mysql service returns the state to green; mysql is now listening on all interfaces.

The Xymon team has incorporated rrdtools reporting into the application.  This is a great benefit for performance trending.

They have also provided a comprehensive host report with the output from Linux systems utilities.

Adding TCP ports and processes to monitor is relatively simple because the application provides netstat and ps outputs.

However, Xymon's notification capability is limited.  If a router or link fails, the monitoring server node will flood failure notifications for every monitored service that is unreachable behind the failed router; it is difficult to filter the root cause of the failure when there are notifications from every service that is unreachable behind the failed router or link.

Since Xymon is primarily a host-based monitoring application, it is not designed to monitor multi-homed hosts.  Thus, in the test network -- with redundant WAN links -- on which it is installed, it may not catch a failed router interface if the configuration monitors hosts, not the individual interfaces.

Even with two failed interfaces, Xymon may not be aware of a problem because all hosts are reachable on at least one interface.

There is a way around this problem: defining interfaces as hosts with unique names.  Thus, each router has an eth0 through eth3 host name configured to monitor the four interfaces.  Simply doing this in the bb-hosts file leads to a rather cluttered monitoring entry page, however.

However, the bb-hosts file supports defining pages below the entry page quite easily.  And since these interface host definitions do not match the host name in the /etc/default/hobbit file, they only report connection info and trends information.  It is a workable and efficient solution; with experience, the formatting syntax is flexible enough to find workable solutions.

Notifications are managed in the hobbit-alerts.cfg file.  Recipients are defined as an e-mail address or script (e.g. sms message sent by a script ).  The following snips of the configuration file provide an overview of some of the options available.

Xymon appears rather limited when first installed, but digging into the well-documented configuration files reveals many more options.  It is simple enough to deploy as an entry-level monitoring tool, with enough features to provide a reasonably comprehensive monitoring, trending, reporting and notification system.  It is not resource-intensive and scales fairly well with the option to deploy multi-server configurations.

Xymon is also a good tool to deploy in an emergency because it sets up fairly quickly and can provide sufficient information to identify problem hosts in a large, complex data center.