Nagios/Icinga Windows Server, Domain Controller, DNS and IIS Performance Monitoring

This article describes Windows Server 2008 R2 Nagios/Icinga Templates that monitor core server functions, Domain Controllers, DNS Servers and IIS 7.5.  While built specifically for those systems, it is likely  the Templates are compatible with other versions of Windows as well.

For those with Nagios/Icinga and Windows experience, the counters used are available from the Monitoring Exchange page.

Windows Server ships with a excellent monitoring and trend analysis tool: Performance Monitor.  As illustrated below, it allows administrators to select and graph counters that include list of system metrics.  These measurements may also be saved as delimited text files for future analysis and visualization.  A centralized server may connect to other servers to remotely collect data.  Since the API is well-documented, it is integrated into other value-added systems monitoring software.

The illustration below depicts the text format of a Windows Performance Monitor counter.  Commands conforming to this syntax may be sent from remote monitoring servers whose applications comply with the Windows API.

Prerequisites

A thorough knowledge of Nagios/Icinga installation and configuration is necessary.  The fundamental knowledge to Nagios/Icinga is presented in the following articles:
Nagios/Icinga Installation and Initial Configuration
Nagios/Icinga Database Integration
Nagios/Icinga Configuration Files
Nagios / Icinga Configuration with NConf - a Graphical Web / Database Tool
Nagios / Icinga Performance Data Graphing with PNP4Nagios


Information specific to Windows Server enterprise monitoring is presented in these two articles:

Nagios/Icinga Templates for Windows 2008 R2 OS, Domain Controllers, DNS and IIS Servers 
Automated Nagios/Icinga NSClient++ Deployment and Configuration for Windows Enterprises

Description of Monitored Windows 2008 R2 Services and Counters

Nagios/Icinga use Check Commands and Service Checks to monitor Windows.  They use an NRPE Windows Services check to determine if all automatically started and disabled services are in their proper state. They use NRPE Windows Performance Monitoring Ccounter checks to monitor Processor, Memory, Disk and Network counters (among others).  The services and counters listed below are common to Windows Server 2008 R2 regardless of applications installed.  They are indicative of overall performance (or problems) but do not necessarily pinpoint the root issue(s); more advanced -- and specific -- checks are required to diagnose application issues.

Three template files containing the required Check Commands, Service Checks and advanced Performance Monitoring Windows Server Service Checks are available for download from the Monitoring Exchange site.

These checks were gathered from a variety of Microsoft sources.

TCP Ports

135 MSRPC
139 NetBIOS-ssn
445 NetBIOS-ssn

Processor

Processor Information Total Percent UtilizationProcessor Information Total Idle Time
System Processor Queue Length
Server Work Queues Length

Memory

Memory Available MBytes
Memory Free System Page Table Entries
Memory Pages Input/sec
Memory Pages/sec
Memory Pool Nonpaged Bytes
Memory Pool Paged Bytes
Memory Cache Bytes
Memory Percent Registry Quota in Use
Memory Percent Committed Bytes in Use
Paging File Total Percent Usage

Disk

LogicalDisk Avg. Disk sec/Read
LogicalDisk Avg. Disk sec/Write
LogicalDisk Disk Transfers/sec
PhysicalDisk Total Disk sec/Read
PhysicalDisk Percent Idle Time
PhysicalDisk Total Disk sec/Write
LogicalDisk C Free Megabytes
LogicalDisk C Percent Free Space

Network

Network Interface Output Queue Length
Network Interface Bytes Total/sec
Network Interface Bytes Sent/sec
Network Interface Bytes Received/sec

Windows Domain Controller Monitoring and Trend Analysis

Three template files containing the required Check Commands, Service Checks and advanced Performance Monitoring Windows Domain Controller Service Checks are available for download from the Monitoring Exchange site.

Microsoft provides a summary of Windows Domain Controller Performance Counters.

Summary of Monitored Services and Counters

TCP Ports

389 LDAP
464 Kerberos Password
636 LDAPS
3268 Global Catalog
3269 Global Catalog

Windows Server Domain Controller (NTDS) Counters

NTDS DRA Inbound Full Sync Objects Remaining
NTDS DS Notify Queue Size
NTDS LDAP Bind Time
NTDS SAM Account Group Evaluation Latency

Summary of DNS Server Services and Counters

Three template files containing the required Check Commands, Service Checks and advanced Performance Monitoring Windows DNS Server Service Checks are available for download from the Monitoring Exchange site.

Microsoft provides a summary of Windows DNS Server Performance Counters.

TCP and UDP Ports

53 (TCP) DNS
53 (TCP) DNS

Windows DNS Server Counters

Caching Memory
Database Node Memory
Record Flow Memory
Recursive Query Errors
Secure Update Failure
TCP Message Memory
Total Query Received
Total Query Received/sec
UDP Message Memory 
Zone Transfer Failure
Zone Transfer Success

Summary of IIS Server Services and Counters

IIS has changed repeatedly over time and Microsoft-recommended performance counters are generally out-of-date.  The list was developed from a variety of sources and intended to reflect the basic IIS 7.5 Server functions.  Other counters (such as ASP.NET, etc.) are more appropriate to various application environments, such as the Windows Application Server Role, which adds the .NET environment.

Three template files containing the required Check Commands, Service Checks and advanced Performance Monitoring Windows IIS Server Service Checks are available for download from the Monitoring Exchange site.

TCP Ports

80 HTTP
443 HTTPS

Windows IIS Server Counters

Bytes Received/sec
Bytes Sent/sec
Bytes Total/sec
Current Connections
GET Requests/sec
POST Requests/sec
Current Files Cached
Current Metadata Cached
Current URIs Cached
File Cache Hits %
Metadata Cache Hits
URI Cache Hits %

Server 2008 R2, Domain Controller and DNS Server Performance Monitoring Templates

Performance Monitoring Counters are included for advanced troubleshooting, trending and capacity planning.  These counters unlikely useful for day-to-day monitoring and should be used only when needed in those scenarios.

Host Group Assignments

Advanced Services -- as defined above -- are assigned to Host Groups.  When a host is added to a Host group, all of the Advanced Services in that group are then applied to the host.  The OS - Windows Role definitions of Host Groups and Advanced Services defines the groups that must be defined.  These include:

1. windows-servers
2. windows-servers-perfmon
3. windows-domain-controllers
4. windows-domain-controllers-perfmon
5. windows-dns-servers
6. windows-dns-servers-perfmon
7. windows-iis-servers

 Applying the checks now requires no more than adding the host to the appropriate groups.

Example

The video below depicts a Processor stress test conducted on a Windows Server 2008 R2 Domain Controller/DNS Server.  The Warning and Critical values returned for Processor Counters are expected.  Under this high load, there are also Warning and Critical values returned for Domain Controller functions, indicating that there may be issues with authentication and services relying upon it.

Nagios/Icinga Templates for Windows 2008 R2 OS, Domain Controllers, DNS and IIS Servers

This article describes how to use Template Check Commands, Service Checks and Host Groups to monitor Windows Server, Domain Controllers, DNS Servers and IIS -- applications that ship with Windows Server.  Links to sample templates are also provided.  In an enterprise Windows environment, a thorough understanding of the Operating System and Applications is required to successfully utilize sets of commands and checks to monitor and diagnose problems.

Nagios/Icinga Template Formats

Although Nagio/Icinga configuration files allow the administrator to insert any properly formatted definition or command into a file ending in .cfg, there are good reasons to observe several formats.  First, many of the add-on graphical configuration applications (for instance, NConf discussed here) require only a single type of definition or command in each file.  Not only does NConf generate configuration files in this format, it requires them to perform imports.  Service Checks are split into two types:

1. Services -- applied only to single hosts
2. Advanced Services -- applied to multiple hosts and/or service groups

Check Commands are also split into two groups, but they are not used in this article and are not discussed.

It is also easier to troubleshoot configuration errors is a strict format is observed.  This need not split each type of command and check into separate and dedicated files.  However, experience and portability to graphical configuration applications indicate such a structure is the easiest to use and most compatible.

Therefore, the templates used for this article consist of Check Commands (specifying Windows NRPE Service and Performance Monitoring Counter definitions) and Advanced Services (Service Checks defined with a Host Group assignment).  Two examples are provided below:

Check Command

define command {
                command_name                          check_WinNRPEPerf_4ArgMax
                command_line                          $USER1$/check_nrpe -H $HOSTADDRESS$ -n -p 5666 -c CheckCounter -a "Counter:$ARG1$=$ARG2$" ShowAll MaxWarn=$ARG3$ MaxCrit=$ARG4$
}

Advanced Service

define service {
                service_description                   Processor Information Total Percent Utilization
                check_command                         check_WinNRPEPerf_4ArgMax!CPUPercent!\\Processor Information(_Total)\\% Processor Time!80!90
                check_period                          24x7
                notification_period                   24x7
                hostgroup_name                        windows-servers
                use                                   generic-service
                contact_groups                        +admins
}

Several points deserve explanation.  In the Check Command command_line definition, the -n option specifies no SSL, which may or may not apply to specific deployments.  The -c option defines the type of command (which must be supported in the NSClient configuration file).  The -a command passes specific arguments to the agent.  $ARG1$ is the command name.  $ARG2$ is the Windows-formatted Performance Counter to be queried.  $ARG3$ and $ARG4$ pass warning and critical threshold values to be processed by Nagios/Icinga.  The Advanced Service check_command definition the calls the Check Command and specifies the arguments.  Finally, the Advanced Service is only applied to hosts belonging to the windows-servers Host Group.

The Importance of Defining Appropriate Sets of Nagios/Icinga Template Information for Windows Enterprises

Windows 2008 R2 Operating System Roles, Role Services and Features

A default Windows Server installation only installs the software necessary to operate as a server.  Windows then provides "Roles" and "Features" to provide specific additional functionality.  For instance, a Server may be assigned the "Active Directory Domain Services" and "DNS Server" roles that includes one set of functionality while another may be assigned the "Web Server (IIS)" role that includes a different set.  Role Services and Features may also be installed, further adding to the complexity of defining sets of monitored information.  Flowing from the definitions, the fundamental hardware and OS data may be collected by a Template "Windows Server 2008 R2."  Additional Templates then define "Active Directory Domain Services," "DNS Server" and "Web Server (IIS)" Roles and Features.

The templates for each Role/Role Service/Feature set should monitor the availability of services using both the  Nagios/Icinga Plugins "check_nrpe" to query the OS Services and "check-tcp"/"check_udp" to query the NIC TCP and UDP service(s) in question.  The NRPE Services check selected queries all services set to "automatically started" and "disabled"; it returns OK if all services are in the proper state or lists those that are not.  The associated TCP and UDP services on ports 389 (LDAP), 636 (LDAPS) and 464 (Kerberos Password), among others, are queried to determine if the Active Directory services necessary for authentication are available.  Finally, the data collection includes the Performance Monitoring Counters.

There is still another set of definitions for Templates -- information important for day-to-day monitoring and information important for troubleshooting, trend analysis and scalability design.  The second set includes primarily Performance Counters.  Thus, for each set of Windows Role data collected, there is a standard set of data including OS service checks, TCP service checks and Performance Counters and a second, more detailed set that includes primarily Performance Counters.  Flowing from the examples above, the "Windows Server" Template would provide important day-to-day triggers and performance information while a second, "Windows Server Performance Counters" provides highly detailed data for troubleshooting, trend analysis, etc.

Windows Application Servers

The Windows Operating System (with its Roles, Role Services and Features) is also a platform for additional Servers, such as the Exchange E-Mail and Collaboration Server and SQL Database Server.  These servers provide common core software and specialized role-based software; the modular aspect of Windows Application Servers provides fault-tolerance and scalability.  For example, Exchange may be deployed on a single server with all roles and services for small-business environments or may be deployed on many servers with roles (individually or in combination) of Mailbox, Hub Transport, Edge Transport, Client Access and Unified Messaging for large enterprises.  Microsoft provides design guidelines for large Exchange 2010 deployments here and for large Exchange 2013 deployments here.

Template designs for Windows Application Servers must also reflect common and specific sets of data in much the same way as (for example) "Windows Server 2008 R2" is common core functionality and "Active Directory Domain Services" is specific to a role.  Windows Application Server Templates thus defined must also provide to "Core" day-to-day Items, Triggers and Performance Counters and also the less-commonly used Performance Counters set of information.

Conclusion

Using the above guidelines, we may define an (incomplete) example set of Templates to be implemented for the Windows Operating System:

1. Windows Server 2008 R2
2. Windows Server 2008 R2 Performance Counters
3. DNS Server
4. DNS Server Performance Counters
5. Active Directory Domain Services
6. Active Directory Domain Services Performance Counters
7. Web Server (IIS)
8. Web Server (IIS) Performance Counters

We may also define an (incomplete) set Templates to be implemented for the Exchange Server 2010 Application Server:

1. Exchange Server Common
2. Exchange Server Common Performance Counters
3. Exchange Hub Transport Server
4. Exchange Hub Transport Server Performance Counters
5. Exchange Mailbox Server
6. Exchange Mailbox Server Performance Counters
7. Exchange Client Access Server
8. Exchange Client Access Server Performance Counters

The following links are Templates hosted on the Monitoring Exchange web site.
Windows Server Check Commands and Service Checks
Windows Domain Controller Check Commands and Service Checks
Windows DNS Server Check Commands and Service Checks

Windows IIS Check Commands and Service Checks

Automated Nagios/Icinga NSClient++ Deployment and Configuration for Windows Enterprises

Much of the NSClient++ deployment and configuration work may be centralized and automated using Active Directory Group Policy Objects (GPOs).  This article describes agent deployment and .ini file configuration to support core modules, check_nt and check_nrpe commands. 

Deploying the NSClient++ Agent through Active Directory GPOs

Deploying the NSClient++ Program through Active Directory GPOs

The NSClient++ Agent is deployed from a Microsoft Software Installer (.msi) package available and documented here. The package provides everything required to install a default NSClient++ agent. For the purposes of this article, all configurations will be applied to the Default Domain Policy Organizational Unit (OU). The OU(s) utilized in a production environment will vary depending upon the structure of the actual Active Directory domain.  Save the .msi file to a shared directory. Then, open the Group Policy Management Console and edit the policy for the selected OU. The package is then selected for deployment by its shared path (<Server>\,<Share>) under the Computer Configuration > Policies > Software Settings > Software Installation policy. Upon rebooting, each server to which the GPO is applied will then install the agent.  However, the configuration file will be the default installation one and not operational.  The customized installation file (described below) requires a second reboot.

If you also wish to support python scripting, also install the .msi file that may be downloaded here.

The .msi install will also add a Windows Firewall rule (pictured above) to allow any inbound traffic to connect to the nscp.exe process.

The following video depicts the Windows boot process and application installation, admittedly not exciting, but included as a demonstration.

Deploying the NSClient++ Agent Configuration File through Active Directory GPOs

The default configuration file -- c:\program files\nsclient++\nsclient.ini -- deployed above is not customized for a production environment.  You may generate new configurations using the nsclient++ program or simply issue the following command to generate a configuration file with all options supported (but not enabled):

c:\program files\nsclient++>nscp settings --generate --add-defaults --load-all

If you have not installed the python .msi file above, you will receive an error that the python .dll file is missing; simply click through the warning an the process will complete.

A configuration file defining the live environment must be deployed to each node before the agents may communicate with the server.  The nsclient.ini defines many different features (modules, Nagios Remote Plugin Executor (NRPE) Server, Nagios Service Check Acceptor (NSCA), scripts, etc.)  A full description of the capabilities is beyond the scope of this article, but I will focus on modules and NRPE Server settings.

Default settings are located at the bottom of the file.  These include:

allowed ciphers =  [set to: ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH]
allowed hosts = [set to IP address(es) of allowed Nagios/Icinga Servers]
cache allowed hosts = true
certificate = [specify path to SSL certificate)
password = [set to shared password]
timeout = 30
use ssl = [true or false; if true, requires certificate]
verify mode = none

Modules provide the core functionality of NSClient++.  To support Nagios/Icinga check_nt, check_nrpe and NSCA commands, the following apply:

CheckSystem = 1 (important for supporting check_nrpe Performance Counter commands)
NRPEServer = 1 (for Nagios/Icinga check_nrpe commands)
NSCAClient = 1 (for Nagios/Icinga NSCA commands)
NSCAServer = 1 (for Nagios/Icinga NSCA commands)
NSClientServer = 1 (for Nagios/Icinga check_nt commands)

To implement the NRPE Server, the following modifications apply:

allowed ciphers = ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH
allow arguments = true
allow nasty characters = true
port = 5666

These settings define the ciphers allowable, whether NSClient++ accepts arguments and normally illegal characters and the port upon which it listens.  By default, use ssl = TRUE requires SSL encryption; if you disable this, any check_nrpe commands issued by the Nagios/Icinga server must include the -n option to disable SSL.

The file is deployed through the GPO Computer Configuration > Preferences > Windows Settings > Files.  The illustration below depicts deploying the Zabbix Agent configuration file, but the process is the same; simply substitute the nsclient.ini file in the shared folder and GPO configuration.

Nagios/Icinga NRPE Commands

Two of the most useful groups of commands check:

1. Windows Service Availability
2. Windows Performance Monitoring Counters

TCP and UDP Port availability is also important, but is checked directly rather than through the agent.

Windows Service Availability

The easiest way to implement service checks is to define a Nagios/Icinga command:
 
define command {
                command_name                          check_WinNRPEService
                command_line                          $USER1$/check_nrpe -H $HOSTADDRESS$ -n -p 5666 -c CheckServiceState -a CheckAll exclude=WLMS exclude=ShellHWDetection
}
Where:

-H $HOSTADDRESS$ -- variable for the checked IP address
-n -- no SSL Encryption
-p 5666 -- TCP port for NRPE communications
-c CheckServiceState -- specifies a Windows Service check
-a CheckAll -- specifies to check that all automatically started services are running and all disabled services are stopped
exclude=WLMS exclude=ShellHWDetection -- excludes the Windows Licensing and Shell hardware Detection Services from the check

This command will return either OK or a list of services in the critical state.

Alternatively, this is a more granular check for individual services:

define command {
  command_name <<CheckServiceState>>
  command_line check_nrpe -H $HOSTADDRESS$ -p 5666 -c CheckServiceState -a ShowAll $ARG1$ $ARG2$=stopped
}

Here, the "-a ShowAll $ARG1$ $ARG2$=stopped" options specify to check if the specific service name specified in variable $ARG1$ is running and that in $ARG2$ is stopped.

Full Service Check documentation is available here.

Windows Performance Monitoring Counters

Collecting Windows Performance Monitoring Counters requires at least two variables supplied to the check_nrpe command:  a name and the formatted Windows Performance Counter path.  However, to get the most out of the check, it may be desirable to supply additional warning and critical threshold values.  If the threshold is simple (such as exceeds 0 or less than 100), a single variable -- either --MinCrit or --MaxCrit -- is supplied.  If there is both a warning and critical threshold, two variables -- either --MinWarn and --MinCrit or --MaxWarn and --MaxCrit -- are supplied.  Thus, there are file possible Performance Check commands:

define command {
                command_name                          check_WinNRPEPerf_2Arg
                command_line                          $USER1$/check_nrpe -H $HOSTADDRESS$ -n -p 5666 -c CheckCounter -a "Counter:$ARG1$=$ARG2$" ShowAll
}

define command {
                command_name                          check_WinNRPEPerf_3ArgMax    
                command_line                          $USER1$/check_nrpe -H $HOSTADDRESS$ -n -p 5666 -c CheckCounter -a "Counter:$ARG1$=$ARG2$" ShowAll MaxCrit=$ARG3$
}

define command {
                command_name                          check_WinNRPEPerf_3ArgMin    
                command_line                          $USER1$/check_nrpe -H $HOSTADDRESS$ -n -p 5666 -c CheckCounter -a "Counter:$ARG1$=$ARG2$" ShowAll MinCrit=$ARG3$
}

define command {
                command_name                          check_WinNRPEPerf_4ArgMax    
                command_line                          $USER1$/check_nrpe -H $HOSTADDRESS$ -n -p 5666 -c CheckCounter -a "Counter:$ARG1$=$ARG2$" ShowAll MaxWarn=$ARG3$ MaxCrit=$ARG4$
}

define command {
                command_name                          check_WinNRPEPerf_4ArgMin
                command_line                          $USER1$/check_nrpe -H $HOSTADDRESS$ -n -p 5666 -c CheckCounter -a "Counter:$ARG1$=$ARG2$" ShowAll MinWarn=$ARG3$ MinCrit=$ARG4$
}

Zabbix Templates for Windows 2008 R2 OS and Domain Controllers

This article describes how to build Zabbix templates for Windows Server and Application enterprises.  Zabbix Template items must be unique because duplicate items may not be assigned to the same host.  In an enterprise Windows environment, a thorough understanding of the Operating System and Applications is required to successfully define templates for automated deployment.

Zabbix Template Formats

Applications 

Applications are logical groupings of Items.  These should reflect physical and logical aspects of the device monitored.  For Windows Server, this includes hardware (e.g. Disk, Processor, Network and Memory) and software (e.g. Services and Processes).

Items

Items are individual, measurable counters that may be retrieved by querying the OS or Zabbix Agent.  The administrator may easily define time-weighted average sampling to smooth "peaks" and "valleys" in the calculated values.  A sampling rate (frequency) and period (time) define a time-weighted average.  For instance, if the sampling period is defined as 300 seconds and the rate at one sample per 30 seconds, a time-weighted average value for each stored Item value consists of the mean of ten samples.

Triggers

Triggers are counter values that represent degraded conditions.  Zabbix provides the following classifications of Trigger values:

1. Not Classified
2. Information
3. Warning
4. Average
5. High
6. Disaster

Each Trigger also has administratively-assigned fields for Description and URL.  These are very useful as the administrator may assign the URL of a reliable source and description of the Trigger (e.g. Microsoft Technet Performance Monitor references for descriptions and thresholds) and a brief description copied therefrom. 

Graphs

Graphs are the primary visualization for collected data and the feature at which Zabbix excels.  Multiple, related items may be assigned to a single graph (e.g. Processor Idle % and Processor Utilization %) for easy comparison.  Graphs should be grouped according to the types of data present (e.g. rates in counters/sec for one graph, percentages in a second, total counters in a third, etc.).

Graphs, unlike those generated by many RRDTool implementations, are generated just-in-time, saving resource utilization.  That is, without a CGI implementation, RRDTool typically generates all graphs simultaneously using cron jobs.  For an example, see this article on Munin and this article on Icinga/Nagios graphing for examples.  Zabbix only generates graphs when called upon by the administrator, thus saving much of the periodic processing time a cron job implementation requires.

Screens

Screens are logical groupings of graphs and may be designed to present a quick overview of a partial or entire subsystem (e.g. several graphs of processor counters that present the entire set of collected processor data).

The Importance of Defining Appropriate Sets of Zabbix Template Information for Windows Enterprises

Windows 2008 R2 Operating System Roles, Role Services and Features

Zabbix will not allow you to assign Templates to a host if there are duplicates in the Templates.  Thus, if Template A defines and item "perf_counter["\Processor Information(_Total)\% Processor Time",300]" and Template B defines the same item, Zabbix will only allow you to assign one of the Templates.  It is critical to design Templates in advance so that Item, Trigger, Graph and Screen names are not duplicated.  For a Windows Enterprise, the author has selected a framework based upon the overall architecture of the Operating System, its Services and Server Applications.  Yet even then it is more complex than just those three general groups.

A default Windows Server installation only installs the software necessary to operate as a server.  Windows then provides "Roles" and "Features" to provide specific additional functionality.  For instance, a Server may be assigned the "Active Directory Domain Services" and "DNS Server" roles that includes one set of functionality while another may be assigned the "Web Server (IIS)" role that includes a different set.  Role Services and Features may also be installed, further adding to the complexity of defining sets of monitored information.  Flowing from the definitions, the fundamental hardware and OS data may be collected by a Template "Windows Server 2008 R2."  Additional Templates then define "Active Directory Domain Services," "DNS Server" and "Web Server (IIS)" Roles and Features.

The templates for each Role/Role Service/Feature set should monitor the availability of services using both the Zabbix "service_state" check to query the OS and "net.tcp.listen" (or variant thereof) to query the NIC TCP service(s) in question.  For instance, you may check the OS Service state "NTDS" and associated TCP services on ports 389 (LDAP), 636 (LDAPS) and 464 (Kerberos Password) to determine if services necessary for authentication are available.  Service state queries may have multiple triggers that depend on the state returned by Windows (Up, Down, Restarting, etc.) and TCP services a single trigger for Up or Down.  Templates also collect fundamental Performance Counters.

There is still another set of definitions for Templates -- information important for day-to-day monitoring and information important for troubleshooting, trend analysis and scalability design.  The second set includes primarily Performance Counters.  Thus, for each set of Windows data collected, there is a standard set of data including OS service checks, TCP service checks and Performance Counters and a second, more detailed set that includes primarily Performance Counters.  Flowing from the examples above, the "Windows 2008 R2 Server" Template would provide important day-to-day triggers and performance information while a second, "Windows 2008 R2 Performance Counters" provides highly detailed data for troubleshooting, trend analysis, etc.

Windows Application Servers

The Windows Operating System (with its Roles, Role Services and Features) is also a platform for additional Servers, such as the Exchange E-Mail and Collaboration Server and SQL Database Server.  These servers provide common core software and specialized role-based software; the modular aspect of Windows Application Servers provides fault-tolerance and scalability.  For example, Exchange may be deployed on a single server with all roles and services for small-business environments or may be deployed on many servers with roles (individually or in combination) of Mailbox, Hub Transport, Edge Transport, Client Access and Unified Messaging for large enterprises.  Microsoft provides design guidelines for large Exchange 2010 deployments here and for large Exchange 2013 deployments here.

Template designs for Windows Application Servers must also reflect common and specific sets of data in much the same way as (for example) "Windows Server 2008 R2" is common core functionality and "Active Directory Domain Services" is specific to a role.  Windows Application Server Templates thus defined must also provide to "Core" day-to-day Items, Triggers and Performance Counters and also the less-commonly used Performance Counters set of information.

Conclusion

Avoiding duplicate definitions that may lead to the inability to assign multiple Templates to a Zabbix host requires planning and a thorough understanding of Windows Server, Roles, Role Services, Features and Applications design.  A thorough understanding of the critical core Items versus specialized troubleshooting ones is also very important to provide both day-to-day functionality and less-commonly used troubleshooting, trend analysis, and scalability design information.

Using the above guidelines, we may define an (incomplete) example set of Templates to be implemented for the Windows Operating System:

1. Windows Server 2008 R2
2. Windows Server 2008 R2 Performance Counters
3. DNS Server
4. DNS Server Performance Counters
5. Active Directory Domain Services
6. Active Directory Domain Services Performance Counters
7. Web Server (IIS)
8. Web Server (IIS) Performance Counters

We may also define an (incomplete) set Templates to be implemented for the Exchange Server 2010 Application Server:

1. Exchange Server Common
2. Exchange Server Common Performance Counters
3. Exchange Hub Transport Server
4. Exchange Hub Transport Server Performance Counters
5. Exchange Mailbox Server
6. Exchange Mailbox Server Performance Counters
7. Exchange Client Access Server
8. Exchange Client Access Server Performance Counters

Each Template contains (where available and appropriate):

1. Item and Trigger Descriptions
2. Time-weighted Item collection (generally a 30 second collection interval averaged over a 300 second collection period)
3. Trigger URL references to the source reference page for the definition
4. Graphs of individual and sets of items
5. Screens of Application Groups that collect various screens into logical and complete visualization sets

Automated Zabbix Deployment and Configuration for Windows Enterprises

Deploying Zabbix in the Windows Enterprise consists of three automated tasks:

1. Deploying the Zabbix Windows Agent and configuration file through Active Directory Group Policy Objects (GPOs)
2. Configuring Windows for the Zabbix Agent through GPOs to collect data and
3. Configuring Zabbix Discovery of Windows Servers and Services

All of these tasks may be automated for enterprise deployments using Active Directory and the Zabbix Server.  This document shall describe these tasks in detail.

Deploying the Windows Zabbix Agent through Active Directory GPOs

The Zabbix Windows Agent is deployed from a Microsoft Software Installer (.msi) package available and documented here.  The package provides everything required to install a default Zabbix agent.  For the purposes of this article, all configurations will be applied to the Default Domain Policy Organizational Unit (OU).  The OU(s) utilized in a production environment will vary depending upon the structure of the actual Active Directory domain.

Save the .msi file to a shared directory.  The, open the Group Policy Management Console and edit the policy for the selected OU.  The package is then selected for deployment by its shared path (<Server>\,<Share>) under the Computer Configuration > Policies > Si=oftware Settings > Software Installation policy.  Upon rebooting, each server to which the GPO is applied will then install the agent.

Deploying the Zabbix Windows Agent Configuration File through Active Directory GPOs

The default configuration file -- c:\program files\zabbix agent\zabbix_agentd.conf -- deployed above is not customized for a production environment. A configuration file defining the live deployment must be deployed to each node before the agent may communicate with the server.

At a minimum, the following items should be configured in the customized agent .conf file:

1. LogFile=<Path and Name of Zabbix Agent Log File>
2. Server=<IP Address of Zabbix Server>
3. ServerActive=<IP Address of Zabbix Server>
4. EnableRemoteCommands=1
5. LogRemoteCommands=1
6. HostnameItem=system.hostname

These settings configure the agent to log to a defined location, communicate with a specified Zabbix server, recive and log remte commands and, finally, report the system NetBIOS name as the host name.

Windows Service Discovery using the Zabbix Agent

Describing the variety of agent configuration options available is beyond the scope of this article.  However, one other option is used:

1. UserParameter=services.NTDS,net start NTDS

This option defines a Zabbix Agent parameter "services.NTDS" that issues the shell command "net start NTDS."  The shell command attempts to start the Active Directory Domain Service.  If the host server is not a Domain Controller, it replies with an error; if it is a Domain Controller, it will reply that the service is already started.


The services.NTDS Zabbix Agent parameter thus defined provides Windows Service Discovery, a feature lacking in Zabbix at the time of writing.  There are other methods of Windows Service Discovery, however this one is chosen for simplicity and ease of configuration and deployment through GPOs.


Once the configuration file is prepared, it is placed in a shared folder.  It is deployed by editing the GPO's Computer Configuration > Preferences > Windows Settings > Files option.  Specify the network path (<server>\<share>) to the file and configure it to Replace within the c:\program files\zabbix agent folder.

Configuring Windows Firewall for Zabbix Agent-Server communications through Active Directory GPOs

Edit the GPO's Computer Configuration > Policies > Windows Settings > Security Settings > Windows Firewall with Advanced Security > Inbound Rules to install a Firewall Rule.  The rule must allow TCP port 10050 Inbound.  It may be more restrictive, but that is the minimum required.

Configuring a Registry Key to Collect Windows Database Advanced Counters

While not necessary for a basic deployment, there are circumstances in which collecting advanced ESENT (database) counters are helpful.  An example will be reviewed in an article on monitoring Exchange Mailbox Servers.

Configuring Advanced Counters support requires a registry edit.  On the Domain Controller, open the Registry Editor (Run regedit.exe) and navigate to \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESENT\Performance.  Add a DWORD Value named "Show Advanced Counters" and set its value to "1."

Then, edit the GPO's Computer Configuration > Preferences > Windows Settings > Registry and select the local Show Advanced Counters Value configured above.

Finally, set the Action to Replace.

Configuring Zabbix Autodiscovery of the Windows Operating System and Defined Services

Although the Windows Servers are now configured to communicate with the Zabbix Server, the server itself does not have any recognized nodes.  Manual configuration is impractical in an Enterprise, so Zabbix provides Autodiscovery.  The video at the bottom of the page illustrates a general configuration case.  The following sections illustrate how to configure Zabbix for Operating System and Service Autodiscovery of Windows Servers.

Configuring Zabbix Autodiscovery of the Windows Operating System

The first step is to define a Discovery Rule.  From the Zabbix Web Interface, select the Configuration tab and Discovery item.  Then, create a Discovery Rule "Windows Server," specify an IP address range and define the Check Zabbix agent "system.uname" to return a verbose operating system description from discovered agents.

 

Next, select the Actions item and define an Action.  Under the Conditions tab, choose the previous Discovery Rule name and define the Received Value "Like" and "Windows."  This will look for the word "Windows" in the returned verbose operating system description and True if "Windows" appears in it.  Also define the Discovery Status as "Up" and the Service Type "Zabbix Agent."

Finally, define Operations for the Action.  Define the actions "Add Host,"  "Add to Host Group," and "Link to Template," to add the discovered node to a predefined host group and template.  You may also wish to add the operation Remove from host group "Discovered Hosts."

Templates for Windows Server are included with the default Zabbix installation.  You may also customize or import templates.

 

Configuring Zabbix Autodiscovery of the Windows Domain Controller Service

The item "UserParameter=services.NTDS,net start NTDS" deployed in the Zabbix Agent configuration file now comes into play.  Service Autodiscovery is no more complicated than the built-in OS Autodiscovery described above.

Once again, the first step is to define a Discovery Rule.  From the Zabbix Web Interface, select the Configuration tab and Discovery item.  Then, create a Discovery Rule "Windows Domain Controller," specify an IP address range and define the Check Zabbix agent "system.NTDS" to return a verbose Service response.

Next, select the Actions item and define an Action.  Under the Conditions tab, choose the previous Discovery Rule name and define the Received Value "Like" and "already been started."  This will look for the phrase "already been started" in the returned verbose service response and True if "already been started" appears in it.  Also define the Discovery Status as "Up" and the Service Type "Zabbix Agent."

 

Finally, define Operations for the Action.  Define the actions "Add Host,"  "Add to Host Group," and "Link to Template," to add the discovered node to a predefined host group and template.  You may also wish to add the operation Remove from host group "Discovered Hosts."

At this point, the Windows Servers and Zabbix Server are configured to automatically deploy all required software and settings and perform Autodiscovery.  Simply enable the Autodiscovery and Actions items and all discovered servers will be added as hosts with host groups and templates applied.

This demonstration includes five Windows 2008 R2 Servers, one of which is a Domain Controller.  As shown below, Zabbix discovers these hosts and adds them to the proper groups.  However, it does NOT recognizes neither the NetBIOS not DNS names and adds the servers by IP address.  This is an ongoing problem and requires manual updates for each host.