Zabbix Templates for Windows LLD Discovery

This article describes Windows Server Zabbix Low Level Discovery (LLD) Templates that monitor core server functions.  While tested on Windows Server 2012 R2, it us likely  the Templates are compatible with other versions of Windows as well.

For those with Zabbix and Windows experience, the templates used are available from the Zabbix Share Templates page.  A previous version of Windows Templates is described in this blog post on Windows Server 2008 R2 Performance Monitoring.  The previous templates used Windows Performance Monitoring (_Total) and (*) instances to collect data.  While this provides overall systems performance indicators, it is not highly precise.  For instance, if the Disk Queue Write Length (_Total) exceeds the warning threshold, it applies to all disks on the server and does not identify the specific disk that is the problem.

Zabbix Low Level Discovery (LLD) provides more specific information about the hardware and software running on Windows Servers.  Three components are used:

1. A Discovery Rule that defines what information will be obtained.
2. A UserParameter statement in the Zabbix Agent zabbix.conf file defines what scripts will be used.
3. A PowerShell script that queries the Windows Operating System and returns JavaScript Object Notation (JSON) formatted variables used in the Discovery Rule.

Instead of relying on (_Total) and (*) instances written into the counters, the Discovery Rule will enumerate individually-returned items, such as Logical Disks C:, D:, E:, etc.

Discovery Rules

Rule Definitions

Discovery Rules are written in the Zabbix Template.  The definition page requires:

• Unique Name such as windowsldisk.discovery for Logical Disk Discovery
• Type (Zabbix Agent for all rules in this article)
• Key to define what UserParameter to run on the Zabbix Agent
• Update Interval in seconds

You may also add Flexible Intervals and define how long items that are no longer discovered are retained.  The last feature is useful if you are, for example, monitoring SMB-mapped drives shared by a failover cluster.

Filters use regular expressions and macros to filter the returned results.  For instance, calling the {#FSTYPE} macro that uses the @File systems for discovery defined regular expression will return results for matching values (e.g. ext4, ntfs) and filter out those not desired (e.g. cdfs).

For Windows Server Discovery, the macro is simply defined from the data returned by a PowerShell script (see below) that filters on the server before returning values to Zabbix.  We do not need to define Zabbix-level filters and simply use the macro name defined in the PowerShell script.

Item Prototypes

Item prototypes are similar to regular items in format, except they typically reference a macro instead of a defined value.  For Windows Discovery, notice both the Name and Key contain the macro {#DISKNUMLET}.  This acts as a variable and references all of the items returned by the PowerShell script in JSON format.  As discussed in more detail below, the PowerShell script will filter all values to return the logical drives recognized by the operating system (C:, E:, F:, etc.) filtering out the CD drive.  Keep in mind logical drives include mapped SMB shares physically hosted on other servers.

Item details define how items will be stored, reported and -- importantly -- the macro and Zabbix operation performed.  The Type of operation is always Zabbix Agent because the process is sent to a remote agent for execution.  The perf_counter key instructs Zabbix to use a Windows Server-formatted Performance Monitoring item and pass the defined operation and macro name to the Agent.  The returned item is a Numeric (float) type and may be assigned appropriate units (Bytes, Bytes/sec, sec, millisec, etc.).  Value mapping may also be assigned.  These define how numeric values returned by the Agent are interpreted.  For Windows, the agent will return a numeric value for the service state.  The Value Mapping maps the numeric value to a human-readable value value (e.g. running, paused, stopped, etc.).

Trigger Prototypes

Collecting data is useful for trend analysis.  Triggers define thresholds at which Zabbix generates alerts.  A complete discussion of Triggers is beyond the scope of this article.  Windows Server-specific Triggers warrant description.

Microsoft's MSDN and Technet provide lists of suggested Performance Counter thresholds that are easily translated into Zabbix Triggers.  The illustration depicts a Warning Trigger for Logical Disk sec/Read (sec) on a specific drive; if the read time is greater than 0.015 seconds, Zabbix generates a Warning alert.  This trigger was then cloned and the threshold value changed to 0.025 seconds and Warning changed to High to create a higher-rated alert.  Decisions about forwarding may be made based upon the severity of the alert.

Graph Prototypes

Graphical displays of information are useful for trend analysis and diagnosing problems.  Zabbix Discovery Graph Prototypes are much like standard graphs but use macros in place of defined objects.  One graph prototype, as illustrated, calling macros will generate a graph for each item returned.  Thus, a Windows Server with three logical drives will have three of each graph and the macro will list each drive name in the title.

Zabbix Agent

The Zabbix Agent controls communications between the Windows Server operating system and Zabbix server.  Its configuration file -- zabix.conf -- defines the UserParameter functions that the Zabbix server passes to the agent in order to execute Windows PowerShell scripts.

Following the Logical Disk Discovery described above, the zabbix.conf file requires a UserParameter statement for each different script used.

UserParameter=windowsldisk.discovery,powershell -NoProfile -ExecutionPolicy Bypass -File c:\scripts\get_ldisks.ps1

This UserParameter line responds to calls from the the Zabbix server windowsldisk.discovery definition and invokes PowerShell to run with the privileges necessary to execute the script c:\scripts\get_ldisks.ps1.  The agent then returns the values to the Zabix server.

PowerShell Scripts

PowerShell is the Microsoft Windows scripting language used to query Performance Monitoring. The following script queries the Logical Disks Performance Counters and returns the macro-defined {#DISKNUMLET} and drive number-letter name:

1. $drives = Get-WmiObject win32_PerfFormattedData_PerfDisk_LogicalDisk | ?{$_.name -ne "_Total"} | Select Name
2. $idx = 1
3. write-host "{"
4. write-host " `"data`":[`n"
5. foreach ($perfDrives in $drives)
6. {
7. if ($idx -lt $drives.Count)
8. {
9. $line= "{ `"{#DISKNUMLET}`" : `"" + $perfDrives.Name + "`" },"
10. write-host $line
11. }
12. elseif ($idx -ge $drives.Count)
13. {
14. $line= "{ `"{#DISKNUMLET}`" : `"" + $perfDrives.Name + "`" }"
15. write-host $line
16. }
17. $idx++;
18. }
19. write-host
20. write-host " ]"
21. write-host "}"

Line 1 invokes the LogicalDisk query command and filters out the _Total item, returning only the counters' names (not the voluminous additional data associated with each).  Line 2 sets an index at 1 and Line 17 increments it.  Lines 3 and 4 write the required headers for JSON format.  Lines 5 through 16 query each drive item returned in Line 1 and writes a formatted pair -- {#DISKNUMLET} and Drive Name -- in JSON format.  Lines 19 through 21 then complete the JSON-formatted response.

{

"data":[

{ "{#DISKNUMLET}" : "C:"},
{ "{#DISKNUMLET}" : "E:"},
{ "{#DISKNUMLET}" : "F:"},
{ "{#DISKNUMLET}" : "G:"}

 ]
}

Don't try to look up a list of Get-WmiObject commands because what Microsoft documents is incomplete.  There are simply too many and they are installed as needed with Roles and Applications.  Fortunately, PowerShell also provides a command syntax that will export all available Get-WmiObject commands to a .csv-format file:

Get-WmiObject -List | Where-Object { $_.name -match 'perfformatted' } | Export-CSV c:\scripts\perfformatted.txt

You may the search this lengthy document for the syntax needed to create other PowerShell scripts.

Summary

You may manually install the Zabbix Agent on Windows host, script the installation or use Group Policy.

1. Create a Zabbix Discovery rule with a named macro, filters (if necessary) items, triggers and graphs.
2. Add UserParameter statements to the client agent zabbix.conf file referencing the Zabbix Discovery rule and calling PowerShell scripts.
3. Add PowerShell scripts to the client.

Windows Server Templates

There are two Templates on Zabbix Share for Windows Server Discovery:

• Windows Server Discovery (LLD)
• Windows Server Discovery (LLD) Performance Monitoring

The first template is adequate for day-to-day monitoring and trend analysis.  The second template is very thorough, Zabbix Serverv-intensive and intended for diagnosing difficult problems.

Each .zip file downlaod contains the template, UserParameter statements to be added to the zabbix.conf file and PowerShell scripts to be placed in the c:\scripts directory.  There is a brief README file explaining what needs to be done.