Zabbix Templates for Windows Cluster Services LLD Discovery

This article provides an overview of Microsoft Clustering and its underlying technologies.  Microsoft Cluster Services provide failover clustering for increased availability.  It supports several types of shared storage and implements older technologies, such as SMB, in new ways.  Finally, it presents Zabbix templates for resource discovery, monitoring, alerts and trend analysis.

The Zabbix templates (with PowerShell scripts and zabbix.conf agent modifications) may be downloaded from the Zabbix Share site:

• Windows Server Discovery (LLD)
• Windows Server Discovery (LLD) Performance Monitoring
• Windows Cluster Services (LLD)
• Windows Cluster Shared Volumes (LLD)

These are zip files containing and contain a README.

Windows Cluster Services Components

Windows Clusters are primarily failover, in which one node is active and the other standing by ready to take over if the active -- Coordinating Node -- goes off line.  Shared resources -- such as services, the shared network address and some storage types -- will only be active on the Coordinating Node.  Monitoring strategies need to address the differences between the Cluster Server System, Coordinating Node and remaining nodes.

Monitoring should include Cluster Software Components and Cluster Objects.  Software Components are generally operating system level software items that control the cluster and its nodes.  Objects are hardware and software features/services controlled by the Cluster Software Components.  Cluster Software Components include:

• Cluster Service
• Disk Driver
• Resource Monitor and DLLs
• Database
• Administration Interface

Cluster Objects include (among others):

• Network Interfaces
• Nodes
• Resources

Cluster Object Resources are the raison d'etre of clusters, the IP addresses and services presented to the network.  The other components are more specific to the individual nodes. 

While all of these components are important, not all of them have Performance Monitor counters that provide actionable information.  Those that merely provide running totals -- not rates or thresholds -- are not particularly useful.  Thus, merely tracking the number of database flushes does not provide particularly useful  diagnostic information.

The operation of individual nodes may be monitored at the operating system level as detailed here and here.  Items such as network interfaces, disk utilization, etc. are not unique to clusters and operate much as any other Windows operating system.  There are two applicable Zabbix Templates:  Windows Server Discovery (LLD) and Windows Server Discovery (LLD) Performance Monitoring.  The first template is designed for detailed day-to-day monitoring while the second (which is Zabbix Server intensive) is intended as an addition to the first for diagnosing more complex problems.

Persistent and global vs. ephemeral or Coordinator/Standby Node items are also an important distinction.  The Windows Cluster Service is an item that runs on all nodes at all times and its operation is mandatory.  If it fails on any node, we want to know about it.  Shared Storage may be more ephemeral.  Consider the case of Physical and Logical Disks used as Shared Storage.  In this case, the Coordinator Node (left) recognizes two Logical Disks: F: and H:.  The Standby Node (right) recognizes none.

The illustration below depicts the Physical Disks recognized by the Coordinator Node on the left and a Standby Node on the right.  Each recognizes four physical disks.  The Coordinator Node lists two (Disks 2 and 4) as Logical Disks F: and H:.  Disk 1 is recognized as an Online Cluster Shared Volume and Disk 3 an Offline Cluster Shared Volume.  The Standby Node recognizes Disk 3 as Online.

The Windows Cluster Manager application provides another view of this:

Monitoring Global and Persistent Cluster Items

The Zabbix Windows Cluster Services template is designed to monitor global and persistent services.  It consists of fourteen items, three triggers, three graphs and a discovery rule; the discovery rule consists of four item, three trigger and one graph prototype.

The items monitor the Windows Cluster Service, Global Update Manager, Resource Control Manager and Total Resources (numeric, not by name).  The Discovery Rule is required because it enumerates the names of member nodes in the cluster and monitors network reconnections and message queues between the nodes.

Monitoring Shared Storage

Shared storage is ephemeral and moves between the Coordinator and Standby Nodes as needed.  Its monitoring is a much more complex issue.  Understanding how to monitor shared storage and interpret the results requires understanding the different types of shared storage Windows Clusters use.

Server Message Block (SMB)

Microsoft Server Message Block (SMB) is a client/server file sharing protocol.  On the client side, a Logical Drive may be mapped to a server SMB share.  SMB operates at the Application and Presentation Layers 6 and 7 of the OSI model.


The Distributed File System (Dfs) is an extension of the SMB protocol that creates a single share -- the Dfs Root -- that logically organizes shares on multiple servers.  These distributed shares may then all be accessed through the DFS Root.  Dfs also provides file replication for increased fault tolerance.


Windows Server 2012 introduced the SMB 3 protocol and among its capabilities is the Scale Out File Server (SOFS).  A SOFS system distributes data across many enclosures and manages all file replication; it can even be used with inexpensive commodity JBOD enclosures lacking RAID hardware.  SOFS is not intended for use as a traditional file share because the amount of metadata traffic (from opening, modifying and saving files) would consume excessive network bandwidth.  It is better suited to managing and replicating large files such as databases and Virtual Hard Disks, as used by failover clusters.

iSCSI, Serial Attached SCSI and Fibre Channel

Microsoft supports iSCSI, Serial Attached SCSI, Fibre Channel and SMB 3 shared storage in failover clusters.  From the Windows Server perspective, these types of storage present themselves (through Host Bus Adapters, Network Interfaces and drivers) as physically-attached SCSI storage addressed by its Logical Unit Number (LUN).  That is, they appear as Physical Drives and may be partitioned and formatted into one or more Logical Drives.  However, such storage is often not partitioned into Logical Drives.

iSCSI operates at the Session Layer 5; it has less network overhead than SMB shares and is often faster on the same hardware.  Fibre Channel does not operate using the OSI model because it is not Ethernet.  However, Fibre Channel switches do operate in layers in a manner somewhat analogous to the OSI model, but lacking layers that correspond to the OSI Session, Presentation and Application layers 5-7:

• FC4: Protocol Mapping layer for protocols such as SCSI.
• FC3: Common Services layer, a thin layer for encryption or RAID redundancy algorithms.
• FC2: Network layer, consists of the core of Fibre Channel, and defines the main protocols.
• FC1: Data Link layer, which implements line coding of signals.
• FC0: PHY, includes cabling, connectors etc.

SAS is a technology that operates more like locally-attached storage; it does not correlate to networking models because it uses Host Bus Adapters and drivers to access storage.


SMB 3 shares operate at the Presentation and Application Layers 6 and 7 of the OSI model.  SMB 3 provides the shared storage space for Virtual Hard Disks (VHDs) used by Hyper-V and Clusters.

File Systems

The dominant Windows file systems are NTFS and ReFS.  NTFS remains the one used for the Windows Server system/boot partition.  It is more feature-rich than ReFS.  ReFS is designed for scalability and resiliency; it is the file system of choice for very large data storage.

These file systems may be mounted as drives by one or more SMB clients.  The SMB server arbitrates file access to prevent more than one client from accessing -- and corrupting -- data simultaneously.

iSCSI, SAS and Fibre Channel storage is also formatted as NTFS or ReFS, but if two or more servers mount them as drives, the file systems themselves will not prevent simultaneous data access and corruption.  NTFS and ReFS are not Clustered File Systems.  Microsoft has provided the Cluster Shared Volume File System (CSVFS) since Windows Server 2008 R2.  This file system allows two or more mount the drive.

CSVFS is really an NTFS- or ReFS-formatted file system managed by Windows Failover Cluster Services.  Each LUN is actually a CSV FS-formatted VHD that resides on an NTFS or ReFS partition.  The server in control of the LUN is called the Coordinator Node.  The  Coordinator Node addresses the LUN using the appropriate SCSI protocol (iSCSI, SAS or Fibre Channel).  It then creates an SMB share under the %SystemDrive%\ClusterStorage directory that may be addressed by other nodes in the cluster using the SMB protocol.

CSVFS is quite different from other cluster-aware file systems such as Oracle's OCFS2 because it uses so many older technologies -- file systems, VHDs and SMB -- to provide simultaneous file system access.  There is involvement of the higher layers of the OSI model with attendant overhead, which is manifest as additional processor, memory, network utilization and potentially Logical Disk bottlenecks.

Yet there is a method to this seeming madness.  Microsoft's goal is to implement cloud storage protocols that scale beyond SAN technology on commodity hardware -- JBOD enclosures.  Thus, the Microsoft Cluster Services utilize technologies and protocols that are tested and closely related to its SMB 3 SOFS strategy.

Monitoring NTFS and ReFS Partitions

NTFS and ReFS partitions -- not being cluster-aware file systems -- are controlled by the operating system and cluster service.  The operating system will recognize the Physical Disks, which may be readily monitored using the Windows OS Discovery (LLD) template.  This template will also recognize Logical Drives, but only on the Coordinator Node.  Thus, standby nodes will either not recognize the presence of offline Logical Disks or (if they have been the coordinator Node in the past) Zabbix will show them as Not Present.  This is a minor issue and if you change the go to the template's LogicalDisk Discovery rule and change the value of "Keep lost resource period (in days)" from the default 30 to 1 (or even 0), the offline Logical Disks will be more quickly removed from the host's discovered items.

The most important information may be obtained by configuring a host for the cluster shared IP address with the Windows Discovery (LLD) template.  The cluster will be aware of all physical and logical disks managed by the Cluster Service and should remain unchanged.

Monitoring CSVFS Partitions

Monitoring the underlying Physical and Logical disks will discover CSVFS partitions.  However, as describe in the sections above, CSVFS relies upon the SMB protocol and any node in the cluster (Coordinator and Standby) may access a CSVFS partition.  Recent versions of Windows Cluster Services intentionally distribute the Online and Offline CSV partitions between nodes in the cluster, both the coordinator and standby.  However, CSVFS partitions differ in that they are NOT recognized as Logical Disks by any node at the operating system level and are logically controlled and addressed solely through the Cluster Service, at a higher level of the operating system than iSCSI, SAS and Fibre Channel shared storage.

CSVFS partitions are so different from iSCSI, SAS and Fibre Channel partitions they require a separate Zabbix template -- CSV Cluster Shared Volumes (LLD).  This template consists of three discovery rules:

1. Volume Manager -- 6 Item, 6 Trigger and 2 Graph Prototypes
2. File System -- 26 Item, 4 Trigger and 5 Graph Prototypes
3. SMB Server -- 10 Item, 3 Trigger and 3 Graph Prototypes

CSVFS is designed to be fault-tolerant and error transparent.  The template -- PARTICULARLY THE TRIGGERS -- are intended to be applied to the shared cluster host and its corresponding IP address.  The triggers target Redirected IO, something that should be minimal on the Coordinator Node but may be expected on others.  Please read Microsoft's Shared Volume Performance Counters article for an in-depth description of these Performance Monitoring Counters.  The triggers are configured according to the article but may provide false positives under some circumstances.

The triggers are indicative of problems, but diagnosis may require additional trend analysis.

Zabbix Template Deployment

There is a README file included in the zip file download that explains how to deploy the templates, modify the zabbix.conf file and add scripts.

In summary, you may manually install the Zabbix Agent on Windows host, script the installation or use Group Policy.

1. Create a Zabbix Discovery rule with a named macro, filters (if necessary) items, triggers and graphs.
2. Add UserParameter statements to the client agent zabbix.conf file referencing the Zabbix Discovery rule and calling PowerShell scripts.
3. Add PowerShell scripts to the client.

You may also wish to review the Windows Server Discovery LLD article for more detailed information.