Linux Layer 2 Open vSwitch with Floodlight Virtualization

Illustrates how to configure Open vSwicth and its web interface Floodlight to connect to VirtualBox and GNS3 virtual machines.  Also illustrates integrating Zabbix to monitor the topology.

Introduction

Open vSwitch is a multilayer switch capable of distribution across multiple physical hosts.  It supports standard management tools and protocols (OpenFlow, NetFlow, SFlow, etc.).   Floodlight is a Java web-based monitoring interface that reports information about bridges, attached hosts and flows.

As of version 3.12, the Linux kernel supports Open vSwitch; older kernels require a compiled kernel module.  At the time of writing, Debian Wheezy (kernel version 3.2.x) does not include native support, so this article focuses on Ubuntu Trusty Tahr (14.04) with a 3.13.x kernel.

Open vSwitch on an Ubuntu Host

Installation

Ubuntu 14.04 includes a kernel with Open vSwitch support, so installation from the repository is simple:

sudo apt-get install openvswitch-switch

Configuration

For a simple Layer 2 implementation, configuration requires only five steps, the last of which is optional:

1. Create the bridge
2. Add ports to the bridge
3. Set the Floodlight controller
4. Assign an IP address
5. Configure SNMP

The first three steps use the ovs-vsctl command line utility while the last two are standard Debian/Ubuntu configurations.

Create the Bridge

Create the bridge with the command:

sudo ovs-vsctl add-br <bridge name> or, specifically,
sudo ovs-vsctl add-br br0

Add Interfaces to the Bridge

There are eight network interfaces on each VirtualBox VM (see this article for information on adding more than four interfaces to VirtualBox appliances).  Add them to the bridge with the commands:

sudo ovs-vsctl add-port <bridge name> <port name> or, specifically,
sudo ovs-vsctl add-port br0 eth0
sudo ovs-vsctl add-port br0 eth1
...
sudo ovs-vsctl add-port br0 eth7

Set the Floodlight Controller

The Floodlight controller is a centralized appliance that controls decision making in the network.  More on that below.  For now, configure each switch to use the Floodlight controller Host-02 at the IP address 10.120.0.102 listening on TCP port 6633:

sudo ovs-vsctl set-controller <bridge name> <controller> or, specifically,

sudo ovs-vsctl set-controller br0 tcp:10.128.0.102:6633

Assign an IP Address

This scenario uses cloned Ubuntu VirtualBox appliances.  Their initial IP addresses must be changed and default gateway (10.128.0.1) reset.  Use the iproute2 set of commands to reconfigure networking:

ip address del <existing IP address/CIDR mask> dev <interface>
ip address add <IP address/CIDR mask> dev <interface>
ip route add 0.0.0.0/0 via <gateway address>

For the cloned Ubuntu VirtualBox switches used in this scenario with the IP address 10.120.0.201/24 assigned to interface eth0:

ip address del 10.128.0.201/24 dev eth0
ip address add 10.128.0.x/24 dev br0
ip route add 0.0.0.0/0 via 10.128.0.1

The Ubuntu switches are numbered starting at 10.128.0.254 and descend as new ones are added. Restart networking and then change the /etc/network/interfaces file to reflect the operational configuration.

Configure SNMP

This step is optional.  However, if you wish to monitor the devices using SNMP application, see this article for more information.  Adding additional SNMP MIBs is not necessary; only SNMP Daemon support is required.

Floodlight Controller on the Linux Host

Floodlight is an OpenFlow-compliant controller.  Open vSwitch, also being OpenFlow compliant, may use Floodlight.  A simple explanation of Software-Defined Networking (SDN) is a system in which the frame and packet forwarding (the data plane) is separated from the forwarding decision logic (the control plane).

Traditional switches MAC address tables and exchange topology information to build forwarding logic on each switch.  They may also run Spanning Tree Protocol (and other protocols) to optimize forwarding decisions.  Two problems (among others) with this approach is that it 1) consumes processing and memory resources on each device and 2) requires time, processor and memory resources for network topology changes to converge.

SDN networking, removes the decision-making logic from individual switches and replaces it with a centralized controller -- in this case Floodlight -- that stores all MAC and topology information on a single server.  There are advantages to this system, including rapid change convergence, faster switch performance and a complete topological map available to administrators for manual configurations (such as prioritization and filtering).

There is a significant drawback to SDN OpenFlow controllers: they are a single point of failure.

Installation and Configuration

Installation requires no more than using apt to get all necessary packages from the repositories:

sudo apt-get install floodlight

Floodlight is a Java application, so a lot of additional packages are required.  That's it.  Once installed, it will listen on TCP port 8080 of all network interfaces.

Floodlight Web Interface

The Floodlight web interface only provides information about the switches and hosts.

Dashboard

The Dashboard provides an overview of network devices including switches, MAC and IP addresses, number of flows, total data transfers and controller processes.

Topology

Topology provides an application-generated overview of switches, hosts, MAC and IP addresses and the connections between devices.  However, it is not easily configurable, crowded, poorly-organized and difficult to interpret even in small networks.

Switches

The Switches overview lists each configured switch by MAC and IP addresses and includes total data transfers and flows on each.

Clicking on the linked MAC address (DPID) of each switch provides port- and flow-level detail of each switch.

Hosts

Hosts provides an overview of each unique MAC address on the network -- be it a switch, router or server.  It also indicates the MAC address to which each device is connected.

Floodlight REST API

The Floodlight REST API provides a URL-based interface to more detailed information about the controller and configured switches.  For instance, the URL http://<controller>:8080/wm/topology/links/json provides a summary of each switch-to-switch link.

The link above lists all of the URLs included in the current Floodlight release.

Floodlight Avior

Avior is a Java application developed and maintained by Marist College and IBM among others.  This graphical interface provides overviews similar to the Floodlight web interface and -- more importantly -- configuration options for flows, filtering and many other Layer 2, 3 and 4 traffic control.  This article provides only a brief overview of the application.  The figures below are the Switches and Devices options -- analogous to Switches and Hosts in the Floodlight web application.

The illustrations below provide an overview of flow configuration.  The various Layer 2, 3 and 4 flow control decision logic may be defined on the controller.

Once the flow rules are defined, simply push them to the applicable switch and the rule is in effect.

Zabbix Integration

This blog contains two previous articles pertinent to discovering and monitoring SNMP networking devices with Zabbix:
SNMP -- Simple Network Management Protocol for Linux Management Stations
Zabbix SNMP Low Level Router Discovery

As the illustration at the top of this posts indicates, there is a Zabbix Server and Zabbix Proxy in the topology.  They are configured to use SNMP Network Device templates (on the Zabbix Share) to automate device discovery:

1. Template_SNMP_Network_Device_Interfaces
2. Template_SNMP_Router_Cisco
3. Template_SNMP_Router_OSPF

The illustrations above depict the Discovered Hosts and a single interface graph on one device.

Linux Host Virtualization Networking

Optimizing and testing Linux Layer 2 and Layer 3 host networking tun/tap, bridge and routing options for host-to-VM and VM-to-VM connectivity for QEMU/KVM, VirtualBox and GNS3.  Includes performance tests for QEMU/KVM networking options that indicate 12.0 Gb/s to 18.8 Gb/sec sustained throughput for Linux host switching and routing.

Introduction

Linux distributions -- through the kernel and additional packages -- includes support for a variety of Layer 2 and Layer 3 networking features.  At Layer 2, it supports bridges, switching and VLANs.  At Layer 3, it supports IP routing and routing protocols such as RIP, OSPF, EIGRP, etc.  This article illustrates configuring various Linux host networking features to provide connectivity between GNS3 and QEMU/KVM networks.

The current Linux networking toolkit is iproute2 -- replacing older net-tools (ifconfig, ARP, etc.) and other (bridge-utils, tunctl, vlan, etc.) with one package.  However, some of the features in the older tools are still useful and will be installed.

The host laptop has quad-core Core i5 3230M, 2.60 GHz processors and 6 GB memory.  The QEMU/KVM VMs used for testing were assigned 4 cores each and 2 GB memory.

Linux Layer 2 Features

Layer 2 of the OSI model -- the Data Link Layer -- provides a variety of lower-level services.  Frames use Media Access Control (MAC) addresses to identify the source and destination addresses on a single Local Area Network.  MAC addresses are hard-coded into network interfaces (although they may be changed is the operating system supports that) and unique.  Destination addresses are discovered with broadcasts (and ARP) that flood the LAN; a single LAN is a broadcast domain.  Switching and bridging refers to transmitting frames from source to destination based upon MAC addresses.  Traditionally, a bridge has two interfaces and a switch more than two.  Thus, Layer two only operates on LANs and requires higher-level networking to transmit between LANs.

Linux Layer 2 Switching and Bridging

The current iproute2 toolkit may be used to create bridged interfaces.  However, there is additional functionality on the older bridge-utils toolkit that also integrates more easily with the Debian-style /etc/network/interfaces configuration file.

Linux tun and tap Devices

These are pseudo devices created in software and not physical ones.  At Layer 3, tun devices work at the packet level and typically used for tunnelling protocols.  For switching, tap devices operates at the Data Link layer and simulate a single interface.

You may use the older bridge-utils tunctl command or newer iproute2 command to create tap devices:

tunctl -t tap0

or

ip tuntap add tap0 mode tap

You may also create these devices at boot time using the Debian-style /etc/network/interfaces file:

auto tap0
iface tap0 inet manual

pre-up tunctl -t tap0

or

pre-up ip tuntap add tap0 mode tap

 The devices may also be configured with IP addresses by specifying "iface tap0 inet static" and the address, netmask and (optionally) gateway.

Linux Bridges and Switches

Bridges and switches -- multiport bridges -- are also available.  For the balance of this section, I will refer to both bridges and switches as bridges.  They are quite flexible in that you do not have to define specific ports as members or even a total number of logical ports in a bridge. The newer iproute2 toolkit supports creating bridges, however the older bridge-utils package is (in the author's opinion) easier to use and provides more granular control of bridge configuration.  From the brctl help command:

• addbr         <bridge>        add bridge
• delbr         <bridge>        delete bridge
• addif         <bridge> <device>    add interface to bridge
• delif         <bridge> <device>    delete interface from bridge
• hairpin       <bridge> <port> {on|off}    turn hairpin on/off
• setageing     <bridge> <time>        set ageing time
• setbridgeprio    <bridge> <prio>        set bridge priority
• setfd         <bridge> <time>        set bridge forward delay
• sethello      <bridge> <time>        set hello time
• setmaxage     <bridge> <time>        set max message age
• setpathcost    <bridge> <port> <cost>    set path cost
• setportprio    <bridge> <port> <prio>    set port priority
• show          [ <bridge> ]        show a list of bridges
• showmacs      <bridge>        show a list of mac addrs
• showstp       <bridge>        show bridge stp info
• stp           <bridge> {on|off}    turn stp on/off

For this article, we will focus on creating bridges at boot time in the /etc/network/interfaces file:

auto br0
iface br0 inet static
address 172.31.254.1
netmask 255.255.255.0
bridge_stp on
bridge_fd 0
bridge_ports tap0

These specify a bridge with an address and netmask (but no gateway) with a single defined interface -- tap0.  Spanning Tree Protocol -- to identify and prevent bridge loops -- is enabled and there is no forwarding delay when the bridge becomes active.  More interfaces may be added to the bridge using virtual networking -- such as GNS3 and QEMU/KVM, but more on that later.

It is not necessary to use tap interfaces with numbered, empty bridges.  Some virtualization technologies, such as VirtualBox, refer to this as Host-Only Networking because the VMs can communicate with each other and the host, but not outside networks.  Simply omit the bridge_ports definition:

auto br0
iface br0 inet static
address 172.31.254.1
netmask 255.255.255.0
bridge_stp on
bridge_fd 0

In the above case, tap0 is the operating system's logical interface connection to the bridge. It does not have to be numbered as the OS will recognize the bridge itself as having the assigned IP address. You are not limited to adding host tap interfaces to the bridge and physical Ethernet devices may be added as well. Wireless interfaces may also be added, but require additional configuration that will not be addressed in this article.

Layer 2 Host Networking Performance

bandwidth is an important consideration and there is a lot of misinformation about Linux tun and tap performance scattered around the Internet -- further proving the satirical adage "I read it on the Internet, it must be true!"  The author's limited searches suggested tap interfaces "theoretically" perform at up to 160 Mb/s, but in reality perform at under 10 Mb/s (less than a 20-year-old Ethernet NIC).

The following output is from a 60-second iperf test between the host (Ubuntu 14.04 Desktop) and a QEMU/KVM virtual machine (Debian Wheezy) over a tap/bridge connection:

~$ iperf -c 172.31.253.2 -t 60 -i 10
------------------------------------------------------------
Client connecting to 172.31.253.2, TCP port 5001
TCP window size:  136 KByte (default)
------------------------------------------------------------
[  3] local 172.31.253.1 port 50211 connected with 172.31.253.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  21.7 GBytes  18.7 Gbits/sec
[  3] 10.0-20.0 sec  22.5 GBytes  19.3 Gbits/sec
[  3] 20.0-30.0 sec  21.8 GBytes  18.7 Gbits/sec
[  3] 30.0-40.0 sec  21.3 GBytes  18.3 Gbits/sec
[  3] 40.0-50.0 sec  21.9 GBytes  18.8 Gbits/sec
[  3]  0.0-60.0 sec   131 GBytes  18.8 Gbits/sec

The following output is from a 60-second iperf test between the host (Ubuntu 14.04 Desktop) and a QEMU/KVM virtual machine (Debian Wheezy) over a bridge (no tap interface) connection:

~$ iperf -c 172.31.254.3 -t 60 -i 10
------------------------------------------------------------
Client connecting to 172.31.254.3, TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 172.31.254.1 port 34266 connected with 172.31.254.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  21.6 GBytes  18.5 Gbits/sec
[  3] 10.0-20.0 sec  20.9 GBytes  18.0 Gbits/sec
[  3] 20.0-30.0 sec  21.0 GBytes  18.1 Gbits/sec
[  3] 30.0-40.0 sec  21.6 GBytes  18.6 Gbits/sec
[  3] 40.0-50.0 sec  21.1 GBytes  18.2 Gbits/sec
[  3] 50.0-60.0 sec  22.0 GBytes  18.9 Gbits/sec
[  3]  0.0-60.0 sec   128 GBytes  18.4 Gbits/sec

The following output is from a 60-second iperf test between the two QEMU/KVM virtual machines (Debian Wheezy) over a tap/bridge connection:

~# iperf -c 172.31.253.2 -t 60 -i 10
------------------------------------------------------------
Client connecting to 172.31.253.2, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[  3] local 172.31.253.3 port 33823 connected with 172.31.253.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  14.1 GBytes  12.1 Gbits/sec
[  3] 10.0-20.0 sec  14.3 GBytes  12.3 Gbits/sec
[  3] 20.0-30.0 sec  14.5 GBytes  12.4 Gbits/sec
[  3] 30.0-40.0 sec  14.3 GBytes  12.3 Gbits/sec
[  3] 40.0-50.0 sec  14.2 GBytes  12.2 Gbits/sec
[  3]  0.0-60.0 sec  84.1 GBytes  12.0 Gbits/sec

The following output is from a 60-second iperf test between the two QEMU/KVM virtual machines (Debian Wheezy) over a bridge (no tap interface) connection:

~# iperf -c 172.31.254.2 -t 60 -i 10
------------------------------------------------------------
Client connecting to 172.31.254.2, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[  3] local 172.31.254.3 port 59334 connected with 172.31.254.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  16.0 GBytes  13.7 Gbits/sec
[  3] 10.0-20.0 sec  16.1 GBytes  13.8 Gbits/sec
[  3] 20.0-30.0 sec  16.1 GBytes  13.8 Gbits/sec
[  3] 30.0-40.0 sec  15.8 GBytes  13.5 Gbits/sec
[  3] 40.0-50.0 sec  15.4 GBytes  13.2 Gbits/sec
[  3]  0.0-60.0 sec  95.0 GBytes  13.6 Gbits/sec

Properly configured, Layer 2 networking performance between Linux hosts and VMs is excellent.  Bandwidth between VMs is less than between host and VM; a 13.3% performance improvement was realized by using bridges without a host tap interface.

Adding Virtual Machine Interfaces to Linux Layer 2 Devices

QEMU/KVM

Using the Qemu/KVM Virtual Machine Manager is the easiest way to connect to host machine networks.  Virtual Machine Manager automatically recognizes configured host tap/bridge devices and offers to bridge its own virtual NICs to a tap interface.

The choice of NIC models is also important and the paravirtualized Virtio device offers better performance than fully virtualized devices such as Intel e1000 NICs.

 The optimal QEMU/KVM configuration -- an empty bridge (i.e. a host bridge qithout an attached tap interface) is illustrated above.

VirtualBox

The selections for VirtualBox guests are much the same as for QEMU/KVM guests.  Under Network, select "Bridged Adapter" and "tap0" (or another tap interface if so desired).  Open the Advanced settings and select the Paravirtualized Network (virtio-net).

GNS3

GNS3 connections to the host are more detailed and have been described in another post.

Linux Layer 3 Routing

The Linux kernel supports static Layer 3 IP routing.  However, virtualized environments (particularly using GNS3) are better served by routing protocols.  This article describes how to implement a relatively portable OSPF configuration that, with minimal configuration, connects virtual machines and networks to the Internet.

Quagga Routing Protocols

Quagga is a fork of the inactive Zebra project.  For the subject host, it will implement OSPF to route between the host, Internet and virtual networks.  A previous post discusses the topic in detail.

For this implementation, the backbone area (range 172.16.0.0/12) will be the host Linux laptop itself.  The wireless network's DHCP network and tap/bridge interfaces will be areas attached to the backbone.

Layer 3 VM Networking Performance

Layer 3 routing between networks typically involves additional overhead and resulting lower speeds than Layer 2 switching.  Under Linux host networking using two different numbered bridges (without tap interfaces), there appears to be a small degree of such reduced performance, albeit quite small.

~# iperf -c 172.31.254.2 -t 60 -i 10
------------------------------------------------------------
Client connecting to 172.31.254.2, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[  3] local 172.31.253.3 port 57056 connected with 172.31.254.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  15.3 GBytes  13.2 Gbits/sec
[  3] 10.0-20.0 sec  15.2 GBytes  13.1 Gbits/sec
[  3] 20.0-30.0 sec  15.4 GBytes  13.2 Gbits/sec
[  3] 30.0-40.0 sec  15.3 GBytes  13.1 Gbits/sec
[  3] 40.0-50.0 sec  14.6 GBytes  12.6 Gbits/sec
[  3] 50.0-60.0 sec  14.4 GBytes  12.4 Gbits/sec
[  3]  0.0-60.0 sec  90.2 GBytes  12.9 Gbits/sec

 

Host Wireless Adapter

The author uses many wireless networks and they are (fortunately) all configured to assign DHCP addresses in the 192.168.x.x range, although on a variety of different 24-bit CIDR masks.  They also assign a static default gateway.  While it is possible to add a wireless adapter to a bridge, it is easier to configure OSPF to treat the wireless adapter as a non-backbone area.  The following Quagga commands (note the slightly different from Cisco syntax) cover the range of 162.168.x.x addresses to operate correctly under OSPF:

network 192.168.0.0/24 area 192.168.0.0
network 192.168.1.0/24 area 192.168.0.0
...
area 192.168.0.0 range 192.168.0.0/16
default-information originate always 

Please note that a separate network definition must be applied for each different subnet the wireless LAN interface encounters.  This requires adding only one command to the Quagga ospfd.conf file each time a new network is encountered.

Since wireless routers assign a static route to the Linux laptop host, it acts as the default gateway originator (OSPF command "default gateway originate always").  This configuration passes the static default gateway from the backbone to connected OSPF areas -- the virtual networks.

Host tap/bridge Interfaces

The numbered host tap/bridge interfaces may be added to the Quagga routing protocols.  For OSPF, assure they are numbered in the Area 0.0.0.0 Backbone range and add their networks accordingly.  These interfaces will then update the host routing tables and any other connected areas.

Guest Interfaces

Three scenarios are depicted in the illustration above:

1. VMs individually connected to a bridge
2. QEMU/KVM or VirtualBox Linux router gateway
3. GNS3 Cisco router gateway

VMs individually connected to a bridge

This is a basic scenario.  The hosts simply need to connect to the tap or bridge interface, have an address in the same subnet and the gateway defined as the host tap or bridge IP address.

QEMU/KVM or VirtualBox Linux router gateway

A VM acting as a gateway to other VMs behind it requires a routing protocol.  One interface must be connected to the host tap or bridge interface and any others connected to one or more bridges for the other hosts.  These additional bridges may be host bridges (as described in this article) or ones defined in the Virtual Networking software used (e.g. QEMU/KVM, VirtualBox, etc.).  Performance of the additional bridges will be the topic of another post.

GNS3 Cisco router gateway

GNS3 on an Ubuntu host is a bit more complicated affair.  Connecting to host networking requires root access (as depicted in the illustrations below), and this has been problematical for the author.

The interfaces are recognized and GNS3 hosts may connect to other VM hosts on the bridge, but not to the host itself.  As described in another article, under Ubuntu the author uses Cloud devices to connect to the host.
GNS3 also is a software emulator and when emulating Cisco IOS hardware-based routers, performance suffers.  Although this test was conducted using an older version of GNS3 (0.86), the author's anecdotal observations of performance of the current (at the time of writing, 1.2.3) indicates low bandwidth routing performance.

Multi Site Open Monitoring Distribution Configuration

Configuring distributed monitoring using the Open Monitoring Distribution's Nagios - Check_MK - Livestatus - Multisite features.

Introduction

The Open Monitoring Distribution (OMD) provides an easy to install and configure Nagios system with Check_MK.  Check_MK uses the Livestatus broker module to efficiently process host and service checks.  Installation is described in this article.

Installing and configuring distributed Nagios Core monitoring -- using the Nagios Service Check Acceptor (NSCA) or mod_gearman is complicated and time-consuming.  OMD, however, includes menu-driven and web-interface configuration of Check_MK's Multisite feature -- distributed monitoring and centralized configuration.  This article explains how to configure Multisite distributed monitoring.

Configuring the Master Server

The Master Server is the one upon which all configurations are performed.  The Slave servers allow configuration, but the changes will be overwritten when the Master Server updates.

The Master Server must be configured as a local connection.  It requires a URL prefix of the host name, localhost or a valid IP address.  In this example, IP addresses are used and 127.0.0.1 selected as the URL prefix http://127.0.0.1/COU -- the site name is the directory.

Configuring Slave Servers

Enabling Livestatus

Slave servers require some additional work.  First, you must enable Livestatus from the server's menu configuration utility.  Stop the site with the command: "omd stop <site_name>."  The, start the configuration utility with the command "omd config <site_name."  Select "Distributed Monitoring."  Then, enable Livestatus as depicted below.

This creates and configures the file /opt/omd/sites/<site_name>/etc/xinetd.d/check_mk.  This xinetd service listens on TCP port 6557 and the file defines all parameters necessary for Master Server connections.

Creating Master to Slave Replication

Several items must be defined to enable replication:

1. The Livestatus IP address and TCP port 6557
2. The URL prefix http://<host_name or IP_address>/<site_name>
3. Slave replication
4. The Multisite URL http://<host_name or IP_address>/<site_name>/check_mk

Once configured and live, login to each Slave Server from the Master to enable replication.

Creating Hosts

Creating hosts is almost identical to creating them on single-server installations.  There is an additional -- and important -- field: the "Monitored on site" definition.  This provides a drop-down list of all configured Master and Slave Servers.

Organization is also important.  The author uses folders for each Master and Slave Server, each predefining the corresponding "Monitored on site" value.

The video below depicts the process of configuring OMD Distributed Monitoring.

Integrating Open Monitoring Distribution (Nagios-Check-MK) with GNS3

This article demonstrates how to configure a Debian VirtualBox Open Monitoring Distribution using Nagios and Check-MK into an existing GNS3 network of Cisco Routers and Linux Layer 2 Switches.

Introduction

Nagios -- and its fork Icinga -- is one of the most common systems monitoring tools available.  Among its strengths are notifications and topological relationships.  Properly configured, it does not send out large numbers of (meaningless) notifications for an entire subnetwork because the router/switch between it and the monitoring host fails.

There is a great deal to learn about Nagios Core (that ships with many Linux distributions.  A few of the author's own articles provide additional details on how to configure it.

Nagios Basics for Infrastructure

The single most important concept to understand for monitoring switches and routers with Nagios is the Parent relationship.  Nagios provides a field in the host definition that specifies a host between the monitoring server and host.  In the image above, the Monitoring Server is a parent to the COU_Switch.  The COU_Switch -- in turn -- is a parent to both the COU_7206 and COU_3745 Routers.  This is a simple topology and redundant links are a more complicated issue beyond the scope of this article.  Properly defined, when a parent host fails, the child hosts register as Unreachable rather than Down.  When registered as Unreachable, Nagios does not send out alerts.  Thus, if a router fails, it registers as Down and an alert is sent; the child hosts register as Unreachable and no notifications are sent.  This provides a mch simplified set of data to determine the root causes of problems.

The second important concept to understand about Nagios for monitoring switches and routers is the Simple Network Management Protocol (SNMP).  SNMP is an IETF standard for collecting and reporting host information.  Networking equipment generally supports SNMP Version 2c or higher.  In its simplest configuration, SNMP is configured to listen on one or more interfaces and requires a community string (the general default is "public") and will report information to a monitoring host.  See the link at the beginning of this paragraph for more information.

Nagios with Check-MK

Check-MK is an addition to Nagios Core.  Perhaps that is an understatement.  Check-MK provides host agents that automatically poll and return information to the monitoring server.  Livestatus is a broker module that interprets returned host agent data and translates it into Nagios for processing.  Multisite and WATO are web-based configuration and reporting tools.

In short, Check-MK is a tool that greatly simplifies host check configurations.

Installing Nagios with Check-MK on Debian

You may install Check-MK from Debian repository packages.  However, a much simpler method is to download the Open Monitoring Distribution (OMD).  This is a package built for a variety of major distributions.  The following steps are for Debian Wheezy using the current package at the time of writing:

• Download the 102 Megabyte OMD package:  wget http://files.omdistro.org/releases/debian_ubuntu/omd-1.20.wheezy.amd64.deb
• Run the command dpkg -i omd* -- this will fail because dependencies are not met but Debian will correct that issue.
• Run the command: apt-get -f install to add the missing dependencies; you will need to configure MySQL with a root user password.
• Run the command dpkg -i omd*

Once this is complete, OMD -- with Nagios, Check-MK and a variety of other utilities is installed.   OMD installs into the /opt/omd directory.  Under that are apache, sites and versions.  The primary work area is /opt/omd/sites, where the files configuring each monitoring site are maintained.

The agent package is also available and this may be installed from the Debian repositories using apt.  The agent runs under xinetd, and the configuration file is /etc/xinetd/check_mk.  The last line must be changed from "disable = yes" to "disable = no" at a minimum.  You may also restrict access to specified IP addresses.  Once changed, simply restart xinetd.

Creating a monitoring site is easy.  Issue the command "omd create <site_name>."  For this article, we shall use the site name COU.  This command:

• creates a directory structure /opt/omd/COU under which configuration files are maintained.
• a new user and group COU; the user is also a member of the group omd.

You may modify configuration files (e.g. /etc/apache/*, /etc/nagios/*) as necessary, but the distribution is ready to run and configure from the web interface.  Start the site with the command "omd start <site_name>."

Configuring OMD using the Check-MK Web Administration Tool (WATO)

The new site is available under the web server's /COU directory.  The default logon is user name "omdadmin" and password "omd."

The initial page is an overview of different web-based tools such as Nagios, Icinga, Thruk and NagVis.

Check-MK is another web interface that may be used with Nagios, but in this case it is also integrated with the Check-MK suite and includes tools such as WATO for configuration, the Livestatus Nagios event broker and a very efficient Check Engine that submits results to Nagios as passive checks, saving a great deal of processing -- an effective alternative to NSCA.

WATO, the administration tool, provides a well-organized interface.  The illustration above depicts the Hosts option, with the monitoring server and two Folders for Cisco and Linux devices added.  The add hosts and folders icons link to the screens for these tasks.

The Cisco Folder contains eight routers with icons and summaries provided for administrative tasks such as modifying and cloning hosts.

The edit and add hosts screens are very similar.  Check-MK is agent or SNMP-based, so there is very little information required to define a host -- the application, agents and SNMP manage discovery.

The discovery process is fairly thorough.  The illustration above depicts SNMP items discovered on a Cisco 3745 Router.  Discovery finds not just interfaces, but also fans, power supplies, CPU, memory and other counters.

By default, the SNMP Discovery process only searches for Ethernet and Frame Relay devices on all hosts.  If you have SONET or other interfaces, you must activate them from the Global Configuration screen.  That process is depicted in the video at the bottom of this article.

Parent relationship definitions are included for each host.  Parent relationships define topology -- as depicted above -- and also control Notifications.  A more thorough treatment discussion of Nagios/Icinga Logic, Alerts and Notifications is available in another blog article.  The illustration above depicts a NagVis-generated topology available from the Check-MK views.

Finally (at least as far as this article is concerned), OMD includes a preconfigured PNP4Nagios installation.  As soon as a host is added the application processes service check results and begins graphing them.  Although not depicted, Check-MK includes its own Perf-O-Meter views that are small and effective overviews of counters.
The video below depicts OMD installation on Debian Wheezy, a rather simple process described above.

The video below depicts configuring OMD for monitoring eight Cisco routers and two Linux Layer 2 Switches using the Check-MK interface.

GNS3 - VirtualBox Part 11: Debian/Ubuntu Bonded NIC Layer 2 Switches

This article demonstrates how to configure Debian/Ubuntu VirtualBox guests to operate as Layer 2 switches with bonded NICs, aggregating several adapters into higher-speed logical interfaces.

Introduction

There have been many implementations of adapter bonding -- often vendor-specific and proprietary; these implementations are not germane to this article.  Over time, published standards have replaced proprietary ones.  Linux supports seven different bonding types and a Linux Bonding HOW-TO document is available at kernel.org.

This article describes Mode 4, IEEE 802.3ad Dynamic link aggregation.  This is a common implementation and requires switch support.  It is widely supported by vendors.

Bonding Modes

Specifies one of the bonding policies. The default is balance-rr (round robin). Possible values are: 

Balance-rr or 0

Round-robin policy: Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.

Active-Backup or 1

Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch. This mode provides fault tolerance.

Balance-XOR or 2

XOR policy: Transmit based on the selected transmit hash policy. The default policy is a simple [(source MAC address XOR'd with destination MAC address XOR packet type ID) modulo slave count]. Alternate transmit policies may be selected via the xmit_hash_policy option. This mode provides load balancing and fault tolerance. 

Broadcast or 3

Broadcast policy: transmits everything on all slave interfaces. This mode provides fault tolerance.

802.3ad or 4

IEEE 802.3ad Dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings. Utilizes all slaves in the active aggregator according to the 802.3ad specification. Slave selection for outgoing traffic is done according to the transmit hash policy, which may be changed from the default simple XOR policy via the xmit_hash_policy option. Most switches will require some type of configuration to enable 802.3ad mode. This mode provides load balancing and fault tolerance.

Balance-TLB or 5

Adaptive transmit load balancing: channel bonding that does not require any special switch support. In tlb_dynamic_lb=1 mode; the outgoing traffic is distributed according to the current load (computed relative to the speed) on each slave. In tlb_dynamic_lb=0 mode; the load balancing based on current load is disabled and the load is distributed only using the hash distribution. Incoming traffic is received by the current slave. If the receiving slave fails, another slave takes over the MAC address of the failed receiving slave.

Balance-ALB or 6

Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support. The receive load is distributed sequentially (round robin) among the group of highest speed slaves in the bond. When a link is reconnected or a new slave joins the bond the receive traffic is redistributed among all active slaves in the bond.

Installing ifenslave and Modifying Configuration Files

The first step is to install the ifenslave package (apt-get install ifenslave).  The package includes the commands and kernel module support.  Boot time kernel module loading requires adding a line -- bonding -- to /etc/modules thus:

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.

lp
rtc
bonding

A new file -- etc/modprobe.d/modules.conf -- is required for additional configuration.  While many configuration commands may be later added to the /etc/network/interfaces file, they may also be added to this one.

alias bond0 bonding
options bonding mode=4 miimon=100 downdelay=200 updelay=200 max_bonds=2
alias bond1 bonding
options bonding mode=4 miimon=100 downdelay=200 updelay=200 max_bonds=2

The illustrated options are not mandatory, but advisable.  The miimon option specifies the interval (milliseconds) in which MII Link Monitoring occurs.  The default value is 0 and disables MII Link Monitoring.  The downdelay and updelay options specify delay times (milliseconds) between MII Link Monitoring detection of a state change and application of the change; it is a multiple of the miimon value and will be automatically rounded if otherwise defined.  Another important option is "maxbonds=#."  The default for this value is 1 and it allows (but does not automatically create) a bond0 interface.  If you plan to add more than one bonded interface, you will need to specify "maxbond=#" as a larger value.
Additional configuration options are detailed in the kernel.org Linux Bonding HOW-TO document.

Configuring from the Command Line Interface

Bonded interfaces may be configured from the command line or added to the /etc/rc.local file.  The following commands utilize iproute2, ifenslave and the deprecated bridge-utils packages:

ifenslave bond0 eth0 eth1
ifenslave bond1 eth2 eth3 eth4
ip link set bond0 up
ip link set bond1 up
ip link add dev br0 type bridge
ip link set dev bond0 master br0
ip link set dev bond1 master br0
ip addr add 10.64.0.4/255.255.255.0 dev br0ip route add default via 10.64.0.1ip link set dev br0 upbrctl stp br0 on

It is not necessary to set the individual Ethernet adapters to "up" when using bonded interfaces.  However, if unbonded Ethernet adapters are to be used in the bridge, they must be set to "up" thus:

ip link set dev eth5 up

Configuring /etc/network/interfaces

Alternatively, bonded Ethernet interfaces may also be configured in the /etc/network/interfaces file, providing support for the ifupdown package for high-level configuration.   The following configuration creates a bridge (br0) with a static IP address, two bonds (bond0 = eth0 and eth1; bond1 = eth2, eth3 and eth4) and three Ethernet interfaces (eth5, eth6 and eth7) that are members of bridge br0.

auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
pre-up ifenslave bond0 eth0 eth1
post-up ip link set dev bond0 master br0
pre-down ip link set dev bond0 nomaster
post-down ifenslave -d bond0 eth0 eth1

auto bond1
iface bond1 inet manual
pre-up ifenslave bond1 eth2 eth3 eth4
post-up ip link set dev bond1 master br0
pre-down ip link set dev bond1 nomaster
post-down ifenslave -d bond1 eth2 eth3 eth4 
iface eth5 inet manual


iface eth6 inet manual

iface eth6 inet manual


auto br0
iface br0 inet static
address 10.64.0.4
netmask 255.255.255.0
gateway 10.64.0.1
dns_nameservers 192.168.1.1 8.8.8.8 4.4.4.4
bridge_stp on
bridge_waitport 0
bridge_fd 0
bridge_ports bond0 bond1 eth5 eth6 eth7

Testing the Switches

The Legacy ifconfig Command

This command is of limited utility for bonded interfaces.  Note that it lists the bond and Ethernet interfaces as up and whether they are masters (bonds) or slaves (Ethernet).  It does not specify details of master-slave relationships.

bond0     Link encap:Ethernet  HWaddr 08:00:27:1c:e8:ec 
          inet6 addr: fe80::a00:27ff:fe1c:e8ec/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:420 errors:0 dropped:3 overruns:0 frame:0
          TX packets:576 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:46060 (46.0 KB)  TX bytes:52141 (52.1 KB)

bond1     Link encap:Ethernet  HWaddr 08:00:27:83:45:d7 
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:72 errors:0 dropped:2 overruns:0 frame:0
          TX packets:473 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:8736 (8.7 KB)  TX bytes:38259 (38.2 KB)

br0       Link encap:Ethernet  HWaddr 08:00:27:1c:e8:ec 
          inet addr:10.64.0.4  Bcast:10.64.0.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe1c:e8ec/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:367 errors:0 dropped:1 overruns:0 frame:0
          TX packets:212 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:34606 (34.6 KB)  TX bytes:27137 (27.1 KB)

eth0      Link encap:Ethernet  HWaddr 08:00:27:1c:e8:ec 
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:88 errors:0 dropped:0 overruns:0 frame:0
          TX packets:32 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:8736 (8.7 KB)  TX bytes:3604 (3.6 KB)

...

eth4      Link encap:Ethernet  HWaddr 08:00:27:83:45:d7 
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:26 errors:0 dropped:2 overruns:0 frame:0
          TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3032 (3.0 KB)  TX bytes:6982 (6.9 KB)

The ip addr Command

This command -- part of the newer iproute2 package -- provides detailed information about interfaces states and master-slave relationships.  For instance, Ethernet interfaces eth0, eth1 and eth2 enumerate bond0 as their masters and bond0 is enumerates br0 as its master.

2: eth0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 08:00:27:1c:e8:ec brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP group default qlen 1000
    link/ether 08:00:27:1c:e8:ec brd ff:ff:ff:ff:ff:ff
...
9: eth7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 08:00:27:81:dd:5a brd ff:ff:ff:ff:ff:ff
10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default
    link/ether 08:00:27:1c:e8:ec brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a00:27ff:fe1c:e8ec/64 scope link
       valid_lft forever preferred_lft forever
11: bond1: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default
    link/ether 08:00:27:83:45:d7 brd ff:ff:ff:ff:ff:ff
12: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 08:00:27:1c:e8:ec brd ff:ff:ff:ff:ff:ff
    inet 10.64.0.4/24 brd 10.64.0.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe1c:e8ec/64 scope link
       valid_lft forever preferred_lft forever

Listing /proc/net/bonding Files

The /proc/net/bonding file contains bond-specific information, this time for a different switch using bonded Ethernet adapters eth2, eth3 and eth4.  Notice that the bond Transmit Hash Policy is the default Layer 2.  Also notice that the default LACP rate -- slow -- applies.  LACP is the protocol that negotiates bundling between complaint switches.  The default interval is slow (30 seconds) while fast (1 second) must be manually-specified.

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200

802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
    Aggregator ID: 1
    Number of ports: 3
    Actor Key: 17
    Partner Key: 17
    Partner Mac Address: 08:00:27:74:2e:0d

Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:83:45:d7
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:1d:37:ae
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 08:00:27:7e:25:d0
Aggregator ID: 1
Slave queue ID: 0

The Legacy brctl Commands

The legacy bridge-utils package provides three commands -- brctl show, brctl showmacs and brctl showstp -- that provide an overview of basic bridge configuration and operation.

brctl show <bridge_name>

brctl show br0
bridge name    bridge id        STP enabled    interfaces
br0        8000.0800271ce8ec    yes              bond0
                                                                 bond1

brctl showmacs <bridge_name>

brctl showmacs br0
port no    mac addr        is local?    ageing timer
  1    08:00:27:1c:e8:ec    yes           0.00
  2    08:00:27:83:45:d7    yes           0.00
  1    ca:02:10:48:00:06    no           0.00

brctl showstp <bridge_name>

brctl showstp br0
br0
 bridge id        8000.0800271ce8ec
 designated root    8000.0800271ce8ec
 root port           0                                 path cost           0
 max age          20.00                           bridge max age          20.00
 hello time           2.00                          bridge hello time       2.00
 forward delay           2.00                     bridge forward delay       2.00
 ageing time         300.00
 hello timer           0.10                         tcn timer           0.00
 topology change timer       0.00            gc timer         141.52
 flags           


bond0 (1)
 port id        8001                                      state             forwarding
 designated root    8000.0800271ce8ec       path cost           4
 designated bridge    8000.0800271ce8ec    message age timer       0.00
 designated port    8001                             forward delay timer       0.00
 designated cost       0                               hold timer           0.00
 flags           

bond1 (2)
 port id        8002                                      state             forwarding
 designated root    8000.0800271ce8ec       path cost         100
 designated bridge    8000.0800271ce8ec    message age timer       0.00
 designated port    8002                             forward delay timer       0.00
 designated cost       0                               hold timer           0.00
 flags           

GNS3 Configuration

The GNS3 configuration depicted at the top of this article depicts three Linux switches -- PHL-Core, PHL-Servers and PHL-Storage.  PHL-Core is connected to two routers over bridged Ethernet ports eth0 and eth1; it is connected to PHL-Servers over the bridged two-adapter bond0 interface (eth2 and eth3 to eth0 and eth1, respectively).  PHL-Servers is connected to bridged three-adapter bond1 interface (eth2, eth3 and eth4 to eth0, eth1 and eth2, respectively).  The /etc/network/interfaces files for the three configurations are below.

PHL-Core Configuration

auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
pre-up ifenslave bond0 eth2 eth3
post-up ip link set dev bond0 master br0
pre-down ip link set dev bond0 nomaster
post-down ifenslave -d bond0 eth2 eth3

auto br0
iface br0 inet static
address 10.64.0.2
netmask 255.255.255.0
gateway 10.64.0.1
dns-nameservers 192.168.1.1 8.8.8.8 4.4.4.4
bridge_stp on
bridge_waitport 0
bridge_fd 0
bridge_ports eth0 eth1 bond0

iface eth0 inet manual

iface eth1 inet manual

PHL-Servers Configuration

auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
pre-up ifenslave bond0 eth0 eth1
post-up ip link set dev bond0 master br0
pre-down ip link set dev bond0 nomaster
post-down ifenslave -d bond0 eth0 eth1

auto bond1
iface bond1 inet manual
pre-up ifenslave bond1 eth2 eth3 eth4
post-up ip link set dev bond1 master br0
pre-down ip link set dev bond1 nomaster
post-down ifenslave -d bond1 eth2 eth3 eth4

auto br0
iface br0 inet static
address 10.64.0.4
netmask 255.255.255.0
gateway 10.64.0.1
dns-nameservers 192.168.1.1 8.8.8.8 4.4.4.4
bridge_stp on
bridge_waitport 0
bridge_fd 0
bridge_ports bond0 bond1

PHL-Storage Configuration

auto lo
iface lo inet loopback

auto bond0
iface bond0 inet manual
pre-up ifenslave bond0 eth0 eth1 eth2
post-up ip link set dev bond0 master br0
pre-down ip link set dev bond0 nomaster
post-down ifenslave -d bond0 eth0 eth1

auto br0
iface br0 inet static
address 10.64.0.5
netmask 255.255.255.0
gateway 10.64.0.1
dns-nameservers 192.168.1.1 8.8.8.8 4.4.4.4
bridge_stp on
bridge_waitport 0
bridge_fd 0
bridge_ports bond0