Zabbix Template for Squid Proxy Server SNMP

This article describes how to configure SNMP on Squid Proxy Servers and a Zabbix Template to monitor data.

Introduction

The Squid Proxy Server does not use the Linux typical Net-SNMP when installed.  Instead, it uses a separate binary that runs as its own process.  The binary is installed by default with Squid 3.x and later, but must be enabled when compiled into Squid 2.x and earlier and the SNMP binary compiled after. Once the SNMP binaries are available, they may be configured in the squid.conf file.  The separate SNMP procces may then be queried by SNMP monitoring tools such as Zabbix.

Configuring the Squid SNMP Process

The Squid Proxy web site provides a detailed HOW-TO page.  Assigning ACLs and listener IP addresses for security is strongly recommended.  For instance, create a host ACL for the Zabbix Server's IP address and limit access to that ACL and set the listener IP address for a single interface. For this article, we will not use a secure configuration because it is for demonstration purposes.  Familiarity with Squid's ACLs is assumed for a live deployment.

Minimum squid.conf Configuration

A minimal Squid SNMP configuration requires the following lines in the squid.conf file:

snmp_port 3401
acl <acl_name> snmp_community <community_name>
snmp_access allow all

 These lines create the SNMP process listener port (3401 is a widely-referenced port) and listens on all interfaces (default), creates an SNMP_Community-type ACL and assigns a community name (public is a de facto, but inssecure, standard) and allows access over port 3401 to all hosts.

Squid Proxy Zabbix Template

The Zabbix Share web site hosts the Squid Proxy Server template.  There are several customizations to query Squid Proxies.

Two macros are included in the template:

{$SQUID_SNMP_COMMUNITY} > public
{$SQUID_SNMP_PORT} > 3401

 These reflect the settings applied in the squid.conf file above.  If the global configurations of Squid Proxies in the enterprise differ, adjust them accordingly.

Each host must also be configured with an SNMP interface (Hosts > Interfaces > SNMP) that specifies the listening IP address / UDP port configured in the squid.conf file.  This must also be the primary listening SNMP interface.

Example Squid Proxy Server Cache Monitoring

The following architecture is a test environment.

There are two Linux host bridges (br0 and br1) on different subnets.  On br0, there is a Zabbix Server, a parent Squid Server (Squid01) and its child Squid Server (Squid02).  On br1, there is another Squid01 child Squid Server, Squid03.  The two child Squid Servers are configured as siblings.  The Squid Servers begin with empty Disk Caches.

Squid01 -- the parent cache -- is configured to only disk cache files larger than 4096 bytes and smaller than 128 MB.  Squid02 and Squid03 -- the sibling child caches -- have the default configuration of only caching files less than 4MB.  There is an overlap of the caching size policies, but in general, large files will only be cached on the parent.   Smaller files will be cached on the children.

The Internet connection (through which the Debian repositories are accessed during installation) is limited to 10 Mb/s (1.25MB/sec).

The Zabbix Server utilizes the Squid Proxy SNMP template.  The custom screens below organize the three Squid Proxy Servers in columns.  The three respective rows present graphs for HTTP Byte Hit Ratio (%), HTTP Rate (KB/s) and HTTP Request Hit Ratio.




The test consists of performing a Debian 8 (Jessie) desktop network installation (packages retrieved from repositories) with the Debian hosts configured to use different Squid Proxies.  First, we perform the installation with empty caches; also, server Squid03 is off line because we only want to cache on Squid02 and the sibling configuration will distribute the load.  As illustrated below, the installation requires approximately 20 minutes to download required packages from the repositories.  Notice the HTTP Rate (KB/sec) is limited to approximately 1.25 MB/sec (1.25KKB/sec or the Internet connection bandwidth of 10 Mb/sec).  Parent Proxy Squid01 (in the left column) returns generally low cache hit ratios and Child Proxy Squid02 returns higher hit ratios because it is passing requests to the parent.

Unpopulated Parent-Child Proxy Caches

The second test installation mimics the first, but is now accessing Proxy Caches populated with files from the first test.  Not only is the installation much faster (approximately five minutes compared to 20 in the first test), but the HTTP Rate is much higher -- 9 to 10 MB/sec (72 - 80 Mb/sec).  Internet bandwidth for this test is not an issue because the files are all cached locally.  Bandwidth is limited by the system Linux Bridge and Virtio NIC and Disk performance.
 

Populated Parent-Child Proxy Caches

The third test installation moves the Debian machine to Linux KVM host bridge br0 and uses Squid03 as its proxy server.  Here we see limitations of disk IO and peer caching because Sibling Proxies Squid02 and Squid03 are synchronizing and both accessing parent cache Squid01.  While the installation is faster than fetching all the files from the Internet-hosted repository, the disk contention and peer cache synchronization slow the total time of installation to approximately 10 minutes (compared to 20 for fetching all files from the Internet and five minutes for the populated two-cache proxy arrangement).

As above with an Unpopulated Child-Sibling Proxy Cache

The fourth test installation mimics the third but is now accessing Proxy Caches populated with files from the first three tests and the Sibling Proxies are also synchronized.  The Parent Proxy and Squid03 Child Proxy are both actively providing files for the installation which completes in approximately five minutes.  HTTP Rate bandwidth is also comparable to the tuned proxy performance in Test 2 -- 7MB/sec - 11 MB/sec.   Squid02 -- the other Sibling Proxy depicted in the center column -- displays very little activity bceuase its disk cache contents are identical to Squid03 and no files are needed.

Populated Parent - Sibling Children Proxy Caches

The above tests are not representative of an enterprise proxy server deployment.  The caches are either unpopulated or fully populated with requested data, whereas in live deployments typically approximately 5% of requests are cached.  There are performance limitations (particularly disk IO) in the Linux KVM test environment.  The all-or-nothing caching depicted above does, however, demonstrate the increased HTTP and Cache Hit Rates associated with proxy caching.

The Zabbix Share web site hosts the Squid Proxy Server template.

Linux - Microsoft Active Directory Integration -- Seamlessly-Authenticated Squid Proxy Server

Introduction

Squid is an open source proxy server that may be configured to authenticate to Active Directory.  Microsoft has a variety of Proxy Server offerings such as ISA Server and Threat Management Gateway.  These are fine products that are well-integrated with Active Directory.  Out-of-the box, Squid is not well-integrated, but with a little added software it works very well with Active Directory.

One problem with Squid is it is not well-integrated with Active Directory.  True, you may apply Group Policy settings to configure Microsoft clients to use a Squid Proxy Server, but there are limitations.  One important limitation from a liability standpoint is the ability to keep track of which user is visiting which sites.  Under Active Directory Group Policy client configuration, Squid will record host names of client computers, but not the actual user logged in.  This is a serious drawback if called upon to enforce company Acceptable Use Policies because the actual user is not logged.

Achieving user-specific logging requires an authentication mechanism.  Squid offers a manually-configured authentication option, but this represents an additional administrative burden.  Installing Squid to seamlessly authenticate to Active Directory provides such logging capabilities.  Additionally, security groups may also be incorporated into Access Control Lists.

Proxy servers are primarily security devices; that they provide a disk and memory cache offers some performance advantages, but not to the extent many administrators think.

Source and Copyright

This article draws heavily upon the original here.  The original was published under the Creative Commons Attribution-ShareAlike 3.0 Unported License, and this derivative is also so licensed.

Test Environment

Network
Domain= mydomain.com
Proxy Server Subnet 10.64.0.0/24
Proxy Server
IP = 10.64.0.2
HOSTNAME = cou-firewall.mydomain.com
Kerberos computer name = COU-FIREWALL-K
Windows Server Subnet 10.128.0.0/24

Windows Server 1
IP = 10.128.0.2
HOSTNAME = exchange01.mydomain.com
Windows Server 2
IP = 10.128.0.3
HOSTNAME = exchange02.mydomain.com

Install Kerberos Packages

apt-get install krb5-user libkrb5-3

Note: Just accept the debconf dialog defaults because we are overwriting them.

Setup Kerberos

Back up the kerberos configuration file and overwrite the original with information specific to this installation.

#cp /etc/krb5.conf /etc/krb5.conf.default
#cat /dev/null > /etc/krb5.conf
#nano /etc/krb5.conf

Edit the file replacing the variables with the client's domain and server.
Note: If you only have 1 Domain Controller remove the additional 'kdc' entry from the '[realms]' section, or add any additional Domain Controlers if you have more than two domain controllers.

[libdefaults]
default_realm = MYDOMAIN.COM
dns_lookup_kdc = no
dns_lookup_realm = no
ticket_lifetime = 24h
default_keytab_name = /etc/squid3/PROXY.keytab
; for Windows 2008 with AES
;
default_tgs_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5
;
default_tkt_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5
;
permitted_enctypes = aes256-cts-hmac-sha1-96 rc4-hmac des-cbc-crc des-cbc-md5
;
[realms]
MYDOMAIN.COM = {
kdc = exchange01.mydomain.com
kdc = exchange02.mydomain.com
admin_server = exchange01.mydomain.com
default_domain = mydomain.com
}
;
[domain_realm]
.mydomain.com = .MYDOMAIN.COM
mydomain.com = MYDOMAIN.COM

Install Squid3

We install squid now as we need the squid3 directories available. Squid configuration takes places after authentication is configured.

#apt-get install squid3 ldap-utils

Authentication

The Proxy uses 3 methods to authenticate clients:

1. Negotiate/Kerberos
2. Negotiate/NTLM and 
3. basic authentication

Some applications cannot use Kerberos and need to rely on NTLM (notably iTunes). A problem also exists in the order in which the authentication helpers are used, one example is when using IE on a non-domain computer it will fail to negotiate kerberos and will not failover to NTLM or basic authentication, this is regardless of the order in which the helpers are provided. The result is the user will endlessly receive a popup window requesting authentication. A negotiate wrapper (http://sourceforge.net/projects/squidkerbauth/files/negotiate_wrapper/) around the Kerberos and NTLM helpers resolves this issue.

Kerberos

Kerberos utilizes msktutil -- an Active Directory keytab manager. We need to install some packages that msktutil requires.

#apt-get install libsasl2-modules-gssapi-mit libsasl2-modules

To install msktutil on Debian Wheezy, you may either download and compile the source or use the Debian Squeeze package here.

Initiate a kerberos session to the server with administrator permissions to add
objects to AD, update the username where necessary. msktutil will use it to create
our kerberos computer object in Active directory.

#kinit administrator
Password for administrator@MYDOMAIN.COM:

 It should return without errors. You can see if you succesfully obtained a ticket with:  

#klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: administrator@MYDOMAIN.COM
Valid starting Expires Service principal
01/09/12 09:01:49 01/09/12 19:01:53 krbtgt/MYDOMAIN.COM@MYDOMAIN.COM
renew until 01/10/12 09:01:49

Now we configure the proxy's kerberos computer account and service principle by
running msktutil (remember to update the highlighted values with yours).
Warning: There are 2 important caveats in regard to the msktutils --computer-name argument.

1. --computer-name cannot be longer than 15 characters due to netbios name limitations --computer-name must be different from the proxy's hostname so computer
2. account password updates for NTLM and Kerberos do not conflict.  Thus,this example appends -k to the hostname.

Execute the msktutil command as follows:

#msktutil -c -b "CN=COMPUTERS" -s HTTP/cou-firewall.mydomain.com -k /etc/squid3/PROXY.keytab --computer-name COU-FIREWALL-K --upn HTTP/cou-firewall.mydomain.com --server exchange01.mydomain.com --verbose --enctypes 28

Pay attention to the output of the command to ensure success, because we are using --verbose output you should review it carefully. Set the permissions on the keytab so squid can read it.

#chgrp proxy /etc/squid3/PROXY.keytab
#chmod g+r /etc/squid3/PROXY.keytab

Destroy the administrator credentials used to create the account.

#kdestroy

On the Windows Server reset the Computer Account in AD by right clicking on the COU-FIREWALL-K Computer object and select "Reset Account", then run msktutil as follows to ensure the keytab is updated as expected and that the keytab is being sourced by msktutil from /etc/krb5.conf correctly. This is not completely necessary but is useful to ensure msktutil works as expected.

Then run the following:

#msktutil --auto-update --verbose --computer-name squidproxy-k

Note: Even though the account was added in capital letters, the --auto-update in msktutil requires the --computer-name to be lower case.

If the keytab is not found try adding -k /etc/squid3/PROXY.keytab to the command to
see if it works and then troubleshoot until resolved or users will not be able to
authenticate with Squid.

Add the following to cron so it can automatically updates the computer account in
active directory when it expires (typically 30 days). I pipe it through logger so I
can see any errors in syslog if necessary. As stated msktutil uses the default
/etc/krb5.conf file for its parameters so be aware of that if you decide to make any
changes in it.

The SHELL and PATH variables are there to ensure cron runs properly, change
this if you know what your doing.

#crontab -e
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
#  m    h   dom   mon    dow    command
   00    4    *       *         *         msktutil --auto-update --verbose --computer-name squidproxy-k | logger -t msktutil

Add the following configuration to /etc/default/squid3 so squid knows where to
find the kerberos keytab.

#nano /etc/default/squid3
KRB5_KTNAME=/etc/squid3/PROXY.keytab
export KRB5_KTNAME

NTLM

Install Samba and Winbind

apt-get install samba winbind samba-common-bin

Stop the samba and winbind daemons

service winbind stop && service samba stop

Copy the default smb.conf out of the way and edit the smb.conf

cp /etc/samba/smb.conf /etc/samba/smb.conf.default
cat /dev/null > /etc/samba/smb.conf
nano /etc/samba/smb.conf

local master = no
workgroup = MYDOMAIN
security = ads
realm = MYDOMAIN.COM
winbind uid = 10000-20000
winbind gid = 10000-20000
winbind use default domain = yes
winbind enum users = yes
winbind enum groups = yes
load printers = no
printing = bsd
printcap name = /dev/null
disable spoolss = yes

Now join the proxy to the domain.

#net ads join -U Administrator
Enter Administrator's password:
Using short domain name -- MYDOMAIN
Joined 'COU-FIREWALL' to realm 'mydomain.com'

Start samba and winbind and test acces to the domain

#service samba start && service winbind start
#wbinfo -t
checking the trust secret for domain EXAMPLE via RPC calls succeeded
#wbinfo -a EXAMPLE\\testuser%'password'
plaintext password authentication succeeded
challenge/response password authentication succeeded

Set Permissions so the proxy user account can read /var/run/samba/winbindd_privileged

#gpasswd -a proxy winbindd_priv

append the following to cron to regularly change the computer account password.

#crontab -e
05    4    *    *    *    net rpc changetrustpw -d 1 | logger -t changetrustpw

Basic

In order to use basic authentication by way of LDAP we need to create an account
with which to access Active Directory.  In Active Directory create a user called "Squid Proxy" with the logon name squid@mydomain.com. Ensure the following is true when creating the account

• User must change password at next logon Unticked
• User cannot change password Ticked
• Password never expires Ticked
• Account is disabled Unticked

Create a password file used by squid for ldap access and secure the file permissions (substitute the word "squidpass" below with your password).

#echo 'squidpass' > /etc/squid3/ldappass.txt
#chmod o-r /etc/squid3/ldappass.txt
#chgrp proxy /etc/squid3/ldappass.txt

Access Groups

Authorisation to use the internet is managed via Security Groups in Active
Directory. By default the squid account will not be able to query the "memberOf" attribute in
AD. Select the top level of your active directory domain in Active Directory Users
and Computers, Right click on it and choose properties, Security Tab, Add the
squid user and give it read permissions (should happen by default) and allow it to
read "This Object and all descendant objects" (Server 2008) by going into Advanced options.
 
Tip: If you do not see the Security Tab on the domain properties window, select
View and tick Advanced features from the AD Users and Computers MMC
Create the following Security Groups and descriptions in AD and add users to the
relevant groups. I suggest adding all your users to Internet Users Standard and
then increasing or decreasing their access level by adding them to additional
groups. The order of access is from least access to highest. So for example, if a
user was a member of Blocked, Standard and Anonymous, Blocked takes priority
and they would have no internet access.

Internet Users Blocked
Description: Members of this group have no internet access
Internet Users Restricted
Description: Members of this group can access the internet allowed sites only
Internet Users Standard
Description: Members of this group can access the internet except for blocked sites
Internet Users Exception
Description: Members of this group can access the internet with exceptions to blocked sites
Internet Users Full
Description: Members of this group have full internet access
Internet Users Anonymous
Description: Members of this group have full internet access and no access is logged

Create the associated files on the proxy. Squid will use these to lookup group membership for users.

#echo 'Internet Users Blocked' > /etc/squid3/blocked_a cess.txt
#echo 'Internet Users Restricted' > /etc/squid3/restricted_access.txt
#echo 'Internet Users Standard' > /etc/squid3/standard_access.txt
#echo 'Internet Users Exception' > /etc/squid3/exception_access.txt
#echo 'Internet Users Full' > /etc/squid3/full_access.txt
#echo 'Internet Users Anonymous' > /etc/squid3/anonymous_access.txt

Note: After making changes to group membership squid needs to be reloaded on the Proxy

#service squid3 reload

Configure Squid

Install negotiate_wrapper
Firstly we need to install negotiate_wrapper. Install the necessary build tools.

apt-get install build-essential linux-headers-$(uname -r)

Then compile and install.

cd /usr/local/src/
wget "http://downloads.sourceforge.net/project/squidkerbauth/negotiate_wrapper/negotiate_wrapper-1.0.1
tar -xvzf negotiate_wrapper-1.0.1.tar.gz
cd negotiate_wrapper-1.0.1/
./configure
make
make install

squid.conf
We then setup squid and it's associated config files.

cp /etc/squid3/squid.conf /etc/squid3/squid.conf.default
cat /dev/null > /etc/squid3/squid.conf
nano /etc/squid3/squid.conf

Note: Update the cache manager variable with the emails address of the person
in charge of the proxy.  Study and update the following text carefully, replacing the example content with your networks configuration - if you get something wrong your proxy will not
work.

## /etc/squid3/squid.conf Configuration File ####

### cache manager
cache_mgr administrator@mydomain.com

### negotiate kerberos and ntlm authentication
auth_param negotiate program /usr/local/bin/negotiate_wrapper -d --ntlm /usr/bi$
auth_param negotiate children 10
auth_param negotiate keep_alive off

### pure ntlm authentication
auth_param ntlm program /usr/bin/ntlm_auth --diagnostics --helper-protocol=squi$
auth_param ntlm children 10
auth_param ntlm keep_alive off

### provide basic authentication via ldap for clients not authenticated via ker$
auth_param basic program /usr/lib/squid3/squid_ldap_auth -R -b "dc=mydomain,dc=$
auth_param basic children 10
auth_param basic realm Internet Proxy
auth_param basic credentialsttl 1 minute

### ldap authorisation
external_acl_type memberof %LOGIN /usr/lib/squid3/squid_ldap_group -R -K -b "dc$

### acl for proxy auth and ldap authorizations
acl auth proxy_auth REQUIRED
# aclname acltype typename activedirectorygroup
acl BlockedAccess external memberof "/etc/squid3/blocked_access.txt"
acl RestrictedAccess external memberof "/etc/squid3/restricted_access.txt"
acl StandardAccess external memberof "/etc/squid3/standard_access.txt"
acl ExceptionAccess external memberof "/etc/squid3/exception_access.txt"
acl FullAccess external memberof "/etc/squid3/full_access.txt"
acl AnonymousAccess external memberof "/etc/squid3/anonymous_access.txt"
acl allowedsites dstdomain "/etc/squid3/allowedsites.txt"
acl blockedsites dstdomain "/etc/squid3/blockedsites.txt"
acl exceptedsites dstdomain "/etc/squid3/exceptedsites.txt"
acl prioritysites dstdomain "/etc/squid3/prioritysites.txt"

### squid defaults
acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl SSL_ports port 443
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost

### http_access rules
# allow unrestricted access to prioritysites
http_access allow prioritysites
http_access allow prioritysites
# enforce authentication, order of rules is important for authorization levels
http_access deny !auth
# prevent access to basic auth prompt for BlockedAccess users
http_access deny BlockedAccess all
http_access allow allowedsites
http_access deny RestrictedAccess all
http_access allow AnonymousAccess auth
http_access allow FullAccess auth
http_access allow ExceptionAccess exceptedsites auth
http_access deny blockedsites
http_access allow StandardAccess auth
# DO NOT REMOVE THE FOLLOWING LINE
http_access deny all

### logging
# don't log allowedsites, prioritysites, AnonymousAccess
#access_log /var/log/squid3/access.log squid !allowedsites !prioritysites !Anon$
#log all traffic
access_log /var/log/squid3/access.log squid

### squid Debian defaults
http_port 3128
hierarchy_stoplist cgi-bin ?
coredump_dir /var/spool/squid3
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
refresh_pattern . 0 20% 4320
dns_nameservers 10.64.0.1
dns_defnames on
emulate_httpd_log on
log_mime_hdrs on
log_fqdn on

Create the blocked and allowed sites files and some blocked and allowed sites to
them.

touch /etc/squid3/allowedsites.txttouch
touch /etc/squid3/blockedsites.txt
touch /etc/squid3/exceptedsites.txt
touch /etc/squid3/prioritysites.txt

Client configurations are not discussed in this article, but will be in another.  This is a complex topic of itself.