virsh is the main command line tool to manage virtual guest domains on Linux. Several Linux distributions use polkit, a toolkit for handling unprivileged access to processes, to manage access to the libvirt virtualisation layer.

libvirt ships with a set of polkit actions defining operations that clients (example: virsh) can request from privileged processes (example: libvirtd). These action files are stored in /usr/share/polkit-1/actions and can be viewed with the pkactions command.

The polkit action we’re interested in is org.libvirt.unix.manage.

$ pkaction --verbose --action-id org.libvirt.unix.manage
org.libvirt.unix.manage:
  description:       Manage local virtualized systems
  message:           System policy prevents management of local virtualized systems
  vendor:
  vendor_url:
  icon:
  implicit any:      auth_admin_keep
  implicit inactive: auth_admin_keep
  implicit active:   auth_admin_keep

polkit can be configured to allow clients that belong to a particular unix group to run an action of org.libvirt.unix.manage.

[libvirt Admin Access]
Identity=unix-group:virt
Action=org.libvirt.unix.manage
ResultAny=yes
ResultInactive=yes
ResultActive=yes

Any invocations of this action are logged to /var/log/secure.

Now we need to add users to the virt group.

# usermod -a -G virt rene

virsh can connect to remote hypervisors running libvirtd. To configure virsh to connect to libvirtd running on a local server by default, we need to define the URI of qemu:///system as a environment variable.

if test -x `which virsh`; then
  export LIBVIRT_DEFAULT_URI=qemu:///system
fi

Users within the virt group should now be able to run virsh commands without having to be root.

Nagios Remote Plugin Executor (NRPE), is used by nagios to run commands remotely on linux servers.

For example, to discover the load, a measure of computional work, on a linux server you would run uptime which provides you with the load average.

$ uptime
 22:41:02 up 14 min,  1 user,  load average: 0.00, 1.02, 2.40

Without NRPE, nagios is unable to run this command remotely.

NRPE works by running as a daemon on the remote linux server listening on a port. Nagios connects to the NRPE daemon and runs a NRPE command which typically calls a nagios plugin such as check_load, returning some info aswell as a status of OK, WARNING, CRITICAL or UNKNOWN.

First, install and start the NRPE server on the remote linux server.

$ sudo yum install nrpe
$ sudo service nrpe start

NRPE by default listens on port 5666/tcp which you will need to allow through if your remote linux server is running a firewall. Be sure to only expose the NRPE service to your nagios server by specifying the IP address of your nagios server with -s in your iptables rules.

-A INPUT -m state --state NEW -m tcp -p tcp --dport 5666 -s 10.0.0.1 -j ACCEPT
$ service iptables restart

On the nagios server, install the check_nrpe plugin.

$ sudo yum install nagios-plugins-nrpe

check_nrpe is the executable that nagios will use to query the remote linux server.

$ /usr/lib64/nagios/plugins/check_nrpe -H 10.0.1.254
CHECK_NRPE: Error - Could not complete SSL handshake.

NRPE will only allow certain hosts to connect which is configured by the allowed_hosts option in /etc/nagios/nrpe.cfg. Let’s add our nagios server and restart the NRPE service.

allowed_hosts=127.0.0.1,10.0.0.1
$ sudo service nrpe restart

Again, let’s run check_nrpe from our nagios server.

$ /usr/lib64/nagios/plugins/check_nrpe -H 10.0.1.254
NRPE v2.14

By default, NRPE should be configured with a check_load command.

command[check_load]=/usr/lib64/nagios/plugins/check_load -w 15,10,5 -c 30,25,20

Call this command from the nagios server via NRPE using the check_nrpe plugin.

$ /usr/lib64/nagios/plugins/check_nrpe -H 10.0.1.254 -c check_load
OK - load average: 0.00, 0.00, 0.00|load1=0.000;15.000;30.000;0; load5=0.000;10.000;25.000;0; load15=0.000;5.000;20.000;0;

Once we’ve confirmed that the nagios server is able to communicate with the remote linux server over NRPE we can configure nagios and reload.

define service {
  use                            generic-service
  host_name                      webserver.example.org
  service_description            NRPE - Load
  check_command                  check_nrpe!check_load
}
$ sudo service nagios reload

virsh provides Linux System Administrators with the ability to dynamically scale allocated memory to virtual guests during runtime.

$ virsh dommemstat web-server
actual 2097152
rss 903040

dommemstat shows that the domain web-server has a memory allocation limit of 2097152 kB and is currently using 903040 kB. You can confirm this with ps.

$ ps -C qemu-kvm -o rss,cmd
  RSS CMD
903040 /usr/bin/qemu-kvm -name web-server -S -M pc-1.2 -enable-kvm -m 2048 -smp

The memory allocation limit is set with setmaxmem and can only be performed whilst the domain is inactive.

$ virsh setmaxmem web-server 1048576 --config

By passing setmaxmem the config option the memory allocation limit is written to the virsh configuration file.

The current memory allocation for a guest domain is set with setmem.

$ virsh setmem web-server 786432 --config --live

When passing setmem the live option, a memory balloon is performed on the running guest and the change happens instantly. This can be verified with dominfo.

$ virsh dominfo web-server
Id:             3
Name:           web-server
UUID:           a1a0ee38-20ff-144b-a0ef-2bb14e4ce167
OS Type:        hvm
State:          running
CPU(s):         2
CPU time:       33.9s
Max memory:     1048576 KiB
Used memory:    786432 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: selinux
Security DOI:   0
Security label: system_u:system_r:svirt_t:s0:c782,c903 (enforcing)

If you have a SSH private key and are missing the public key you can create the public key based off the private key with ssh-keygen.

$ ssh-keygen -y -f id_rsa > id_rsa.pub

Ever needed to find what branch a certain git commit belong to?

Check out git-what-branch.

$ cd /srv/git/project.git
$ perl git-what-branch --all ff00bb
remote/banch/example

Nagios is a powerful open source monitoring system that is used by many IT organisations to monitor IT infrastructure. Without a doubt, Nagios is the most widely used monitoring application that Linux System Administrators deploy today.

There are some configuration objects that nagios uses which we will cover today. These objects are contacts, hosts, services and commands.

To install and setup your first server within nagios, start by enabling the EPEL repository if you haven’t already.

Install the nagios RPM and it’s ping plugin.

# yum install nagios nagios-plugins-icmp

This will install the nagios web interface which you can access at http://server/nagios which has a username and password of nagiosadmin. There is still some further configuration that needs to be done before any servers show up within the web interface.

Nagios contacts are used to send notifications of alerts and recoveries. Create the first nagios contact in the file /etc/nagios/conf.d/contacts.cfg

define contact{
  contact_name    john
  alias           John Example (sysadmin)
  use             generic-contact
  email           john@exmaple.com
}

Nagios hosts define servers, routers, switches and other devices which nagios monitors. Create the first nagios host in the file /etc/nagios/conf.d/hosts.cfg

define host{
  use                   generic-host
  host_name             server.example.com
  address               server.example.com
  max_check_attempts    5
}

Nagios services objects are services that nagios monitors on the remote server. Create the first nagios service in the file /etc/nagios/conf.d/services.cfg

define service{
  use                   generic-service
  host_name             server.example.com
  service_description   PING
  check_command         check_icmp!500.0,80%!1000.0,100%
}

Nagios service objects call nagios commands which are a defined set of objects that run commands which return OK, WARNING, CRITICAL or UNKNOWN. The first Nagios command that we will setup is a simple ping command.

define command{
  command_name    check_icmp
  command_line    $USER1$/check_icmp -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$
}

Restart the nagios service.

# service nagios restart

Bring up the nagios web interface again and go to “Services” where you will find the server you are now monitoring. It will take up to 5 minutes for nagios to run the service check “PING” which hopefully returns an OK.

Linux Sysadmin Protip

You can force Nagios to execute a service check and not have to wait 5 minutes by going to the “Service Information” page for a particular service and choosing “Re-schedule the next check for this service”.

Graphite is highly scalable real-time graphing system. It is seen as a replacement for more traditional graphing systems such as cacti and munin.

To install graphite, first you will need to enable the EPEL repository first. Once enabled pull in the required RPMS.

# yum install graphite-web graphite-web-selinux mysql mysql-server MySQL-python

Start up mysql and setup a root password.

# service mysqld start
# /usr/bin/mysql_secure_installation

Configure graphite to use the MySQL database within the /etc/graphite-web/local_settings.py config file.

DATABASES = {
  'default': {
    'NAME': 'graphite',
    'ENGINE': 'django.db.backends.mysql',
    'USER': 'graphite',
    'PASSWORD': 'complexpassw0rd',
    'HOST': 'localhost',
    'PORT': '3306',
  }
}

Create the graphite account and database within MySQL.

# mysql -e "CREATE USER 'graphite'@'localhost' IDENTIFIED BY 'complexpassw0rd';" -u root -p 
# mysql -e "GRANT ALL PRIVILEGES ON graphite.* TO 'graphite'@'localhost';" -u root -p
# mysql -e "CREATE DATABASE graphite;" -u root -p
# mysql -e 'FLUSH PRIVILEGES;' -u root -p

Setup the graphite database with what is required by graphite.

# /usr/lib/python2.6/site-packages/graphite/manage.py syncdb

You should now have a working graphite installation.

http://localhost/graphite

You may want to download RPMs only and not install them using yum. Perhaps you’re within a change freeze but would like to ‘warm’ up your yum cache to make your production security upgrades alot faster and smoother.

The yum-plugin-downloadonly yum plugin provides yum with a –downloadonly option.

# yum install yum-plugin-downloadonly
# yum install --downloadonly puppet

You can also download the RPMs to a directory of your choosing and not /var/cache/yum.

# yum install --downloadonly --downloaddir=/var/tmp/puppet puppet

Extra Packages for Enterprise Linux (EPEL) is the group that creates, maintains and manages high quality packages for RHEL, CentOS and Scientific Linux.

To enable the repo all that is needed is the epel-release RPM.

# rpm -ivh https://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

The epel-release RPM will setup the RPM EPEL gpg key and the required yum repo files.

Traditionally, Operations Teams setup nagios to send alerts using SMS. Whilst SMS is out-of-band and would withstand outages to your IP links, in some cases it becomes cost prohibitive especially for small startups with little cash to burn.

Using push notifications over IP to smart phones is an alternative which in most cases is free though does result in your monitoring servers ability to notify your operations teams of critical issues relying on your IP link being up.

One push notification service is boxcar.io.

Sign up with boxcar.io and install the app on your iOS device.

Drop the small boxcar shell script on the nagios server. This script is a copy and paste of the script posted by Jedda Wignall.

#!/bin/bash
while getopts "e:h:m:" optionName; do
  case "$optionName" in
    e) boxcarEmail=( "$OPTARG" );;
    h) host=( "$OPTARG" );;
    m) message=( "$OPTARG" );;
  esac
done

curl --sslv3 --data-urlencode "email=$boxcarEmail" \
  --data-urlencode "&notification[from_screen_name]=$host" \
  --data-urlencode "&notification[message]=$message" https://boxcar.io/devices/providers/MH0S7xOFSwVLNvNhTpiC/notifications

You will need to **subscribe** to the boxcapp Monitoring generic provider.

$ curl -d "email=your-boxcar-registered-email@example.com" http://boxcar.io/devices/providers/MH0S7xOFSwVLNvNhTpiC/notifications/subscribe

Create the nagios host and service command objects that nagios will call when it needs to send an alert.

define command {
  command_name  notify-host-by-boxcar
  command_line /bin/bash /usr/local/bin/boxcar.sh -e "$CONTACTADDRESS1$" -h "$HOSTNAME$" -m "$NOTIFICATIONTYPE$: $HOSTSTATE$"
}

define command {
  command_name  notify-service-by-boxcar
  command_line /bin/bash /usr/local/bin/boxcar.sh -e "$CONTACTADDRESS1$" -h "$HOSTNAME$" -m "$NOTIFICATIONTYPE$: $SERVICEDESC$ $SERVICESTATE$ ($SERVICEOUTPUT$)"
}

Finally you will then need to update your nagios contact object adding an address1 definition.

define contact{
  contact_name                    rene
  use                             generic-contact
  alias                           Rene Cunningham
  service_notification_commands   notify-service-by-email,notify-service-by-boxcar
  host_notification_commands      notify-host-by-email,notify-host-by-boxcar
  email                           rene@linuxsysadmintutorials.com
  address1                        rene@example.com
}

Restart the nagios service. The next nagios host or service alert will be sent by boxcar.

# service nagios restart

Linux Sysadmin Protip

You can quickly test whether boxcar notifications are working with nagios by generating a failed service or host with iptables. Choose a host or a service that nagios is monitoring and use iptables to block outbound connections.

For example, if nagios was monitoring HTTP on web-server-1.example.com block all outbound packets from the monitoring server to web-server-1.exmaple.com on port 80.

# iptables -I OUTPUT -d web-server-1.example.com -p tcp --dport 80 -j REJECT

Log into the nagios web interface and Re-schedule the next check of this service for this particular service. Delete the iptables rule once you’ve finished testing.

# iptables -D OUTPUT -d web-server-1.example.com -p tcp --dport 80 -j REJECT

all code and configs can be found on github.com/renecunningham/linuxsysadmintutorials.com/tree/master/boxcar-nagios