Ghetto-timemachine Ruby Backup Script

I decided to try my hand at some Ruby scripting and figured, what better way than to port one of my existing Perl scripts.

The result is the Ruby ghetto-timemachine.rb backup script!

The code is maintained here:
https://github.com/flakrat/ghetto-timemachine

So, what does it do? In a nutshell, it performs daily backups using a combination of hard links and rsync that result in a Time Machine like backup that uses very little additional space beyond the initial backup and the size of any files that change.

Example usage:

• Local source to local destination with comma separated list of excludes

  
  ./ghetto-timemachine.rb --src ~/Pictures --dest /media/USB/backup --excludes \*.raw,\*.bmp
  

• Local source to remote destination

  
  ./ghetto-timemachine.rb --src ~/Pictures --dest flakrat@rathole:/backup --excludes \*.raw,\*.bmp
  

Nightly cron job running at 1:10AM

10 1 * * * /home/flakrat/bin/ghetto-timemachine.rb --src ~/Pictures --dest flakrat@rathole:/backup --excludes \*.raw,\*.bmp

See the README for a more detailed description of the backup and rotation process.

Fedora 16 Gnome Terminal Redraw Doesn't Always Completely Render

I've noticed an issue on my Fedora 16 workstation where Gnome Terminal redraws don't always completely render.

For example, it could be as simple as pressing the up arrow key to scroll through the CLI history (or ESC K for the 'set -o vi' crowd). A good percentage of the time when I'd do that, the command line wouldn't completely render, CTRL C and scrolling again usually cleared it up.

I also experienced the behavior a lot when in VI. Scroll or jump to a line and the text would end up being a combination of the previous page and the new location.

I experienced the issue using Nouveau and binary Nvidia drivers.

The following Bugzilla ticket provided a possible workaround. I've only just implemented the ENV variables, but after 1 day of use I have yet to experience the issue, where as before it was pretty consistent.

To globally apply the settings, add the following variables to end of /etc/gdm/Xsession

# Attempting to fix the Gnome terminal redraw issue:
# https://bugzilla.redhat.com/show_bug.cgi?id=720605
export CLUTTER_DEBUG=disable-culling
export CLUTTER_PAINT=disable-clipped-redraws:disable-culling

Reboot, log back in and hopefully the issue goes away :-)

CentOS 6.2 Installation issue with Dell Precision Workstation T7500

Attempting to kickstart a new Dell Precision Workstation T7500 with an nVidia Quadro 2000 with the CentOS6.2 DVD media resulted in a kernel panic. All signs pointed to the community built nouveau module "nouveau_probe_i2c_addr panic"

I'm not so worried about the nouveau module working, since the kickstart process installs the official nVidia binary release.

To work around the issue and allow the system to boot fully into the installer, simply add "rblacklist=nouveau" to the argument list:

> vmlinuz initrd=initrd.img ks=http://srv1/ks/wks-el6.cfg rdblacklist=nouveau reboot=pci

How to Configure Nagios Check_MK to Report Number of Package Updates Need on Client

The Check_mk Updates plugin was posted to the Check_mk mailing list by Jonathan Mills. The following Blog entry covers the steps I took to integrate it in my environment. I opted to distribute the two plugin files to the clients via puppet rather than RPM.

• OMD version: 0.51.20111117
  
• Puppet version: 2.6.12
  
• Server is CentOS 5.7 x86_64
  
• Clients are CentOS 5.7, CentOS 6.2, RHEL5.7, Fedora 16 ix86,x86_64
  

Refrences

• Jonathan Mills check_mk mailing list post containing the source files
  
• Jonathan Mills check_mk mailing list post feedback providing additional check_mk and nagios configurations
  

This check currently only works with Yum based clients (tested on CentOS 5 and 6, RHEL5 and Fedora 16) and requires the yum-security package (EL5) or yum-plugin-securities (EL6) The plugin attempts to identify security and non-security packages that are pending install. For RHEL, it's simple to get this info via "yum --security check-update"

$ sudo yum --security check-update
Loaded plugins: dellsysid, rhnplugin, security
Limiting package lists to security relevant ones
Needed 6 of 17 packages, for security

kernel.x86_64                        2.6.18-274.17.1.el5             rhel-x86_64-client-5
kernel-devel.x86_64                  2.6.18-274.17.1.el5             rhel-x86_64-client-5
kernel-headers.x86_64                2.6.18-274.17.1.el5             rhel-x86_64-client-5
libxml2.i386                         2.6.26-2.1.12.el5_7.2             rhel-x86_64-client-5
libxml2.x86_64                       2.6.26-2.1.12.el5_7.2             rhel-x86_64-client-5
libxml2-python.x86_64                2.6.26-2.1.12.el5_7.2             rhel-x86_64-client-5

For CentOS and most likely Scientific Linux, the security errata are not provided with the repos, so the above command will always report 0 security updates. This is solved in the plugin by parsing the results of the -v (verbose) output.

1. Add the client side scripts to the puppet server (Puppet isn't necessary, you can install the RPM provided in the tar file on the check_mk post)
   
   • Create the file directories under site
     

     
     $ mkdir -p var/lib/puppet/files/site/etc/check_mk
     $ mkdir -p var/lib/puppet/files/site/usr/lib/check_mk_agent/plugins 
     

   • Create the check_updates.cfg etc file
     

     
     $ vim var/lib/puppet/files/site/etc/check_mk/check_updates.cfg 
     
     

     
     # +------------------------------------------------------------------+
     # |             ____ _               _        __  __ _  __           |
     # |            / ___| |__   ___  ___| | __   |  \/  | |/ /           |
     # |           | |   | '_ \ / _ \/ __| |/ /   | |\/| | ' /            |
     # |           | |___| | | |  __/ (__|   <    | |  | | . \            |
     # |            \____|_| |_|\___|\___|_|\_\___|_|  |_|_|\_\           |
     # |                                                                  |
     # | Copyright Mathias Kettner 2010             mk@mathias-kettner.de |
     # +------------------------------------------------------------------+
     #
     # This file is part of Check_MK.
     # The official homepage is at http://mathias-kettner.de/check_mk.
     #
     # check_mk is free software;  you can redistribute it and/or modify it
     # under the  terms of the  GNU General Public License  as published by
     # the Free Software Foundation in version 2.  check_mk is  distributed
     # in the hope that it will be useful, but WITHOUT ANY WARRANTY;  with-
     # out even the implied warranty of  MERCHANTABILITY  or  FITNESS FOR A
     # PARTICULAR PURPOSE. See the  GNU General Public License for more de-
     # ails.  You should have  received  a copy of the  GNU  General Public
     # License along with GNU Make; see the file  COPYING.  If  not,  write
     # to the Free Software Foundation, Inc., 51 Franklin St,  Fifth Floor,
     # Boston, MA 02110-1301 USA.
     
     # check_updates.cfg
     # This file configures mk_check_updates.
     
     # interval (seconds) between runs of 'yum check-update'
     INTERVAL=7200
     
     # path to log file
     LOG="/var/log/check_updates.log"
     

   • Create the mk_check_updates script (The script has updates that I made to resolve some issues related to the priorities yum plugin and yum output beginning with Keeping or Removing, so it's slightly different than the original source)
     

     
     $ vim var/lib/puppet/files/site/usr/lib/check_mk_agent/plugins/mk_check_updates 
     
     

     
     #!/bin/bash
     #
     # OUTPUT:
     # (security) (non-security) (runtime) (check age)
     # <<<updates>>>
     # 7 40 7 209
     
     # Unix time (seconds since Unix epoch)
     START=$(date +%s)
     
     TIME=
     AGE=
     
     INTERVAL=86400                          # default interval once a day
     LOG="/var/log/check_updates.log"        # default path to log file
     
     # Source config file if it exists
     if [ -e "/etc/check_mk/check_updates.cfg" ]; then
         . /etc/check_mk/check_updates.cfg
     fi
     
     # function run_check_update
     run_check_update () {
     if which yum >/dev/null; then
     
       if [ ! -e "/var/run/yum.pid" ]; then
     
         cat /dev/null > $LOG
     
         # Check for security RPMS
         yum -v --security check-update | egrep '(i.86|x86_64|noarch)' | egrep -v '\(priority\)' |\
      egrep -v '(^Keeping|^Removing|^Nothing|^Excluding|^Looking)' | sed 's/^.*--> //g' | while read L
         do
     
           RPM=$(echo $L | awk '{print $1}')
           Q=$(echo ${L} | grep 'non-security' > /dev/null; echo $?)
           if [ $Q -eq 0 ]; then
             echo "non-security $RPM" >> $LOG
           else
             echo "security $RPM" >> $LOG
           fi
     
         done
     
       fi
     fi
     }
     
     # function timeyet
     timeyet () {
     LAST=$(stat -c '%Y' $LOG)
     NOW=$(date +%s)
     AGE=$((NOW - LAST))
     [ $AGE -gt $INTERVAL ] && TIME=1 || TIME=0
     }
     
     # See if it's time to run 'yum check-updates' yet
     if [ ! -e $LOG ]; then
       touch $LOG
       run_check_update
       timeyet
     else
       timeyet
       if [ $TIME = 1 ]; then
         run_check_update
         timeyet
       fi
     fi
     
     # Gather results from log file
     SEC=$(grep '^security' $LOG | wc -l)
     NON=$(grep '^non-security' $LOG | wc -l)
     
     # Unix time (seconds since epoch)
     END=$(date +%s)
     
     RUNTIME=$((END - START))
     
     echo '<<<updates>>>'
     echo $SEC" "$NON" "$RUNTIME" "$AGE
     exit 0
     
     

   • Add the scripts to git
     

     
     $ git add var/lib/puppet/files/site/usr/lib/check_mk_agent/plugins/mk_check_updates
     $ git add var/lib/puppet/files/site/etc/check_mk/check_updates.cfg 
     $ git commit -a -m "Adding check_mk client side scripts to report yum updates"
     $ git push
     

   • Add the scripts to the check_mk class to ensure that the clients get the code
     

     
     $ vim etc/puppet/manifests/classes/check_mk.pp
     
     

     
     # etc/puppet/manifests/classes/check_mk.pp
     
     class check_mk {
        case $operatingsystem {
           "centos",
           "fedora",
           "redhat": {
              package {["check_mk-agent", "check_mk-agent-logwatch"]:
                 ensure   => latest,
                 notify   => Service["xinetd"],
              }
              service { "xinetd":
                 ensure     => running,
                 enable     => true,
              }
              file { "/etc/check_mk/check_updates.cfg":
                 owner => "root",
                 group => "root",
                 mode => 755,
                 source => "puppet:///site/etc/check_mk/check_updates.cfg",
              }
              file { "/usr/lib/check_mk_agent/plugins/mk_check_updates":
                 owner => "root",
                 group => "root",
                 mode => 755,
                 source => "puppet:///site/usr/lib/check_mk_agent/plugins/mk_check_updates",
              }
          }
           default: { }
        }
     }
     

   • Ensure that the check_mk class is included in the node definitions (currently included in the baseclass template)
     
   • Git commit the changes to check_mk.pp class and push to the git server
     
2. Install the python script on the nagios server (note user defined checks go in local/share/check_mk/checks, if you put them into $SITE/share.... they won't survive the next OMD upgrade)
   

   
   $ su - sitename
   $ vim local/share/check_mk/checks/updates
   
   

   
   #!/usr/bin/python
   # -*- encoding: utf-8; py-indent-offset: 4 -*-
   
   # Jonathan Mills 10/2011
   
   # Example output from agent:
   # [security] [non-security] [runtime (seconds)] [age of results (seconds)]
   # <<<updates>>>
   # 7 40 0 13
   #
   
   updates_default_values = (5, 20)
   
   # inventory
   def inventory_updates(checktype, info):
       #if len(info) >= 1 and len(info[0]) >= 1:
       #    return [ (None, None) ]
       inventory = []
       inventory.append( (None, "updates_default_values") )
       return inventory
   
   
   # check
   def check_updates(_no_item, params, info):
       # unpack check parameters
       min_num_sec, min_num_nonsec = params
   
       for line in info:
           perfdata = []
           sec = int(line[0])
           nonsec = int(line[1])
           age = int(line[3])
           infotext = "%s Security Updates, %s Non-Critical Updates  (Last Checked %s seconds ago)" % (sec, nonsec, age)
           perfdata.append( ( "Runtime (sec)", int(line[2]) ) )
           if sec > min_num_sec:
               return (2, "CRITICAL - " + infotext, perfdata)
           elif nonsec > min_num_nonsec:
               return (1, "WARNING - " + infotext, perfdata)
           else:
               return (0, "OK - " + infotext, perfdata)
   
   # declare the check to Check_MK
   check_info['updates'] = (check_updates, "Updates", 1, inventory_updates)
   

3. Add a new time period 'nightly' to nagios that can be used to limit this check to running daily from 3AM to 4AM
   

   
   $ vim etc/nagios/conf.d/timeperiods.cfg 
   

   
   ###############################################################################
   # TIMEPERIODS.CFG - SAMPLE TIMEPERIOD DEFINITIONS
   #
   # NOTES: This config file provides you with some example timeperiod definitions
   #        that you can reference in host, service, contact, and dependency
   #        definitions.
   #
   #        You don't need to keep timeperiods in a separate file from your other
   #        object definitions.  This has been done just to make things easier to
   #        understand.
   #
   ###############################################################################
   
   # This defines a timeperiod where all times are valid for checks,
   # notifications, etc.  The classic "24x7" support nightmare. :-)
   define timeperiod{
       timeperiod_name 24x7
       alias           24 Hours A Day, 7 Days A Week
       sunday          00:00-24:00
       monday          00:00-24:00
       tuesday         00:00-24:00
       wednesday       00:00-24:00
       thursday        00:00-24:00
       friday          00:00-24:00
       saturday        00:00-24:00
   }
   
   # 'workhours' timeperiod definition
   define timeperiod{
          timeperiod_name workhours
          alias           Normal Work Hours
          monday          08:00-17:00
          tuesday         08:00-17:00
          wednesday       08:00-17:00
          thursday        08:00-17:00
          friday          08:00-17:00
   }
   
   # 'none' timeperiod definition
   define timeperiod{
       timeperiod_name  none
       alias            No Time Is A Good Time
   }
   
   # 'nightly' timeperiod definition
   define timeperiod{
            timeperiod_name         nightly
            alias                   Nightly Check
            sunday                  03:00-04:00  ; Every Sunday of every week
            monday                  03:00-04:00  ; Every Monday of every week
            tuesday                 03:00-04:00  ; Every Tuesday of every week
            wednesday               03:00-04:00  ; Every Wednesday of every week
            thursday                03:00-04:00  ; Every Thursday of every week
            friday                  03:00-04:00  ; Every Friday of every week
            saturday                03:00-04:00  ; Every Saturday of every week
   }
   
   

4. Add the new check to check_mk main.mk file
   

   
   $ vim etc/check_mk/main.mk
   
   

   
   # check-updates (OMD 0.52 requires user defined vars to prepend and underscore)
   _updates_default_values = (6, 20) # check-updates: critical when 6 or more sec updates, warning when 20 or more non-sec updates
   
   extra_service_conf["check_period"] = [
     ( "nightly", ALL_HOSTS, [ "Updates" ] ), # check-updates: Only check for updates from 3 to 4AM as set in timeperiods.cfg
   ]
   
   extra_host_conf["max_check_attempts"] = [
     ( "1", ALL_HOSTS, [ "Updates" ] ), # check-updates: Only check for updates once
   ]
   
   # Enable notifications for specific services
   extra_service_conf["notifications_enabled"] = [
     ( "1", ALL_HOSTS, ["Check_MK"]),
     ( "0", ALL_HOSTS, ["Updates"]), # check-updates: Don't notify for security OS updates
     ( "1", ALL_HOSTS, ["Memory used"]),
     ( "1", ALL_HOSTS, ["IPMI Sensor Summary","fs_*"]),
     ( "1", ["linsrv"], ["IPMI Sensor Summary","ambient_temp"]),
     ( "1", ALL_HOSTS, ["Multipath *"]),
     ( "1", ["kvm"], ALL_HOSTS, ["CPU load"]),
     ( "1", ["kvm"], ALL_HOSTS, ["CPU utilization"]),
     ( "1", ["mailsrv"], ["Postfix Queue"]),
     ( "1", ["linsrv"], ALL_HOSTS, ["Dell OMSA"]),
     ( "0", ALL_HOSTS, ALL_SERVICES), # and disable notifications for everything else
   ]
   
   service_groups = [
     ( "updates", ALL_HOSTS, [ "Updates" ] ), # check-updates: Create updates service group to make viewing in web interface easier
   ]
   
   define_servicegroups = {
      "updates" : "RHEL/CentOS Yum Updates", # check-updates: Can now statically link to a service group web view: http://nagios.server/sitename/check_mk/view.py?view_name=servicegroup&servicegroup=updates
   }
   
   

5. Rerun the inventory for the nodes
   

   
   $ check_mk -II node-01
   ...
   
   or for all nodes
   $ check_mk -II
   

6. Reload the services
   

   
   $ check_mk -O
   
   

7. Check the web page for the nodes, alternatively you can go straight to the Updates overview page:
   https://nagios.server/sitename/check_mk/view.py?view_name=servicegroup&servicegroup=updates
   

VM Guests Take Forever to Shutdown in VMware Workstation 8

I recently had a situation where a user purchased a new i7 + SSD based laptop. Needless to say, the laptop is fast.

As part of the purchase, the user also upgraded to the latest VMware Workstation, version 8.x. The users RHEL4 guest was transferred over from the old laptop (running an older version of Workstation). It powered up and ran just fine on the new hardware / version of VMware, with the exception of the guest shutdown.

It literally took an hour from the point where the RHEL4 issued the halt to the underlying hardware (i.e. the OS portion of the shutdown is done) to a fully stopped virtual guest. From an appearance stand point, the VM screen turns black and remains that way until it fully stops.

The laptop hard drive indicator was also flashing madly during the entire process.

I found suggestions to prevent virus scanners from scanning .vmdk files, that didn't change the behaviour.

The laptop also uses PGP whole disk encryption. Perhaps Workstation 8 and PGP don't play nicely together? I couldn't find any references. The old laptop (much slower than this) also had PGP with the older VMware Workstation, so PGP didn't seem to be a prime candidate.

I eventually discovered this post on the VMware message boards that provided the solution.

The thread discusses a similar issue happening in VMware Workstation 6.0.4 on Windows XP.

The solution was to add the following lines to either the global VMware config.ini file, or each individual guests .vmx file. Exit out of VMware Workstation before modifying config.ini or the .vmx files

prefvmx.minVmMemPct = "100"
mainMem.useNamedFile = "FALSE"
mainMem.partialLazySave = "FALSE"
mainMem.partialLazyRestore = "FALSE"

With that code added to the configuration file, the virtual machine shuts down immediately after the operating system issues the halt.

Placing the code in the config.ini file affects all virtual machines, new and existing, unless the settings are overridden in the individual .vmx files.

For Windows 7, the config.ini file can be found here
C:\ProgramData\VMware\VMware Workstation\config.ini

For Windows XP it can be found here:
C:\Documents and Settings\All Users\Application Data\VMware\VMware Workstation\config.ini

Dell Optiplex 790 Workstations hang while rebooting with CentOS 6

I'm working on deploying a large number of Dell Optiplex 790 workstations using kickstart and CentOS 6.

During the initial testing I found that the 790's wouldn't completely reboot with CentOS 6 installed or booted into the install media. They'd get as far as "Restarting".

The solution is to pass an option to the kernel:

reboot=pci

This can be added manually to the grub configuration file for systems already installed. For kickstarting:

1. Add the option in your kickstart file

bootloader --location=mbr --driveorder=sda --append="crashkernel=auto rhgb quiet reboot=pci" --md5pass=$1$.xxxxx

2. During the initial boot off of the CD/DVD, press TAB to alter to boot options (this is all one continuous line broken into multiple for readability)

> vmlinuz initrd=initrd.img ks=http://192.168.1.5/ks/el6/wks1.cfg
    ip=192.168.1.100 netmask=255.255.255.0 gateway=192.168.1.1 nameserver=192.168.1.1
    ksdevice=eth0 reboot=pci

Fedora 16 does not Boot if /boot is on Software RAID

In previous versions of Fedora, you could configure /boot to exist on a software RAID device (say a software mirror), however in Fedora 16 this will result in failure to boot. This wasn't a supported configuration, but it used to work.

This is a known "issue" and is explained as follows:

Cannot boot with /boot partition on a software RAID array
link to this item - Bugzilla: #750794

Attempting to boot after installing Fedora 16 with the /boot partition on a software RAID array will fail, as the software RAID modules for the grub2 bootloader are not installed. Having the /boot partition on a RAID array has never been a recommended configuration for Fedora, but up until Fedora 16 it has usually worked.

To work around this issue, do not put the /boot partition on the RAID array. Create a simple BIOS boot partition and a /boot partition on one of the disks, and place the other system partitions on the RAID array. Alternatively, you can install the appropriate grub2 modules manually from anaconda's console before rebooting from the installer, or from rescue mode. Edit the file /mnt/sysimage/boot/grub2/grub.cfg and add the lines:

insmod raid
insmod mdraid09
insmod mdraid1x
Now run these commands:

chroot /mnt/sysimage
grub2-install /dev/sda
grub2-install /dev/sdb
Adjust the device names as appropriate to the disks used in your system.

I had a system where I'd created a mirror for /boot that had been reinstalled from Fedora 13, 14, 15 and now 16. As reported, it failed to boot following the F16 install.

Destroying the mirror and creating a simple /dev/sda2 partition for /boot got it booting.