Showing posts with label rocks cluster. Show all posts
Showing posts with label rocks cluster. Show all posts

Monday, February 21, 2011

Building Mellanox OFED 1.5.2 for Rocks 5.4

Here are my notes from Rocks 5.4 and Mellanox OFED 1.5.2

Perform the build steps on a compute node. That way if the build process, run as root, has a bug, we don't risk having to rebuild the head node.

The MLNX_OFED-1.5.2 comes with modules for kernel 2.6.18-194.el5, we are using 2.6.18-194.17.1.el5, so we need to build new kernel modules.

1. Download the ISO file MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5.iso from this page

2. Ensure that the build system is running the correct kernel

# uname -r

2.6.18-194.17.1.el5

3. Mount the ISO and copy the contents to a scratch work area

# mount -t iso9660 -o loop /root/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5.iso /mnt/cdrom 
# mkdir /root/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5
# cp -r /mnt/cdrom/* /root/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5/
# umount /mnt/cdrom
# rm /root/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5.iso

4. Install some dependencies

# yum -y install libtool tcl-devel libstdc++-devel mkisofs gcc-c++ rpm-build

5. Uninstall some RPM files that will fail to uninstall during the ISO build

# yum remove \*openmpi\*

6. Build the new ISO file

# cd /root/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5

# ./docs/mlnx_add_kernel_support.sh -i /root/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5.iso
Note: This program will create MLNX_OFED_LINUX ISO for rhel5.5 under /tmp directory.
      All Mellanox, OEM, OFED, or Distribution IB packages will be removed.
Do you want to continue?[y/N]:y
Building OFED RPMs...
Removing OFED RPMs...
Running mkisofs...
Created /tmp/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5.iso

# mkdir /share/apps/mellanox/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5
# mv /tmp/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5.iso /share/apps/mellanox/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5.iso

7. Copy the new files from the iso to the NFS share

# mount -t iso9660 -o loop /share/apps/mellanox/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5.iso /mnt/cdrom
# rsync -a /mnt/cdrom/ /share/apps/mellanox/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5/

# umount /mnt/cdrom

8. List the new kernel modules

# cd /share/apps/mellanox/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5
# find . -name kernel-* | grep 194.17
./x86_64/kernel-ib-1.5.2-2.6.18_194.17.1.el5.x86_64.rpm
./x86_64/kernel-mft-2.6.2-2.6.18_194.17.1.el5.x86_64.rpm
./x86_64/kernel-ib-devel-1.5.2-2.6.18_194.17.1.el5.x86_64.rpm

9. Test the installer on one of the compute nodes

# cd /share/apps/mellanox/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5
# ./mlnxofedinstall --force --hpc

This will automatically update the firmware on the HCA.

10. This OFED can be installed on the compute nodes by adding this section to extend-compute.xml (note, I normally put other driver updates into this 'post-98-installdrivers' script). Also notice the yum install, the MLNX OFED install will remove any package containing 'openmpi' in the package name, this line reinstalls said packages


<file name="/etc/rc.d/rocksconfig.d/post-98-installdrivers" perms="0755">
#!/bin/sh

# Install Mellanox
if [ "$(/sbin/lspci | grep -i connectx)" != "" ] ; then
  /usr/bin/yum -y remove openmpi\* rocks-openmpi\*
  /share/apps/mellanox/MLNX_OFED_LINUX-1.5.2-2.0.0-rhel5.5-2.6.18-194.17.1.el5/mlnxofedinstall --hpc --force

  /sbin/chkconfig --add openibd
  /sbin/chkconfig openibd on
  /sbin/service openibd start
fi

/usr/bin/yum -y install my-custom-openmpi my-custom-application-openmpi

/bin/mv /etc/rc.d/rocksconfig.d/post-98-installdrivers /root/post-98-installdrivers

# Reboot one final time
/sbin/shutdown -r now

</file>

Adding Infiniband over IP to Rocks

20120611 - Based on a question to the Rocks mailing list, I'm adding this section to explain how to enable TCP/IP over Inifiniband via Rocks. This process should add the IP addresses to the Rocks managed DNS / hosts. The IP addresses of my compute-0-x nodes start at 254 and work backwards, so that's what I used for the IB ip addresses: First add the new network, calling it 'infiniband', or whatever name you'd like
# rocks add network infiniband subnet=192.168.3.0 netmask=255.255.255.0
# ip=254 && for node in {1..16}; do
   rocks add host interface compute-0-${node} ib0 \
     ip=192.168.3.${ip} subnet=ib-cheaha ;
   let ip=${ip}-1 ;
done
Repeat for the next set of nodes
# ip=238 && for node in {1..16}; do
   rocks add host interface compute-1-${node} ib0 \
     ip=192.168.3.${ip} subnet=ib-cheaha ;
   let ip=${ip}-1 ;
done
And so on... Change the sshd_config on the compute nodes to not use DNS. I have found that ssh to compute nodes take close to a minute when this is set to true
# rocks set attr ssh_use_dns false
Synchronize the configuration

# rocks sync config
Now open the firewall the ib0 for all ports and protocols
# rocks open appliance firewall compute \
   network=infiniband service="all" protocol="all"

# rocks sync host firewall compute

# rocks list host firewall compute-0-1
SERVICE PROTOCOL CHAIN ACTION NETWORK   OUTPUT-NETWORK FLAGS                                COMMENT SOURCE
ssh     tcp      INPUT ACCEPT public     -------------- -m state --state NEW                 ------- G     
all     all      INPUT ACCEPT public     -------------- -m state --state RELATED,ESTABLISHED ------- G     
all     all      INPUT ACCEPT infiniband -------------- ------------------------------------ ------- A     
all     all      INPUT ACCEPT private    -------------- ------------------------------------ ------- G     
Hope this helps

Building Mellanox OFED 1.4 for Rocks 5.3

Here are my notes from building Mellanox OFED 1.4 on a Rocks 5.3 x86_64 cluster utilizing CentOS 5.4 and kernel 2.6.18-128-7.1:

1. Download the ISO file MLNX_OFED_LINUX-1.4-rhel5.3.iso from this page

The MLNX_OFED-1.4 comes with modules for kernel 2.6.18-128, we are using 2.6.18-128-7.1, so we need to build new modules.

2. Mount the ISO and copy the contents to a scratch work area

# mount -t iso9660 -o loop /root/MLNX_OFED_LINUX-1.4-rhel5.3.iso /mnt/cdrom 
# mkdir /root/MLNX_OFED-1.4
# cp -r /mnt/cdrom/* /root/MLNX_OFED_LINUX-1.4/
# umount /mnt/cdrom

3. Edit the script so that it will work with CentOS (our centos-release says 5.4, we are still running a 5.3 kernel), this is the script that will build a new ISO file

# cd /root/MLNX_OFED-1.4

TabularUnifieddocs/mlnx_add_kernel_support.sh
Index: docs/mlnx_add_kernel_support.sh
===================================================================
--- docs/mlnx_add_kernel_support.sh.orig 2009-12-17 15:51:46.000000000 -0600
+++ docs/mlnx_add_kernel_support.sh 2009-12-17 15:52:00.000000000 -0600
@@ -279,7 +279,7 @@
         redhat-release-5Server-5.2.0.4)
         distro="rhel5.2"
         ;;
-        redhat-release-5Server-5.3.0.3)
+        redhat-release-5Server-5.3.0.3 | centos-release-5-4.el5.centos.1 )
         distro="rhel5.3"
         ;;
         sles-release-10-15.2)
4. Install some dependencies

# yum -y install libtool tcl-devel libstdc++-devel mkisofs gcc-c++

5. Uninstall some RPM files that will fail to uninstall during the ISO build

/bin/rpm --nodeps -e --allmatches openmpi-libs-1.3.2-2.el5 \
 openmpi-devel-1.3.2-2.el5 rocks-openmpi-1.3.2-1 openmpi-libs-1.3.2-2.el5 \
 openmpi-devel-1.3.2-2.el5 openmpi-1.3.2-2.el5 openmpi-1.3.2-2.el5 \
 openmpi-gnu-1.3.3-1.el5.uabeng

6. Build the new ISO file

# cd /root/MLNX_OFED_LINUX-1.4
# ./docs/mlnx_add_kernel_support.sh -i /root/MLNX_OFED_LINUX-1.4-rhel5.3.iso

Note: This program will create MLNX_OFED_LINUX ISO for rhel5.3 under /tmp directory.
      All Mellanox, OEM, OFED, or Distribution IB packages will be removed.
Do you want to continue?[y/N]:y
Building OFED RPMs...
Removing OFED RPMs...
Running mkisofs...
Created /tmp/MLNX_OFED_LINUX-1.4-rhel5.3.iso

# mv /tmp/MLNX_OFED_LINUX-1.4-rhel5.3.iso /share/apps/mellanox/MLNX_OFED_LINUX-1.4-rhel5.3-kernel-2.6.18_128.7.1.iso

7. Copy the new files from the iso to the NFS share

# mount -t iso9660 -o loop /share/apps/mellanox/MLNX_OFED_LINUX-1.4-rhel5.3-kernel-2.6.18_128.7.1.iso /mnt/cdrom
# cp -r /mnt/cdrom /share/apps/mellanox/MLNX_OFED_LINUX-1.4-rhel5.3-kernel-2.6.18_128.7.1

# cd /share/apps/mellanox
# find ./MLNX_OFED_LINUX-1.4-rhel5.3-kernel-2.6.18_128.7.1 -name kernel-* | grep x86
./MLNX_OFED_LINUX-1.4-rhel5.3-kernel-2.6.18_128.7.1/x86_64/kernel-ib-1.4-2.6.18_128.7.1.el5.x86_64.rpm
./MLNX_OFED_LINUX-1.4-rhel5.3-kernel-2.6.18_128.7.1/x86_64/kernel-ib-1.4-2.6.18_128.el5.x86_64.rpm
./MLNX_OFED_LINUX-1.4-rhel5.3-kernel-2.6.18_128.7.1/x86_64/kernel-ib-devel-1.4-2.6.18_128.el5.x86_64.rpm
./MLNX_OFED_LINUX-1.4-rhel5.3-kernel-2.6.18_128.7.1/x86_64/kernel-ib-devel-1.4-2.6.18_128.7.1.el5.x86_64.rpm

8. Test the installer on the compute node

# cd /share/apps/mellanox/MLNX_OFED_LINUX-1.4-rhel5.3-kernel-2.6.18_128.7.1
# ./mlnxofedinstall --hpc

This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed. 
Do you want to continue?[y/N]:y

Uninstalling the previous version of OFED 

Starting MLNX_OFED_LINUX-1.4 installation ... 

Installing mpi-selector RPM 
Preparing...                ########################################### [100%]
   1:mpi-selector           ########################################### [100%]
Installing kernel-ib RPM 
Preparing...                ########################################### [100%]
   1:kernel-ib              ########################################### [100%]
Installing ib-bonding RPM 
Preparing...                ########################################### [100%]
   1:ib-bonding             ########################################### [100%]
Installing mft RPM 
Preparing...                ########################################### [100%]
   1:mft                    ########################################### [100%]
Install user level RPMs: 
Preparing...                ########################################### [100%]
   1:libibverbs             ########################################### [  2%]
   2:libibcommon            ########################################### [  4%]
   3:libibumad              ########################################### [  6%]
   4:opensm-libs            ########################################### [  8%]
   5:librdmacm              ########################################### [ 10%]
   6:openmpi_intel          ########################################### [ 12%]
   7:libibmad               ########################################### [ 14%]
   8:infiniband-diags       ########################################### [ 16%]
   9:openmpi_gcc            ########################################### [ 18%]
  10:mpitests_openmpi_gcc   ########################################### [ 20%]
  11:mpitests_openmpi_pgi   ########################################### [ 22%]
  12:mpitests_openmpi_intel ########################################### [ 24%]
  13:qperf                  ########################################### [ 26%]
  14:perftest               ########################################### [ 28%]
  15:ibutils                ########################################### [ 30%]
  16:libmthca               ########################################### [ 32%]
  17:libmlx4                ########################################### [ 34%]
  18:openmpi_pgi            ########################################### [ 36%]
  19:mstflint               ########################################### [ 38%]
  20:mlnxofed-docs          ########################################### [ 40%]
  21:ofed-scripts           ########################################### [ 42%]
  22:libibverbs             ########################################### [ 44%]
  23:libibcommon            ########################################### [ 46%]
  24:libibumad              ########################################### [ 48%]
  25:mvapich_intel          ########################################### [ 50%]
  26:opensm-libs            ########################################### [ 52%]
  27:librdmacm              ########################################### [ 54%]
  28:libibcommon-devel      ########################################### [ 56%]
  29:libibumad-devel        ########################################### [ 58%]
  30:libibverbs-devel       ########################################### [ 60%]
  31:librdmacm-utils        ########################################### [ 62%]
  32:opensm                 ########################################### [ 64%]
  33:mvapich_gcc            ########################################### [ 66%]
  34:mpitests_mvapich_gcc   ########################################### [ 68%]
  35:mpitests_mvapich_pgi   ########################################### [ 70%]
  36:mpitests_mvapich_intel ########################################### [ 72%]
  37:mvapich_pgi            ########################################### [ 74%]
  38:libibverbs-utils       ########################################### [ 76%]
  39:librdmacm-devel        ########################################### [ 78%]
  40:librdmacm-devel        ########################################### [ 80%]
  41:opensm-devel           ########################################### [ 82%]
  42:opensm-devel           ########################################### [ 84%]
  43:libibumad-devel        ########################################### [ 86%]
  44:libibcommon-devel      ########################################### [ 88%]
  45:libibverbs-devel       ########################################### [ 90%]
  46:libibmad               ########################################### [ 92%]
  47:libmthca               ########################################### [ 94%]
  48:libmlx4                ########################################### [ 96%]
  49:libibmad-devel         ########################################### [ 98%]
  50:libibmad-devel         ########################################### [100%]
Device (15b3:673c):
        0c:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev a0)
        Link Width: 8x
        Link Speed: 2.5Gb/s


Installation finished successfully. 

The firmware version 2.6.0 is up to date. 
Note: To force firmware update use '--force-fw-update' flag.
Configuring /etc/security/limits.conf. 
warning: /etc/infiniband/openib.conf saved as /etc/infiniband/openib.conf.rpmsave

Tuesday, March 2, 2010

Build and Install Latest Grid Engine Roll for Rocks 5.3

This FAQ discusses the process of building the Sun Grid Engine roll for a Rocks 5.3 hpc cluster.

Why would you want to rebuild the roll? Possibilities include adding a later version of grid engine, or customizing the roll different from that provided by Rocks.

The latest version of Rocks at this time is 5.3 which includes Grid Engine 6.2u4.

The Grid Engine team released 6.2u5 in January. This version contains many new features and bug fixes over 6.2u4.

I built and tested this roll on a Rocks 5.3 x86_64 virtual machine prior to deploying it to the production cluster.

The overview:
  • Download the Rocks source code
  • Download the Grid Engine source code
  • Build the Roll
  • Add the Roll to the head node
  • Update the RPM on the head node
  • Rebuild the compute nodes

Obtain the Rocks Source Code

  1. Download the Rocks source code tree (http://www.rocksclusters.org/roll-documentation/base/5.3/source-access.html) and make a backup of the original sge roll source in case you need to restore or compare files
  2. $ mkdir -p ~/software/rocks-cluster
    $ cd ~/software/rocks-cluster
    $ hg clone http://fyp.rocksclusters.org/hg/rocks-5.3
    
    destination directory: rocks-5.3
    real URL is http://fyp.rocksclusters.org/hg/rocks-5.3/
    requesting all changes
    adding changesets
    adding manifests
    adding file changes
    added 1 changesets with 2815 changes to 2815 files
    2815 files updated, 0 files merged, 0 files removed, 0 files unresolved
  3. Create a new sge roll build area by copying the existing sge roll directory
  4. $ cd ~/software/rocks-cluster/rocks-5.3/src/roll
    $ cp -a sge sge-v62u5
  5. Download the Grid Engine source code (all versions of source are available at the Documents and Files page http://gridengine.sunsource.net/servlets/ProjectDocumentList ). Extract the source so we can later copy the installer script
  6. $ mkdir -p ~/software/ge6.2u5-source
    $ cd ~/software/ge6.2u5-source
    $ wget http://gridengine.sunsource.net/files/documents/7/215/ge-V62u5_TAG-src.tar.gz
    $ tar -zxf ge-V62u5_TAG-src.tar.gz

Build the SGE Roll

  1. cd to the sge roll directory and copy the 6.2u5 source code to the roll src directory
  2. $ cd ~/software/rocks-cluster/rocks-5.3/src/roll/sge-v62u5/
    $ cp ~/software/ge6.2u5-source/ge-V62u5_TAG-src.tar.gz ~/software/rocks-cluster/rocks-5.3/src/roll/sge-v62u5/src/sge/
  3. Edit the roll version info
  4. vi ~/software/rocks-cluster/rocks-5.3/src/roll/sge-v62u5/version.mk
    
    ROLLNAME = sge
    RELEASE = 62u5
    COLOR   = plum
    
    REDHAT.ROOT = $(PWD)
  5. Edit the SGE version info
  6. vi ~/software/rocks-cluster/rocks-5.3/src/roll/sge-v62u5/src/sge/version.mk
    
    NAME            = sge
    VERSION         = V62u5
    RELEASE         = 1
  7. Create a new patch-files directory for V62u5 and copy the aimk SGE installer script and then edit as follows
  8. $ mkdir -p src/sge/patch-files/V62u5/gridengine/source
    $ cp ~/software/ge6.2u5-source/gridengine/source/aimk src/sge/patch-files/V62u5/gridengine/source/
  9. Edit the aimk install script as follows
  10. $ vim src/sge/patch-files/V62u5/gridengine/source/aimk
    
    set KRBLIB            = "-lkrb5 -lz"
  11. Edit the sge-client.xml file if you want to do any customizations to the clients, leave alone to accept the Rocks customizations
  12. Edit the sge-server.xml file to if you want to customize the head node settings (create custom parallel environments, etc...)
  13. Build the Roll (note you have to be ROOT to run this command)
  14. # cd ~me/software/rocks-cluster/rocks-5.3/src/roll/sge-v62u5
    # make roll > make-roll.log 2>&1

Once this command completes, you should have a new ISO file in the build directory named "sge-5.2-62u5.x86_64.disk1.iso".

If you are installing a new Rocks 5.3 cluster, simply burn this ISO to CD and use it when selecting the SGE roll.

If you are upgrading your existing Rocks 5.3 cluster from 6.2u4 to 6.2u5, proceed to the next section.

Update GE on Existing Rocks 5.3 Cluster

Rocks 5.3 ships with GE62u4, following the procedure above we have built a roll to install GE62u5 to a new cluster. This roll can also be used to upgrade GE on an existing 5.3 cluster. I haven't tested or even thought about the idea of using this roll on a Rocks 5.2 cluster, so I really do not know if it would work.

The downside to this approach is that it doesn't allow you to install the newer version 'side by side' with the previous version. So I would recommend testing this on a virtual machine Rocks 5.3 cluster before doing so on the production cluster.

  1. copy the new roll to the head node if it isn't already there
  2. Add the roll using the rocks command. This will add the updated files from the roll that differ from the already installed sge roll
  3. # rocks add roll sge-5.3-1.x86_64.disk1.iso
    # cd /export/rocks/install
    # rocks create distro
  4. Make sure that all running jobs have drained from the system before proceeding by first disabling the queues.
  5. $ qmod -d '*'
  6. Verify via qstat that running jobs have finished
  7. Stop the SGE services on the compute nodes, again be sure that the jobs have stopped otherwise this process will terminate them (repeat for other compute appliance types)
  8. $ sudo rocks run host compute command="/sbin/service sgeexecd.$(hostname -s) stop"
    $ sudo rocks run host verari-compute command="/sbin/service sgeexecd.$(hostname -s) stop"
  9. Stop the SGE master process on the head node
  10. $ sudo /sbin/service sgemaster.$(hostname -s) stop
  11. Backup your /opt/gridengine directory
  12. $ cd /opt
    $ sudo tar -cjf gridengine-backup-62u4.tar.bz2 gridengine
  13. Run the roll to generate an install script. Under normal "Adding a roll" circumstances, you would procede by running the script, since we are upgrading an existing roll, it isn't necesary to do so. In fact, running the script may potentially overwrite some customizations that have been made to the grid engine since install time.

    We will simply examine the script to find out what is necessary to perform the upgrade.
  14. # rocks run roll sge > /tmp/install-sge-6.2u5.sh
  15. DON'T run the script!
  16. Examination of the script reveals that the only RPM that we really need to upgrade is sge-V62u5-1.x86_64.rpm, the rest of the script was already performed when we installed the head node (creation of parallel environments, sge user account, etc...)
  17. Use the loop below to update the rpms, again the only one that will actually update is sge-V62u5-1.x86_64.rpm, but just in case try them all
  18. # for n in $(grep ^rpm /tmp/install-sge-6.2u5.sh |awk '{print $5}'); do echo $n; rpm -Uvh $n; done
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/db4-utils-4.3.29-10.el5.x86_64.rpm
    Preparing...                ########################################### [100%]
    package db4-utils-4.3.29-10.el5.x86_64 is already installed
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/javacc-4.0-3jpp.3.x86_64.rpm
    Preparing...                ########################################### [100%]
    package javacc-4.0-3jpp.3.x86_64 is already installed
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/junit-3.8.2-3jpp.1.x86_64.rpm
    Preparing...                ########################################### [100%]
    package junit-3.8.2-3jpp.1.x86_64 is already installed
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/libXp-devel-1.0.0-8.1.el5.x86_64.rpm
    Preparing...                ########################################### [100%]
    package libXp-devel-1.0.0-8.1.el5.x86_64 is already installed
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/openmotif-2.3.1-2.el5.i386.rpm
    Preparing...                ########################################### [100%]
    package openmotif-2.3.1-2.el5.i386 is already installed
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/openmotif-devel-2.3.1-2.el5.i386.rpm
    Preparing...                ########################################### [100%]
    package openmotif-devel-2.3.1-2.el5.i386 is already installed
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/pam-devel-0.99.6.2-6.el5.i386.rpm
    Preparing...                ########################################### [100%]
    package pam-devel-0.99.6.2-6.el5.i386 is already installed
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/rocks-command-sge-5.3-0.x86_64.rpm
    Preparing...                ########################################### [100%]
    package rocks-command-sge-5.3-0.x86_64 is already installed
    file /opt/rocks/lib/python2.4/site-packages/rocks/commands/remove/host/plugin_sge.pyc from install of rocks-command-sge-5.3-0.x86_64 conflicts with file from package rocks-command-sge-5.3-0.x86_64
    file /opt/rocks/lib/python2.4/site-packages/rocks/commands/remove/host/plugin_sge.pyo from install of rocks-command-sge-5.3-0.x86_64 conflicts with file from package rocks-command-sge-5.3-0.x86_64
    file /opt/rocks/lib/python2.4/site-packages/rocks/commands/report/sge/__init__.pyc from install of rocks-command-sge-5.3-0.x86_64 conflicts with file from package rocks-command-sge-5.3-0.x86_64
    file /opt/rocks/lib/python2.4/site-packages/rocks/commands/report/sge/__init__.pyo from install of rocks-command-sge-5.3-0.x86_64 conflicts with file from package rocks-command-sge-5.3-0.x86_64
    file /opt/rocks/lib/python2.4/site-packages/rocks/commands/report/sge/machines/__init__.pyc from install of rocks-command-sge-5.3-0.x86_64 conflicts with file from package rocks-command-sge-5.3-0.x86_64
    file /opt/rocks/lib/python2.4/site-packages/rocks/commands/report/sge/machines/__init__.pyo from install of rocks-command-sge-5.3-0.x86_64 conflicts with file from package rocks-command-sge-5.3-0.x86_64
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/rocks-sge-5.3-2.x86_64.rpm
    Preparing...                ########################################### [100%]
    package rocks-sge-5.3-2.x86_64 is already installed
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/rocks-sge-5.3-2.x86_64.rpm
    Preparing...                ########################################### [100%]
    package roll-sge-usersguide-5.3-0.x86_64 is already installed
    file /var/www/html/roll-documentation/sge/5.3/index.html from install of roll-sge-usersguide-5.3-0.x86_64 conflicts with file from package roll-sge-usersguide-5.3-0.x86_64
    file /var/www/html/roll-documentation/sge/5.3/rocks-copyright.html from install of roll-sge-usersguide-5.3-0.x86_64 conflicts with file from package roll-sge-usersguide-5.3-0.x86_64
    file /var/www/html/roll-documentation/sge/5.3/roll-sge-usersguide.pdf from install of roll-sge-usersguide-5.3-0.x86_64 conflicts with file from package roll-sge-usersguide-5.3-0.x86_64
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/sge-V62u5-1.x86_64.rpm
    Preparing...                ########################################### [100%]
    1:sge                    ########################################### [100%]
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/sge-drmaa-5.3-0.noarch.rpm
    Preparing...                ########################################### [100%]
    package sge-drmaa-5.3-0.noarch is already installed
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/sge-ganglia-5.3-2.x86_64.rpm
    Preparing...                ########################################### [100%]
    package sge-ganglia-5.3-2.x86_64 is already installed
    file /opt/ganglia/lib64/ganglia/python_modules/sge.pyc from install of sge-ganglia-5.3-2.x86_64 conflicts with file from package sge-ganglia-5.3-2.x86_64
    file /opt/ganglia/lib64/ganglia/python_modules/sge.pyo from install of sge-ganglia-5.3-2.x86_64 conflicts with file from package sge-ganglia-5.3-2.x86_64
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/sge-ganglia-5.3-2.x86_64.rpm
    Preparing...                ########################################### [100%]
    package sge-ganglia-5.3-2.x86_64 is already installed
    file /opt/ganglia/lib64/ganglia/python_modules/sge.pyc from install of sge-ganglia-5.3-2.x86_64 conflicts with file from package sge-ganglia-5.3-2.x86_64
    file /opt/ganglia/lib64/ganglia/python_modules/sge.pyo from install of sge-ganglia-5.3-2.x86_64 conflicts with file from package sge-ganglia-5.3-2.x86_64
    
    /export/rocks/install/rocks-dist/x86_64/RedHat/RPMS/sge-insert-ethers-5.3-0.x86_64.rpm
    Preparing...                ########################################### [100%]
    package sge-insert-ethers-5.3-0.x86_64 is already installed
    file /opt/rocks/var/plugins/insertethers/sge.pyc from install of sge-insert-ethers-5.3-0.x86_64 conflicts with file from package sge-insert-ethers-5.3-0.x86_64
    file /opt/rocks/var/plugins/insertethers/sge.pyo from install of sge-insert-ethers-5.3-0.x86_64 conflicts with file from package sge-insert-ethers-5.3-0.x86_64
  19. Start the SGE master process on the head node
  20. $ sudo /sbin/service sgemaster.$(hostname -s) start
  21. Rebuild the compute nodes
  22. # rocks run host compute '/boot/kickstart/cluster-kickstart'
    # rocks run host verari-compute '/boot/kickstart/cluster-kickstart'
  23. Once all of the compute nodes are rebuilt (and finished rebooting multiple times, i.e. OFED install, lustre, etc...) and you are ready to release the jobs, enable the queues
  24. $ qmod -e '*'
Grid Engine 6.2u5 is now installed and running on the head node and compute nodes.

Tuesday, April 14, 2009

The migration from Rocks to xCAT

Thinking of using an alternative HPC solution to Rocks? Check out xCAT.

xCAT appears to be much more flexible, and less entangled with the underlying operating system (in Rocks case, either Red Hat EL5 or CentOS5). This is appealing in that you can apply core OS updates without the worry that you'll break compatibility with the cluster stack.

There's a long history of discussions on the Rocks mailing list about how to apply core OS updates, and which packages to avoid. Over the releases, Rocks has moved much of the functionality into its own space or commands (for example, old Rocks had a modified /usr/sbin/useradd). Some users now claim that they apply updates to the nodes on a weekly basis without any problems, while others say that it's still possible to break components by applying the updates.

xCAT lives in its own space, under $XCATROOT (I think I have that right). Since xCAT is perl based, your biggest exposure would be updating the core perl to a much newer release, but you'd have to do that outside the normal yum repos.

One of Rocks strengths is that a novice Linux user can install and manage a basic cluster fairly easy, and conversley xCAT appears to have a steep learning curve and certainly requires a good deal of familiarity with Linux in a CLI environment (sorry no GUI yet).

More to come as I get time to play with xCAT.

Wednesday, May 21, 2008

VMware Roll for Rocks 5 Clusters

Rocks 5 (www.rocksclusters.org) released April 30, 2008 using Red Hat EL5 (CentOS5) as the underlying operating system and support for Xen virtual machines.

Adding virtual machines to a computing cluster is a terrific idea. Think about the possibilities from an admin stand point. Say that user jsmith needs his compute nodes configured with Red Hat 7 because his application is old and won't work in current releases. With the virtual machine approach, it's very easy to satisfy this request without having to carve out physical compute nodes for reinstallation.

Not to be out done, Takahiro Hirofuchi of AIST announced that they are working on a VMware Roll for Rocks 5 (they also have plans for a Rock 4.2 version).

I'm a VMware guy and haven't done a whole lot with Xen, so this roll is very appealing.

They list the features as:
The project page is hosted on Code Google (http://code.google.com/p/grivon/wiki/VMwareRollRocks5).