Friday, April 19, 2013

How To Compile Gromacs 4.6.1 with OpenMPI 1.6.4 on CentOS

I took the following notes while installing Gromacs 4.6.1 recently. Hopefully others will find them useful. The software was compiled and installed on a CentOS 5.9 x86_64 containing QDR Infiniband. The system uses SGE (Grid Engine) as it's scheduler. Intel compilers 13.1.1 were used to compile both OpenMPI and Gromacs.

Intel Compilers

I used version 13.1.1 of the Intel compilers. I'm including the module file here for reference 
#%Module

set ver 13.1.1
set arch intel64

set root /share/apps/intel/parallel_studio_xe_2013_update3/composer_xe_2013.3.163
set ccroot /opt/intel/Compiler/$ver/$ccver
set fcroot /opt/intel/Compiler/$ver/$fcver
set mklroot $root/mkl

set msg "This module adds Intel compilers v$ver to various paths"

proc ModulesHelp { } {
   puts stderr $msg
}

module-whatis   $msg

setenv MKLROOT $mklroot
prepend-path MANPATH $root/man/en_US
prepend-path INTEL_LICENSE_FILE $root/licenses

prepend-path NLSPATH $root/compiler/lib/$arch/locale/en_US:$root/ipp/lib/$arch/locale/en_US:$root/mkl/lib/$arch/locale/en_US:$root/debugger/lib/$arch/locale/en_US

prepend-path LIBRARY_PATH $root/compiler/lib/$arch:$root/ipp/lib/$arch:$root/mkl/lib/$arch:$root/tbb/lib/$arch/gcc4.1
prepend-path MIC_LD_LIBRARY_PATH $root/compiler/lib/mic:$root/mkl/lib/mic:$root/tbb/lib/mic
prepend-path LD_LIBRARY_PATH $root/compiler/lib/$arch:$root/mpirt/lib/$arch:$root/ipp/lib/$arch:$root/mkl/lib/$arch:$root/tbb/lib/$arch/gcc4.1
prepend-path PATH $root/bin/$arch:$root/mpirt/bin/$arch:$root/bin/${arch}_mic:$root/debugger/gui/$arch

prepend-path CPATH $root/mkl/include:$root/tbb/include
prepend-path INCLUDE $root/mkl/include

prepend-path IPPROOT $root/ipp
prepend-path TBBROOT $root/tbb
prepend-path IDB_HOME $root/bin/$arch

Build OpenMPI 1.6.4

First, build the latest OpenMPI with the Intel compilers. This version of OpenMPI will reside in the Gromacs 4.6.1 install path.
  1. Create the install base directory 
    cd /share/apps/gromacs/4.6.1/build
wget http://www.open-mpi.org/software/ompi/v1.6/downloads/openmpi-1.6.4.tar.bz2
  2. Create the build script for Intel compilers: /share/apps/gromacs/4.6.1/build/build-openmpi-intel.sh 
    #!/bin/sh
. /etc/profile.d/modules.sh
ver=1.6
rel=1.6.4
grover=4.6.1
src="openmpi-${rel}.tar.bz2"
url="http://www.open-mpi.org/software/ompi/v${ver}/downloads/${src}"
compiler="intel"
basedir="/share/apps/gromacs/${grover}/openmpi"
prefix="${basedir}/${rel}-${compiler}"

if [ -d $prefix ]; then
  echo "Installation directory already exists, exiting to prevent overwrite: $prefix";
  exit 1
else
  echo "Verified installation directory does not exist: $prefix"
fi

if [ ! -f $src ]; then
  echo "Downloading source code from: $url";
  wget $url;
else
  echo "Using existing source code: $src";
fi

if [ -d "openmpi-${rel}" ]; then
  echo "Removing existing build directory ./openmpi-${rel}"
  rm -rf openmpi-${rel};
fi

echo "Extracting $src"
tar -jxf $src
cd openmpi-${rel}

echo "Building OpenMPI $rel for Intel compilers"
echo "Running configure"
module purge
module load intel/intel-compilers-13.1.1
CC=icc CXX=icpc F77=ifort FC=ifort ./configure --with-sge --with-openib --prefix=$prefix --enable-static --enable-shared

make all install
module purge

cd ..
  3. Run the compile and install 
    ./build-openmpi-intel.sh | tee openmpi-1.6.4-intel.log
  4. Create the module file (based off of [http://www.levlafayette.com/node/145 this example]) in the Gromacs base directory 
    mkdir -p /share/apps/gromacs/4.6.1/modulefiles/gromacs-openmpi-intel
vim /share/apps/gromacs/4.6.1/modulefiles/gromacs-openmpi-intel/1.6.4

    #%Module1.0#####################################################################
##
## $name modulefile
##
# /share/apps/gromacs/4.6.1/openmpi/1.6.4-intel/bin
# /share/apps/gromacs/4.6.1/openmpi/gromacs-openmpi-intel/1.6.4-intel/bin
set ver [lrange [split [ module-info name ] / ] 1 1 ]
set name [lrange [split [ module-info name ] / ] 0 0 ]
set loading [module-info mode load]
set subname [lrange [split $name - ] 0 0 ]
#set compiler [lrange [split $name - ] 1 1 ]
set compiler "intel"
set compilerver "13.1.1"

proc ModulesHelp { } {
  puts stderr "\tThis module sets the envinronment for $name v$ver compiled\n\twith Intel v13.1.1 compilers"
}

module-whatis   "Set environment variables to use $name version $ver"

if { $loading && ![ is-loaded $compiler/$compilerver ] } {
  module load $compiler/$compilerver
}

set basedir /share/apps/gromacs/4.6.1/openmpi/${ver}-$compiler

prepend-path --delim " " CPPFLAGS -I$basedir/include
prepend-path --delim " " LDFLAGS -L$basedir/lib

prepend-path LD_LIBRARY_PATH $basedir/lib
prepend-path MANPATH $basedir/share/man
prepend-path PATH $basedir/bin

setenv  MPI_DIR         $basedir/
setenv  MPI_HOME        $basedir/
setenv  OPENMPI_ROOT    $basedir/
  5. Run a test
    • Create the input C file /share/apps/openmpi/build/mpi-helloworld.c 
      #include <stdio>
#include <string>
#include <stddef>
#include <stdlib>
#include "mpi.h"

main(int argc, char **argv ) {
    int rank, size;
    int buflen = 512;
    char name[buflen];
    
    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD,&size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    gethostname(name, buflen);
    
    printf( "P rank %d of %d, host %s\n", rank, size, name);

    MPI_Finalize();
}
    • Compile mpi-helloworld.c 
      module load /share/apps/gromacs/4.6.1/modulefiles/gromacs-openmpi-intel/1.6.4
mpicc -o mpi-helloworld-intel mpi-helloworld.c
    • run it as a job 
      module purge
module load /share/apps/gromacs/4.6.1/modulefiles/gromacs-openmpi-intel/1.6.4
qsub -V -b y \
   -l h_rt=00:15:00 \
   -l vf=256M \
   -j y \
   -pe openmpi\* 8 \
   mpirun -n 8 /share/apps/openmpi/build/mpi-helloworld-intel

Build Gromacs 4.6.1

Official Gromacs 4.6 build instructions They have changed dramatically from versions 4.5 and prior. One requirement listed is that we need CMake version 2.8 or later. EPEL provides CMake 2.6.4, so we'll need to build CMake from scratch.
  1. Download the latest CMake from here: http://www.cmake.org/cmake/resources/software.html 
    sudo mkdir /share/apps/cmake
sudo chown mhanby:atlab /share/apps/cmake
cd /share/apps/cmake
wget http://www.cmake.org/files/v2.8/cmake-2.8.10.2-Linux-i386.tar.gz
tar -zxf cmake-2.8.10.2-Linux-i386.tar.gz
  2. Create the build directory and download the source 
    mkdir -p /share/apps/gromacs/4.6.1/build
cd /share/apps/gromacs/4.6.1/build
wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-4.6.1.tar.gz
tar -zxf gromacs-4.6.1.tar.gz 
mkdir gromacs-4.6.1/build
cd gromacs-4.6.1/build
  3. Create the module files 
    mkdir /share/apps/modulefiles/gromacs-intel
vim /share/apps/modulefiles/gromacs-intel/4.6.1

    #%Module

set ver 4.6.1
set app "Gromacs"
set url "http://gromacs.org/"
set approot /share/apps/gromacs/$ver
set msg "This module adds $app v$ver to various paths\nWebsite: $url\n\n"

proc ModulesHelp { } {
   puts stderr $msg
}

module-whatis  $msg

module use --append $approot/modulefiles
module load intel/13.1.1
#module load $approot/modulefiles/gromacs-openmpi-intel/1.6.4
module load gromacs-openmpi-intel/1.6.4

setenv GROMACSHOME $approot
#setenv MKL_HOME $::env(MKLROOT)
prepend-path PATH $approot/bin
prepend-path LD_LIBRARY_PATH $approot/lib
prepend-path MANPATH $approot/man
prepend-path MODULEPATH $approot/modulefiles
prepend-path PKG_CONFIG_PATH $approot/lib/pkgconfig
  4. The next steps will build single and double precision in serial and parallel
    • Serial single precision
      1. Run cmake to configure the build 
        module purge
module load gromacs-intel/4.6.1
export PATH=/share/apps/cmake/cmake-2.8.10.2-Linux-i386/bin:$PATH

cd /share/apps/gromacs/4.6.1/build/gromacs-4.6.1/build
rm -rf *

cmake .. \
   -DCMAKE_C_COMPILER=icc \
   -DCMAKE_CXX_COMPILER=icpc \
   -DGMX_BUILD_OWN_FFTW=ON \
   -DCMAKE_INSTALL_PREFIX=/share/apps/gromacs/4.6.1 \
   -DREGRESSIONTEST_DOWNLOAD=ON
      2. View the configuration 
        ccmake ..

 BUILD_SHARED_LIBS                ON
 CMAKE_BUILD_TYPE                 Release
 CMAKE_INSTALL_PREFIX             /share/apps/gromacs/4.6.1
 CMAKE_PREFIX_PATH
 CUDA_HOST_COMPILER               /share/apps/intel/parallel_studio_xe_2013_update3/composer_xe_2013.3.163/bin/intel64/icc
 GMX_CPU_ACCELERATION             SSE4.1
 GMX_DEFAULT_SUFFIX               ON
 GMX_DOUBLE                       OFF
 GMX_FFT_LIBRARY                  fftw3
 GMX_GPU                          OFF
 GMX_GSL                          OFF
 GMX_MPI                          OFF
 GMX_OPENMP                       ON
 GMX_QMMM_PROGRAM                 none
 GMX_THREAD_MPI                   ON
 GMX_X11                          OFF
 REGRESSIONTEST_DOWNLOAD          OFF
      3. make 
        make
      4. Run the tests 
        make check
[ 70%] Built target gmx
[ 71%] Built target gmxfftw
[ 80%] Built target md
[ 92%] Built target gmxana
[ 92%] Built target editconf
[ 96%] Built target gmxpreprocess
[ 97%] Built target grompp
[ 97%] Built target pdb2gmx
[ 97%] Built target gmxcheck
[100%] Built target mdrun
[100%] Built target gmxtests
Test project /share/apps/gromacs/4.6.1/build/gromacs-4.6.1/build
    Start 1: regressiontests/simple
1/5 Test #1: regressiontests/simple ...........   Passed   72.17 sec
    Start 2: regressiontests/complex
2/5 Test #2: regressiontests/complex ..........   Passed  868.76 sec
    Start 3: regressiontests/kernel
3/5 Test #3: regressiontests/kernel ...........   Passed  583.48 sec
    Start 4: regressiontests/freeenergy
4/5 Test #4: regressiontests/freeenergy .......   Passed  695.84 sec
    Start 5: regressiontests/pdb2gmx
5/5 Test #5: regressiontests/pdb2gmx ..........   Passed  379.29 sec

100% tests passed, 0 tests failed out of 5

Total Test time (real) = 2599.69 sec
[100%] Built target check
      5. Install 
        make install

...
-- Installing: /share/apps/gromacs/4.6.1/lib/pkgconfig/libgmxana.pc
-- Installing: /share/apps/gromacs/4.6.1/bin/GMXRC
-- Installing: /share/apps/gromacs/4.6.1/bin/GMXRC.bash
-- Installing: /share/apps/gromacs/4.6.1/bin/GMXRC.zsh
-- Installing: /share/apps/gromacs/4.6.1/bin/GMXRC.csh
-- Installing: /share/apps/gromacs/4.6.1/bin/completion.csh
-- Installing: /share/apps/gromacs/4.6.1/bin/completion.bash
-- Installing: /share/apps/gromacs/4.6.1/bin/completion.zsh
-- Installing: /share/apps/gromacs/4.6.1/bin/demux.pl
-- Installing: /share/apps/gromacs/4.6.1/bin/xplor2gmx.pl
    • Serial double precision (will append _d to applicable installed files, example, mdrun_d)
      1. Run cmake to configure the build 
        module purge
module load gromacs-intel/4.6.1
export PATH=/share/apps/cmake/cmake-2.8.10.2-Linux-i386/bin:$PATH

cd /share/apps/gromacs/4.6.1/build/gromacs-4.6.1/build
rm -rf *

cmake .. \
   -DCMAKE_C_COMPILER=icc \
   -DCMAKE_CXX_COMPILER=icpc \
   -DGMX_BUILD_OWN_FFTW=ON \
   -DGMX_DOUBLE=ON \
   -DCMAKE_INSTALL_PREFIX=/share/apps/gromacs/4.6.1 \
   -DREGRESSIONTEST_DOWNLOAD=ON
      2. View the configuration 
        ccmake ..

 BUILD_SHARED_LIBS                ON
 CMAKE_BUILD_TYPE                 Release
 CMAKE_INSTALL_PREFIX             /share/apps/gromacs/4.6.1
 CMAKE_PREFIX_PATH
 GMX_CPU_ACCELERATION             SSE4.1
 GMX_DEFAULT_SUFFIX               ON
 GMX_DOUBLE                       ON
 GMX_FFT_LIBRARY                  fftw3
 GMX_GPU                          OFF
 GMX_GSL                          OFF
 GMX_MPI                          OFF
 GMX_OPENMP                       ON
 GMX_QMMM_PROGRAM                 none
 GMX_THREAD_MPI                   ON
 GMX_X11                          OFF
 REGRESSIONTEST_DOWNLOAD          OFF
      3. make 
        make
      4. Run the tests 
        make check

Test project /share/apps/gromacs/4.6.1/build/gromacs-4.6.1/build
    Start 1: regressiontests/simple
1/5 Test #1: regressiontests/simple ...........   Passed    3.33 sec
    Start 2: regressiontests/complex
2/5 Test #2: regressiontests/complex ..........   Passed    6.80 sec
    Start 3: regressiontests/kernel
3/5 Test #3: regressiontests/kernel ...........   Passed   52.89 sec
    Start 4: regressiontests/freeenergy
4/5 Test #4: regressiontests/freeenergy .......   Passed    9.19 sec
    Start 5: regressiontests/pdb2gmx
5/5 Test #5: regressiontests/pdb2gmx ..........   Passed   22.82 sec

100% tests passed, 0 tests failed out of 5

Total Test time (real) =  95.08 sec
[100%] Built target check
      5. Install 
        make install
    • Parallel single precision (will append _mpi to applicable installed files, example, mdrun_mpi)
      1. Run cmake to configure the build 
        module purge
module load gromacs-intel/4.6.1
export PATH=/share/apps/cmake/cmake-2.8.10.2-Linux-i386/bin:$PATH

cd /share/apps/gromacs/4.6.1/build/gromacs-4.6.1/build
rm -rf *

cmake .. \
   -DCMAKE_C_COMPILER=icc \
   -DCMAKE_CXX_COMPILER=icpc \
   -DGMX_BUILD_OWN_FFTW=ON \
   -DGMX_MPI=ON \
   -DCMAKE_INSTALL_PREFIX=/share/apps/gromacs/4.6.1 \
   -DREGRESSIONTEST_DOWNLOAD=ON
      2. View the configuration 
        ccmake ..

 BUILD_SHARED_LIBS                ON
 CMAKE_BUILD_TYPE                 Release
 CMAKE_INSTALL_PREFIX             /share/apps/gromacs/4.6.1
 CMAKE_PREFIX_PATH
 CUDA_HOST_COMPILER               /share/apps/intel/parallel_studio_xe_2013_update3/composer_xe_2013.3.163/bin/intel64/icc
 GMX_CPU_ACCELERATION             SSE4.1
 GMX_DEFAULT_SUFFIX               ON
 GMX_DOUBLE                       OFF
 GMX_FFT_LIBRARY                  fftw3
 GMX_GPU                          OFF
 GMX_GSL                          OFF
 GMX_MPI                          ON
 GMX_OPENMP                       ON
 GMX_QMMM_PROGRAM                 none
 GMX_THREAD_MPI                   OFF
 GMX_X11                          OFF
 MPI_EXTRA_LIBRARY                /usr/lib64/librdmacm.so;/usr/lib64/libibverbs.so;/usr/lib64/librt.so;/usr/lib64/libnsl.so;/usr/lib64/libutil.so;/usr/lib64/libdl.so;/usr/lib64/libm.so;/usr/lib64/librt.so;
 MPI_LIBRARY                      /share/apps/gromacs/4.6.1/openmpi/1.6.4-intel/lib/libmpi.so
 REGRESSIONTEST_DOWNLOAD          OFF
      3. make 
        make
      4. Run the tests 
        make check

Test project /share/apps/gromacs/4.6.1/build/gromacs-4.6.1/build
    Start 1: regressiontests/simple
1/5 Test #1: regressiontests/simple ...........   Passed   22.68 sec
    Start 2: regressiontests/complex
2/5 Test #2: regressiontests/complex ..........   Passed   29.49 sec
    Start 3: regressiontests/kernel
3/5 Test #3: regressiontests/kernel ...........   Passed  222.08 sec
    Start 4: regressiontests/freeenergy
4/5 Test #4: regressiontests/freeenergy .......   Passed   19.17 sec
    Start 5: regressiontests/pdb2gmx
5/5 Test #5: regressiontests/pdb2gmx ..........   Passed   75.39 sec

100% tests passed, 0 tests failed out of 5

Total Test time (real) = 368.83 sec
[100%] Built target check
      5. Install 
        make install
    • Parallel double precision (will append _mpi_d to applicable installed files, example, mdrun_mpi_d)
      1. Run cmake to configure the build 
        module purge
module load gromacs-intel/4.6.1
export PATH=/share/apps/cmake/cmake-2.8.10.2-Linux-i386/bin:$PATH

cd /share/apps/gromacs/4.6.1/build/gromacs-4.6.1/build
rm -rf *

cmake .. \
   -DCMAKE_C_COMPILER=icc \
   -DCMAKE_CXX_COMPILER=icpc \
   -DGMX_BUILD_OWN_FFTW=ON \
   -DGMX_MPI=ON \
   -DGMX_DOUBLE=ON \
   -DCMAKE_INSTALL_PREFIX=/share/apps/gromacs/4.6.1 \
   -DREGRESSIONTEST_DOWNLOAD=ON
      2. View the configuration 
        ccmake ..

 BUILD_SHARED_LIBS                ON 
 CMAKE_BUILD_TYPE                 Release
 CMAKE_INSTALL_PREFIX             /share/apps/gromacs/4.6.1
 CMAKE_PREFIX_PATH
 GMX_CPU_ACCELERATION             SSE4.1
 GMX_DEFAULT_SUFFIX               ON
 GMX_DOUBLE                       ON
 GMX_FFT_LIBRARY                  fftw3
 GMX_GPU                          OFF
 GMX_GSL                          OFF
 GMX_MPI                          ON
 GMX_OPENMP                       ON
 GMX_QMMM_PROGRAM                 none
 GMX_THREAD_MPI                   OFF
 GMX_X11                          OFF
 MPI_EXTRA_LIBRARY                /usr/lib64/librdmacm.so;/usr/lib64/libibverbs.so;/usr/lib64/librt.so;/usr/lib64/libnsl.so;/usr/lib64/libutil.so;/usr/lib64/libdl.so;/usr/lib64/libm.so;/usr/lib64/librt.so;/usr/lib64/libnsl.so;/usr/lib64/libutil.so                       
 MPI_LIBRARY                      /share/apps/gromacs/4.6.1/openmpi/1.6.4-intel/lib/libmpi.so
 REGRESSIONTEST_DOWNLOAD          OFF
      3. make 
        make
      4. Run the tests 
        make check

Test project /share/apps/gromacs/4.6.1/build/gromacs-4.6.1/build
    Start 1: regressiontests/simple
1/5 Test #1: regressiontests/simple ...........   Passed   22.65 sec
    Start 2: regressiontests/complex
2/5 Test #2: regressiontests/complex ..........   Passed   32.25 sec
    Start 3: regressiontests/kernel
3/5 Test #3: regressiontests/kernel ...........   Passed  232.23 sec
    Start 4: regressiontests/freeenergy
4/5 Test #4: regressiontests/freeenergy .......   Passed   21.13 sec
    Start 5: regressiontests/pdb2gmx
5/5 Test #5: regressiontests/pdb2gmx ..........   Passed   82.39 sec

100% tests passed, 0 tests failed out of 5

Total Test time (real) = 390.67 sec
[100%] Built target check
      5. Install 
        make install
Posted by at 11:14 AM
Labels: , , , ,

3 comments:

Anonymous said...

FYI: in my experience, gcc 4.7 generates faster code than Intel 12 and 13 (the latter seems to be even slightly slower than the former).

April 19, 2013 at 12:57 PM
Unknown said...

Thanks for the reply, I'll compile a GCC 4.7 version and have one of our Gromacs users do a benchmark test.

That's good news, score one for open source :-)

April 19, 2013 at 4:09 PM
Unknown said...

Updated the post with new SyntaxHighlighter support:
http://alexgorbatchev.com/SyntaxHighlighter/

June 13, 2013 at 10:43 AM

Post a Comment

Subscribe to: Post Comments (Atom)