Underworld

a long term geodynamics simulation platform

MPICH Download and Installation

What is MPICH and do I need to install it?

StGermain-based codes such as Underworld require an implementation of the MPI-1 or MPI-2 standard for parallel message-passing, even if you only intend to run the code in serial. Underworld is compatible with Mpich 2, which is suitable for "all flavours of UNIX" and Macs.

Further information on MPICH is available here, including documentation and manual pages.

Downloading the MPICH source code

To download the Mpich 2 source code for "all flavours of UNIX" including macs, go here and look for a file like mpich2-1.4.1p1.tar.gz.

Untar and unzip the file (root / administrator access may be required) (if you are on a mac this may be done for you):

tar zxovf mpich2-1.4.1p1.tar.gz

Configuring MPICH 2

There are two ways (for most systems) to facilitate the communications in both versions of mpich: via a network socket or via shared memory. The former is the most portable and the only way that works for ethernet based clusters. The latter suits multiprocessors, but cannot be used on macs. Both can be used on single processors that will fake having multiple processors, however, the latter is a much faster implementation.

On linux, mpich 1 required you to select between the two, but with Mpich2 (on the linux OS only) you can select ssm (sockets and shared memory), which does both (shared memory is used for communication with the same machine, sockets are used for communication between different machines).

gforker is recommended over mpd, when running in serial (not parallel); the --with-device (linux) and --with-pm (mac and linux) flags are for machines with shared memory and explicitly exclude mpd, ensuring that gforker is used and mpd is not needed to be run as a daemon (when running Underworld with MPI you will be able to just run mpiexec without starting mpd first).

Navigate into the unzipped directory, and type one of the following to create the makefiles for the build you want, where /usr/local/mpich-* should be changed to the place where mpich 2 is to be installed. Note: You can have each kind of build side-by-side, by building each one seperately, but you cannot combine different builds into one i.e. you cannot create one build that is both optimised and debug.

optimised build: good if you are not going to be developing a lot of code but are going to be doing a lot of runs

linux, gforker:

./configure --enable-shared --enable-sharedlibs=gcc 
--enable-fast 
--with-device=ch3:nemesis 
--with-pm=gforker 
--prefix=/usr/local/mpich2-optimised

linux, mpd:

./configure --enable-shared --enable-sharedlibs=gcc 
--enable-fast 
--with-device=ch3:ssm 
--prefix=/usr/local/mpich2-optimised

macs, gforker:

./configure --enable-shared --enable-sharedlibs=osx-gcc 
--enable-fast=all 
--with-mpe 
--with-pm=gforker 
--prefix=/usr/local/mpich2-optimised

macs, mpd:

./configure --enable-shared --enable-sharedlibs=osx-gcc 
--enable-fast=all 
--with-mpe 
--prefix=/usr/local/mpich2-optimised

debug build: for code developing

linux, gforker:

./configure --enable-shared --enable-sharedlibs=gcc 
--enable-debug-info 
--enable-g=all 
--with-device=ch3:nemesis 
--with-pm=gforker 
--prefix=/usr/local/mpich2-debug

linux, mpd:

./configure --enable-shared --enable-sharedlibs=gcc 
--enable-debug-info 
--enable-g=all 
--with-device=ch3:ssm 
--prefix=/usr/local/mpich2-debug

macs, gforker:

./configure --enable-shared --enable-sharedlibs=osx-gcc 
--enable-debug-info=all 
--enable-g=all 
--with-mpe 
--with-pm=gforker 
--prefix=/usr/local/mpich2-debug

macs mpd:

./configure --enable-shared --enable-sharedlibs=osx-gcc 
--enable-debug-info=all 
--enable-g=all 
--with-mpe 
--prefix=/usr/local/mpich2-debug

When configuration has finished you should see:

Configuration completed.

Installing MPICH 2

From within the unzipped directory (for each build type) (root or administrator access may be required; on macs type "sudo make install" in place of the "make install" step):

make

make install

and mpich 2 will be installed in the directory that was given with the prefix command when you configured. If you want to have more than one build side-by-side, repeat the steps for the build desired.

Setting up Environment Variables

Environment variables must be set, so that the build system knows where to find your MPI implementation's parent directory, to allow the StGermain-based code Underworld to use it.

You can set these by adding the following to your ~/.bashrc (if you are using bash) or ~/.tcshrc (if you are using tcsh), where /usr/local/mpich2-optimised is the path where the MPI implementation was installed that you want to use when compiling PETSc and Underworld.

Note: if you later decide to compile a copy of Underworld with a different build of Mpich 2 (such as /usr/local/mpich2-debug), you will need to change this in your ~/.bashrc or ~/.tcshrc file and activate the change (see below) prior to compiling PETSc and Underworld.

bash:

export MPI_DIR=/usr/local/mpich2-optimised

To activate this change, save and close your .bashrc file and then either exit your terminal application or type the following:

source ~/.bashrc

tcsh:

setenv MPI_DIR /usr/local/mpich2-optimised

To activate this change, save and close your .tcshrc file and then either exit your terminal application or type the following:

source ~/.tcshrc

Adding MPICH 2 to your Path

You will also need to add the build of Mpich 2, that you are going to want to use when compiling and using Underworld, to your path (see below).

Note: if you later decide to compile a copy of Underworld with a different build of Mpich 2 (such as /usr/local/mpich2-debug), you will need to change the path (make sure that you use a new terminal window after activating the change to ensure that the old path doesn't still apply) prior to compiling PETSc and Underworld.

To view your PATH, type the following:

echo $PATH

This will probably look something like PATH=/usr/bin:/bin:/usr/local/bin, which is a list of ':' separated directories of where commands can be executed from without typing the full path. To add Mpich 2 to your path add the following to your ~/.bashrc or ~/.tcshrc file:

linux:

export PATH=$MPI_DIR/bin:$PATH

export LD_LIBRARY_PATH=$MPI_DIR/lib:$LD_LIBRARY_PATH

macs:

export PATH=$MPI_DIR/bin:$PATH

export LD_LIBRARY_PATH=$MPI_DIR/lib:$LD_LIBRARY_PATH

export DYLD_LIBRARY_PATH=$LD_LIBRARY_PATH

Setting up and Testing mpd

If you configured Mpich2 to use mpd, you need to set up mpd, which is a default external process manager that comes with Mpich 2. Further information about using mpd with Mpich 2 is available in mpich2-1.4.1p1/src/pm/mpd and here.

First, create a file in your home directory called .mpd.conf, replacing <secretword> with a word known only to yourself that is different to your normal password, by typing:

cat << EOF >> ~/.mpd.conf

MPD_SECRETWORD=<secretword>

EOF

Make this file readable and writable only by you by typing:

chmod 600 ~/.mpd.conf

To set mpd running from the Mpich 2 bin directory, type:

mpd &

To test your Mpich 2 installation (if you have more than one build of Mpich 2, repeat these steps for each build), navigate into any directory in the Mpich 2 share directory that starts with "examples" (the names may vary) (e.g. /usr/local/mpich2-optimised/share/examples_logging) and then type:

make

then look for any executable and try to run it as follows (using here the executable cpilog as an example, again the file names may vary), and check that a result is returned in the terminal, without any errors:

serial:

./cpilog

parallel:

../../bin/mpiexec -n 4 ./cpilog

which should return four (from -n 4) lines something like:

time taken by 1X0 MPI_Bcast() at rank 0 = 0.000002

time taken by 1X0 MPI_Bcast() at rank 2 = 0.000002

time taken by 1X0 MPI_Bcast() at rank 1 = 0.000002

time taken by 1X0 MPI_Bcast() at rank 3 = 0.000002

creation / use of dynamic libraries: (mac only)

otool -L ./cpilog

and check that one of the lines returned includes "libmpich.dylib".

If either of these tests fail, and / or you would like more information on configuration options or testing your copy of Mpich2, please refer to the Mpich2 user documentation supplied with the Mpich2 source, or from here.

You have now completed MPI Implementation.

-- Wendy Mason - 28 October 2010
-- Steve Quenette - 14 October 2010
-- Alan Lo - 18 Jan 2006
-- Robert Turnbull
-- Patrick Sunter
-- Steve Quenette