Underworld

a long term geodynamics simulation platform

Compiling and Setting up MPICH 2

Configuring MPICH 2

Navigate into the unzipped directory, and type one of the following to create the makefiles for the build you want, where /usr/local/mpich-* should be changed to the place where mpich 2 is to be installed. Note: you can have each kind of build side-by-side, by building each one seperately, but you cannot combine different builds into one i.e. you cannot create one build that is both optimised and debug. See downloading MPICH for further details and for instructions for downloading MPICH 2.

On a Mac using sock The recommended method for macs to facilitate the communications in Mpich 2 is via a network socket (sock) (all communication goes over TCP sockets).

optimised build: - good if you are not going to be developing a lot of code but are going to be doing a lot of runs

./configure --enable-sharedlibs=osx-gcc --enable-fast --with-device=ch3:sock --prefix=/usr/local/mpich2-optimised

debug build: - for code developing

./configure --enable-sharedlibs=osx-gcc --enable-debug-info --enable-g=all --with-device=ch3:sock --prefix=/usr/local/mpich2-debug

On a Mac

optimised build: - good if you are not going to be developing a lot of code but are going to be doing a lot of runs

./configure --enable-sharedlibs=osx-gcc --enable-fast=all --with-mpe --prefix=/usr/local/mpich2-optimised

debug build: - for code developing

./configure --enable-sharedlibs=osx-gcc --enable-debug-info=all --enable-g=all --with-mpe --prefix=/usr/local/mpich2-debug

When this has finished you should see:

Configuration completed.

On Linux using ssm

The recommended method for linux to facilitate the communications in Mpich 2 is via ssm (sockets and shared memory) (shared memory is used for communication with the same machine, sockets are used for communication between different machines). See downloading MPICH for further details and for instructions for downloading MPICH 2.

optimised build: - good if you are not going to be developing a lot of code but are going to be doing a lot of runs

./configure --enable-sharedlibs=gcc --enable-fast --with-device=ch3:ssm --prefix=/usr/local/mpich2-optimised

debug build: - for code developing

./configure --enable-sharedlibs=gcc --enable-debug-info --enable-g=all --with-device=ch3:ssm --prefix=/usr/local/mpich2-debug

When this has finished you should see:

Configuration completed.

Installing MPICH 2

From within the unzipped directory (for each build type) (root or administrator access may be required; on macs type "sudo make install" in place of the "make install" step):

make
make install 

and mpich 2 will be installed in the directory that was given with the prefix command when you configured. If you want to have more than one build side-by-side, repeat for the build desired.

Setting up Environment Variables

Environment variables must be set, so that the build system knows where to find your MPI implementation, to allow the StGermain-based code Underworld to use it.

You can set these by adding the following to your ~/.bashrc (if you are using bash) or ~/.tcshrc (if you are using tcsh), where /usr/local/mpich2-optimised is the path where the MPI implementation was installed that you want to use when compiling PETSc and Underworld.

Note: if you later decide to compile a copy of Underworld with a different build of Mpich 2 (such as /usr/local/mpich2-debug), you will need to change this in your ~/.bashrc or ~/.tcshrc file and activate the change (see below) prior to compiling PETSc and Underworld.

bash:

export MPI_DIR=/usr/local/mpich2-optimised

To activate this change, save and close your .bashrc file and then either exit your terminal application or type the following:

source ~/.bashrc

tcsh:

setenv MPI_DIR /usr/local/mpich2-optimised

To activate this change, save and close your .tcshrc file and then either exit your terminal application or type the following:

source ~/.tcshrc

Adding MPICH 2 to your Path

You will also need to add the build of Mpich 2, that you are going to want to use when compiling and using Underworld, to your path (see below).

Note: if you later decide to compile a copy of Underworld with a different build of Mpich 2 (such as /usr/local/mpich2-debug), you will need to change the path (make sure that you use a new terminal window after activating the change to ensure that the old path doesn't still apply) prior to compiling PETSc and Underworld.

To view your PATH, type the following:

echo $PATH

This will probably look something like PATH=/usr/bin:/bin:/usr/local/bin, which is a list of ':' separated directories of where commands can be executed from without typing the full path. To add Mpich 2 to your path add the following to your ~/.bashrc or ~/.tcshrc file:

linux

export PATH=$MPI_DIR/bin:$PATH
export LD_LIBRARY_PATH=$MPI_DIR/lib:$LD_LIBRARY_PATH

macs

export PATH=$MPI_DIR/bin:$PATH
export LD_LIBRARY_PATH=$MPI_DIR/lib:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=$LD_LIBRARY_PATH

Setting up and Testing mpd

For Mpich2 (unlike Mpich1), you need to set up mpd, which is a default external process manager that comes with Mpich 2. Further information about using mpd with Mpich 2 is available in mpich2-1.0.5/src/pm/mpd and at http://www-unix.mcs.anl.gov/mpi/mpich2/.

First, create a file in your home directory called .mpd.conf, replacing <secretword> with a word known only to yourself that is different to your normal password, by typing:

cat << EOF >> ~/.mpd.conf
MPD_SECRETWORD=<secretword>
EOF

Make this file readable and writable only by you by typing:

chmod 600 ~/.mpd.conf

To set mpd running from the Mpich 2 bin directory, type:

mpd &

To test your Mpich 2 installation (if you have more than one build of Mpich 2, repeat these steps for each build), navigate into any directory in the Mpich 2 share directory that starts with "examples" (the names may vary) (e.g. /usr/local/mpich2-optimised/share/examples_logging) and then type:

make

then look for any executable and try to run it as follows (using here the executable cpilog as an example, again the file names may vary), and check that a result is returned in the terminal, without any errors:

serial:

./cpilog 

parallel:

../../bin/mpiexec -n 4 ./cpilog

which should return four (from -n 4) lines something like:

time taken by 1X0 MPI_Bcast() at rank 0 = 0.000002
time taken by 1X0 MPI_Bcast() at rank 2 = 0.000002
time taken by 1X0 MPI_Bcast() at rank 1 = 0.000002
time taken by 1X0 MPI_Bcast() at rank 3 = 0.000002

creation / use of dynamic libraries: (mac only)

otool -L ./cpilog

and check that one of the lines returned includes "libmpich.dylib".

If either of these tests fail, and / or you would like more information on configuration options or testing your copy of Mpich2, please refer to the Mpich2 user documentation supplied with the Mpich2 source, or at http://www-unix.mcs.anl.gov/mpi/mpich2/.

You have now completed MPI Implementation.

-- Wendy Mason - 13 Nov 2008
-- Robert Turnbull
-- Patrick Sunter
-- Steve Quenette