Underworld

a long term geodynamics simulation platform

Compiling PETSc 2.3.2 and above

The following instructions detail how to compile the recent versions of PETSc (e.g. 2.3.3), set up environment variables and add the dynamic libraries to your path, for 'all flavours of UNIX' including macs. Note: Underworld-1.2.0 will not work with PETSc 3.0.

Locating the Unzipped PETSc 2.3.3 Source Directory and Deciding on an Installation Method

There are two ways you can install PETSc 2.3.3, as it now offers a "make install" step:

1. move the unzipped source directory to the location in which you want to install PETSc 2.3.3, and build PETSc into the unzipped directory (as was necessary for previous versions of PETSc); different builds of PETSc (such as optimised and debug) can live side-by-side within the same directory, and will be differentiated by a library subdirectory for each build
2. locate the unzipped source directory anywhere you like and build PETSc into a different directory in the location of your choice; different builds of PETSc (such as optimised and debug) can live side-by-side in their own different directories

Note: You can only compile one build type (e.g. optimised or debug) at a time. If you want more than one build, follow all steps prior to "Setting up Your Environment Variables", repeat for the second build and then set up the environment variables for the build that you are going to compile Underworld with.

Setting the PETSc Directory and Your Machine Architecture with which to compile PETSc 2.3.3

In order to compile PETSc 2.3.3, it needs to be told where the unzipped (source) directory is located (irrespective of where you are going to install it) and the architecture of your machine for the build type you are going to compile (e.g. optimised or debug).

Note: if you do not know your machine's architecture (on linux you can try "linux"), PETSc will tell you after you configure, in which case just set PETSC_DIR now and set PETSC_ARCH after the configure step below.

You can just set these temporarily from the command line for the terminal window you are using in the current session as follows. Navigate into the unzipped source directory and type (where pwd means "print name of current/working directory" and the example machine architecture darwin8.7.1-c is used):

bash:

optimised - good if you are not going to be developing a lot of code but are going to be doing a lot of runs

export PETSC_DIR=`pwd`
export PETSC_ARCH=darwin8.7.1-c-opt

debug - for code developing

export PETSC_DIR=`pwd`
export PETSC_ARCH=darwin8.7.1-c-debug

tcsh:

optimised - good if you are not going to be developing a lot of code but are going to be doing a lot of runs

setenv PETSC_DIR `pwd`
setenv PETSC_ARCH darwin8.7.1-c-opt

debug - for code developing

setenv PETSC_DIR `pwd`
setenv PETSC_ARCH darwin8.7.1-c-debug

Configuring PETSc 2.3.3

Prior to configuring PETSc 2.3.3, please ensure that you have installed an MPI implementation and set up its environment variables (if applicable):

• on linux: Mpich 1 or Mpich 2
• on macs: Mpich 2 for copies of Underworld downloaded on or after 17 January 2007, or OpenMPI for earlier copies of Underworld.

A PETSc instruction manual for installation is available. The following is a summary of the procedure.

From within the unzipped petsc-2.3.3 source directory, type one of the following, noting:

• the MPI implementation used in these examples is Mpich 2, installed in /usr/local (if you only have one MPI implementation built and have set its MPI_DIR environment variable in your .bashrc or .tcshrc, you can replace /usr/local/mpich2 with ${MPI_DIR})
• if you didn't compile fortran into your MPI implementation then the PETSc configure will ask you to add the flag --with-fc=0

optimised build:

• good if you are not going to be developing a lot of code but are going to be doing a lot of runs
• use an optimised build of your MPI implementation, in this example /usr/local/mpich2-optimised
• if you want to compile this build into a directory other than the unzipped source directory, add the flag --prefix=/usr/local/petsc2-optimised to the end of the command line (where /usr/local/petsc2-optimised is the directory into which you want to compile PETSc)

./config/configure.py --with-debugging=0 --with-mpi-dir=/usr/local/mpich2-optimised --with-shared=1

debug build:

• for code developing
• use a debug build of your MPI implementation, in this example /usr/local/mpich2-debug
• if you want to compile this build into a directory other than the unzipped source directory, add the flag --prefix=/usr/local/petsc2-debug to the end of the command line (where /usr/local/petsc2-debug is the directory into which you want to compile PETSc)

./config/configure.py --with-debugging=1 --with-mpi-dir=/usr/local/mpich2-debug --with-shared=1

You should see a line saying "PETSc shared libraries: enabled". You needn't necessarily be concerned at this stage if you also see "PETSc dynamic libraries: disabled".

Installing PETSc 2.3.3

Next type the following (for either build, still from within the unzipped source directory):

make

(If you are using Mac OS 1.5.2 and you get an error about .dylib files when compiling petsc, see the sub-section "Mac OSX Troubleshooting" below, and then continue the make install step here if necessary).

Then if you specified an installation directory above using the --prefix flag, now type (root or administrator access may be required; on macs type "sudo make install" in place of the "make install" step):

make install_src
make install

Mac OSX 10.5.2 Troubleshooting

If you are using Mac OS 10.5.2 and you get an error about .dylib files when compiling petsc, navigate into the unzipped source directory and try adding the following line to the file bmake/common/rules.shared.basic:

shared_darwin9: shared_darwin7

Then navigate back to the top of the unzipped source directory and rebuild the shared libraries by typing (replacing the machine architecture with the one you specified with your PETSC_ARCH variable):

make PETSC_ARCH=darwin8.7.1-c-opt shared

If you specified an installation directory during configuration using the --prefix flag, continue the steps above for the "make install" step.

Testing Your Copy of PETSc 2.3.3

Still from the unzipped source directory, type the following to ensure mpd is running and then test your copy of PETSc 2.3.3:

mpd &
make test

Setting Environment Variables

Now set the environment variable for the build of PETSc with which you want to build Underworld. If you specified an installation directory when configuring using the --prefix flag, set PETSC_DIR to that directory; otherwise set PETSC_DIR to the unzipped source directory into which you compiled.

Add the following to your ~/.bashrc (if you are using bash) or ~/.tcshrc (if you are using tcsh) (the example of an optimised build compiled into /usr/local/petsc2-optimised on a machine with the architecture darwin8.7.1-c is used):

bash:

optimised

export PETSC_ARCH=darwin8.7.1-c-opt
export PETSC_DIR=/usr/local/petsc2-optimised

debug

export PETSC_ARCH=darwin8.7.1-c-debug
export PETSC_DIR=/usr/local/petsc2-debug

tcsh:

optimised

setenv PETSC_ARCH darwin8.7.1-c-opt
setenv PETSC_DIR /usr/local/petsc2-optimised

debug

setenv PETSC_ARCH darwin8.7.1-c-debug
setenv PETSC_DIR /usr/local/petsc2-debug

Note: if you later decide to compile a copy of Underworld with a different build of PETSc (such as a debug build instead of an optimised build), you will need to change these environment variables according to the new build of PETSc to be used.

Adding the PETSc 2.3.3 Dynamic Libraries to your Path

You will also need to add the location of the PETSC 2.3.3 dynamic libraries to your path. To view your PATH, type the following:

echo $PATH

This will probably look something like PATH=/usr/bin:/bin:/usr/local/bin, which is a list of ':' separated directories of where commands can be executed from without typing the full path. To add the PETSc 2.3.3 dynamic libraries to your path add the following to your ~/.bashrc or ~/.tcshrc file:

linux

export LD_LIBRARY_PATH=${PETSC_DIR}/lib/${PETSC_ARCH}:$LD_LIBRARY_PATH

macs

export LD_LIBRARY_PATH=${PETSC_DIR}/lib/${PETSC_ARCH}:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=$LD_LIBRARY_PATH

Note: This setting does not need to be changed if you later decide to compile a copy of Underworld with a different build of PETSc (such as a debug build instead of an optimised build), so long as you update the PETSC_DIR and PETSC_ARCH environment variables, as detailed in the section above. However you will need to open a new terminal window after setting the environment variables to ensure that the old paths no longer apply.

Testing the PETSc 2.3.3 Dynamic Libraries (on a mac)

To ensure that dynamic libraries have been created and are being used, navigate into the directory into which you compiled PETSc 2.3.3 (using here the example of an optimised build on a machine with the architecture darwin8.7.1-c) and type:

otool -L ./lib/darwin8.7.1-c-opt/libpetsc.dylib

and ensure that a line is returned with libmpich.dylib (the path should be to this file in your built directory).

You have finished installing PETSc.

-- Wendy Mason - 07 October 2009
-- Owen Kaluza - 07 October 2009