Underworld

a long term geodynamics simulation platform

PETSc Download and Installation

What is PETSc?

PETSc is a set of libraries, including equation solvers and assembly routines, which is used by Underworld. It is a toolkit, developed by the United States' Argonne National Laboratory, for doing mathematical calculations in parallel. Its particular focus is the solution of applications modelled by partial differential equations, and to provide this it includes extensive matrix and vector support. The library is scalable, callable from C, C++, and Fortran, and includes many useful support features for debugging, profiling etc.

Further information is available on PETSc's website.

Downloading the PETSc source code

Underworld is compatible with the following versions of PETSc, "all flavours of UNIX":

  • 3.1 for Underworld-1.5 and above
  • 2.3.3 for Underworld-1.4

Go to PETSc's download page and download the code (for example, petsc-3.1-p5.tar.gz) (via FTP).

Untar and unzip the file (root / administrator access may be required) (if you are on a mac this may be done for you): 

tar zxvf petsc-3.1-p5.tar.gz

Setting up and Compiling PETSc

Prior to setting up and compiling PETSc, please ensure that you have installed an MPI implementation. Click on the relevant link below for a summary of the procedure for your version of PETSc:

The following instructions detail how to compile the recent versions of PETSc (e.g. 3.1), set up environment variables and add the dynamic libraries to your path, for 'all flavours of *NIX systems' including Macs.

Locating the Unzipped PETSc Source Directory and Deciding on an Installation Method

There are two ways you can install PETSc, as it now offers a "make install" step:

  1. move the unzipped source directory to the location in which you want to install PETSc, and build PETSc into the unzipped directory (as was necessary for previous versions of PETSc); different builds of PETSc (such as optimised and debug) can live side-by-side within the same directory, and will be differentiated by a library subdirectory for each build
  2. locate the unzipped source directory anywhere you like and build PETSc into a different directory in the location of your choice; different builds of PETSc (such as optimised and debug) can live side-by-side in their own different directories

Note: You can only compile one build type (e.g. optimised or debug) at a time. If you want more than one build, follow all steps prior to "Setting up Your Environment Variables", repeat for the second build and then set up the environment variables for the build that you are going to compile Underworld with.

Setting the PETSc Directory and Your Machine Architecture with which to compile PETSc

In order to compile PETSc, it needs to be told where the unzipped (source) directory is located (irrespective of where you are going to install it) and the architecture of your machine for the build type you are going to compile (e.g. optimised or debug).

Note: if you do not know your machine's architecture (on linux you can try "linux"), PETSc will tell you after you configure, in which case just set PETSC_DIR now and set PETSC_ARCH after the configure step below.

You can just set these temporarily from the command line for the terminal window you are using in the current session as follows. Navigate into the unzipped source directory and type (where pwd means "print name of current/working directory" and the example machine architecture darwin8.7.1-c is used):

bash: 

export PETSC_DIR=`pwd`

tcsh: 

setenv PETSC_DIR `pwd`

Configuring PETSc

Prior to configuring PETSc, please ensure that you have installed an MPI implementation and set up its environment variables (if applicable).

A PETSc instruction manual for installation is available. The following is a summary of the procedure.

From within the unzipped petsc source directory, type one of the following, noting:

  • the MPI implementation used in these examples is Mpich 2, installed in /usr/local (if you only have one MPI implementation built and have set its MPI_DIR environment variable in your .bashrc or .tcshrc, you can replace /usr/local/mpich2 with ${MPI_DIR})
  • if you didn't compile fortran into your MPI implementation, then the PETSc configure will ask you to add the flag --with-fc=0

optimised build:

  • good if you are not going to be developing a lot of code but are going to be doing a lot of runs
  • use an optimised build of your MPI implementation, in this example /usr/local/mpich2-optimised 
./config/configure.py --with-debugging=0 --with-mpi-dir=/usr/local/mpich2-optimised --with-shared=1

debug build:

  • for code developing
  • use a debug build of your MPI implementation, in this example /usr/local/mpich2-debug 
./config/configure.py --with-debugging=1 --with-mpi-dir=/usr/local/mpich2-debug --with-shared=1

If you want to compile the build into a directory other than the unzipped source directory, add the prefix flag. For example: 

--prefix=/usr/local/petsc2

If you're using PETsc version 3.2 and above, you must replace the 

--with-shared=1

flag with the following: 

--with-shared-libraries

You should see a line saying "PETSc shared libraries: enabled". You needn't necessarily be concerned at this stage if you also see "PETSc dynamic libraries: disabled".

Installing PETSc

Next type the following (for either build, still from within the unzipped source directory): 

make

(If you are using Mac OS 1.5.2 and you get an error about .dylib files when compiling petsc, see the sub-section "Mac OSX Troubleshooting" below, and then continue the make install step here if necessary).

Then if you specified an installation directory above using the --prefix flag, now type (root or administrator access may be required; on macs type "sudo -E make install" in place of the "make install" step, or if this does not work login as root or copy the directory to the location you want it to be installed and run these commands there, although the prefix flag will still be required): 

make install

Mac OSX 10.5.2 Troubleshooting

If you are using Mac OS 10.5.2 and you get an error about .dylib files when compiling petsc, navigate into the unzipped source directory and try adding the following line to the file bmake/common/rules.shared.basic: 

shared_darwin9: shared_darwin7

Then navigate back to the top of the unzipped source directory and rebuild the shared libraries by typing (replacing the machine architecture with the one you specified with your PETSC_ARCH variable): 

make PETSC_ARCH=darwin8.7.1-c-opt shared

If you specified an installation directory during configuration using the --prefix flag, continue the steps above for the "make install" step.

Testing Your Copy of PETSc

Still from the unzipped source directory, type the following to ensure mpd is running and then test your copy of PETSc: 

mpd &
make test

Setting Environment Variables

Now set the environment variable for the build of PETSc with which you want to build Underworld. If you specified an installation directory when configuring using the --prefix flag, set PETSC_DIR to that directory; otherwise set PETSC_DIR to the unzipped source directory into which you compiled.

Add the following to your ~/.bashrc (if you are using bash) or ~/.tcshrc (if you are using tcsh) (the example of an optimised build compiled into /usr/local/petsc2-optimised on a machine with the architecture darwin8.7.1-c is used):

bash:

optimised 

export PETSC_ARCH=darwin8.7.1-c-opt
export PETSC_DIR=/usr/local/petsc2-optimised

debug 

export PETSC_ARCH=darwin8.7.1-c-debug
export PETSC_DIR=/usr/local/petsc2-debug

tcsh:

optimised 

setenv PETSC_ARCH darwin8.7.1-c-opt
setenv PETSC_DIR /usr/local/petsc2-optimised

debug 

setenv PETSC_ARCH darwin8.7.1-c-debug
setenv PETSC_DIR /usr/local/petsc2-debug

Note: if you later decide to compile a copy of Underworld with a different build of PETSc (such as a debug build instead of an optimised build), you will need to change these environment variables according to the new build of PETSc to be used.

Adding the PETSc Dynamic Libraries to your Path

You will also need to add the location of the PETSC dynamic libraries to your path. To view your PATH, type the following: 

echo $PATH

This will probably look something like PATH=/usr/bin:/bin:/usr/local/bin, which is a list of ':' separated directories of where commands can be executed from without typing the full path. To add the PETSc dynamic libraries to your path add the following to your ~/.bashrc or ~/.tcshrc file:

linux 

export LD_LIBRARY_PATH=${PETSC_DIR}/lib/${PETSC_ARCH}:$LD_LIBRARY_PATH

macs 

export LD_LIBRARY_PATH=${PETSC_DIR}/lib/${PETSC_ARCH}:$LD_LIBRARY_PATH
export DYLD_LIBRARY_PATH=$LD_LIBRARY_PATH

Note: This setting does not need to be changed if you later decide to compile a copy of Underworld with a different build of PETSc (such as a debug build instead of an optimised build), so long as you update the PETSC_DIR and PETSC_ARCH environment variables, as detailed in the section above. However you will need to open a new terminal window after setting the environment variables to ensure that the old paths no longer apply.

Testing the PETSc Dynamic Libraries (on a mac)

To ensure that dynamic libraries have been created and are being used, navigate into the directory into which you compiled PETSc (using here the example of an optimised build on a machine with the architecture darwin8.7.1-c) and type: 

otool -L ./lib/darwin8.7.1-c-opt/libpetsc.dylib

and ensure that a line is returned with libmpich.dylib (the path should be to this file in your built directory).

You have finished installing PETSc.

-- Jerico Revote 29 November 2011
-- Wendy Mason - 28 June 2011
-- Julian Giordani - 28 June 2011
-- Owen Kaluza - 07 October 2009
-- Wendy Sharples - 03 September 2010
-- Alan Lo - 25 Jan 2006