Introduction to PETSc

Introduction to PETSc VIGRE Seminar, Wednesday, November 8, 2006

Parallel Computing How (basically) does it work?

Parallel Computing How (basically) does it work? • Assign each processor a number

Parallel Computing How (basically) does it work? • Assign each processor a number • The same program goes to all

Parallel Computing How (basically) does it work? • Assign each processor a number • The same program goes to all • Each uses separate memory

Parallel Computing How (basically) does it work? • Assign each processor a number • The same program goes to all • Each uses separate memory • They pass information back and forth as necessary

Parallel Computing Example 1: Matrix-Vector Product

Parallel Computing Example 2: Matrix-Vector Product

Parallel Computing Example 3: Matrix-Matrix Product

Parallel Computing Example 4: Block Diagonal Product

Parallel Computing When is it worth it to parallelize?

Parallel Computing When is it worth it to parallelize? • There is a time cost associated with passing messages

Parallel Computing When is it worth it to parallelize? • There is a time cost associated with passing messages • The amount of message passing is dependent on the problem and the program (algorithm)

Parallel Computing When is it worth it to parallelize? • Therefore, the benefits depend more on the structure of the problem and the program than on the size/speed of the parallel network (diminishing returns).

Parallel Networks How do I use multiple processors?

Parallel Networks How do I use multiple processors? • This depends on the network, but… • Most networks use some variation of PBS, a job scheduler, and mpirun or mpiexec.

Parallel Networks How do I use multiple processors? • This depends on the network, but… • Most networks use some variation of PBS, a job scheduler, and mpirun or mpiexec. • A parallel program needs to be submitted as a batch job.

Parallel Networks • Suppose I have a program myprog, which gets data from data.dat, which I call in the following fashion when only using one processor: ./myprog –f data.dat I would write a file myprog.pbs that looks like the following:

Parallel Networks #PBS –q compute (name of the processing queue [not necessary on all networks]) #PBS -N myprog (the name of the job) #PBS –l nodes=2:ppn=1,walltime=00:10:00 (number of nodes and number of processes per node, maximum time to allow the program to run) #PBS -o /home/me/mydir/myprog.out (where the output of the program should be written) #PBS -e /home/me/mydir/myprog.err (where the error stream should be written) These are the headers that tell the job scheduler how to handle your job.

Parallel Networks Although what follows depends on the MPI software that the network runs, it should look something like this: cd $PBS_O_WORKDIR (makes the processors run the program in the directory where myprog.pbsis saved) mpirun –machinefile $PBS_NODEFILE –np 2 myprog –f mydata.dat (tells the MPI software which processes to use and how many processes to start: notice that command line arguments follows as usual)

Parallel Networks • Once the .pbs file is written, it can be submitted to the job scheduler with qsub: qsub myprog.pbs

Parallel Networks • Once the .pbs file is written, it can be submitted to the job scheduler with qsub: qsub myprog.pbs • You can check to see if your job is running with the command qstat.

Parallel Networks • Some systems (but not all) will allow you to simulate running your program in parallel on one processor, which is useful for debugging: mpirun –np 3 myprog –f mydata.dat

Parallel Networks What parallel systems are available?

Parallel Networks What parallel systems are available? • RTC : Rice Terascale Cluster: 244 processors.

Parallel Networks What parallel systems are available? • RTC : Rice Terascale Cluster: 244 processors. • ADA : Cray XD1: 632 processors.

Parallel Networks What parallel systems are available? • RTC : Rice Terascale Cluster: 244 processors. • ADA : Cray XD1: 632 processors. • caamster: CAAM department exclusive: 8(?) processors.

PETSc What do I use PETSc for?

PETSc What do I use PETSc for? • File I/O with “minimal” understanding of MPI

PETSc What do I use PETSc for? • File I/O with “minimal” understanding of MPI • Vector and matrix based data management (in particular: sparse)

PETSc What do I use PETSc for? • File I/O with “minimal” understanding of MPI • Vector and matrix based data management (in particular: sparse) • Linear algebra routines familiar from the famous serial packages

PETSc • At the moment, ada and caamster (and harvey) have PETSc installed

PETSc • At the moment, ada and caamster (and harvey) have PETSc installed • You can download and install PETSc on your own machine (requires cygwin for Windows), for educational and debugging purposes

PETSc • PETSc builds on existing software BLAS and LAPACK: which implementations to use can be specified at configuration

PETSc • PETSc builds on existing software BLAS and LAPACK: which implementations to use can be specified at configuration • Has (slower) debugging configuration and (faster, tacit) optimized configuration

PETSc • Installation comes with documentation, examples, and manual pages.

PETSc • Installation comes with documentation, examples, and manual pages. • The biggest part of learning how to use PETSc is learning how to use the manual pages.

PETSc • It is extremely useful to have an environmental variable PETSC_DIR in you shell of choice, which gives the path to the installation of PETSc, e.g. PETSC_DIR=/usr/local/src/petsc-2.3.1-p13/ export PETSC_DIR

PETSc Makefile

Introduction to PETSc