From HPC

Revision as of 11:00, 9 April 2015 (view source) Aitsswhi (Talk | contribs) (→Intel) ← Older edit		Revision as of 11:03, 9 April 2015 (view source) Aitsswhi (Talk | contribs) (→GNU) Newer edit →
Line 83:		Line 83:
	===GNU===		===GNU===
-	With the module openmpi/1.2.6-1/gnu loaded	+	With the module (currently Open MPI: 1.6.4)
-		+
-	/opt/openmpi-1.2.6-1/gcc/bin/	+
	mpic++, mpiCC, mpicc, mpicxx, mpif77, mpif90		mpic++, mpiCC, mpicc, mpicxx, mpif77, mpif90

---

Revision as of 11:03, 9 April 2015

Contents 1 Parallel queue 2 Running Shared Memory or OpenMP job (single node, most common) 2.1 OpenMP 2.2 Compiling 3 OpenMPI 3.1 Process that a parallel job goes through 3.2 Compiling OpenMPI 3.2.1 GNU 3.3 Submitting OpenMPI Job

Parallel queue

The parallel queue is for running two types of parallel enviroment job Shared Memory/openMP (single node) and openMPI (single/multiple nodes).

Basic jobs spec

#$ -q parallel.q
#$ -pe smp 4
#$ -R y

or

#$ -q parallel.q
#$ -pe openmpi 16
#$ -R y

Running Shared Memory or OpenMP job (single node, most common)

For OpenMP jobs - The enviroment OMP_NUM_THREADS must be set to define the number of threads the program should use. Use as many as you have processors i.e 8

Memory requests - The memory specified by h_vmem is per slot, so if ask for 8 slots (-pe smp 8) then the node requires 8 x h_vmem (8 x 1GB = 8GB).

The "#$ -pe smp 2" specifies the job should run on one node and will consume two slots/cpus.

Example submission script (8 CPUS/Slots for 8 OpenMP threads)

#!/bin/bash
#$ -cwd -V
#$ -l mem_free=1G,h_vmem=1G
#$ -q parallel.q
#$ -pe smp 8
#$ -R y
export OMP_NUM_THREADS=8
myOpenMPapp

Example of requesting two cpus (non OpenMP)

#!/bin/bash
#$ -cwd -V
#$ -l mem_free=1G,h_vmem=1G
#$ -q parallel.q
#$ -pe smp 2
#$ -R y
myMultiCPuApp

To submit job to the queue

qsub script

OpenMP

Shared memory, single node. This is parallel across the cores on a single node, can often be archived by just compiling your code with openMP flags. However better performance can be achieve if you programs are written with openMP in mind.

Compiling

enabling openMP while compiling:

GNU: gfortran -fopenmp -o <exec> <src>

OpenMPI

Distributed Memory, multiple nodes. This method of parallel requires you to write your programs to work with openMPI.

Memory requests - The memory specified by h_vmem is per slot, so if ask for 8 slots (-pe smp 8) then the node requires 8 x h_vmem (8 x 1GB = 8GB).

Openmpi documentation for Grid Engine - https://www.open-mpi.org/faq/?category=sge (not pe is openmpi not orte)

Process that a parallel job goes through

• SGE produces a list of hosts $PE_HOSTFILE
• SGE executes a "start" script for the PE
• SGE runs the users job script
• On termination a "stop" script is executed

Compiling OpenMPI

Setup up the system environment, using modules to load the OpenMPI libraries you want to use. This is likely to be a choice between GNU and Intel.

Compile using via wrapper scripts

GNU

With the module (currently Open MPI: 1.6.4)

mpic++, mpiCC, mpicc, mpicxx, mpif77, mpif90

Submitting OpenMPI Job

Run using batch scheduler

ompisub C <app> (C = cores)

ompisub 8 <prog> <args> (8 cores, which equals on node)

ompisub NxC <app> (N = Nodes C = Cores)

ompisub 4x3 <prog> <args> (4 nodes each setup to use 3 cores)