Parallel Jobs

From HPC

(Difference between revisions)
Jump to: navigation, search
(→Running OpenMP job)
(→Parallel queue)
 
(12 intermediate revisions not shown)
Line 1: Line 1:
-
parallel queue
+
=Parallel queue=
-
The parallel queue is for running two types of parallel job openMP and openMPI.
+
The parallel queue is for running two types of parallel enviroment job Shared Memory/openMP (single node) and openMPI (single/multiple nodes).
-
=OpenMP=
+
Basic jobs spec
-
Shared memory, single node. This is parallel across the cores on a single node, can often be archived by just compiling your code with openMP flags.  However better performance can be achieve if you programs are written with openMP in mind.
+
#$ -q parallel.q
 +
  #$ -pe smp 4
 +
#$ -R y
-
==Compiling==
+
or
-
enabling openMP while compiling:
+
#$ -q parallel.q
 +
#$ -pe openmpi 16
 +
#$ -R y
-
GNU: gfortran -fopenmp -o <exec> <src>
+
==Important notes==
-
Intel: ifort -openmp -o <exec> <src>
+
'''Your code MUST limit the number of CPUS/Cores/Thread/Processes used to the same number of slots requested.'''
-
==Running Shared Memory or OpenMP job (most common)==
+
If you don't do this your code may run on the same host as other users jobs and you will consume more resource than you requested affecting the other users work. This can lead to resource starvation causing your jobs and others users jobs to take exponentially longer or even crash the host.
 +
 
 +
You cannot assume the number of CPUS on a host (we have various combination for example 12 or 32 CPU hosts). So you cannot simply assume that the number of slots requested will get you exclusive use of a host to save limiting your code. Requesting 32 slots will severely limit the hosts your job can run on and will take a long time to start running.
 +
 
 +
'''Jobs take a long time to start'''
 +
 
 +
The more CPUSs/slots you request the longer you will have to wait for a host that has that many free slots. You will be waiting for multiple jobs to finish so the scheduler can give you have the requested resource for your job.
 +
 
 +
The HPC contains hosts with different numbers of Slots (CPUS/Cores) some will have 8, 12 or 32 for example. The more slots you request the less hosts that can run the job. If you request 20 slots, the job cannot run on a host with less than 20 CPUs. The exception to this run is OpenMPI jobs which are designed to run across multiple hosts.
 +
 
 +
=Running Shared Memory or OpenMP job (single node, most common)=
'''For OpenMP jobs''' - The enviroment OMP_NUM_THREADS must be set to define the number of threads the program should use. Use as many as you have processors i.e 8
'''For OpenMP jobs''' - The enviroment OMP_NUM_THREADS must be set to define the number of threads the program should use. Use as many as you have processors i.e 8
Line 28: Line 42:
  #$ -cwd -V
  #$ -cwd -V
  #$ -l mem_free=1G,h_vmem=1G
  #$ -l mem_free=1G,h_vmem=1G
 +
#$ -q parallel.q
  #$ -pe smp 8
  #$ -pe smp 8
 +
#$ -R y
  export OMP_NUM_THREADS=8
  export OMP_NUM_THREADS=8
  myOpenMPapp
  myOpenMPapp
-
Example of requesting two cpus
+
Example of requesting two cpus (non OpenMP)
  #!/bin/bash
  #!/bin/bash
  #$ -cwd -V
  #$ -cwd -V
  #$ -l mem_free=1G,h_vmem=1G
  #$ -l mem_free=1G,h_vmem=1G
 +
#$ -q parallel.q
  #$ -pe smp 2
  #$ -pe smp 2
 +
#$ -R y
  myMultiCPuApp
  myMultiCPuApp
Line 43: Line 61:
  qsub script
  qsub script
 +
 +
 +
==OpenMP==
 +
 +
Shared memory, single node. This is parallel across the cores on a single node, can often be archived by just compiling your code with openMP flags.  However better performance can be achieve if you programs are written with openMP in mind.
 +
 +
==Compiling==
 +
 +
enabling openMP while compiling:
 +
 +
GNU: gfortran -fopenmp -o <exec> <src>
=OpenMPI=
=OpenMPI=
-
Distributed Memory, multiple nodes. This method of parallel requires you to write your programs to work with openMPI. Using openMPI the nodes will comunicate with each other via the infiniband network
+
Distributed Memory, multiple nodes. This method of parallel requires you to write your programs to work with openMPI.  
 +
 
 +
'''Memory requests''' - The memory specified by h_vmem is per slot, so if ask for 8 slots (-pe smp 8) then the node requires 8 x h_vmem (8 x 1GB = 8GB).
 +
 
 +
 
 +
Openmpi documentation for Grid Engine - https://www.open-mpi.org/faq/?category=sge (not pe is openmpi not orte)
==Process that a parallel job goes through==
==Process that a parallel job goes through==
Line 54: Line 88:
*SGE runs the users job script
*SGE runs the users job script
*On termination a "stop" script is executed
*On termination a "stop" script is executed
-
 
-
Scripts that are automatically used are in
 
-
 
-
/usr/local/sge6.0/streamline/mpi/
 
-
 
-
The ompi_start.sh script
 
-
 
-
#!/bin/sh
 
-
# Local info
 
-
# This is executed on the front end server
 
-
# SERVER=`hostname -s`
 
-
function ncpus() {
 
-
  n=`cat /proc/cpuinfo | grep processor | tail -1 | cut -f 2- -d :`
 
-
  echo $((n+1))
 
-
  return
 
-
}
 
-
SMP=${SMP:-`ncpus`}
 
-
. /etc/profile
 
-
#
 
-
pe_hostfile=$1
 
-
echo $pe_hostfile
 
-
cat $pe_hostfile
 
-
job_id=$2
 
-
user=`basename ${HOME}`
 
-
if [ -d /users/$user ]; then
 
-
  user_dir=/users/$user
 
-
else
 
-
  user_dir=${HOME}
 
-
fi
 
-
mpich_dir=${user_dir}/.mpich
 
-
mkdir -p $mpich_dir
 
-
 
-
cat $pe_hostfile | cut -f 1 -d " " | sort > $mpich_dir/mpich_hosts.$job_id
 
==Compiling OpenMPI==
==Compiling OpenMPI==
Line 93: Line 94:
Compile using via wrapper scripts
Compile using via wrapper scripts
-
 
-
===Intel===
 
-
 
-
With the module openmpi/1.2.6-1/intel loaded
 
-
 
-
/opt/openmpi-1.2.6-1/intel/bin/
 
-
 
-
mpic++, mpiCC, mpicc, mpicxx, mpif77, mpif90
 
===GNU===
===GNU===
-
With the module openmpi/1.2.6-1/gnu loaded
+
With the module (currently Open MPI: 1.6.4)
-
 
+
-
/opt/openmpi-1.2.6-1/gcc/bin/
+
  mpic++, mpiCC, mpicc, mpicxx, mpif77, mpif90
  mpic++, mpiCC, mpicc, mpicxx, mpif77, mpif90
Line 112: Line 103:
==Submitting OpenMPI Job==
==Submitting OpenMPI Job==
-
Run using batch scheduler
+
You should specify parallel environment openmpi (-pe openmpi x)
-
+
-
ompisub C <app> (C = cores)
+
-
ompisub 8 <prog> <args> (8 cores, which equals on node)
+
'''Memory requests''' - The memory specified by h_vmem is per slot, so if ask for 8 slots (-pe smp 8) then the node requires 8 x h_vmem (8 x 1GB = 8GB).
-
ompisub NxC <app> (N = Nodes C = Cores)
+
Job script (16 slots)
 +
 
 +
#!/bin/bash
 +
#$ -cwd -V
 +
#$ -l mem_free=1G,h_vmem=1G
 +
#$ -q parallel.q
 +
#$ -pe openmpi 16
 +
#$ -R y
 +
mpirun -np 16 <program>
-
ompisub 4x3 <prog> <args> (4 nodes each setup to use 3 cores)
+
'''Note: -np value should match -pe value'''
[[Category:Getting Started]]
[[Category:Getting Started]]

Current revision as of 07:43, 24 May 2019

Contents

Parallel queue

The parallel queue is for running two types of parallel enviroment job Shared Memory/openMP (single node) and openMPI (single/multiple nodes).

Basic jobs spec 

#$ -q parallel.q
#$ -pe smp 4
#$ -R y

or 

#$ -q parallel.q
#$ -pe openmpi 16
#$ -R y

Important notes

Your code MUST limit the number of CPUS/Cores/Thread/Processes used to the same number of slots requested.

If you don't do this your code may run on the same host as other users jobs and you will consume more resource than you requested affecting the other users work. This can lead to resource starvation causing your jobs and others users jobs to take exponentially longer or even crash the host.

You cannot assume the number of CPUS on a host (we have various combination for example 12 or 32 CPU hosts). So you cannot simply assume that the number of slots requested will get you exclusive use of a host to save limiting your code. Requesting 32 slots will severely limit the hosts your job can run on and will take a long time to start running.

Jobs take a long time to start

The more CPUSs/slots you request the longer you will have to wait for a host that has that many free slots. You will be waiting for multiple jobs to finish so the scheduler can give you have the requested resource for your job.

The HPC contains hosts with different numbers of Slots (CPUS/Cores) some will have 8, 12 or 32 for example. The more slots you request the less hosts that can run the job. If you request 20 slots, the job cannot run on a host with less than 20 CPUs. The exception to this run is OpenMPI jobs which are designed to run across multiple hosts.

Running Shared Memory or OpenMP job (single node, most common)

For OpenMP jobs - The enviroment OMP_NUM_THREADS must be set to define the number of threads the program should use. Use as many as you have processors i.e 8

Memory requests - The memory specified by h_vmem is per slot, so if ask for 8 slots (-pe smp 8) then the node requires 8 x h_vmem (8 x 1GB = 8GB).

The "#$ -pe smp 2" specifies the job should run on one node and will consume two slots/cpus.

Example submission script (8 CPUS/Slots for 8 OpenMP threads) 

#!/bin/bash
#$ -cwd -V
#$ -l mem_free=1G,h_vmem=1G
#$ -q parallel.q
#$ -pe smp 8
#$ -R y
export OMP_NUM_THREADS=8
myOpenMPapp

Example of requesting two cpus (non OpenMP) 

#!/bin/bash
#$ -cwd -V
#$ -l mem_free=1G,h_vmem=1G
#$ -q parallel.q
#$ -pe smp 2
#$ -R y
myMultiCPuApp

To submit job to the queue 

qsub script


OpenMP

Shared memory, single node. This is parallel across the cores on a single node, can often be archived by just compiling your code with openMP flags. However better performance can be achieve if you programs are written with openMP in mind.

Compiling

enabling openMP while compiling:

GNU: gfortran -fopenmp -o <exec> <src>

OpenMPI

Distributed Memory, multiple nodes. This method of parallel requires you to write your programs to work with openMPI.

Memory requests - The memory specified by h_vmem is per slot, so if ask for 8 slots (-pe smp 8) then the node requires 8 x h_vmem (8 x 1GB = 8GB).


Openmpi documentation for Grid Engine - https://www.open-mpi.org/faq/?category=sge (not pe is openmpi not orte)

Process that a parallel job goes through

  • SGE produces a list of hosts $PE_HOSTFILE
  • SGE executes a "start" script for the PE
  • SGE runs the users job script
  • On termination a "stop" script is executed

Compiling OpenMPI

Setup up the system environment, using modules to load the OpenMPI libraries you want to use. This is likely to be a choice between GNU and Intel.

Compile using via wrapper scripts

GNU

With the module (currently Open MPI: 1.6.4) 

mpic++, mpiCC, mpicc, mpicxx, mpif77, mpif90

Submitting OpenMPI Job

You should specify parallel environment openmpi (-pe openmpi x)

Memory requests - The memory specified by h_vmem is per slot, so if ask for 8 slots (-pe smp 8) then the node requires 8 x h_vmem (8 x 1GB = 8GB).

Job script (16 slots) 

#!/bin/bash
#$ -cwd -V 
#$ -l mem_free=1G,h_vmem=1G
#$ -q parallel.q
#$ -pe openmpi 16
#$ -R y
mpirun -np 16 <program>

Note: -np value should match -pe value

Retrieved from "http://wiki.lshtm.ac.uk/hpc/index.php5/Parallel_Jobs"
Personal tools