We use Sun Grid Engine (SGE) as the job scheduler, the manual can be found here.
Submitting a Job
To submit a job run the following command from a login node:
prompt$ qsub myJobScript.sh
You will get confirmation that the job has been submitted successfully as well as the a job number e.g.
Your job 1234567 ("MyTESTJOBNAME") has been submitted
Once the job has been submitted you can get extra info about the job with:
prompt$ qstat -j <jobid>
This will show you the status of the job.
qstat by itself will show you the status of your whole queue. A status of r indicates that the job is running, qw indicates the job is waiting to run and Eqw indicates an error.
Some example scripts are stored in: /share/apps/examples
Please also take time to look at the other sections of this website particularly File Systems & Storage and Cluster Etiquette
Job Script
A simple example of a job script along with an explanation can be seen below:
#$ -l tmem=2G
#$ -l h_vmem=2G
#$ -l h_rt=1:0:0
#$ -S /bin/bash
#$ -j y
#$ -N MyTESTJOBNAME
hostname
date
sleep 20
date
The #$ will be interpreted as command flags for the scheduler. You must include at least the first 3 command flags in all of your scripts except if you are using Tensorflow or PyTorch. Beneath the command flags should be your script. The command flags -l tmem=2G and h_vmem=2G are used to request RAM, you should amend the amount requested as appropriate. -l h_rt=x:x:x is the requested run time of the job in the format hr:min:sec. If you exceed either h_rt or h_vmem your job will be stopped.
Array Jobs
The scheduler will store the task number in an environmental variable called $SGE_TASK_ID you can use this to reference the data that you want to process. You can use this to make each task within an array job refer to different pieces of data.
For example:
#$ -l tmem=2G
#$ -l h_vmem=2G
#$ -l h_rt=1:0:0
#$ -S /bin/bash
#$ -N MyTESTJOBNAME
#$ -t 1-1000
hostname
date
~/programs/process_data.sh "~/data/mydata${SGE_TASK_ID}"
date
This will submit an array job of 1000 tasks, and each task will run ~/programs/process_data.sh, and process a different file. i.e Task 1 will process ~/data/mydata1 and task2 will process ~/data/mydata2 etc.
You can also use SGE_TASK_ID to refer to lines in a file, say if you have a datapoint on each line.
This can be done in a few different ways, for example:
sed -n ${SGE_TASK_ID}'{p;q}' input.data
The -n flag suppresses the ‘normal’ output of sed with a p command, which is to print the file you give it. The q quits sed as soon as the specified line is reached, which minimises unnecessary processing of the input file.
or
head -${SGE_TASK_ID} input.data | tail -1
head -N prints the first N lines. tail -N prints the last N lines. In this case, you are running tail -1 on a subset of the initial file (output of the head command), which has the Nth line as the final entry.
SMP
SMP (Symmetric Multiprocessing) is a parallel environment for running multi-threaded jobs within a single node. Before you request an SMP environment you should ensure that your code is capable of running multi-threaded. To request an SMP environment use:
#$ -pe smp 5
Where 5 is the number of slots (amend this as needed)
You should always enable resource reservation on the job with
#$ -R y
This will stop smaller jobs from jumping ahead and causing the job to queue for longer.
MPI
MPI (Message Passing Interface) is a parallel environment for running a multi-threaded job using the MPI protocol. This is typically used for communicating between nodes but can be used within a node as well. You request an MPI environment with:
#$ -pe mpi 100
or
#$ -pe mpirr 100
where 100 is the number of slots (amend this as needed). The mpi environment will try to keep the slots close together, spread across as few nodes as possible. The mpirr environment will do the opposite and will try to allocate the slots as widely as possible across many nodes. Generally the mpi environment works better in most scenarios.
As with SMP jobs you should enable resource reservation on the job with
#$ -R y