Storage on the Cluster
To make best use of the cluster it is important to be aware of how the file systems attached to the cluster are arranged. If you read/write simultaneously from 100+ nodes to a server with one disk you will get, at best, the speed of the disk divided by the number of nodes. In reality your performance will be much worse.
In an ideal world your job should be CPU bound and not I/O bound. That is to say you don’t want a processor sitting idle while it waits to get data from a disk.
Home Directory
Your home directory is located at /home/<username>. This is separate from your main CS file store and is not backed up. Your home folder has a quota of 50GB and is hosted on a BlueArc Titan 4080 clustered NAS. Please do not use your home directory to store data in the long term.
Project Stores
For example: /cluster/project<n>/<projectname>
This is specially requested project space and is the only storage on the cluster that is backed up. Space here is available on request. You should request as many project spaces as you need, one per project.
Certain limited central file systems are available directly on the cluster.
Where they are available the general file stores from /cs/research/…/…/ will be mounted under
/SAN/<filestorename>
Click here to fill in a storage request form.
Node Local Scratch Space
Each node has its own local disk or SSD, with typically around 200GB of space available.
This is mounted on /scratch0 and is typically the best place for large amounts of data to be accessed from by your job. N.B. Scratch means just that it will get deleted on a regular basis. Note that /scratch0 is different for each node.
To use /scratch0 add #$ -l tscratch=<n>G to your submission script. Additionally, before you move any data to scratch, create a directory at the top level of /scratch0 with your username and job ID. For example:
mkdir -p /scratch0/alice/$JOB_ID
If you are running an array job, add the task id to the directory name too:
mkdir -p /scratch0/alice/$JOB_ID.$SGE_TASK_ID
This is to prevent multiple jobs running on the same node from overwriting each other.
Please remember that scratch space is for transient job data only, and the data could be deleted without warning.
When your job is finished, remember to delete said data within your job script. To ensure this happens you can incorporate a trap statement into your submission script. For example, with a non-array job:
function finish {
rm -rf /scratch0/alice/$JOB_ID
}
trap finish EXIT ERR INT TERM
...
This will run the finish function whenever the job exits (regardless of whether the job finished successfully or not).
Reference Datasets
We are trying to co-ordinate some of the reference sets available on the cluster to avoid duplication.
There is a /share/ref folder with links to all the sets and an index in the README.TXT
If you are creating a reference set of public data or a closed reference set for UCL or group use only let us know and we can add it here.
The list of current sets will soon be made available.
Please do not be afraid to discuss what your storage needs are. Also please remember to request money for storing you data on your grants.