The HPC is changing

We will soon be switching to a new High Performance Cluster, called Double Helix. This will mean that some of the commands you use to connect to the HPC and call modules will change. We will inform you by email when you are switching over, allowing you to make the necessary changes to your scripts. Please check our HPC changeover notes for more details on what will change.

How to submit jobs to LSF

You can submit jobs to LSF using different queues. Follow the instructions in this guide for different methods.

Things to note when submitting jobs on the HPC

1. You must use the login nodes as a portal to the HPC for the submission of jobs, the login nodes have not been designed to handle the processing of data directly.
2. Unauthorised tools will not be permitted to run on the login nodes and, if they are found to be running, will be terminated without warning.
3. Memory management is enforced on the HPC. This is to safeguard against rogue jobs consuming all the available memory on a node and crashing that node for everyone. When a job is submitted, you will need to specify -R rusage[mem=<memory_in_mb>] and -M <memory_in_mb>. -R informs LSF of how much memory you wish to consume, and -M will terminate your job if you consume more memory than asked for. Each CPU has an allocation of 16GB (16000 Mb) of memory, so in order to use larger amounts, you will need to ask for additional CPUs with -n. For example, to ask for 32GB of RAM, the following flags should be used: -R rusage[mem=32000] -M 32000 -n 2. If -R and -M re not specified, default values of -R rusage[mem=1000] and -M 2000 will be added to your job.

Interactive vs batch jobs

Interactive jobs are jobs that you interact with either via:

• command-line
• GUI
• Jobs that stay connected to a submission shell

Interactive command-line jobs in the HPC have a dedicated queue (name of the queue is inter) with dedicated resources during core hours for faster dispatch. These sessions should be closed as soon as they are no longer required.

Batch jobs are jobs that you don’t interact with. The job is disconnected from submission shell.

Jobs are batch by default

Submitting a job interactively

To submit an interactive LSF job, use the command bsub:

bsub -q inter -P <project_code> -o <output.stdout> -e <output.stderr> -R rusage[mem=<memory_in_mb>] -M <max_memory_in_mb> <myjob>

Submitting a script as a job

To submit an LSF job using a script use the following template:

#!/bin/bash  
#BSUB -q <your_queue>  
#BSUB -P <yourProject>  
#BSUB -o <path_to/job.%J.out>  
#BSUB -e <path_to/job.%J.err>  
#BSUB -J <jobName>  
#BSUB -R "rusage[mem=10000] span[hosts=1]"  
#BSUB -M <max_memory_in_mb>
#BSUB -n 2  
#BSUB -cwd <"your_dir">  

# Only retain the TMPDIR corresponding to your GECIP or Discovery Forum membership  
export TMPDIR=/re_scratch/re_gecip/<your_GECIP>/<your_username>  
export TMPDIR=/re_scratch/re_discovery_forum/<your_discovery_forum_folder>/<your_username>  

module load <moduleName>  

script

Use the following command to submit the script:

bsub < myscript.sh

Note the < sign above indicating that your script is fed into the submission command.

Files output.stdout and output.stderr are file names of your choice where any messages, warnings and errors will be logged.

Project codes

For a list of all LSF queues and project codes see LSF Project Codes. You can also see what queues are assigned to you using:

bugroup -w | grep <your_username> | awk '{print $1}'

You will only be able to submit to queues that you have LDAP access to.

For more on how to submit jobs, see Advanced Job Submission Guidelines.

Queues available

Genomics England HPC queues are time based queues (short, medium, long). This means you need to submit the jobs to the queue that reflects the runtime of your job.

Queue name	LDAP group	Description
inter	N/A	This queue is for lightweight interactive or GUI tools. The queue has a per-user concurrent job limit of five
short	discovery_lsf_access, GECIP_lsf_access, research_lsf_access	This queue is for jobs with maximum RUNTIME of four hours
medium	discovery_lsf_access, GECIP_lsf_access, research_lsf_access	This queue is for jobs with maximum RUNTIME of 24 hours
long	discovery_lsf_access, GECIP_lsf_access, research_lsf_access	This queue is for jobs with unlimited RUNTIME (default to seven days if not specified - or specify the limit as -W [hours:]minutes )

To see all available queues in the grid, run bqueues

QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP
inter            50  Open:Active       -    -    -    -     0     0     0     0
short            30  Open:Active       -    -    -    -     0     0     0     0
medium           20  Open:Active       -    -    -    -     0     0     0     0
long             10  Open:Active       -    -    -    -     0     0     0     0

Some basic LSF commands

command	description
bsub	submits a job to the cluster
bqueues	shows info on the cluster queues
bjobs	shows info on the cluster jobs
bhosts	shows info on the cluster hosts
bhist	shows info on the finished cluster jobs
bacct	shows statistics and info on finished cluster jobs
bkill	removes a job from the cluster
lshosts	shows static resource info
lsload	shows dynamic resource info

bjobs is a very handy command to view job information (both pending and running jobs). Using the long option (-l), it shows high level view of why (in case of job pending in the queue), where, turnaround time, resource usage detail (for running jobs)

Usage:

bjobs -l <JOBID>

<JOBID> is the job number generated when you submit you job. You can always get access to the ID of your running jobs by running a simple bjobs command.