Skip to content

How to submit jobs to LSF

You can submit jobs to LSF using different queues. Follow the instructions in this guide for different methods.

Things to note when submitting jobs on the HPC

  1. You must use the login nodes as a portal to the HPC for the submission of jobs, the login nodes have not been designed to handle the processing of data directly.
  2. Unauthorised tools will not be permitted to run on the login nodes and, if they are found to be running, will be terminated without warning.
  3. Memory management is enforced on the HPC. This is to safeguard against rogue jobs consuming all the available memory on a node and crashing that node for everyone. When a job is submitted, you will need to specify -R rusage[mem=<memory_in_mb>]. -R informs LSF of how much memory you wish to consume. Each CPU has an allocation of 1 GB (1000 MB) of memory, so in order to use larger amounts, you will need to ask for additional CPUs with -n. For example, to ask for 2 GB of RAM, the following flags should be used: -R rusage[mem=2000] -n 2. If -R is not specified, default values of -R rusage[mem=1000] will be added to your job.

Submitting a script as a job

To submit an LSF job using a script use the following template:

#!/bin/bash  
#BSUB -q <your_queue>  
#BSUB -P <yourProject>  
#BSUB -o <path_to/job.%J.out>  
#BSUB -e <path_to/job.%J.err>  
#BSUB -J <jobName>  
#BSUB -R "rusage[mem=1000] span[hosts=1]"  
#BSUB -M <max_memory_in_mb>
#BSUB -n 2  
#BSUB -cwd <"your_dir">  

# Only retain the TMPDIR corresponding to your GECIP or Discovery Forum membership  
export TMPDIR=/re_scratch/re_gecip/<your_GECIP>/<your_username>  
export TMPDIR=/re_scratch/re_discovery_forum/<your_discovery_forum_folder>/<your_username>  

module load <moduleName>  

script

Use the following command to submit the script:

bsub < myscript.sh

Note the < sign above indicating that your script is fed into the submission command.

Files output.stdout and output.stderr are file names of your choice where any messages, warnings and errors will be logged.

Project codes

For a list of all LSF queues and project codes see LSF Project Codes. You can also see what queues are assigned to you using:

bugroup -w | grep <your_username> | awk '{print $1}'

You will only be able to submit to queues that you have LDAP access to.

For more on how to submit jobs, see Advanced Job Submission Guidelines.

Queues available

Genomics England HPC queues are time based queues (short, medium, long). This means you need to submit the jobs to the queue that reflects the runtime of your job.

Queue name LDAP group Description
inter N/A This queue is for lightweight interactive or GUI tools. The queue has a per-user concurrent job limit of five
short discovery_lsf_access, GECIP_lsf_access, research_lsf_access This queue is for jobs with maximum RUNTIME of four hours
medium discovery_lsf_access, GECIP_lsf_access, research_lsf_access This queue is for jobs with maximum RUNTIME of 24 hours
long discovery_lsf_access, GECIP_lsf_access, research_lsf_access This queue is for jobs with unlimited RUNTIME (default to seven days if not specified - or specify the limit as -W [hours:]minutes )

To see all available queues in the grid, run bqueues

1
2
3
4
5
QUEUE_NAME      PRIO STATUS          MAX JL/U JL/P JL/H NJOBS  PEND   RUN  SUSP
inter            50  Open:Active       -    -    -    -     0     0     0     0
short            30  Open:Active       -    -    -    -     0     0     0     0
medium           20  Open:Active       -    -    -    -     0     0     0     0
long             10  Open:Active       -    -    -    -     0     0     0     0

Some basic LSF commands

command description
bsub submits a job to the cluster
bqueues shows info on the cluster queues
bjobs shows info on the cluster jobs
bhosts shows info on the cluster hosts
bhist shows info on the finished cluster jobs
bacct shows statistics and info on finished cluster jobs
bkill removes a job from the cluster
lshosts shows static resource info
lsload shows dynamic resource info

bjobs is a very handy command to view job information (both pending and running jobs). Using the long option (-l), it shows high level view of why (in case of job pending in the queue), where, turnaround time, resource usage detail (for running jobs)

Usage:

bjobs -l <JOBID>

<JOBID> is the job number generated when you submit you job. You can always get access to the ID of your running jobs by running a simple bjobs command.