Skip to content

HPC changeover 2024

We will be changing our HPC, completing in May 2024. The new HPC, called Double Helix, will use Cloud compute, making it faster. This will change the way you connect to the HPC and call modules. You will receive an email when it is time for you to switch over.

This document reflects our current plans. We reserve the right to change these plans as things arise during the transition process.

What will change?

  • Connect to the HPC: The address you use to connect to the HPC will change. If you have an ssh config file set up, you will need to change this.
  • Call software modules: The module paths will be changing. We recommend that you run module avail to find the new paths for your favourite modules. You will need to change these in any workflows you have built.
  • Pre-built workflows: We will bring out new versions of the workflows that use the new software module calls. If you have a copy of these in your own working directories, you will need to get a new copy.
  • Next-flow workflow manager: We will be upgrading to a new version of Nextflow which uses DSL2. If you have any of your own workflows that use DSL1, you will still be able to use them with v23.04 for now, but we recommend that you update to DSL2, following guidance from Nextflow.

What will stay the same?

  • The filesystem: The filesystem and folder structure will not be changed. You do not need to move or copy your files to a new location. You can continue to write to your working folder before, during and after transition.
  • The queues: We will keep the hpc job queues short, medium, long and inter, although we may add more queues.
  • Tools available: We will make our best effort to install the same tools and modules on the new HPC, however this may not be possible with some older versions. The paths to call all tools will change; please run module avail to find the correct call.
  • Cloud costs: You will not need to do anything to pay for any Cloud compute you use. Cloud compute is all covered by Genomics England. This will continue to be free for members of our Research Network, and is covered by existing agreements with Discovery Forum members.

What do I need to do?

You will receive an email informing you when it's time for you to switch to the new HPC. You should not run any jobs on the HPC while you are being switched over. Any jobs that you already have running will finish, but eventually your access to the old HPC will be revoked.

FAQs

Cloud compute costs

Will I be charged by Amazon for any cloud compute I use on Double Helix?

No. All cloud compute costs are currently being covered by Genomics England.

What will happen to my ongoing jobs on the HPC?

Any jobs that you start before you transition to the new HPC will finish running. As the filesystem is not changing, they will write to your working directory as normal.

You should not start any new jobs on the old HPC during transition. You can start new jobs on the new HPC as soon as you have transitioned.

I only use the desktop applications, will this affect me?

No. This change will only affect researchers who use the HPC via the command line.

Will I still be able to use the same tools for my analysis?

Mostly yes, we will try to make all the same tools available on the new HPC, however this may not be possible with some older versions of tools. However the paths to the tools is changing. We recommend running module avail to find the new paths for your tools.

Will I still be able to access my data?

Yes, the new HPC will access the same folders and filesystem as before. You do not need to move or copy any data.

Will I be able to access Helix (the old HPC) after I have migrated to Double Helix (the new HPC)?

No. Your jobs will finish on Helix during your transition, but after transition you should not attempt to connect to Helix again and your access will be revoked.

Will I still be able to use tools I have installed myself?

You will need to reinstall any tools you have compiled and installed yourself.

Getting help

Please reach out to us with any questions you have about the new HPC via Service Desk, tagging your tickets as "Double helix".