Using nabu at ESRF

This page contains specific instructions on how to use nabu at ESRF. It focuses on the usage of nabu itself, without the graphical user interface tomwer, which has a dedicated documentation.

One command to remember: source /scisoft/tomotools_env/utils/activate_nabu.sh latest

1. On the compute cluster

For large scale reconstructions (multiple volumes), nabu is run on the ESRF compute cluster. Reconstruction can be done either through an interactive session, or through submission of a shell script.

The first thing to do is to connect to a front-end machine, which in turn is able to request computational resources:

ssh -XC cluster-access

1.1 - Requesting resources in SLURM

You can open an interactive session on a GPU compute node. Currently, the preferred partition is p9gpu (power9 machines with Tesla V100 GPUs).

salloc --partition p9gpu --gres=gpu:1 --time=01:00:00 --mem=200G  srun --pty bash

Warning

The important bit is --gres=gpu:1 - you have to explicitly request a GPU, even if the node is in the GPU partition.

This command will only request one CPU core. If you wish to do some CPU processing (eg. involving numpy), then the -c parameter has to be provided. To ensure that all the cores are on the same CPU socket, a working command is

salloc --partition p9gpu --gres=gpu:1 --time=01:00:00 --mem=200G --cores-per-socket 16 -c 64 srun --pty bash

Caution

The maximum duration of jobs in the p9gpu is one hour. For longer jobs, the p9gpu-long partition has to be used.

1.2 - Requesting resources in OAR

The OAR cluster is made of older machines (Kepler generation GPUs). This cluster is bound to be removed in the future. Nabu is less regularly updated on this cluster.

Request one hour of interactive session on the gpu queue:

oarsub -l cpu=1,walltime=01:00:00 -I -q gpu

1.3 - Pick and activate a nabu version

On the compute clusters (and most ESRF machines), the command to use is

source /scisoft/tomotools_env/utils/activate_nabu.sh VERSION

where VERSION is replaced with a version of nabu. For example:

source /scisoft/tomotools_env/utils/activate_nabu.sh 2021.2.0

Please see section 2 for instructions specific to machines that cannot access the /scisoft disk.

1.4 - Run the reconstruction

Once you have a compute node and activated a version of nabu, you can run the usual command nabu config_file.conf

2. On a local machine (LBS/dedicated machines)

Some machines cannot mount the /scisoft disk.

The following machines have special paths for activating nabu (here the dev version, but it can be another version):

  • lbs191: source  /nobackup/lbs191tmp1/nabu/utils/activate_nabu.sh dev

  • lbs051: source /nobackup/lbs0511/tomwer/nabu/deployment_scripts/activate_nabu.sh dev

Please contact us if you wish to install nabu on a specific ESRF machine.

3. Troubleshooting

Nabu fails to start with a cublasNotInitialized error

It very likely means that you did not request for a GPU in the SLURM/OAR allocation command. A list of available GPUs can be obtained with nvidia-smi. If a No devices were found message is shown, then indeed no GPU is available.

4. Frequently Asked Questions (FAQ)

How can I report a problem ?

For any problem with nabu (crash, bad reconstruction results, installation), there are several ways to report it:

In any case, you have to provide the configuration file giving the error so that the error can be reproduced, and ensure that the dataset to process is accessible by “other” users.

How can I do a quick check that nabu works correctly ?

To check that nabu is installed, simply type nabu -V. It returns the current version of nabu. If it fails, then nabu is not installed/activated.

To check that nabu is actually able to reconstruct, you can run some tests, the recommended one is nabu-test reconstruct. Ensure that all the tests in the line test_fbp are passing.

Is any of these above questions really frequently asked ?

No, but let’s hope the answers can help.