Performing distributed AI on a dataset

Performing distributed AI on a dataset#

This page explains how to distribute azimuthal integration of a dataset.

You first have to create a configuration file which describes how the azimuthal integration should be performed.

Then, Integrator provides two commands: integrate-slurm and integrate-mp. Both work the same way: with a configuration file.

Run distributed AI on the local machine#

The azimuthal integration can be distributed on a local powerful machine.

Modify the configuration file to set partition = local:

[computations distribution]

partition = local
n_workers = 4
cores_per_worker = 4

Then run integrate-mp conf_file.conf

Use the integrate-mp command, using partition = gpu or partition = p9gpu.

Warning

in this case, n_workers has a different meaning. It will actually spawn 8 * n_workers (8 workers by SLURM job).

Use the integrate-slurm command, with partition = nice or partition = p9gpu.

With this mode, you will likely have to use many workers to achieve decent speed (especially when using partition = nice).