integrator.app.utils module#

integrator.app.utils.config_to_clusterconfig(conf)#

Converts a dictionary to a ClusterConfiguration object

integrator.app.utils.config_to_aiconfig(conf)#

Converts a dictionary to a AIConfiguration object

integrator.app.utils.get_datasets_list(fname, entry=None, detector_name=None)#
integrator.app.utils.browse_datasets_wildcard(wildcards, entry=None, detector_name=None, errors='print', dataset_cls=<class 'integrator.hdf5.ID15Dataset'>)#

Browse folders and look for datasets. This assumes that the datasets are stored in this layout: /path/to/samplename_num/samplename_num.h5

Parameters

wildcards (str or list of str) – Wildcard or list of wildcards,

integrator.app.utils.get_folders_id15a_layout(datasets, output_prefix, processed_name='azint_', create_dirs=False)#

From a list of datasets, returns a list of output paths corresponding to each file name.

Parameters

  • datasets (list of HDF5Dataset) – List HDF5Dataset objects. The file path is expected to be in the ID15A layout “proposal/sample/dataset/scan(s)” example: /data/id15/inhouse6/ihxxx/id15/sampletest/sampletest_0008/sampletest_0008.h5 (possibly several scans therein)

  • output_prefix (str) – Common path where output files will be saved

  • create_dirs (bool, optional) – Whether to create output directories. Default is False.

Returns

  • output_files (list of str) – List of output files corresponding to each dataset

  • datasets_info (dict) – Nested dictionary with the information on samples/datasets/scans

integrator.app.utils.get_distributed_integrator_params(conf_file, force_create_cluster=False, overwritten_params=None, conf_dict=None, print_already_processed=True, ignore_already_processed=True, parsing_error_handling='print')#

Parse and validate a configuration file, return several data structures detailed below.

Parameters

  • conf_file (str) – Path to the configuration file

  • force_create_cluster (bool, optional) – Whether to force creation a cluster of workers. Default is False

  • overwritten_params (dict, optional) – Dictionary of parameters overwriting some options of the configuration file

  • conf_dict (dict, optional) – Configuration dict to use instead of first parameter ‘conf_file’.

  • print_already_processed (bool, optional) – Print datasets already processed (if existing_output = skip)

  • ignore_already_processed (bool, optional) – Remove already-processed datasets from the results list. Default is True

  • parsing_error_handling (str, optional) – How to handle datasets parsing errors. Can be “print”, “raise” or “pass”. Default is “print”.

Returns

  • conf (dict) – Nested dictionary extracted from the configuration file

  • ai_config (AIConfiguration) – Data structure with the azimuthal integration configuration

  • cluster (ClusterConfiguration) – Data structure with the cluster configuration

  • datasets (list of Dataset objects) – List where each item is a data structure with information on the dataset to process

  • output_files (list of str) – List where each item is the path to the output file, either a .nx file or a directory