Change-log of versions

2023.03

• Rework all 2D integrator to propagate variance, expose standard deviation and standard error of the mean.

• Support all version of python between 3.7 and 3.11

• Support Cython 3.0b1 (with performance degradation) in addition of cython 0.29.31+

2023.02 27/02/2023

• Core:

  • sigma_clip_ng is not the default sigma_clip

  • expose documentation for sigma_clip_legacy in addition to sigma_clip_ng

  • Worker can now use sigma_clip_legacy in addition to sigma_clip_ng, integrate1d or integrate2d

  • New calibrant: Vanadinite (used in high-presure, often as single crystal)

• GUI:

  • Fix labels on axes for some rarely used units

  • refactor IntegrationMethodDialog and share the IntegrationMethodTabs with pyFAI-calib2

• Test:

  • silent some noisy test (OpenCL on windows)

2023.01 16/01/2023

• Developer and packager tools:

  • Switch build system from numpy.distutils to meson-python

  • Keep the former setup.py for compatibility reasons: it will be removed in a future release

  • Drop Python 3.6 (default parameters in namedtuple feature used)

  • Require silx 1.1 (for OpenCL), scipy and matplotlib

• GUI side:

  • several minor improvements in pyFAI-calib2

  • Fixed calibration in jupyter-lab

• Core improvements:

  • Refactoring of the Geometry class

  • Geometry pseudo-inversion optimization

  • Improved support from Medipix-based Lambda-detectors

  • New detectors from Dectris (Pilatus 900k and Eiger 250k)

  • Support Nexus format in output: NXmonpd and NXcansas

  • Single-threaded CSC sparse matrix multiplication engine

• Improved uncertainty propagation:

  • Refactor error model management (uses enum)

  • Hybrid error model (azimuthal for sigma-clipping but reports Poissonian noise)

  • Export peakfinder data to the CXI format (used by CrystFEL)

• Improvement in the doc:

  • Update installation instructions

  • Multi-threaded integration tutorial

  • GPU implementation tutorial

• Facts and figures:

  • 400+ commits, 100 PR

  +with the contribution of:

  Clemens Prescher, Elena Pascal, Jérôme Kieffer, Malte Storm, Marco Cammarata, Michael Hudson-Doyle, Picca Frédéric-Emmanuel, Rodrigo Telles, Thomas A Caswell, Tommaso Vinci, Valentin Valls, Wout de Nolf.

0.21.3 24/03/2022

• Support hexagonal pixels in splitbbox and nosplit mode (collaboration with EuXFEL)

• Support extra dtype in OpenCL (contribution from Desy)

• Fix version of setuptools (enables to build pyFAI -> python 3.12)

• Support some of the Lambda detectors from XSpectrum (collaboration with Xmas)

• Restore back compatibility in histoBBox1d and histoBBox2d (contribution from NSLS-II)

• Fix bug in OpenCL distortion correction (collaboration with Soleil)

• Several other bugs fixed

0.21.2 03/02/2022

Fix documentation of sparsify-Bragg and peakfinder CLI tools

0.21.1 02/02/2022

Bug-fix release with: - issue in propagated error not reported with certain engines - diff_map tool can now perform 2D integration - issue with dummy/dela_dummy in Worker - fix improper reset of the integrator due to cabling issue inside AzimuthalIntegrator - Remove deprecation warning about scipy - Use silx for median-filter which is faster than scipy - skip 3 tests which are known to be fragile on 32-bit platforms (spotted by debian on i386 and armel)

0.21.0 20/01/2022

• One year of development: 523 commits, close to 100 pull-requests. +53000 lines of code and 20000 suppressed.

• Sigma-clipping allows separation of Bragg/amorphous signal:

  • Implementation in Python, Cython and OpenCL with poissonian and azimuthal error-model

  • Sparsification, compression of single crystal data

    • Application to peak-picking and quality assessement of SSX data

    • Analysis of grid-scan to find single crystal

    • Single pass variance propagation in azimuthal bin

    • Integration of the Jungfrau detector (ID29)

• 2D integration:

  • New integrators with error propagation by default

  • Full pixel splitting in addition to BBox and no splitting

• Refactor of all LUT and CSR to share the same code base, makes tests more robust.

• Calibration of experimental setup using Jupyter notebooks

  • User interaction with plots in matplotlib (thanks Phil Hans)

  • Factorize code between pyFAI-calib and jupyter calibration

  • Tutorial as notebook and video recording

• Parallax correction for thick detector (still experimental, thanks to Vadim)

• Improved tutorial on detector geometry calibration (based on Kabsch alignment)

• Better performances on HPC nodes by limiting simple OpenMP to fewer cores

• Many improvement in test, typos fixed, doc …

• Deprecate all legacy integrators since the new version sees its matrix complete.

• Supports Python 3.6 … 3.10 under Windows, MacOS and Linux.

0.20.0 22/01/2021

• One year of development, about 500 commits & 370 files modified.

• Generalization of the new generation 1D integrators (better error propagation)

• Sigma clipping and sparsification of single crystal data (OpenCL only)

• Fix issue introduced with the scipy 1.15 (constrained calibration broken)

• Improved distortion correction (also on GPU, …)

• Major re-work of the documentation (thanks Thomas Kluyver and Loic Huder)

• Improve the calibration of Pilatus and Eiger detectors based on a grid of holes.

• New cylindrical detector from Rigaku

• Drop deprecated OpenCL integrator

• Support all Eiger2 detectors (thanks Clemens Weninger and Marie Ruat)

• CI: move to Gihub workflow and gitlab-runners (bob) for building (thanks Thomas Vincent).

• Build for debian 10 and 11 (also ubuntu 20.04), drop debian 9

• Remove Python2 related code

• Lower Numpy ABI dependency as much as possible (remains _distortion’s C++ code)

• Drop tests for Python 3.5, checked on 3.9 as well.

0.19.0 31/01/2020

• Minor revison with only 150 commits, mainly bug-fixes

• Improvement on the GUI with many small bug-fixes

• Support newer h5py (mode mandatory, [()], …)

• Build for debian 10 and 11 (also ubuntu 20.04)

• Drop tests for Python 2 and 3.4, checked on 3.8

• Improved compatibility with ImageD11

• Use hdf5plugin to provide hdf5 io-filters in apps

• Rework diffraction mapping tools to use a worker

• New generation azimuthal integrator using CSR algorithm implemented in Python, Cython and OpenCL.

• Sigma-clipping implemented in OpenCL

• Publication on new generation integration, the GUI for calibration and the goniometers accepted in J. Synch. Rad. DOI: 10.1107/S1600577520000776

• A big thank to Florian from Germany, Bertrand from Xenocs, Alex from Soleil and Jon from ESRF for their contributions.

0.18.0 15/05/2019

• Last release with Valentin as he finishes his contract soon

• almost 800 commits, 60 PR since the last release: this is a huge release !

• Major rework on all GUIs, mainly pyFAI-calib2 and pyFAI-integrate.

• Possibility to integrate image stacks (i.e. from HDF5), …

• Rework the method to specify the algorithm, pixel splitting and implementation with sensible fall-backs. Also available via the different GUIs

• 3D visualization of detectors and experimental setup, useful for non flat detectors.

• integrate1d_ng is available with histogramming without pixel splitting in Python, Cython and OpenCL. Now, propagates the variance properly !

• IO sub-packages with associated refactoring for ponifile, json, …

• Improved management of OpenMP: simplify the code for histograms.

• Improved geometry description and tutorial for writing exchange with other software (ImageD11, thanks to Carsten Detlefs).

• More reliable simple ellipse fitting with tests and doc.

• Better POCL integration (debugged on cuda, x87, Power9, …)

• Rely on silx mechanics for the build, test, download, GUI, opencl …

• Many new tutorials, now available on binder-hub: new calibrants, Pilatus calibration, …

• Fix many issues reported in third-party software (Dioptas, …)

• Drop support of debian8, Python 2.7 and Python 3.4 on all platforms. It is likely most functionalities still work but without guaranty.

0.17.0 19/12/2018

• Only 200 commits in a couple of month, this is a small release

• Fix major bugs in pyFAI-calib2 (double validator, initial guess, ring position)

• Constrains have been added to the geometry fitting of pyFAI-calib2

• New pyFAI-integrate graphical application

• Much better support for user defined detector (HDF5)

• Start the rewrite of all integrators to allow proper error propagation (2D done)

• Factorize the preprocessing steps for many integrators

• Remove tons of code which has been deprecated for years in AzimuthalIntegrator

• Featuring contribution from Soleil and Berkeley

• Stop supporting Python2.7 on Windows (there won’t be wheels!)

• All scripts are now using Python 3.x (x>=4)

• This is the last release supporting Python 2.7, 3.4 hence debian 8

0.16.0: 26/10/2018

• Almost 800 commits since 0.15 !

• Huge improvements on the graphical application for calibration

• New detector definition (with manufacturers)

• Improved tests: the GUI is now tested

• Preparation for changing all rebinning engines (see variance tutorials)

• Azimuthal integrators (and most other objects) are now serializable with Pickle

• New distortion correction using the SparseBuilder C++ code

• New PONI-file format (detector definition changed)

• Isocontour is now provided by silx

• Peak-picker clean up (better peak selection near gaps)

• new Goniometer refinement with enhanced rotation using Euler angles

• Updated documentation: new cookbooks and tutorials about: - The use of the calibration application (cookbook) - log-scale integration of SAXS data (notebook) - Variance propagation (notebook)

0.15.0: 01/02/2018

• 150 commits since last revision

• New tutorials on image inpainting, sensor thickness correction, …

• Improve scripts

• Improve the new calibration GUI (pyFAI-calib2)

• Use scipy physical constants instead of hard-coded values

• Improved detector serialization and binning assessement

• Update documentation (images, better rendering of notebook & tutorials)

• Converge project with silx and fabio

• Remove generated rst- and C-files from repository

• This is the last version supporting python2.7

0.14.2: 14/09/2017

• Fix seg-fault with manylinux1 wheels, in fastcrc module (thanks Thomas)

• Fix Qt4-Qt5 compatibility (thanks Vadim)

• Easier to understand geometry transformation (thanks Jon)

• Lower memory consumption, better cache management

• Unified debian packaging working for 6->9

• New detector: Mythen & CirPad (thanks Fred)

• Clean up debug code which avoid to use pyFAI-calib2

• pyFAI-calib2 now expect fabio >= 0.5

• Fix issue with metadata saving in 1d

• Fix performance regression with pyopencl >2015.2 (Thanks Andreas)

• pyFAI saxs and waxs scripts guess now the binning of the detector (thanks Fred).

0.14.1: 25/07/2017

• Fixes Debian 7 and 8 packages

0.14.0: 20/07/2017

• Graphical user interface for calibration (pyFAI-calib2)

• Goniometer calibration tools and multi-geometry enhancements

• Integration scripts and averaging scripts are now able to normalize the data from monitors found in the header.

• Propagate metadata information as part of the integrated data

• Common pre-processing factorization on Python, Cython and OpenCL

• Test clean up and acceleration (avoid tests on too large images)

• Many new tutorials http://pyfai.readthedocs.io/en/latest/usage/tutorial/index.html

• New averaging / integration methods: - Azimuthal median filtering - Azimuthal trimmed mean - sigma-clipping on azimuthal angle - Radial averaging

• Diffraction image inpainting to fill-up the gaps with plausible values.

• This release correspond to 572 commits

• Change of license: now all pyFAI is MIT license.

0.13.0: 01/12/2016

• Global improvement of tests, packaging, code quality, documentation and project tools

• Scripts

  • Add support for multiframe formats on pyFAI-average

  • Add support for monitoring correction from header file (on pyFAI-average)

  • Add progressbar in the shell (on pyFAI-average and pyFAI-integrate)

  • Script drawMask_pymca is renamed into pyFAI-drawmask

  • Rework of the drawmask GUI using silx

  • pyFAI-drawmask do not have anymore hard dependency on PyMCA

  • pyFAI-integrate can now be used without qt dependency (–no-gui)

  • Fix the script to support both Python 2 and 3 (pyFAI-calib, pyFAI-benchmark)

  • Fix selection of units on diff-map (the user selection was not propagated)

• For users

  • More source code in MIT license

  • Update name and specification for cameras

  • Add cameras: Eiger500k, RaspberryPi5M, RaspberryPi8M

  • Fix Xpad S540 flat detector geometry

  • Fix definition of CeO2 calibrant

  • Add mask and flat on multi-geometry

  • Fix solid angle of the multi-geometry

  • Fix geometry processing for custom output space

  • Fix normalization factor and variance

  • Add support for Qt5

  • Add support for Debian 9 packaging

• For developers

  • Create common preprocessing for distortion correction

  • Create common image preprocessing using Cython (NaN filter, flatfield, dark, polarisation)

  • Refactoring of units module. It allows to register custom units.

  • Worker can now use Writer

  • Worker polarization argument is renamed into polarization_factor

  • Remove the dependency from python-fftw3, use numpy instead

  • Remove QtWebKit dependency

  • Fix un-correction of images using sparse matrix from scipy

0.12.0: 06/06/2016

• Continuous integration on linux, windows using Python 2.7 and 3.4+

• Drop support of Python 2.6, 3.2, 3.3 and debian6 packaging

• New radial output units: Reciprocal spacing squared and log(q) ID02

• GPU accelerate version of ai.separate (Bragg & amorphous) ID13

• Quantile filtering in pyFAI-average ID02

• New graphical application for diffraction imaging ID21

• Migrate to a common structure with silx (reorganize tests, benchmarks, …)

• Extensions (binary sub-modules) have all been moved to ext directory

• Many improvements multigeometry integrators

• Compatibility with the copy module (copy.deepcopy) for azimuthal integrator ID02

• Distortion correction works also for non-contiguous detectors

• Update documentation and provide advanced tutorials:

  • Introduction to pyFAI using the jupyter notebook

  • detector calibration ID15, BM02

  • Correction of detector distortion, examples of pixel detectors.

  • calibrant calculation ID30

  • error handling ID02, BM29

• pyFAI-integrate can now be used with or without GUI

• Many new detectors (ADSC, Pilatus CdTe, Apex II, Pixium):

  • support for non-flat/curved detectors (Aarhus)

  • non-contiguous detectors (WOS Xpad)

• Include tests and benchmarking tools as part of the library

• Better testing.

0.11.0: 07/2015

• All calibrant from NIST are now available, + Nickel, Aluminum, … with bibliographic references

• The Cell class helps defining new calibrants.

• OpenCL Bitonic sort (to be integrated into Bragg/Amorphous separation)

• Calib is available from the Python interface (procedural API), not only from the shell script.

• Many new options in calib for reset/assign/delete/validate/validate2/chiplot.

  • reset: set the detector, orthogonal, centered and at 10cm

  • assign: checks the assignment of groups of points to rings

  • delete: remove a group of peaks

  • validate: autocorrelation of images: error on the center

  • validate2: autocorrelation of patterns at 180° apart: error on the center function of chi

  • chiplot: assesses the quality of control points of one/multiple rings.

• Fix the regression of the initial guess in calib (Thanks Jon Wright)

• New peak picking algorithm named “watershed” and based on inverse watershed for ridge recognition

• start factorizing cython regridding engines (work ongoing)

• Add “–poni” option for pyFAI-calib (Thanks Vadim Dyakin)

• Improved “guess_binning”, especially for Perkin Elmer flat panel detectors.

• Support for non planar detectors like Curved Imaging plate developped at Aarhus

• Support for Multi-geometry experiments (tested)

• Speed improvement for detector initialization

• better isotropy in peak picking (add penalization term)

• enhanced documentation on http://pyfai.readthedocs.org

0.10.3: 03/2015

• Image segmentation based on inverse watershed (only for recalib, not for calib)

• Python3 compatibility

• include testimages into distribution

0.10.2: 11/2014

• Update documentation

• Packaging for debian 8

0.10.1: 10/2014

• Fix issue in peak-picking

• Improve doc & manpages

• Compatibility with PyMca5

0.10.0: 10/2014

• Correct Caglioti’s formula

• Update tests and OpenCL -> works with Beignet and pocl open source drivers

• Compatibility with MacOSX and windows

0.9.4: 06/2014

• include spec of Maxwell GPU

• fix issues with intel OpenCL icd v4.4

• introduce shape & max_shape in detectors

• work on marchingsquares/sorted controurplot for calibration

• Enforce the use the Qt4Agg for Matplotlib and other GUI stuff.

• Update shape of detector in case of binning

• unified distortion class: merge OpenCL & OpenMP implementation #108

• Benchmarks for distortion

• Raise the level to warning when inverting the mask

• set of new ImXpad detectors Related issue #111

• Fix issue with recalib within MX-calibrate

• saving detector description in Nexus files issue #110

• Update some calibrants: gold

• about to make peak-picking more user-friendly

• test for bragg separation

• work on PEP8 compliance

• Do not re-cythonize: makes debian package generation able to benefit from ccache

• conversion to SPD (rotation is missing)

• pixelwise worker

• correct both LUT & OCL for memory error

• replace os.linsep with “n” when file file opened in text mode (not binary)

• rework the Extension part to be explicit instead of “black magic” :)

• implement Kahan summation in Cython (default still use Doubles: faster)

• Preprocessing kernel containing all cast to float kernels #120

• update setup for no-openmp option related to issue #127

• Add read-out mode for mar345 as “guess_binning” method for detector. Also for MAR and Rayonix #125

• tool to benchmark HDF5 writing

• try to be compatible with both PySide and PyQt4 … the uic stuff is untested and probably buggy #130

• Deactivate the automatic saturation correction by default. now it is opt-in #131

0.9.3: 02/2014

• Better control for peak-picking (Contribution from Gero Flucke, Desy)

• Precise Rayonix detectors description thanks to Michael Blum

• Start integrating blob-detection algorithm for peak-picking: #70

• Switch fron OptParse to ArgPrse: #83

• Provide some calibrant by default: #91

• Description of Mar345 detector + mask#92

• Auto-registration of detectors: #97

• Recalib and check-calib can be called from calib: #99

• Fake diffraction image from calibrant: #101

• Implementation of the CSR matrix representation to replace LUT

• Tight pixel splitting: #43

• Update documentation

0.9.2: (01/2014)

• Fix memory leak in Cython part of the look-up table generation

• Benchmarks with memory profiling

0.9: 10/2013

• Add detector S140 from ImXpad, Titan from Agilent, Rayonix

• Fix issues: 61, 62, 68, 76, 81, 82, 85, 86, 87

• Enhancement in LImA plugins (better structure)

• IO module with Ascii/EDF/HDF5 writers

• Switch some GUI to pyQtGraph in addition to Qt

• Correction for solid-angle formula

0.8: 10/2012

• Detector object is member of the geometry

• Binning of the detector, propagation to the spline if needed

• Detector object know about their masks.

• Automatic mask for some detectors like Pilatus or XPad

• Implementation of sub-pixel position correction for Pilatus detectors

• LUT implementation in 1D & 2D (fully tested) both with OpenMP and with OpenCL

• Switch from C++/Cython OpenCL framework to PyOpenCL

• Port opencl code to both Windows 32/64 bits and MacOSX

• Add polarization corrections

• Use fast-CRC checksum on x86 using SSE4 (when available) to track array change on GPU buffers

• Support for flat 7*8 modules Xpad detectors.

• Benchmark with live graphics (still a memory issue with python2.6)

• Fat source distribution (python setup.py sdist –with-test-images) for debian

• Enhanced tests, especially for Saxs and OpenCL

• Recalibration tool for refining automatically parameters

• Enhancement of peak picking (much faster, recoded in pure Cython)

• Easy calibration for pixel detector (reconstruction of inter-module space)

• Error-bar generation using Poisson law

• Unified programming interface for all integration methods in 2theta, q or radius unit

• Graphical interface for azimuthal integration (pyFAI-integrate)

• Lots of test to prevent non regression

• Tool for merging images using various method (mean, median) and with outlayer rejection

• LImA plugin which can perform azimuthal integration live during the acquisition

• Distortion correction is available alone and as LImA plugin

• Recalibration can refine the wavelength in addition to 6 other parameters

• Calibration always done vs calibrant’s ring number, lots of new calibrant are available

• Selection by hand of single peaks for calibration

• New detectors: Dexela and Perkin-Elmer flat panel

• Automatic refinement of multiple images at various geometries (for MX)

• Many improvements requested by ID11 and ID13

0.7.2: 08/2012

• Add diff_tomo script

• Geometry calculation optimized in (parallel) cython

0.7: 07/2012

Implementation of look-up table based integration and OpenCL version of it

0.6: 07/2012

• OpenCL flavor works well on GPU in double precision with device selection

0.5: 06/2012

• Include OpenCL version of azimuthal integration (based on histograms)

0.4: 06/2012

• Global clean up of the code regarding options from command line and better design

• Correct the orientation of the azimuthal angle chi

• Rename scripts in pyFAI-calib, pyFAI-saxs and pyFAI-waxs

0.3: 11/2011

• Azimuthal integration splits pixels like fit2d

0.2: 07/2011

• Azimuthal integration using cython histogramming is working

0.1: 05/2011

• Geometry is OK