Introduction to the tutorials¶
The iPython/Jupyter notebook¶
The document you are seeing it may be a PDF or a web page or another format, but what is important is how it has been made ...
According to this article in Nature the Notebook, invented by iPython, and now part of the Jupyter project, is the revolution for data analysis which will allow reproducable science.
This tutorial will introduce you to how to access to the notebook, how to use it and perform some basic data analysis with it and the pyFAI library.
Getting access to the notebook¶
There are many cases ...
Inside ESRF¶
The simplest case as the data analysis unit offers you a notebook server: just connect your web browser to http://scisoft13:8000 and authenticate with your ESRF credentials.
Outside ESRF with an ESRF account.¶
The JupyterHub server is not directly available on the internet, but you can login into the firewall to forward the web server:
ssh -XC -p5022 -L8000:scisoft13:8000 user@firewall.esrf.fr
Once logged in ESRF, keep the terminal opened and browse on your local computer to http://localhost:8000 to authenticate with your ESRF credentials. Do not worry about confidentiality as the connection from your computer to the ESRF firewall is encrypted by the SSH connection.
Other cases¶
In the most general case you will need to install the notebook on your local computer in addition to silx, pyFAI and FabIO to follow the tutorial. WinPython provides it under windows. Please refer to the installation procedure of pyFAI to install locally pyFAI on your computer. Anaconda provides also pyFAI and FabIO as packages.
Getting trained in using the notebook¶
There are plenty of good tutorials on how to use the notebook. This one presents a quick overview of the Python programming language and explains how to use the notebook. Reading it is strongly encouraged before proceeding to the pyFAI itself.
Anyway, the most important information is to use Control-Enter to evaluate a cell.
In addition to this, we will need to download some files from the internet. The following cell contains a piece of code to download files using the silx library. You may have to adjust the proxy settings to be able to connect to internet, especially at ESRF, see the commented out code.
Introduction to diffraction image analysis using the notebook¶
All the tutorials in pyFAI are based on the notebook and if you wish to practice the exercises, you can download the notebook files (.ipynb) from Github
Load and display diffraction images¶
First of all we will download an image and display it. Displaying it the right way is important as the orientation of the image imposes the azimuthal angle sign.
In [1]:
import os, time
#Nota: comment out when outside ESRF
os.environ["http_proxy"] = "http://proxy.esrf.fr:3128"
start_time = time.time()
import pyFAI
print("Using pyFAI version", pyFAI.version)
Using pyFAI version 0.16.0-dev0
/users/kieffer/VirtualEnvs/py3/lib/python3.5/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
from ._conv import register_converters as _register_converters
In [2]:
from silx.resources import ExternalResources
downloader = ExternalResources("pyFAI", "http://www.silx.org/pub/pyFAI/testimages/", "DATA")
In [3]:
moke = downloader.getfile("moke.tif")
print(moke)
/tmp/pyFAI_testdata_kieffer/moke.tif
The moke.tif image we just downloaded is not a real diffraction image but it is a test pattern used in the tests of pyFAI.
Prior to displaying it, we will use the Fable Input/Output library to read the content of the file:
In [4]:
#initializes the visualization module to work with the jupyter notebook
%pylab nbagg
Populating the interactive namespace from numpy and matplotlib
In [5]:
import fabio
from pyFAI.gui import jupyter
img = fabio.open(moke).data
jupyter.display(img, label ="Fake diffraction image")
Out[5]:
<matplotlib.axes._subplots.AxesSubplot at 0x7f36964c38d0>
As you can see, the image looks like an archery target. The origin is at the lower left of the image. If you are familiar with matplotlib, it correspond to the option origin=”lower” of imshow.
Displaying the image using imsho without this option ends with having the azimuthal angle (which angles are displayed in degrees on the image) to turn clockwise, so the inverse of the trigonometric order:
In [6]:
fig, ax = subplots(1,2, figsize=(10,5))
ax[0].imshow(img, origin="lower")
ax[0].set_title("Proper orientation")
ax[1].imshow(img)
ax[1].set_title("Wrong orientation")
Out[6]:
Text(0.5,1,'Wrong orientation')
Nota: Displaying the image properly or not does not change the content of the image nor its representation in memory, it only changes its representation, which is important only for the user. DO NOT USE numpy.flipud or other array-manipulation which changes the memory representation of the image. This is likely to mess-up all your subsequent calculation.
1D azimuthal integration¶
To perform an azimuthal integration of this image, we need to create an AzimuthalIntegrator object we will call ai. Fortunately, the geometry is explained on the image.
In [7]:
import pyFAI, pyFAI.detectors
detector = pyFAI.detectors.Detector(pixel1=1e-4, pixel2=1e-4)
ai = pyFAI.AzimuthalIntegrator(dist=0.1, detector=detector)
# Short version ai = pyFAI.AzimuthalIntegrator(dist=0.1, pixel1=1e-4, pixel2=1e-4)
print(ai)
WARNING:pyFAI.DEPRECATION:function, AzimuthalIntegrator is deprecated since silx version 0.16! Use 'pyFAI.azimuthalIntegrator.AzimuthalIntegrator' instead.
File "<ipython-input-7-403adc458771>", line 3, in <module>
ai = pyFAI.AzimuthalIntegrator(dist=0.1, detector=detector)
Detector Detector Spline= None PixelSize= 1.000e-04, 1.000e-04 m
SampleDetDist= 1.000000e-01m PONI= 0.000000e+00, 0.000000e+00m rot1=0.000000 rot2= 0.000000 rot3= 0.000000 rad
DirectBeamDist= 100.000mm Center: x=0.000, y=0.000 pix Tilt=0.000 deg tiltPlanRotation= 0.000 deg
Printing the ai object displays 3 lines:
- The detector definition, here a simple detector with square, regular pixels with the right size
- The detector position in space using the pyFAI coordinate system: dist, poni1, poni2, rot1, rot2, rot3
- The detector position in space using the FIT2D coordinate system: direct_sample_detector_distance, center_X, center_Y, tilt and tilt_plan_rotation
Right now, the geometry in the ai object is wrong. It may be easier to define it correctly using the FIT2D geometry which uses pixels for the center coordinates (but the sample-detector distance is in millimeters).
In [8]:
help(ai.setFit2D)
Help on method setFit2D in module pyFAI.geometry:
setFit2D(directDist, centerX, centerY, tilt=0.0, tiltPlanRotation=0.0, pixelX=None, pixelY=None, splineFile=None) method of pyFAI.azimuthalIntegrator.AzimuthalIntegrator instance
Set the Fit2D-like parameter set: For geometry description see
HPR 1996 (14) pp-240
Warning: Fit2D flips automatically images depending on their file-format.
By reverse engineering we noticed this behavour for Tiff and Mar345 images (at least).
To obtaine correct result you will have to flip images using numpy.flipud.
:param direct: direct distance from sample to detector along the incident beam (in millimeter as in fit2d)
:param tilt: tilt in degrees
:param tiltPlanRotation: Rotation (in degrees) of the tilt plan arround the Z-detector axis
* 0deg -> Y does not move, +X goes to Z<0
* 90deg -> X does not move, +Y goes to Z<0
* 180deg -> Y does not move, +X goes to Z>0
* 270deg -> X does not move, +Y goes to Z>0
:param pixelX,pixelY: as in fit2d they ar given in micron, not in meter
:param centerX, centerY: pixel position of the beam center
:param splineFile: name of the file containing the spline
In [9]:
ai.setFit2D(100, 300, 300)
print(ai)
Detector Detector Spline= None PixelSize= 1.000e-04, 1.000e-04 m
SampleDetDist= 1.000000e-01m PONI= 3.000000e-02, 3.000000e-02m rot1=0.000000 rot2= 0.000000 rot3= 0.000000 rad
DirectBeamDist= 100.000mm Center: x=300.000, y=300.000 pix Tilt=0.000 deg tiltPlanRotation= 0.000 deg
With the ai object properly setup, we can perform the azimuthal integration using the intergate1d method. This methods takes only 2 mandatory parameters: the image to integrate and the number of bins. We will provide a few other to enforce the calculations to be performed in 2theta-space and in degrees:
In [10]:
res = ai.integrate1d(img, 300, unit="2th_deg")
#Display the integration result
fig, ax = subplots(1,2, figsize=(10,5))
jupyter.plot1d(res, label="moke",ax=ax[0])
#Example using pure matplotlib
tth = res[0]
I = res[1]
ax[1].plot(tth, I, label="moke")
ax[1].set_title("Display 1d powder diffraction data using pure matplotlib")
Out[10]:
Text(0.5,1,'Display 1d powder diffraction data using pure matplotlib')
As you can see, the 9 rings gave 9 sharp peaks at 2theta position regularly ranging from 4 to 12 degrees as expected from the image annotation.
Nota: the default radial unit is “q_nm^1”, so the scattering vector length expressed in inverse nanometers. To be able to calculate q, one needs to specify the wavelength used (here we didn’t). For example: ai.wavelength = 1e-10
To save the content of the integrated pattern into a 2 column ASCII file, one can either save the (tth, I) arrays, or directly ask pyFAI to do it by providing an output filename:
In [11]:
ai.integrate1d(img, 30, unit="2th_deg", filename="moke.dat")
# now display the content of the file
with open("moke.dat") as fd:
for line in fd:
print(line.strip())
# == pyFAI calibration ==
# Distance Sample to Detector: 0.1 m
# PONI: 3.000e-02, 3.000e-02 m
# Rotations: 0.000000 0.000000 0.000000 rad
#
# == Fit2d calibration ==
# Distance Sample-beamCenter: 100.000 mm
# Center: x=300.000, y=300.000 pix
# Tilt: 0.000 deg TiltPlanRot: 0.000 deg
#
# Detector Detector Spline= None PixelSize= 1.000e-04, 1.000e-04 m
# Detector has a mask: False
# Detector has a dark current: False
# detector has a flat field: False
#
# Mask applied: False
# Dark current applied: False
# Flat field applied: False
# Polarization factor: None
# Normalization factor: 1.0
# --> moke.dat
# 2th_deg I
3.831631e-01 6.384597e+00
1.149489e+00 1.240657e+01
1.915815e+00 1.222277e+01
2.682142e+00 1.170348e+01
3.448468e+00 9.964797e+00
4.214794e+00 8.913503e+00
4.981120e+00 9.104074e+00
5.747446e+00 9.242975e+00
6.513773e+00 6.136262e+00
7.280099e+00 9.039030e+00
8.046425e+00 9.203415e+00
8.812751e+00 9.324570e+00
9.579077e+00 6.470129e+00
1.034540e+01 7.790758e+00
1.111173e+01 9.410036e+00
1.187806e+01 9.464832e+00
1.264438e+01 7.749059e+00
1.341071e+01 1.151200e+01
1.417703e+01 1.324891e+01
1.494336e+01 1.038730e+01
1.570969e+01 1.069764e+01
1.647601e+01 1.056094e+01
1.724234e+01 1.286720e+01
1.800867e+01 1.323240e+01
1.877499e+01 1.548398e+01
1.954132e+01 2.364553e+01
2.030764e+01 2.537155e+01
2.107397e+01 2.512984e+01
2.184030e+01 2.191267e+01
2.260662e+01 7.605135e+00
This “moke.dat” file contains in addition to the 2th/I value, a header commented with “#” with the geometry used to perform the calculation.
**Nota: ** The ai object has initialized the geometry on the first call and re-uses it on subsequent calls. This is why it is important to re-use the geometry in performance critical applications.
2D integration or Caking¶
One can perform the 2D integration which is called caking in FIT2D by simply calling the intrgate2d method with 3 mandatroy parameters: the data to integrate, the number of radial bins and the number of azimuthal bins.
In [12]:
res2d = ai.integrate2d(img, 300, 360, unit="2th_deg")
#Display the integration result
fig, ax = subplots(1,2, figsize=(10,5))
jupyter.plot2d(res2d, label="moke",ax=ax[0])
#Example using pure matplotlib
I, tth, chi = res2d
ax[1].imshow(I, origin="lower", extent=[tth.min(), tth.max(), chi.min(), chi.max()], aspect="auto")
ax[1].set_xlabel("2 theta (deg)")
ax[1].set_ylabel("Azimuthal angle chi (deg)")
Out[12]:
Text(0,0.5,'Azimuthal angle chi (deg)')
The displayed images the “caked” image with the radial and azimuthal angles properly set on the axes. Search for the -180, -90, 360/0 and 180 mark on the transformed image.
Like integrate1d, integrate2d offers the ability to save the intgrated image into an image file (EDF format by default) with again all metadata in the headers.
Radial integration¶
Radial integration can directly be obtained from Caked images:
In [13]:
target = 8 #degrees
#work on fewer radial bins in order to have an actual averaging:
I, tth, chi = ai.integrate2d(img, 100, 90, unit="2th_deg")
column = argmin(abs(tth-target))
print("Column number %s"%column)
fig, ax = subplots()
ax.plot(chi, I[:,column], label=r"$2\theta=%.1f^{o}$"%target)
ax.set_xlabel("Azimuthal angle")
ax.set_ylabel("Intensity")
ax.set_title("Radial intgration")
ax.legend()
Column number 34
Out[13]:
<matplotlib.legend.Legend at 0x7f366c05c208>
Nota: the pattern with higher noise along the diagonals is typical from the pixel splitting scheme employed. Here this scheme is a “bounding box” which makes digonal pixels look a bit larger (+40%) than the ones on the horizontal and vertical axis, explaining the variation of the noise.
Integration of a bunch of files using pyFAI¶
Once the processing for one file is established, one can loop over a bunch of files. A convienient way to get the list of files matching a pattern is with the glob module.
Most of the time, the azimuthal integrator is obtained by simply loading the poni-file into pyFAI and use it directly.
In [14]:
all_files = downloader.getdir("alumina.tar.bz2")
all_edf = [i for i in all_files if i.endswith("edf")]
all_edf.sort()
print("Number of EDF downloaded: %s"%len(all_edf))
Number of EDF downloaded: 52
In [15]:
ponifile = [i for i in all_files if i.endswith(".poni")][0]
splinefile = [i for i in all_files if i.endswith(".spline")][0]
print(ponifile, splinefile)
#patch the poni-file with the proper path.
with open(ponifile, "a") as f:
f.write("SplineFile: %s\n"%splinefile)
ai = pyFAI.load(ponifile)
print(ai)
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_00_max_51_frames.poni /tmp/pyFAI_testdata_kieffer/alumina/distorsion_2x2.spline
Detector Detector Spline= /tmp/pyFAI_testdata_kieffer/alumina/distorsion_2x2.spline PixelSize= 1.034e-04, 1.025e-04 m
Wavelength= 7.084811e-11m
SampleDetDist= 1.168599e-01m PONI= 5.295653e-02, 5.473342e-02m rot1=0.015821 rot2= 0.009404 rot3= 0.000000 rad
DirectBeamDist= 116.880mm Center: x=515.795, y=522.995 pix Tilt=1.055 deg tiltPlanRotation= 149.271 deg
In [16]:
fig, ax = subplots()
for one_file in all_edf:
destination = os.path.splitext(one_file)[0]+".dat"
image = fabio.open(one_file).data
t0 = time.time()
res = ai.integrate1d(image, 1000, filename=destination)
print("%s: %.3fs"%(destination, time.time()-t0))
jupyter.plot1d(res, ax=ax)
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0000.dat: 0.402s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0001.dat: 0.014s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0002.dat: 0.012s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0003.dat: 0.012s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0004.dat: 0.013s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0005.dat: 0.014s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0006.dat: 0.013s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0007.dat: 0.017s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0008.dat: 0.011s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0009.dat: 0.012s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0010.dat: 0.018s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0011.dat: 0.009s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0012.dat: 0.012s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0013.dat: 0.010s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0014.dat: 0.013s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0015.dat: 0.011s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0016.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0017.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0018.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0019.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0020.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0021.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0022.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0023.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0024.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0025.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0026.dat: 0.009s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0027.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0028.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0029.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0030.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0031.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0032.dat: 0.020s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0033.dat: 0.010s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0034.dat: 0.012s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0035.dat: 0.009s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0036.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0037.dat: 0.017s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0038.dat: 0.009s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0039.dat: 0.012s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0040.dat: 0.011s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0041.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0042.dat: 0.014s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0043.dat: 0.009s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0044.dat: 0.012s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0045.dat: 0.011s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0046.dat: 0.008s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0047.dat: 0.019s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0048.dat: 0.020s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0049.dat: 0.011s
/tmp/pyFAI_testdata_kieffer/alumina/al2o3_0050.dat: 0.011s
/tmp/pyFAI_testdata_kieffer/alumina/dark_0001.dat: 0.008s
This was a simple integration of 50 files, saving the result into 2 column ASCII files.
Nota: the first frame took 40x longer than the other. This highlights go crucial it is to re-use azimuthal intgrator objects when performance matters.
Conclusion¶
Using the notebook is rather simple as it allows to mix comments, code, and images for visualization of scientific data.
The basic use pyFAI’s AzimuthalIntgrator has also been presented and may be adapted to you specific needs.
In [17]:
print("Total execution time: %.3fs"%(time.time()-start_time))
Total execution time: 6.214s