Calibration of the pixel position for a Pilatus detector

This tutorial summarizes the work done by Frederic Sulzman during his internship at ESRF during the summer 2015 entilted "Calibration for geometric distortion in multi- modules pixel detectors".

The overall strategy is very similar to "CCD calibration" tutorial with some specificities due to the modular nature of the detector.

  1. Image preprocessing
  2. Peak picking
  3. Grid assignment
  4. Displacement fitting
  5. Reconstruction of the pixel position
  6. Saving into a detector definition file

Each module being made by lithographic processes, the error within a module will be assumeed to be constant. We will use the name "displacement of the module" to describe the rigide movement of the module.

This tutorial uses data from the Pilatus3 2M CdTe from the ID15 beam line of the ESRF. They provided not only the intersnip subject but also the couple of images uses to calibrate the detector.

This detector contains 48 half-modules, each bound to a single CdTe monocrystal sensor and is designed for high energy X-ray radiation detection. Due to the construction procedure, these half-modules could show a misalignment within the detector plane. While the manufacturer (Dectris) garanties a precision within a pixel (172µm), the miss-alignment of certain modules can be seen while calibrating Debye-Scherrer ring using refereance sample. So the aim of this work is to provide a detector description with a better precision better than the original detector.

This work will be performed on the image of a grid available: http://www.silx.org/pub/pyFAI/detector_calibration/Pilatus2MCdTe_ID15_grid_plus_sample_0004.cbf and the scattering of ceria (CeO2) at 72.1keV available here. http://www.silx.org/pub/pyFAI/detector_calibration/Pilatus2MCdTe_ID15_CeO2_72100eV_800mm_0000.cbf

It is a good exercise to calibrate all rings of the later image using the pyFAI-calib2 tool. A calibration close to perfection is needed to visualize the module miss-alignement we aim at correcting.

In [1]:
%matplotlib notebook
In [2]:
#many imports which will be used all along the notebook
import time
start_time = time.time()
import os
import pyFAI
import fabio
import glob
import numpy
from math import sin, cos, sqrt
from scipy.ndimage import convolve, binary_dilation
from scipy.spatial import distance_matrix
from scipy.optimize import minimize
from matplotlib.pyplot import subplots
from pyFAI.ext.bilinear import Bilinear
from pyFAI.ext.watershed import InverseWatershed
from silx.resources import ExternalResources

print("Using pyFAI verison: ", pyFAI.version)
Using pyFAI verison:  0.18.0-dev1
In [3]:
#Nota: comment out when running from outside ESRF
#os.environ["http_proxy"] = "http://proxy.esrf.fr:3128"
In [4]:
downloader = ExternalResources("detector_calibration", "http://www.silx.org/pub/pyFAI/detector_calibration/")
ring_file = downloader.getfile("Pilatus2MCdTe_ID15_CeO2_72100eV_800mm_0000.cbf")
print(ring_file)
grid_file = downloader.getfile("Pilatus2MCdTe_ID15_grid_plus_sample_0004.cbf")
print(grid_file)
/tmp/detector_calibration_testdata_kieffer/Pilatus2MCdTe_ID15_CeO2_72100eV_800mm_0000.cbf
/tmp/detector_calibration_testdata_kieffer/Pilatus2MCdTe_ID15_grid_plus_sample_0004.cbf
In [5]:
rings = fabio.open(ring_file).data
img = fabio.open(grid_file).data
fig,ax = subplots(1,2, figsize=(10,5))
ax[0].imshow(img.clip(0,1000), interpolation="bilinear")
ax[0].set_title("grid")
ax[1].imshow(numpy.arcsinh(rings), interpolation="bilinear")
ax[1].set_title("rings")
Out[5]:
Text(0.5, 1.0, 'rings')

Image processing

There are 3 pre-processing steps which are needed.

  1. Define for each module a unique identifier which will be used later on during the fitting procedure
  2. Define the proper mask: each module is the assembly of 4x2 sub-modules and there are (3) interpolated pixels between each sub-module, such "unreliable pixels should be masked out as well
  3. Correct the grid image by the smoothed image to have a constant background.
  4. Convolve the raw image with a typical hole shape to allow a precise spotting of the hole center.
In [6]:
# This is the default detector as definied in pyFAI according to the specification provided by Dectris:
pilatus = pyFAI.detector_factory("Pilatus_2m_CdTe")
print(pilatus)

mask1 = pilatus.mask
module_size = pilatus.MODULE_SIZE
module_gap = pilatus.MODULE_GAP
submodule_size = (96,60)
Detector Pilatus CdTe 2M	 PixelSize= 1.720e-04, 1.720e-04 m
In [7]:
#1 + 2 Calculation of the module_id and the interpolated-mask:
mid = numpy.zeros(pilatus.shape, dtype=int)
mask2 = numpy.zeros(pilatus.shape, dtype=int)
idx = 1
for i in range(8):
    y_start = i*(module_gap[0] + module_size[0])
    y_stop = y_start + module_size[0]
    for j in range(3):
        x_start = j*(module_gap[1] + module_size[1])
        x_stop = x_start + module_size[1]
        mid[y_start:y_stop,x_start: x_start+module_size[1]//2] = idx
        idx+=1
        mid[y_start:y_stop,x_start+module_size[1]//2: x_stop] = idx
        idx+=1
        mask2[y_start+submodule_size[0]-1:y_start+submodule_size[0]+2,
              x_start:x_stop] = 1
        for k in range(1,8):
            mask2[y_start:y_stop,
              x_start+k*(submodule_size[1]+1)-1:x_start+k*(submodule_size[1]+1)+2] = 1
In [8]:
#Extra masking
mask0 = img<0
#Those pixel are miss-behaving... they are the hot pixels next to the beam-stop
mask0[915:922,793:800] = 1
mask0[817:820,747:750] = 1
In [9]:
fig,ax = subplots(1,3, figsize=(10,4))
ax[0].imshow(mid, interpolation="bilinear")
ax[0].set_title("Module Id")

ax[1].imshow(mask2+mask1+mask0, interpolation="bilinear")
ax[1].set_title("Combined mask")

nimg = img.astype(float)
nimg[numpy.where(mask0+mask1+mask2)] = numpy.nan


ax[2].imshow(nimg)#, interpolation="bilinear")
ax[2].set_title("Nan masked image")
Out[9]:
Text(0.5, 1.0, 'Nan masked image')
In [11]:
# The Nan-masked image contains now only valid values (and Nan elsewhere). We will make a large median filter to 
# build up a smooth image without gaps.
#
# This function is backported from future version of numpy ... it allows to expose a winbowed view 
# to perform the nanmedian-filter
from numpy.lib.stride_tricks import as_strided
def sliding_window_view(x, shape, subok=False, readonly=True):
    """
    Creates sliding window views of the N dimensional array with the given window
    shape. Window slides across each dimension of `x` and extract subsets of `x`
    at any window position.
    Parameters
    ----------
    x : array_like
        Array to create sliding window views of.
    shape : sequence of int
        The shape of the window. Must have same length as the number of input array dimensions.
    subok : bool, optional
        If True, then sub-classes will be passed-through, otherwise the returned
        array will be forced to be a base-class array (default).
    readonly : bool, optional
        If set to True, the returned array will always be readonly view.
        Otherwise it will return writable copies(see Notes).
    Returns
    -------
    view : ndarray
        Sliding window views (or copies) of `x`. view.shape = x.shape - shape + 1
    See also
    --------
    as_strided: Create a view into the array with the given shape and strides.
    broadcast_to: broadcast an array to a given shape.
    Notes
    -----
    ``sliding_window_view`` create sliding window views of the N dimensions array
    with the given window shape and its implementation based on ``as_strided``.
    Please note that if readonly set to True, views are returned, not copies
    of array. In this case, write operations could be unpredictable, so the returned
    views are readonly. Bear in mind that returned copies (readonly=False) will
    take more memory than the original array, due to overlapping windows.
    For some cases there may be more efficient approaches to calculate transformations
    across multi-dimensional arrays, for instance `scipy.signal.fftconvolve`, where combining
    the iterating step with the calculation itself while storing partial results can result
    in significant speedups.
    Examples
    --------
    >>> i, j = np.ogrid[:3,:4]
    >>> x = 10*i + j
    >>> shape = (2,2)
    >>> np.lib.stride_tricks.sliding_window_view(x, shape)
    array([[[[ 0,  1],
             [10, 11]],
            [[ 1,  2],
             [11, 12]],
            [[ 2,  3],
             [12, 13]]],
           [[[10, 11],
             [20, 21]],
            [[11, 12],
             [21, 22]],
            [[12, 13],
             [22, 23]]]])
    """
    np = numpy
    # first convert input to array, possibly keeping subclass
    x = np.array(x, copy=False, subok=subok)

    try:
        shape = np.array(shape, np.int)
    except:
        raise TypeError('`shape` must be a sequence of integer')
    else:
        if shape.ndim > 1:
            raise ValueError('`shape` must be one-dimensional sequence of integer')
        if len(x.shape) != len(shape):
            raise ValueError("`shape` length doesn't match with input array dimensions")
        if np.any(shape <= 0):
            raise ValueError('`shape` cannot contain non-positive value')

    o = np.array(x.shape) - shape  + 1 # output shape
    if np.any(o <= 0):
        raise ValueError('window shape cannot larger than input array shape')

    if type(readonly) != bool:
        raise TypeError('readonly must be a boolean')

    strides = x.strides
    view_strides = strides

    view_shape = np.concatenate((o, shape), axis=0)
    view_strides = np.concatenate((view_strides, strides), axis=0)
    view = as_strided(x, view_shape, view_strides, subok=subok, writeable=not readonly)

    if not readonly:
        return view.copy()
    else:
        return view
In [12]:
%%time 
#Calculate a background image using a large median filter ... takes a while
shape = (19,11)
print(nimg.shape)
padded = numpy.pad(nimg, tuple((i//2,) for i in shape), mode="edge")
print(padded.shape)
background = numpy.nanmedian(sliding_window_view(padded,shape), axis = (-2,-1))
print(background.shape)
fig,ax = subplots()
ax.imshow(background)
ax.set_title("Background image")
(1679, 1475)
(1697, 1485)
(1679, 1475)
CPU times: user 26.9 s, sys: 5.72 s, total: 32.6 s
Wall time: 45 s
In [13]:
fig,ax = subplots(1,2, figsize=(9,5))

normalized = (nimg/background)

low = numpy.nanmin(normalized)
high = numpy.nanmax(normalized)
print(low, high)
normalized[numpy.isnan(normalized)] = 0
normalized /= high

ax[0].imshow(normalized)
ax[0].set_title("Normalized image")

ax[1].hist(normalized.ravel(), 100, range=(0,1))
ax[1].set_title("Histogram of intensities in normalized image")
0.0 17.728813559322035
Out[13]:
Text(0.5, 1.0, 'Histogram of intensities in normalized image')

For a precise measurement of the peak position, one trick is to convolve the image with a pattern which looks like a hole of the grid.

In [14]:
#print the profile of the normalized image: the center is difficult to measure due to the small size of the hole.
fig,ax = subplots(2)
ax[0].plot(normalized[:,545])
ax[1].plot(normalized[536,:])
Out[14]:
[<matplotlib.lines.Line2D at 0x7f1439d384e0>]
In [15]:
#Definition of the convolution kernel
ksize = 5
y,x = numpy.ogrid[-(ksize-1)//2:ksize//2+1,-(ksize-1)//2:ksize//2+1]
d = numpy.sqrt(y*y+x*x)

#Fade out curve definition
fadeout = lambda x: 1/(1+numpy.exp(5*(x-2.5)))

kernel = fadeout(d)
mini=kernel.sum()
print(mini)

fig,ax = subplots(1,3)
ax[0].imshow(d)
ax[0].set_title("Distance array")

ax[1].plot(numpy.linspace(0,5,100),fadeout(numpy.linspace(0,5,100)))
ax[1].set_title("fade-out curve")

ax[2].imshow(kernel)
ax[2].set_title("Convolution kernel")
19.63857792789662
Out[15]:
Text(0.5, 1.0, 'Convolution kernel')
In [16]:
my_smooth = convolve(normalized, kernel, mode="constant", cval=0)/mini
print(my_smooth.shape)
fig,ax = subplots(1,2)
ax[0].imshow(normalized.clip(0,1))
ax[0].set_ylim(1050,1100)
ax[0].set_xlim(300,350)
ax[1].imshow(my_smooth.clip(0,1))
ax[1].set_ylim(1050,1100)
ax[1].set_xlim(300,350)
numpy.where(my_smooth == my_smooth.max())
(1679, 1475)
Out[16]:
(array([1065]), array([338]))
In [17]:
#mask out all pixels too close to any masked position

all_masks = numpy.logical_or(numpy.logical_or(mask0,mask1),mask2)
print(all_masks.sum())
big_mask = binary_dilation(all_masks, iterations=ksize//2+1+1)
print(big_mask.sum())
smooth2 = my_smooth.copy()
smooth2[big_mask] = 0
fig,ax = subplots()
ax.imshow(smooth2)
335009
782371
Out[17]:
<matplotlib.image.AxesImage at 0x7f1439ba22e8>
In [18]:
#Display the profile of the smoothed image: the center is easy to measure thanks to the smoothness of the signal
fig,ax = subplots(2)
ax[0].plot(my_smooth[:,545])
ax[1].plot(my_smooth[536,:])
Out[18]:
[<matplotlib.lines.Line2D at 0x7f1439b27860>]

Peak picking

We use the watershed module from pyFAI to retrieve all peak positions. Those regions are sieved out respectively for:

  • their size, it should be larger than the kernel itself
  • the peaks too close to masked regions are removed
  • the intensity of the peak
In [19]:
iw = InverseWatershed(my_smooth)
iw.init()
iw.merge_singleton()
all_regions = set(iw.regions.values())

regions = [i for i in all_regions if i.size>mini]

print("Number of region segmented: %s"%len(all_regions))
print("Number of large enough regions : %s"%len(regions))
Number of region segmented: 82126
Number of large enough regions : 41333
In [20]:
#Remove peaks on masked region
sieved_region = [i for i in regions if not big_mask[(i.index//nimg.shape[-1], i.index%nimg.shape[-1])]]
print("Number of peaks not on masked areea : %s"%len(sieved_region))
Number of peaks not on masked areea : 30001
In [21]:
# Histogram of peak height:
s = numpy.array([i.maxi for i in sieved_region])

fig, ax = subplots()
ax.hist(s, 100)
Out[21]:
(array([1.0000e+00, 0.0000e+00, 1.0000e+01, 3.9940e+03, 2.2274e+04,
        1.4800e+03, 1.1700e+02, 3.4000e+01, 8.0000e+00, 2.0000e+00,
        2.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 1.0000e+00,
        1.0000e+00, 0.0000e+00, 0.0000e+00, 1.0000e+00, 0.0000e+00,
        1.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 1.0000e+00,
        0.0000e+00, 1.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,
        1.0000e+00, 2.0000e+00, 2.0000e+00, 1.0000e+00, 0.0000e+00,
        1.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,
        0.0000e+00, 0.0000e+00, 0.0000e+00, 1.0000e+00, 0.0000e+00,
        1.0000e+00, 3.0000e+00, 1.0000e+00, 2.0000e+00, 0.0000e+00,
        0.0000e+00, 2.0000e+00, 1.0000e+00, 0.0000e+00, 0.0000e+00,
        0.0000e+00, 1.0000e+00, 0.0000e+00, 0.0000e+00, 2.0000e+00,
        0.0000e+00, 0.0000e+00, 2.0000e+00, 0.0000e+00, 0.0000e+00,
        2.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00,
        0.0000e+00, 1.0000e+00, 1.0000e+00, 3.0000e+00, 3.0000e+00,
        7.0000e+00, 1.0000e+01, 1.0000e+01, 2.1000e+01, 1.9000e+01,
        2.9000e+01, 3.5000e+01, 5.7000e+01, 7.3000e+01, 9.9000e+01,
        1.1800e+02, 1.5900e+02, 1.9400e+02, 1.9500e+02, 1.9700e+02,
        1.9500e+02, 1.6300e+02, 1.6200e+02, 1.0200e+02, 8.1000e+01,
        6.6000e+01, 3.9000e+01, 7.0000e+00, 1.0000e+00, 1.0000e+00]),
 array([0.04771975, 0.05009548, 0.05247121, 0.05484694, 0.05722266,
        0.05959839, 0.06197412, 0.06434985, 0.06672558, 0.06910131,
        0.07147704, 0.07385276, 0.07622849, 0.07860422, 0.08097995,
        0.08335568, 0.08573141, 0.08810713, 0.09048286, 0.09285859,
        0.09523432, 0.09761005, 0.09998578, 0.1023615 , 0.10473723,
        0.10711296, 0.10948869, 0.11186442, 0.11424015, 0.11661588,
        0.1189916 , 0.12136733, 0.12374306, 0.12611879, 0.12849452,
        0.13087025, 0.13324597, 0.1356217 , 0.13799743, 0.14037316,
        0.14274889, 0.14512462, 0.14750035, 0.14987607, 0.1522518 ,
        0.15462753, 0.15700326, 0.15937899, 0.16175472, 0.16413044,
        0.16650617, 0.1688819 , 0.17125763, 0.17363336, 0.17600909,
        0.17838482, 0.18076054, 0.18313627, 0.185512  , 0.18788773,
        0.19026346, 0.19263919, 0.19501491, 0.19739064, 0.19976637,
        0.2021421 , 0.20451783, 0.20689356, 0.20926929, 0.21164501,
        0.21402074, 0.21639647, 0.2187722 , 0.22114793, 0.22352366,
        0.22589938, 0.22827511, 0.23065084, 0.23302657, 0.2354023 ,
        0.23777803, 0.24015376, 0.24252948, 0.24490521, 0.24728094,
        0.24965667, 0.2520324 , 0.25440813, 0.25678385, 0.25915958,
        0.26153531, 0.26391104, 0.26628677, 0.2686625 , 0.27103822,
        0.27341395, 0.27578968, 0.27816541, 0.28054114, 0.28291687,
        0.2852926 ]),
 <a list of 100 Patch objects>)
In [22]:
#sieve-out for peak intensity
int_mini = 0.1
peaks = [(i.index//nimg.shape[-1], i.index%nimg.shape[-1]) for i in sieved_region if (i.maxi)>int_mini]
print("Number of remaining peaks with I>%s: %s"%(int_mini, len(peaks)))

peaks_raw = numpy.array(peaks)
Number of remaining peaks with I>0.1: 2075
In [23]:
# Finally the peak positions are interpolated using a second order taylor expansion 
# in thevinicy of the maximum value of the signal:

#Create bilinear interpolator
bl = Bilinear(my_smooth)

#Overlay raw peak coordinate and refined peak positions

ref_peaks = [bl.local_maxi(p) for p in peaks]
fig, ax = subplots()
ax.imshow(img.clip(0,1000), interpolation="nearest")
peaks_ref = numpy.array(ref_peaks)
ax.plot(peaks_raw[:,1], peaks_raw[:, 0], ".r")
ax.plot(peaks_ref[:,1],peaks_ref[:, 0], ".b")
ax.set_title("Extracted peak position (red: raw, blue: refined)")
print("Refined peak coordinate:")
print(ref_peaks[:10])
Refined peak coordinate:
[(1066.1397777199745, 514.6457688212395), (1066.2501765489578, 544.1776854991913), (1066.1244528591633, 573.5188286304474), (1066.2436281442642, 602.8817960470915), (1066.2303752303123, 632.3413894176483), (1066.2659375667572, 661.732112288475), (1066.3028543293476, 691.1117671355605), (1066.2113457471132, 720.5017040967941), (1066.3163948357105, 750.0201806090772), (1066.2818910479546, 779.437959253788)]

At this stage we have about 2000 peaks (with sub-pixel precision) which are visually distributed on all modules. Some modules have their peaks located along sub-module boundaries which are masked out, hence they have fewer ontrol point for the calculation. Let's assign each peak to a module identifier. This allows to print out the number of peaks per module:

In [24]:
dt = numpy.dtype([('y', numpy.float64),
                  ('x', numpy.float64),
                  ('i', numpy.int64),
                 ])
yxi = numpy.array([i+(mid[round(i[0]),round(i[1])],) 
                   for i in ref_peaks], dtype=dt)
print("Number of keypoint per module:")
for i in range(1,mid.max()+1):
    print("Module id:",i, "cp:", (yxi[:]["i"] == i).sum())
Number of keypoint per module:
Module id: 1 cp: 48
Module id: 2 cp: 30
Module id: 3 cp: 48
Module id: 4 cp: 46
Module id: 5 cp: 42
Module id: 6 cp: 47
Module id: 7 cp: 47
Module id: 8 cp: 30
Module id: 9 cp: 48
Module id: 10 cp: 48
Module id: 11 cp: 41
Module id: 12 cp: 39
Module id: 13 cp: 48
Module id: 14 cp: 30
Module id: 15 cp: 47
Module id: 16 cp: 48
Module id: 17 cp: 42
Module id: 18 cp: 48
Module id: 19 cp: 47
Module id: 20 cp: 30
Module id: 21 cp: 48
Module id: 22 cp: 47
Module id: 23 cp: 42
Module id: 24 cp: 47
Module id: 25 cp: 48
Module id: 26 cp: 30
Module id: 27 cp: 48
Module id: 28 cp: 47
Module id: 29 cp: 41
Module id: 30 cp: 50
Module id: 31 cp: 46
Module id: 32 cp: 30
Module id: 33 cp: 48
Module id: 34 cp: 42
Module id: 35 cp: 47
Module id: 36 cp: 48
Module id: 37 cp: 48
Module id: 38 cp: 28
Module id: 39 cp: 48
Module id: 40 cp: 42
Module id: 41 cp: 44
Module id: 42 cp: 48
Module id: 43 cp: 47
Module id: 44 cp: 26
Module id: 45 cp: 46
Module id: 46 cp: 42
Module id: 47 cp: 45
Module id: 48 cp: 48

Grid assignment

The calibration is performed using a regular grid, the idea is to assign to each peak of coordinates (x,y) the integer value (X, Y) which correspond to the grid corrdinate system.

The first step is to measure the grid pitch which correspond to the distance (in pixels) from one peak to the next. This is easily obtained from a pair-wise distribution function.

In [25]:
# pairwise distance calculation using scipy.spatial.distance_matrix

dist = distance_matrix(peaks_ref, peaks_ref)

fig, ax = subplots()
ax.hist(dist.ravel(), 100, range=(0,100))
ax.set_title("Pair-wise distribution function")
Out[25]:
Text(0.5, 1.0, 'Pair-wise distribution function')

The histogram of the pair-distribution function has a first peak at 0 and the second peak between 29 and 30. Let's start the fit with this value

Two other parameters correspond to the offset, in pixel for the grid index (X,Y) = (0,0). The easiest is to measure the smallest x and y for the first module.

In [26]:
#from pair-wise distribution histogram
step = 29 
#work with the first module and fit the peak positions
first = yxi[yxi[:]["i"] == 1]
y_min = first[:]["y"].min()
x_min = first[:]["x"].min()
print("offset for the first peak: ", x_min, y_min)
offset for the first peak:  16.269793540239334 7.186804354190826

The grid looks very well aligned with the axes which makes this step easier but nothing garanties it is perfect, so the rotation of the grid has to be measured as well.

The default rotation will be zero and will be fitted later on.

Once the indexes X,Y determined for eack peak, one can fit the parameter to properly align the grid with the first module. Those 4 parameters are step-size, x_min, y_min and angle

In [27]:
#Assign each peak to an index
dl = numpy.dtype([('y', numpy.float64),
                  ('x', numpy.float64),
                  ('i', numpy.int64),
                  ('Y', numpy.int64),
                  ('X', numpy.int64),
                 ])
indexed1 = numpy.zeros(len(first), dtype=dl)

for i,v in enumerate(first):
    Y = int(round((v["y"]-y_min)/step))
    X = int(round((v["x"]-x_min)/step))
    indexed1[i]["y"] = v["y"]
    indexed1[i]["x"] = v["x"]
    indexed1[i]["i"] = v["i"]
    indexed1[i]["Y"] = Y
    indexed1[i]["X"] = X
    print("peak id: %s %20s Y:%d (Δ=%.3f) X:%s (Δ=%.3f)"%
          (i,v, Y, (v["y"]-Y*step-y_min)/step, X, (v["x"]-X*step-x_min)/step))
peak id: 0 (124.69245848, 16.47663069, 1) Y:4 (Δ=0.052) X:0 (Δ=0.007)
peak id: 1 (124.74490786, 45.56257215, 1) Y:4 (Δ=0.054) X:1 (Δ=0.010)
peak id: 2 (124.81674041, 75.13362026, 1) Y:4 (Δ=0.056) X:2 (Δ=0.030)
peak id: 3 (124.75223885, 104.64793801, 1) Y:4 (Δ=0.054) X:3 (Δ=0.048)
peak id: 4 (124.84565204, 133.79748473, 1) Y:4 (Δ=0.057) X:4 (Δ=0.053)
peak id: 5 (124.85729225, 163.3060154, 1) Y:4 (Δ=0.058) X:5 (Δ=0.070)
peak id: 6 (124.89991033, 192.57832247, 1) Y:4 (Δ=0.059) X:6 (Δ=0.080)
peak id: 7 (124.96431521, 222.11254385, 1) Y:4 (Δ=0.061) X:7 (Δ=0.098)
peak id: 8 (65.94893547, 16.45467511, 1) Y:2 (Δ=0.026) X:0 (Δ=0.006)
peak id: 9 (65.94483702, 45.61336857, 1) Y:2 (Δ=0.026) X:1 (Δ=0.012)
peak id: 10 (66.03506869, 75.15414216, 1) Y:2 (Δ=0.029) X:2 (Δ=0.030)
peak id: 11 (65.89957739, 104.38019937, 1) Y:2 (Δ=0.025) X:3 (Δ=0.038)
peak id: 12 (66.12287057, 133.89461011, 1) Y:2 (Δ=0.032) X:4 (Δ=0.056)
peak id: 13 (66.1012452, 163.39174998, 1) Y:2 (Δ=0.032) X:5 (Δ=0.073)
peak id: 14 (66.10113759, 192.66102123, 1) Y:2 (Δ=0.032) X:6 (Δ=0.082)
peak id: 15 (66.11095764, 222.20659751, 1) Y:2 (Δ=0.032) X:7 (Δ=0.101)
peak id: 16 (36.49696946, 16.50531429, 1) Y:1 (Δ=0.011) X:0 (Δ=0.008)
peak id: 17 (36.54493174, 45.66864574, 1) Y:1 (Δ=0.012) X:1 (Δ=0.014)
peak id: 18 (36.57917181, 75.22501931, 1) Y:1 (Δ=0.014) X:2 (Δ=0.033)
peak id: 19 (36.48521471, 104.38776881, 1) Y:1 (Δ=0.010) X:3 (Δ=0.039)
peak id: 20 (36.5958997, 133.89308376, 1) Y:1 (Δ=0.014) X:4 (Δ=0.056)
peak id: 21 (36.56109262, 163.54238695, 1) Y:1 (Δ=0.013) X:5 (Δ=0.078)
peak id: 22 (36.62988734, 192.63348889, 1) Y:1 (Δ=0.015) X:6 (Δ=0.082)
peak id: 23 (36.66130301, 222.22433692, 1) Y:1 (Δ=0.016) X:7 (Δ=0.102)
peak id: 24 (7.18680435, 16.41273034, 1) Y:0 (Δ=0.000) X:0 (Δ=0.005)
peak id: 25 (7.23123039, 45.74501115, 1) Y:0 (Δ=0.002) X:1 (Δ=0.016)
peak id: 26 (7.29059145, 75.19019249, 1) Y:0 (Δ=0.004) X:2 (Δ=0.032)
peak id: 27 (7.19876912, 104.56766725, 1) Y:0 (Δ=0.000) X:3 (Δ=0.045)
peak id: 28 (7.28363913, 134.01557775, 1) Y:0 (Δ=0.003) X:4 (Δ=0.060)
peak id: 29 (7.25490889, 163.38282073, 1) Y:0 (Δ=0.002) X:5 (Δ=0.073)
peak id: 30 (7.36717248, 192.76562056, 1) Y:0 (Δ=0.006) X:6 (Δ=0.086)
peak id: 31 (7.37469381, 222.16684446, 1) Y:0 (Δ=0.006) X:7 (Δ=0.100)
peak id: 32 (183.49989897, 16.39247844, 1) Y:6 (Δ=0.080) X:0 (Δ=0.004)
peak id: 33 (183.50560439, 45.46551961, 1) Y:6 (Δ=0.080) X:1 (Δ=0.007)
peak id: 34 (183.60742423, 75.08415242, 1) Y:6 (Δ=0.083) X:2 (Δ=0.028)
peak id: 35 (183.51583004, 104.62201929, 1) Y:6 (Δ=0.080) X:3 (Δ=0.047)
peak id: 36 (183.66220078, 133.73585817, 1) Y:6 (Δ=0.085) X:4 (Δ=0.051)
peak id: 37 (183.65553248, 163.30831155, 1) Y:6 (Δ=0.085) X:5 (Δ=0.070)
peak id: 38 (183.60811904, 192.46339762, 1) Y:6 (Δ=0.083) X:6 (Δ=0.076)
peak id: 39 (183.75545964, 222.06266521, 1) Y:6 (Δ=0.089) X:7 (Δ=0.096)
peak id: 40 (154.25518942, 16.26979354, 1) Y:5 (Δ=0.071) X:0 (Δ=0.000)
peak id: 41 (154.21079713, 45.57556251, 1) Y:5 (Δ=0.070) X:1 (Δ=0.011)
peak id: 42 (154.28871083, 75.04928852, 1) Y:5 (Δ=0.072) X:2 (Δ=0.027)
peak id: 43 (154.21604921, 104.4366582, 1) Y:5 (Δ=0.070) X:3 (Δ=0.040)
peak id: 44 (154.37423909, 133.8660319, 1) Y:5 (Δ=0.075) X:4 (Δ=0.055)
peak id: 45 (154.35255641, 163.24378996, 1) Y:5 (Δ=0.075) X:5 (Δ=0.068)
peak id: 46 (154.32582822, 192.64246738, 1) Y:5 (Δ=0.074) X:6 (Δ=0.082)
peak id: 47 (154.46003968, 222.04596291, 1) Y:5 (Δ=0.078) X:7 (Δ=0.096)

The error in positionning each of the pixel is less than 0.1 pixel which is already excellent and will allow a straight forward fit.

The cost function for the first module is calculated as the sum of distances squared in pixel space. It uses 4 parameters which are step-size, x_min, y_min and angle

In [28]:
guess1 = [step, y_min, x_min, 0]

def cost1(param):
    """contains: step, y_min, x_min, angle for the first module
    returns the sum of distance squared in pixel space
    """
    step = param[0]
    y_min = param[1]
    x_min = param[2]
    angle = param[3]
    XY = numpy.vstack((indexed1["X"], indexed1["Y"]))
    rot = [[cos(angle),-sin(angle)],
           [sin(angle), cos(angle)]]
    xy_min = [[x_min], [y_min]]
    xy_guess = numpy.dot(rot, step*XY+xy_min)
    delta = xy_guess - numpy.vstack((indexed1["x"], indexed1["y"]))
    return (delta*delta).sum()
In [29]:
print("Before optimization", guess1, "cost=", cost1(guess1))
res1 = minimize(cost1, guess1, method = "slsqp")
print(res1)
print("After optimization", res1.x, "cost=", cost1(res1.x))
print("Average displacement (pixels): ",sqrt(cost1(res1.x)/len(indexed1)))
Before optimization [29, 7.186804354190826, 16.269793540239334, 0] cost= 250.36710826038237
     fun: 0.633754214619366
     jac: array([ 2.10084021e-04, -1.17766112e-03,  1.04576349e-03, -3.24301951e-01])
 message: 'Optimization terminated successfully.'
    nfev: 51
     nit: 6
    njev: 6
  status: 0
 success: True
       x: array([2.93980817e+01, 7.13605735e+00, 1.63978976e+01, 8.76803818e-04])
After optimization [2.93980817e+01 7.13605735e+00 1.63978976e+01 8.76803818e-04] cost= 0.633754214619366
Average displacement (pixels):  0.11490523401729846

At this step, the grid is perfectly aligned with the first module. This module is used as the reference one and all other are aligned along it, using this first fit:

In [30]:
#retrieve the result of the first module fit:
step, y_min, x_min, angle = res1.x
indexed = numpy.zeros(yxi.shape, dtype=dl)

rot =  [[cos(angle),-sin(angle)],
        [sin(angle), cos(angle)]]
irot =  [[cos(angle), sin(angle)],
         [-sin(angle), cos(angle)]]

print("cost1: ",cost1([step, y_min, x_min, angle]), "for:", step, y_min, x_min, angle)

xy_min = numpy.array([[x_min], [y_min]])
xy = numpy.vstack((yxi["x"], yxi["y"]))
indexed["y"] = yxi["y"]
indexed["x"] = yxi["x"]
indexed["i"] = yxi["i"]
XY_app = (numpy.dot(irot, xy)-xy_min) / step
XY_int = numpy.round((XY_app)).astype("int")
indexed["X"] = XY_int[0]
indexed["Y"] = XY_int[1]
xy_guess = numpy.dot(rot, step * XY_int + xy_min)

thres = 1.2
delta = abs(xy_guess - xy)
print((delta>thres).sum())
suspicious = indexed[numpy.where(abs(delta>thres))[1]]
print(suspicious)
cost1:  0.633754214619366 for: 29.39808169745086 7.136057345210408 16.397897589727894 0.0008768038177288225
6
[(1654.4202804 , 954.44135761, 46, 56, 32)
 (1624.68617365, 807.47743243, 46, 55, 27)
 (1595.5792897 , 954.49445057, 46, 54, 32)
 (1448.30209509, 748.8784325 , 40, 49, 25)
 (1448.66381988, 895.86545633, 40, 49, 30)
 (1448.84506877, 954.56920397, 40, 49, 32)]
In [31]:
fig,ax = subplots()
ax.imshow(img.clip(0,1000))
ax.plot(indexed["x"], indexed["y"],".g")
ax.plot(suspicious["x"], suspicious["y"],".r")
Out[31]:
[<matplotlib.lines.Line2D at 0x7f143835e470>]

Only 6 peaks have an initial displacement of more than 1.2 pixel, all located in modules 40 and 46. The visual inspection confirms their localization is valid.

There are 48 (half-)modules which have each of them 2 translations and one rotation. In addition to the step size, this represents 145 degrees of freedom for the fit. The first module is used to align the grid, all other modules are then aligned along this grid.

In [32]:
def submodule_cost(param, module=1):
    """contains: step, y_min_1, x_min_1, angle_1, y_min_2, x_min_2, angle_2, ...
    returns the sum of distance squared in pixel space
    """
    
    step = param[0]
    y_min1 = param[1]
    x_min1 = param[2]
    angle1 = param[3]
    
    mask = indexed["i"] == module
    substack = indexed[mask]
    
    XY = numpy.vstack((substack["X"], substack["Y"]))
    rot1 = [[cos(angle1), -sin(angle1)],
            [sin(angle1), cos(angle1)]]
    xy_min1 = numpy.array([[x_min1], [y_min1]])
    xy_guess1 = numpy.dot(rot1, step * XY + xy_min1)
    #This is guessed spot position for module #1
    if module == 1:
        "Not much to do for module 1"
        delta = xy_guess1 - numpy.vstack((substack["x"], substack["y"]))
    else:
        "perform the correction for given module"
        y_min = param[(module-1)*3+1]
        x_min = param[(module-1)*3+2]
        angle = param[(module-1)*3+3]     

        rot = numpy.array([[cos(angle),-sin(angle)],
                           [sin(angle), cos(angle)]])
        xy_min = numpy.array([[x_min], [y_min]])
        xy_guess = numpy.dot(rot, xy_guess1+xy_min)
        delta = xy_guess - numpy.vstack((substack["x"], substack["y"]))

    return (delta*delta).sum()

guess145 = numpy.zeros(48*3+1)
guess145[:4] = res1.x
for i in range(1, 49):
    print("Cost for module #",i, submodule_cost(guess145, i))
Cost for module # 1 0.633754214619366
Cost for module # 2 1.8137345287978905
Cost for module # 3 1.8517786230140598
Cost for module # 4 4.912779678723252
Cost for module # 5 1.773716286969788
Cost for module # 6 7.486929157622746
Cost for module # 7 14.439350005921112
Cost for module # 8 5.219243256400658
Cost for module # 9 23.403698041674616
Cost for module # 10 5.689205807511105
Cost for module # 11 4.128371757842894
Cost for module # 12 8.41467387105107
Cost for module # 13 29.862711769405962
Cost for module # 14 12.62947128481549
Cost for module # 15 3.137577847401579
Cost for module # 16 5.280064180026195
Cost for module # 17 8.497453518207243
Cost for module # 18 4.6558015981617
Cost for module # 19 6.00851405310983
Cost for module # 20 4.9823079299639605
Cost for module # 21 5.137223797698782
Cost for module # 22 5.050520776157658
Cost for module # 23 1.4370003085447867
Cost for module # 24 6.050442140272764
Cost for module # 25 38.26261029663973
Cost for module # 26 19.51495609602089
Cost for module # 27 2.431126083353555
Cost for module # 28 5.491289468477838
Cost for module # 29 10.770584216483565
Cost for module # 30 14.051533587238408
Cost for module # 31 22.948566069502792
Cost for module # 32 18.49886234695026
Cost for module # 33 12.949367384216359
Cost for module # 34 15.380286444980001
Cost for module # 35 39.681025037194516
Cost for module # 36 37.29859111666596
Cost for module # 37 63.71623178050871
Cost for module # 38 23.55575004101026
Cost for module # 39 34.01666918845312
Cost for module # 40 43.819042055018926
Cost for module # 41 15.516115274724731
Cost for module # 42 8.057402166015052
Cost for module # 43 22.746163517473015
Cost for module # 44 10.802018609756525
Cost for module # 45 41.036167448779246
Cost for module # 46 46.276810920543255
Cost for module # 47 24.562082002847546
Cost for module # 48 34.788763306988784

On retrieves that the modules 40 and 46 have large errors. Module 37 as well.

The total cost funtion is hence the sum of all cost function for all modules:

In [33]:
def total_cost(param):
    """contains: step, y_min_1, x_min_1, angle_1, ...
    returns the sum of distance squared in pixel space
    """
    return sum(submodule_cost(param, module=i) for i in range(1,49))
total_cost(guess145)
Out[33]:
778.6683388937577
In [34]:
%%time
print("Before optimization", guess145[:10], "cost=", total_cost(guess145))
res_all = minimize(total_cost, guess145, method = "slsqp")
print(res_all)
print("After optimization", res_all.x[:10], "cost=", total_cost(res_all.x))
Before optimization [2.93980817e+01 7.13605735e+00 1.63978976e+01 8.76803818e-04
 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
 0.00000000e+00 0.00000000e+00] cost= 778.6683388937577
     fun: 27.183652120008176
     jac: array([-2.22609043e-01, -1.41382217e-04, -7.23886490e-03,  6.30186272e+00,
        1.49965286e-04,  2.99215317e-04,  1.03988647e-02,  3.04222107e-04,
        2.16960907e-04,  1.62777901e-02, -1.30891800e-04, -9.97781754e-04,
        3.07233334e-02,  1.20878220e-04,  1.26552582e-03,  7.25142956e-02,
       -1.57356262e-04, -6.82115555e-04,  1.34363413e-01, -6.98328018e-04,
       -3.70502472e-04,  2.73823738e-02, -4.72307205e-04, -9.15765762e-04,
        1.92334652e-02,  2.06947327e-04, -6.36577606e-05,  8.33108425e-02,
       -5.96046448e-06, -1.35898590e-05,  6.88681602e-02,  1.33991241e-04,
       -4.91142273e-05,  3.70295048e-02,  1.63316727e-04, -6.67572021e-05,
        4.39488888e-02, -1.25408173e-04, -2.62260437e-04,  7.39893913e-02,
        9.84668732e-05,  3.44038010e-04,  2.05819607e-02,  1.44958496e-04,
        1.50442123e-04,  5.73618412e-02, -1.08480453e-04, -2.77996063e-04,
        3.36527824e-02, -6.96182251e-04, -1.89757347e-03,  8.91270638e-02,
        3.74078751e-04,  1.55448914e-04,  1.42634630e-01,  1.77383423e-04,
       -1.16109848e-04,  6.13884926e-02,  3.43322754e-04,  1.29222870e-04,
        3.99765968e-02,  2.74181366e-05, -7.96318054e-05,  9.58600044e-02,
        1.70230865e-04, -2.49862671e-04,  1.01646185e-01, -3.33786011e-04,
       -9.95397568e-04,  7.76944160e-02, -6.38961792e-05, -8.52346420e-04,
        1.02715254e-01, -9.53674316e-06, -1.15633011e-04,  1.40109539e-01,
       -2.94923782e-04, -3.42607498e-04,  7.00299740e-02,  2.28881836e-05,
        8.67843628e-05,  8.27276707e-02, -2.39849091e-04, -2.20060349e-04,
        3.17461491e-02,  3.54766846e-04,  1.12771988e-04,  1.80860043e-01,
        3.85284424e-04, -3.44038010e-04,  1.38649225e-01, -1.11293793e-03,
        4.95910645e-05,  1.51008368e-01, -2.75135040e-04, -9.96589661e-05,
        9.11080837e-02,  1.29699707e-04,  6.41345978e-05,  1.03739262e-01,
        6.22272491e-05, -1.38282776e-04,  8.80627632e-02, -2.52723694e-05,
       -3.29494476e-04,  2.83084631e-01,  4.94003296e-04,  5.52892685e-04,
        2.91340590e-01,  3.33786011e-05, -3.22818756e-04,  2.63633490e-01,
       -1.94311142e-04, -4.76837158e-05,  1.69058084e-01,  4.02927399e-04,
        1.05142593e-04,  2.43228436e-01,  2.63834000e-03,  2.07734108e-03,
        2.55576134e-01, -8.01324844e-04, -6.29425049e-04,  1.96581125e-01,
        2.82526016e-04,  4.60624695e-04,  2.39082575e-01, -3.11136246e-04,
       -1.70230865e-04,  2.45981932e-01, -3.71932983e-05, -5.53607941e-04,
        7.57119656e-02, -1.53088570e-03, -5.53131104e-04,  2.73381710e-01,
       -2.98523903e-03, -1.34921074e-03,  2.76092052e-01, -5.57422638e-04,
       -2.66551971e-04,  2.06231594e-01,  7.22646713e-04, -1.66893005e-06,
        1.97270870e-01])
 message: 'Optimization terminated successfully.'
    nfev: 15031
     nit: 99
    njev: 99
  status: 0
 success: True
       x: array([ 2.94082871e+01,  7.10545045e+00,  1.63621941e+01,  8.77069273e-04,
       -2.61078393e-01, -1.72466913e-01,  2.74598922e-04, -4.65209657e-01,
       -1.49519174e-01,  9.22459862e-04, -1.40923632e+00, -3.93463169e-02,
        1.93717313e-03, -4.71684303e-01, -4.34636643e-01,  3.15328305e-04,
       -2.55772812e+00, -4.42370917e-01,  2.08052092e-03, -7.62305573e-01,
        5.00458319e-01,  1.34287296e-03, -8.36113931e-01,  2.10440744e-01,
        1.08250550e-03, -1.27257665e+00, -1.27987216e-01,  8.93671013e-04,
       -2.05270815e+00,  2.04317076e-01,  2.10457489e-03, -3.29511619e-01,
       -1.11614083e-01,  4.68385298e-04, -5.87612275e-01,  3.68144844e-02,
        6.44980513e-04, -8.24186370e-01,  4.58249844e-02,  1.14477356e-03,
       -9.99964202e-01,  1.20518109e-01,  1.31112806e-03, -9.06392481e-01,
        1.88514372e-01,  9.65141038e-04, -1.14456758e+00,  1.26295952e-01,
        1.27059935e-03, -8.21307339e-01, -5.18676989e-01,  4.31430923e-04,
       -2.16164119e+00,  1.33792625e-01,  1.53336072e-03, -5.67581749e-01,
        3.19180359e-01,  7.06534099e-04, -4.90619992e-01,  6.00730062e-02,
        7.15055182e-04, -2.58762718e-01, -6.04035880e-01, -3.01713070e-04,
       -1.66714232e+00,  6.64107310e-01,  1.62285007e-03, -1.58622996e+00,
        6.43406741e-01,  1.23593211e-03, -2.16579067e+00,  8.25504451e-01,
        1.65149729e-03, -8.55913479e-01,  1.36421601e-01,  9.44230561e-04,
       -1.04919358e+00,  8.15282869e-01,  1.75084134e-03,  2.14256801e-01,
       -6.90352540e-01, -5.26059213e-04, -4.87443162e-01,  3.35467711e-01,
        5.93016506e-04, -7.51463029e-01, -2.88064653e-01,  5.54630341e-04,
       -3.03353890e+00,  1.59271721e+00,  2.32781114e-03, -4.67011021e-01,
       -1.52645625e+00, -7.47721942e-04, -1.21122713e+00,  8.98494049e-01,
        1.43544007e-03,  2.05403150e-01, -1.38106953e+00, -6.39075503e-04,
       -1.40235382e+00,  1.03365224e+00,  1.55524860e-03, -9.46923813e-01,
       -5.14003258e-01,  6.35545868e-04, -8.51540960e-01, -6.13815477e-01,
        5.56684334e-04, -1.02276557e+00, -1.03647395e+00, -3.62861813e-05,
       -9.28167391e-01, -8.96820978e-01, -4.11375915e-05, -6.65434980e-01,
       -8.47907836e-01,  1.06002756e-04, -2.37720467e+00,  2.09986078e+00,
        2.43053124e-03, -5.84679910e-01, -4.18668666e-01,  3.36615060e-04,
       -1.08726840e+00,  6.61343772e-02,  6.10144196e-04, -9.52937683e-01,
        9.58584610e-01,  9.84347242e-04, -8.38408429e-01,  1.64867751e-01,
        5.47194710e-04, -3.42996542e-01, -1.67353317e+00, -3.58513066e-04,
       -1.50797477e+00,  7.37773616e-01,  1.28049360e-03, -1.40269966e+00,
        8.68238942e-01,  1.15054461e-03, -2.29192883e+00,  2.04931069e+00,
        1.83716372e-03])
After optimization [ 2.94082871e+01  7.10545045e+00  1.63621941e+01  8.77069273e-04
 -2.61078393e-01 -1.72466913e-01  2.74598922e-04 -4.65209657e-01
 -1.49519174e-01  9.22459862e-04] cost= 27.183652120008176
CPU times: user 37.7 s, sys: 0 ns, total: 37.7 s
Wall time: 37.7 s
In [35]:
for i in range(1,49):
    print("Module id: %d cost: %.3f Δx: %.3f, Δy: %.3f rot: %.3f°"%
          (i, submodule_cost(res_all.x, i), res_all.x[-2+i*3], res_all.x[-1+i*3], numpy.rad2deg(res_all.x[i*3])))
Module id: 1 cost: 0.683 Δx: 7.105, Δy: 16.362 rot: 0.050°
Module id: 2 cost: 0.473 Δx: -0.261, Δy: -0.172 rot: 0.016°
Module id: 3 cost: 0.650 Δx: -0.465, Δy: -0.150 rot: 0.053°
Module id: 4 cost: 0.613 Δx: -1.409, Δy: -0.039 rot: 0.111°
Module id: 5 cost: 0.566 Δx: -0.472, Δy: -0.435 rot: 0.018°
Module id: 6 cost: 0.666 Δx: -2.558, Δy: -0.442 rot: 0.119°
Module id: 7 cost: 0.696 Δx: -0.762, Δy: 0.500 rot: 0.077°
Module id: 8 cost: 0.341 Δx: -0.836, Δy: 0.210 rot: 0.062°
Module id: 9 cost: 0.566 Δx: -1.273, Δy: -0.128 rot: 0.051°
Module id: 10 cost: 0.661 Δx: -2.053, Δy: 0.204 rot: 0.121°
Module id: 11 cost: 0.616 Δx: -0.330, Δy: -0.112 rot: 0.027°
Module id: 12 cost: 0.495 Δx: -0.588, Δy: 0.037 rot: 0.037°
Module id: 13 cost: 0.763 Δx: -0.824, Δy: 0.046 rot: 0.066°
Module id: 14 cost: 0.353 Δx: -1.000, Δy: 0.121 rot: 0.075°
Module id: 15 cost: 0.645 Δx: -0.906, Δy: 0.189 rot: 0.055°
Module id: 16 cost: 0.492 Δx: -1.145, Δy: 0.126 rot: 0.073°
Module id: 17 cost: 0.440 Δx: -0.821, Δy: -0.519 rot: 0.025°
Module id: 18 cost: 0.779 Δx: -2.162, Δy: 0.134 rot: 0.088°
Module id: 19 cost: 0.748 Δx: -0.568, Δy: 0.319 rot: 0.040°
Module id: 20 cost: 0.423 Δx: -0.491, Δy: 0.060 rot: 0.041°
Module id: 21 cost: 0.520 Δx: -0.259, Δy: -0.604 rot: -0.017°
Module id: 22 cost: 0.676 Δx: -1.667, Δy: 0.664 rot: 0.093°
Module id: 23 cost: 0.503 Δx: -1.586, Δy: 0.643 rot: 0.071°
Module id: 24 cost: 0.822 Δx: -2.166, Δy: 0.826 rot: 0.095°
Module id: 25 cost: 0.575 Δx: -0.856, Δy: 0.136 rot: 0.054°
Module id: 26 cost: 0.320 Δx: -1.049, Δy: 0.815 rot: 0.100°
Module id: 27 cost: 0.642 Δx: 0.214, Δy: -0.690 rot: -0.030°
Module id: 28 cost: 0.599 Δx: -0.487, Δy: 0.335 rot: 0.034°
Module id: 29 cost: 0.498 Δx: -0.751, Δy: -0.288 rot: 0.032°
Module id: 30 cost: 0.830 Δx: -3.034, Δy: 1.593 rot: 0.133°
Module id: 31 cost: 0.687 Δx: -0.467, Δy: -1.526 rot: -0.043°
Module id: 32 cost: 0.422 Δx: -1.211, Δy: 0.898 rot: 0.082°
Module id: 33 cost: 0.616 Δx: 0.205, Δy: -1.381 rot: -0.037°
Module id: 34 cost: 0.393 Δx: -1.402, Δy: 1.034 rot: 0.089°
Module id: 35 cost: 0.447 Δx: -0.947, Δy: -0.514 rot: 0.036°
Module id: 36 cost: 0.587 Δx: -0.852, Δy: -0.614 rot: 0.032°
Module id: 37 cost: 0.603 Δx: -1.023, Δy: -1.036 rot: -0.002°
Module id: 38 cost: 0.295 Δx: -0.928, Δy: -0.897 rot: -0.002°
Module id: 39 cost: 0.558 Δx: -0.665, Δy: -0.848 rot: 0.006°
Module id: 40 cost: 0.498 Δx: -2.377, Δy: 2.100 rot: 0.139°
Module id: 41 cost: 0.468 Δx: -0.585, Δy: -0.419 rot: 0.019°
Module id: 42 cost: 0.569 Δx: -1.087, Δy: 0.066 rot: 0.035°
Module id: 43 cost: 0.633 Δx: -0.953, Δy: 0.959 rot: 0.056°
Module id: 44 cost: 0.413 Δx: -0.838, Δy: 0.165 rot: 0.031°
Module id: 45 cost: 0.671 Δx: -0.343, Δy: -1.674 rot: -0.021°
Module id: 46 cost: 0.542 Δx: -1.508, Δy: 0.738 rot: 0.073°
Module id: 47 cost: 0.537 Δx: -1.403, Δy: 0.868 rot: 0.066°
Module id: 48 cost: 0.593 Δx: -2.292, Δy: 2.049 rot: 0.105°

Analysis: Modules 40, 46 and 48 show large displacement but the fitting precedure allowed to reduce the residual cost to the same value as other modules.

Reconstruction of the pixel position

The pixel position can be obtained from the standard Pilatus detector. Each module is then displaced according to the fitted values, except the first one which is left where it is.

In [36]:
pixel_coord = pyFAI.detector_factory("Pilatus2MCdTe").get_pixel_corners()
pixel_coord_raw = pixel_coord.copy()
for i in range(2, 49):
    # Extract the pixel corners for one module
    module_idx = numpy.where(mid == i)
    one_module = pixel_coord_raw[module_idx]
    #retrieve the fitted values
    dy = res_all.x[1+(i-1)*3]
    dx = res_all.x[2+(i-1)*3]
    angle = res_all.x[3+(i-1)*3]
    
    z = one_module[...,0]
    y = one_module[...,1]
    x = one_module[...,2]
    
    #apply the correction the other way around
    irot = [[cos(angle), sin(angle)],
           [-sin(angle), cos(angle)]]
    trans = [[dx * pilatus.pixel2],
              [dy * pilatus.pixel1]]
    xy_guess = numpy.vstack((x.ravel(), y.ravel()))
    xy_cor = numpy.dot(irot, xy_guess) - trans
    xy_cor.shape = ((2,)+x.shape)
    one_module[...,1] = xy_cor[1, ...] #y
    one_module[...,2] = xy_cor[0, ...] #x
    #Update the array
    pixel_coord_raw[module_idx] = one_module

Update the detector and save it in HDF5

In [37]:
pilatus.set_pixel_corners(pixel_coord_raw)
pilatus.mask = all_masks
pilatus.save("Pilatus_ID15_raw.h5")
In [38]:
displ = numpy.sqrt(((pixel_coord - pixel_coord_raw)**2).sum(axis=-1))
displ /= pilatus.pixel1 #convert in pixel units
fig, ax = subplots()
ax.hist(displ.ravel(), 100)
ax.set_title("Displacement of pixels versus the reference representation")
ax.set_xlabel("Error in pixel size (172µm)")
Out[38]:
Text(0.5, 0, 'Error in pixel size (172µm)')
In [39]:
unmasked = numpy.logical_not(all_masks)
misaligned = numpy.vstack((pixel_coord_raw[unmasked][..., 2].ravel(), #x
                           pixel_coord_raw[unmasked][..., 1].ravel())) #y

reference = numpy.vstack((pixel_coord[unmasked][..., 2].ravel(), #x
                          pixel_coord[unmasked][..., 1].ravel())) #y

def global_displacement(param):
    """correct all pixel position with dy. dx, angle"""
    dy, dx, angle = param
    rot = [[cos(angle), -sin(angle)],
           [sin(angle), cos(angle)]]
    trans = [[dx],
              [dy]]
    xy_corr = numpy.dot(rot, misaligned) + trans
    delta = xy_corr - reference
    return (delta*delta).sum()
global_displacement([0,0,0])
Out[39]:
0.16846723092395738
In [40]:
%%time
guess_global = [0,0,0]
print("Before optimization", guess_global, "cost=", global_displacement(guess_global))
res_global = minimize(global_displacement, guess_global, method = "slsqp")
print(res_global)
print("After optimization", res_global.x, "cost=", global_displacement(res_global.x))
Before optimization [0, 0, 0] cost= 0.16846723092395738
     fun: 0.027839035331955204
     jac: array([ 0.24042948, -0.02983177, -0.02220915])
 message: 'Optimization terminated successfully.'
    nfev: 45
     nit: 5
    njev: 5
  status: 0
 success: True
       x: array([-1.16046878e-04, -2.56570524e-05,  5.32495440e-04])
After optimization [-1.16046878e-04 -2.56570524e-05  5.32495440e-04] cost= 0.027839035331955204
CPU times: user 14.5 s, sys: 11.1 s, total: 25.5 s
Wall time: 8.07 s
In [41]:
dy, dx, angle  = res_global.x
rot = [[cos(angle), -sin(angle)],
       [sin(angle), cos(angle)]]
trans = [[dx],
          [dy]]

raw = numpy.vstack((pixel_coord_raw[..., 2].ravel(), #x
                    pixel_coord_raw[..., 1].ravel())) #y
aligned = numpy.dot(rot, raw) + trans

pixel_coord_ref = pixel_coord.copy()
pixel_coord_ref[...,1] = aligned[1,:].reshape(pixel_coord_ref.shape[:-1])
pixel_coord_ref[...,2] = aligned[0,:].reshape(pixel_coord_ref.shape[:-1])
In [42]:
displ = numpy.sqrt(((pixel_coord-pixel_coord_ref)**2).sum(axis=-1))
displ /= pilatus.pixel1 #convert in pixel units
fig, ax = subplots()
ax.hist(displ.ravel(), 100)
ax.set_title("Displacement of pixels versus the reference representation")
ax.set_xlabel("Pixel size (172µm)")
Out[42]:
Text(0.5, 0, 'Pixel size (172µm)')
In [43]:
pilatus.set_pixel_corners(pixel_coord_ref)
pilatus.mask = all_masks
pilatus.save("Pilatus_ID15_ref.h5")

Conclusion

This tutorial presents the way to calibrate a module based detector using the Pilatus2M CdTe from ESRF-ID15. The HDF5 file generated is directly useable by any parts of pyFAI, the reader is invited in calibrating the rings images with the default definition and with this optimized definition and check the residual error is almost divided by a factor two.

To come back on the precision of the localization of the pixel: not all the pixel are within the specifications provided by Dectris which claims the misaliment of the modules is within one pixel.

In [ ]:
print("Total execution time: ", time.time()-start_time)