This work emphasizes many aspects that have been neglected so far in BCDI, which need to be understood for a quantitative analysis of displacement and strain based on this technique. It concludes with the optimization of experimental parameters to improve throughput and to establish BCDI as a reliable 3D nano-imaging technique.Bragg coherent X-ray diffraction imaging (BCDI) has emerged as a powerful technique to image the local displacement field and strain in nanocrystals, in three dimensions with nanometric spatial resolution. However, BCDI relies on both dataset collection and phase retrieval algorithms that can induce artefacts in the reconstruction. Phase retrieval algorithms are based on the fast Fourier transform (FFT). We demonstrate how to calculate the displacement field inside a nanocrystal from its reconstructed phase depending on the mathematical convention used for the FFT. We use numerical simulations to quantify the influence of experimentally unavoidable detector deficiencies such as blind areas or limited dynamic range as well as post-processing filtering on the reconstruction. We also propose a criterion for the isosurface determination of the object, based on the histogram of the reconstructed modulus. Finally, we study the capability of the phasing algorithm to quantitatively retrieve the surface strain ( Bragg Coherent X-ray Diffraction Imaging (BCDI) fulfills these requirements. The technique appeared in the 2000s2 and is finding application in in situ and operando studies5. This lens-less technique is based on inverse microscopy and employs digital methods to replace X-ray imaging lenses. Illuminating an isolated object by an X-ray beam with coherence lengths larger than the object, and measuring its oversampled 3D diffraction pattern, it is possible to reconstruct a complex amplitude object using iterative algorithms6. In Bragg geometry, the reconstructed phase is related to the projection of the displacement field, 7. Taking the gradient of this displacement allows direct extraction of one of the corresponding component of the strain tensor with the high sensitivity common to X-ray diffraction measurements8.Understanding the role of atomic displacement in determining physical properties and chemical reactivity of nanocrystals requires quantitative characterisation of the strain field with a good spatial resolution in 3D (≤10 nm) and a high strain sensitivity  on the reconstructed strain is therefore an important milestone for the credibility of the technique when applied to more complex environments. Methodical investigation of the performance and capability of phase retrieval has been approached with numerical simulations19. For instance, it has been demonstrated11 that a minimum dynamic range of 106 is needed for the intensity measurement to reach ultimate reconstruction performance and that the best resolution can be achieved with an overall oversampling ratio beyond 30. However, there is no tool to understand and distinguish the phase components caused by the real lattice displacement from the phase fluctuations that are statistical uncertainties introduced by the phase retrieval process itself. These artefacts affect the accuracy and reliability of the recovered modulus and phase and cannot be associated to physical phenomena in the measured crystal.Often, a qualitative understanding of the strain and displacement components is possible, but since the technique is extending towards 20, and we use here a simple model to demonstrate it. Then, we use numerical simulations on a realistic system with full control of measurement conditions to quantify the phase fluctuations: detector size, detector gap width and position relative to the Bragg peak, dynamic range, presence or not of noise. We also probe the effect of applying a filter after phase retrieval on the reconstruction quality. Finally, we propose a criterion for isosurface determination, based on the histogram of the reconstructed modulus. We study the capability of the phasing algorithm to retrieve quantitatively the strain of the surface voxels, which is an important parameter in surface physics and chemistry of materials. The study proposes optimum experimental conditions and tools to achieve optimum BCDI.First, we discuss the formal mathematical relation between the retrieved phase and displacement field. The displacement and the retrieved phase may have an opposite sign depending on the mathematical convention of the fast Fourier transform (FFT) used in the phasing algorithm. The relation between the retrieved phase and displacement has been evoked in ref.et al. mentioned briefly that the retrieved phase and the retrieved displacement are of opposite sign as a result of the FFT convention used by their Python-based phasing algorithms20. This is a critical point if one wants to perform a quantitative analysis of the displacement field and strain to distinguish compressive or tensile regions in a nanocrystal. We believe that it deserves a more in-depth description. Starting with a 2D model with a known real-space orientation, electron density and displacement ux  of the complex-valued real-space object, whatever the (positive or negative) convention used for the phase term: ux), i.e. Figure ψ = −qx . ux. This is explained by the fact that Python uses a negative sign convention for the FFT. The reconstructions obtained by phase retrieval are shown in Fig. Labat see Fig. , which ie.g., when there is an epitaxial relationship between the crystal and the substrate used for crystal growth). In many cases where there is no systematic orientation and asymmetry in crystal shape , it is not possible to pick up the correct solution arising from the phasing algorithm. Without prior knowledge about the behavior of the displacement (or strain) in the nanocrystal, it is therefore not possible in these cases to discuss the sign of the result. Not knowing the real-space orientation of the crystal is less of a problem when studying defects, because their signature is a jump in the phase22, which appears whatever the convention of sign used.For the sake of completeness, we show in Fig. one inside the crystal and zero outside. This crystal is a Pt tetrahexahedral (THH) particle (width of ~400 nm), grown on a glassy carbon substrate by electrochemical deposition and faceting23. A scanning electron microscope (SEM) image of the THH particles is shown in Fig. i.e., Z downstream, Y vertical up and X outboard24 (right handed). The Y axis is along the [111] direction of the crystal. For simplicity, we decide to fix the real-space phase, 111 Pt Bragg peak at the ESRF – the European synchrotron, ID01 beamline25. The scattered X-rays were detected using a 2D Maxipix pixel detector (516 × 516 pixels of 55 µm × 55 µm)26. The detector is composed of four modules separated by gaps with a size of 220 µm. Pixels at the edge of the module have a size of 165 µm × 55 µm to cover this gap. Depending on the oversampling, the data can be binned and edge pixels used (for an oversampling larger than 3), or masked for smaller oversampling because then it is not possible to redistribute efficiently the measured intensity in virtual pixels of 55 µm × 55 µm. The detector gaps are visible in the contour plot of Fig. etc.) to mimic realistic experimental conditions. The generated data are then retrieved using a well-established phase retrieval process to get the real-space object, whose quality is quantitatively evaluated. This allows one to investigate the origin of artefacts in the reconstruction and characterize the performance of BCDI. We do not discuss specifically the phasing algorithm in this work; therefore, all trends presented here apply to the error metric used for phase retrieval (see Methods). The resolution of the reconstruction is estimated using the normalized Phase-Retrieval Transfer Function (PRTF)28 at a cutoff value of 1/e. The PRTF is a measure of how well the retrieved Fourier amplitudes match the square root of the measured diffraction intensity. Another common method to determine the resolution of experimental data is the Fourier shell correlation29.To investigate the origin and impact of artefacts on the reconstruction from BCDI measurements, we create a complex-valued real-space object of density see Fig. . The sup30): frequently parallel to facets, they are observed both in the modulus and the phase of the reconstruction and affect the reconstruction statistics. Since the phasing algorithm is based on FFT, it is legitimate to suspect the Gibbs phenomenon and the aliasing effect. The Gibbs phenomenon is known as a non-physical oscillation artefact near the sharp discontinuity31, which degrades the quality of the reconstruction. In particular, this phenomenon is noticeable by the cut-off of the waveform in a limited window during the discrete Fourier transform process32. Aliasing occurs when the diffraction intensity does not completely decay to zero within the detector area and violates the continuous boundary condition of the Fourier transform33. As the phase retrieval algorithms are based on the fast Fourier transform (FFT) algorithm, the reconstructed density and phase are affected by the two effects mentioned above. This is enhanced in the case of a faceted crystal : the reciprocal space pattern of a faceted crystal is characterized by streaks of intensity, i.e. crystal truncation rods which are normal to the each surface facet. These streaks whose intensity follows a q−2 law34 can elongate far in reciprocal space and are often truncated by the detector.As mentioned in the introduction, it is quite common to observe phase fluctuations in BCDI reconstructions. They often have the form of oscillation stripe artefacts . In order to assess the effect of the size of the FFT window, which corresponds to the size of the experimentally exploited detector area, we crop the array, and finally phase it using a Python-based phase-retrieval algorithm, the PyNX package36.Phasing of the experimental data used to generate our model is realized with a FFT window of 270 × 432 × 420 pixels. To avoid as much as possible introducing artefacts in our simulation, we zero-pad the support 3D array to a size of 10111 Bragg reflection encodes the displacement field along the [111] direction (u111) of the crystal, which is parallel to the Y axis of Fig. YY calculated after phasing. Diffraction patterns for each configuration and slices in XY plane are shown in Figs. i.e., different sizes of detector). For this particular example, we see that the presence of artefacts is independent of the cropped FFT window size, and that their amplitude increases slowly with the FFT window size. Since we are dealing with a 3D object, stripes interfere with each other, yielding a complex pattern . We use the root-mean-square error (RMSE) of the retrieved strain as an indicator for this trend. The line plot of Fig. i.e. the extent of diffracted photons in reciprocal space. If the resolution kept increasing with the window size, it would mean that we could generate reconstructions at any resolution with the same amount of information by just padding the data, which of course is not physical. When the data is not cut by the FFT window, the resolution is determined by the data and not by the detector size. The amplitude of artefacts generated by the FFT window size , however, does not seem able to explain by itself all artefacts in reconstructions, which can be experimentally larger by one order of magnitude.The phase obtained from the inversion of the simulated diffraction patterns at the i.e., when the intensity does not completely decay to zero within the detector area and violates the continuous boundary condition of the Fourier transform. Aliasing cannot be totally suppressed because of the use of the FFT during iterative phasing: unlike the measurement data, the retrieved complex amplitude array fills totally the FFT window, thus violating the periodic boundary condition.In the following, we decided to fix the detector window to 400 pixels to limit as much as possible aliasing effects, YY strain. Diffraction patterns for each configuration and slices in XY plane are presented in Figs. 26) is at least one order of magnitude larger than without gaps. The effect of detector gaps is often underestimated, but the above simulations show that this is actually the main contribution to stripe artefacts. Note that the resolution, however, starts to be slightly affected only with gaps larger than 12 pixels. This has a direct implication for experiments using detectors with large gaps : it is necessary to avoid positioning the diffracted/scattered signal in gaps, otherwise it could lead to misinterpretation of the reconstructed data.The next candidate for potential experimental artefacts is the presence of gaps in the detector (area where no intensity is measured), since they introduce non-random features in the data during the phasing, even if gap voxels are set free during the phase retrieval (the amplitude constraint is not applied on these voxels). Since the 3D diffraction pattern is sampled by successive 2D slices through it, detector gaps are also continuous in the dimension corresponding to the scan. Here, we fix the FFT window size to 400 pixels of 55 µm × 55 µm each and we introduce a gap in the detector. We show the influence of the gap width in Fig. 37, and missing high-intensity pixels can have a large impact on the algorithm’s ability to reconstruct the crystal. In the following simulation, we fix the FFT window size to 400 pixels of 55 µm × 55 µm, the gap width to 6 pixels, and we vary the distance of the gap to the central Bragg peak position. Figure YY strain calculated after phasing. Diffraction patterns for each configuration and slices in the XY plane are presented in Figs. The position of the gap relative to the Bragg peak is also important. When the crystal has few defects, streaks can go far from the central Bragg peak position in reciprocal space, and the experimentalist may be tempted to put the Bragg peak near the central gap of the detector, in order to collect more signal. In Bragg CDI, there is no redundancy in the data as for ptychographyi.e., a FFT window size of 400 pixels of 55 µm × 55 µm each, a gap width of 6 pixels located 50 pixels away from the Bragg peak, we vary the number of scattered/diffracted photons measured in the detector by renormalizing the diffraction pattern. This is equivalent to tuning the acquisition time of the detector for each 2D slice of the 3D diffraction pattern. Figure YY strain calculated after phasing. Diffraction patterns for each configuration and slices in the XY plane are shown respectively in Figs. via the formula: (wavelength × sample to detector distance)/(pixel number × pixel size), the resolution of the reconstruction is linked to the dynamic range of the data and the signal to noise ratio. From these noise-free simulations, we see that in otherwise similar conditions, a larger dynamic range slightly reduces the amplitude of artefacts. This effect is however rather limited considering the huge difference in diffracted intensity between various cases.Then, we study the effect of the dynamic range of the data. Keeping the same simulation conditions, In order to study the impact of various experimental parameters, all simulation results presented above considered a noise-free measurement which is not encountered in experiments. It is legitimate to question whether, in practice, artefacts are not instead dominated by the noise. We present in Fig. 12. Considering the simulated nanoparticle from this study (Pt nanoparticle of ~400 nm diameter), with an oversampling of ~3 and a resolution limited by the dynamic range, a reasonable gapless detector size would be 400 pixels × 400 pixels of 55 µm × 55 µm each. For 200 angular steps, it corresponds to ~800 Mb of required memory using PyNX for phase retrieval.From the previous simulations, we conclude that the major contribution to stripe artefacts in reconstructions is the presence of a detector gap, its width and distance to the central Bragg peak position. We showed that the size of the FFT window has a smaller impact than detector gaps on strain artefacts, but it can limit the resolution: one wants to enlarge the FFT window in order to avoid cutting intensity streaks. This is equivalent to ensuring a sufficient oversampling in real space: the object should be less than half of the FFT window size38. Improvements can be made either by (1) preprocessing the measured data in reciprocal space or (2) post-processing of the outcome of the phasing algorithm. From a conceptual point of view however, it seems more acceptable to work on the reconstructed object than to modify the raw experimental data. Approaches during phasing are not relevant here: genetic algorithm40 and partial coherence compensation of the incoming X-ray wavefront41 do not help to remove stripes since they are just geometric artefacts due to FFT aliasing and appear whatever phasing process is used.There are several perspectives for improving the quality of the reconstruction. We exclude from the present discussion refraction and absorption corrections applied to the output of a phasing algorithm, as they have already been described in the literature42. It is similar to the decomposition of modes when reconstructing the probe in ptychography43. However, since the most relevant artefacts (as mentioned in this article) are caused by the geometric configuration of the experiment  and thus part of the experimental data itself, this approach also fails to remove them from the reconstruction. We now approach methods to spatially filter the phase to remove high frequency noise. We introduce first the use of an apodization step after phase retrieval, and then discuss the effect of filtering the reconstructed phase.For post-processing, it is common to average independent reconstructions that have enough correlation between them (in the sense of modulus correlation). One drawback of this approach is that the resulting reconstruction may not be anymore the best fit to the diffraction data, in terms of the figure of merit chosen for convergence (the modulus of the average being different from the average of the moduli). Moreover, it results inevitably in a coarsening of the spatial resolution due to tiny differences and slight misalignment between reconstructions . Recently, the decomposition of the complex-valued real-space solutions into modes has been introduced44). Various filtering windows exist, but as an example, we choose here to compare two windows, a Blackman window and a Tukey Window of parameter α = 0.7. The shape of these functions is presented in Fig. Apodization is used commonly in signal processing to compensate discontinuities in the measured spectrum. It smoothly brings a sampled signal down to zero at the edges of the sampled region. For instance, in extended X-ray absorption fine structure (EXAFS), an apodization window is used to avoid spurious mathematical ripples , hence the average has to be done with less voxels, and is more prone to statistical noise. Averaging the phase can therefore be used only for a qualitative approach, when no feature with a spatial frequency larger than artefacts is expected in the phase. In order to be closer to experimental cases, we compare the effect of phase averaging and apodization on real experimental data in Fig. 28. To assess these questions, we use the same model as previously but with a known phase, which is shown in Fig. In a number of cases, the quantity of interest is the strain at the surface of the nanocrystal. The surface in BCDI corresponds to the surface voxel layer defined by a modulus isosurface. Indeed, the normalized modulus ranges from 0 to 1, but there is no criterion yet on how one should define the proper isosurface to extract the proper strain values. Moreover, for a sharp interface, the strain will be defined only from the top sublayer to the core, because the gradient of the phase at the surface is calculated between a point on the crystal and a point outside. BCDI reconstructions from real data show a smoother interface because the crystal shape is convoluted with the resolution function, which is also direction dependentYY of the order of 0.1% maximum as shown in Fig. YY for a too low isosurface level . The methodology used in the paper helps to isolate statistical errors from systematic errors that are inherent in experimental setups. The resolution in the retrieved strain is limited by artefacts which can reach 10−3 when large masked areas are present in the dataset. The best approach is, when possible, to avoid using a detector with gaps. Regarding the size of the FFT window used for phasing, the best practice is to select the smallest window containing signal, while not truncating the diffraction pattern in order to avoid any unnecessary loss in resolution. Finally, adding a post-processing filtering step can help further reducing high-frequency noise. BCDI is a technique applied increasingly to in situ or operando systems, in which the initial qualitative approach tends to be replaced by a quantitative analysis of displacement and strain for probing subtle changes under real application conditions. We provide two important milestones for this, namely the relation between the retrieved phase and displacement field depending on the FFT sign convention of the phasing algorithm, and an empirical criterion for isosurface determination. We believe that this study will contribute to improving the reliability of the BCDI technique towards a quantitative strain analysis method.We present in this article a detailed study of artefacts in BCDI reconstructions induced by both dataset collection and phase retrieval algorithms  electrodes by a square-wave-potential method with a saturated calomel reference electrode (SCE) and a Pt foil counter electrode in 2 mM H25 of the ESRF synchrotron. The required beam size was obtained with a Kirkpatrick-Baez mirror, which focused the beam down to ≈300 nm  × 165 nm  full width at half maximum, as determined by the ptychographic reconstruction of the probe using a Siemens star. The sample was positioned out of the focus of the beam to ensure the full illumination of a single nanoparticle by the beam. A coherent portion of the beam was selected with high precision slits by matching their horizontal and vertical gaps with the transverse coherence lengths of the beamline: 200 μm  and 60 μm . The intensity distribution around the 111 Pt reflections was measured in a vertical coplanar diffraction geometry. The BCDI experiment was performed at a beam energy of 9 keV. The diffracted beam was recorded with a 2D MAXIPIX26 photon-counting detector positioned on the detector arm at a distance of 0.5 m. The detector distance as well as the rocking angle increment were chosen in order to ensure oversampling of interference fringes (oversampling ~3). A typical counting time was 1 s per angle, to get good resolution while preserving the stability of the particle. The sample was mounted on a Physik Instrumente Mars xyz piezoelectric stage with a lateral stroke of 100 μm and a resolution of 2 nm, sitting on a hexapod that was mounted on a (3 + 2 circles) goniometer.BCDI measurements were performed at the upgraded ID01 beamline3 × 103 × 103 grid to avoid as much as possible introducing aliasing and converted into photons to a defined total diffracted intensity. No noise was introduced into the simulated diffraction pattern. Then, phase retrieval was performed as described in the next section.3D simulations were realized using the same pipeline as for experimental data. Using a support retrieved from a real dataset, a complex object was created using a particular phase pattern (flat or not). Then, the data was interpolated back into the detector plane using the same geometric parameters as the experiment. The diffraction pattern was calculated on a 10For 2D simulations, we used an asymmetric 2D core-shell model with a particular phase. The diffraction pattern was then calculated using FFT or a kinematic sum. Finally, phase retrieval was performed as described in the next section.36, imposing at each iteration that the calculated Fourier intensity of the guessed object agrees with the measured data. The metric used to estimate the goodness of fit during phasing was the free log-likelihood42, available in PyNX. Defective pixels for experimental data and gaps in the detector were not used for imposing the reciprocal space constraint mentioned above and thus were evolving freely during phasing. The initial support, which is the constraint in real space, was estimated from the autocorrelation of the diffraction intensity. A series of 1400 Relaxed Averaged Alternating Reflections (RAAR45) plus 200 Error-Reduction (ER47) steps, including shrink wrap algorithm48 every 20 iterations were used.Phase retrieval was carried out on the raw diffracted intensity data using PyNX package41. The reconstruction was then corrected for refraction and absorption, the small size of the particles ensuring that dynamical diffraction effects could be neglected38. After removing the phase ramp and phase offset, the data was finally interpolated onto an orthogonal grid for easier visualization. For each simulated case, we calculated the resolution (PRTF) and error bars for strain based on these five independent reconstructions. We used a single reconstruction to make figures.For 3D simulations and experimental data, we kept only the best 5 reconstructions from 200 with random phase start and a known support. For experimental data, which was measured in a focused beam out of focal plane, the phasing process included a partial coherence algorithm to account for the partially incoherent incoming wave front42.For 2D simulations, convergence of the phasing algorithm was more challenging because of the complex electron density and phase configuration of the model. To ensure the best reconstruction possible, we kept only the best 50 reconstructions from 1000 with random phase start and a known support (800 RAAR + 200 ER), and performed the decomposition in modesWe applied apodization either before or after phase retrieval. Before phase retrieval, it consists of multiplying the 3D raw data array by the 3D apodization window, which was chosen as a Blackman window or a Tukey window of parameter α = 0.7.For post-processing apodization, the Fourier transform of the reconstructed object was calculated in order to go back to the frequency domain. Then, we multiplied it by the 3D filtering window of the same size as the reconstructed array. Finally, we calculated back the complex object in real space by applying a FFT.supplementary information


The production of a  Measurements of vector boson production in association with jets provide fundamental tests of quantum chromodynamics (QCD). The high centre-of-mass energy at the CERN LHC allows the production of an electroweak boson along with a large number of jets with large transverse momenta. A precise knowledge of the kinematic distributions in processes with large jet multiplicity is essential to exploit the potential of the LHC experiments. Comparison of the measurements with predictions motivates additional Monte Carlo (MC) generator development and improves our understanding of the prediction uncertainties. Furthermore, the production of a massive vector boson together with jets is an important background to a number of standard model (SM) processes  of the first, second, and third jets, where the jets are ordered by decreasing The cross section is measured as a function of the jet multiplicity  . When comentclasspt{minimaEvents of interest are selected using a two-tiered trigger system . The firThe cross section is measured for jet multiplicities up to 6 and differentially as a function of the transverse momentum of the N jets and as “exclusive” for events with exactly N jets.Jet kinematic variables are measured for event samples with at least one, two, and three jets. In the following, the jet multiplicity will be referred to as “inclusive” to designate events with at least The balance between the }|y|<2.4 , 15. TheThe distribution of The n theory , 17. TheThe measured MadGraph 5_amc@nlo version 2.2.2 (denoted MG5_aMC) [pythia8 (version 8.212) [pythia8 is used to include initial- and final-state parton showers and hadronisation. Its settings are defined by the CUETP8M1 tune [The first two calculations use MG5_aMC) , which in 8.212) . pythia88M1 tune , in partt}kT-MLM , 23 sche0 LO PDF  is used the FxFx  scheme wgeneva 1.0-RC2 MC program (GE), where an NNLO calculation for Drell–Yan production is combined with higher-order resummation [pythia8 using the same tune as for MG5_aMC.The third calculation uses the ummation , 27. Log in Ref. , are resC15 NNLO  is used N-jettiness subtraction scheme  , 31. The}Njetti)  is used fewz  [MG5_aMC predictions are given in Table fewz, the cross section is computed in the dimuon channel, using a mass threshold applied after QED FSR, but including the photons around the lepton at a distance Given the large uncertainty in the LO calculation for the total cross section, the prediction with LO MEs is rescaled to match the th fewz   (versionmentclass2pt{minimtheo. unc. in the figure legends) are estimated for the NLO MG5_aMC, NNLO, and geneva calculations following the prescriptions recommended by the authors of the respective generators. The uncertainty coming from missing terms in the fixed-order calculation is estimated by varying the normalisation  [powheg interfaced to pythia8   are geneThe events collected at the LHC contain multiple superimposed proton-proton collisions within a single beam crossing, an effect known as pileup. Samples of simulated pileup are generated with a distribution of proton-proton interactions per beam bunch crossing close to that observed in data. The number of pileup interactions, averaging around 20, varies with the beam conditions. The correct description of pileup is ensured by reweighting the simulated sample to match the number of interactions measured in data.The particle-flow (PF) algorithm  is used The online trigger selects events with two isolated electrons (muons) with transverse momenta of at least 17 and 12 (17 and 8) in Refs. , 12. TheJets at the generator level are defined from the stable particles . It typically amounts to 5% for a jet multiplicity of one and increases with the number of reconstructed jets. The uncertainty in the jet resolution (JER), which is responsible for the bin-to-bin migrations that is corrected by the unfolding, is estimated and the resulting uncertainty is typically 1%.The most important uncertainty after the JES arises from the measured efficiency (Eff) of trigger, lepton reconstruction, and lepton identification, which results in a measurement uncertainty of about 2% up to 4% for events with leptons of large transverse momenta. The uncertainty in the measurement of the integrated luminosity (Lumi) is 2.3% . The resThe largest background contribution to the uncertainty (Bkg) comes from the reweighting procedure for the Unf model. An additional uncertainty comes from the finite size of the simulation sample used to build the response matrix. This source of uncertainty is denoted Unf stat in the table and is included in the systematic uncertainty of the measurement.Because of the finite binning a different distribution will lead to a different response matrix. This uncertainty is estimated by weighting the simulation to agree with the data in each distribution and building a new response matrix. The weighting is done using a finer binning than for the measurement. The difference between the nominal results and the results unfolded using the alternative response matrix is taken as the systematic uncertainty, denoted The measurements from the electron and muon channels are found to be consistent and are combined using a weighted average as described in Ref. . For eacMG5_aMC and pythia8 the number of partons included in the ME calculation and the order of the calculation is indicated by distinctive labels  MG5_aMC predictions, which give a fair description of the data for multiplicities above three (four).Figure mentclass2pt{minimMG5_aMC, showing a better agreement than the NNLLgeneva. The shape of the distribution in the region below 10geneva than by the other predictions, as shown by the flat ratio plot. This kinematic region is covered by events with extra hadronic activity in addition to the jet required by the event selection. The estimation of the uncertainty in the shape in this region shows that it is dominated by the statistical uncertainty, represented by error bars on the plot since the systematic uncertainties are negligible. In the intermediate region, geneva predicts a steeper rise for the distribution than the other two predictions and than the measurement. The high-geneva and NLO MG5_aMC are expected to have similar accuracy (NLO), is equally well described by the two. The LO predictions undershoot the measurement in this region despite the normalisation of the total The measured cross section as a function of the transverse momentum of the MG5_aMC predicted spectrum differs from the measurement, showing a steeper slope in the low MG5_aMC and geneva prediction shows good agreement for the measured The jet transverse momenta for the urements . The comMG5_aMC calculation predicts fewer events than found in the data for the region MG5_aMC are compatible with the measurement, although the contribution in the region MG5_aMC. The contribution at lower values of geneva generator predicts a steeper spectrum than measured. For jet multiplicities of at least one, we also compare with MG5_aMC. The uncertainty for The total jet activity has been measured via the mentclasspt{minimaMG5_aMC predictions. The slopes of the distributions for the first two jet multiplicities predicted by LO MG5_aMC do not fully describe the data. This observation indicates that the NLO correction is important for the description of hadronic activity beyond the jet acceptance used in this analysis, MG5_aMC sample and LO accuracy for the two other samples. In the case of the geneva simulation, when at least two jets are required, as in the second plot of Fig. MG5_aMC prediction, which has an effective LO accuracy for this observable, starts to show discrepancies with the measurement. The estimated theoretical uncertainties cover the observed discrepancies.The balance in transverse momentum between the jets and the mentclasspt{minimamentclasspt{minimaMG5_aMC prediction provides a better description of the geneva and LO MG5_aMC. This applies to both configurations, The We have measured differential cross sections for the production of a y) for inclusive jet multiplicities of 1, 2, and 3. The balance in transverse momentum between the reconstructed jet recoil and the The cross section has been measured as functions of the exclusive and inclusive jet multiplicities up to 6, of the transverse momentum of the MG5_aMC. The third is a combination of NNLO calculation with NNLL resummation, based on geneva. The fourth is a fixed order NNLO calculation of one pythia8.The results are compared to the predictions of four different calculations. The first two merge matrix elements with different final-state parton multiplicities. The first is LO for multiplicities up to 4, the second NLO for multiplicities up to 2 and LO for a jet multiplicity of 3, and both are based on The measurements are in good agreement with the results of the NLO multiparton calculation. Even the measurements for events with more than 2 jets agree within the The transverse momentum balance between the Inclusive measurement for events with at least one jet are compared with the NNLO The NNLO+NNLL predictions provide similar agreement for the measurements of the kinematic variables of the two leading jets, but fail to describe observables sensitive to extra jets. At low transverse momentum of the The results suggest using multiparton NLO predictions for the estimation of the


Quantum calculations plus lipophilicity (log P) lead to usefully accurate predictions of binding affinity that allow correction of crystal structures. P values for the ligand (which can be computed easily) this approach can significantly improve the agreement between computed and measured binding energies. When the approach is applied to lactate dehydrogenase A, it can make quantitative predictions about conformational, tautomeric and protonation state preferences as well as stereoselectivity and even identifies potential errors in structures deposited in the Protein Data Bank for this enzyme. By broadening the evidence base for these structures from only the diffraction data, more chemically realistic structures can be proposed.Accurately computing the geometry and energy of host–guest and protein–ligand interactions requires a physically accurate description of the forces in action. Quantum mechanics can provide this accuracy but the calculations can require a prohibitive quantity of computational resources. The size of the calculations can be reduced by including only the atoms of the receptor that are in close proximity to the ligand. We show that when combined with log  Others have previously lamented the reliance on these statistics and particularly when they distract from achieving a sound chemical interpretation that correctly accounts for the physical interactions at play.14The interaction of small guest molecules with larger hosts in aqueous systems is a key component of many applications including: sensors for Correctly quantifying the interaction strength between a host and a guest or a protein and a ligand computationally requires a physically accurate description of the molecules, of the kind provided by QM calculations. Despite significant progress, such calculations remain challenging because of their computational cost. Including only the portion of the host or protein directly involved in binding reduces this cost and examples of this approach are mentioned in recent reviews.Alternative approaches to understanding the behavior of proteins and quantifying their interaction strength with ligands have been pursued vigorously over the past decade. The majority of this effort has been in the area of simulation and recent advances in free energy perturbation (FEP) are particularly noteworthy.Structural biology studies often entail generating crystalline solids containing protein–ligand complexes. These are irradiated to create X-ray diffraction patterns, arising from interactions with electrons in the molecules. Software is employed to link from atomic coordinates to electron density and hence X-ray diffraction patterns; thus a likely set of atomic positions can be identified. These interpretations can be problematic and are rarely unambiguous.P values provides a usefully accurate treatment that accounts for the differences between binding energies computed with quantum mechanics and experimentally observed values. We style the resulting type of calculation as a “theoceptor”  by analogy to the theozymes  of Houk and co-workers.We propose an approach in which a correlation with computed log The current study is presented in three parts. In the first, we describe our approach. Subsequently, we describe our theoceptor for LDHA. Finally, we show how these calculations can increase the value of protein–ligand crystal structures and provide alternative interpretations of the electron density for several examples taken from LDHA.2Quantum mechanical calculations generally deal with the gas phase electronic energy of the system whereas experimental observations in biological systems are related to free energies in solution (eqn (1)).Gbind(aq)) is related to the change in gas phase electronic energy ) computed with QM. Corrections (ΔHcorr) are added to obtain gas phase enthalpies and a gas phase entropy change (ΔS(gas)) is computed. The change in solvation free energies (ΔΔGsolv) completes the link to the experimental environment.The free energy of binding in solution (ΔHcorr(gas) – TΔS(gas)), it is important to consider the relative contribution of polar and non-polar groups. Furthermore, the solvation free energy of the ligand and of the complex in water will be enhanced by the presence of polar groups on the ligand and diminished by non-polar regions. The solvation free energy of the protein is a constant and so accounting for whether the groups involved in binding are polar or non-polar offers a simple way to account for the change in solvation free energy (ΔΔGsolv).The electronic energies in eqn (2) can account for the energy of the complex but do not indicate the shape of the potential energy surface (PES), which describes how rigid the complex is. When two molecular species come into proximity, the potential energy surface can involve a narrow minimum or a wide one . A narroP has been used to differentiate ligands that rely primarily upon polar interactions from those that are hydrophobic.P, and its logarithm are convenient parameters because they are easily measured and can be accurately predicted.P can serve as a surrogate for the three terms on the right of eqn (1) when computing its binding free energy.α, β and γ are constants for that system. We propose obtaining values for these empirically and ΔE is defined by eqn (2), where E indicates a quantum mechanical energy that is obtained with no vibrational corrections included. The energy of the receptor (Ereceptor) in this equation is arbitrary because it is constant for all ligands but it will influence the resulting value of both α and γ. A range of affinity assessment types  can be used but should all be transformed such that they will have a linear relationship with energy. We do this by transformation to pIC50, pKi, pKd such that improved affinity corresponds to increasing values of each of these properties; α should therefore be negative and β positive if more negative values of ΔE and increased lipophilicity correspond to tighter binding. In order to obtain values for the three constants, at least three compounds of known affinity are required and thus, like the linear interaction energy approach, only relative affinity can be computed.E would usually be computed using the lowest energy protein–ligand complex and lowest energy conformation of the free ligand and therefore, be relevant to cases where the bound state is dominated by one geometry.Medicinal chemists have been concerned about the influence of non-polar groups upon binding affinity for some time and log E will be large and negative and this will offset the low log P term. A polar ligand that does not make a satisfactory set of polar interactions will have a less beneficial ΔE term that will not be sufficient to counteract the small log P contribution. A largely hydrophobic ligand will also have a smaller contribution to binding affinity from ΔE but this might be offset by the hydrophobicity term. There have been a number of reports in which quantum mechanical calculations have been used to compute binding energies for protein–ligand and other host–guest interactions. Energy differences are computed with no vibrational corrections and provide data with which this concept can be tested . The examples  agonists, using a range of DFT methods, with M062X performing the best.et al.et al. who used DFT to explore the relationship between the measured and predicted affinities for a set of positively charged amidine and guanidine cores binding to the β-site of APP cleaving enzyme (BACE-1).53The effect of eqn (3) is that for a polar ligand that makes a full set of polar interactions with the receptor, Δexamples  include:P improved the description of measured binding affinity by the QM energies. To make the comparison in the most straightforward way, two-parameter linear regression has been performed using firstly ΔE and calculated log P as independent variables and secondly ΔE and a random value that spans the same range as the log P values. The statistics for the second cannot be worse than those for the single parameter model using only ΔE. The resulting R2 and RMSE values are shown in P improved the correlation. For system A, Roos et al. also performed FEP calculations and achieved a correlation with R2 of 0.60 and RMSE of 0.73 (worse correlation is found when correlating with pKi–log D instead of just pKi).52As shown in β = 0.24) are binding to a highly polarized dianionic binding site while the iNOS inhibitors (β = 1.22) bind to a site dominated by the hydrophobic porphyrin of heme and MR agonists (β = 1.58) are binding to sites which lack substantial polarity (a lone threonine provides a point of polar interaction). The CDK2 binding site (β = 0.88) includes hydrophobic residues and several polar interactions. The more hydrophobic the binding site, the higher the value of β.et al.β are higher than for the protein–ligand systems. This likely reflects the absence of polar groups inside the cavities (unlike proteins which must contain amide groups). The sets of compounds studied in host–guest interactions are smaller than for the protein–ligand binding examples and this likely explains the larger standard deviations observed for the values of the coefficients in The values obtained for the coefficients in eqn (3) are shown in 1–11) studied include seven from high-throughput screening (HTS) of the Genentech/Roche corporate compound collection;To investigate the utility of the theoceptor approach, it has been applied to a system of therapeutic interest: Lactate Dehydrogenase A (LDHA). LDHA catalyses the reversible conversion of pyruvate to lactate, with the concomitant conversion of NADH to NAD+. Several classes of cancers are characterised by elevated levels of lactate, and LDHA is overexpressed in human tumors.1–7 are all available in complex with human LDHA.8 and 10 are available in complex with rat LDHA ,11 is available in complex with rabbit LDHA .σ of the ligands with lower RSCC values  are shown in section S5.Structures of 4QO7 . Protonation states were assigned such that Arg168 is protonated, Asp165 and Asp194 are both deprotonated and His192 is protonated in order to form a stabilizing interaction with Asp165 and the net charge of the residues was 0. There are two noteworthy changes in the sidechain conformations amongst the deposited structures but one arrangement is clearly most relevant to the bound state in humans: Arg105 protrudes into the binding site in rat and rabbit structures as well as in the apo form of human LDHA but in the holo forms of the human protein, it interacts with Glu191 instead (and so it is excluded from the theoceptor) while Tyr238 moves out of the binding site only in the apo form (and hence is included). Asp165 forms a hydrogen bond with His192 and is stabilised by an additional hydrogen bond with one water molecule. Preliminary studies showed that the water molecule maintains the sidechain in an optimal position to interact with the His192 sidechain and so is considered structural and was kept in the theoceptor  with Ereceptor being obtained from a theoceptor calculation in which the ligand had been deleted. The values of the coefficients were α = –0.080, β = 0.892 and γ = 0.419 and yield the relationship shown in β, when compared to those in Residues directly involved in ligand binding were selected as a part of the theoceptor: Arg168, His192, Val234, Asp165, Asp194, Tyr238, and Asn137 . These weoceptor , insert.2.11 is crystallographically modeled in a boat conformation that yields a predicted affinity far from experiment. This is corrected when modeled as a chair; the free conformation is 9 kcal mol–1 lower than the boat and the theoceptor with 1 in a chair conformation is 10 kcal mol–1 lower in energy than with a boat conformation . Inappropriate restraints and the statistics used to analyze the outcome misled the original refinement resulting in the wrong ligand conformation. Occasionally (see below) the correct conformation of a ring will not be so trivial and in these circumstances a quantum mechanical binding energy that accounts for the conformational preference of the ligand in the field of the protein can provide a very useful companion to a structural refinement.In order to investigate this further, the deposited structure was re-refined (using the PHENIX suite of software)2.22 of 0.83 suggests that it does not fit the density map accurately and inspection reveals that the density around the indane moiety is missing. The first step to model this compound was to determine the preferred position of the oxygen atom in the dihydropyrone ring, because this can easily be misplaced when interpreting electron density maps.R by Fauber et al. on the basis of the protein–ligand crystal structure.–1, indicating that the ring oxygen is not interacting with the theoceptor and the proposed positioning is reasonable.–1 but in the complex with LDHA, the conformation with the phenyl in the pseudo-axial conformation is 8 kcal mol–1 lower in energy than in the pseudo-equatorial conformation (R stereochemistry retained for ease of comparison). We therefore propose a bound structure in which the position of the ring oxygen is maintained but the phenyl group is in the pseudo-axial position. This provides a better interpretation of the experimental data and suggests that the absolute stereochemistry is likely to have been misassigned based on the deposited crystal structure and is actually S.The RSCC for ligand ormation . This enRwork and Rfree obtained during this phase are provided in To investigate this possibility, the deposited structure was re-refined. In this case, the ligand is only present in one of the proteins in the asymmetric unit and geometries of the two stereoisomers and two conformations were created in GaussView and were overlaid onto the ligand in the deposited structure using the pair_fit command in pymol.2.32, the pyridine nitrogen in 11 is impossible to place with certainty at 2.2 Å resolution.–1 favouring the ligand with nitrogen oriented as in the deposited structure. This preference is due to an interaction of the pyridine nitrogen with a positively charged environment formed by the sidechains of His192 and Asn137. The theoceptor provides extra support for the deposited structure.As with the oxygen atom in 2.43 is in an axial position relative to the 3-hydroxycyclohexenone ring in the deposited structure. Somewhat surprisingly, axial and equatorial ligand conformations were computed to be energetically indistinguishable in the unbound state . In 3 there are no axial hydrogens located on the same face of the ring as the phenyl group and consequently, no clashes. However, there is a clear preference in the bound state: the theoceptor with the ligand in an equatorial conformation is 4 kcal mol–1 higher in energy. This difference originates from protein–ligand edge to face stacking present when the ligand is in an axial conformation  of +44 kcal mol–1 is not in the range found for other ligands described here. This arises due to the loss of solvation energy that is not compensated by sufficient interactions in the receptor. These findings suggest that the malonate moiety may bind in a mono-deprotonated form. Three different malonate positions were investigated for monoanionic 9 in the theoceptor. The lowest energy theoceptor was the one where the malonate moiety made four hydrogen bonds with the nearby sidechains . The energy difference of 15 kcal mol–1 between the highest and lowest energy theoceptor emphasizes the effect subtle conformational changes of the interacting groups can have on the overall binding energy, particularly when strongly interacting groups such as anions are involved.Initial positioning of the malonate derivatives was carried out by superimposing the malonate moiety on the 3-hydroxy cyclohexenone ring of ligand 2.6–1). The correct orientation of malonate in the binding cavity could also not be confidently determined: optimization of 8 different theoceptors, each with a different malonate starting position resulted in complexes with ΔE values ranging from –8 to +16 kcal mol–1. These findings, in conjunction with experimental ambiguity concerning the state of the protein during the experimental measurements, suggested that malonate may bind in the presence of and proximal to cofactor NAD+/NADH.E with co-factor in its reduced form was 5.4 kcal mol–1 and in its oxidized form almost –40 kcal mol–1, which is consistent with the experimental affinities. Finally, the binding of oxamate was investigated. Again, binding is facilitated by the co-factor: the energy difference between the theoceptor with and without co-factor bound was more than 50 kcal mol–1 .When considering the binding of malonate itself, a total of 8 different malonate tautomeric monoanions were optimized in solution and all were found to be energetically indistinguishable (within a range spanning 0.5 kcal molal mol–1 . These ftyrosine . The fav3P values can provide usefully accurate predictions of host–guest and protein–ligand binding energies. By assigning relative energies to protein–ligand structures in which different conformations, tautomers, protonation states and stereoisomers are sampled, useful insights about the interactions between the protein and ligand are generated. This can include regions that are not observed in the experimental electronic density. Computed structures reveal the limitations of the statistics currently used during the refinement of X-ray crystal structures. Our approach is sufficiently accurate to make useful predictions about the affinity for new compounds. Chemists must become proactive creators of protein–ligand crystal structures rather than passive consumers; quantum mechanics provides a useful tool to do this in an objective fashion.The combination of computed quantum mechanical binding energies with measured or predicted ligand log AGL is a former employee and shareholder of AstraZeneca and is a shareholder of Medchemica Limited.


The advance of scene understanding methods based on machine learning relies on the availability of large ground truth datasets, which are essential for their training and evaluation. Construction of such datasets with imagery from real sensor data however typically requires much manual annotation of semantic regions in the data, delivered by substantial human labour. To speed up this process, we propose a framework for semantic annotation of scenes captured by moving camera(s), e.g., mounted on a vehicle or robot. It makes use of an available 3D model of the traversed scene to project segmented 3D objects into each camera frame to obtain an initial annotation of the associated 2D image, which is followed by manual refinement by the user. The refined annotation can be transferred to the next consecutive frame using optical flow estimation. We have evaluated the efficiency of the proposed framework during the production of a labelled outdoor dataset. The analysis of annotation times shows that up to 43% less effort is required on average, and the consistency of the labelling is also improved. Annotation of Ground Truth (GT) data is now an important task in research. This can be attributed to machine learning becoming a mainstream approach to solving a wide range of problems, especially in machine perception and sensing. The popularity of deep neural networks resulted in the development of efficient platforms for their design, training and evaluation, ultimately reducing the original problem into searching for a sufficient number of samples required to tune the network parameters and structure.In the case of computer vision, the aim is to develop methods that work on data captured by real sensors, e.g., to detect pedestrians from a stereo camera mounted on a car. Supervised training of a deep network for this task typically requires 10–100 k sample GT images with annotated objects of interest. Researchers have reduced this problem by synthesizing images from virtual models, where a perfect GT can be also rendered. Real sensors however produce a wide range of artefacts  that are still difficult to model in a virtual camera; hence, a number of real images are still required to fine-tune the network to make it work in practice.Visual data can be semantically annotated in several ways. The ideal description of the observed 3D scene as humans understand it would be a hierarchical segmentation of the scene typically into regions of adjacent matter, as associated with individual objects, groups of objects or object parts, each associated with a semantic label or category. In practice, the extent of the hierarchy is limited in depth (semantic resolution ), detailA 2D view of the scene from a camera captures a snapshot of the same 3D hierarchy, derived by projection from the geometry and the structure of the scene. The corresponding 2D digital image can be annotated at different levels or scales. Depending on the task, researchers choose from a range of labels spanning from individual pixel-wise labelling to whole image categorization, including parametric image regions delineated by rectangular or polyhedral bounding boxes, landmarks localized with points or circles, etc. Purely manual methods to produce the annotations require users to enter the labels exactly, independently for all samples, with the total time proportional to typical “unit” sample annotation time. The average unit time can be decreased by providing efficient tools and interfaces to the annotators . These cAutomation can however introduce some bias in the resulting ground truth. It will likely guide the annotator to what is preferred by the method’s data model , different from what the manual result might be. The bias can be both negative, e.g., when the used edge is not the actual boundary of the object, or positive, e.g., by automatically discovering an object or part that would otherwise be overlooked by the annotator. The aim of annotation research is to propose approaches where the benefits and efficiency improvements outweigh the potential negative bias. In our approach, we derive a good 2D initialization of smaller objects and parts from their 3D representation and apply associated semantic labels. This is particularly useful with a large set of classes, when the correct label cannot be easily identified from the 2D appearance.https://github.com/rtylecek/rosemat) that reads the input from ROS bagfile archives and uses the contained metadata to associate image streams with an external 3D model of the scene. This allows us to generate initial pixel-wise semantic annotations. Annotations can be manually refined using a semantic paintbrush, and the refined result can be transferred to consecutive frames using optical flow. In this way, we eliminate much of the labour normally spent on repeated labelling of slightly changing views (from video rate data as the camera moves) of the same scene. In particular, we highlight the following advantages over existing approaches:It handles the point cloud representation preferred for natural outdoor scenes (mesh not needed).It provides an interface for efficient 2D refinement of annotations (does not rely on a good 3D model).It supports frame-to-frame transfer of annotations in video using optical flow.Integration with the ROS platform reduces the data preparation time for robotic applications.We have developed an annotation tool . We introduce a point cloud projection technique that can deal with the artefacts usually arising from the sparse nature of the point cloud, like holes.From the above review, we can see that there has been little work on 2D image labelling from 3D point cloud labels, and here, we introduce a 3D labelling process based on transfer from a manually-created 3D sketch map. To the best of our knowledge, the 2D optical flow label propagation has not been used so far in the context of semantic annotation tools. These are methods that we promote in this paper.Primarily, we employ a process where the time-consuming task of human annotation of image sequences can be facilitated by projection of an annotated 3D geometry (semantic point cloud) into images given the camera poses. The subsequent key observation we exploit is that we can estimate the image motion by optical flow and use it to transfer labels between consecutive frames. For this purpose, we present a workflow described in the sections below following the schematic overview in Assume the general case of a rig with s Kalibr .c at time The primary inputs are streams of colour images Dense and accurate point clouds can be obtained with a stationary laser scanner, such as the Leica ScanStation, which was used in the experiments reported below. Scans from multiple locations to cover all surfaces with measurements are merged to obtain a single point cloud The 6 DOF robot pose Alternatively, Structure-from-Motion (SfM) algorithms such as  can be uOur goal is to help the user specify a 3D semantic model of the scene. Ultimately, this means a semantic label L is defined by the user in a two-level hierarchy, where the first level general classes can have the second level specific subclasses. In practice, they are listed in a configuration file as general-specific label pairs, as shown in The label set With millions of points in a typical input set ground: the horizontal terrain with different types of surfaces, e.g., grass or pavement,objects: semantically meaningful parts of the scene, e.g., trees or bushes,background: the part of the scene outside of the region of interest.We exploit the usual outdoor scene structure to sequentially split the input point cloud background points are cropped out. The remaining foreground part is processed with the segmentation method [objects are identified as connected components of the remaining point cloud above the ground. This process is implemented in CloudCompare [The segmentation is obtained by sequentially splitting the input point cloud. First, the perimeter of the region of interest is manually specified, and outlying n method , which tdCompare  softwareG and a set of primitive shapes The 3D Map Editor  of the ground segment. Vertices of the DT are uniformly sampled from ground points.Ground mesh The 3D map is initialized from the segmented point cloud (background part excluded):Both parts are then manually adjusted, e.g., to prevent overlaps of the object bounding boxes, and the shape can be also changed to a sphere, cylinder or cone. A semantic label is manually assigned to every object shape M. First, object bounding shapes Point cloud semantic labels The remaining ground points get the label of the surface mesh face The next step is to annotate 2D images in the video stream(s). The previously produced semantic point cloud is loaded into the 2D Image Annotation Tool  standards and uses several publicly available modules with a user interface. ROS is a popular framework to manage and run components required for robot control and machine perception. It defines also standards for data exchange, which are useful to record data streams from multiple sensors simultaneously, e.g., images from colour and depth cameras, their poses, along with metadata like timestamps and coordinate system references. These can be stored in an archive called a rosbag.https://github.com/dsgou/annotator) allows only manual video annotation with bounding boxes. Some other tools can be used to attach string tags to the recorded timeline.There are only a few annotation tools available for the ROS platform. The multimedia stream annotator ,Move a selected vertex ,Adjust the elevation (Z ) of a selected vertex or face,Insert objects of primitive shapes ,Change dimensions of the shapes and orientation ,Assign a semantic label from the list to a selected face of the ground mesh or object.The top orthogonal 2D view is shown in Import a segmented point cloud and initialize objects from its components,Export a semantic point cloud with labels corresponding to the current 3D map.The 3D view mode shown in The 2D Image Annotation Tool we have created allows to load an ROS bagfile with multiple image streams together with camera calibration and the semantic 3D model from a point cloud, which can be projected into the images. The workspace  can be saved in a configuration file (YAML) and loaded later.Switch between multiple camera topics and image frames,Transparently overlay semantic labels on the original image with adjustable opacity f,Initialize the frame from 3D projection c,Translate and rotate the current semantic map in the image frame,Automatically refine annotation boundaries to align with contours of the original image (super-pixel boundaries ),Draw user-selected semantic labels with a brush of adjustable size d,Draw region boundaries and fill the semantic or image region,Transfer labels to the next frame e,Export annotations and overlays with the option of label set reduction (top classes only or custom).The drawing interface shown in The 3D camera pose associated with the current frame (translation and rotation) can be also manually adjusted to better fit the projection of the semantically-labelled point cloud to the image .The typical annotation of a sequence will start with the 3D projection initialization of the first frame, aligned to the image, followed by the manual refinement. The result is then propagated to the next frame using optical flow and refined again. When the view changes too much, e.g., after rotation or some frames are skipped, the frame can be again initialized from 3D. This process is repeated until the end of the sequence is reached. The annotations are immediately available in the workspace folder as indexed bitmaps (PNG) with an embedded colour map .The proposed framework was used to annotate the dataset presented in Empty annotation ,Projection of the 3D semantic model to the image (3D-2D projection),Transferring labels from the previous frame using calculated optical flow (2D flow transfer).We have compared frame annotation initialized in three different ways:We have measured consistency, accuracy and time to quantify the comparison.U is the set of annotators, j label annotated by user u and We asked a group of three users to annotate a set of 10 non-consecutive frames independently, both manually and with the initialization from the semantically-labelled 3D model. Multiple annotations allowed us to calculate the variance between individual annotators. It was calculated as the mean pixel-wise label variation Segmentation inaccurate: variation of object boundaries.Under-segmentation: objects or a part missing.Semantic class mismatch: different labels assigned to objects or parts.The results are shown in The visual comparison of label variance We annotated a sequence of 50 consecutive frames In the typical scenario when the motion was limited by a high frame rate and the low velocity of the moving cameras to approximately 20 pixels in the image, the results suggested that the estimated optical flow was accurate enough to adjust the moved boundaries of objects in the image. The pixel-wise measure showed over 40% improvement on average, but if we consider the usual variation of the boundaries due to human factors as in The observed annotation statistics are summarized in The bottom block compares labelling of 180 consecutive frames completed manually (20 min each) versus refining the transferred labels (11 min each). A subset of 50 frames (one sequence) was annotated using initialization with a copy of the previous frame annotation, which needed manual refinement of 3.7% pixels to compensate the motion of the camera. The needed refinement was reduced to 2.1% when optical flow was used to transfer the labels from the previous frame.In addition to the reduction of labour measured pixel-wise, we see that the annotation time per frame was reduced by 5% in the case of initialization by 3D-2D projection and by 45% in the case of consecutive frame transfer. The run-time of dense optical flow estimation using  was typihttps://github.com/rtylecek/rosemat). First, the 3D Map Editor allows us to define semantic labels for a point cloud of the scene. Second, the 2D Image Annotation Tool can load the image streams from a bagfile along with camera calibration and poses. Annotation of a given frame is then initialized using projection of the point cloud and manually refined by the user. The refined annotation can then be transferred to the consecutive frames using estimated optical flow.We have presented a framework for semantic pixel-wise annotation of images. It is designed in particular for data captured by a moving robot. The implemented annotation workflow is based on two publicly available components (This pipeline was used to produce annotations for an outdoor dataset of a garden presented in The accuracy of the projection is however limited by a static projection model. In the future, we would like to improve the projection part to better adapt to the dynamics of the scene by matching the statically-projected contours to the currently-visible moved contours, e.g., in the case of branches moving in the wind. The projection and label transfer could be also done simultaneously and the result fused for the new frame, i.e., the part that could not be transferred from the previous frame would be initialized from projection. This would be particularly useful after rotation of the camera when new objects or parts enter the view.


Ipsilateral fracture of the shaft of femur and dislocation of the hip are very rare injuries. There always exists a dilemma regarding the treatment to reduce hip and choosing the appropriate method of ﬁxation for a femur fracture, and a clear consensus is yet to be reached. A number of treatment methods such as the open reduction of femur and ﬁxation followed by hip reduction have been tried so far. Ipsilateral fractures and dislocation occur due to high-energy trauma, and reduction of hip dislocation is considered as an orthopedic emergency. Here, we report a case in which we tried a novel approach by temporarily ﬁxing the femur with an external ﬁxator and reducing the hip dislocation. In the next sitting, we performed femur ﬁxation in a closed manner with an interlocking intramedullary nail. We recommend that this novel method of treatment can be used for such types of injuries. Ipsilateral fracture of the shaft of femur and dislocation of the hip are very rare injuries . There aAn 18-year-old male patient reported to our ED following a road traffic accident with severe pain in the right hip and right thigh. The patient had a deformity of the right hip and thigh, and was unable to walk. On clinical examination, we found that he had ﬂexion-adduction deformity of the right hip. His leg and foot showed external rotation and abduction. There was tenderness and abnormal mobility at the right thigh. In addition, there was tenderness at the right hip joint and the femoral head was palpable in the gluteal region. The range of movements at the right hip was grossly restricted and painful. There was no distal neurovascular deﬁcit noted.After providing the advanced trauma life support in causality, the patient was sent for radiography. X-rays of the pelvis with both hips and of the right thigh with hip and knee were taken Figure .CT scan of the hip and pelvis was also taken to exclude acetabulum fractures Figure .From the radiograph, we conﬁrmed that there was a fracture in the shaft of femur and a posterior dislocation in the hip. The patient was taken to an emergency OT and subjected to closed manipulation of the hip joint. As there was a fracture in the shaft of femur, the force was not transmitted to the hip joint and we are unable to reduce the hip dislocation. At that moment, we had two options: open reduction and external ﬁxation, and closed manipulation of the dislocation of the hip joint. We placed a temporary external ﬁxator in the femur under anesthesia, and performed closed manipulation and reduction of dislocation of the hip using the Allis method Figure .The patient was subjected to ﬁnal ﬁxation of the femur shaft fracture after one day in elective OT. Femur shaft fracture was ﬁxed with a closed reduction and an internal ﬁxation with an intramedullary femur interlocking nail Figure .The operation was done on a traction table under the guidance of ﬂuoroscopy.Postoperatively, the patient’s legs were placed on an abduction pillow and physiotherapy was started. Low-molecular-weight heparin was given as deep vein thrombosis (DVT) prophylaxis. The patient was not allowed weight-bearing for two weeks. Gradually, he was allowed partial weight-bearing based on his tolerance.Ipsilateral fracture of the shaft of femur and dislocation of the head are very rare injuries. So far, no consensus has been reached on the treatment of such injuries. Reduction of dislocation is considered an emergency; however, it is difficult  to perfoDifferent methods such as placing Steinmann pin in the proximal femur and reduction have been tried by various authors. These are followed by open reduction and internal ﬁxation. Moreover, closed or indirect reduction is more favorable than open reduction unless there is a need for internal ﬁxation for an acetabular or proximal femur fracture. Many complications are often associated with an open reduction  and inteIpsilateral fracture of the femur shaft and hip dislocation are rare injuries, and always difficult to be diagnosed and treated. These injuries are always considered as an orthopedic emergency. In such conditions, ﬁxator-assisted reduction of hip dislocation followed by closed reduction and internal ﬁxation with an intramedullary nail gives satisfactory results with minimal complications.


Flexible, self-powered and miniaturized sensors are extensively used in the areas of sports, soft robotics, health care and communication devices. Measurement of vibration is important for determining the mechanical properties of a structure, specifically the string tension in strings. In this work, a flexible, lightweight and self-powered sensor is developed and attached to a string to measure vibrations characteristics in strings. Electrospun poly(vinylidene) fluoride (PVDF) nanofibers are deposited on a flexible liquid crystal polymer (LCP) substrate for the development of the sensor. The electrospinning process is optimized for different needle sizes (0.34–0.84 mm) and flow rates (0.6–3 mL/h). The characterization of the sensor is done in a cantilever configuration and the test results indicate the sensor’s capability to measure the frequency and strain in the required range. The comparison of the results from the developed PVDF sensor and a commercial Laser Displacement Sensor (LDS) showed good resemblance (±0.2%) and a linear voltage profile (0.2 mV/με). The sensor, upon attachment to a racket string, is able to measure single impacts and sinusoidal vibrations. The repeatability of the results on the measurement of vibrations produced by an impact hammer and a mini shaker demonstrate an exciting new application for piezoelectric sensors. The advent of flexible electronics and Internet of Things (IOT) has led a surge in the development of smart devices in different facets of life. The applications of these devices cover key areas such as healthcare, sports, robotics and communication devices ,2,3,4. TStrings are important components of many natural and artificial structures around us, for example, human ligaments, cable suspension bridges, sports rackets, musical instruments, humanoid robots and fabrics. Measurement of impact and vibrations in strings are important because they can provide information about the mechanical properties of the strings such as the string tension. Taking an example of tennis rackets, larger vibrations are produced when the impact of the ball on the racket is near the center than an off-center impact. The sweet spot which is the desired impact point for a tennis player produces the least vibration and most power in the shot . Hence, There have been various research works involving sensors used to determine strain and frequency of vibration. Traditionally, piezoresistive films ,17 have Amongst the studies pertaining to sensors utilized to measure string properties, a few studies taking a racket string as an example have focused on developing sensors for determining strain and frequency. One commercial device by Stringmeter  involvesThis research focuses on the development of a self-powered, flexible and lightweight sensor that can be attached to a string. The sensor should be able to withstand high impact forces and yet be able to measure strain and vibration characteristics in the desired application range. In the present study, a vibration sensor made from electrospun PVDF nanofibers has been developed for the measurement of vibration on a string. Improved β-phase is obtained by varying the needle size and flow rate in the electrospinning of the PVDF nanofibers which in turn is known to improve the voltage output of the sensor. The sensor is tested in cantilever mode first for the required range of vibration frequencies and strain. The performance of the sensor is measured against a commercial Laser displacement sensor (LDS) and the sensor shows correct and repeatable performance. A linear voltage-strain relationship is obtained in the required range of strain. The sensor is attached to the string and tested for vibration measurement. The sensor shows accurate and repeatable readings on single impact and sinusoidal displacement of the string.PVDF nanofibers are formed using an electrospinning set-up with a needle and a rotating drum collector that is shown in The voltage applied is kept constant at 15 kV for all the experiments as the electrospinning process is found to be most stable at this voltage. The flow rate of the solution is kept constant at 2.0 mL/h which enables a smooth formation of the nanofibers. The distance between the needle tip and the collector is set at 150 mm. To ensure the quality of the nanofiber samples, scanning electron microscopy is used to examine the morphology of the nanofibers generated. The sample is prepared at a flow rate of 2.0 mL/h and a needle size of 0.34 mm. The SEM image in The sensor is composed of liquid crystal polymer (LCP) as the substrate, a set of copper electrode tapes and PVDF nanofibers as the sensing material. LCP is chosen as the substrate since it can be of low thickness, lightweight and highly bendable ,47. It hThe structure and the composition of the different phases of the polymorphic PVDF nanofibers after electrospinning is obtained by Fourier-transform infrared  spectroscopy. The surface morphology of the nanofiber samples is obtained using Field Emission Scanning Electron Microscope . The frequency of different sizes of nanofibers present in the sample is obtained using Image J which is an open-source image processing software.The vibration and strain inputs are provided using a mini shaker controlled by a frequency generator and amplifier . A laser displacement sensor (LDS)  is used to compare the frequency and strain readings of the sensor. A Babolat 19 Gauge string  is used for the string testing. The string is subjected to the required tension using a spring weight machine.The following sections describe the results of the composition study of PVDF, frequency and strain characterization of the sensor in cantilever mode and single impact and sinusoidal vibration testing of the sensor attached to the string.PVDF is a polymorphic polymer which can be found in five phases, α, β, γ, δ and ε phase . The nat−1 corresponds to the β-phase which is the main contributing phase to the piezoelectricity of PVDF. The peak at 840 cm−1 signifies β-phase, however, it is debated in the literature that this peak also belongs to γ-phase [−1 signifies the dominance of β-phase in the electrospun nanofibers. This occurs due to the stretching under high voltage applied in the electrospinning process, as the raw powder consists purely of α-phase. The β-phase content in the electrospun nanofibers can be calculated from the FTIR spectra with the assumption that the absorption spectrum follows the Beer-Lambert law. The absorption coefficients were calculated by Gregorio [−1. The relative fraction of β-phase in the sample is given by the following formula [F(β) represents the fraction of β-phase in the sample; αA and βA correspond to the absorbance values at 766 and 840 cm−1 and αK and βK are the absorption coefficients at 766 and 840 cm−1. The absorption coefficients values were given as 6.1 × 104 and 7.7 × 104 respectively [Three different sizes of needles with internal diameters of 0.84, 0.60 and 0.34 mm (18 G to 23 G) are used to form the nanofibers through electrospinning. Seven different flow rates ranging between 0.6 mL/h and 3.0 mL/h are used. The corresponding FTIR plots for the electrospun samples for different flow rates are shown in  γ-phase . The larGregorio  at the r formula :(1)F(β)=ectively . The corThe effect of flow rate and needle size is initially studied by the use of two needle sizes of 0.84 mm and 0.34 mm and flow rates of 0.6 mL/h and 1.2 mL/h. The results show a higher β-phase percentage at the lower needle size of 0.34 mm and a higher flow rate of 1.2 mL/h. The effect of flow rate is then calculated by keeping the needle size fixed at 0.34 mm and increasing the flow rate up to 3 mL/h. It is seen in 2 dipoles. Hence, there is no need for any additional poling for PVDF nanofibers electrospinning as is required in the case of the PVDF films [cX)for a given sample can be determined from its enthalpy of melting by the following relation:mH the melting enthalpy of the material and ΔH0 is the enthalpy of melting of the perfectly crystalline form of the material.Besides the larger fraction of β-phase for good piezoelectric response from the nanofiber web, the crystallinity or the preferential poling of the piezoelectric material is also an important factor that establishes whether the piezoelectric material will produce sufficient piezoelectric response to a stimulus such as strain. For this purpose, a differential scanning calorimetry (DSC) was performed for different nanofiber samples. It is also to be noted that according to one of the studies, it was reported that electrospinning of nanofibers with a rotating drum collector does provide a preferential poling for the CFDF films . The DSCH0 for PVDF is taken as 104.7 J/g [cα form [To determine the degree of crystallinity in this paper, the value of Δ04.7 J/g . The DSC04.7 J/g . A small[αc form . Althoug[αc form .cX with flow rate, this is shown in cX is seen to slightly reduce with an increase in the flow rate (0.6–3.0 mL/h). The slight reduction in the cX values, with an increase in the flow rate, can be attributed to the reduced solvent evaporation at higher flow rates. This reduces the effective stretching of the nanofibers between the nozzle and the collector under a constant electric field which can lead to a reduction in the degree of crystallinity of the nanofibers. This is in agreement with [On the variation of ent with  where thcX with needle size as shown in cX is observed to be almost constant with the needle size ranging from (0.34–0.84 mm). The ineffectiveness of the size of the needles on the crystallinity of the sample suggests that the shear force exerted by the capillary does not affect the chain orientation of the dipoles. This result is in contrast to [cX values with a reduction in needle diameter.For the variation of trast to  where a It has been shown in the past by various researchers that a larger percentage of β-phase in the FTIR calculations and a high degree of crystallinity leads to a larger voltage output hence the flow rate of 2.0 mL/h and a needle size of 0.34 mm is used to obtain nanofiber samples in further vibration sensor fabrication. These parameters are also utilized because the electrospinning jet is found to be continuous and process flow was smooth during electrospinning. The different compositions used for the preparation of the samples and their corresponding β-phase percentage and thermal properties have been summarized in The sensor is first subjected to several frequencies in the cantilever configuration and tested against a commercial laser displacement sensor (LDS) system to characterize its vibration response. The schematic of the experimental set-up is shown in The results of vibrating the sensor with six different frequencies are shown in The voltage output of the sensor corresponding to its tip displacement at a frequency of 90 Hz is shown in The comparison between the frequency measured by the PVDF sensor and that measured by the commercial LDS sensor for the frequency plots shown in h of the free end of the cantilever is given as:ε is the strain in the nanofibers, y is the height above or below the neutral axis and L is the total length of the cantilever.The second set of tests involves subjecting the proposed sensor to a range of tip displacement amplitude profiles ranging from 0.4–2.7 mm. The range of tip displacement amplitude is chosen as the span of deflection experienced near the end of a taut string at the edge of the racket frame considering a maximum displacement of 30 mm at the string centre. The range of strain was also suggested in other research work such as Maeda  where thThe results from the tests are shown in The characterization of the sensor for frequency and strain in cantilever mode is necessary to understand the behaviour and the performance of the sensor. The tests are also necessary to determine the suitability of the sensor in the working range required for the applications mentioned in A racket string is fixed between two C-clamps with a tension of 40 lbs achieved by pulling the string with a spring weight scale to the required tension value and clamping. The PVDF sensor is attached to the string with a resin-based glue and left overnight for the bond to strengthen. The output from the sensor is connected to a data acquisition system for recording the voltage output obtained from the sensor. The data acquisition is controlled by a computer connected to the DAQ unit. The mini shaker connected to a signal generator and amplifier is used to provide a sinusoidal displacement to the string. The mini shaker can be replaced with another object such as a hammer to characterize an impact on the string. The schematic of the experimental set-up is shown in An impact hammer test is first done to examine the performance of the sensor on a single impact force subjected to the string. The string is struck at the center with a light hammer constituting of a flat tip at intervals of around 5 s. The output of 5 such hits with the hammer is shown in The string is then subjected to a vibrational impact provided by the mini shaker and the output of the sensor is recorded. The output performance of the sensor is shown in The output voltage generated by the sensor varied from 10 mV for 180 Hz frequency to 50 mV peak to peak for a frequency of 40 Hz. The voltage output follows similar trend to the cantilever testing pattern with the output increasing with the frequency till 40 Hz and then dropping due to reduction in displacement amplitude as explained in Secondly, the interface between the string and the sensor could lead to a reduction in the overall strain experienced by the sensor. The results in A self-powered, flexible and light-weight PVDF nanofiber sensor on LCP substrate, that is capable of accurately measuring the vibration frequency and strain in the given range of applications has been developed. The sensor’s characteristics of being light-weight and self-powered makes it suitable for miniaturization. These characteristics enable its attachment to a string and measurement of the vibration characteristics of an impact force as well as a periodic force on the string.PVDF nanofibers were formed with the use of electrospinning technique. The characterization of the nanofibers was done for morphology using FESEM which showed fibers were formed without any beads and the mean diameter of the fibers was (218 ± 53 nm). The percentage of β-phase was optimised between three different needle sizes of 0.84, 0.60 and 0.34 mm and flow rates ranging between (0.6–3.0 mL/h). Since, for electrospun nanofibers in general, a higher β-phase leads to a higher voltage output from the nanofibers, the needle size of 0.34 mm and flow rate of 2.0 mL/h was used for sensor development.The experimental results of testing the sensor in the cantilever configuration showed a good agreement between the developed PVDF sensor and a commercial LDS system in identifying vibration frequencies between 2 to 180 Hz (± 0.2%). The output voltage of the sensor was seen to vary with the strain as well as the frequency of strain or the strain rate.For a free cantilever motion, the sensor readings show a linear relationship between voltage output and strain (0.2 mV/με) in the range of strain between 200 to 2300 με. This indicates the suitability of the sensor for the range of applications involving a string.The attachment of the developed PVDF sensor on a racket string was successfully achieved and the sensor showed a good response to vibration testing on the string. The results from a single impact hammer force on the string showed a repeatable voltage output of 25–35 mV from the sensor obtained with a light hit. The sensor also produced repeatable voltage output on the application of sinusoidal displacement of the string with a mini shaker. The development of a sensor that can be attached to a smaller curved surface like a string, opens a new area in the miniaturization of flexible sensors in the series of smart devices for IOT.