Electromyography (EMG) data has been extensively adopted as an intuitive interface for instructing human-robot collaboration. A major challenge to the real-time detection of human grasp intent is the identification of dynamic EMG from hand movements. Previous studies predominantly implemented the steady-state EMG classification with a small number of grasp patterns in dynamic situations, which are insufficient to generate differentiated control regarding the variation of muscular activity in practice. In order to better detect dynamic movements, more EMG variability could be integrated into the model. However, only limited research was conducted on such detection of dynamic grasp motions, and most existing assessments on non-static EMG classification either require supervised ground-truth timestamps of the movement status or only contain limited kinematic variations. In this study, we propose a framework for classifying dynamic EMG signals into gestures and examine the impact of different movement phases, using an unsupervised method to segment and label the action transitions. We collected and utilized data from large gesture vocabularies with multiple dynamic actions to encode the transitions from one grasp intent to another based on natural sequences of human grasp movements. The classifier for identifying the gesture label was constructed afterward based on the dynamic EMG signal, with no supervised annotation of kinematic movements required. Finally, we evaluated the performances of several training strategies using EMG data from different movement phases and explored the information revealed from each phase. All experiments were evaluated in a real-time style with the performance transitions presented over time. With the rapid development of human–robot interaction (HRI) technology, collaborative robotics have been widely utilized in the assistive environment and smart prosthetic hands. The activity detection of the human hand and arm  of human body structure. More specifically, the human hand alone consists of 21 DOFs controlled by 29 muscles . All subjects were right-handed and only the dominant hand was used for data collection. None of the subjects had any known motor or psychological disorders. The experimental procedure and tasks were explained to all subjects, and we obtained their consent to participate before the experiments.f = 1562.5 Hz. The visualization of the C = 12 targeted muscles of our experiment is shown in We collected surface EMG  in bipolar derivations, with a sampling rate of l∈{0, 1, ..., 13}, where l = 0 indicated the open-palm/rest gesture without any movement, and l∈{1, ..., 13} were accordingly identified as the other 13 gestures listed in The experimental protocol focused on 14 gestures and 4 dynamic motion phases involving commonly used gestures and wrist motions  was performed four times using four different objects, totaling 52 objects in each session. The subject performed 6 trials for each of the 52 objects per session, where each trial was executed along its corresponding predefined path, as shown in During the sessions, each gesture of At the beginning of the experiment, the subject was seated facing a table and electrodes were connected to the right arm while the arm was at the rest position with an open palm, as illustrated in Since the experimental procedure requires subjects to execute real movements, the data is prone to noises and motion artifacts. Thus, we applied a fourth-order band-pass (40 Hz to 500 Hz). Butterworth filter to remove the unwanted data contamination and clear any other frequency noise outside of the normal EMG range. No default filtering was applied to the data from the acquisition device.Due to the nature of the human muscle system, upper arm muscles generate stronger signals than hand muscles and thus a fair source contribution would be only possible by normalizing each muscle with respect to its maximum power. Therefore, the maximum voluntary contraction (MVC) test was conducted for each muscle where the subjects were asked to perform isometric constructions of muscles lasting for 3 s. The envelope of both the experiment and the MVC data were generated, and each channel of the experimental data was then normalized with respect to the maximum value of MVC envelopes.T = 320 ms, with a step size of 40 ms between two consecutive windows. The following EMG feature extraction and classification were all conducted based on each time window.In order to implement the EMG identification in real-time, we further divided the processed experimental data of EMG envelopes into sliding time windows of The selection of EMG features involves considering both processing time and data representability, at the same time avoiding redundancy to maximize the classification performance  by Hallac et al.  was adopIn this research, we segmented all EMG trials using GGS by assigning 3 breakpoints to each trial, which gave us 4 sequences corresponding to reaching, grasping, returning, and resting. By doing so, the order of different movement phases was taken into account by imposing a higher probability of movement transitions that were expected to follow one another, e.g., grasping movement following reaching movement, later followed by returning action. Given the specified number of breakpoints, the GGS algorithm was utilized to unsupervised segment the EMG trial and provide the optimal segment boundaries.In l∈{0, 1, ..., 13}, where l = 0 was defined as the open-palm/rest gesture and l∈{1, ..., 13} were accordingly identified as the other 13 gestures listed in l∈{1, ..., 13} executed during the movement, and tagged the stationary phase of resting with the open-palm label l = 0.Throughout the reach-to-grasp movement, the limb configurations of fingers and wrist together change continuously with respect to the shape and distance of the targeted object , where Xi represents the ith EMG window and C = 12 is the number of channels. Then, based on the gesture label l∈{0, 1, ..., 13} of the corresponding EMG window Xi, pairs of data and label n is the total number of windows. Later, the selected three time-domain features of RMS, MAV, and VAR were extracted as We constructed a classifier to recognize dynamic hand gestures using the collected EMG signals. The EMG signals were first broken down into time windows of In this research, we utilized the extra-trees method  for training and 112 trials (33.3%) for validation in total. Thereafter, the classifier was trained on the aforementioned training set and evaluated on the left-out validation set which was unseen to the classifier. Since our main objective was to detect the upcoming grasping intention at an earlier stage and pre-shape the robot before the final grasp was accomplished, the data from the returning phase were excluded from training, during which the grasp was already completed and the object was already released from the hand. However, to provide a full scope of the effectiveness of the classification approach, the classifier was then evaluated on the entire EMG trial of four movement phases irrespectively.As the subjects were not required to perform the four movement phases of reach-grasp-return-rest at a fixed timing or speed during the data collection, the duration of each phase for different trials varies based on the subject's natural pace. Therefore, during the evaluation, we aligned all validation EMG trials based on the breakpoint occurring at the beginning of the grasping phase, as shown at 0 ms in As mentioned earlier, we trained the classifier with reaching, grasping, and resting phases only, whereas validated the model with all motion phases . In order to specifically inspect the performance of the dynamic-EMG classifier and explore the informative levels of EMG data from different motion phases, the constructed gesture classifier was trained with three different strategies independently:Trained with EMG of reaching and resting phases;Trained with EMG of grasping and resting phases;Trained with EMG of reaching, grasping, and resting phases.9, respectively. We present the validation results as functions of time, in order to inspect the performance variation during different dynamic phases during a trial. In those figures, we show the predicted probabilities and the classification accuracy on the validation set, where each time point represents an EMG time window. The performances of each time point in a trial were averaged over all 3-fold validation trials of all subjects, given their aligned timeline. We define the beginning of the grasping phase as 0 ms, and the averaged breakpoints between different motion phases are illustrated by vertical dashed lines.The performances of classifiers trained by the three different strategies introduced in Section 2.2.6 are shown in 9, the grasp gesture is identified as the ground-truth gesture l∈{1, ..., 13} executed during the non-resting phases, the rest gesture represents the open-palm/rest gesture l = 0 during the resting phase, and the top competitor is defined as the gesture with the highest predicted probability among all other 12 gestures which were not performed during the entire trial. For example, a large diameter was executed in order to grasp a bottle during a trial, so in this case, the grasp gesture is the large diameter with l = 1 and the rest gesture denotes open-palm/rest gesture l = 0. Since the classification was applied to each EMG time window independently, every time point t in 9 corresponds to a 14-dimensional predicted probability output , where the rest gesture probability is pl = 0, the grasp gesture probability is pl = 1 (given the large diameter as grasp gesture), and the top competitor probability is tm and tn, the top competitor gesture top competitor does not represent a particular grasp type and varies over time windows. The probability distance between the grasp gesture and the top competitor indicates the classifier's capability of distinguishing ground-truth gestures from other distracting grasps. The predicted probabilities and classification accuracies in 9 were averaged over all subjects and validation trials, i.e., the classification performance was evaluated and averaged on all possible grasp gesturesl∈{1, ..., 13}, not referring to the performance on a specific gesture.In grasp gesture and rest gesture intersect with each other is defined as ti (the intersection is marked by a solid red dot), and the distance from the probability peak of grasp gesture to the simultaneous probability of top competitor is defined as dp. In grasp gesture and rest gesture are independently given.In In this section, the performances of the three training strategies are further compared and analyzed, and the information revealed by dynamic EMG of different motion phases is also discussed.grasp gesture probability was achieved during the reaching phase at around 0.51. During the grasping phase and the first half of the returning phase, the grasp gesture probability was stable at around 0.3, and then gradually decreased as the movement converted to the resting phase. Notably, the estimated probability of grasp gesture was constantly higher than the top competitor throughout the entire trial. The time point when the probability curves of grasp gesture and rest gesture intersect was ti = −729 ms and the distance from the probability peak of grasp gesture to the top competitor probability was dp = 0.36. Simultaneously, the predicted probability of the rest gesture reduced dramatically to below 0.2 as the grasp movement happened until the hand returned to the rest position when the open-palm probability gradually went up again.The performance of the classifier trained by the first strategy (trained with reaching and resting phases) is shown in grasp gesture during the reaching phase presented an outstanding performance with a peak over 0.8 and remained higher than 0.6 within most of the reaching phase. This accuracy decreased in grasping and returning phases, fluctuating around 0.5. The performance of detecting rest gesture was highly accurate, with the accuracy value over 0.8 throughout most of the two resting phases.In ti between the grasp gesture and rest gesture probabilities. Since after this point, the grasp gesture starts to outperform the rest gesture, the system could start to prepare for pre-shaping the robot from this point on according to the type of detected grasp gesture. Ideally, this intersection is expected to appear right at the junction where the resting phase ends and the reaching phase starts in order to indicate the beginning of the hand motion. However, in practice, the hand movement could only be predicted based on the past motion, so the intersection is expected to appear after the start of the reaching phase but the closer to it the better. In ti = −729 ms earlier than the start of the grasping phase, which was after but very close to the beginning of the reaching phase and allowed enough time to pre-shape the robotic hand before the actual grasp. Therefore, the dynamic EMG of the reaching phase is proved to be informative for forecasting the gesture in advance during the reach-to-grasp movement. Furthermore, even though the model was trained by reaching and resting phases only, the probability of grasp gesture was still steadily higher than top competitor during all the four dynamic phases, representing the significant hand-shaping information revealed by the movement during the reaching phase.We are especially interested in the intersection point grasp gesture probability first gradually increased, until the grasping phase where the probability peak of grasp gesture was reached at around 0.56. When the hand movement entered returning and resting phases, the grasp gesture probability declined to lower than 0.2. The probability curve of rest gesture shows an opposite trend—the estimated rest gesture probability started from a peak at the first resting phase, then decreased to below 0.2 as the grasp movement happened, and then finally reached another peak when the hand went back to rest again. The intersection point between grasp gesture and rest gesture probabilities appeared at ti = −334 ms, and the probability peak of grasp gesture was dp = 0.41 higher than the top competitor probability at the same time point.The performance of the classifier trained by the second strategy (trained with grasping and resting phases) is shown in grasp gesture during the grasping phase performed outstandingly better than other phases, showing a similar pattern with its probability curve in grasp gesture accuracy was also higher than 0.6 during the end of the reaching phase and the entire grasping phase. The rest gesture accuracy was similar to the model trained by the first strategy as indicated in In ti = −729 ms, the probability intersection of this second-strategy model appeared 395 ms later in time, at ti = −334 ms. This illustrates that the EMG data of the grasping phase contained less information for detecting gestures in the earlier stage compared to the reaching-phase data, and thus the model trained by the second strategy provided less response time for the system. However, the grasp gesture probability peak of this second model outperformed the top competitor by dp = 0.41, which was 0.05 higher than dp = 0.36 of the first model. This demonstrates that the dynamic EMG from the grasping phase is more accurate and confident for decision-making during the grasp movement, outperforming other interference options to a larger degree.Compared to the classifier trained by reaching phase  is shown in grasp gesture was stably higher than 0.8 within most of the reaching and grasping phases, which are the most critical phases for making a robotic-grasp decision. It is worth noting that, the validation accuracy was still higher than 0.75 at the beginning of the returning phase even though the model was not trained on any data from that phase. The average-accuracy curve of the rest gesture presented a similar trend as shown in For the dynamic gesture classification, as shown in ti = −743 ms moved toward the beginning of the reaching phase even further compared to the first training strategy, and simultaneously the model was able to distinguish between grasp gesture and top competitor with a higher degree of confidence (dp = 0.49) throughout the entire trial compared to the second training strategy. Therefore, the model could provide even more response time before the grasp happens with more precise performance. Even during the returning phase which was unseen in the training, the model could still perform decently, illustrating the gain of using dynamic EMG for training. This higher degree of freedom in EMG data could enable more information and stability of the model to a wider range of postures during the dynamic grasp activity. In addition, the accuracy for detecting the resting phase was also highly accurate and sensitive to perform as a detector of muscle activation for triggering the robotic grasp.Compared to the first and second training strategies, the third strategy combined the advantages from both the reaching and grasping phases, leading to a boosted performance in detecting dynamic EMG compared to every single phase of reaching and grasping. The intersection point The summary of model performances under different classification scenarios is shown in t = −743 ms to t = 0 ms, in order to inspect the model's capability of detecting upcoming gestures during the reaching phase before the actual grasp happens.In order to investigate the model performance regarding different gestures, in As shown in But there are gestures that are very similar to each other and, thus, less distinguishable as illustrated in In our future experiment, in order to better distinguish the subtle differences between similar gestures, the time period of the pre-shaping stage could be divided into finer-grained sub-phases for capturing precise changes in finger configurations.via comparative experiments. The proposed method was assessed in a real-time manner and the corresponding performance variation over time was presented. Results illustrated the effectiveness of the framework built with the EMG data of a high degree of freedom.In this article, we proposed a non-static EMG recognition method for identifying real-time hand/arm movements regarding the dynamic muscular activity variation in practice. The presented framework was trained and validated by EMG signals collected from continuous grasping tasks with variations on dynamic hand postures so that the transitions from one intent to another could be encoded into the model based on natural sequences of human grasp movements. The obtained EMG data was segmented unsupervised into different dynamic motion sequences and further labeled based on the specific motion. A classifier was then constructed for recognizing the gesture label based on the dynamic EMG signal, and the impact of different movement phases was examined The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. The patients/participants provided their written informed consent to participate in this study.MH and MZ conducted the research and wrote the manuscript. MH, MZ, and SG organized the data collection. MH realized the methods and completed the experiments. GS and DE supervised this study and conceived the main idea of this study. All authors contributed to the article and approved the submitted version.This study is partially supported by NSF .The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


With the aggravation and evolution of global warming, natural disasters such as hurricanes occur more frequently, posing a great challenge to large-scale power systems. Therefore, the pre-position and reconfiguration of the microgrid defense resources by means of Mobile Energy Storage Vehicles (MEVs) and tie lines in damaged scenarios have attracted more and more attention. This paper proposes a novel two-stage optimization model with the consideration of MEVs and tie lines to minimize the shed loads and the outage duration of loads according to their proportional priorities. In the first stage, tie lines addition and MEVs pre-position are decided prior to a natural disaster; in the second stage, the switches of tie lines and original lines are operated and MEVs are allocated from staging locations to allocation nodes according to the specific damaged scenarios after the natural disaster strikes. The proposed load restoration method exploits the benefits of MEVs and ties lines by microgrid formation to pick up more critical loads. The progressive hedging algorithm is employed to solve the proposed scenario-based two-stage stochastic optimization problem. Finally, the effectiveness and superiority of the proposed model and applied algorithm are validated on an IEEE 33-bus test case. In recent years, hurricanes, earthquakes, and other natural disasters have caused large-scale power interruptions for a long time, which has brought huge economic losses to society. Forexample, in 2008, China’s extreme snow and ice disaster damaged power grid components in 13 provinces, witha total of 36,740 line faults, 2018 substations affected, andpower grids in some areas collapsed and even entered isolated network operation[Most literature focuses on the planning of defense resources before power system damage and the allocation of reconfiguration resources and network topology reconfiguration after power system damage. Leietal. proposedA series of models have been proposed to formulate this issue. In, GuoetalAccordingly, some algorithms were designed to solve these problems. Dingetal. used a hWith regard to the use of reconfiguration resources, Taherietal. not onlyBased on the above discussion, ascenario-based stochastic two-stage mixed-integer linear program model based on hybrid microgrid defense resources is proposed in this paper. This model includes the pre-position of the MEVs and the location selection of tie line addition before damage, andthe dynamic dispatching of MEVs and operation of added tie lines after damage. Amore detailed problem statement will be presented in (1)The multi-stage application of tie lines including addition and operation is firstly proposed and studied. This novel method effectively enhances power system resilience and the diversity of reconfiguration resources;(2)A two-stage framework for the combination use of microgrid defense resources by dispatching MEVs and employing tie lines is firstly proposed, which can restore the critical loads immediately with the limited use of MEVs in severely damaged scenarios;(3)A scenario-based two-stage mixed-integer linear program model is formulated and explained in detail, which is generalizable and easy to understand.(4)Several experiments have been conducted to compare the performance of the proposed method with other methods, demonstrating that the combined optimization model with MEVs and tie lines is competitive.The major contributions of this study can be summarized as:The remainder of this paper is organized as follows. In this section, we first depict how natural disasters damage the power grid. Based on this, theaction mechanisms of MEVs and tie lines on power grid restoration are described, anda two-stage microgrid reconfiguration framework of combination use of MEVs and tie lines is proposed andillustrated.Natural disasters and man-made attacks can usually destroy parts of lines in the power system, causing some nodes to lose connection with the main network and become isolated island loads. Asshown in As shown in In the new mode of microgrid reconfiguration with the combination use of MEVs and tie lines, after the original line faults occur, several local power supply areas can be formed by the deployment of MEVs and tie lines more flexibly, so as to realize the rapid restoration of unsupplied loads. As shown in In this section, the model formulation for the proposed problem is proposed.i under scenario n. Overall, the optimization objective of this problem is to restore more loads in the shortest possible time. The objective  is to miThe defined objective is subject to the following constraints which are divided into several groups as follows:Constraints –4) refe refe4) rm in a staging location node, the MEV m can be allocated to one of the allocation nodes in each scenario in the second stage. Constraints (k from the staging location node s.Constraints  and 6)  6) referk, then the node k is a power node supplied by MEVs. It is noted that the candidate locations include the staging locations and allocation locations where the MEVs can connect to the nodes on them. Constraints  rep rep11) rstraints  mean thastraints  and 10)k, then tAlgorithm 1: Algorithm For Finding Sub-graphs.i are equal to realize power balance. Constraint  rep rep16) rnstraint  determinstraints –20) equ equi arestraints  guarantestraints  and (16)i, j). Constraints (Constraints –24) rep rep24) rn occurs.In the first stage, we have determined the optimal pre-positioning decisions of MEVs Algorithm 2: Progressive Hedging Algorithm.The proposed two-stage microgrid optimization problem is where some of the decision variables are binary variables and the data is uncertain is a multi-stage mixed-integer stochastic programming problem, which is difficult to solve in a short time. To represent the uncertainty in data, we use the common approach by formulating a finite number of scenarios with corresponding probabilities for the values of uncertain parameters in this paper. Since the Progressive Hedging Algorithm (PHA) has proved an effective method for solving multi-stage stochastic programming problems ,28, espen. As for the consensus criteria in step2, it is based on the average per-scenario deviation from the average denoted as l denotes the lth element of the corresponding vector; N denotes the number of scenarios.We consider the stochastic program of the following form:6min∑Ni=1 such as  and 10)(26)min∑NIn this section, case studies are performed in the IEEE 33-node distribution system to validate the effectiveness and superiority of the two-stage 0-1 stochastic programming problem. As in , the topDue to the lack of actual damaged data, this paper uses the normal distribution to randomly generate 10 random damaged scenarios and load priorities . Considering the influence of vehicle transport speed in different damaged scenarios, this paper generates vehicle transport speed in the speed set {60 Km/h, 90 Km/h, 100 Km/h} with equal probability randomly. The maximum power of MEV is set as 1000 kW. When setting the parameters of the routing network, we first generate the adjacency matrix between nodes according to the damaged scenario and then calculate the transportation time of each vehicle on the corresponding routing according to the given speed parameters. In order to compare the effects of different reconfiguration strategies on the optimization objectives, multiple comparative experiments are carried out.As shown in After the extreme situation, the faulted lines in the real scenario are detected which are marked with a cross in In order to further reflect the superiority of our proposed model, we compare it with other methods which only consider using MEVs or tie lines. The models of these two methods are easy to build according to the proposed two-stage optimization framework.The objectives of the two stages are the same as objectives  and 25)25). The The optimization results of this model are shown in Since the model of only using tie lines does not include MEVs, the time of dispatching MEVs of objectives  and 25)25) is suThe optimization results of this model are shown in Overall, both the methods are advantageous to load restoration. However, only using the single configuration resource with the same budgets cannot fully restore all the disconnected loads. The combined use of these two resources can play a greater role in power grid restoration.In this section, the budgets of MEVs and tie lines are changed to investigate their effects of them on the final reconfiguration results. We use load shedding to measure the effects, which is obtained from the average results of 20 repeated experiments and calculated by Equation . As seenTo rapidly recover a microgrid under damaged scenarios, this paper proposes a novel two-stage stochastic mixed-integer model based on the combined optimization of MEVs and tie lines, minimizing the load shedding and the outage duration of loads according to their priorities. Numerical studies on the IEEE 33-bus distribution system demonstrate the effectiveness of the proposed model on restoring critical loads in the microgrid reconfiguration. Compared with the methods which only dispatch MEVs or use tie lines, the combined optimization model can restore more loads at less cost. Moreover, the microgrid recovery effect of the proposed approach improves with the increase of the budgets of reconfiguration resources. In future work, we will consider other reconfiguration resources, such as repair crews and more complex damaged scenarios after multiple extreme events, and design more computationally efficient algorithms.


In this article, we present an overview of recent progress in non-contact remote optical detection of ultrasound in application to nondestructive testing and evaluation of materials. The focus of the review is on the latest advances in optical detection that offer mature and robust field-applicable solutions for diagnostics and imaging of engineered structures. We provide a detailed description of these solutions, including their operation principles, their evolution from the previously known designs to commercial devices, and their contribution to solving the most important problems associated with non-contact optical detection of ultrasound. Several application examples are presented to demonstrate the capabilities of optical detection and provide ideas to a reader on how it can be used in practice. We also discuss the main challenges of modern non-contact detectors which have not yet been addressed, as well as the directions and prospects for their development. Although various new techniques have been introduced over the past few decades, ultrasound testing (UT) remains one of the main methods in NDT. However, some disadvantages of UT methods in some cases make their application inefficient and, therefore, must be overcome. Indeed, in order to launch ultrasound (US) signals into the sample under study an acoustic coupling  is necessary. This is not a problem when small samples are investigated, especially when the sample can be immersed into a liquid. A much more complicated situation arises when large industrial components (turbine blades or airplane wings) are inspected. Liquids such as oil or water, commonly used in these applications, not only make the inspection difficult but are also cumbersome or/and can contaminate the material under examination.The development of coupling-free (or air-coupled) inspection methods in UT has been a target problem of multiple studies for years Regardless of the boundary conditions at the air/material interface, the laser excitation parameters must be restrained to qualify the technique as nondestructive. The laser pulse energy and the delivered power per unit surface area must be adjusted to ensure the thermoelastic regime of operation and avoid ablation or degradation of the surface. This requirement has motivated the research and development of new detectors with sufficient sensitivity to receive US signals of relatively low amplitude.Note that when contact with the test material is allowed, ultra-broadband laser-generated US signals alone have a significant advantage over conventional piezoelectrically-generated signals when they are detected by well-designed broadband receivers. PVDF (polyvinylidene difluoride) transducers Although LUT methods with contact receivers have shown a great advantage over conventional UT, there are many applications where contact between the sample and the measurement probe is not desirable or even impossible. These applications include, but are not limited to, high-temperature applications, remote sensing of radioactive chambers, large constructions (such as turbine blades), airspace and car industry NDT, metallurgy, inspection of fragile substances, etc.Non-contact detection of ultrasound has a long history, beginning with laboratory devices and now including commercial systems for field applications. Advances have been made in both material evaluation and biomedical applications, as detailed in a series of reviews Recent advances in the field of diode and fiber optical sources, optoelectronics and telecommunication optics have made it possible to propose new solutions and make sufficient progress in non-contact optical detection of US signals. Some solutions have only been implemented in the laboratory environment and it is too early to conclude how they may advance the remote detection. However, others have already reached a level of high technological readiness, allowing them to be used in the field.This work does not describe the basics of laser ultrasonics and numerous laboratory designs of optical detectors, which can be found in previous reviews 2One of the key characteristics of a receiver is the sensitivity to detect mechanical vibrations, or the minimum signal  magnitude that can be distinguished from the background noise floor. Almost every publication introducing a new detection approach first addresses its sensitivity compared to existing methods. Similarly, the main question for non-contact optical detection of ultrasound, which must be discussed among others, is whether optical detection can achieve the sensitivity of conventional piezoelectric transducers.Most non-contact optical approaches use a portion of the light reflected from the sample surface. The larger the effective numerical aperture (NA) of the detection optics is used, the higher the reception efficiency can be achieved. The choice of NA is determined by the specific problem in which the detector is used. Since the wavelength of the probe light is on the order of 1 In the piezoelectric approach, US transducers usually focus the probe US beam onto the sample to reach the desired lateral resolution and the depth of field. Acoustic beam characteristics can be calculated based on the transducer’s central frequency, its bandwidth and NA, and material properties.The ultimate limit for any acoustic detector, independent of its nature, is set by the Johnson–Nyquist noise power associated with molecular thermal vibrations, as given by the well-known formula In case of the optical approach, Note that it is usually very difficult to reach the noise limit in the US signal detection. Because the ultra-broadband detection is required for laser-generated US signals, US detectors are usually far from the ‘ideal’ performance. Optical detectors are broadband, and therefore their performance closer approaches the noise limit for the given reception aperture. For example, apples-to-apples comparison between a broadband PVDF transducer and the Sagnac optical detector operated in a similar frequency range was performed in Ref. , and a ∼33.13.1.1Most non-contact optical detectors of ultrasound rely on interferometry. Its basic principle is well known For example, the basic architecture of Mach-Zehnder Interferometer using a Quadrature Demodulation scheme is presented in The setup shown in To collect multiple speckles, the detection aperture size can be enlarged, and an array of photodetectors used instead of a single detector, as presented in An extension of the multi-detector principle was proposed by Pouet et al. To further improve the system’s flexibility, the MCRQ interferometer was adapted to use the multimode optical fiber delivery d. In thaThe MCRQ interferometer uses two possible demodulation schemes: a rectified demodulation, and a linear demodulation, as shown in The second demodulation scheme, i.e., the linear demodulation, involves a logic control, which detects the signal phase. Depending on whether the signal is in-phase or out-of-phase, the logic switches between the two summing amplifiers to add the appropriate signals constructively. In this demodulation scheme, switching is based on the low-frequency Doppler shift introduced by the piezo-ring attached to the fiber. The low-frequency signal is separated from the high-frequency signal and used to switch the summation logic circuit based on the low-frequency sign. Thus, the displacement direction of the signal can be correctly recovered.The approach described above was commercialized by Sound & Bright .3.1.2One of the first applications of the multi-channel random quadrature interferometer was presented by Pouet et al. A more recent application of the MCRQ interferometer was presented by Lukacs et al. 3.23.2.1Most interferometers are designed to measure the out-of-plane component of vibrations, as both the incident and reflected beams are aligned close to the surface normal to maximize the signal-to-noise ratio. However, the in-plane vibration components are useful in some applications as they provide an additional information on the behavior of specific wave modes. To measure the in-plane vibration, laser beams reflected at several angles from the measurement surface should be recorded at once. Usually, optical systems using multiple laser sources aligned at different incident angles are used Optical interferometers can detect the phase of reflected light by combining the sample beam with the reference beam. The combined beams are projected on the photodetector and the interference pattern indicates the phase shift between the two beams. Wave mixing can be performed using a polarization beam splitter, as demonstrated in the previous section. For successful detection, the sample and reference beams must coincide with the wavefronts on the photodetector, and the average phase between the beams must be constant. Photorefractive crystals (PRC) An example of a two-wave mixing interferometer using a PRC is presented in A two-wave mixing PRC interferometer based on the design presented in For small angles, the These equations can be implemented using a signal processing circuit, presented in 3.2.2Application of the two-wave mixing PRC interferometer described above was presented by Blum et al. Chen et al. Chen et al. Stampfli et al. Garcia de la Yedra et al. 3.33.3.1Knife-edge detection (KED) is a non-interferometric technique for the measurement of vibrations based on the optical beam deflection. Surface vibrations result in the alteration of the incident optical beam direction when the beam size is smaller than the ripple spacing The limitation of KED for probing optically rough surfaces has been addressed by a speckle knife edge detector (SKED) If the inspected surface is rough, the sample beam is reflected diffusively, which results in multiple speckles on the CMOS array, as shown in the The solution to this problem offered by SKED is to tune the CMOS array to the speckle pattern by identifying individual speckles and using some of them as positive and some of them as negative signals, which is done using the following algorithm. Assuming that the speckle pattern shifts horizontally, for each pair of two neighboring pixels, the circuit checks which of the two receives a higher light intensity. If the right-hand side pixel has the higher intensity, the output from the right-hand detector is processed as positive (‘red’) signal. If the left-hand side pixel has the higher intensity, then the right-hand side pixel is processed as ‘green’ (negative) signal. As a result, the CMOS array is adjusted to the speckle pattern resulting from the measured beam, as shown in In the latest reported version, SKED consisted of a CMOS array with 32 × 32 elements According to Ref. A more recent version of SKED, named SKED2 Note that the presence of parasitic impedances and different amplitudes of the signals at different pixels reduce the coherence of individual pixels during their summation. Thus, some decrease in SNR compared to a single ideal detector with an equivalent aperture should be expected at a minimum surface roughness. However, the described SKED principle demonstrates a valuable improvement in detection of US vibrations from rough surfaces.3.3.2The effectiveness of SKED was demonstrated in detection of laser-generated Rayleigh waves, in particular for spatially resolved acoustic spectroscopy (SRAS) Patel et al. The detection of Rayleigh waves in steel samples using SKED was demonstrated by Lee et al. 3.43.4.1Heterodyne interferometers offer a slightly different approach compared to homodyne interferometers described in previous sections. Heterodyne interferometers are based on the measurement of surface vibrations using the principle of light beam modulation due to the Doppler effect An exemplary scheme of the heterodyne interferometer in Mach-Zehnder architecture is presented in Heterodyne interferometry is a mature technology, and there are many commercially available instruments provided by several manufacturers, including Polytec, Dantec, Ometron and Optomet. Unfortunately, LDVs are very sensitive to dark speckles, especially when measurements are performed on a rough surface. Due to surface vibration, the speckle pattern may evolve, resulting in variations of the measured signal. The fluctuation of speckles influence the Doppler signal through the amplitude or phase modulation •Combination of different polarizations , especially in case of diffusively reflecting surfaces, where the reflected light is randomly polarized.•Application of spatial separation, e.g., different parts of the probe beam or beams measured by different apertures can be processed independently.•Application of modal separation, meaning that different orthogonal beam modes can be processed separately, as the energy of the speckles is distributed into multiple modes.The issue of signal drop-outs was addressed in new Polytec® vibrometers by applying a principle of signal diversity o, separated using PBS1, is directed through the Bragg cell and then projected onto the measurement surface using an optical lens. The reflected sample beam A prototype of a heterodyne interferometer using polarization diversity was demonstrated by Dräbenstedt et al. In order to combine signals from different independent channels, each of them is independently demodulated and then multiplied by a weighting factor based on the power level of the frequency-shifted carrier signal, RSSI, which works as a signal quality indicator. Therefore, if a signal is poorly measured due to the dark speckle, its RSSI value is low, and less weight is assigned to it. The combination of signals from different measurement channels is performed according to the following weighting scheme ,,,The spatial diversity can be achieved by dividing the aperture into several sub-apertures 3.4.2A prototype of heterodyne interferometer using polarization diversity was used to measure vibrations of an aluminum disk rotating at 20 RPM 3.53.5.1Apart from the ‘classical’ two-beam interferometers discussed above of which the designs are constantly being improved, there is an additional Sagnac-type (or ring-type) interferometer, which has been in the shadow of others for decades. It has been recently reconsidered due to the development of fiber-optic technology that was essential for its proper functioning. This type of optical interferometers was not included in previous reviews and, therefore, we provide a more detailed description of its principle here.The Sagnac interferometer, named after French physicist Georges Sagnac, represents a circular path along which two interfering beams propagate in opposite directions . If the However, as it was noticed later, if a temporally short vibration is applied to the mirror located in the right bottom corner in In 2006, Tachizaki et al. The presented Sagnac interferometer was used in several applications, including measuring GHz-range surface waves in metamaterials As can be seen, the first applications of the Sagnac interferometer were limited to the reception of US waves of very high frequency. Indeed, the detection bandwidth For example, a 1 m difference in path lengths results in a Bowers et al. In 1997, Alcoz In 2014, Pelivanov, Buma and O’Donnell revisited the Sagnac technology to develop a fiber-optic interferometer independently of Alcoz. The proposed design, as shown in The linear polarization design also allowed the balanced detection to remove the non-birefringent component from the detected signal and reduce the common noise. A little footprint super-luminescent diode (SLD) was used as a stable low-coherent source, which provided more than 50 dB dynamic range of the detection and did it insensitive to in-fiber reflections due to the short coherence length. The only constraint on the coherence length in the proposed design is that it should be larger compared to variations in the topography of sample surface to provide coherent reflections within the area covered by the probe beam. Finally, a fully PM-fiber design with an in-fiber polarization controllers (OZ Optics) made of short piece of SM fiber was commercialized by LuxSonics Inc.The modern fiber-optic design makes the Sagnac interferometer competitive with other designs and even outperforms them in several characteristics. Indeed, most designs discussed above require a reference arm, which adds complexity into the system and requires complex stabilization techniques. In the Sagnac approach, both interfering beams come from the sample surface and no reference arm is required. Because the same surface is used to reflect both interfering beams that are reflected from the same target spatial point, the thermal lensing effect introduced by the pump beam is almost eliminated for frequencies above a few kHz. In addition, in interferometers that use a reference arm, it is necessary to “protect” it from environmental vibrations. The Sagnac interferometer is free of noise coming from the reference arm. However, in two-beam interferometers, the most serious negative effect is driven by the principle difference between the ideal mirror-like reflector of the reference arm and the speckle-based reflection introduced by most surfaces of untreated materials. This leads to the dramatic decrease of sensitivity of conventional designs; and most innovations in two-beam interferometer designs (discussed above) were aimed to compensate for the speckle effect. In contrast, the phase retardation effect has no influence on the Sagnac interferometer made of PM fibers. Indeed, optical fields of both Sagnac beams are the same without surface vibration. PM fibers propagate both fields with minor distortions and make them interfere in the second polarization controller. Only the difference between the fields of the two beams is important, and not the phase compound of the individual fields. Note that the amount of detected light is still affected by surface roughness. This problem can potentially be solved by increasing the probe light source power up to 300 mW (damage limit for PM components), but unfortunately, stable low coherent 1550 nm sources with the power of more than 40 mW are currently not available on the market.The Sagnac loop automatically equalizes the paths of the interfering beams, which will be stable under any environmental conditions except for the rotation of the interferometer. This maximizes the interference to the theoretical limit because interfering beams travel exactly the same distance and meet at exactly the same time. This allows using a very compact, low coherent SLD source to greatly reduce the effects of parasitic interferences within the interferometer itself and minimizes speckles when examining regular rough surfaces.A unique feature of the Sagnac design is the ability to quickly tune the detection bandwidth over an extremely broad range (from kHz to GHz) by changing the fiber length within the same device and without additional adjustment. This allows to apply the interferometer for both low and high-frequency measurements. Using PM fibers makes it possible to focus the probe beam spot to the diffraction limit, which enable its potential application for spectroscopy of microscale objects, such as single cells.Traditionally, the noise figure of interferometers is compared to the shot noise of photodetectors, which is also an intrinsic characteristic of the detector. Such estimations are important and represent the ultimate sensitivity of the detector, but these estimations do not indicate how large the detection signal is compared to the ambient noise. Indeed, Johnson-Nyquist noise defines the thermal noise power, which is the minimum acoustic signal power that can be detected regardless the type of the detector used. The authors of Ref.  , estimaDespite the fact that the noise figure of the Sagnac detector is quite small, the overall NEP of the detection system is still modest compared to contact US probes because of the small detection area (probe light focusing spot). For a mirror-like sample surface, the probe beam diameter can be increased, and the overall optical power can be increased to significantly reduce the NEP. However, the main problem with LU inspection of industrial materials is that their surface is usually quite rough, which requires a focused probe beam and high NA optics to efficiently collect backscattered light. On the other hand, the larger the NA, the shorter the depth of field. The trade-off between a high NA and sufficient depth of field is a subject for future optimizations of confocal systems, such as the Sagnac detector.A disadvantage of the current Sagnac principle is the difficulty of measuring the absolute displacement without calibration, which is sometimes impossible due to highly variable light reflection properties of the target. Another disadvantage is its sensitivity to a possible depolarization of the probe light induced by its reflection from the target surface.3.5.2A laser-ultrasound scanner with a Sagnac interferometer on receive see  was firsrate see a,b.Fig. The reduced sensitivity of the Sagnac detector to surface roughness was demonstrated in Ref. , where detected c.The broadband nature of generated LU signals and their non-contact detection with a high spatial resolution made it possible to not only detect the flaws, but to visualize the material structure, which in turn enabled new imaging capabilities not available to reach with conventional UT. Consequently, the LUT scanner was then demonstrated in the evaluation of material porosity 3.6Direct optical vibration sensors rely heavily on the surface quality to perform successful measurements. Despite the significant improvements to the interferometric techniques, described above, there are surfaces that are very hard to measure using optical sensors – materials with high roughness, such as concrete, living tissue, or materials with highly light-absorbing surfaces. Conventional non-contact vibration measurements use air-coupled (AC) transducers that are applicable to various surfaces by measuring waves leaking into the air from inspected materials.Conventional AC transducers may use piezoelectric, capacitive or PVDF sensors. The main limitation for manufacturing AC receivers is a large impedance mismatch between air and most of the solids that can be used as piezoelectric materials or membranes. In addition, the AC transducers are usually highly directional, have a very narrow bandwidth and use relatively large aperture to achieve a reasonable sensitivity.To address the limitations of conventional AC transducers, an optical membrane-free sensor was proposed by Xarion Laser Acoustics GmbH 3.6.1The all-optical akinetic sensor is based on a Fabry-Perot (FP) interferometer architecture, as demonstrated in In the given sensor, the distance between the mirrors and wavelength are fixed. Therefore, the transfer function depends on the refractive index, which can be influenced by the acoustic waves propagating between the mirrors.The optical akinetic sensor The advantage of the optical akinetic sensor compared to traditional air-coupled transducers is the elimination of one air-solid high-impedance interface when measuring leaky waves. The sensor exhibits maximum sensitivity in a plane perpendicular to the axis of the laser source with a nearly uniform directional response profile. Amplitude drops for certain directions may be caused by the sensor casing that blocks the acoustic path 3.6.2The optical akinetic sensor can be used in the same way as traditional air-coupled transducers. Rus et al. The combination of laser-ultrasound with the optical microphone on receive was presented by Rus et al. Haindl et al. 4This paper reviewed recent developments in the field of optical detection of ultrasound vibrations, with the focus on addressing some of the key issues associated with the application of LUT techniques in the field. Key areas of improvement included new schemes for measuring both in-plane and out-of-plane components of vibration, reducing the sensitivity of optical detection to surface roughness, and increasing its stability to environmental noise. Various ideas have been proposed based on the use of an array of optical detectors and improved signal processing schemes for efficient processing of speckles in the reflected probe beam. Recently, an alternative approach to the problem of uneven, rough and poorly reflecting surfaces has been proposed, in which the optical akinetic sensor is positioned in the air near the measuring surface and receives acoustic waves leaking from the sample in the air. Recent advances in the Sagnac architecture have resulted in a flexible system with the improved both sensitivity and stability, and capable of quick optimization of the detection range from a few MHz to GHz. The use of the signal diversity principle through creating multiple independent beams significantly reduced the probability of signal drop-outs in the optical heterodyne detection. These systems have found many practical applications, and most of them have been applied commercially, giving a broad range of options for optical sensing of US vibrations.One of the most significant problems to be solved in optical detection systems is the long inspection time, especially in the case of multi-dimensional imaging. One possible solution to this problem is to use a parallel detection scheme. Several examples of such systems have been presented, including multi-beam systems A possible solution to the problem of the long inspection time is to use a faster screening method for the initial examination followed by a high-resolution localized imaging using LUT. In the case of thin-walled structures, full-field Lamb wave imaging has been proven to be effective for damage detection using mapping algorithms such as local wavenumber estimation The need to reduce scanning time in laser ultrasound has led to alternative approaches based on line- of full-field detection. While existing systems using parallel detection are still at an early stage of development and are mainly suited for low-frequency applications with low sensitivity, technological advances in high-speed cameras and electronics might change the situation in future. Therefore, we reference to these methods even though they have not yet demonstrated clear advantages and reliability in NDT field applications. For example, systems based on holography have already been demonstrated for measuring low frequency vibrations from the entire sample surface An interesting approach was recently proposed in Optical Coherence Tomography (OCT). A conventional approach used for imaging objects with OCT for years was in fast (currently up to a few MHz) scanning of a focused beam over the sample area 12SiO20), where it was mixed with the reference beam. Their overlap established a refractive index grating which caused part of each of the beams to diffract towards the other beam. Thus, interference occurred between the weak diffracted beam and the strong transmitted beam, resulting in a heterodyning effect. This effect converted changes in optical phase difference between the overlapping beams induced by the surface motion of the sample into intensity differences of the overlapping beams. The intensity modulations were recorded by a CCD camera. However, unlike the OCE approach which allows propagating waves using a single excitation Non-scanning detection of surface waves in the MHz frequency range was demonstrated for a metal plate in Ref To address the problem of the sample surface quality in full field imaging, a simple non-scanning speckle-based approach was proposed Similar principle is exploited in laser shearography Another key point that would radically change the transfer of remote LUS methods to field use would be the ability to inspect parts and objects of complex shape. This problem does not have a universal solution with existing NDT methods. Indeed, though small parts can be imaged in X-ray tomographs Indeed, the SNR in optical detection is directly proportional to the amount of light collected from the sample surface. As discussed above, industrial grade surfaces are typically rough, which require a high numerical aperture optics to efficiently collect the reflected light. On the other hand, there is a trade-off between the numerical aperture of the imaging lens and the depth of field of detection. This makes confocal optical detectors sensitive to both angular and axial misalignment.An approach based on the diffuse optical reflection created by surface roughness has been demonstrated by Tecnatom using a confocal Fabry-Perot interferometer In contrast to the approach described above, an interesting distance and angle correction system (DACS) was proposed by Canfield et al. The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Ivan Pelivanov is a co-founder and shareholder of LuxSonics Inc. Other authors declare no conflict of interests.


This research aimed to improve the classification performance of a developed near-infrared (NIR) spectrometer when applied to the geographical origin identification of coffee bean samples. The modification was based on the utilization of a collection of spectral databases from several different agricultural samples, including corn, red beans, mung beans, black beans, soybeans, green and roasted coffee, adzuki beans, and paddy and white rice. These databases were established using a reference NIR instrument and the piecewise direct standardization (PDS) calibration transfer method. To evaluate the suitability of the transfer samples, the Davies–Bouldin index (DBI) was calculated. The outcomes that resulted in low DBI values were likely to produce better classification rates. The classification of coffee origins was based on the use of a supervised self-organizing map (SSOM). Without the spectral modification, SSOM classification using the developed NIR instrument resulted in predictive ability (% PA), model stability (% MS), and correctly classified instances (% CC) values of 61%, 58%, and 64%, respectively. After the transformation process was completed with the corn, red bean, mung bean, white rice, and green coffee NIR spectral data, the predictive performance of the SSOM models was found to have improved (67–79% CC). The best classification performance was observed with the use of corn, producing improved % PA, % MS, and % CC values at 71%, 67%, and 79%, respectively. Coffee is among the most popular beverages consumed by people from all over the world. Coffee can be grown in many areas, especially in the equatorial region. However, it has been reported that the conditions of the growing environment can affect the chemical composition of coffee beans and consequently influence the characteristic quality of the coffee products . Up to tHowever, the cost of a commercial benchtop NIR spectrometer can be relatively expensive when compared to other analytical instruments that employ similar detection methodology, such as ultraviolet (UV) and visible (VIS) spectrometers. This could be among the main drawbacks that make the practical use of NIR detection rather limited, especially for agricultural applications wherein the products are not expensive. Recently, various low-cost NIR spectrometers have been developed ,7,8. ForTheoretically, more useful variations can be extracted from a commercial benchtop NIR instrument, resulting in a better degree of predictive performance when compared to those obtained from homemade or low-cost NIR instruments. Therefore, the predictive ability of the counterpart NIR systems could be improved by incorporating the essential information found in the reference system with greater spectral detection ability. Calibration transfer involves a group of chemometric methods that mathematically create a connection between the data obtained from different detection instruments. These are often referred to as master and slave instruments . PiecewiIdeally, the NIR spectra of the identical samples—the so-called “transfer samples”—should be recorded from both the master and the slave instruments to establish an accurate calibration transfer model . In addiThis research aimed to classify green coffee bean samples based on their growing origins with the developed NIR detection, using artificial neural network classification. The predictive performance of the developed NIR instrument was improved based on spectral transformation using the PDS calibration transfer method. The effects of different types of transfer samples on the instrumental calibration process were investigated. The PDS transformation using green coffee beans and the classification based on the NIR reference instrument was performed for comparison and for demonstration of the improvement achieved by the developed method. This development allowed the homemade NIR system to provide robust and accurate prediction results based on utilization of the existing NIR spectral databases acquired from the reference instruments.The NIR spectra recorded from the transfer samples using the reference and the homemade systems are presented in F). In this case, after PDS transformation, an improvement could be clearly noticed from the PCA score plot. Although all samples were still organized into a single cluster, the regions of the coffee samples obtained from the different plantation areas became clearer when compared to the PCA results in The PDS transformation subsequently created local multivariate models. The created local models assumed that the spectral information at a certain wavelength of the slave instrument was contained in a small spectral region of the neighboring wavelengths of the master instrument. However, the low-cost NIR instrument only captured the short-wave spectral information between 900–1700 nm, which was considered the second overtone NIR region a. The abPCA is an unsupervised learning method and directly reflects the differences of spectra. There were differences among the spectra recorded by the reference spectrometer, which led to the sample cluster on the PCA score plots. For the spectra recorded by the homemade spectrometer, the differences did not seem significant. After spectra transfer process, the unsupervised learning method PCA was changed to a supervised learning method, improving the performance of the instrument. On the score plot, the green coffee beans obtained from YP (red circles) could be separated from those that were obtained from NK (blue circles).In considering the use of different agricultural products as transfer samples, in particular corn, red beans, and white rice as shown in The developed method aimed to classify the geographical origins of the coffee bean samples. It should be noted here that the transfer samples were chosen based on two important criteria. Firstly, the transfer samples should share some common properties with the test samples, such as moisture and graininess. Secondly, their main compositions should be similar. For example, carbohydrates, proteins, and fats were among the main components in the grain samples. It was not required that all the grain samples contain caffeine as in the coffee beans, because the NIR detection mainly captured the variation from the major components in the samples.After the PDS transformation used different transfer samples, the classification results were changed. Using MT green coffee as the transfer sample, the classification accuracy increased to 78% CC. This outcome suggested that transfer samples with the same characteristics as the test or unknown samples could enhance the classification performance of the homemade instrument, thereby demonstrating that the calibration transfer method could be used to improve the classification results. Interestingly, using corn, red beans, and white rice as the transfer samples, the classification performance in terms of the % CC was also clearly improved (75–79% CC), where the best classification value was obtained from SSOM using corn as the transfer sample (79% CC). However, the use of black beans and roasted coffee did not significantly improve the predictive performance when compared with the predictions that were based on the non-transformed spectra. In contrast, adzuki beans, paddy rice, and soybeans unsuccessfully offered lower % CC values (52–59%).To monitor improvement after PDS transformation, DBI values of the transformed spectral data using different transfer samples were calculated and are listed in 2 value of 0.816 indicated a strong relationship between the −log(DBI) values and the % CC predictive performance. This outcome implied that DBI could be used to evaluate the degree of improvement of the NIR spectral data after PDS transformation.The transfer samples resulted in increased % CC values of the MT green coffee beans, corn, red beans, mung beans, and white rice by providing DBI values ranging from 2.49 to 4.09, whereas adzuki beans, paddy rice, and soybeans exhibited an increase in DBI values ranking from 6.38 to 13.36. In this research, a methodology to improve the classification performance of a homemade NIR spectrometer aiming to identify the geographical origins of coffee beans was described. The development was based on the learning and transformation of essential information found in the reference system with greater spectral detection ability, called the transfer samples. In addition, a variety of agricultural samples was employed to serve as the transfer samples in the transformation process. This research has demonstrated that it was not required that transfer samples should be the same type as the test samples. Different types of transfer samples could be used for the transformation process and improve the classification ability.In ideal work conditions, it would be recommended that large numbers of different transfer samples be tested and compared by the calculation of the DBI values. A strong relationship between the −log(DBI) values and the % CC predictive performance could be observed. Therefore, it was possible to forecast the degree of improvement of the NIR spectral data after PDS transformation when the different transfer samples were used. The transformed NIR spectra with low DBI values tended to offer improved classification results. However, it should be noted that the developed method could not point out the reason why transformation based on the selected samples could result in optimal predictive results. The suitability of the transfer samples should be re-evaluated and compared if new transfer samples are introduced to the database.Green coffee beans were collected from four different subdistricts located in Chiang Mai, the largest province in the northern part of Thailand, in 2020, including Sop Khong (SK), Na Kian (NK), Yang Piang (YP), and Mae Tuen (MT). Although these plantation areas are in the same province, their environmental conditions are rather different due to their elevation above sea level . Characteristic details of the coffee samples have been presented in In this research, two NIR spectrometers were used as the reference and homemade spectrometers. For the reference NIR system, a Foss NIRSystem 6500  benchtop instrument was used. The detection of the reference instrument was in the region of 400–2500 nm at 2 nm sampling intervals using an NIR reflectance transportation module in a 25 °C temperature-controlled room. This detection resulted in a total of 1050 absorption parameters. For 100 g of each sample, the spectral data from 64 detection points were averaged to provide a mean spectrum.For the low-cost NIR system, a homemade NIR spectrometer that had been described in previously published literature  was usedThe performance of these two spectrometers has been previously compared and reported. Generally, the detection performance of the homemade NIR spectrometer could not compete with that of the commercial NIR spectrometer ,21. HoweCalibration transfer (CT) methods were used to investigate the relationship between the spectral data obtained from the reference (master) and the other (slave) instruments. This relationship information was then used to adjust for the differences between the variations recorded from both instruments. Piecewise direct standardization (PDS) involves the improvement of a conventional calibration transfer algorithm called direct standardization (DS), where the data are segmented into small windows so that the spectral adjustment of the slave instrument can be based on the local characteristics rather than on the entire spectrum of the reference .XMS and XSL, respectively. After the PDS process, the relationship information was expressed in terms of a transformation matrix (F). In this research, partial least squares (PLS) regression was used to evaluate this relationship information [In this research, the NIR spectra established from the reference and the low-cost spectrometers were defined as ormation . A totalormation  and the ormation .To investigate the transfer ability using different types of samples, different agricultural products, including corn, red beans, mung beans, black beans, soybeans, green and roasted coffee, adzuki beans, and paddy and white rice were adopted as the transfer samples. The developed method aimed to classify the geographical origins of the green coffee bean samples. Therefore, the transfer samples used in the current study were restricted to grain products. The number of transfer samples used could affect the modelling ability of the standardized spectra ; howeverThe Kohonen map, or self-organizing map (SOM), is among the most well-known artificial neural networks that can be used to project data from a high-dimensional space onto a low-dimensional array of neurons (or map units) . SuperviTo evaluate whether the clusters of the samples could be well-separated after the transformation process, the Davies–Bouldin index (DBI) was employed ,30. The The characteristics of the classification models could be dependent upon the training samples used to establish the models. Different selections of the training samples, in many cases, could result in different optimal solutions for the same problem. In this research, the predictive performance of the developed models was evaluated using a bootstrap methodology . In thisThe developed NIR spectrometer could be applied to classifying green coffee beans based on their geographical origins using SSOM. Spectral transformation could improve the classification performance of the homemade NIR spectrometer by incorporating a fraction of the information in the NIR spectra acquired from the reference NIR instrument. A variety of the transfer samples could then be used for the transformation process. Moreover, it was not necessary that the transfer samples share the same characteristics as the test samples. The suitability of the transfer samples could be evaluated using the DBI cluster validation index. In this demonstration, the transformation of the green coffee spectra, based on the use of corn as the transfer sample, could improve the classification performance by up to 23% in terms of classification accuracy when compared to the classification performance of the non-transformation data. Therefore, the developed method could improve the accuracy and reliability of the developed NIR instrument without additional costs, based on the previously provided database of the agricultural transfer samples.


The demand for food delivery services (FDSs) during the COVID-19 crisis has been fuelled by consumers who prefer to order meals online and have it delivered to their door than to wait at a restaurant. Since many restaurants moved online and joined FDSs such as Uber Eats, Menulog, and Deliveroo, customer reviews on internet platforms have become a valuable source of information about a company’s performance. FDS organisations strive to collect customer complaints and effectively utilise the information to identify improvements needed to enhance customer satisfaction. However, only a few customer opinions are addressed because of the large amount of customer feedback data and lack of customer service consultants. Organisations can use artificial intelligence (AI) instead of relying on customer service experts and find solutions on their own to save money as opposed to reading each review. Based on the literature, deep learning (DL) methods have shown remarkable results in obtaining better accuracy when working with large datasets in other domains, but lack explainability in their model. Rapid research on explainable AI (XAI) to explain predictions made by opaque models looks promising but remains to be explored in the FDS domain. This study conducted a sentiment analysis by comparing simple and hybrid DL techniques  in the FDS domain and explained the predictions using SHapley Additive exPlanations (SHAP) and Local Interpretable Model-Agnostic Explanations (LIME). The DL models were trained and tested on the customer review dataset extracted from the ProductReview website. Results showed that the LSTM, Bi-LSTM and Bi-GRU-LSTM-CNN models achieved an accuracy of 96.07%, 95.85% and 96.33%, respectively. The model should exhibit fewer false negatives because FDS organisations aim to identify and address each and every customer complaint. The LSTM model was chosen over the other two DL models, Bi-LSTM and Bi-GRU-LSTM-CNN, due to its lower rate of false negatives. XAI techniques, such as SHAP and LIME, revealed the feature contribution of the words used towards positive and negative sentiments, which were used to validate the model. COVID-19 lockdowns and quarantines have increased the demand for online food delivery service (FDS) organisations, such as Uber Eats, Deliveroo and Menulog, because restaurants were instructed to stop providing dining services . AccordiCustomers primarily look for reviews and recommendations of others when they order cuisines from online FDSs. Positive reviews drive customers to make a decision on ordering food from one restaurant, whereas negative reviews help them look for other options . FDS comSentiment analysis is an automated process used to determine the emotions and sentiments of customers towards the food or service . MachineOver the past couple of decades, there has been a good amount of research published on the application of ML techniques to perform sentiment analysis in the FDS domain ,29,30,31In another significant work, Noor  comparedDuring the COVID-19 pandemic, many businesses received “1 star” reviews for being closed during lockdown. FDS Yelp received poor reviews for slow service or heat waves in the seating areas. The restaurants need to know about their customers’ complaints and expectations, which can be found through sentiment analysis . AlthougAccording to Nurdin , DL in NWhich DL classifier will be best suited to pick FDS customer complaints from feedback and work on its solution?Can XAI techniques, such as LIME or SHAP, provide explanations for sentiment prediction and help to build trust in the DL model created from the previous question?The research aims of this study were to address the gaps identified in the literature by answering the following research questions:The main contributions of this study that differentiate it from similar studies are as follows:Contribution I: This study compared the DL techniques and selected the best DL classifier suitable for the FDS domain to predict the negative sentiments from customer feedback that can be further used to improve customer satisfaction.Contribution II: The prepared DL model was tested with XAI techniques to validate the model’s logic for prediction and build trust for the organisations and industries who use it.The novelty and main contribution of this research work come from building a DL model and then explaining it using the XAI technique to validate the model’s logic for food industry application. On the basis of the recommendations and gaps found in a recently published review paper , we conshttps://www.productreview.com.au, accessed on 1 June 2022) is an Australian customer review website that stores the customer reviews for multiple brands across various products and services. Six FDS organisations, such as Uber Eats, Menulog, Youfoodz, Deliveroo, My Muscle Chef and Macros, were selected from the ProductReview website for their customer review data. A DL model was built on 13,621 customer reviews pulled from the selected FDS organisations. The accuracy of the DL models was compared by using a confusion matrix, and then black box DL models were interpreted by using XAI techniques, such as SHAP and LIME. The contribution information of every word (feature) in a customer review sentence using SHAP and LIME techniques was utilised to analyse the outcome of the DL model.Product Review  perform sentiment analysis using DL models and compare, and (b) explain the predictions by showing the most significant features that contribute to a customer sentiment. The design of the research development methodology that was used to conduct the research presented in this study is shown in The ProductReview dataset was split into test and training datasets. Three different DL models  were developed, trained and tested on the dataset. Several rounds of testing and fine-tuning of the hyper parameters were performed to finalise the DL model’s architecture and then fine-tune its performance. The DL models were compared in terms of their accuracy and other key parameters for the FDS domain. The best DL model was picked after comparison, and XAI techniques (SHAP and LIME) were used to interpret the DL model. The explanations were provided, thereby obtaining the word features that contributed to positive or negative sentiments. The sections that follow provide more information.Productreview.com.au is an Australian website that gathers consumer feedback on a variety of products and services. Overall, 13,621 customer reviews were collected from various FDS companies, such as Uber Eats, Menulog, Youfoodz, Deliveroo, My Muscle Chef and Macros, from the ProductReview website via web scraping. The below dataset example  shows thThe dataset was grouped into the following binary sentiment tasks: positive and negative classes. The positive class was labelled as ratings of 4 and above, and the negative class was labelled as a rating of 2 or below. The dataset was then divided into 8995 positive reviews and 4626 negative reviews . Rating The labelled customer review data were cleaned by reducing the noise and normalising each word to lowercase. Further punctuation, such as question marks, commas, colons, hash signs and website URLs, were removed to reduce the noise of the data. Some review data sequences were truncated or padded to provide a fixed length and mark all the sequence data into a standard length. For the training data, one of the requirements for LSTM models is to have a fixed length for the input sentence length of the review data. We set the customer review data length to 100.In Study 1, a sentiment analysis using DL techniques was performed, such as LSTM, Bi-LSTM and Bi-GRU-LSTM-CNN;In Study 2, the results of the above DL models were compared and the different XAI methods were applied, such as SHAP and LIME, to validate the best DL model.A sequence of data input works well with an RNN . In tradt. The input at time step t is Assuming we have a sentence of five words, then the above figure will have five layers, with one layer for each word. In LSTM is a gated RNN, and Bi-LSTM is an extension of the model. LSTM models can learn long dependencies from the previous states as compared to the traditional RNN model . The Bi-Sum;Multiplication;Averaging;Concatenation (default).The forget and output gate manages the information that needs to be kept or deleted . The modGRU, which was introduced in 2014, is similar to LSTM without an output gate. GRU has update and reset gates that help to combine new inputs with the previous ones . The updThe results of the DL models were verified by adjusting the hyper parameters after multiple rounds of training and testing. The DL models were trained and tested several times before finalising the hyperparameters, which included epochs, batch size, layers, dropouts, number of units, and the activation function. The LSTM model was built with one embedding layer for word embedding, one spatialdropout1d layer to train fewer features, an LSTM layer, flatten layer, two dense layers with the second one using SoftMax, and one dropout layer with 50% located between the dense layers. One embedding layer for word embedding, one spatialdropout1d layer for training fewer features, one Bi-LSTM layer, a flatten layer, two dense layers with the second one using SoftMax, and one dropout layer with 50% located between the dense layers were used to create the Bi-LSTM model. The Bi-GRU-LSTM-CNN model was developed with one embedding layer for word embedding, one spatialdropout1d layer to train fewer features, one bi-directional GRU layer with two LSTMs (one forward and one backward), a 1D convolutional layer, one global average polling 1D later and one global max pooling 1D layer, two dense layers with the last one using SoftMax, and one dropout layer with 50% located between the dense layers.The models achieved optimum results with 100 epochs and a batch size of 32 after testing various combinations of hyperparameters. The model was compiled with the Adam optimiser  and sparTo understand the accuracy of the models, we used the confusion matrix and F1 score of precision and recall metrics of the ML and DL models. The formulae for calculating precision Equation (1) and recall Equation (2) are as follows:1F score (Equation (3)) is used to seek a balance between the two metrics.As can be seen from the above equations, precision should be used when the cost of false positives for the business is greater, whereas recall should be used when the cost of false negatives is greater for a business. The 1F score to compare the accuracy of the DL models.We used the x by computing each feature’s contribution to the prediction  of the word tokens  t ∈ T with respect to a given class c of interest [T that push C’s prediction towards c and a negative score to tokens in T that push C’s prediction towards any other class c′ c. LIME provides a positive score to the tokens in T that push the prediction of C towards c and a negative score to those pushing to any other class c′ ≠ c.A set of scores are produced by LIME, denoted as interest . LIME prIn this section, three DL techniques for sentiment analysis of the FDSs were evaluated. The accuracy of models was compared using the confusion matrix and model metrics. XAI techniques, such as SHAP and LIME, were used to explain the outcome of the DL models. The classification performance and errors of the classifiers are represented in the confusion matrix, as shown in The confusion matrix shown in Similarly, the confusion matrix B shows tThe confusion matrix C shows tThe results from the above performance metrics show that all the DL models developed for performing a sentiment analysis attained a high overall accuracy . However, FDS organisations will pick the LSTM model as the best classifier due to its fewer type 1 errors with 21 false negatives as compared to BiLSTM with 33 and Bi-GRU-LSTM-CNN with 58. Although the accuracy of the LSTM model is high, it lacks model interpretability and explainability of the decisions made. The explanations of the LSTM-based black box model will help build trust in the system.This section describes the interpretation and explainability of the LSTM-based black box model by using different XAI techniques (SHAP and LIME). We applied SHAP on the model to interpret the feature importance considered by LSTM while making the predictions after training and testing the LSTM classifier. A DeepExplainer class from the SHAP library, which took approximately 20 min, was used to generate the SHAP values for the test dataset. As can be seen from The LIMETextExplainer class from the LIME library was used to predict the class with variations in the probability value on the same two customer reviews previously used by SHAP. LIMETextExplainer took only 2–3 min to train and generate local explanations for predictions.For customer review 1 in In the next example of customer review 2, the LSTM model was 100% certain that the customer review shown in  indicateIn this study, we addressed the subject of sentiment analysis in the FDS domain by utilising DL approaches and interpreted them with XAI techniques. The study was able to solve the black-box nature of the DL methods  by impleDL techniques  were used to perform a sentiment analysis of the customer reviews after performing data preprocessing. Compared with previous work in the literature represented in Although the overall accuracy of Bi-GRU-LSTM-CNN was 96.33%, its false negative percentage was 2.13, which was higher than that of the LSTM model (0.77). The overall accuracy of the LSTM model was 96.07%. In the case of sentiment analysis, the generation of fewer false negatives is preferred over false positives because businesses do not like to miss any negative customer reviews as compared to positive reviews. The LSTM model is recommended over the Bi-LSTM and Bi-GRU-LSTM-CNN models due to its lower false negative percentage. This model can be used to perform a sentiment analysis of any FDS organisation. The customer review data can be pulled from different social media channels, such as the ProductReview website, Twitter and Facebook. The negative customer reviews can be further analysed to understand which aspects of customer service can be improved. Positive customer reviews can be used by food delivery organisations for rewards and recognition.Although the LSTM model achieved high accuracy in determining the sentiment of customer reviews, it did not provide explainability of the model. Placing trust in the highly accurate DL black-box model without knowing its decision-making logic is difficult for FDS organisations. Businesses prefer to verify the parameters or features of a model that provides accurate results. Following the training and testing of the LSTM classifier, we used SHAP and LIME techniques to interpret the feature importance considered by LSTM when making predictions for customer reviews. A DeepExplainer class from the SHAP library took more time to generate the SHAP values for the test dataset as compared to LIME.SHAP and LIME provided similar prediction results with their explanation of the model . The useThe strengths of this study depend on the LSTM model, which was effectively used to identify the positive and negative sentiments from customer reviews with an accuracy of 98.07% with a lesser false negative rate of 0.77%. The accuracy attained by the DL model was higher than the models used in past work in the FDS domain ,29,30,31The limitation of this research is that, in the presence of sarcasm, the model can misinterpret a customer review as positive and vice versa, thereby resulting in an improper training set. The risk of spam accounts, false accounts and bots, which can generate irrelevant data and affect the training set and compromise accuracy results, is always high.Negative or positive sentiments can be predicted by using the LSTM model combined with XAI techniques with high accuracy and explainability. The recommendation for future work is to categorise the negative and positive sentiments into various topic groups so that they can be sent to the right channel to address supply chain issues. The negative sentiment topics can be used for customer satisfaction improvement, whereas the positive sentiment topics can be used to reward staff or restaurants.The purpose of this study was to predict the tone of customer evaluations in the FDS industry and to explain the decisions. Even once the COVID-19 pandemic abates, the rise in FDS use will continue. Given the volume of review data dispersed across numerous platforms and the absence of customer service professionals to examine and respond to each comment, AI can help FDS organisations to solve problems and save money. In the FDS domain, a false positive result indicates more operational efforts, whereas a false negative increases the risk of an organisation missing important customer complaints. The results showed that the LSTM model with lower false negatives outperforms the BiLSTM and Bi-GRU-LSTM-CNN models. SHAP and LIME were successfully applied to the LSTM model to determine the positive or negative contributions of each word to the predictions made by the model. Original customer reviews were analysed, and the logic behind the predictions made by the DL models, such as LSTM, was found to be explainable. Therefore, this research revealed that the behaviour of such models can be understood by implementing DL models for sentiment analysis along with XAI techniques. The LIME explainer uncovered the features that contribute to a particular prediction, and the SHAP explainer can further deepen an organisation’s understanding of models’ behaviour. SHAP required more training time with the dataset compared with LIME. This research concludes that the sentiment analysis of customer reviews in FDS can be best achieved with the LSTM model combined with LIME and SHAP techniques to achieve high accuracy and explainability. In future research, the focus should be on topic categorisation techniques that can be added to DL with an XAI solution to direct customer complaints to the right problem-solving group so that they can address supply chain issues. Additionally, the same methodology can be used to predict customer reviews in other domains to compare the results. Furthermore, it is worth exploring the models in a real-time working scenario where, as soon as a customer lodges their complaint, the complaint is directed to the relevant department for resolution.