The wealth of properties in functional materials at the nanoscale has attracted tremendous interest over the last decades, spurring the development of ever more precise and ingenious characterization techniques. In ferroelectrics, for instance, scanning probe microscopy based techniques have been used in conjunction with advanced optical methods to probe the structure and properties of nanoscale domain walls, revealing complex behaviours such as chirality, electronic conduction or localised modulation of mechanical response. However, due to the different nature of the characterization methods, only limited and indirect correlation has been achieved between them, even when the same spatial areas were probed. Here, we propose a fast and unbiased analysis method for heterogeneous spatial data sets, enabling quantitative correlative multi-technique studies of functional materials. The method, based on a combination of data stacking, distortion correction, and machine learning, enables a precise mesoscale analysis. When applied to a data set containing scanning probe microscopy piezoresponse and second harmonic generation polarimetry measurements, our workflow reveals behaviours that could not be seen by usual manual analysis, and the origin of which is only explainable by using the quantitative correlation between the two data sets. Among these materials, ferroelectrics are of particular interest. They possess a spontaneous polarization switchable under the application of an electric field, and a wealth of associated functional properties including strong nonlinear electro-optical effects, very high piezo- and pyroelectric responses, and in some cases magnetoelectric coupling4. Recently, the focus has shifted towards the investigation of ferroelectric domains and domain walls—interfaces separating regions of differing polarization orientation—as their intrinsically nanoscale nature and unique functional properties make them potentially useful for device applications6.Technological advances rely heavily on the development of new materials, both in terms of understanding their fundamental properties and through the engineering of their composition and structure. In the last decades, complex oxide materials have shown promise in photovoltaics applications8, mechanical shear9 or magnetic ordering closely tied to the distinct structure of domain walls10. Whilst most of these investigations were carried out using a combination of high resolution scanning probe microscopy (SPM) based techniques11 to explore the electromechanical and electrochemical responses, optical approaches—in particular second harmonic generation (SHG) microscopy with polarimetry analysis—are increasingly being used as a way to non-invasively probe the internal structure, chirality and polarization of domain walls16.Ever more detailed studies have revealed unusual behaviours such as localized electrical conduction17. As the spatial resolution of optical techniques increased, joint usage of optical and SPM techniques allowed the gathering of a wealth of knowledge about the structure, hierarchy and organisation of ferroelectric domains, as well as their properties in uniaxial ferroelectrics such as lithium niobate and tantalate19. Overall, such multifaceted approaches are expected to become the norm in the community, in particular due to the development of ever more precise spatially resolved functional measurement techniques, allowing the probing of a large variety of electronic and optical phenhomena at the nanoscale resolutions20.Combining the two felicitously complementary techniques—with sequential SPM and SHG measurements of the same intrinsic or engineered domain structure—provides an opportunity to extract a complete structural and functional portrait of domain walls. Indeed, such multi-technique studies have already been leveraged at varying length scales and with a varying degree of correlation. At the most basic level, SPM imaging has been used in order to locate areas of interest for coarser resolution micro-Raman measurements, enabling a better understanding of the resulting signal from the qualitative model of the underlying domain structrures21, in particular in ferroelectric and multiferroic thin films24. As a result, disentangling the SHG signal of a specific domain wall from the response arising from neighboring walls, adjacent domains, and surface/interface (background) contributions can be an extremely tedious task, made more so by the very small size of the investigated objects . In contrast, SPM based techniques possess a high spatial resolution and can be used to directly probe the polarization orientation, mechanical response, surface charge dynamics, electrical conduction and other surface properties. However, SPM is prone to artifacts such as signal contributions from electrochemical and electrostatic interactions, or mechanical cross-talk, which pose challenges for the quantitative and sometimes even qualitative interpretation of the ferroelectric polarization. Above all, the very different observation scales and analysis approaches make a quantitative correlation between SHG and SPM datasets a formidable problem.However, despite their complementarity and successful application at submicron lengthscales, the very different nature of the various optical and SPM techniques has so far made a fully quantitative and correlative analysis of their observables at the level of subnanometric domain walls extremely challenging. Specifically, in SHG polarimetry experiments, stacks of images are recorded as a function of the laser polarization and the analyzer angle, with multiple potential contributors combining into the resulting SHG signal and making analysis and interpretation highly nontrivial28. Computer vision can be used to match data based on common features across the two completely different techniques29. Machine learning techniques can not only identify and quantify behaviours in large data sets across multiple parameters, but through additional pre-processing methods such as dimensional stacking can provide physical insights into the intrinsic correlations at multiple length scales and across characterization methods30.A solution to this challenge is provided by the recent advances in the use of computer vision and machine learning for the investigation of ferroelectric materialsc-domains. Classical analysis methods—including manual data extraction and superpixel analysis—are presented, and compared to K-means clustering on the stacked SHG dataset, which is not only consistent with the previously published manual analysis16, but provides a significant improvement in terms of identifying physically meaningful and distinct behaviours. This stacked K-means clustering analysis—requiring no manual input other than number of behaviours—greatly enhances the ability to analyze SHG signals, especially in more complex cases with the presence of a background signal and/or combination of multiple physical behaviours. In particular, it allows us to identify additional behaviours revealed by machine learning that might have been discarded as “noise” or “experimental error” during manual analysis. Finally, we extend this approach to include both the SHG signal and co-localized PFM response through spatially correlative machine learning—enabling a better discrimination of domain and domain wall functional responses, and a truly nanoscale understanding of their structure-property relationships.In this paper, we leverage a combination of machine learning based techniques with data stacking approaches in order to enable a fast, correlative, and unbiased analysis of SHG and SPM data sets. This correlative approach is first introduced and discussed from the workflow point of view, then demonstrated via a comparative study of the different data analysis methods. Based on a data set acquired on a Pb, performed on each consecutive pair of SHG images which have been subjected to a median subtraction and a modulus operation. The generated list of cumulative offsets is then applied to the original SHG data, yielding a position-corrected SHG dataset.First, we address instrumental and thermal drift resulting from sequential SHG acquisition performed for each analyzer/polarizer configuration. The drift is corrected through the use of a translation geometric transform (cv2.MOTION_HOMOGRAPHY), as more than simple translation shifts may be involved in the use of heterogeneous data.Second, the corresponding SPM images are matched to the SHG dataset. As the images were acquired through piezoresponse force microscopy, not only surface topography but also the phase and amplitude of the sample electromechanical response are available. The latter represent the qualitative orientation and magnitude of the polarization. As the matching has to be performed based on information common to both datasets, proxy images have to be generated to allow comparison between them. For the SHG dataset, the median of the stacked and corrected dataset modulus is used, highlighting the position of the ferroelectric domain walls with a width characteristic of the optical resolution of the technique. For the SPM dataset, the phase representing the polarization orientation is binarized, then cleaned to filter small holes/objects, and finally eroded/dilated to provide a precise position of the domain wall. The final domain wall signal is then Gaussian-blurred to the spatial equivalent of the optical resolution. The resulting SPM proxy is further distortion-corrected to the SHG proxy through the use of a homography transformation  within the written triangular down-polarized domains. These are visually identified in Fig. As an alternative method, each image can be subdivided into a checkerboard consisting of homogeneous squares of multiple pixels in which the intensity is averaged to enhance the signal-to-noise ratio. The process is repeated for each analyzer/polarized configuration, yielding a separate polar plot for each of these superpixels. All of the polar plots are then represented at their superpixel center position, yielding a plot field like the one shown in Fig. Accessing the internal structure of a domain wall with SHG polarimetry analysis requires a detailed modeling of the SHG emission based on local symmetry and polarization orientation (see “Methods”). This modeling procedure is particularly difficult in the case of thin ferroelectric films due to the contribution of the background signals, and the nanoscale nature of the domain wall itself. Indeed, even if the SHG signal arising from symmetry breaking at surfaces and interfaces is known to be small, its contribution in thin films and multilayers can often induce an anisotropic response, which could for instance result in artificially elongated polar plots. Therefore, the subtraction of the surface SHG contribution from the signal emitted by the selected ROIs can strongly alter the resulting anisotropy of the object, i.e., the polarimetry response of a domain wall in the case of this study  and attention (focus on specific objects) of the researcher making the decisions. The two methods presented above are clear examples of this human-machine barrier. In the traditional method, manual delimiting of the ROIs introduces a potential bias—usually favoring regions of high and uniform intensity. The superpixel analysis, in contrast, will favor the selection of larger ROIs due to the visual similarity of polar plots appearing less important than their intensity alone.Automating data segmentation is therefore indispensable if errors arising from human intervention are to be reduced. A partial improvement is provided by standard image processing software, which allows for a semi-artisanal segmentation, generally based on histogram analysis and thresholding. This method provides satisfactory results in unambiguous cases, where the image contains objects of discrete gray levels and an almost perfectly homogeneous background. However, it cannot be properly applied to complex systems such as the local polarimetry analysis presented here. Domain walls with different orientations superimposed onto an inhomogeneous optical response of the background , and each point is attributed to the cluster of the closest centroid. The centroids are then displaced to the resulting cluster center. This expectation-maximization process is repeated until convergence, with the cluster number for each parameter space vector assigned to its spatial position. The K-means analysis of the SHG polarimetry data set is shown in Fig. n has the effect of identifying a larger number of regions with a distinct SHG signal, with Clustering methods are particularly suited for the identification of groups with distinct properties in a given data set, based on a concept of similarity between elements within each cluster. In K-means clustering, this similarity is understood as the Euclidean distance between data points within their multidimensional parameter space. When applied to the SHG polarimetry data set, each analyzer/polarizer configuration is considered as one parameter dimension, with the complete data set represented by spatially referenced parameter space voxels. To use K-means clustering, one has to define n. These could for instance be small regions with similar polar plot behaviors throughout the analyzer/polarizer parameter space. These features are shown, for 35. Moreover, the clustering seems to present distinct features right at the edges where the domain walls along the different directions meet and have to adapt crystallographically. Such features may stem from a complex domain structure, and would require supplementary atomistic resolution studies, for instance with transmission electron microscopy in order to be fully understood. Additionally, we can clearly identify heterogeneities within the background regions, as well as a noticeable difference between the up-polarized 36 for the former, and from the structural modifications and strains generated through the writing of ferroelectric domain structures with an SPM tip37 for the latter. Overall, the additional features can be explained through physical considerations, and reveal behaviours that were previously overlooked as a result of the manual selection process in the artisanal segmentation methods.Besides enabling a full automation of the image segmentation, K-means clustering gives us the possibility to resolve additional features that are impossible to identify by the human eye, simply by increasing the number of clusters n number above this value. The benefit of additional feature detection with increasing n must be balanced against the risk of over-fitting, where an excessively large number of clusters could lead to their noise-induced separation, while in reality they belong to a group with the same physical properties. In fact, the absolute limit of n is, trivially and meaninglessly, the number of data points provided to the algorithm—with each data point occupying its cluster. Thus, to answer the question of how many clusters are enough, one has to make use of physical intuition as well as quantitative metrics such as silhouette score and plots38. An example of such an analysis is detailed in Supplementary Fig. One can naturally question the pertinence of increasing the As discussed above, the overarching goal of this work is to devise an approach to perform a quantitative and correlative analysis of SHG and SPM data sets acquired over the same regions. This is enabled through the use of a combination of computer vision distortion correction, machine learning based techniques,and data stacking approaches.39. The difference between the proxy and SHG images is shown in Fig. The correlation of the two data sets is performed on the spatial dimensions, as both SPM and SHG are spatially resolved techniques. However, not only the length scales but also the spatial resolution of these techniques are radically different. Whereas SPM is capable of nanometer-scale precision, SHG is limited by the optical wavelength of the laser beam. It is thus necessary to generate a proxy image from the SPM data set to perform spatial correlation with SHG. To this end, the piezoresponse phase—allowing a clear identification of the ferroelectric domains—is first binarized and then skeletized through dilation and erosion, yielding a pixel-wide domain wall image. The image is then Gaussian-blurred to the resolution of SHG, finally resulting in the proxy image shown in Fig. Once registered, the SPM data can for example be used to understand the origin of the various optical responses found through the K-means clustering discussed above. Here, the Once combined, the two SPM and SHG sets can be treated together in order to further extract correlations across the different length scales. The above analysis can for example be extended by the inclusion of the SPM channels as additional parameter dimensions in a full K-means analysis, as illustrated in the Supplementary Note We have demonstrated a workflow for the automated recovery of SHG features through the use of machine learning and data stacking techniques. By applying the workflow on a previously published data set, we have unearthed significant additional insights into the structure-property relationships and functional responses at domain walls in PZT thin films, with much more limited intervention and effort. By combining the workflow with computer vision approaches, we have presented a quantitative correlative analysis of SHG and SPM data sets—two techniques with radically different modes of operation and spatial resolutions.In fine, the approach presented here opens up new possibilities at the cross-roads between materials physics and data science. We envision that in the future, a use of holistic, cross-technique analysis will become standard practice in the quest for the understanding of material physics. The correlative workflow discussed in the current work is driven by data science, and is applicable to any set of spatially resolved techniques. This opens a wide range of possibilities for correlated studies of complex oxides or mixed phase polymers by a combination of optical, magnetic, morphological, and functional probing techniques.m point group symmetry, which corresponds to a Néel-type domain wall structure. A detailed description of the method can be found in16.The SHG measurements were conducted with a scanning confocal microscope in reflection geometry. In order to analyze the internal structure of the domain walls, polarimetry experiments and simulations based on symmetry arguments are conducted. The modeling of the SHG polarimetry response is based on the analytic form of the SHG by assuming a non-Ising character  of the domain walls. In the case of tetragonal Pb(15.The scanning probe microscopy measurements were performed in piezoresponse force microscopy mode on a Bruker Dimension 3100 atomic force microscope. The details of the measurement parameters may be found inSupplementary Information.


Environmental enrichment is a combination of techniques that aim to improve the quality of life of zoo animals. However, institutions might be reluctant to add certain enrichment items due to the belief that their presence could negatively affect the visitor experience in the zoo. To explore the veracity of this belief, we assessed visitor attitudes towards two types of enrichment items  in an outdoor walk-through enclosure for ring-tailed lemurs in Zoo Planckendael (Belgium). We developed a questionnaire that was answered by 371 visitors. We also took into consideration the behaviour of the animals and their visibility. We found that the visitor attitudes were more influenced by the behaviours displayed by the lemurs than the appearance of the enrichment items. We suggest that more emphasis should be placed on designing enrichment items that provide the animals with opportunities to display more active and appropriate behaviours, regardless of the appearance of the objects, in order to improve animal welfare while simultaneously improving the visitor experience.Lemur catta). Naturalistic and artificial looking enrichment items were alternately provided in the enclosure. A total of 371 visitors filled out the questionnaire: 174 in the naturalistic and 197 in the artificial conditions. Both researchers and visitors conducted behavioural observations of the lemurs. Our results suggest that the appearance of the items did not have an effect on visitor attitudes and that visitors recognised both naturalistic and artificial items as enriching for the animals. Moreover, the behaviour and visibility of the lemurs had a greater effect on the visitors’ attitudes. We suggest that during the design of enrichment items, less concern should be placed on the appearance of the items and more on their effect on animal behaviour. Ultimately, this would improve both animal welfare in captivity and the visitor experience.Decisions on environmental enrichment programmes are sometimes based on the assumption that non-natural or artificial looking items negatively affect visitor experiences. In this study, we developed a questionnaire to assess zoo visitor attitudes towards enrichment appearance in an outdoor walk-through enclosure for ring-tailed lemurs ( It is generally accepted that modern zoos have four roles, including conservation, education, research, and entertainment of the visitors . The rolUrsus arctos maritimus), the more the visitor attention was directed to the animals and their behaviours, as showed by an increase in the behaviour content in the conversations among the visitors. Godinez et al. [The effects of zoo animal behaviour on the perceptions and behaviours of visitors have also been studied on several occasions. For instance, Altman  observedz et al.  and Millz et al.  examinedz et al.  or displz et al. ,7. Tofiez et al.  examinedEnvironmental enrichment is a combination of techniques with the aim to enhance the quality of life of zoo-housed animals by providing environmental stimuli necessary for optimal psychological and physiological wellbeing ,10,11,12Mandrillus sphinx) enclosure at Beijing zoo (China). They observed that the improved enclosure attracted more visitors, who stayed longer, suggesting that the enriched design was more attractive than the previous exhibit. However, they did not compare artificial with naturalistic looking enrichment, but a barren enclosure with a remodelled naturalistic exhibit with enrichment devices.Environmental enrichment, as noted by Mellen and MacPhee , is a hoPan troglodytes) enclosure in Lincoln Park Zoo (USA). They concluded that the visitors were not affected by the aesthetics of the enrichment and showed that the largest proportion of visitors thought the naturalism of the enclosure was important. Similar results are reported in a study investigating the effects of enrichment in a polar bear enclosure on visitor perception: enrichment type, either artificial or naturalistic looking, did not alter visitor perceptions about the zoo, the enclosure, the species on display, or the individual animals [Panthera tigris altaica), lynx (Lynx canadensis), and fishing cat (Prionailurus viverrinus) exhibits in Brookfield Zoo (USA), but their results did not report an influence of the type of enrichment on visitor impressions.Most studies on the impact of environmental enrichment on zoo visitors’ experience suggested that a mismatch between enrichment appearance and exhibit design has little effect on the attitudes of the visitor ,27,29,31 animals . McPhee In another study conducted at Brookfield Zoo , visitorAccording to Luebke and Matiasek , the mosSince cultural background has been shown to have an effect on attitudes towards animals and animal welfare , visitorLemur catta) in an European zoo. Our goal was also to determine if the visibility and behaviour of the lemurs made a difference on visitor attitudes towards the enrichment devices. We expected that visitor attitudes did not differ between naturalistic and artificial looking enrichment [The aim of this study was to assess visitor attitudes towards enrichment appearance on a walk-through enclosure for ring-tailed lemurs  for 10 days in April 2019. The enclosure consisted of a walk-through outdoor enclosure, a visible indoor enclosure  and an off-exhibit indoor enclosure, which was not visible from the visitor side. The outdoor enclosure had a natural looking design that included vegetation, grass, climbing structures (including trees), and a pond. Visitors were allowed to walk on a designated path through the outdoor enclosure, where six ring-tailed lemurs roamed . It was In total, 371 zoo visitors participated in the study; 174 visitors participated on days with naturalistic looking enrichment and 197 visitors participated on days with artificial looking enrichment. The information related with the visitors that were interviewed can be found in To collect visitor attitudes, a questionnaire was developed, consisting of two parts. The first part included 14 attitude statements and one multiple-choice question on the behaviour of the lemurs. The participant had to indicate for each of the 14 statements on a 7-point Likert scale to what degree they agreed with the statement, ranging from ‘Completely disagree’ (score = 1) to ‘Completely agree’ (score = 7), with a neutral midpoint at score 4. In the multiple-choice question on lemur behaviour, visitors were asked to report whether they had seen any social behaviours, feeding behaviours, inactivity and/or activity. This question was included to collect data on the visitor perception on the lemur behaviour. The second part of the questionnaire focused on environmental enrichment and included a short text explaining the purpose of enrichment and five supporting images of examples of enrichment items in other species. After reading this text, participants were asked to score three enrichment-related statements using the 7-point Likert scale. In this section, the open-ended question ‘Do you see any enrichment items in this enclosure?’ was also included to ensure that researchers and visitors were considering the same objects as enrichment items.To avoid order-effects , three vFive enrichment items were designed, taking into account the ring-tailed lemurs’ needs and the behaviours that we wanted the animals to display when using them. The designs were approved for safety by the zoo veterinarian, the curator and the caretakers’ coordinator. We built two versions of each item: one naturalistic and one artificial looking . Among tAny time a food-related item was provided to the lemurs, two or more of the same devices were present to avoid feeding competition ,17,19. IIn order to determine the effect of enrichment appearance, we designed two experimental conditions that consisted of the presence of either naturalistic looking or artificial looking enrichment items in the ring-tailed lemur outdoor enclosure. To avoid habituation to the enrichment items ,22, threWhile the visitors were completing the questionnaire, we observed the behaviour of the ring-tailed lemurs. We used a focal continuous sampling method  and a sishapiro.test) showed the Likert-scale data to be non-normally distributed . As the normality assumption for performing t-test was not met, alternatively nonparametric Mann–Whitney U tests were run to compare scores of agreement on each of the statements between the two experimental conditions . As a means for correction for multiple testing, we recalculated p-values by the Holm–Bonferroni method (using the build-in R function p.adjust).First, to assess if visitor attitudes differed between conditions , a Mann–Whitney U test was performed on the 17 attitude statements. Shapiro–Wilk tests  Feeding, drinking, foraging; (b) Inactive ; (c) Active  Social behaviour ; and/or (e) Other. For this question, we did not want to draw the visitor’s attention towards the enrichment items, so we did not include ‘Interaction with enrichment’ as a behaviour in the list. A researcher observed the lemur behaviour during the time window where each participant was taking the survey . For each lemur, it was observed if they were active, inactive, displaying social behaviours, or out of view (if they were indoors), and if they were using an enrichment device or not. The total number of seconds that the lemurs exhibited behaviour of a certain category—sum of all lemurs—was divided by the total sum of all lemur behaviours (in seconds) exhibited during the questionnaire interval to get the proportion of each behavioural category for each participant. Then, we analysed the relation between the proportions of lemur behaviour of each category and the scores of the participants for each of the Likert-type questions.Finally, for the open question ‘Do you see any enrichment items in this enclosure?’, percentages were calculated for four pre-defined response categories. These categories included: (1) Enrichment items designed for this study and swinging rings permanently present on the enclosure; (2) Other items or features of the enclosure; (3) Sounds and musical background; (4) People (including visitors and caretakers).Mann–Whitney U tests, explorative analyses, linking researcher observed lemur activity to participant questionnaires and ordinal logistic regressions . Regarding visitor participation, trained zoo staff and research interns approached adult visitors in the outdoor walk-through exhibit for the ring-tailed lemurs and asked them if they were willing to participate voluntarily in the survey. All subjects who agreed to participate gave their verbal informed consent for inclusion in this study before completing the printed questionnaire. No personal information was collected, and all participants were assured in the introductory text that participation in the questionnaire would be completely anonymous. This study did not require ethical approval since it did not involve any interventions or handling of the animals. The enrichment items used in this study were evaluated and approved by the veterinarian, curator, and caregiver coordinator of Zoo Planckendael.Visitor attitude scores were compared between the naturalistic and the artificial condition for all 17 questionnaire statements . No statOrdinal logistic regression analyses were conducted to investigate the relation between lemur behaviour—the proportions of time the lemurs were active, inactive, inside (out of view for the participants) and engaging with the enrichment items during the time window where the participant was completing the survey—and participants’ attitude scores on each of the survey questions.‘I think the enrichment in this enclosure is fit for the lemurs’ and ‘The lemurs look happy’. The time that the lemurs spent interacting with the enrichment was significantly related to higher Likert-scores on the following statement: ‘I think the enrichment in this enclosure is fit for the lemurs’  mentioned people as enriching. In the artificial looking condition, 1% (2 visitors) considered sounds and the presence of people enriching for the lemurs.The aims of our study were to determine if visitor attitudes were influenced by the appearance of enrichment items and to explore whether the perceived activity levels and the observed activity levels of ring-tailed lemurs had an effect on visitor attitudes towards these items. The results obtained from the open-ended question ensured that researchers and visitors considered the same objects as enrichment items. Above three quarters of the visitors in each condition  mentioned at least one of our items as enriching.The analysis of the attitude statements of the 371 completed questionnaires showed that statements regarding the behaviour of the animals and their natural habitat elicited more neutral responses from the visitors, indicating that the visitors did not have a strong opinion about these aspects. As for the other statements, visitors responded generally in favour of the zoo, its management and care: lemurs appeared well taken care of, relaxed and happy, not stressed, and their exhibit and the enrichment in it were rated good-looking.Our results showed that enrichment appearance did not have an influence on visitor attitudes. Visitors generally reacted positively to all enrichment items, naturalistic and artificial alike, and recognised the purpose of both naturalistic and artificial items as being enriching for the animals. The fact that the lemur enclosure in Zoo Planckendael is an immersive experience, where visitors walk through the enclosure following a pathway, might make the visitors more flexible to accept artificial enrichment devices, compared with naturalistic enclosures that lack such constructed elements. Nevertheless, these results are in line with those already published on visitor attitudes towards enrichment performed in other, non-European, zoos ,27,31.Providing appropriate enrichment items can increase the activity level and natural behaviour of the animals by offering them opportunities to engage in species-appropriate behaviours ,35,38. IImportantly, when lemurs were interacting with the enrichment items, visitors provided higher scores for enclosure suitability. Moreover, when seeing the lemurs feeding on a food item from a food-based enrichment device, visitors found the enrichment items to disturb the view of the enclosure less and agreed more on that the enrichment was suited for lemurs.Eulemur coronatus) [The more active behaviours were displayed by the lemurs as observed by the researchers, the more positive visitor attitudes about the enclosure, the enrichment, and the lemurs’ emotional state. We argue that the functionality, rather than the appearance, of the enrichment items is important, considering its efficiency to elicit the display of more active and species-specific natural behaviours , which ironatus)  lemurs iSeveral tools can be used to ensure that zoo visitors understand the importance of environmental enrichment for the animals, including educational signs, animal training and presentations, or having staff or volunteers draw the attention of visitors to the enrichment devices in an enclosure and clarify how the animals use them ,21, espeOur results suggest that visitors of the ring-tailed lemur enclosure were not affected by the appearance of the environmental enrichment items. Instead, visitor attitudes were affected to a greater extent by the activity and the behaviours that the animals displayed. Behavioural data on the activity of the ring-tailed lemurs, continuously recorded by the researchers during the experiment, as well as the visitors’ own reports of activity were shown to influence the visitor attitudes. Visitors found that the lemurs were well taken care of, scored the welfare of the lemurs high, and reported that both the enclosure and enrichment items were good-looking and appropriate for the species. We consider that our study supports the idea that zoo management decisions on environmental enrichment programmes should not only be based on the appearance of the enrichment devices, but on how they can stimulate certain species-specific behaviours in the animals. This would ensure better welfare for the animals in zoo settings, since they would benefit from good enrichment items regardless of their appearance, as well as enhance the experience of visitors.


Artificial Intelligence applied to Structural Health Monitoring (SHM) has provided considerable advantages in the accuracy and quality of the estimated structural integrity. Nevertheless, several challenges still need to be tackled in the SHM field, which extended the monitoring process beyond the mere data analytics and structural assessment task. Besides, one of the open problems in the field relates to the communication layer of the sensor networks since the continuous collection of long time series from multiple sensing units rapidly consumes the available memory resources, and requires complicated protocol to avoid network congestion. In this scenario, the present work presents a comprehensive framework for vibration-based diagnostics, in which data compression techniques are firstly introduced as a means to shrink the dimension of the data to be managed through the system. Then, neural network models solving binary classification problems were implemented for the sake of damage detection, also encompassing the influence of environmental factors in the evaluation of the structural status. Moreover, the potential degradation induced by the usage of low cost sensors on the adopted framework was evaluated: Additional analyses were performed in which experimental data were corrupted with the noise characterizing MEMS sensors. The proposed solutions were tested with experimental data from the Z24 bridge use case, proving that the amalgam of data compression, optimized  machine learning architectures and environmental information allows to attain high classification scores, i.e., accuracy and precision greater than 96% and 95%, respectively. Damage detection has a pivotal role in Structural Health Monitoring (SHM) systems as a fundamental means to implement on-condition maintenance. In particular, many novel damage detection procedures are gaining momentum thanks to the recent developments in the Machine Learning (ML) field ,2,3,4,5.Another emerging approach, Spiking Neural Networks (SNN), was recently adopted for damage assessment purposes. SNNs are peculiar in that they try to tackle the problem of structural inference by means of a more realistic mathematical representations of the human brain, which specifically mimics biological spike-based event-driven processes to communicate between neurons  . Nev. Nev24])Furthermore, condition monitoring data are typically affected by environmental and operational parameters (EOPs), whose effect on modal parameters can be even more pronounced than the one due to pure structural degradation . Thus, Etarget  or outlier  class. OCC solutions based on standard neural network (NN) models were shown to achieve good classification scores in numerous application scenarios, such as the monitoring of industrial plants . The objective of OCC is, therefore, to find which specific class a given input object belongs to by selecting either the l rotors , wind fal rotors ,33) and l rotors ,35. In tl rotors  and valil rotors . Indeed,l rotors ,39. In fl rotors  are veryl rotors  does notInput data. It is investigated how CS methodologies, which reduce the probability of network congestion, may affect the classification process. Temperature values are provided as additional input data for the NN machine to inherently model the dependency of modal features on environmental factors.Knowledge distillation. A reduction in the complexity of the NN models is performed by shrinking the number of neurons in the hidden layers, without affecting the classification accuracy with respect to more redundant configurations.MEMS noise density. Acceleration waveforms are corrupted with the intrinsic noise density characterizing MEMS-based sensors, which are the most widely adopted sensing technology in this kind of application and, thus, need to be properly handled in view of real installations. Hence, the robustness of the classification process under this technological limitation is evaluated.In this work, an OCCNN-based damage detection procedure for dense accelerometer networks is proposed and tested against the possible limitation of commercial off-the-shelf MEMS sensors. Such a procedure is compatible with the deployment of large sensor networks and embedded processing solutions. More specifically, the main contributions can be listed as follows:The paper is organized as follows. The complete processing flow, from the data compression/decompression stage up to the classification process, is thoroughly illustrated in data compression and recovery phase, which is aimed at retrieving the original time waveform from compressive acquisitions; (ii) the modal identification procedure, returning the structural features of interests; and (iii) the final classification stage, which leverages ML techniques as enabling tools for structural integrity assessment.The monitoring framework proposed in this work  and compression procedures that are performed by local sensor nodes installed on the structure . Subsequsensing matrix   is suggeThe strategy to be implemented for the derivation of the optimal sensing matrix as dictated by the MRAK-CS approach is schematically represented in Modal identification can be performed in the time or frequency domain. The selection of one category over the other is usually influenced by the required frequency resolution, which is a consequence of mode proximity and the allowable computational complexity . The basConversely, stochastic subspace identification (SSI) algorithms  tackle tF containing the h-th vibration component of interest: At each step, the filter is designed to update the mean value The crucial point of SSI methodologies is the selection of the most appropriate model order, on the basis of which the system matrices are computed and the corresponding modal parameters can be retrieved by means of an eigenvalue decomposition. It is worth noting that the model order depends on the operational and environmental conditions, therefore such a parameter must be adaptively selected. To tackle this issue, the so-called stabilisation diagram , i.e., aQ first-order differential equations that emulate the dynamics of the underlying physical  system.Amidst the various SSI implementations, the SSI-COV method in l and time shift s, the block Toeplitz matrix of dimension Compute, for fixed time lag Perform the Singular Value Decomposition (SVD) of A, the state matrixC and the next state–output matrix G. While C and G can be easily extracted from the first Q rows (columns) of the controllability and observability matrix, respectively, the computation of A is given by Apply the state–space factorization of the covariance matrix. Starting from the pure algebraic manipulation of the SVD, one may write Q eigenvalues Execute the eigenvalue decomposition of the above-computed state matrix. This is decomposed as f and mode shapes Estimate the sought natural frequencies of vibration The approach involves the following steps:As a competitive variant to SSI-COV, in SSI-DATA the computation of the covariance matrix is replaced by the projection of the row space of future outputs into the row space of past outputs. The problem with this method is that it involves the factorization of a very large matrix and, hence, becomes very computationally onerous. For this reason, SSI-COV inherently provides a much faster and efficient algorithmic solution, since the derivation of irection . Beside,irection  have verModal parameters are extremely sensitive to environmental factors  since they are constitutive elements determining the stiffness and damping property of the structure .E) as additional input features of the AI block. In this manner, the neural network is instructed to autonomously learn this frequency vs EOP relationship, without requiring any further processing steps to be performed aside.To cope with environmental factors, the conventional approach is to resort to regression methods . Conversadversarial population). In these cases, a possible alternative consists in artificially generating these adversarial points. Among the OCC implementations presented in the literature, the very recent OCCNN proposed in [OCCs can be seen as standard neural networks trained with samples acquired for the pristine structure since no training data is usually available for damaged conditions . The latter value can be computed as [n-th point and its OCCNN  is an MLribed in , APG genputed as  λi=∑n=1Nal ones) .These adversarial data, together with healthy instances, are plugged as inputs to the second NN component, that is a two-layer fully connected NN with The adversarial point generator cycles are iterated until the desired level of fitting with respect to the training data distribution is reached. The higher the number of cycles, the higher the resolution of the boundary contours will become. It follows that two key variables might significantly affect the classification performance of OCCNN, which are the number In essence, an ANN  Figure  represenThe openly available dataset related to the Z24 bridge  providesWhen compared with previous works related to the same dataset, the analysis presented here is novel in that it jointly evaluates three pivotal aspects for the design of the next generation of SHM architectures, which are: (i) The investigation of the effects of adapted compressed sensing on the quality of the identified bridge health status, (ii) the exploitation of temperature data as direct input of the NN models, (iii) the inclusion of instrumental non-idealities on the entire monitoring processing flow, and in particular of the residual noise density characterizing commercial off-the-shelf MEMS accelerometers.A dataset preparation phase was necessary, too. In particular, the machine learning models adopted in this work only need data from the normal class during training. Thus, 70% of these data from the normal class was randomly sampled to favor diversity in terms of environmental conditions. This subset was further subdivided: 70% of it was used for training and the leftover 30% allocated to validation. Conversely, the remaining 30% of the normal class together with all the data acquired in damaged conditions were employed for testing purposes.For the target scenario explored in this work, modal analysis studies applied to the Z24 dataset proved that the most relevant modal components of the bridge are located below 20 Hz. In particular, at a reference temperature value of 25 °C and in nominal working conditions (no vehicle passing through), the three dominant bending modes are located at rated in , while tThe acquired waveforms were divided into 512-sample long segments and the compression ratio, intended as the ratio between the number of columns  over the whole structure, so as to precisely keep trace of EOP effects on the vibration signature. As already proven in previous works for the Z24 use-case [It is worth underlining that, thanks to the relatively high thermal inertia of the structure, just one temperature value per acceleration series has to be stored. Trends in modal frequencies induced by temperature fluctuations are depicted in O and A. The initial number of parameters for the NN models was selected to provide a fair comparison with the reference work in [The details of the explored NN models are presented in this subsection. For the sake of brevity and from this point onward, the acronyms OCCNN and ANN will be denoted as  work in ,46, in wFor the ANN case, T  devices are characterized by high sensitivity, low-power consumption and very high integration levels, which made this sensing technology a cost-effective yet reliable and extremely advantageous alternative to the piezolectric counterpart for the design of accelerometer sensors . Indeed,® Sensor Fusion and Tracking Toolbox™, which offers specific routines to simulate the mechanical behavior of inertial measurement units, including accelerometers and gyroscopes.Notwithstanding their successful adoption, signals acquired by MEMS accelerometers are affected by comparatively higher intrinsic noise density values, which thus need to be properly accounted for in the signal processing chain to assess the actual performances of the implemented algorithms. To this end, further analyses were performed in this work, in which the original data were degraded by adding the residual noise floor inherent in two different kinds of digital accelerometers. The noise was generated via the MatlabIn more detail, the mechanical characteristics in a priori.For the sake of NN validation, noise-corrupted data can be treated as novel datasets and, for this reason, they are taken as inputs to the previously trained i) Assess the improvement brought by the introduction of temperature values as additional input features of the AI block; (ii) evaluate the effect of compression/recovery stages on the classification performance of the designed SHM framework; (iii) reduce the complexity  of the NN models to be compatible with embedded processors without impinging on the accuracy of the classification; (iv) evaluate the effect of MEMS noise floor on classification performances to cope with real issues.Four main objectives were pursued within the experimental validation phase: . Moreover, it performs even better than alternative solutions with much higher parameters (see models B and C). This is further justified by the very similar classified instances reported in the corresponding confusion matrices of Remarkably, model ∞) to MB MEMS-type, a trend which is also clarified by the confusion matrices (e) and (f) in The impact of non-negligible noise floors in MEMS accelerometers can be quantified by observing the last three columns in As such, it was demonstrated how the selection of the sensing unit might play a crucial role in the effectiveness of the adopted signal processing techniques, which must cope with the inherent source of non-idealities while moving from theoretical analyses to real case studies.In this work, an investigation on how data compression may affect the performance of ML-based damage detection procedures in structural health monitoring is presented. In particular, one of the most promising compression techniques (namely MRAK-CS) was applied to the dataset of the Z24 bridge and the recovered data were used to feed low-complexity neural networks to perform the anomaly detection task.In particular, it was shown: (i) How the compression ratio influences the detection performances; (ii) the different degradation in the performance achieved with different classifiers (ANN and OCCNN) as a function of the network complexity; (iii) the practical implementation of the model in low-power and resource constrained devices; (iv) the beneficial effect brought by the inclusion of temperature data among the inputs of the network; and (v) the impact of non-negligible noise floors on MEMS-based accelerometers on classification scores.From the results, it can be concluded that the OCCNN architecture with eight neurons per layer and a compression ratio equal to 6 achieves a negligible degradation with respect to much deeper networks applied to features extracted from uncompressed data. The low computational cost of the implemented network is compatible with the storage and processing resources of low-cost microcontrollers, and the compression stage allows to minimize the risk of network congestion. Further works will include the validation of the proposed approach in multiple application scenarios and the inclusion of additional features, such as damping factors and mode shapes, as inputs for the NNs, in order to further improve the classification results.


The prediction of hand grasping and control of a robotic manipulator for hand activity training is of great significance to assist stroke patients to recover their biomechanical functions. However, the human hand and the figure joints have multiple degrees of freedom; therefore, it is complex to process and analyze all the collected data in hand modeling. To simplify the description of grasping activities, it is necessary to extract and decompose the principal components of hand actions. In this paper, the relationships among hand grasping actions are explored by extracting the postural synergy basis of hand motions, aiming to simplify hand grasping actions and reduce the data dimensions for robot control. A convolutional neural network (CNN)-based hand activity prediction method is proposed, which utilizes motion data to estimate hand grasping actions. The prediction results were then used to control a stimulated robotic model according to the extracted postural synergy basis. The prediction accuracy of the proposed method for the selected hand motions could reach up to 94% and the robotic model could be operated naturally based on patient’s movement intention, so as to complete grasping tasks and achieve active rehabilitation. The hand is one of the most important parts of the human body and it has the characteristics of multiple degrees of freedom and flexible motions. It is difficult to recover the hand functions when the biomechanical and neurophysiological abilities are injured or damaged due to diseases such as stroke. At present, the aging problem in China is serious and the number of elderly aged 60 and over will exceed 300 million by 2025 . The incIn recent years, new techniques for identifying hand motions have been developed ,6,7, theHand gesture recognition methods based on EMG and neural networks, data gloves embedded with sensors and color marking are here compared in depth. Kang et al. proposed a method based on feature calibration to improve the accuracy of gesture recognition based on electromyography (EMG), which is robust to noise . Sheng eIn fact, though the movement of the human hand is the result of many joints working together, we tend to use the first action as the principal gesture feature and complete the task via the cooperation of other finger joints ,20. PrevUsing postural synergy basis can decompose and simplify grasping actions, but how to understand the neurologic gesture coordination and how to accurately predict hand actions is still an open problem. The convolutional neural network (CNN) is one of the representative algorithms of deep learning, which is constructed by imitating biological visual perception and imitating the connection between neurons. Hand grasping activities are affected by many factors, such as the shape and weight of the object, or the smoothness of the contact surface. A CNN can effectively predict human hand movements and improve the efficiency of online prediction thanks to its capacity to process complex and high-dimensional data. Shen et al.  designedIn this paper, a motion intention prediction method based on a convolutional neural network is proposed. The collected motion data of hand grasping activities were input into neural network for gesture prediction. Then, according to the prediction results, the attitude coordination base was extracted from the motion data. By coupling the relationship between finger joints, the reduced-dimension robot model was controlled to ensure the real-time and natural interaction of robot-assisted hand rehabilitation. The main contributions of this paper include the following: (1) By analyzing various hand grasping actions, common hand grasping activities are classified. Three common grasping gestures are selected as the input of the CNN predictor for gesture recognition. (2) An improved CNN-based method for hand grasping activity prediction is proposed to identify hand motions to enable the patient-dominated control of a robotic manipulator for active rehabilitation. (3) An extraction method of attitude cooperation base is proposed which significantly decomposes grasping motion data and reduces the dimension of controlling grasping. A simulated robot manipulator for grasping activity planning and training was designed and posture coordination was used to enhance the real-time and natural control of patients’ active rehabilitation. To the authors’ knowledge, existing studies have not yet reported such a practical solution for various patient-in-charge hand gasping activity training via CNN-based prediction and control via postural synergy extraction. The recognition results in this paper could be used to control various gestures of a modular soft exoskeleton for the active hand rehabilitation of stroke patients  and coulThe rest of this paper is organized as follows: When conducting different grasping tasks, the human hand takes different grasping postures and forces from the palm and the fingers. Hand grasping tasks can be roughly divided into two types, force grasping and fine grasping. A standard human hand grasping classification table was established to mine the grasping posture types of the human hand for different objects. Feix et al. proposed a classification method for 33 grasping postures, which combined previous research and different definitions of hand grasping postures. They further refined the grasping classification table and defined the “Grasp” taxonomy according to the sponsored project . The “GrThe “Grasp” taxonomy is established on the following four bases: (1) according to the grasping force, as grasping actions can be divided into force type, fine type and the type between the first two; (2) according to the relative position between the object and the human hand, as the hand has pad operation, palm operation and side opposition; (3) according to the number of virtual fingers when fingers work together as a functional unit; (4) the thumb can be abducted or adducted according to the state of thumb.For patients following neural diseases such as stoke, their hand’s strength and ranges of motion are very limited. To meet the needs of patients, we simplified grasping activities into two types. (1) According to the strength of hand grasping, we divide hand activities into force type, fine type and the type between the first two. The force-type grasping action needs to be carried by the hand with larger grasping force, which means that the contact area between the hand and the object should be larger, so that the object can be grasped more stably. The fine-type action usually does not need a larger grip force, but it needs to accurately complete the motion to grasp the object, which means that the palm may not make contact with the object and the contact area between the hand and the object is smaller. (2) According to the position of the thumb, we define the movements with one being the abduction of the thumb and the other the adduction and these are also related to the position of the palm; when the thumb and the palm are on the same side, the gesture of the hand is thumb adduction and when the thumb is on the opposite side of the palm, the gesture is thumb abduction. This study focuses on the situation whereby the patient cannot complete all the actions listed in the classification of hand grasping movements in the process of rehabilitation. In addition, we considered that the hand manipulator cannot complete some too-fine movements. In In order to meet the needs of rehabilitation training for patients with hand movement disorders, we designed a hand manipulator based on hand movement prediction to complete the auxiliary grasping task, as shown in To predict complex and high-dimensional hand grasping activities accurately, an improved CNN-based classification method is proposed in which higher-level features are used to describe the data to possess better discriminant characteristics. The hand grasping dataset was employed to train and optimize the CNN predictor. Since the data dimension here was not as high as the image, a 2 × 2 convolution kernel was used. We designed two convolution layers, two pooling layers and a full connection layer to classify the three gestures and output the classification accuracy. In addition, the number of iterations was adjusted to avoid the problem of overfitting. The CNN model parameters are shown in Open datasets of hand grasping motions are available on line. The hand grasping datasets provided by  were use(a) Input LayerAfter the first element time node was deleted for each row of data in the dataset, the rest were 1 × 16 arrays, which represent the angles of hand joints at each moment. In order to facilitate data convolution, we converted the 1 × 16 vector into a 4 × 4 matrix. All the motion data of all subjects in the same grasping posture were extracted from the dataset. As the completion time of each group of gestures was different, we used n to represent the number of sampling points. The n × 16 joint angle matrix (b) Convolution Layeri-th input, j-th output, j-th output, f is the excitation function and ∗ stands for the convolution operation. For the input data with 16 dimensions, a 2 × 2 convolution kernel and two convolution layers were adopted here. The first layer output n × 4 × 4 × 32 feature maps, while the second layer output n × 4 × 4 × 64 feature maps.The function of the convolution layer in a CNN is to extract the corresponding features after calculating the convolution kernel and input data. The convolution layer adopts the way of local connection, so the neurons are connected with the local area in the feature map. The expression of convolution is(c) Pooling LayerMax pooling selects the maximum value in the neighborhood window to replace the whole values of that window, which can further improve the nonlinear fitting ability of a CNN. It is the most widely used pooling method in CNNs. A 2 × 2 maximum pooling layer and two pooling layers were adopted. The first layer output n × 4 × 4 × 32 feature maps with a step size of 2 × 2 and the second layer output n × 4 × 4 × 64 feature maps with a step size of 1 × 1.(d) Excitation LayerThe role of the excitation layer is to map the data processed by the convolution layer in a non-linear way, which gives the CNN the function of non-linear change and enables the CNN to classify non-linear separable data. The most commonly used function is the ReLU function, which converges faster and the gradient solution is simpler. We chose the ReLU function as the excitation function, whose expression is(e) Fully Connected LayerEach node of the fully connection layer is connected with all the nodes of the previous layer and all the features extracted can be integrated. The fully connection layer maps the characteristics of the data to the labels of the samples by weighted summation and then uses the classifier to classify the data. We set the full connection layer with a large number of neurons and then added a dropout layer to reduce the overfitting of the model. We set the number of nodes in the full connection layer to 1024.(f) Output Layersoftmax function is used to control the output ranges from 0 to 1, which denotes the probability of each category. In gesture prediction, the softmax activation function is used to generate the probability distributions over three categories. k is the label of the sample. If multiple data have the same label of the input, making the number of data n, then the loss function of the softmax function isThe output layer is the last layer of a CNN. The softmax function be a. By deriving the above equation, we obtainLet the output of the To make the output predicted value approach the real value, the model weight was iteratively adjusted. The smaller the cross-entropy loss, the better the model is. So,softmax to obtain the classification accuracy. Finally, we chose the gesture with the highest probability as our prediction target. In total, 2000 iterative training sessions were conducted for each classification and the accuracy of classification was output every 100 training sessions, as shown in This study focuses on the situation whereby the patient cannot complete all the actions listed in the classification of hand grasping movements in the process of rehabilitation. In addition, we considered that the underactuated manipulator cannot complete some too-fine actions. We selected three groups of common hand movements for CNN prediction. The CNN output was transformed into probability distribution by For Group 1 of hand activities with large-diameter, power-disk and power-sphere actions, the prediction accuracy reached 94%. Small diameter, legal and palmar actions were selected for Group 2, with an accuracy of 89%. The accuracy was only 43% for Group 3, which included large-diameter, extension and parallel actions. This indicates that the CNN-based prediction method was greatly influenced by the differences in grasping postures. According to the prediction results and considering the common hand grasping activities that patients can complete in their daily life, the three groups in In the next section, we present our use of the gesture results recognized by the CNN to control the simulated manipulator to complete grasping tasks, which were formulated according to the recognition results. The patient motion data were input to the CNN for prediction. The dimension of the data was reduced according to the recognized gesture. Finally, the dimension-reduced joint angle data were input into the simulation manipulator to realize the assistive grasping of the simulation manipulator controlled by the patient.The prediction results of patient gesture data obtained by the CNN showed the patient’s motion intention, but there were many redundancies in patients’ hand motion data. When controlling an underactuated manipulator, it is necessary to input high-dimensional features for control. In order to simplify the control of the manipulator, we used a principal component analysis (PCA) to reduce the dimension of the patients’ motion data. Finally, we used the dimension-reduced data to control the manipulator, complete the grasping task of the patient and realize the active rehabilitation of the patient.In a grasping task, not all joint angles of the hand are controlled independently of each other. We analyzed the data of all joints in the hand grasping movement dataset, including the joint angle data of 30 subjects repeatedly grasping three objects with different shapes in 33 types of gestures. Finally, the power-disk action in the dataset was selected to draw the correlation coefficients among finger joints, as shown in The correlation coefficient diagram intuitively shows the correlation among joints. In the power-disk action, the metacarpophalangeal joint (MCP) angles of adjacent fingers were often highly correlated, as were the adjacent proximal interphalangeal joint (PIP) angles. There was a high correlation between the proximal interphalangeal joint (PIP) and interphalangeal joint (DIP) angles of the same finger. The correlation between the thumb joint and the other finger joints was weak. The correlation decreased with the spacing between finger pairs. The positive correlations between joints indicate that the motion of one joint was usually accompanied by the others, considering the coupling characteristics of hand activities. Meanwhile, it was possible to further reduce the dimensions of the hand motion data and accelerate the processing of hand grasping features.For joints with no obvious correlation, it does not mean there is no coupling relationship; instead, it is possible that a simple linear relationship cannot describe the coupling between these joints. Therefore, more advanced data mining tools are needed, as to obtain the coupling relationship between joints. Since the principal component analysis (PCA) is a common data mining method, which can reduce the actual dimension of data, it was utilized here to extract the principal components with dominant characteristics of hand grasping actions.i-th joint; The mathematical expression of the PCA is as follows:The postural synergy basis is the dominant principal component with the greatest influence on hand motions. Using the postural synergy basis, we could project the high-dimensional hand data into the low-dimensional space and then complex hand motions from low-dimensional data could be constructed. In order to determine the correlations among the joints, the angle data of 15 joints that followed the time change were paired to calculate the correlation coefficient. The correlation coefficient shows the coupling relationship between two joints. Data dimensionality reduction was achieved by solving the eigenvalues and eigenvectors of the joint angle covariance matrix. For example, we selected the first two principal components to restore the original hand grasping action; then,In order to extract the gesture collaboration basis, we calculated the correlation coefficients among the joints in the same gesture grasping task. The correlation coefficient normalizes the covariance matrix to facilitate the observation of the correlation. T, I, M, R and L represent thumb, index finger, middle finger, ring finger and little finger, respectively. The square with color is used to represent the correlation coefficient between two joints, while the colors change from blue to red. Dark blue denotes that the correlation coefficient is 0, while dark red denotes 1. The data between two joints are axisymmetric. The CMC joint, MCP joint and IP joint of the thumb are from left to right and the MCP joint, PIP joints and DIP joints of the other four fingers are from top to bottom. It can be seen that the correlation between PIP joints and DIP joints of four fingers except for the thumb was high (dark red). The correlation between MCP joints of the middle and ring fingers was 0.87 (bright red). The correlation between the PIP joint and DIP joint of the ring finger and the PIP joint and DIP joint of the little finger was also high (bright red), indicating that there were obvious correlations among these joints.The PCA was used to extract the postural synergy basis. As we tried to retain 95% of the data, the number of extracted principal components and the variance Five principal components were needed to restore the complete hand motion and these five components constituted the postural synergy basis matrix, while the first three principal components postural synergy basis accounted for more than 83% of the sampling data. This shows that, although the hand joint freedom is high for grasping activities, the motion of the human hand can be potentially reconstructed by using the postural synergy basis with fewer data dimensions.In order to verify the dimension-reduced data, the original gesture could be reconstructed. We used the R-squared coefficient to determine the reconstruction of gesture collaboration basis which is defined asFor different grasping tasks, the first three principal component bases were used to reconstruct the R-squared coefficient of hand motions, as shown in We drew the R-squared coefficient under different actions in In order to realize the cooperative task between patient and manipulator, we input the collected patient motion data into the CNN model for gesture prediction. The PCA dimension of patient motion data could be reduced with the output gestures. Low-dimensional joint data could be used well to control the manipulator to assist the patient in grasping the object. In order to achieve real-time control and repeated rehabilitation training, we designed a simulated manipulator to replace the actual manipulator. Patients could repeat training according to pre-designed gesture movements, which made controlling a simulated manipulator more convenient.Based on the extracted postural synergy basis, the grasping models of the under-actuated manipulator could be established. Then, we defined the grasping symbols with the manipulator and object. Each finger of the manipulator could be regarded as a joint link model. The complete manipulator joint connecting rod model is shown in i-th contact point, the coordinate system of the local object is Let The symbol The symbol Similarly, Let us suppose that the manipulator has When both the object and the manipulator are rigid bodies, the motion constraints at the contact points are determined by the contact model. By selecting matrix Then, the constraint equation of contact point motion isThe under-actuated manipulator can imitate the human hand to accomplish flexible and diverse grasping actions and its structure design is simple. Based on the simulation toolbox SynGrasp  for grasAccording to the CNN’s prediction results, we selected the grasping activities of large-diameter, power-disk and power-sphere objects as training tasks. We carried out grasping planning and optimization for the manipulator. The flow chart of manipulator grasping simulation and planning based on gesture cooperation is shown in The simulation system mainly included a model generation part, a grasping planning part and a grasping solver part. Among them, the model generation part mainly constructed the manipulator model by inputting D–H parameters. The object model is determined by its type, size, spatial position and other parameters. The grasping planning part was responsible for determining the pre-grasping stage and grasping implementation stage after obtaining the manipulator model and object model and used the optimal grasping path to realize the grasping task. The grasping solver part was based on the analysis of the manipulator grasping model to solve the model parameters in the grasping process and improve the solution speed of the system.The complete grasp planning consists of pre-grasping and grasp implementation. In the pre-grasping stage, there is no substantial contact between the manipulator and the object. After obtaining the position of the object relative to the manipulator, the joints move toward the object, but there is no contact point. During grasp implementation, the contact points are calculated to make the grasping stable. In fact, crawl planning can also be regarded as an optimization problem.We selected a special objective function to evaluate the pre-stage grab quality asAfter the pre-grasping stage, the manipulator starts to grasp and the manipulator further approaches the object until the contact point appears. However, when the contact point first appears, the manipulator cannot grasp the object steadily, so it needs to continue to move towards the object until more contact points appear. This stage is called the first stage of real-time grasping and the optimization objective function isIn order to realize the second stage of grasping, we defined a state called force closure, which refers to the force exerted on the object by the manipulator in order to overcome the influence of gravity and other external forces in the grasping process. For example, when we grasp an object to make it stable in the air, this state is force closure. When the grasping force is closed, the first stage of grasping is finished and the second stage starts. The conditions of force closure are as follows:G and It can be obtained, from the static balance equation of the object, thati-th contact point, the contact force is For the We defined a 15-degree of freedom (DOF) under-actuated manipulator in the SynGrasp toolbox and its kinematics model was established by a D-H method, as shown in In order to verify the performance of the CNN network and realize the real-time control of manipulator, hand movement data were collected via the Qualisys motion capture system and the grasping actions were predicted via the CNN predictor. Our system configuration is shown in The subjects were asked to complete three kinds of actions, large diameter, power disk and power sphere, for three objects, cylinder, disk and ball, each of which was repeated three times. The subjects sat on the chair in a relaxed state and the markers on their right hands were set with the configuration shown in Then, we used the predicted motion results to control a robotic manipulator for patient-dominant active hand training. The simulation model of the under-actuated manipulator was established and the performance of CNN-based grasping activities prediction was verified. As shown in The excessively high complexity of the human hand brings great difficulty to the analysis and prediction of hand movements, which makes the control of hand robots difficult. The use of postural synergy basis not only can decompose and simplify grasping activities, but can also decompose complex hand actions into a lower dimension, so as to control the manipulator to complete grasping tasks with less input. Hand motion prediction is the basis for patient-dominant robot-assisted rehabilitation, which can be used to optimize the rehabilitation experience and avoid secondary injury. Common prediction methods include the use of biological signals and image recognition. In , an indiBased on the classification of hand grasping activities, a CNN-based prediction method is proposed in this paper. It is worth mentioning that the types of hand grasping movements were adjusted according to the patients, which is more in line with the daily needs of the patients. Taking the under-actuated manipulator as the control object, the postural synergy basis extracted from hand grasping motions was used to reduce the hand data dimensions, so as to simplify the control input of the manipulator. Li et al.  used a 3For robotic hand control, traditional hand-gesture-based methods are complicated. One of the challenges in robotic grasping is how to coordinate so many finger joints in order to generate an appropriate grasping posture for a specific object . To resoBy extracting the postural synergy basis of hand motions, this paper explores the cooperative relationship in the hand grasping action, simplifies the hand grasping action and reduces the hand data dimensions. Based on the improved CNN predictor, the hand grasping motion category could be predicted with an accuracy up to 94% for the selected activities. A simulated robotic hand manipulator was established and controlled on the postural synergy basis to help patients complete the grasping tasks according to the prediction, so that patient-in-charge assistance could be realized. The experimental results prove that the proposed prediction algorithm could predict the motion intention of the patients and the control method based on postural synergy could help them complete the grasping action easily. In the future, subsequent research will be conducted on the prediction of more hand movements for daily life activities and intelligent robot-assisted rehabilitation. We will further improve the control structure of the manipulator, so that the robot can better assist patients in patient-in-charge training and daily life. Meanwhile, the recognition results of this paper can be used to control patients to perform interactive tasks in a virtual/enhanced environment and to carry out rehabilitation training by repeatedly grasping objects in the virtual scene. It can also be used to control soft rehabilitation gloves to aid daily grasping behavior and complete drinking actions. The proposed method can also be combined with remote rehabilitation to help patients realize prediction and control of hand movement in residential settings.


Single-Photon Avalanche Diodes (SPAD) in Complementary Metal-Oxide Semiconductor (CMOS) technology are potential candidates for future “Light Detection and Ranging” (Lidar) space systems. Among the SPAD performance parameters, the Photon Detection Probability (PDP) is one of the principal parameters. Indeed, this parameter is used to evaluate the SPAD sensitivity, which directly affects the laser power or the telescope diameter of space-borne Lidars. In this work, we developed a model and a simulation method to predict accurately the PDP of CMOS SPAD, based on a combination of measurements to acquire the CMOS process doping profile, Technology Computer-Aided Design (TCAD) simulations, and a Matlab routine. We compare our simulation results with a SPAD designed and processed in CMOS 180 nm technology. Our results show good agreement between PDP predictions and measurements, with a mean error around 18.5%, for wavelength between 450 and 950 nm and for a typical range of excess voltages between 15 and 30% of the breakdown voltage. Due to our SPAD architecture, the high field region is not entirely insulated from the substrate, a comparison between simulations performed with and without the substrate contribution indicates that PDP can be simulated without this latter with a moderate loss of precision, around 4.5 percentage points. Laser remote sensing is remodeling many fields, from Earth observation with Low-Earth Orbit (LEO) satellites to self-driving cars. The working principle is quite simple as it consists of emitting an optical pulse and analyzing the reflected and back-scattered light to probe a scene .SPAD high sensitivity with low photon flux detection capabilities , excelleCMOS SPAD performance prediction is crucial to optimizing SPAD design before manufacture since the CMOS industry workflow is time-consuming and costly. This paper is focused on the Photon Detection Probability (PDP) which is the probability for an incident photon to trigger an avalanche breakdown. Commercial finite element simulator tools (TCAD) accurately simulate electrical and physical parameters as the electric field or leakage current, and its outputs can be translated into SPAD metrics to compute performance parameters as the PDP. Therefore, in the last few years, multiple efforts contributed to improving the state-of-the-art in SPAD performance parameter modeling and simulations. The first attempt we have reported is from Kindt et al. , on a cui et al.  simulateo et al. , using dConcerning standard CMOS processes, Rochas et al.  performeu et al.  and Hsieu et al.  proposedc et al. , Signorec et al. , and Xiec et al. .Doping profile measurement. The main distinction with the previous approach is the use of measured doping profile from commercial standard CMOS processes applied to a structure available in such processes, instead of custom processes in , an abruA commercial device simulator tool (TCAD).Analytical/numerical relations computed using a Matlab routine.In line with these previous studies, we propose a modeling and simulation method based on a combination of:Our approach philosophy is close to Gulinatti et al. , as theyThe model aims at being generic and usable for any SPAD, regardless of its architecture (layout) or size, notably for SPAD envisioned for future space-borne Lidars. This paper is structured as follows: To be detected, the photon must make its path throughout the passivation layer on top of the silicon surface before it reaches the multiplication region of interest and triggers the breakdown. Hence, we broke down the photon detection into two phases: the photon transmission and the avalanche triggering. In the first phase, we define d PDP is ,9,10:(1)The following subsections describe the models used to estimate transmission i. The relation between layers i and i.We implemented the matrix approach  and the Applying the matrix approach to evaluate the electric field and considering the reflection in the silicon is null The theory predicts oscillations on the transmission as several parameters like the phase shift or the angles of incidence are modeled with trigonometric functions. Their frequencies depend upon the phase shift, and increases when the wavelength decreases and/or the layer thickness increases. Figure 12 shows these oscillations on the simulated PDP for be absorbed, creating an electron–hole pair;the resultant particle has to diffuse to the depletion layer without recombination;once the particle reaches the depletion region, it needs to be accelerated by the high electric field, and finally, triggers an avalanche.The Internal Quantum Efficiency, z within the depleted region, triggers an avalanche. For the junction on The section below gives details about each contribution.C axis in The mathematical formalism of C axis represents a typical cross-section of the photosensitive area of the sensor.Our SPAD was circular and exhibits a rotational symmetry, hence the z is:Within the first region corresponding to the N undepleted layer, photons have to be absorbed, and then the generated hole must diffuse to the upper limit The probability for a hole to reach the upper limit T, the elementary charge q and the Boltzmann constant k:The diffusion coefficient We define Once the hole arrives in the depletion region, it must drift inside the high electron field and trigger an avalanche, consequently:z in the depleted region to trigger an avalanche breakdown. The modeling of this probability is given in Similarly, we obtain With An additional contribution has to be taken into account when the high field region is not entirely insulated from the substrate, which is the case of The introduction of the parameter Inside the depleted region, a photon is absorbed and generates an electron-hole pair with the probability The avalanche breakdown probability is an essential parameter used for each performance parameter calculation. The TCAD tool available to us does not implement its calculation, consequently we use a Matlab routine. The relations describing avalanche triggering depend on the chosen model. In this paper, we employ the Oldham and McIntyre local model ,21 with z inside the depletion region, triggers an avalanche breakdown:z inside the depletion region to trigger an avalanche. In the local model, these probabilities are related to the following system of equations:quations:dPpedz=1−To acquire the doping profile, the measurement technique used is the Secondary Ion Mass Spectrometry (SIMS measurement) performed on a dedicated structure . These md, the silicon and oxide material refractive index The transmission probability C showed in carriers mobility ti model ,24 for mti model , is alsoti model ;carrier lifetimes pendence . Carrierpendence . Neverthpendence . More dependence ;F and the depletion layer width the electric field  profile .The PDP calculation is executed alongside the vertical axis T, the bias voltage These electrical data are then injected in the Matlab routine. The program evaluates the diffusion length For the purposes of this study it is assumed that PDP does not substantially change with device area and that understanding the central PDP is meaningful.Feasibility studies performed by Airbus DS suggest that large SPAD should be used for their future projects of spaceborne Lidars: between 100 and 200 Provided the SPAD design allows us to avoid phenomena such as optical diffraction effects for very small active areas and/or guard rings merge leading to a lower electric field and/or higher breakdown voltage, the model should be applicable to simulate the various set of SPAD dimensions envisioned for spaceborne Lidar. Further measurements will have to be performed on various SPAD sizes, to confirm this statement. However, other works confirm this trend, as in , where tOnce the simulations are completed, we compared the simulated and measured PDP to validate the model, as presented in We developed a CMOS SPAD in The high electric field is related to the junction between the N+ implant and the P-well. This region must be photosensitive as it is the active area. With the doping profile measurements and TCAD simulations we performed, no implant were available to us to insulate the high field region without creating a PEB or other issues. In consequence, the main drawback of our structure lies in the incomplete high field region insulation from the substrate. To lower the number of parasitic charges captured by the high electric field, we add an N-well insulation ring. This insulation method is not perfect, as only the charges passing in its neighborhood are captured and deeper parasitic substrate charges can diffuse to the high electric field. Metal layers are used to shield the non-photosensitive areas and enable the reduction of the number of photo-generated charges inside the substrate.to avoid premature edge breakdown;to avoid diode and insulation ring depletion layer merging.Our SPAD has a circular shape of The driving electronic to quench the avalanche and shape the signal is composed of a resistor (passive quenching) and a buffer co-integrated with the SPAD.The PDP measurements are achieved with excess voltages between 15 and 30% of the breakdown voltage. We illuminate the SPAD in flat field with a wide-band light source with halogen-tungsten lamp, coupled with an integrating sphere Gooch and Housego—OL Series 462. Interference filters enable the wavelength selection, between 450 and We performed around 3000 oscilloscope acquisitions of 1 ms each, to obtain the The dark count rate and afterpulses previously measured for the detector are removed from terature ,31,32,33silicon nitride silicon dioxide For this study, the passivation layer thickness provided by the foundry is used. These layers consist of:In the following sections, the excess voltage ficients .The measured breakdown voltage of C.The model was applied on the cross-section presented The electric field is illustrated in this zone is longer than Top and Dep regions and more charges can diffuse to the depleted region to trigger an avalanche;the electrons generate the breakdown in this region. Electrons have a better mobility and avalanche triggering probability than holes.Moreover, The potential barrier between Bot and substrate regions, make the task more difficult for electrons to reach the depleted region. As a consequence, only a small fraction of electrons generated within the substrate, modeled by the PDP exhibits oscillations predicted by the thin-film theory presented As presented in The PDP can be simulated without the substrate contribution with a mean error of 18.5% and the introduction of a fitting parameter to adjust the substrate contribution improves the precision around 4.5%. This result shows that the PDP can be simulated without the substrate contribution, with a moderate loss of precision, as only a small fraction of charges from the substrate, from 5% to 15% depending on the bias voltage, can cross the potential barrier between P-well and substrate.As the use of the fitting parameter would make the model specific to a given type of SPAD, the simulation without the substrate contribution can be used as a predictive tool to simulate PDP for a broad range of excess voltage, with a moderate loss of precision.On the whole, we exhibit good predictions, especially for excess voltages greater than 20% of the breakdown voltage, which are the bias condition values usually used for SPAD.For the last data point, the measured PDP goes down while the excess voltage increases over 25%, we did not identify any reason for this as it should slightly increase as the simulated one. However, this decrease is small and could be related to slight errors in the measurements.the FWHM of the light intensity received by the sensor, presented in the slight dispersion on the breakdown voltage measurement discussed in Other uncertainties can influence the characterization results:doping profiles measurement;the Inter-Layer-Dielectric (ILD) of silicon dioxide incidence angle used for simulations is Uncertainties can also affect the simulation:These uncertainties are identified and remain relatively small, but provide a first indication for the mean error.The local model was chosen, with a specific set of ionization coefficient, based on the excellent correlation between voltage breakdown measurements and simulation results. Although, it may not be the case for all CMOS processes and non-local model investigation could also be an alternative when breakdown voltages diverge too much.In this paper, we develop a model to predict CMOS SPAD Photon Detection Probability PDP accurately.We designed our SPAD to validate the model predictions. Particular attention was given to reduce parasitic capacitances, reduce recharge time, and avoid premature edge breakdown PEB, to enable PDP measurements.A comparison between the simulation results indicates that the PDP can be simulated without the substrate contribution with a moderate loss of precision. We demonstrate a mean error of 18.5%, without the substrate contribution, for a typical range of excess voltages between 15% and 30% of the breakdown voltage.We also have some identified uncertainties related to the thin-film on top of the silicon, doping profile, or even implant depth, which are indications to explain the mean error.The local model for ionization coefficients is used in our study according to the good correlation between our breakdown voltage measurements and simulations.We think that this work can help future designers to improve the conception method of future SPAD as factors important to PDP are identified and modeled.