The detection of psychological stress using the electrocardiogram (ECG) signal is most commonly based on the detection of the R peak—the most prominent part of the ECG waveform—and the heart rate variability (HRV) measurements derived from it. For stress detection algorithms focused on short-duration time windows, there is potential benefit in including HRV features derived from the detection of smaller peaks within the ECG waveform: the P, Q, S, and T waves. However, the potential drawback of using these small peaks is their smaller magnitude and subsequent susceptibility to noise, making them more difficult to reliably detect. In this work, we demonstrate the potential benefits of including smaller waves within binary stress classification using a pre-existing data set of ECG recordings from 57 participants (aged 18–40) with a self-reported fear of spiders during exposure to videos of spiders. We also present an analysis of the performance of an automated peak detection algorithm and the reliability of detection for each of the smaller parts of the ECG waveform. We compared two models, one with only R peak features and one with small peak features. They were similar in precision, recall, F1, area under ROC curve (AUC), and accuracy, with the greatest differences less than the standard deviations of each metric. There was a significant difference in the Akaike Information Criterion (AIC), which represented the information loss of the model. The inclusion of novel small peak features made the model  Psychological stress causes changes in the autonomic nervous system that can be observed in the electrocardiogram (ECG) signal. The R peak—the most prominent feature in the ECG signal, shown in Many studies have explored the utility of the ECG signal for psychological stress detection and have shown that traditional R-peak-based HRV features can reliably indicate stressed state ,2. To thThis more scientific exploration of small peak utility must be accompanied by an understanding of the operational utility of these features. As previously mentioned, the development of wearable ECG-monitoring systems is increasing rapidly, but wearable systems are plagued with noise due to poor electrode contact and motion artifacts . For thiThe research questions we address in this work are threefold: (1) Do features derived from the intervals between the P, Q, S, and T waves within the ECG waveform augment binary stress classification? (2) Can we characterize the correlations between the novel and traditional features to improve our understanding of the influence of the novel features? (3) What is the impact of noise on the detectability of these smaller parts of the ECG waveform? Our goal is to understand both the efficacy of adding these features to stress detection algorithms and the practicality of their use in operational environments where noise is ubiquitous.The paper is organized as follows: Each of the stress classification models was developed by down-selecting time-domain ECG features that were extracted from a pre-existing data set, assessing validity coefficients for each feature, inspecting correlations between the features, and ultimately assessing model performance with three widely used classification algorithms. Outside of model development, an assessment of automated peak detection performance was also conducted to understand the potential operational utility of small peak features. The TFM and NFM were both developed using the set of traditional features, but the NFM was also given the full set of potential novel features before the model down-selection, while the TFM was not. The NFM was not based purely on small peak features but instead on a combination of traditional and novel features. The following sections step through these development procedures in more detail for both the TFM and NFM and the methods used to evaluate their performance.The data set used for this study is a data set publicly available on PhysioNet, a database of physiological signal data sets managed by the National Institutes of Health . This dath- order Butterworth filter with a normalized cutoff frequency of 0.3 and subsequently split into 10 s samples, one from each stressed and nonstressed period in the data set. We selected sample durations of 10 s to accommodate the shortest intervals between video clips. These short time intervals were also relevant for psychological stress detection in real time. Peak detection was conducted using the “findpeaks” and “delineate” functions based on the Pan–Tompkins algorithm for peak detection available in the neurokit2 Python package [Each ECG recording was filtered using a 5 package ,13. NoveThis section provides an explanation of each of the features included in both models. There were a total of 24 R-peak-based features and a total of 12 small-peak-based features for each of the models to down-select from. In this paper, we have only included the features that were ultimately selected by the two models in this explanation. Each feature is described in The same method of feature selection was used for both the TFM and the NFM. We split the 960 samples into a training set of 67% and a testing set of 33%. The full set of available features for both models was reduced using a forward and backward stepwise feature selection algorithm with the Akaike Information Criterion (AIC) as the evaluation metric. The smaller the AIC value, the less information loss in the model, so at each step the feature with the highest AIC value was removed from the set.Validity coefficients were computed for each feature by calculating the point-biserial correlation, We evaluated three different classification methods with both the TFM and NFM. These three methods were linear discriminant analysis (LDA), logistic regression (LR), and a support vector classification (SVC). These formulations are different in their approaches, and the goal with this investigation was to see how well the two feature sets performed under different, but commonly used, classification algorithms. LDA is based on least squares regression, whereas LR is based on maximum likelihood estimation. LDA assumes normality and homoscedasticity in the data set under the two conditions , while LR does not. SVC is a maximum margin classifier that maximizes the distance between each feature and the hyperplane decision boundary. Explainability is highest in the LDA and LR models and lowest in the SVC model. We compared the performance of these three classification algorithms in precision, recall, F1, AUC, and accuracy with both the TFM and NFM. Each of these metrics is presented as an average value from 10-fold cross-validation conducted on the reserved test data.If the small peak features show promise for stress detection, the capabilities of automated peak detection algorithms must be evaluated to ensure operational feasibility. It is time-consuming to identify features through visual inspection, so the smaller parts of the ECG waveform must be detectable by unsupervised peak detection algorithms in order to be effectively incorporated into stress detection algorithms that function without constant supervision. To assess the detectability of each feature, we randomly selected 10 of the 57 participants’ data sets to assess precision, recall, and F1 scores for the neurokit2 peak detection function when compared with peak detection through visual inspection. The peak detection algorithm begins by calculating the gradient of the ECG signal to identify the location of the QRS complexes within the signal and identifies each of the peaks by calculating local maxima within that region.The TFM consists of nine features down-selected from the 24 traditional features derived from R-R interval values. The features present in this model are RMSSD, MeanNN, CVSD, MedianNN, IQRNN, pNN50, pNN20, TINN, and HTI. In this set of features, CVSD has the highest coefficient in the model, and MeanNN and MedianNN have the strongest correlation with stress. Each of these metrics displays a negative correlation with stress, meaning that lower values for each feature are associated with greater stress. The results are summarized in The NFM consists of 13 features down-selected from the 36 features drawn from R-R-interval-based HRV metrics and small-peak-based metrics. The features present in this model are SDNN, MadNN, MCVNN, IQRNN, pNN50, pNN20, TINN, PR mean, PR sd, ST mean, ST sd, PT mean, and PS sd. Four of the traditional HRV features overlap with those selected in the TFM: IQRNN, pNN50, pNN20, and TINN. None of the novel features derived from the Q peak contributed positively to the AIC metric and none were selected as part of the final NFM. As with the features in the TFM, the features included in the NFM are negatively correlated with stress level. These results are shown side-by-side with the TFM results in The top of The bottom of In addition to inspecting the correlation coefficients between the features, we calculated the validity coefficients between each feature and stress. The coefficients for each feature are shown in nentials . This caThe distributions of precision, recall, and F1 scores for each part of the ECG waveform are shown in The results of this study suggest that the inclusion of small peak features in stress classification algorithms can reduce information loss, potentially increasing the predictive power of these algorithms, even with short ECG samples. We can see this result in the comparison of the TFM with only R-peak-based features and the NFM with both R-peak- and small-peak-based features.The NFM performs slightly better than the TFM in the commonly used precision, recall, F1, AUC, and accuracy metrics, but they are not statistically significantly improved, given that the differences are within the standard deviations of each value. The differences between the two models in these metrics are largest with LR classification, and they perform almost identically with the SVC. These results do not show conclusively that the inclusion of small peak features augments the binary classification in this case, but they do suggest that these features can be used effectively in conjunction with traditional HRV features. However, in the AIC metric comparison, the NFM does perform better than the TFM. The NFM has a substantially better AIC score than the TFM, showing that the NFM is plicity) . This diMayya et al., 2015, explored the efficacy of time domain, frequency domain, and nonlinear heart rate variability (HRV) features over 60 s test intervals and found that RMSSD was the most influential feature for determining stressed state when model performance was measured using classification accuracy . In our The This, the difference in information loss between the two models, the reduction in multicollinearity, higher validity coefficients, and the inclusion of small peak features over other commonly included features, suggest that the small peak intervals could be indicative of somewhat independent processes within the heart that reflect the reaction of the autonomic nervous system to stress as seen in the ECG signal.The novel features included in the NFM were all derived from the P, S, and T peaks. These peaks reflect known physiological processes, but they have not been studied deeply as psychophysiological correlates of stress. The duration of the P-R interval is dependent upon signal conduction through the atria, atrioventricular node, His bundle, and Purkinje fibers . This coFrom an operational standpoint, the S and T peaks are most reliably detected with an automated detection algorithm, making them the most reliable for inclusion in stress classification models. The Q peak is least reliably detected, due to its smaller magnitude in comparison with the other peaks. The intervals associated with the Q peak were not included in the NFM, so this result does not negatively impact the potential for operational use of the NFM.In this investigation, we made the assumption that exposing spider-fearful individuals to videos of spiders would create a psychophysiological response shown in the ECG signal similar to responses indicative of psychological stress. In the study conducted by Ihmig et al., HRV metrics were extracted for their correlation with “disturbance and emotional arousal”, so we believe that our assumption is valid . AdditioFuture work may include the validation of the TFM and NFM on other data sets. Testing with other data sets involving other types of psychological stress  could add to our depth of understanding in the utility of small peak features in this paradigm. Additionally, we plan to investigate the utility of small peaks for nonbinary stress classification, determining whether their inclusion augments the detection of multiple levels of stress. Another important area of future work is the validation of the NFM with different types of stress  and the investigation of whether small peaks contribute to the classification of different types of stress.In parallel with further assessments of the utility of the NFM in different experimental scenarios with nonbinary stress classification, an investigation of novel peak detection algorithms could expand the potential for use in the NFM. There are several groups working to develop peak detection algorithms that amplify the small peaks within the ECG waveform, and leveraging their work in this area is an important next step. In the future, it may be valuable to conduct a study of the efficacy of adding small peaks in conjunction with the use of such an algorithm.This investigation focused on the benefit and utility of incorporating features derived from small peaks within the ECG waveform for binary stress classification. To the best of our knowledge, these features have not been investigated in depth for stress detection, so we hope the results of this study contribute renewed interest in their investigation as wearable sensor systems develop and become more capable of detecting small ECG peaks reliably. We found that the NFM performed slightly better than the TFM in the commonly used metrics of precision, recall, F1, AUC, and accuracy, but not statistically significantly so. Alternatively, we found that the NFM exhibited a better AIC score than the TFM, indicating that the NFM has less information loss than the TFM. This difference could be due to the higher validity coefficients between the small peak features and most of the traditional HRV features and the lower correlations between these features, potentially reducing multicollinearity within the model. These results show that the incorporation of these features into future psychological stress detection algorithms could augment these algorithms’ capabilities. As to the question of operational feasibility, we found that the Q peak was least robust to noise, so its use is not feasible for stress detection outside of the laboratory environment; however, the Q peak was not included in the NFM, so it does not show promise for stress detection and its poor detectability is not of major concern.


A rise in the reported numbers of children with Autism Spectrum Disorder (ASD) highlights the need for dental practitioners to be more familiar with the treatment approaches for these special needs children to ensure comfortable, well-accepted and efficient management while in dental office.This paper aimed to acquire a deeper understanding of some of the innovative and best approaches to managing children with ASD in dental settings.A systematic literature search was performed in PubMed, Scopus, Web of Science, Cochrane databases, and grey literature based on the PRISMA 2020 statement, using main keywords such as: ‘management’, ‘dental’, ‘children’, and ‘Autism Spectrum Disorder’. Original full-text papers including randomised controlled trials (RCT) and all other designs of non-randomised controlled studies (NRS) reporting relevant intervention studies in English were included without any publication time limit. The quality of the evidence found eligible for the review were then assessed using the ROB-2 and ROBINS-I tools. Subsequently, the details of management interventions and impact of treatment approaches were compared and discussed.Out of the 204 articles found, 109 unrelated articles were excluded during the initial screening. The full papers of remaining 28 were retrieved and only 15 (7%) articles were eligible to be reviewed; eight RCTs with ‘some concerns’ and ‘high risk’ categories particularly concerning their randomisation design, and seven NSRs with ‘serious’ to ‘critical’ bias largely due to confounding factors.Our review found inconclusive evidence on the strength of recent psychological and non-pharmacological approaches used to manage children with ASD in dental settings. Small sample size and lack of a control group in certain studies affected the strength of evidence and credibility of the findings. Nevertheless, this review shared informative details on some innovative approaches for better understanding of the management of children with ASD for dental professionals. Explores deeper knowledge and understanding of psychological approach for managing children with ASD in a clinical dental setting.Highlight the impact of such intervention on dental anxiety, the level of children’s cooperation, and the success of the implementation of dental procedures, which will help the dentists to meet and treat children with ASD according to their individual needs.Discuss the evidence in favour of the use of behaviour management in reducing anxiety and enhancement of cooperation in children with ASD at the dental setting.Children with autism spectrum disorders (ASD) commonly face anxiety and fear when undergoing dental treatment, as manifested via difficult behaviours and uncooperative reactions , 2. The Communication between the child and dental team in clinic can be very difficult or restricted  if thereSo far, the effectiveness of more recent pharmacological and psychological  strategies to improve the dental management of children with ASD has not been reported systematically and are not well known to most dental professionals. Therefore, this systematic literature review aimed to evaluate the effectiveness of available reported behaviour management and modification strategies for children with ASD to overcome the anxiety and discomfort associated with the treatment in dental clinics. This review may provide the necessary evidence for clinical guidelines on the management of dental anxiety, the acceptance, success rates, and impact of each approach with the aim of improving the oral health status and wellness of the children.This systematic literature review was conducted in compliance with the “Preferred Reporting Items for Systematic Reviews and Meta-Analysis” (PRISMA 2020 statement). It is registered under the “International Prospective Register of Systematic Reviews” (CRD42021273415), and received approval for conduct by the research ethics committee (UKM PPI/111/8/JEP-2020-757).The PICO strategy was utilised in answering the research questions: What is the impact of special techniques in dental management for children with autism spectrum disorder on their cooperation while undergoing treatment in dental clinic? The study population (P) of interest was children with ASD within the range of 2–18 years old who were receiving interventions (I) including special dental management techniques in the dental setting as well as other intervention aimed at improving the success and cooperation of children while receiving dental treatment. The results from this survey were compared (C) with healthy children, children with any other disabilities, or another ASD group receiving other intervention(s). The expected outcome (O) from the intervention was the improvement in cooperation during dental procedures as rated by dental professionals or caregivers, improvement in the behaviour scale, and a decreased level of anxiety.https://meshb.nlm.nih.gov/search) was also used to identify words and phrases from articles of interest  AND (child*) AND ("Autism Spectrum Disorder" OR ASD OR autism OR "Asperger syndrome") AND . The Medical Subject Headings, MeSH  or all designs of non-randomised controlled study (NRS), i.e. non-RCT, interventional study, studies with comparative groups, interrupted time series study, cohort study, controlled before-and-after study, and case series . Furthermore, the full-text article must be written in the English language and report the impact of the intervention in the form of behaviour scales or cooperation rate. Studies that focused only on the perceptions and concerns of the caregivers or those with insufficient information on the outcome were excluded from the review.The articles obtained from the search were exported into Microsoft Excel. The list of articles was screened for replicates and their relevance to the study title. Any duplicates or non-ASD-related articles were rejected. Two researchers (MS and SE) screened the titles and abstracts of all the retrieved full-text articles to filter out those that were not relevant to the research question. If there was some disagreement on the relevance of the articles between the two researchers, it would be resolved through discussion with the other three reviewers .For each of the included articles, the following information was obtained: general characteristics , the sample size of subjects, comparative groups, assessment tools used in the study, dental procedures done in each study, type of management or techniques as intervention, outcome measures , and lastly key findings.The reviewers assessed the risk of bias of the included studies independently. Studies with NRS designs were evaluated using the ROBINS-I “Risk Of Bias In Non-randomised Studies-of Interventions” and the studies were rated with the same coding of the data extraction process. The seven domains of ROBINS-I assessed are risk of bias arising from  respectively. In addition, the bias of the RCT studies was evaluated using version 2 of the Cochrane Risk-of-bias tool for randomised trials (ROB-2) and the data in the table were generated using the Excel tool provided by the same team. The five domains of ROB-2 assessed are risk of bias arising from  respectively. Criteria for reaching the overall judgements for studies included in both (ROB-2 or ROBINS-I) tools were performed in compliance with the guidelines for each tool , 17. MeaFinal search date was 1st January 2022. The initial search retrieved 202 papers from four databases; 65 were found to be duplicates. One hundred and nine papers were excluded due to the irrelevance of titles and/ or abstracts . Fifteen were excluded based on full-text ratings . Additionally, two papers were added scanning the references lists of eligible papers. The step-by-step search and selection strategy is shown in Fig. Of the 15 articles selected, 8 were RCT –26 and 7In most studies, the cooperation of children during dental assessment was the most frequent tool used to assess the impact of the approach used , 29, 30,In this systematic review, the main outcome was determined by the improvement in the child’s cooperation during dental procedures as rated by dental professionals or caregivers. Another main outcome was the improvement in the behaviour and decrease in the anxiety level of the children in the dental setting. Accordingly, the measures of effect for the outcomes reported in the studies were the increase in the success rate or completion of dental procedure, i.e., the increase in the number of components achieved in a dental visit, and/ or improvement on the behaviour rating scales.All the approaches were evaluated according to the planned procedure. Most of the studies focused on the clinical oral assessment and examination as main dental procedures to be assessed –31, 33. A variety of approaches have been proposed to improve the management of children with ASD. So far, visual pedagogy appeared as the most common approach . It can The characteristics of the studies were assessed individually to evaluate the outcomes and effects of the interventions using the specific tools based on the study design Table .The reviewers assessed the quality of the eight RCTs using Version 2 of ROB-2 –26  and behavioural level of the children, as manifested by an improvement in children’s cooperation during the dental examination , 31, espNext, the positive reinforcements supported by TSD also showed an improvement in cooperation during dental examination compared to negative reinforcements . FinallyFor the NRSI, it was rare for the overall judgement of bias to be low due to confounding. For this review, we accepted the outcomes at all levels from all the included papers, unless the paper did not show sufficient ability to produce a valid conclusion.There are several limitations to this study. Most of the included studies had a small sample size hence may not be able to fully demonstrate the optimal benefit of specific behavioural strategies on the children from compared groups. Furthermore, some studies lacked control groups. Qualitative assessment could also benefit from the studies in addition to qualitative parameters measured to provide in-depth response on behavioural modification effects –39.This systematic review provided current available approaches yet inconclusive evidence on the effectiveness of the psychological approach for managing children with ASD at dental setting. Although the impact of the approach on the management of dental anxiety, the level of children’s cooperation, and the success of the implementation of dental procedures was reported, the study design of these behavioural modification techniques requires better randomisation and bias control to suggest effectiveness of intervention.


Distributed fiber-optic sensing (DFOS) technologies have been used for decades to detect damage in infrastructure. One recent DFOS technology, Optical Frequency Domain Reflectometry (OFDR), has attracted attention from the structural engineering community because its high spatial resolution and refined accuracy could enable new monitoring possibilities and new insight regarding the behavior of reinforced concrete (RC) structures. The current research project explores the ability and potential of OFDR to measure distributed strain in RC structures through laboratory tests on an innovative beam–column connection, in which a partial slot joint was introduced between the beam and the column to control damage. In the test specimen, fiber-optic cables were embedded in both the steel reinforcement and concrete. The specimen was tested under quasi-static cyclic loading with increasing displacement demand at the structural laboratory of the Pacific Earthquake Engineering Research (PEER) Center of UC Berkeley. Different types of fiber-optic cables were embedded both in the concrete and the rebar. The influence of the cable coating and cable position are discussed. The DFOS results are compared with traditional measurements (DIC and LVDT). The high resolution of DFOS at small deformations provides new insights regarding the mechanical behavior of the slotted RC beam–column connection, including direct measurement of beam curvature, rebar deformation, and slot opening and closing. A major contribution of this work is the quantification of the performance and limitations of the DFOS system under large cyclic strains. Performance is quantified in terms of non-valid points , maximum strain that can be reliably measured, crack width that causes cable rupture, and the effect of the cable coating on the measurements. Structural damage indices are also proposed based on the DFOS results. These damage indices correlate reasonably well with the maximum sustained drift, indicating the potential of using DFOS for RC structural damage assessment. The experimental data set is made publicly available. After a disastrous event , cities are burdened with the need to quickly evaluate infrastructure damage. For modern cities, the indirect cost from business interruption often exceeds the cost from structural damage. Current evaluation procedures for structural damage still largely rely on the judgment of civil engineers, which is expensive and time-consuming. Meanwhile, in recent years, various types of infrastructure-sensing technologies have been developed, such as terrestrial laser scanning , unmanneDamage quantification for reinforced concrete (RC) structures require a deep understanding of the mechanical behavior of the structural component. One emerging DFOS technology, Optical Frequency Domain Reflectometry (OFDR), is especially suitable for this task due to its unprecedented spatial resolution (around 1 mm) and high accuracy  invehe world ,16) on tA beam–column connection was selected for investigation due to its significant influence on the strength and overall stability of RC-framed structures, as observed in past earthquakes  introduced between the beam and the column to control damage. The slotted beam connection system experiences minimal beam elongation and achieves a non-tearing action under seismic loading ,24. The This paper starts with introducing the test set-up and fiber layout . The DFOThe dimensions and reinforcement details of the tested specimen are shown in The layout of the optical cables is presented in In this section, the DFOS results at the following representative drift levels are presented and discussed: ±0.14%, ±0.95%, and ±1.76%. For comparison, the DIC results at corresponding drift levels are also presented, which are obtained through the free software Optecal. While this section discusses three representative drift levels, the DFOS results at all drift levels are organized in a Matlab app which is publicly shared.At the drift level ± 0.14%, no visible damage or strain information can be obtained from DIC . HoweverTo investigate the influence of cyclic loading on the DFOS measurements, two time instants were distinguished for the same drift level . The two different time instants are indicated by vertical lines in the bottom subfigures in From top to bottom along the beam, the gradual increase in strain in the optical fibers indicates increasing curvature along the beam caused by increasing moment. The asymmetric behavior of the beam is clear when comparing the maximum tension under loading in different directions . Strain measurement using the PVC-coated cable was only possible up to the slot location in the beam. The cable must have been damaged at the slot, and no signal was obtained beyond the slot. Meanwhile, strain measurement using the Silicon-PFA-coated fiber was still possible. This was partially caused by the different directions of the cable layout . StartinComplete loss of data after the first cycle was also observed for the PVC-coated cable, which is not surprising since less debonding between the PVC cable and the concrete was observed in the previous section. The resulting large strains damaged the cable completely. Meanwhile, only a partial loss of data was observed for the Silicon-PFA cable. Some data were obtained “beyond” the slot for both the first and third cycles . Some deFor the fiber embedded in the rebar (polyimide), a deterioration in data quality could also be observed in the form of missing  data points along the length of the fiber-optic cable but particularly across the slot for both loading directions. The high strains and strain gradients limited the ability of the analyzer to detect peaks in the frequency spectra. Note that from the LVDT results, the maximum opening of the slot was 3.20 mm for a drift level of −1.76%, which can be used to infer a strain limit for the PVC- and Silicon-PFA-coated cables. A more detailed investigation on the data quality is presented in It is worth noting that, even with a significant amount of damage at a drift level of ±1.76%, the strain for the Silicon-PFA cable was still similar to the rebar strain (polyimide) on the left side of the beam . On the From the DFOS results in the column, the difference in strain distributions for the two loading directions was also more evident at a drift level of ±1.76%. More specifically, negative drift (loading from right to left) caused a larger strain in the column compared with positive drift (loading from left to right). Because the reinforcement in the column was symmetric, this indicates a stronger stiffness of the beam under negative drift, which is expected due to the asymmetry of reinforcement.This subsection investigates the data quality of the DFOS results in more detail. As shown in x-axis in the figure is set so that the laser direction is continuous . From Fx-axis) was, again, different  was adopted here to achieve simultaneous interpolation both in the temporary and spatial domains. At a drift level of 1.76%, the strain before and after 2D interpolation to fill in the missing data are shown in After signal reconstruction, it was possible to integrate the strain in the fiber-optic cable near the slot to obtain the slot opening and closing. The previous sections presented the DFOS results at selected drift levels and focused on explaining the mechanical behavior and quality of the DFOS results. This section investigates the feasibility of using DFOS to quantify structural damage, which would be useful for post-earthquake structural evaluation. As a first study, several indices are proposed based on the DFOS results, and their correlations with the maximum sustained drift of the specimen are investigated. The maximum sustained drift is a typical indicator of damage severity .While an innovative RC joint was tested in this paper, to make the discussion more representative for common RC structures, only the information on the left side of the beam : the loading points when the force first returns to zero after reaching “positive peaks”. Note that this is essentially the residual drift after unloading from a positive peak;Zero force (neg): the loading points when the force first returns to zero after reaching “negative peaks”. Note that this is essentially the residual drift after unloading from a negative peak.For each loading cycle, four different characteristic loading instants were investigated  the measurement frequency of the DFOS technology adopted might not be high enough for real time evaluation , (2) the DFOS analyzer might not be available or in operation during the earthquake, and measurements are more likely to be taken after the earthquake event. In these cases, only the residual DFOS results would be available  from pos peaks  were damaged due to high strains at the slot before a drift level of −1.76% was achieved. Silicon-PFA revealed more information about the strain distribution thanks to the layout direction of the cable, which indicates the importance of the DFOS design, as well as the coating (cable jacket) material that caused the cable to partially debond from the concrete before being completely damaged. The polyimide cable (embedded in rebar) successfully recorded data for larger drift levels. It started to lose a significant amount of data at a drift level of −3.5%, which corresponds to a maximum slot opening of 6.15 mm. With moderate damage , the difference between the strain measured in the concrete and the rebar was small on the left side of the beam (without the slot), which is representative of typical RC members. With the appropriate cable coating, this suggests the possibility of sensing rebar deformation without embedding optical cable in the rebar, at least until high levels of damage are reached.Three different damage indices based on DFOS were proposed and evaluated: integration of strain , integration of the absolute strain , and an approximation of the plastic hinge length (PHL). These indices were evaluated both at the maximum and residual displacements. The integration of the absolute strain at the residual displacement correlated well with the maximum drift sustained by the beam–column joint. Though preliminary, this suggests the possibility of evaluating the maximum sustained damage using the residual strain obtained from DFOS measurements after an earthquake.The aim of the present study was to evaluate the capability of DFOS in sensing the deformation and damage of an RC beam–column joint subjected to cyclic loading. The conclusions are summarized as follows:


Cerebrovascular accidents have physical, cognitive and emotional effects. During rehabilitation, the main focus is placed on motor recovery, yet the patient’s emotional state should also be considered. For this reason, validating robotic rehabilitation systems should not only focus on their effectiveness related to the physical recovery but also on the patient’s emotional response. A case series study has been conducted with five stroke patients to assess their emotional response towards therapies using RobHand, a robotic hand rehabilitation platform. Emotional state was evaluated in three dimensions  using a computer-based Self-Assessment Manikin (SAM) test. It was verified that the emotions induced by the RobHand platform were successfully distributed in the three-dimensional emotional space. The increase in dominance and the decrease in arousal during sessions reflects that patients had become familiar with the rehabilitation platform, resulting in an increased feeling of control and finding the platform less attractive. The results also reflect that patients found a therapy based on a virtual environment with a realistic scenario more pleasant and attractive. The possible implications of stroke are not only physical and motor limitations but also cognitive impairment and emotional and social consequences for both the patients and their families. Stroke rehabilitation has mainly focused on the importance of physical recovery, emphasizing the relevance of motor rehabilitation to enable patients to perform daily living activities. Regarding physical recovery as the principal outcome of stroke rehabilitation has led to neglecting the emotional and social negative effects of stroke. Many patients, regardless of their grade of physical recovery, suffer from emotional disorders, such as anxiety, agoraphobias and depression .There is evidence to show that emotional states such as stress, discomfort, boredom or lack of motivation can have an impact on motor learning ,3,4. AnxHowever, there is no clear evidence of how emotions, feelings and attitudes influence functional recovery after suffering a stroke . DespiteDuring traditional rehabilitation, a major role of the healthcare professional is to maintain the patients’ motivation while undertaking the required physical exercises to recover the motor function of the impaired limbs. An aging population results in an increase in the number of stroke survivals with severe disabilities ,14, whilOne of the great benefits of robotic rehabilitation platforms is allowing the healthcare professional to be simultaneously supervising several patients in the so-called robotic gyms. However, this also has the disadvantage of preventing them from spending enough time with patients, as they do during traditional rehabilitation. Thus, healthcare professionals are forced to neglect the emotional care of the patients, which requires time and individual attention.The evaluation of motor function by using widely known functional tests, such as the Fugl-Meyer Assessment (FMA) or the Action Research Arm Test (ARAT), is essential for determining stroke severity, describing motor recovery and planning treatment. On the other hand, the emotional state of patients typically receives less attention during post-stroke rehabilitation ,16. We cThere are two main approaches for describing emotions: (1) a discrete approach of a universal set of basic emotions  and (2) Emotions can be reported using three dimensions: arousal (sleepy–focused), valence (miserable–happy) and dominance (controlling–controlled) . This thEmotions are accompanied by a set of somatic responses associated with the autonomic nervous system (ANS) activity and can be determined directly or indirectly. The Self-Assessment Manikin (SAM) test  is used After successfully testing the RobHand (Robot for Hand Rehabilitation) with healthy subjects in terms of usability, comfort and finger kinematics ,35, we aA study was performed at the Hospital Clínico Universitario de Valladolid (HCUV), Valladolid, Spain, in order to test and validate the usability of the platform under the passive training modality of RobHand and to evaluate the emotional response of patients when exercising using it.A total of five patients were involved in the study . The medThe inclusion criteria were: patients with a diagnosis of cerebral infarction or spontaneous intracerebral hemorrhage that causes significant, but not complete, disabling paresis of an upper limb as a sequel and people with stroke in the chronic and stabilized phase from the ischemic/hemorrhagic event. The exclusion criteria were: patients with zero mobility and cognitive impairment prior to stroke were excluded.RobHand is a rehabilitation platform for performing passive and active assisted training exercises involving the flexion and extension the metacarpophalangeal (MCP) and proximal interphalangeal (PIP) joints of each hand finger. It is composed of a hand exoskeleton and a software environment.The RobHand exoskeleton  is basedThe passive training modality is based on programmed rehabilitation exercises, which involve the repetition of finger flexion and extension movements at three predefined velocities of the MCP joint: low (20°/s), medium (25°/s) and high (35°/s). The types of movement-based exercises available are: (a) flexion and extension of the five hand fingers simultaneously; (b) fingers opening and closing: flexion and extension of hand fingers individually; and (c) flexion and extension of the thumb against index finger (precision grip) or against the four fingers (pinch grip).obj) and speed (vobj) to the real-time controller (L2) during the duration of the exercise according to the introduced parameters. The robot moves the hand fingers to the target position at the specified speed by an open-loop position control implemented in L2. The position controller calculates the control signal (u), which determines the pulse width that the ePWM module must generate in order to move the actuator to the target position at the chosen speed.The open-loop control schema for the passive training is shown in In the passive training modality, the hand is continuously moved. During passive movements, patients are instructed to relax their hand, while the motors move their fingers with a comfortable ROM and speed selected by the healthcare professional according to the patient’s residual skill. The MCP joint angle and percentage of duty cycle for the control signal (u) of the middle finger while performing passive therapies are shown in The software environment allows the healthcare staff to select and configurate therapies according to the patient’s needs a. There The four therapies are based on virtual reality environments; while therapies T1–T3 show just a virtual hand replicating the real movements of the hand exoskeleton b, therapThe relationship between the four available therapies and the type of movement-based exercises and virtual environment used can be found in The Self-Assessment Manikin (SAM) test was used to subjectively assess the emotional response of the patients. This test is a picture-oriented questionnaire to independently evaluate three emotional dimensions: arousal , valence  and dominance  . We provThe SAM evaluation test was used to obtain the ratings (from 1 to 5) in the valence, arousal and dominance dimensions. The higher the rating, the more intense the emotion would be .Patients were seated in a chair wearing the RobHand exoskeleton in their paretic hand . The forEach patient performed the SAM test three times  for the three emotional dimensions.The SAM tests scores were analyzed through Multifactorial Additive ANOVA. Numerical results extracted from this analysis are reported in p-value for a hypothesis that the true difference of means for the corresponding groups is equal to zero. The distributions of the score results of SAM1, SAM2 and SAM3 are shown in Numerical and graphical results extracted from the Tukey’s HSD post hoc test to compare test stages are reported in With respect to the Tukey’s HSD post hoc test to compare rehabilitation phases, numerical and graphical results are reported in From the results obtained from the Multifactorial Additive ANOVA, it can be concluded that rehabilitation phase  and test stage  statistically influence the three emotional dimensions . The patients became familiar with the RobHand platform during the 30 rehabilitation sessions. The results extracted from the post hoc test to compare rehabilitation phases support this idea. First, the average dominance for the five patients increased during the sessions. In fact, there was a significant difference between the initial and final phase. This growth in dominance implies that the patients, at the end of the treatment, had a greater feeling of control over the rehabilitation tasks. With respect to the arousal, the mean scores decreased over the sessions. This decrease was significant over the initial phase and both the middle and final phases. This decline could be a consequence of the fact that when patients were becoming familiar with the robotic system, they found it less attractive because it was not a novelty anymore. Although the mean of the valence values during the first sessions was high, these values suffered a significant decrease in the middle phase and grew again, reaching higher levels than the initial ones during the last phase. No clear conclusions can be drawn from the results of the valence emotional dimension.Significant differences were found between the results of the SAM1, SAM2 and SAM3 tests on the three evaluated emotional dimensions: valence, arousal and dominance. Both valence and arousal showed a significant difference from SAM1 to SAM3. The SAM3 test was performed right after the squeeze orange therapy T4, which is the only therapy with a realistic scenario, as it represents an everyday activity. This increase in both valence and arousal indicate that the patients felt happier and more stimulated performing T4. These results are consistent with many studies which claim that patients found rehabilitation therapies based on videogames more exciting ,37,38. TAlthough emotions are elusive, the circumplex model of affect is a theory that claims that all human emotions can be described as a linear combination of two independent basic emotions: valence and arousal. Valence expresses whether the emotion is pleasant or unpleasant, and arousal determines how much the emotion is activated ,39. FollThere are some limitations in the current study. The first one is related to the sample size since only five patients were included. The second one is related to the subjectively self-reported data, although it is an intrinsic risk in this type of study. Another limitation is that the subject may not clearly understand what is being asked about their emotional state. However, we chose the SAM questionnaire for this study because it was specially designed to minimize this risk as much as possible by using schematic pictures. Lastly, the order of the therapies (T1–T4) may influence the results. Since results with the robotic platform have been positive, further investigation is required with a larger sample size and considering the possible effect of therapy order.The conducted case series study aimed to assess, using the SAM test, the emotional response of patients under rehabilitation using the RobHand platform. The outcomes are encouraging: the scores of arousal, valence and dominance are positive. It was verified that the emotions induced by the RobHand platform were successfully distributed in the three-dimensional emotional space. Over the 30-session treatment, a significant increase in dominance was found, coupled with a significant reduction in valence. Additionally, significant differences were found in the three emotional dimensions, indicating a positive variation when performing SAM3 since the therapy was carried out just after one based on a virtual environment, in which the user has the objective of squeezing oranges. Furthermore, the healthcare professional responsible for conducting and supervising this study subjectively detected that patients were much more excited when they performed the squeeze oranges therapy. The outcomes of the study encourage further development of videogame-based therapies in order to increase the patient’s positive feeling.


The continuous development of Human-Computer Interaction (HCI) and information technologies impact the digital learning environment. The network and multimedia technologies change the Autonomous Learning System (ALS) structure. The learning process uses several techniques; however, the interactive function requires continuous improvement to enhance autonomous learning. Therefore, Optimized Deep Learning Network (ODNN) is introduced to build the Autonomous English Learning System (AELS) in this work. The ODNN system uses the learning and activation functions that derive the student’s learning capabilities and gives the proper training to the student. The HCI-based created autonomous learning process provides the guidelines to the student for making independent learning. The ALS improves the student’s learning ability and skills compared to classroom-based learning. The discussed ODNN-based AELS system effectiveness is evaluated using the Japanese-English Bilingual Corpus with a set of assessment questionaries. Then the HCI-based autonomous English learning is a quantitative analysis with the classroom-based learning. The discussed system is implemented using the Python tool, in which the AELS system ensures 98.51% learning efficiency compared to classroom learning. Therefo machine .In IES, students must continuously update their skills and knowledge to survive in this globalized society . The stuvia the Internet source. Therefore, several researchers are interested in creating the HCI system for developing the autonomous learning system (ALS). However, the interactive functions are still weak to ensure effective results. In this educational system, pupils’ ability to learn on their own is promoted via the use of ongoing assessments of a variety of competencies. Evaluation of competency is the process of gathering evidence on the accomplishment of an individual’s brilliant practice to create a judgment about his/her abilities with respect to a professional profile. When making a choice, an assessment is a process of gathering information, processing it, interpreting it, and employing it.Finally, self-learning strategies are identified for improving their learning behavior. During this process, users create their self-learning strategies, goals and learning process to enhance their learning skills . The autThen the AELS issues are overcome by applying the HCI with the machine learning techniques. Optimized Deep Learning Network is applied to investigate student learning patterns. During the learning process, the deep neural model is updated using the bee colony algorithm to reduce the deviation between the actual learning and independent learning outputs. For an argumentative design, the interplay between cognitive psychology and HCI must be considered. Evaluation of the student’s comprehension of course content and analysis of difficulty levels for test questions are the app’s primary functions. This research developed a control experiment to test the influence of HCI teaching on students’ English proficiency and to examine students’ performance and instructors’ sentiments and their views on HCI learning. Computing, cognitive psychology, and ergonomics are major influences on HCI design. User interfaces, as they are perceived and controlled by humans, are its primary focus. Using this method, we may see humans and machines as two interdependent systems that exchange information in real-time. Then, a continuous testing procedure is implemented to evaluate the student learning strategies. The system’s effectiveness was assessed using performance analysis and implemented using Python.(i)Student learning and activation functions are derived from the ODNN system, which provides the student with appropriate instruction.(ii)HCI-based autonomous learning offers students a framework for self-directed study.(iii)Thus, a quantifiable comparison may be made between HCI and conventional teaching methods.Then the rest of the paper is organized as follows: Section “Related works” analyses the various researcher’s opinions regarding the HCI-based English learning system. Section “Human-computer interaction-based autonomous English learning system” discusses the working process of the HCI-based ODNN approach related to AELS and the system’s effectiveness evaluated in section “Results and discussions.” The conclusion is described in section “Conclusion.”Various researchers’ work, ideas, frameworks and interactive technologies are analyzed in this section to improve the autonomous learning education system. The research intends to construct the HCI-based AELS. The AELS system is developed by incorporating the Optimized Deep Learning Networks (ODNN). The constructed intelligent learning system is used to understand the audience’s mindset and strategies for improving the learning rate. The ALS developed in terms of two stages: academic-related  and basic knowledge-related learning . Initialvia the network. It is possible to employ technology to assist and enhance one’s ability to learn a foreign language. Teaching foreign languages has never been easier than it is now, in large part to the flexibility afforded by technological advances. Teachers are increasingly relying on technology to assist their students in learning a new language. However, most students lack knowledge of English, which ultimately affects the student’s learning progress. Generally, students face difficulties in grammar; therefore, the HCI-interaction system should concentrate on essential learning skills such as sentence structure, grammar and vocabulary. Especially in the foreign language department, students face vocabulary difficulties, sentence structure formation and grammar issues. Therefore, tutors should address these challenges while teaching English because the problems affect the students learning performance. Accordingly, a computer interaction system should be developed to cover the curriculum reform that covers several leading such as writing, reading, listening and speaking. These modules should cover the grammatical phase because it is the backbone of the autonomous English learning process.Generally, multimedia classrooms utilize the campus Internet while accessing the learning information. The campus network resources act as the bridge between the students and staff, and the multimedia-based learning process improves students’ interests. Then the student learning capabilities are continuously enhanced by performing the educational reforms and the respective high-quality resources shared This section discusses the HCI Optimized Deep Learning Networks (ODNN) approach to AELSs. The system consists of four modules: maintenance according to knowledge, diagnosis practices, strengthening of knowledge points and improvements. It’s this participatory learning on the web that teaches users the right technique to model data. Simply said, it’s easy to use and very efficient. In this regard, the emphasis of this work is on Human-Computer Interface (HCI) design concepts and execution (HCI). HCI design concepts and principles are laid forth, and an interactive e-Learning application’s HCI design is evaluated.These four modules are critical points for the HCI with an autonomous learning process which works according to the machine learning algorithms. The collected English dataset has been investigated using Natural Language Processing (NLP) to perform the preprocessing, which improves the overall AELS system performance. This work uses optimized deep learning networks to examine students’ learning behavior. The network examines the relationship between the students learning process with their performance. The effective utilization of the network activation and learning function reduces the system complexity. During the computation, the network uses fault tolerance, parallelism, numerous approximation, self-learning and hardware properties to improve the AELS performance. Then the HCI with ODNN-based AELS system working process is illustrated in The first module of the AELS is knowledge maintenance which only the English experts do. The above The following important module is diagnosis practices, in which the system continuously learns the samples from expert knowledge. The system learns the several knowledge points in the example question banks. Then the questions are given to the students, and the student’s answers are compared with the training set, which helps to make the self-diagnosis. According to the self-diagnosis process, students learning ability is understood, and their learning level has to be identified effectively.The neural network should consider the different knowledge points to ensure effective AELS. Therefore, the learning points are classified as various points such as untested knowledge points, good knowledge points and low knowledge points. These knowledge points are utilized for examining the students learning capabilities. Students can select the respective modules to make their learning and testing process. After the learning process, self-diagnosis is taken to evaluate their learning performance.The last module of the AELS is practice improvement, which is done between students and skilled users. The students learning ability is checked continuously, and the results are matched with the database. During the analysis, student ability is analyzed at a specific time to determine their learning efficiency. The improvements are examined in terms of historical practices and knowledge point exercises. The student results are consolidating their memory and forgetting category. Finally, the AELS system combines the students’ knowledge and improvement points in English learning.f1,f2,…..fn} and the {w1,w2,…..wn}. From the set of features, the Term Frequency and Inverse Term Frequency (TF-ITF) value is estimated for identifying the student learning performance. Then the Term Frequency (TF) is calculated using Eq. 1.These four modules require a database to evaluate the student learning efficiency. The dataset consists of student personal information  and a user static knowledge table (the knowledge table maintained by experts). The stationary knowledge tables consist of a set of English vocabulary details and grammatical information used to identify the user effectiveness. The static knowledge table has knowledge points and question banks . In addition, rule knowledge points  are maintained to balance the learning efficiency. The constructed knowledge information is processed by applying the Optimized Deep Learning Networks (ODNN). The neural network process the collected question banks (text) along with the Natural Language Processing (NLP). The constructed HCI system-based AELS helps understand the user requirements, and the learning process is established accordingly. The NLP techniques investigate the student’s answers in terms of examining sentence and word count, Parts of Speech (PoS) Tag, spelling mistakes and frequent occurred words. The sentence and word count use the “text mining” library tool for investigating the number of words in the sentence. The library has 276 stop words in English; according to these words, sentences and words are counted continuously. Then Parts-of-Speech (PoS) tag details are extracted from the student’s answer. Here Natural Language Tool Kit (NLTK) is utilized to extract the PoS tags from the given input English sentence. Then the student’s structural and textural features are extracted to determine the frequency of occurrence of words. Consider the student’s answer; it has a set of features and weight values that are represented as {In the English language, words are presented in various positions and have different lengths; hence, the TF value should be standardized using Eq. 2.It is possible to identify the “signature” of a writer’s cultural level, slang or technical words, and other literary qualities using frequency analysis. It is feasible to infer a person’s overall vocabulary from the number of words used in a particular document. With the help of the TF value, the Inverse Document Frequency (IDF) has been computed using Eq. (3).ti,k, term m-composition has been computed using Eq. (4).From the computed term frequency value wi,k = TFi,k + IDFi,k. Then sentence length, unique terms and characteristics are extracted from student answers. After applying the NLP process, word embedding is performed to extract the vocabulary from the student document. The embedding process helps get the syntactic and semantic similarity values and the relationship between the words. The Word2Vec constructs the embeddings from the sentence using the Common Bag of Words (CBOW). The CBOW model derives the context values of words, and the related terms are predicted during the training and testing process. For every input, CBOW uses the neural network layers that have been analyzing each word, and the respective context words are predicted using the database. During this process, the input word hot encoding value is utilized to estimate the output or target word. The input word is one of the encoded vectors, and the one-hot encoded vector has V in size. These input vectors are processed using the hidden layers with N neurons, and the computed output value is estimated with the same V size, which is done using the softmax function. The network has the weight value between the input x to the hidden layer  and the hidden layer to the output layer . Then the structure of the CBOW-related neural network is illustrated in In Eq. 4, ∝ it is represented as the constant, and its value is 0.01; the weight value is computed as This work applies an optimized deep neural model to manage students’ English learning process. The effective utilization of the neural word embedding process retains the system’s robustness and effectiveness and achieves good performance. The ODNN model must be utilized to evaluate the student’s answer; their performance evaluates the learner’s efficiency. A deep learning network processes the extracted input features with several layers such as input, hidden and output layers; each layer performs a specific function to attain the output. The extracted inputs are investigated in the convolution layer that uses the filters to reduce the data size. Then the max-pooling process is applied to minimize the data size and eliminate the over-fitting issue. The outputs are obtained using the softmax activation function, and the computed output is compared with the training set. The deviation between the output values is calculated to update the network parameter. The updating procedure is performed continuously to improve the overall AELS performance. This work uses the bee colony optimized algorithm to select the suitable network parameters. The bee colony optimization algorithm selects the parameter based on the food searching process. It has bee, and a set of food sources are the central concept while selecting the features. During the search process, nectar values are used to choose the best food source to resolve the optimization problem. The algorithm has scouted, employed and onlooker bees that highly influence the food searching. The employee bee checks the parameters, and the computed values are transferred to the scouts while the searching space does not have the food resource. The searching process is initialized by using Eq. 5.In Eq. (5), solutions are predicted from ith iteration in search space, the search space having the number of solutions which are belonging to the neural model weight value, and the answer is computed from the minimum n (number of solutions), a random decimal value xi. The computed food resources are compared with the new resources to identify the best food resources. The computed solution is transferred to the onlooker bee that identifies the optimized solution from the solution set. The new solutions are selected based on the nectar value; if the solution has a maximum value, it has been added to the list, and previous solutions are eliminated. Then the new solutions are computed using Eq. (7).In Eq. 6, k is the random index value taken from 1 to According to Eq. 7, the optimal solutions are identified, and the local optimal solution issue is resolved using the scouting stage activity. After identifying the solutions, the optimal new solutions are obtained by performing the mutual learning that is defined in Eq. (8).Fiti and Fitk. A Uniform random number is defined as In Eq. 8, current and neighboring food source fitness values are represented as the According to Therefore, HCI based autonomous English learning process provides the proper guidelines to learners. Then the effectiveness of the system is evaluated using experimental results and discussions.1  based AELS. This process uses the Japanese-English Bilingual Corpus1  with a sIn Eqs 9–11, True Positive (TP) is defined as a correctly identified student answer, True Negative (TN) is defined as a model that effectively identifies the wrong answer, False Positive (FP) is defined as falsely rejecting the null value, and False Negative (FN) is defined as wrongly identifies the student’s answers.Then the HCI with ODNN-based AELS system learning efficiency is compared with the traditional classroom environment. During the analysis, around 195 students learning efficiency is compared, and the results are illustrated in via the AELS process. The HCI-based AELS system utilizes the ODNN approach to investigating the student knowledge points. According to the student’s query, the constructed AELS system reacted by analyzing the database information. Here, the collected dataset information is divided into two testing (30%) and a training set (70%). Once the dataset is collected, it is processed by applying the NLP techniques to eliminate the irrelevant information from the student’s answer. Then the efficiency of the introduced ODNN approach is evaluated in three phases such as familiar, basic, and unfamiliar. Then the HCI- with ODNN-based AELS system effectiveness is illustrated in Thus, Thus the paper constructs the HCI with AELS by using Optimized Deep Learning Networks (ODNN). The analysis system uses the Japanese-English Bilingual Corpus information to create an effective independent learning set. The sentences in the English set are processed by NLP, removing irrelevant information. Then the word embeddings are derived from the words according to the optimized neural model. The neural model generates the patterns from the derived word that predicts the correct grammar and next word, which is used to train the student effectively. During the analysis, the bee colony optimization algorithm updates the network parameter, minimizing the deviation between the student and systematic answers. The minimum error rate indicates that the student independently learns English with maximum accuracy (98.51%). This work is able to deal with the four modules that is more relevant to the learning process; however, the method requires improvement while evaluating the student performance. In the future, the efficiency of the independent learning system will be further improved by utilizing the meta-heuristics optimization algorithm.The original contributions presented in this study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.XW: writing – original draft preparation. SS: editing data curation and supervision. Both authors contributed to the article and approved the submitted version.


With increasing numbers of people living with dementia, there is growing interest in the automatic monitoring of agitation. Current assessments rely on carer observations within a framework of behavioural scales. Automatic monitoring of agitation can supplement existing assessments, providing carers and clinicians with a greater understanding of the causes and extent of agitation. Despite agitation frequently manifesting in repetitive hand movements, the automatic assessment of repetitive hand movements remains a sparsely researched field. Monitoring hand movements is problematic due to the subtle differences between different types of hand movements and variations in how they can be carried out; the lack of training data creates additional challenges. This paper proposes a novel approach to assess the type and intensity of repetitive hand movements using skeletal model data derived from video. We introduce a video-based dataset of five repetitive hand movements symptomatic of agitation. Using skeletal keypoint locations extracted from video, we demonstrate a system to recognise repetitive hand movements using discriminative poses. By first learning characteristics of the movement, our system can accurately identify changes in the intensity of repetitive movements. Wide inter-subject variation in agitated behaviours suggests the benefit of personalising the recognition model with some end-user information. Our results suggest that data captured using a single conventional RGB video camera can be used to automatically monitor agitated hand movements of sedentary patients. Agitated and repetitive behaviours are some of the most common and demanding symptoms of dementia . AgitatiRGB cameras are widely available and provide a rich source of data. Video-based approaches are particularly suitable for patients who are mainly sedentary. For many patients, a video camera directed towards a favourite chair, perhaps situated above a television screen, offers the potential to capture rich data over significant periods. In addition to capturing the whole body, video can provide detailed information about hand movement. Using opensource algorithms to extract skeletal keypoint locations, a person can be represented as a “stickman.” Reducing video data to a sequence of skeletal poses provides informative features ideal for monitoring behaviour. Skeletal models can protect patient privacy by removing the need to retain images.This research establishes the potential for automatic classification of repetitive hand movements and automatic assessment of the intensity of repetitive hand movements using skeletal keypoints extracted from video. We conjecture that first recognising the type of behaviour will aid assessing intensity. The feasibility of using video-based keypoints to monitor agitated hand movements is explored using a small dataset containing video in which healthy volunteers demonstrate repetitive hand movements according to the expected norms of agitated behaviour. Due to the size of the dataset, methods are confined to those suitable for small datasets. Five types of repetitive hand movements are studied: picking, scratching, rubbing, wringing, and clapping. The first four movements are indicative of agitated behaviours observed in people living with dementia ; clappinA range of sensors has been employed to assess repetitive behaviours, including accelerometers –11, deptThe detection, recognition, and measurement of the intensity of repetitive movements are fundamental components for monitoring agitated behaviours. The first component, the detection of repetitive movements, is the most extensively researched. However, most studies have focused on showing the correlation between motor activity and clinical agitation scores. Studies using accelerometers have shown a correlation between disruptive sleep patterns and night-time incontinence , activitRecognising the type of repetitive movement is the second component of agitation monitoring. A study comparing the effectiveness of accelerometers and 3D keypoint models for agitation recognition found that data obtained from accelerometers were computationally more efficient but slightly less accurate than skeletal pose data . SkeletoThe third component for monitoring agitation is measuring the intensity of agitated episodes. The intensity of a movement is defined as the frequency of repetitions during a specified period. IMUs , 11, 27 This study demonstrates the potential for using video-based keypoints to identify the type and assess the intensity of agitated hand movements. Reducing a video image to a sequence of keypoints significantly reduces the number of features in each frame whilst retaining important pose information, reducing computational cost. However, recognising agitated hand movements from skeletal poses remains challenging due to the number of hand keypoints and the range of possible hand movements.This section describes a system for automatically assessing both the type and intensity of unknown hand movements. Figure Most hand datasets contain hand gestures used for human–computer interaction , 33 or sclapping, picking arm, scratching arm, hand wringing, and rubbing an object. The dataset was split into training and test sets. Data from fifteen participants were used for the training and validation of classifiers to identify the type of movement and to learn parameters for measuring the intensity of repetitive movements. Data from the remaining five participants were reserved for testing the robustness of the system with new subjects. The dataset was recorded by participants in their own homes, creating a varied and demanding dataset that is representative of home or care settings. Variations included participant position (standing or seated), dominant hand (the hand which moves the most), hand position (the location and orientation of the hand in relation to the body), and whether one or both hands move. Similarities between different types of hand movements and variations in the intensity of movement add to the complexity of the dataset. Each movement was performed for 30 s at four different intensities: slow, medium, medium-fast, and fast, creating a dataset of 400 sequences. A metronome set correspondingly at 30, 50, 70, and 90 repetitions per minute (rpm) was used to pace the hand movements. Most videos were recorded at a speed of 30 frames per second; those not were up- or down-sampled to this rate.Twenty healthy participants were recorded by an RGB camera demonstrating five hand movements: ith keypoint in frame f. When a body keypoint is occluded, OpenPose records the keypoint as missing. Conversely, all the hand keypoints are returned, even when the hand is occluded, if the wrist is detected. Missing values are imputed using the median location of keypoints in neighbouring frames.Each 30-second training data sequence was divided into four non-overlapping 6-second (180 frame) sequences. Three seconds of data were discarded at both ends of the original sequence to remove any different movements from the start and end of a sequence, creating a labelled dataset of 1200 6-s sequences. OpenPose  is used f, is pre-processed to create a scale-normalised skeletal pose which is invariant to camera distance and participant size. The keypoint positions are translated so that the neck keypoint becomes the origin and scaled by dividing by the shoulder width, sh is shoulder keypoint location.Each frame, A Savitzky-Golay filter  is applied to each keypoint to smooth the data and remove noise. A Savitzky-Golay filter was selected due to its ability to smooth noisy signals with large frequency spans whilst maintaining the shape and height of the waveform peaks  that arex co-ordinate of the chosen wrist of frame f. The dominant hand is identified, and where necessary, the hands are reflected so that the right hand appears dominant. Finally, hand, wrist and elbow keypoints are re-centred around the midpoint between the wrists, resulting in the scale-normalised skeletal arm and hand pose x-coordinate of the left wrist in frame f.During a single-handed action, it is assumed that the hand that moves the most is performing the action. The hand with the most wrist movement is described as the dominant hand. Where necessary, skeleton sequences are reflected so that the right hand always appears to be the dominant hand. The dominant hand is deemed to be the wrist with the largest path distance during a sequence. Path distance is calculated for each wrist in a 180-frame sequence as:ci of each the 21 keypoints for each hand:Analysing the pattern of a single keypoint or body part can enable the tracking of repetitive body movements , 23, 38.The most straightforward approach for classifying movement is to reduce each sequence to a single feature vector and apply a vector classifier. Sequences of skeletal poses have been summarised into vectors using bag of visual words  and tempn frames, S = c* is the class with the minimum total distance between observations and nearest neighbours:NNc is the nearest neighbour of c. NBNN has been widely used for keypoint based activity recognition , enabling comparisons between sequences where the subjects moved their hands with different levels of mobility.In Fig. Range intensity determines whether the category of intensity of a sequence has been correctly predicted. Repetitions are considered to be correctly counted if the number of detected repetitions falls between a range whose boundaries are determined by the metronome’s speed.Intensity is measured according to the number of repetitions of a movement within a sequence. Two different measures of accuracy are used. Personalised intensity assesses the ability of the model to differentiate between a subject’s movement at different speeds. Personalised boundaries are based upon the average number of repetitions detected at each intensity and are reset for each activity and participant. An intensity is detected correctly when the number of repetitions detected falls between two boundaries; the lower boundary is the mean number of repetitions detected for sequences for which the correct intensity is one category slower than the correct intensity of the examples being considered; the higher boundary is the mean number of repetitions detected for sequences for which the correct intensity is one category faster than the correct intensity of the examples being considered. Criteria for determining the range and personalised boundaries are set out in Table Whilst range accuracy is the more stringent measure of the accuracy, personalised accuracy is sufficient for identifying changes in the intensity of a movement, and so is ideal for monitoring agitation of an individual. Range accuracy is used during the grid search to set the hyperparameters, whilst personalised accuracy is reported in the test data results.hand centroid and horizontal wrist parameters did not perform best for any of the five types of repetitive hand movement and so were not used with the test data in the final model. The violin plots in Fig. Hyperparameters were tuned by a grid search separately for each class of hand movement and distance parameter from the training data. Table The proposed system, illustrated by the pipeline in Fig. Using signals created from the distance measure determined by the predicted type of repetitive hand movement, peak detection was used to assess changes in intensity. Personalised accuracy was used to measure the ability of the proposed approach to differentiate between movements of different intensities. Both the type and the intensity of movement were classified correctly in 75.25% of the test sequences. The type of movement was classified correctly in 80.0% of sequences, whilst the intensity of movement was classified correctly in 92.5% of sequences. Results for each stage of the system are presented in the confusion matrices in Fig. As the type of movement was predicted incorrectly in 20% of the sequences, we explore the importance of first predicting the type of movement before assessing intensity by comparing the proposed model with models built under three different scenarios. In the first scenario, we assume that the type of hand movement was known. The class-specific parameters, tuned on the training data and given in Table The accuracy of our proposed model and the three models built under the different scenarios are reported in Table discriminative poses and peak detection. Our model was able to correctly identify both the type and intensity of repetitive hand movements in 75% of the sequences performed by unseen subjects. We have shown that despite relatively accuracy for classifying the type of movement (80%), first predicting the type of movement can aid assessment of the intensity of repetitive movements by enabling the selection of appropriate distance measures for measuring the intensity of movements.We have proposed a model for efficiently monitoring the type and intensity of repetitive hand movements, based upon The lower accuracy for recognising the type of movement obtained in the cross-subject tests (Test 3) than in the non-cross subject tests (Tests 1 and 2) suggests that classification of the type of movement could be improved by training the model with subject-specific data. Due to variations in how people perform different movements, the inclusion of person-specific observations in the training dataset can significantly improve recognition accuracy. Whilst the ability of an activity recognition model to generalise to new subjects is generally considered important for activity recognition, some researchers have argued that for a model to be robust in real-world applications, personalisation is essential . As NBNNThe experiments in this study have used healthy volunteers demonstrating repetitive hand movements indicative of agitation, where the speed of movement was prescribed by the researcher. Whilst the study has demonstrated that the type and intensity of repetitive movement can be automatically detected, further studies using data captured from people who experiencing agitation are necessary to confirm whether the system is suitable for use within a care setting. However, the high levels of personalised accuracy achieved for assessing the intensity of repetitive behaviours suggest that the system is likely to be able to assess changes in behaviours across a wide variety of different behaviours. This study has shown the potential for video-based systems to supplement existing care of people living with dementia by automatically monitoring the type and intensity of agitated episodes. In this study, the first to focus on agitated hand movements, we have concentrated on recognising and assessing the intensity of repetitive movements. Future work will explore the detection of repetitive hand movements in order to automatically differentiate between agitated movements and normal behaviours. Whilst this study used data from healthy participants, the results have shown the value in additional studies using people living with dementia.