The aim of this study was to evaluate its reliability in localizing the anterior superior iliac spine (ASIS) at the pelvis and the hip joint centers (HJC). Additionally, we evaluated its accuracy to get a rough estimation of the potential to use of 3DFUS to segment bony surface. This could offer potential to register medical images to motion capture data in future. To evaluate reliability, a test–retest study was conducted in 16 lean and 19 obese individuals. The locations of the ASIS were determined by manual marker placement (MMP), an instrumented pointer technique (IPT), and with 3DFUS. The HJC location was also determined with 3DFUS. To quantify reliability, intraclass correlation coefficients (ICCs), the standard error of measurement (SEm), among other statistical parameters, were calculated for the identified locations between the test and retest. To assess accuracy, the surface of a human plastic pelvic phantom was segmented with 3DFUS in a distilled water bath in 27 trials and compared to a 3D laser scan of the pelvis. Regarding reliability, the MMP, but especially the IPT showed high reliability in lean (SEm: 2–3 mm) and reduced reliability in obese individuals (SEm: 6–15 mm). Compared to MMP and IPT, 3DFUS presented lower reliability in the lean group  but slightly better values in the obese group . Correlations between test–retest reliability and torso body fat mass (% of body mass) indicated a moderate to strong relationship for MMP and IPT but only a weak correlation for the 3DFUS approach. The water-bath experiments indicated an acceptable level of 3.5 (1.7) mm of accuracy for 3DFUS in segmenting bone surface. Despite some difficulties with single trials, our data give further rise to the idea that 3DFUS could serve as a promising tool in future to inform marker placement and hip joint center location, especially in groups with higher amount of body fat. These errors can have clinically relevant effects on estimated HJC locations and potentially introduce errors in tracking body segments during locomotion9. An alternative to manual marker placement (MMP) is the calibrated anatomical systems technique10, referred to in this study as the “instrumented pointer technique (IPT)”. It consists of a pointer with a minimum of two markers attached to it and allows to digitize the tip of the pointer as a 3D coordinate in a reference coordinate system. In clinical practice it is primarily used to define anatomical landmarks, which are prone to STA. For example, the IPT is used to define the anterior superior iliac spine (ASIS) as virtual markers within a cluster-based reference coordinate system placed at the sacrum11. A big advantage of these virtual ASIS markers is that they are only prone to the STA introduced by the sacrum cluster which is expected to be lower compared to STA at the abdominal region. Thus, more reliable results are expected. Tawy and Rowe12 have demonstrated that the IPT is a reliable method, but they only used a box as a measurement phantom to evaluate reliability. However, clinical practice may involve determining and palpating anatomical structures in individuals with a high amount of subcutaneous fat. This makes the identification of the underlying anatomical structure challenging and presumably both, MMP and IPT, more unreliable3.In clinical gait analysis, biomechanical models assume that markers are placed above bony landmarks. The combination of a minimum of three markers are used to define body segment coordinate systems. The relative movement of the skin to the bone and wobbling mass, so-called soft tissue artifacts (STA), as well as inaccurate and unreliable marker placement can strongly affect the accuracy of 3D gait analysis13. 3DFUS can be used to measure geometrical aspects of the human skeletal system and identify locations of anatomical landmarks without being affected by soft tissue. This information then can be used to personalize biomechanical models. Compared to volumetric medical imaging such as computed tomography (CT) or magnetic resonance imaging (MRI), 3DFUS offers a potentially cost- and time-efficient option in measuring in-vivo aspects of the human skeletal system without exposure to radiation. These factors make 3DFUS an attractive possibility in clinical and research settings for gait analysis and may be the reason for the increased popularity in recent years. 3DFUS has proved to be an applicable approach in 3D gait analysis to localize the hip joint centers (HJC)15, identify the medio-lateral axis of the femur16, measure femoral torsion17, and was used to create subject-specific models by measuring bone geometry and muscle-tendon architecture18. As already demonstrated by Passmore et al.18 and Jia et al.19, 3DFUS can also be used to identify bone geometry of the femur condyles or the trochanter major. 3DFUS is also being used in other research domains, i.e., to visualize anatomical structures in 3D, measure quantitative 3D attributes, and for fusion with other 3D image modalities. A review by Mozaffari and Lee20 provides a general overview. Recently, it was outlined by Kubicek et al.21 that in particular in the domain of computer assisted orthopedic surgery 3DFUS is being successfully used to identify anatomical landmarks, more particular to measure pelvic tilt or to track anatomical planes to inform surgeons. 3DFUS can reliably identify pelvic tilt during standing, sitting and in supine position23 with an accuracy in the range of approximately 025 and with good to excellent reliability26. Research also demonstrated that 3DFUS-based pelvic tilt tracking is less affected by higher amounts of soft tissue than traditional pointer-based imaging modalities27. These results are encouraging and leverage the relevance to further investigate the potential of 3DFUS to support clinical and research practice in the 3D gait analysis community. Yet, the currently available 3DFUS related research in clinical 3D gait analysis is mostly limited to normal weighted study groups. However, increased subcutaneous fat makes the identification of anatomical structures more challenging. Research on the application of 3DFUS in individuals with increased body mass is scarce, has a limited number of participants, or was used to measure pelvic tilt only for the purpose of computer assisted orthopedic surgery24 and did not compare to skin-marker-based measurements. In clinical gait analysis, the location of the ASIS landmarks is highly important as these locations are required to define the pelvic anatomical coordinate system. Furthermore, ASIS locations are used to calculate pelvic width, which is subsequently used to estimate the HJC location, i.e., via regression equations28. Currently available research on 3DFUS does not report any details on reliability in identifying ASIS locations.Three dimensional free-hand ultrasound (3DFUS) is a combination of a conventional ultrasound probe with a 3D motion capture system. It enables the transformation of 2D pixel coordinates of an ultrasound image to the 3D coordinate system of the motion capture system29, this might in future allow to register medical images such as MRI or CT scans to motion capture data.Therefore, this study adds to the 3DFUS-related research field by evaluating the reliability of a 3DFUS-based approach to identify the ASIS locations and the HJC for the purpose of clinical 3D gait analysis. In contrast to earlier research, we conducted our 3DFUS experiments in a large number of lean and obese participants with the purpose to investigate the utility of 3DFUS in the context of applying external markers and in identifying the HJC location for clinical gait analysis. The reliability in identifying the ASIS landmarks was also compared to MMP and the IPT. We hypothesized that placing markers manually and using the IPT, both will be either similar or even more reliable in the lean study sample compared to 3DFUS. However, we also expected that errors will increase to a lesser extent for the 3DFUS-based approach in obese individuals and therefore placing it as the superior method in this sample. The reason to hypothesize that 3DFUS will not be superior in lean participants is associated with the difficulties in using the probe on edgy bony landmarks with little body fat covering them. Additionally, we also conducted simple water-bath experiments with a plastic measurement phantom of a pelvis to get a rough estimation of the potential usability of 3DFUS in segmenting entire surface areas of bony landmarks. As highlighted by the work of Gasparutto et al.http://www.telemedultrasound.com/) unit synchronized to a 10-camera Vicon TX10 system  operating at 150 Hz. We have used a linear probe (L12-5L40N-4) with a frequency range of 5–12 MHz and field of view of 39 mm. The ultrasound probe was equipped with a rigid 4-marker-cluster  and was performed in accordance with the relevant guidelines and regulations. All participants, and when necessary their legal representatives (< 18 years of age), were informed prior to the study and gave written informed consent.To evaluate the test–retest reliability, we recruited a convenience sample of 16 lean and 19 obese children and adolescents, see Table 15. During data analysis of the 3DFUS-HJC trials we had to remove 11 trials (legs) from the obese and two trials (legs) from the lean group due to problems with image quality and a technical problem, respectively. In the obese cases the HJC were located too deep. This prevented us to acquire images with sufficient quality. A minimum of approximately 10 and a maximum of 50 ultrasound image frames were used to digitize the curvature of the femoral heads. A M-estimator Sample Consensus (MSAC) algorithm32 was then used to fit a sphere into these data to determine the location of the femoral heads (HJC). ASIS were defined by selecting the most anterior structure manually in the ultrasound images. The assessor had several years of experience in 3D gait analysis and approximately four months of experience with the 3DFUS system. A rigid marker-cluster attached to the sacrum during the test and which stayed in place for the retest, served as a common reference system, see Fig. 34. Note that the SEm should not be confused with the Standard Error of the Mean (SEM). ICCs were calculated based on a single-measure, absolute-agreement, and two-way mixed effects model  device . For this purpose participants lay horizontally, with legs and arms slightly apart from the body, for five minutes without any movements. Eight adhesive electrodes, two on each limb, were attached and allowed to measure total and torso fat mass and fat free mass in percentage of body mass (%), see Table To statistically evaluate if test–retest errors in terms of the Euclidean distances differed between the methods to localize the ASIS  a Friedman test was conducted for the lean and the obese group separately. A Shapiro–Wilk test indicated violation of normality for several variables which was the reason to use non-parametric tests. In case of a significant result, additional pairwise Mann-Whitney tests were performed. To avoid multiple testing and, thus, high family-wise error rates, tests were only performed for the Euclidean distances. However, to reduce type II errors, post hoc tests were not Bonferroni-corrected. Alpha level was set to 0.05.For a more intuitive interpretation, the data were expressed in an local pelvis coordinate system based on the sacrum-cluster. Pearson product–moment correlations were run to determine the relationship between the test–retest Euclidean distances for all methods  and fat mass of the torso (FMT).36. To quantify accuracy, a 3D laser scan  of the pelvis phantom served as the ground truth . The velocity of sound in water was calculated by the formula of Lubbers and Graaffsee Fig. . A 3D-prICCs values presented excellent values (> 0.93) for all methods in the lean group except for the superior–inferior direction of 3DFUS-HJC estimation which showed good agreement (0.77). ICCs values in the obese group were in general lower and ranged between 0.76 and 0.99 indicating good to excellent agreement. Details can be found in Table p < 0.001, but significance was slightly missed for the obese group, p = 0.087. In the lean group the median (IQR) Euclidean distances for MMP, IPT, and 3DUS were 6 mm (3–9), 6 mm (4–8) and 10 mm (7–19), respectively. Post hoc tests indicated no significant difference between the MMP and IPT (p = 0.708). However, there was a statistically significant difference between MMP and 3DUS (p = 0.001) and between IPT and 3DFUS . In the obese group the median (IQR) Euclidean distances presented higher values for MMP and lower ones for IPT and 3DUS with 22 mm (13–34), 17 mm (10–26) and 17 mm (11–26), respectively. Regarding the SEm, in the obese group 3DFUS showed a lower SEm (9 mm) compared to MMP and IPT  of the minimum Euclidean distances between the laser scan model and all 3DFUS bone segmentation trials was 3.5 1.7) mm with a range of 0–11.9 mm. Fig. .7 mm witThis study evaluated the reliability of an 3DFUS-based approach in identifying anatomical landmarks at the pelvis and the location of the HJC. In addition, we also conducted simple water-bath experiments with a plastic measurement phantom of a pelvis to get a rough estimation of the potential usability of 3DFUS in reconstructing bony surface. For evaluating the reliability, the position of the ASIS were determined by manual marker placement (MMP), an instrumented pointer technique (IPT), and by a 3D free-hand ultrasound (3DFUS) technique. Test–retest experiments were carried out in a lean and an obese study sample. We hypothesized that MMP and the IPT will both be either similar or more reliable in the lean study sample compared to the 3DFUS approach. In addition, we expected that errors will increase to a lesser extent for the 3DFUS based approach in obese individuals and therefore placing it as the superior method in this sample. The obtained data support this hypothesis only partly.14 who reported a range of approximately 4 mm for their 3DFUS-based HJC reliability study. Studies originating from the field of computer assisted orthopedic surgery, where 3DFUS is used to identify pelvic tilt in various positions, also demonstrated that 3DFUS can be a reliable tool27. In example, Dardenne et al.24 have recently evaluated the intra- as well as inter-observer reliability of an ultrasound-based device to measure patient-specific pelvic tilt in different daily positions. They reported average ICC values of greater than 0.9, except for standing where a slightly lower ICC of 0.7 was found. Similar high reliability indices were found by Marques et al.25. They reported ICC values of greater than 0.85 and SEm values of approximately less than 428 who have reported an inter-trial average difference of below 5 mm with the Harrington regression equation and below 20 mm for the geometric fit method to determine the HJC location. This shows that 3DFUS is able to reach the currently available standards in the identification of the HJC position in lean individuals. In an earlier study we showed that different non-invasive HJC estimation methods localized the HJC position with up to an alarming 6 cm of difference in an obese population2. These errors were primarily due to problems in identifying anatomical landmarks and STA. However, the higher variability and the presence of outliers in the 3DFUS-based estimates in the test–retest differences indicate a lower robustness of 3DFUS  and the FMT in Fig. 9. These can be regarded as clinically relevant. Thus, the approximately 4 mm of error for reconstructing bone surface seem promising. However, it has to be clearly noted that our accuracy estimates are only based on the surface of the pelvis phantom and not on in-vivo measurements of the ASIS or the HJC position. Yet, our accuracy results are in good agreement with earlier research from Peters et al.14. They have validated a 3DFUS-based approach in localizing the HJC by using MRI derived HJC-distances and reported an accuracy of approximately 4 mm. However, this approach is limited as it only takes into account a distance derived from two systems as a surrogate outcome for accuracy. This approach does not allow to directly quantify the accuracy in reconstructing bone surface in 3D space. Our results extend their results and show that 3DFUS is able to reconstruct a surface within an accuracy of less than 4 mm. The accuracy estimates of our study are also in agreement with study results from the field of computer assisted orthopedic surgery. Dardenne et al.24 have evaluated the accuracy of an ultrasound-based device to measure patient-specific pelvic tilt on a testing mechanical bench. They reported an accuracy of approximately 1.139 have recently evaluated a 3DFUS based navigation system for inter-operative navigation. They performed intensive experiments and found that the 3DFUS-based system can localize 3D reconstructions of vertebral phantoms with an accuracy of approximately 2 mm. One reason why our results show slightly lower accuracy is, that our estimates also include errors of the manual identification of bony surface in each frame of the ultrasound images. The assessor who conducted the 3DFUS trials and digitized each frame was only moderately experienced. Thus, our results might still slightly underestimate the possible achievable accuracy as it can be difficult to accurately identify the bony surface in each ultrasound image. Nonetheless, our obtained results suggest that 3DFUS can serve as an accurate technique to register bony surface in individuals where palpation is difficult.To investigate the accuracy of our 3DFUS-based approach in segmenting bone surface we have conducted extensive water-bath experiments with a plastic pelvis measurement phantom. A 3D laser scan of this phantom served as the ground truth. The results indicated an acceptable level of accuracy  could be easily fused to the position of the anatomical landmarks identified by 3DFUS in the motion capture data. Fusion could be performed by optimizing the point cloud of a volumetric medical image into the 3DFUS points by global point-wise optimization such as iterative closest point algorithms40. The combination of statistical shape models and sparse 3DFUS data of the pelvis might also allow to reconstruct the pelvic geometry42. Such approaches could open new perspectives for personalizing biomechanical models in 3D clinical gait analysis. However, applicability and feasibility of such approaches are subject to future research.The identification of anatomical landmarks in motion capture data via 3DFUS might allow to fuse volumetric medical imagine data to gait analysis data. Gasparutto et al.19. They have shown that 3DFUS bone segmentation can be performed to a certain extent with automated image processing techniques to reduce post-processing time. However, they did this for the trochanter major only, which is relatively easy to identify. Results presented in this study are limited to static calibration of a biomechanical model, but STA are especially an issue during movement trials. Therefore, 3DFUS-based approaches also need to be expanded to dynamic situations. First steps in that direction were already presented by Jia et al.19. Our study cohort only included children and adolescents. In several cases ossification of bone was still not completed and made the identification of the bone surface areas more difficult. Next to that, the relative size and composition of the pelvic bones in this young group might also limit generalizability to a certain extent.This study has some limitations which need to be recognized. So far, our 3DFUS-based approach seems to have image-quality related limitations. The utilized ultrasound probe showed a maximum penetration depth of approximately 6–7 cm. We experienced difficulties in obtaining good ultrasound images in six individuals, wherein we had to exclude 11 trials (legs) from the estimation of the HJC. The excluded cases had a BMI of 15 that 3DFUS can serve as an accurate alternative to identify the HJC position, especially when volumetric medical images such as bi-planar X-ray or CT scans, which expose patients to radiation, are not possible. Yet, 3DFUS is still time-consuming, needs a lot of manual post-processing, and is susceptible to outliers. Therefore, future developments should emphasis to increase technical robustness and further develop computer-assisted post-processing techniques to decrease processing time. This will help to make its application to clinical practice more viable.In conclusion, our results give raise to the idea that an 3DFUS-based approach could serve as a reliable and accurate alternative in identifying anatomical landmarks such as the ASIS and the HJC in individuals especially with higher amounts of body fat. It also further corroborates the results of Peters et al.Supplementary Information 1.


Prior work on automated methods demonstrated that it is possible to recognize pain intensity from frontal faces in videos, while there is an assumption that humans are very adept at this task compared to machines. In this paper, we investigate whether such an assumption is correct by comparing the results achieved by two human observers with the results achieved by a Random Forest classifier (RFc) baseline model  and by three proposed automated models. The first proposed model is a Random Forest classifying descriptors of Action Unit (AU) time series; the second is a modified MobileNetV2 CNN classifying face images that combine three points in time; and the third is a custom deep network combining two CNN branches using the same input as for MobileNetV2 plus knowledge of the RFc. We conduct experiments with X-ITE phasic pain database, which comprises videotaped responses to heat and electrical pain stimuli, each of three intensities. Distinguishing these six stimulation types plus no stimulation was the main 7-class classification task for the human observers and automated approaches. Further, we conducted reduced 5-class and 3-class classification experiments, applied Multi-task learning, and a newly suggested sample weighting method. Experimental results show that the pain assessments of the human observers are significantly better than guessing and perform better than the automatic baseline approach (RFc-BL) by about 1%; however, the human performance is quite poor due to the challenge that pain that is ethically allowed to be induced in experimental studies often does not show up in facial reaction. We discovered that downweighting those samples during training improves the performance for all samples. The proposed RFc and two-CNNs models (using the proposed sample weighting) significantly outperformed the human observer by about 6% and 7%, respectively. Several vulnerable groups, such as infants, intensive care patients, or people suffering from dementia, are not able to self-report their pain , and reqAt the heart of this research is evidence that the machine is effective compared to human in recognising phasic pain intensity based on analysis of facial expressions in video. Further, the proposed sample weighting method improved the performance of some automated models of pain assessment by downweighting noisy samples. We introduce some automatic pain intensity recognition methods that outperform human observers on the X-ITE database , which iAfter observing that misclassified samples often contain low or no facial response to pain   has beenSharing knowledge among tasks in Multi-Task Learning is often useful when the tasks are similar. It is generally applied by sharing hidden layers while keeping several task-specific output layers. It can help speed up the learning process and reduce overfitting by using shared parameters ,37. In tThis paper, to the best of our knowledge, reports the first comparison between the performance of automatic methods and the performance of human observer for recognising pain intensity on the X-ITE pain database based on facial expression analysis. We have three reasons for comparing machine pain recognition with human performance: (1) Humans are able to recognise and report pain due to their sensitivity to read pain from facial expression ; (2) HumAnother major contribution is the extensive experimental validation of several automated recognition approaches including Random Forest and deep neural network classifiers, multi-task learning, and a newly suggested sample weighting method.We use the Experimentally Induced Thermal and Electrical (X-ITE) Pain Database see . Within In this section, we give an overview of the X-ITE Pain Database in healthy human participants, which is used to evaluate the performance of different pain intensity recognition methods , see FigThis subsection introduces an overview of the two proposed Random Forest classifier (RFc) methods that have the same structure with different input types: Random Forest classifier (RFc) baseline method  using a new representation for RGB images, and Random Forest classifier (RFc) method using Facial Activity Descriptor (FAD) of facial action unit time-series.In this work, RFc  with 500We additionally applied the RFc  with 500This section describes the pre-processsing process and the proposed deep learning architectures that were used to predict the pain intensity from facial video.We applied pre-processing on raw data to obtain the right format of the input to MobileNetV2 (MNV2) architecture. The input image to MNV2 with transfer learning has to be 3 channel (RGB) format, whereas each sample of our data is a video of 7 s length. OpenFace is used for face detection and registering face images from each video. Three images were selected from second 1, 3 and 4 for each video based on the study by Werner et al. . They anAdditional input for a second branch network was generated to improve the performance of deep learning models. It consists of 28 × 28 single channel images containing the RFc output predictions of the corresponding sample encoded as numbers 0 to 6. These images are used as the second input of the networks, along with the corresponding image that was generated as illustrated in Our own CNN architecture comprises six 2D convolution layers (Conv), each followed by ReLU. The first two Conv have 16 output channels. After each two Conv, we apply a 3 × 3 max-pooling with stride 2 and double the output channels of the next two Conv. The last two Conv have 64 output channels and are followed by global average pooling. The final layer is one dense layer with 1024 neurons that is activated by ReLU. This architecture is used to build the final proposed deep learning architectures, which are presented in the following sections.This section describes how we used transfer learning with a reduced MobileNetV2 (MNV2)  to prediOur proposed two-CNNs architecture for the pain intensity recognition task is a combination of two simple CNN architectures, one used for the X-ITE images (RGB images) and the other for RFc prediction images (single channel). We combined the two simple CNNs by adding three dense layers that are activated by ReLU, containing 1024, 512 and 128 neurons, respectively. The final dense output layer is activated by the softmax function. Similar to reduced MNV2 with simple CNN, the obtained model is trained for 150 epochs with For learning new tasks, we humans often apply the knowledge we have acquired by learning related tasks. This principle is similarly applied in MTL, which has been used successfully across many applications of machine learning and computer vision. In our MTL approach, the deep learning models learn to distinguish pain types, i.e., no pain versus electrical pain versus heat pain, and can be apply this knowledge to another task, which is no pain versus pain intensities. Often one of the sub-tasks can help to improve the performance of other sub-tasks, for example by hard parameter sharing, which is the most commonly used MTL approach in deep learning . It is gWe applied MTL on the proposed deep learning architectures Reduced MobileNetV2 (MNV2) with Simple CNN (MNV2-CNN) and Two-Convolutional Neural Networks (Two-CNNs), however with two output layers instead of one: The first output layer is to recognise baseline (pain intensity 0) versus heat stimuli (H) and electricity stimuli (E). The second output layer is to recognise baseline (pain intensity 0) and pain of the intensities 1, 2, and 3. The two output layers (for two classification tasks) are independently activated with softmax functions. We also experimented with modelling the intensity recognition as a regression task with two alternative versions. In one, the intensity output is activated linearly and trained by optimizing mean squared error (mse) loss. The other alternative uses sigmoid activation and cross-entropy loss. In all cases, the modality classification (first output) uses softmax activation.After observing that misclassified samples often contain low or no facial response to pain , lower reaction (low heat-pain stimuli = H1), low reaction , less moderat reaction (moderate heat-pain stimuli = H2), moderate reaction , severe reaction (severe heat-pain stimuli = H3), and more severe reaction . After the observers completed their class assignment, the labels were compared with ground truth labels. The human observers reached an accuracy of 21.1%. So they outperformed guessing  by about 7% and outperformed the automatic baseline approach (RFc-BL) by about 1%. The result is far from the desired 100% classification rate, which illustrates how hard this classification problem actually is. Human observers had difficulty distinguishing between no pain versus lower pain due to stimulation, because there often was no facial expression reaction. Pain is expressed differently in individuals. Some persons show no facial reactions for low stimulation, others show even no facial expression during high stimulation. Therefore, human observers and automatic methods cannot accurately distinguish pain intensity in the X-ITE experimental pain dataset, which is in line with findings made on the BioVid Heat Pain Database . To compThis section describes the experiments. The experiments were carried out in order to gain insights into pain intensity recognition and to compare the performance of the 3 automatic pain recognition models to the performance of a baseline model and human observation. Further experiments were: Multi-Task learning (MTL) was applied on the proposed deep learning networks, and deep learning models were trained again using the extended databases that were obtained after applying the proposed sample weighting. The accuracy measure is used to measure performance, because it is intuitive and suitable for the data, whose classes were approximately balanced. Stratified 5-fold cross-validation without subject overlap is applied to measure generalization. We also evaluate if the results of proposed models are statistically significantly different compared to human observation results based on the p-values, which were obtained through a paired t-test, with the 7-class classification task.The next section shows the experiments results of (1) full 7-class classification of phasic pain, (2) 5-class pain intensity recognition, and (3) 3-class pain intensity recognition. We considered the heat and electrical pain , and the four pain intensities (ranging from no pain to high pain). No pain  is denoted with the letter B, electrical pain with E and heat pain with H. The letters E and H are followed by a number between 1 and 3 denoting the stimulus intensity. Due to the results of the 7-class classification, only RFc, CNN and CNN* models were trained with the 5-class classification task. For each pain stimulation modality , we excluded one of the two lower pain intensities based on their low performance in ; the sugBased on the high performance of the 5-class classification, 3 classes for each stimulation method  are considered; we excluded H1 or H2 in phasic heat stimuli and E1 or E2 in phasic electrical stimuli, see intensity with facial expressions in X-ITE database, and (2) to compare the performance of automatic methods to that of human observer. In line with prior work, the results show that it is possible to recognise no pain versus pain intensities and qualities  in acute phasic pain using automatic approaches. According to the obtained results, the recognition of electrical pain stimuli performs better than the recognition of heat pain stimuli using automatic pain recognition. This leads to the hypothesis that electrical pain stimuli may work better due to the earlier and more rapid responses than thermal pain responses. The best obtained result for 7-class classification task is 27.8% when using CNNs* model, and the average accuracy from the best models for 5-class classification task is 37.7%; both significantly above the chance level by 13.6% and 17.8% and above automatic baseline model (RFc-BL) by 8.4% and 11%, respectively. By excluding one of lower phasic pain intensities, pain recognition improved more by about 4% above chance and 3% above RFc-BL. The reason is the difficulty in distinguishing between the lower pain intensity stimulation and baseline, in many cases due to a lack of an observable reaction [We conducted several experiments in order to (1) gain insights into recognising pain In our experiments, RFc and CNN models performed similar. This may be surprising, because deep learning has outperformed conventional methods in many computer vision and machine learning tasks in the last years. However, in most cases the success of deep learning is based on its ability to benefit from huge datasets. In automated pain recognition, the dataset sizes are quite small, in our case probably too small to learn significantly beyond the domain knowledge used in the RFc method. With larger datasets or better regularization, deep learning may outperform RFc in future studies.One major goal of automatic pain recognition is to support medical practitioners with assessing pain of patients who cannot utter on their pain experience. The current clinical practice requires a human to observe the behaviour of these patients for assessing their pain . In the When comparing with the human performance, we found that some of our automatic models not only performed similar, but significantly outperformed two human observers. The best of our proposed models (CNN*) reached an accuracy of 27.8%, which is about 7% above the performance of two humans. Thus, we believe that the pain expression recognition technology is ready for clinical trials comparing automated tools with clinically established pain assessment.As we have seen in the quantitative results in discussion section, automatic pain recognitions models and human observers often confuse no pain and low, moderate, and severe pain of both pain modalities . As we have seen in the quantitative results in this section, Some subjects show a lack of facial responses to pain: some have low pain sensitivity resulting in a high tolerance threshold requiring a temperature cutoff to avoid burns; others show low tolerance threshold intentionally or unintentionally during stimulus calibration, possibly because they do not want to feel severe pain. Such inconsistencies between the label and the video may be considered outliers or label noise. Data cleaning by removing such outlier samples may be used for improving the facial expression-based recognition performance. However, this may remove some samples, which are useful for improving multimodal pain recognition because there may be pain responses in other modalities like EMG or EDA although no facial responses shows up. Thus, the proposed sample weighting method could be useful for improving the performance of the multimodal pain assessment by downweighting noisy samples rather than eliminating them as in the cleaning up strategy. In addition, for future data acquisition, advanced study designs and more reliable pain ground truth would be beneficial.There are some risk factors of using this technology: (1) The database (X-ITE) does not contain any of a vulnerable group, we believe that this system can help to predict pain particularly in vulnerable patients but have not yet implemented it to them; (2) Kunz and Lautenbacher  and Rash


Previous studies on grass species suggested that the total centromere size  may be determined by the genome size, possibly because stable scaling is important for proper cell division. However, it is unclear whether this relationship is universal. Here we analyze the total centromere size using the CenH3-immunofluorescence area as a proxy in 130 taxa including plants, animals, fungi, and protists. We verified the reliability of our methodological approach by comparing our measurements with available ChIP-seq-based measurements of the size of CenH3-binding domains. Data based on these two independent methods showed the same positive relationship between the total centromere size and genome size. Our results demonstrate that the genome size is a strong predictor (R-squared = 0.964) of the total centromere size universally across Eukaryotes. We also show that this relationship is independent of phylogenetic relatedness and centromere type , implying a common mechanism maintaining stable total centromere size in Eukaryotes. Thus, the centromere plays a vital role in the cell division of Eukaryotes, as it mediates the proper segregation of chromosomes into daughter cells. In most Eukaryotes, the centromeric function is defined epigenetically by the presence of centromeric histone H3 (CenH3 or CENP-A), which recruits other kinetochore proteins2, although CenH3-independent systems also exist, e.g., holocentric insects3, some holocentric plants4, kinetoplastids5, and some fungi6. Centromeric DNA is usually AT-rich1, but specific DNA sequences are not necessary nor sufficient for the kinetochore assembly2.The centromere is the chromosomal region where the kinetochore, a protein complex that mediates the chromosome’s attachment to spindle microtubules, assembles10. Forty years ago, Bennett et al.11 analyzed nine species of grasses (Poaceae) with electron microscopy and found that the total centromere volume  linearly scales with the nuclear DNA content (genome size) across these species. Zhang and Dawe7 confirmed this relationship by analyzing a partially overlapping set of ten grass species and showing that the total size of the CenH3-immunostained area  strongly positively correlates with genome size. A positive, although much weaker, relationship has also been observed within species across 26 maize lines differing in genome size10. These results indicate that there may be a mechanism maintaining the stable proportion of total centromere size to the genome size that is based on general intracellular scaling principles7. This notion was supported by observation in maize/oat and maize/maize hybrids that centromeres may expand when chromosomes are introduced into larger genomes10. However, it is unclear whether the scaling of total centromere size to genome size is a universal phenomenon because it was observed on grasses only.The size of CenH3-containing domains determines the kinetochore size and thus the functional centromere size12), or holocentric chromosomes .Therefore, in the present study, we have measured the total centromere size (using immunostained-CenH3 areas as a proxy) in 130 eukaryotic species, including plants, animals, fungi, and protists Fig. , and tes7 with the authors’ own values obtained as averages of measurements of multiple nuclei from high-resolution images7. We observed a very good agreement  , we wanted to verify the validity of our approach. First, we compared the total centromere size for 10 grass species that we measured from Fig. 2 = 0.964), suggesting that the genome size is a strong predictor of the total centromere size.We found that the centromere size strongly positively correlated with the genome size Fig. . The detTo further verify our results, for 15 monocentric species of our dataset, we collected data reported in previous studies on total centromere sizes derived independently from ChIP-seq measurements of CenH3-binding areas  means that it remains the same whether we look at closely or distantly related species, implying that Eukaryotes share a mechanism maintaining a stable proportion of functional centromere to genome size. The potential mechanism also appears the same for taxa with monocentric, metapolycentric, or holocentric chromosomes because the relationship between total centromere size and genome size does not change with chromosome type Fig. , and the26. Chromosomes whose kinetochore size falls below this limit are more likely to be lost during repeated rounds of cell division26. Thus, a sufficiently significant increase in chromosome size could require a corresponding increase in kinetochore size, and/or an increase in kinetochore size could allow an increase in chromosome size. Changes in the size of individual kinetochores could occur either by drift or as a result of deterministic processes such as centromeric or holokinetic drive28. It is, therefore, possible there are two antagonistic processes affecting centromere size. The first equalizes the size of individual centromeres (and possibly even entire chromosomes) to ensure proper chromosome behavior during cell division on a cellular level. By contrast, the second process operates on the level of individual chromosomes and may cause centromere and chromosome size divergence within a single karyotype. As the correlation between the total centromere size and genome size  is much stronger than the within-karyotype correlation between sizes of individual chromosomes and their centromeres16, it seems likely that the mechanism keeping centromeres of similar sizes prevails.The reason for such a within-karyotype correlation is unclear, but it appears that for a chromosome of a specific size, there is a lower limit of kinetochore size reflecting the minimum number of kinetochore microtubules required for proper chromosomal segregation29 as follows: (i) We applied split channels and adjust threshold functions to obtain and separate the CenH3-staining area  from the DAPI-stainined DNA area (a proxy for genome size). First, we used the auto-adjust threshold option, and then we fine-tuned the threshold manually to properly circumscribe the DAPI or CenH3 areas. (ii) We measured the size of CenH3-staining and DAPI-stainined DNA areas. (iii) To enable the comparison of area measurements between photographs from different studies/species, we standardized all the measurements using the known 1C genome size for each species as follows: we equalized the measured DAPI area with the known 1C genome size (in Mbp) and calculated the total centromere size in Mbp as [/DAPI staining area]. We had one measurement of the DNA and CenH3 area for each species whose figure was obtained from the literature. For each of the eight grass and nine agavoid species that we newly measured, we measured the DNA and CenH3 area in 6 to 27 nuclei per species (Supplementary Table 30, from the Plant C-values Database31, from Genome Size of the Czech Vascular Flora32, or from the Fungal Genome Size Database33. In the case of the nine Agavoideae species, we measured their genome size using flow cytometry (see Supplementary Text We reviewed the available literature and collected studies containing microscopic photographs of immunolabeled CenH3 (Supplementary Table The values for the total centromere size based on ChIP-seq measurements were obtained from the published papers where they were reported either in the text, tables, or supplementary data. The values in Mbp as well as the references to the respective papers are listed in Supplementary Table pgls function implemented in the package caper34 in R v4.0.235. We set the total centromere size as a response variable and genome size and chromosome type as explanatory variables. Both the total centromere size and genome size were log-transformed before the regression analysis to increase the homogeneity of variances in the response variable. First, we explored the model with the interaction between explanatory variables to check whether the effect of genome size on the total centromere size depends on chromosome type. The interaction was not significant (Supplementary Table 36, we used a dated phylogeny, as it fulfills this assumption. The dated phylogenetic tree (Supplementary Text 37 and the comprehensive dated phylogeny of Angiosperms38. If a species was not present in the trees, we replaced their tip with the closest relative or added it manually based on the published phylogenies as in the case of Allium39 and Cuscuta40.To test the relationship between the total centromere size and genome size, while accounting for a potential non-independence of species due to their shared ancestry, we used a phylogenetically corrected linear regression using the All methods were performed in accordance with the relevant guidelines and regulations.Supplementary Information.