The cycle detection technique showed an accuracy of 97.6% in a Bland–Altman analysis and a 94% accuracy (F1-score of 0.93) in predicting the severity of the FNT. Furthermore, the dependency of the upper-extremity tests was investigated through statistical analysis, and the results confirm dependency and potential redundancies in the upper-extremity SARA assessments. Our findings pave the way to enhance the utility of objective measures of SCA assessments. The proposed wearable-based platform has the potential to eliminate subjectivity and inter-rater variabilities in assessing ataxia.The study presents a novel approach to objectively assessing the upper-extremity motor symptoms in spinocerebellar ataxia (SCA) using data collected via a wearable sensor worn on the patient’s wrist during upper-extremity tasks associated with the Assessment and Rating of Ataxia (SARA). First, we developed an algorithm for detecting/extracting the cycles of the finger-to-nose test (FNT). We extracted multiple features from the detected cycles and identified features and parameters correlated with the SARA scores. Additionally, we developed models to predict the severity of symptoms based on the FNT. The proposed technique was validated on a dataset comprising the seventeen ( Spinocerebellar ataxia (SCA) is characterized by motor dysfunction caused by degenerative changes in the cerebellum . SCA-relMoreover, objective measures may reveal patterns of motor control dysfunction below the threshold of detection even by subspecialty-trained clinical ataxiologists. More specifically, quantitative assessments in pre-symptomatic or early disease stages in neurodegenerative diseases may detect subtle changes in motor abilities that would not be sufficient to change the ordinal rating scale score from one severity level to the next on available clinical rating scales ,10,11. TAlthough these systems can provide the intended information for specific disease severity aspects, they have their limitations. For instance, using a system consisting of special video cameras, touch screens, and a full-size computer to sync various components to track a patient’s finger trajectory in the FCT may provide precise quantification of movement dynamics. However, it still requires the patients to travel to a clinic because such a data acquisition system is neither widely accessible nor easy to operate for non-specialists. The other challenge, especially in remote assessments, is to devise an automated technique in all its subcomponents. For instance, the average cycle time for a cyclic test such as the FNT, which is the focus of this study, is a widely used feature to analyze the results of this test. An objective assessment algorithm should detect cycles to extract relevant features from the associated data within each cycle and predict the severity score. Such a comprehensive procedure represents an important gap in the literature.The objectives of this study were to develop a wearable-based solution that can objectively assess upper-extremity motor symptoms using wearable sensors. In addition, we investigated possible redundancies in the upper-extremity SARA assessments. In doing so, we made the following contributions: First, we developed an algorithm for detecting/extracting the cycles of the FNT. We also extracted multiple features from the detected cycles, investigated their correlation with the SARA scores, and developed models to predict the severity of symptoms based on the FNT. Second, we studied the dependency between the upper-extremity SARA tests through correlation analyses and building predictive models to estimate the severity of symptoms based on the FCT and DDKT using the extracted features from the FNT. Our findings pave the way for the enhancement of the utility of objective measures of SCA assessments using wearable sensors and complement our previous study focused on objective assessments of gait and balance in SCA .n = 1, SCA2: n = 3, SCA3: n = 5, SCA6: n = 5) and three healthy controls were enrolled. A subspecialty ataxiologist (JDS) determined the eligibility for the study participation. Exclusion criteria were less than 18 or more than 75 years of age, inability to walk independently even with a walker (SARA or BARS gait score > 6), inability to comply with all study activities, or unwillingness to provide informed consent. The inability to walk was an exclusion criterion because this study was part of a comprehensive investigation of both upper and lower limb dysfunction and gait.This study protocol was approved by the Partners HealthCare System Institutional Review Board. All participants signed the informed consent form before participating in the study. Seventeen participants were recruited from the Massachusetts General Hospital (MGH) Ataxia Center and studied at the MGH Neurological Clinical Research Institute (NCRI). Fourteen patients with a genetically confirmed diagnosis of SCA  on their wrists during the assessments  which utilizes half-points according to designated descriptors included in the scale ,9. All pessments . Each seThis study only assessed the upper-extremity SARA assessment of the FNT, FCT, and DDKT. The FNT measures how smooth and coordinated the upper-extremity movement is by having the subject perform a cyclic task. In this task, the clinician holds his/her finger at 90% of their arm’s length from the patient, and the patient is instructed to touch the clinician’s finger and then his/her nose in several consecutive trials. For the FCT, the clinician performs five consecutive sudden and fast pointing movements in unpredictable directions in a frontal plane. The amplitude and frequency of the movements are, respectively, about 30 cm and 1 movement per 2 s. The patient is asked to follow the movements with his/her index finger as fast and precisely as possible, and the average performance of the last three iterations is rated. Finally, the DDKT measures how quickly and accurately one can repeat a rapid alternating pronation/supination task by asking the patient to tap the palm of one hand to his/her thigh, raise and flip the hand, tap the back of the hand to the thigh, raise and flip the hand, and repeat this cycle ten times with the regularity of the movement and time taken to perform 10 iterations recorded.Phase 1 (Decline): from the time instant that the patient’s finger starts moving from the clinician’s finger until reaching their nose.Phase 2 (Pause at the nose): the period between the instant that the patient’s finger reaches their nose until they start moving towards the clinician’s finger.Phase 3 (Rise): from the instant that the patient’s finger starts moving from their nose until they reach the clinician’s finger.Phase 4 (Pause at finger): the period between the instant that the patient’s finger reaches the clinician’s finger until they start moving towards their nose.In this study, we focused on the cyclic FNT task. The objective of the proposed algorithm was to extract each cycle of the FNT from the sensor-derived kinematic signal. The proposed algorithms detect each cycle and identify different phases within each cycle as follows:The duration of phases 2 and 3 may be zero if the participant does not pause when they reach their nose or the clinician’s finger. The segmented signal is then used to extract features dependent on the disease severity.We used the Euler angles to extract the four phases of each cycle derived from the quaternion values provided by the sensors. First, the sensor quaternion is calibrated for the local fixed frame of the clinician’s finger, and then the Euler angles are estimated using the calibrated quaternion. A calibrated time series for the right hand of a healthy individual and a participant with SCA is shown in To further clarify this, consider the angle signal of one FNT cycle for an SCA participant in Finally, there is a rapid change in angle from D, which corresponds to phase 3, until the participant reaches the clinician’s hand again at E. Therefore, by detecting the A, B, C, D, and E points in each cycle of the signal, the four phases of the FNT motion can be detected. Note that the direction of rotation from finger to nose is counterclockwise for the right hand, while it is clockwise for the left hand; thus, the left-hand measurements can be similarly analyzed after changing the sign of the Euler angles.The measured signals were filtered with a sixth-order band-pass Butterworth filter with cut-off frequencies of 0.2 and 20 Hz. The lower bound cut-off was used to minimize the drift effects, and the exclusion of high-frequency components was due to the restriction in the maximum frequency that can be generated by human movements .x, y, and z are displacement components along the sensor’s local axes, the double dot is the second time derivative, (θ) is the angular velocity, its subscripts represent the axis of rotation, and a single dot is the first time derivative. The speed metric is defined as the mean value of speed divided by the maximum speed [Two types of features are extracted from the sensor data. The first type is extracted from each FNT cycle and was extracted using the cycle detection algorithm, and the second type of features is extracted from the entire signal of a complete FNT task. We only considered the clinically relevant features and categorized them into five phenotypes: timing, speed, variability, rhythmicity, and steadiness. All features considered in this study are summarized in um speed , where sy2˙+θz2˙MILA=x¨2+ity data .Extracting these features from the angular velocity and linear acceleration in all directions and the motion intensities results in 140 features. Note that there are two types of features in general; the first type has a single value for an entire FNT, such as the maximum and range of AV, and the second type has a single value for each cycle and, hence, multiple values for each test, such as the cycle time. Herein, we call the second type of features cycle-based features. To combine the two types of features, first, an interquartile range (IQR) outlier detection was used to remove the outliers from the cycle-based features. The IQR, denoted by The intraclass correlation coefficient (ICC) was performed to determine the test-retest and inter-rater reliability of the FNT Score from SARA and BARS scales . The ICC reflected the degree of correspondence and the degree of agreement between the scorings by the three raters. Reliability was considered good if the ICC was greater than 0.75 and fair if the ICC was between 0.40–0.75 . Table 2The proposed cycle detection algorithm was tested on the collected dataset, and the results were compared with a ground truth obtained by extracting each cycle’s start/end from the recorded videos. d effect size, the mean value, STD, and the mean difference percentage of the selected features for the two control and SCA participants are also shown in A Spearman rank correlation analysis was conducted to identify the features with the highest correlation to the FNT SARA score. A Mann–Whitney signed-rank test was used to determine if the extracted features are statistically significant for discriminating between the control and SCA participants. The features that were correlated with the FNT SARA score and statistically significant (<0.05) according to the Mann–Whitney test are listed in The available dataset is relatively small and highly unbalanced. Having 17 participants who are tested two times for each hand and noting that sometimes the rater asks a participant to redo the test in either of the trials, the dataset consists of 34 and 31 distinct tests for the first and second trials, respectively. The distribution of SARA scores in this dataset is summarized in For training and predicting the SARA scores, the data points associated with SARA scores of two and three were grouped and categorized as moderate SCA to make the dataset more balanced. SARA scores of zero and one were considered control and mild SCA categories, respectively. The distribution of the dataset in these categories is illustrated in We used a gradient boosting classifier (GBC), logistic regression (LR) with elastic-net regularization, and support vector classifier (SVC) with radial basis function (RBF) kernel to predict the severity of SCA in our dataset. The models were trained using the data from trial 1, and their performances were evaluated using the data from trial 2. Principal component analysis (PCA) was conducted on the feature matrix to reduce the dimensionality and manage collinearity between features. The GBC was trained on both the original features and all principal components (PCs). The LR was trained using the PCs only due to the sensitivity of this model to features’ collinearity. Finally, the SVC was trained along with a random subspace feature selection scheme using the PCs. All the hyper-parameters of these models were optimized through cross-validation on the data of trial 1. p-values are presented, except for the features with p-values > 0.05, i.e., not significantly correlated with a specific test. For instance, “STD of cycle time” is correlated with the FNT and DDKT but not with the FCT.Although each motor task  provides unique information about movement dysfunction, there might be an inherent redundancy among these motor tasks ,20. The To better capture the dependencies of the upper-extremity SARA tests, classification analyses were performed to predict the SCA severity of the FCT and DDKT using the features that were extracted from the FNT. The distribution of the sample points for the FCT and DDKT is presented in The results show a strong dependency between the FNT features and the SARA scores of DDKT and the FCT, although they may not be significantly correlated. Additionally, this dependency can be captured by a nonlinear kernel such as the RBF. This demonstrates the redundancies in upper-extremity SARA tests for predicting the SARA scores or SCA severity despite their unique aspects in showing different types of disabilities. Note that we only used clinically meaningful features and a small dataset in this study; thus, the predictions are expected to be improved by using more advanced features on a larger dataset.We have developed a wearable-based solution for the objective assessment of the upper-extremity SARA test. Seventeen participants were recruited from the MGH Ataxia Center and studied in the MGH NCRI. Participants underwent a standardized ataxia evaluation using the SARA and BARS2 while wearing a single sensor on each wrist. We observed excellent inter-rater reliability for the SARA and BARS2 for all participants among the three raters, one of whom rated the participants in person and the others rated based on videos . These rA cycle detection algorithm was developed for extracting the cycles of the FNT from the sensor measurements and segmenting each cycle data into four main phases, i.e., decline, pause at the nose, rise, and pause at the finger. The algorithm extracted 140 features from the segmented and full sensor measurements. The estimated features were correlated with the SARA scores and could accurately discriminate the control and SCA severity groups related to the FNT. The benefit of the proposed objective assessment over conventional subjective evaluation includes the precise measurement of movement. Due to high sensitivity in tracking hand movement and detecting smaller vibrations , the proposed solution may identify subtle motor symptoms, which are often challenging to detect visually.Unlike previous studies that used wearable-based validated gait assessment tools to assess SCA severity ,21, the Furthermore, the dependency of the upper-extremity SARA tests was investigated via training machine learning algorithms to predict the SARA scores related to the FCT and DDKT using the features obtained from the FNT. The high accuracy of the SVC with the nonlinear RBF kernel implies that the upper-extremity SARA tests are dependent, although their linear correlation may be insignificant. The results suggest that the number of motor tasks could be potentially reduced while estimating the upper-extremity motor symptoms with acceptable accuracy. A reduced number of motor tasks may improve the feasibility of using the platform in a non-clinical setting and for remote patient monitoring. Digital health technologies are poised to become an integral part of modern health care, expedited by the surge in remote care necessitated by the COVID-19 pandemic ,29,30,31Previously, Kasyap et al. (2020) proposed a comprehensive approach to the evaluation of cerebellar ataxia by objectively assessing five domains  through the instrumented versions of nine bedside neurological tests using wearable IMUs and microphones . SimilarA major limitation of the present study is the small sample size. While the high-performance accuracy of the current models is encouraging, the results need to be confirmed in a larger dataset. Furthermore, the models were trained on trial 1 and tested on trial 2. Since we have the same participants in trial 1 and trial 2, the model might have learned individual characteristics in addition to disease characteristics. A further limitation was the absence of late-stage severe SCA patients in whom somatosensory impairment, vestibular involvement, or other central nervous systems (CNS) lesions may contribute to the overall disability . It is iOur findings pave the way to enhancing the utility of objective measures of SCA assessments using wearable sensors, and complement our previous study focused on objective assessments of gait and balance in SCA. The proposed wearable-based platform has the potential to eliminate subjectivity and inter-rater variabilities in assessing ataxia. It can also facilitate frequent remote patient assessments that may compensate for the impact of biological variability in intermittent single visits. The findings of this study need to be validated in a larger sample.


Using a cross-polarization transmission geometry, stress maps for the normalized birefringence and intrinsic stress direction of polymeric materials may be obtained using terahertz nondestructive evaluation. The analysis method utilizes a deconvolution method to determine the arrival times and amplitude of the cross-polarized terahertz pulses through a birefringent material. Using amber  as a material of interest, stress maps show that inclusion-free Lebanese amber samples behave as classic uniaxial birefringent (photoelastic) materials whose principal stress directions, as inferred in the terahertz spectral range, agree well with visible photoelasticity measurements. Since amber samples, depending upon their source, may be either transparent or opaque to visible light, comparing birefringence measurements in the visible and terahertz spectral ranges cross-validates the stress measurements, thereby establishing a strong and unique stress analysis methodology for visibly opaque samples. While the material of interest for this paper is amber, the method is generally applicable for any terahertz-transparent polymer. The cross-polarization experimental configuration enables stress levels within the amber matrix to be visualized while also outlining highly localized regions of stress surrounding inclusions. Birefringence stress maps clearly show localized increases in stress magnitude and directional changes surrounding inclusions. Over the past twenty years, terahertz (THz) imaging and spectroscopy has been applied to the nondestructive evaluation (NDE) of materials . TeraherAmber derives from fossilized plant resin and uniquely preserves aspects of ancient ecosystems. Plants exude resins in response to damage. These resins, when subjected to heat and pressure over extended periods of time, undergo polymerization that leads to fossilization . Biotic Amber-based research focuses primarily on the description and characterization of organic inclusions. Amber is frequently optically transparent, and as such light microscopy is most often employed in sample analysis. However, other bands of electromagnetic radiation are used to resolve inclusions and determine the provenance and material properties of amber itself. Techniques outside of the visible light spectrum include confocal laser scanning microscopy (CLSM) , infrareWhile light microscopy, X-ray imaging, Raman spectroscopy, and infrared spectroscopy  remain tHere, the application and utility of terahertz birefringence in the characterization of amber material properties and the detection of inclusions in amber is investigated. Because fossil specimens, particularly those relating to the description of new species (holotypes), are irreplicable and frequently trimmed to better visualize and conserve specimens , the mapIn a homogeneous material, the speed at which an electromagnetic wave propagates through the material  is independent of the polarization direction of the wave’s electric field direction. However, it is well known that when mechanical stress is present (either intrinsic or applied stress), the refractive index of the material depends on the polarization of the wave relative to the stress direction present within creating what is known as a photoelastic material . In its Since amber precursors are exuded from plants, there is a flow associated with their creation. This flow, as the materials polymerize over time, will create a birefringence within the resulting amber due to varying levels of stress within, creating a photoelastic material. Birefringence induces a relative phase difference between waves polarized parallel to and perpendicular to the stress direction. In this paper, the measurement of the intrinsic birefringence and stress of amber samples is emphasized. This measurable birefringence can aid in both the understanding of the origins of the amber samples as well as in the detection of inclusions since intrinsic stress tends to concentrate at the inclusions, both in the visual and terahertz spectral ranges. Measuring the birefringence can also allow for the determination of the direction of the stress within the amber, which can further be used to obtain insight into the flow and origin of the amber measured. Prior measurements on birefringence from amber inclusions show that birefringence in polarized visible light is a property of certain organic inclusions. Setae, which are found in arthropods and are analogous to hairs found in vertebrates, exhibit birefringence. Because setal composition is used in the identification of certain groups of organisms, measurable birefringence has been used to identify amber inclusions, for example in oribatid mites .The primary aim of this study was to describe the birefringent properties of amber with and without inclusions. The secondary aim of this study was to use an index-matching liquid to remove the refractive effects of the air–amber boundary. A major complication that terahertz imaging faces is refraction at the boundaries of materials due to differences in refractive index. This refraction can radically change the direction of wave propagation, thereby violating the parallel ray projection approximation which is typically assumed in tomographic image reconstruction . Index-mFour amber samples were used in this study. The first sample  was from Chiapas, Mexico and was deposited during the Miocene 16 million years ago (Ma). Sample AMNH-Bal_Mex1_B had been trimmed and polished so that it had two opposing and approximately parallel faces. The second sample  was from Bcharre, Lebanon, was deposited during the Early Cretaceous 125–129 Ma and was used for visual and THz birefringence imaging. The third amber sample (BALDR_0076) was from the Dominican Republic, was also dated to the Miocene 16 Ma, and was used for THz-CT both for its distinct shape and present inclusions in the form of two termite workers from the species Mastotermes electrodominicus. The fourth sample (BALDR_0239) was also from the Dominican Republic and was chosen for polarization imaging analysis due to the large number of bubbles which are trapped in the amber. The amber samples were typically mounted to various standard optical posts using dental wax.The samples were imaged and spectroscopically characterized using a terahertz time-domain spectroscopy system. In this methodology, short time-duration  electromagnetic pulses are generated and detected using optoelectronic techniques ,41. UsinAn illustration of the experimental setup is shown in g et al. . Rather For simplicity, it was assumed that the sample can be treated as a uniaxial birefringent material in which the principal stress axes are perpendicular to the propagation direction of the terahertz waves. Clearly, this assumption may not always be true for amber samples considering the potential for variable resin flows and/or stresses which may be present during polymerization. However, as will be shown for the Lebanese amber samples, the observed birefringence behavior was consistent with the simplifying assumption of a uniaxial birefringent material. The electric field after transmission through polarizer P1 in The amplitudes L is the sample thickness, The propagation times for terahertz pulses traveling along the two principal stress axes in the material are given by The extraction of the stress direction from Equation (2) and the relative birefringence from Equation (3) requires the measurement of four terahertz pulse waveforms. The time-domain waveforms through the sample with polarizers P1 and P2 oriented orthogonally, as illustrated in The amplitude and arrival time for each of the four waveforms was mathematically determined by a deconvolution process using the T-Ray 5000 software. The deconvolution was performed in the frequency domain using three filters. The first was a Gaussian filter in the time-domain with a 0.8 ps bandwidth. The second was a Bessel bandpass filter from 0.2–0.8 THz. The last filter was a Wiener filter. The arrival time As an example, It is well known that the scattering of electromagnetic waves from a boundary can randomize the incident polarization. While the cross-polarization geometry of The best fit curves shown in A and The extracted best-fit parameters from the data for the deconvoluted pulse amplitudes and arrival times Using a standard propagation of errors analysis, the uncertainty in the stress direction −4. For the best-fit parameters and uncertainties of Since amber is a photoelastic material, birefringence images acquired with visible light can assist in determining the locations of various inclusions or detect areas of varying stress induced within the amber during its creation and polymerization process. Samples of Lebanese and Mexican Chiapas amber were imaged using a polarized visible light setup. The imaging setup consisted of two film based linear polarizers placed on each side of the sample with a 6500 K light box placed on one side and a digital camera on the other. With an isotropic (non-birefringent) material placed between two polarizers whose transmission axes are orthogonal, the light which passes through the first polarizing filter maintains its polarization as it propagates through the isotropic material. Consequently, no light emerges from the second cross-polarizer, resulting in a dark image captured by the digital camera. However, if the material experiences either an applied or intrinsic stress, the induced birefringence rotates the plane of polarization of light propagating through the material resulting in a degree of transmission through the second cross-polarizing filter, resulting in a multicolored image. Similarly, if the amber’s birefringence properties can be approximated as a uniaxial crystal with the principal stress axes in a plane parallel to the planar faces of the sample, the incoming terahertz radiation may be propagated perpendicular to the stress axes. In this configuration, no light is transmitted through the crossed polarizers P1 and P2 unless the polarization direction of the light is modified by the birefringence. However, if the principal stress axis is either exactly parallel or perpendicular to the incident direction of polarization, then no light is transmitted; this corresponds to either As shown in Using this technique, areas of highly localized stress were observed in the Mexican Chiapas amber. As with the Lebanese amber, similar areas of localized birefringence were observed but with larger spatial variation within the sample. This spatial variation is denoted by the rapid change in color and contrast variation and is shown in Using the configuration of In optics, it is well known that the scattering of electromagnetic waves can change the polarization state of scattered radiation relative to the incident polarization. Using this effect, it may be possible to detect the presence of inclusions. Since it is the presence of inclusions that is of interest with regard to amber (refractive index 1.58), and not the surface topology, mineral oil (refractive index 1.46) was used as an index-matching fluid. Mineral oil was chosen for its relatively close refractive index, but also it was tested and proven to not damage these rare samples. Attempts were made to create a perfectly index-matched mineral oil solution using ceramic nanoparticles but this is not presented in this research. The sample was submerged in a plastic container filled with mineral oil. −4) is two orders of magnitude larger than the birefringence measured from just the plastic box (×10−6) used in the experimental setup.A second method for imaging inclusions is to measure a spatial map of the sample birefringence. It is well known that inclusions in an otherwise uniform material tends to concentrate the stress at the point of inclusions. Using the cross-polarization configuration of For these two images, two points labelled A and B in the images are highlighted. The deconvoluted In this section, the prospect of using THz-CT to image inclusions in 3D is explored. A simple method to minimize the artifacts from beam refraction is to immerse the sample in an index-matching material. The amber sample was mounted to the end of a rotation stage and immersed into mineral oil within a plastic container for imaging .Transmitted terahertz waveforms were acquired on a pixel-by-pixel basis for each rotational position of the sample. The 2D images  were acqFollowing the same procedure as detailed above, cross-polarization imaging was utilized to image termite inclusions and air bubbles, consistent with methane, in the Dominican amber. As with the previous sample in 2 amber matrix surrounding the termite. This inversion of pulse shape is shown in Note that the stress angle in −4 to 4 × 10−4. Analysis of the uncertainty in the extracted values described above indicates that these variations are larger than the uncertainty and represent spatial variations in the stress direction and magnitude. Similar terahertz images for the Mexican amber sample are shown in In order to compare the visible photoelasticity image with the terahertz image, the terahertz data can be processed to create an equivalent image to the visible image. Starting from Equation (A19) in the L, of the sample), and the A terahertz photoelastic image b that miirection c. The teAs shown in Insect inclusions in amber are heterogeneous in their internal composition. Entombed insects occasionally retain elements of internal soft tissue; however, it is more common for insect exoskeletons to be essentially hollow. Soft tissue that is degraded by microbes during the resin-capture stage is effectively replaced by air, leaving void space between the exoskeleton . Such voUsing terahertz radiation transmission in a cross-polarization geometry, stress maps for the normalized birefringence and stress direction may be obtained nondestructively. The analysis method utilizes a deconvolution method to determine the arrival times and amplitude of the cross-polarized terahertz pulses through a birefringent material. Using amber as a material of interest, the stress maps show that Lebanese amber samples with no inclusions behave as classic uniaxial birefringent (photoelastic) materials whose inferred stress axes in the terahertz spectral range agree well with visible photoelasticity measurements. Moreover, the terahertz data can be processed into stress contour maps which closely follow the color bands of visible photoelasticity measurements in Mexican amber.The cross-polarization experimental configuration enables stress levels within the amber matrix to be visualized while also outlining highly localized regions of stress surrounding inclusions. Scattered cross-polarization radiation, while an important contrast mechanism for the reduction in transmitted power, enabling the visualization of air bubble inclusions within amber, can be separated from the photoelastic response. Birefringence stress maps clearly show localized increases in stress magnitude and direction located at points of inclusions.A methodology for creating 3D reconstructions of amber samples was created using a refractive index matching material. With further investigation into this approach, one should be able to extract a 3D reconstruction of inclusions independently from the surrounding amber matrix. This would allow for the study of inclusions’ shapes, sizes, etc.


Desmoid tumors are characterized by indolent growth, progressive invasion of surrounding tissues and a high rate of relapse. We present the case of a desmoid tumor rising from the trapezius of a young woman. Following resection, we performed a functional reconstruction using an innervated gracilis free flap. A desmoid tumor (DT), also known as desmoid-type fibromatosis, is a rare benign neoplasm characterized by monoclonal growth of fibroblasts with the progressive invasion of soft tissues, a high rate of recurrence following treatment and the inability to metastasize. It can occur in any part of the body, with those arising in the limbs and trunk portending the poorest prognosis .Active surveillance is now considered by most authors the best way to deal with asymptomatic disease . On the A 34 years old woman was admitted to our institute for a soft palpable mass localized in her left trapezius. She complained about localized discomfort and pain that interfered with her daily activities. In order to better characterize the nature of the mass and its anatomical boundaries, an MRI  was perfWe discussed with the patient the possibility of a watchful follow-up before considering surgery. She opted for surgery in the first place, which we considered a reasonable choice in consideration of the symptoms she complained about.Under general anesthesia, we performed an en-bloc resection of the mass with macroscopic margins of resection of about 1 cm. The mass was located in the upper third of the trapezius at the base of the neck, thus resection would involve the neuromuscular hilum leading to paralysis of the muscle, a condition seen after neck dissections with the sacrifice of the spinal accessory nerve . FunctioThe upper third of the trapezius was resected en-bloc with the overlying skin including the scar from the previous biopsy. The spinal accessory nerve was prepared as close to the muscle hilum as possible and so were the transverse cervical vessels. The right transverse musculocutaneous gracilis free flap was transferred to the defect. The vascular anastomosis was performed end to end between the transverse cervical artery and vein and the gracilis pedicle. Neurorraphy was performed end to end between the spinal accessory nerve and the obturator nerve. The flap survived completely and the post-operative period was uneventful.Three months following surgery the gracilis muscle recovered its contractility, as demonstrated by the follow-up EMG. After rehabilitation, the patient regained full range of motion and was able to go back to her daily routine, including swimming, her favourite hobby, without any restriction six months postoperatively. Pain and discomfort disappeared completely.Supplementary Videos 1–3).After 9 months, to improve the cosmetic result and obtain a more attractive contour of the neck area, a skin expander was implanted in the subcutaneous region adjacent to the flap and progressively inflated until enough skin was gained in order to replace the flap island and reduce dimensions of the scar. Three months later the flap’s skin was resected and the resulting defect was covered with the expanded skin. The final aesthetic result was satisfying, consisting of just one linear eutrophic scar of about 7 cm. After a 7-year follow-up based on MRI and clinical evaluations, no signs of recurrence were observed . A 10 yeThe desmoid tumor usually presents as an indolent, asymptomatic, growing mass. Diagnostic workup usually begins with imaging, with MRI being the mainstay. The suspicion of DT is then confirmed by core biopsy and is based on the detection of nuclear accumulation of b-catenin protein caused by activating mutations in the b-catenin gene .Given the wide spectrum of clinical presentations and variable progression of the disease, treatment should be tailored to each patient; options encompass surgery, radiotherapy, medical therapy and isolated limb perfusion, the latter being reserved for DT of the extremities.In recent series, up to 50% of asymptomatic patients had no progression of disease at 5 years and in 20–30% of the regression of disease was observed . Hence, Selection of the best treatment option should be based on tumor location, dimensions, age and functional necessities of the patient. Surgery is still considered the first-line approach for intraabdominal DT ,6 while Since the morbidity of upper trapezius palsy following neck dissection with spinal accessory nerve sacrifice is well known, we sought a functional restoration of the upper trapezius by means of an innervated gracilis free flap ,14. AdeqThe innervated gracilis free flap represents a great choice for functional reconstruction of small and medium defects of trapezius muscle following tumor resection. The availability of muscular innervated tissue with minimal donor site morbidity, allows the surgeon to perform radical resection with greater disease-free margins and better control of disease without concern about reconstruction options. However, the rarity of the disease and the related prognosis requires more evidence to outline a defined therapeutic and reconstructive algorithm.Click here for additional data file.


In this work, we propose a framework to enhance the communication abilities of speech-impaired patients in an intensive care setting via reading lips. Medical procedure, such as a tracheotomy, causes the patient to lose the ability to utter speech with little to no impact on the habitual lip movement. Consequently, we developed a framework to predict the silently spoken text by performing visual speech recognition, i.e., lip-reading. In a two-stage architecture, frames of the patient’s face are used to infer audio features as an intermediate prediction target, which are then used to predict the uttered text. To the best of our knowledge, this is the first approach to bring visual speech recognition into an intensive care setting. For this purpose, we recorded an audio-visual dataset in the University Hospital of Aachen’s intensive care unit (ICU) with a language corpus hand-picked by experienced clinicians to be representative of their day-to-day routine. With a word error rate of 6.3%, the trained system reaches a sufficient overall performance to significantly increase the quality of communication between patient and clinician or relatives. Intensive care units (ICUs) cater to patients whose lives are in immediate danger and require constant supervision and care from the medical staff. The length of a patient’s stay in the ICU depends on the severity of the patient’s condition and can vary from several hours to several weeks or months in extreme cases.1 although 53.9% of MV patients were found to meet”basic communication criteria”with the patient being awake, alert, and responsive to verbal communication from clinicians2. Solutions to this problem are grouped under the term”augmentative and alternative communication (AAC) strategies” and divided into low-technology AAC, such as point charts or handwriting with pen and paper, and high-technology AAC facilitating communication via handheld apps, text-to-speech software or eye-tracking commands3. For a patient who might already be in a state of shock with the ailments under treatment, not being able to communicate effectively can worsen symptoms like anxiety, panic and distress. Ultimately, this leads to frustration for both patients and clinicians1 as the patient is not able to report his symptoms and needs in an unconstrained way. This situation comes with an increased chance of the patient developing the state of delirium, having severe short-term and long-term effects on the patient’s physical and mental health4, which adversely affects the MV dependence duration and thus causes an overall increase in healthcare costs5.Respiratory failure is a common ailment in ICU patients whose spontaneous breathing is insufficient or completely failed due to various reasons. To avoid a decrease in the blood oxygen saturation of patients suffering from acute respiratory failure (ARF), they undergo the treatment of Mechanical Ventilation (MV). It works by either positive or negative pressure ventilation, where air is pushed or sucked into the patient’s lungs. In non-invasive MV, this happens through facial or nasal masks through the patient’s mouth. However, in severe cases, physicians resort to invasive ventilation methods. Two common forms of invasive MV are the insertion of an endotracheal tube through the patient’s mouth and the tracheotomy, a surgery in which a tube is inserted through a tracheal incision. Invasive MV methods result in the patient losing his voice and, in effect, the ability to use the most effective form of communication with clinicians or relatives. The difficulty in communication is the most commonly reported distressing symptom for ICU patients undergoing MV6. Estimates put the number of mechanically ventilated patients worldwide at 13–20 million per year7. The current Sars-CoV2 pandemic highlights the importance of mechanical ventilation in the critical care environment as it causes severe respiratory restrictions, often leading to MV treatment. Depending on the region, healthcare system and age of the patients, different studies report the share of Covid-19 patients that receive MV to 10%–60%10.A French cross-sectional study reported that between 2006 and 2015, the number of patients admitted to intensive care units for respiratory infection increased 2.7-fold3. Given a patient that merely has problems with producing sounds but can still make lip movements, a machine-based lip-reading system could help such patients to communicate more efficiently with the medical staff, thus constituting a fast and intuitive, high-technology AAC solution that comes with small time overhead and requires little to no training for caregivers and patients to use.With a particular emphasis on the above-mentioned problem of MV and delirium, both in terms of the patient’s health and the economic impact, new technologies to aid patients with temporary speech impairment in communication are needed, especially since existing AAC methods are rarely used by caregivers due to time constraints, lack of tool availability, or insufficient training1. Thus the sentence corpus is hand-picked by experienced intensive care clinicians and designed to reflect the context of essential ICU communication. Training the two-stage model on our VSRICU data set yields a trained model being able to perform free-text lip-reading to aid speech-impaired patients in a clinical setting. To compare to the baselines of existing work in this field, we further train our model to perform VSR on the GRID audio-visual corpus11.From a technology perspective, the process of predicting the transcript of the patient’s speech using only the silent videos of the patient’s mouth region movements is known as Visual Speech Recognition (VSR) or lip-reading. The contribution of this work is two-fold. First, we introduce a two-stage framework for lip-reading utilizing the ground-truth audio features of the video data set as an intermediate prediction target at training time. We further introduce a custom lip-reading data set (VSRICU) recorded in an ICU setting in the University Hospital of Aachen (UKA). Studies analyzing the typical communication between patients and nurses indicate brief and informative statements about physical care, yes/no questions, reassurances and commands to be importantThe experimental evaluation conducted in this work aims to test the following statement: the proposed two-stage model is able to perform the task of lip-reading on both the GRID corpus and our VSRICU dataset with acceptable performance w.r.t. patient communication  datasets13, which have a broad and general corpus and challenging speaker movements. In contrast to the conditions found in these datasets, lip reading in an ICU setting has the advantage of a fixed camera angle, less head movement and a more narrow selection of potential conversation topics, which facilitates the training of our model with a comparably small amount of videos to ultimately perform well in the specific domain of essential patient communication. We thus recorded a new dataset dedicated explicitly to the purpose of lip reading in the intensive care setting. Our dataset, dubbed as Visual Speech Recognition for Intensive Care Unit (VSRICU), provides a total of 6900 videos of six male and female speakers (native German), each uttering 238 different sentences in German and English (around 50% German and 50% English). Recordings for the VSRICU dataset have been conducted in a room of a real intensive care unit of the University Hospital Aachen (UKA) in 2019 (pre-Covid). We used two webcams to obtain videos from two near-frontal angles being representative for a potential position of a lip reading device at the ICU bed. Frame samples from the dataset are shown in Fig. To the best of our knowledge to date, there is no database that fits the nature of our problem statement. Various audio-visual datasets for training lip-reading systems are available, e.g. the An informed consent for publishing both the full dataset and identifying images from the dataset was obtained from all subjects and/or their legal guardian(s) in both open access and none open access publications before participating in the recording sessions of Visual Speech Recognition for Intensive Care (VSRICU) dataset. All methods were carried out in accordance with relevant guidelines and regulations.11 contains a total of 34,000 video recordings of 34 speakers, each uttering 1000 distinct sentences. The dataset comes with transcripts of the spoken sentences to be used for training either lip reading or ASR systems. The sentences in GRID are constrained to a specified structure, with each sentence being formed as: ‘command(4) + color(4) + preposition(4) + letter (25) + digit (10) + adverb(4)’. Here the number in brackets denotes the number of options for each category . Using GRID as a dataset for visual speech recognition on the one hand simplifies preprocessing by providing perfectly aligned face recordings of uniform length (3 seconds) with an excellent video and audio quality. On the other hand, the sentences spoken in GRID are not representative of real-world scenarios of a human to human communication.The GRID CorpusCompared to GRID, the camera angle and lighting conditions in the VSRICU dataset are more challenging. A lower audio signal-to-noise ratio complicates the use of audio as an intermediate prediction target. However, the biggest challenge is given by the much smaller number of videos in the dataset . A more detailed comparison of technical characteristics is provided in Table 14 to estimate the face’s bounding box and subsequently a facial landmark detector based on an ensemble of regression trees15. For both the face detector and the landmark prediction, we use the implementations provided by the dlib library12. Based on four key landmarks , we estimate the position of the subject’s mouth and crop the image, such that the padding (distance between mouth and border of the image) equals half the mouth width or height. Finally, the cropped images are resized to a shape of width, height, channels). We relinquish every video in which detectors fail to recognize the face. Before being fed to the AFE, frames are normalized to the range of After extracting the sequence single input frames from the source video, frames are converted into grayscale reducing the images to the luminance information as shown in Fig. In order to facilitate the prediction of audio features from a sequence of silent frames, window size and stride are adjusted such that one set of audio features matches the length of exactly one frame. For the given datasets with a frame rate of 25 frames per second and a sample rate of 16, 000 Hz, the resulting window size is 40 ms or 640 audio samples for each set of audio features. Audio extracted from the source videos is normalized to the range of cepstrum. Finally, Mel Frequency Cepstral Coefficients (MFCC) are given by the amplitudes of this cepstrum. Usually, the first MFCC is omitted, as it merely represents the average power of the input signal. The higher-order MFCCs are renounced as well, as they represent spectral details usually not contributing to a higher performance of the model while increasing the dimensionality of the input. For the evaluation conducted in this work, we consider both Mel Spectrograms and MFCCs as potential intermediate audio feature prediction targets. We use the librosa16 library to perform all of the above-mentioned steps in the audio preprocessing pipeline. Finally, each coefficient is normalized individually by subtracting the mean and diving by the standard deviation of the coefficient across the whole dataset, causing the preprocessed audio features to have zero mean and unit standard deviation. We choose this type of normalization over the mapping to the range of The resulting spectrum is mapped to the Mel scale, which divides the frequency domain into distinct bins of pitches using overlapping triangular windows. The Mel scale is based on the human way of hearing, as we perceive each of these bins as being equally distant in pitch. After summing the logarithmic power spectrum within all bins, a discrete cosine transform is applied to decorrelate the Mel frequency bands, resulting in a so-called 17, proposing a model to map the sequence of silent frames to a mel-spectrogram reconstruction for the GRID dataset. In18, a pretrained speech recognition model is used in a teacher-student approach to guide the training of a visual speech recognition pipeline in order to improve performance and to yield a way of training lip reading models without ground-truth annotated video data. Similarly, in19 a pretrained speech-recognition model is used to provide complementary and discriminant clues for improving the training of a lip reading system.The proposed model for visual speech recognition consists of two stages, exploiting the availability of (otherwise unused) audio data in the training set to serve as an intermediate prediction target in the lip reading pipeline. A similar approach can be found inThe Audio Feature Estimator approximates the mapping from frames to audio features using a spatio-temporal feature extraction part based on Convolutional layers with a three-dimensional kernel followed by a sequence processing part with Gated Recurrent Unit (GRU) layers. In contrast to convolutional layers with two-dimensional kernels being able to learn and extract spatial features from each individual frame, the spatio-temporal (3D) convolutional layers learn a kernel performing calculations across the spatial dimensions (width and height) and the temporal dimension (across subsequent timesteps), thus allowing for learning and processing short-term temporal dependencies in the videos. The visual feature extraction part of the Audio Feature Estimator is followed by multiple Gated Recurrent Unit (GRU) layers to learn both long- and short-term dependencies between the visual encoding of the frame sequence. The output layer yields a sequence of N yieldsThe Audio Feature Estimator is trained end-to-end using a pixel-wise mean-absolute-error (MAE) loss between the ground-truth and the predicted audio features. The loss function minimized for each training batch of size 20 which allows for minimizing the loss for all possible input-output alignments without the need for explicitly segmented text labels with timestamps of word boundaries. For each feature set extracted from a single video, the output layer yields a sequence of shape  representing a probability distribution t. The set of characters  is composed of the 26 characters of the English alphabet along with whitespace (” ”), a padding character (<pad>) and the CTC blank token (_).The second stage of the visual speech recognition model maps the sequence of audio features to a sequence of probabilities for each character. Input features pass through two layers of 1D-convolution to learn short-term temporal dependencies, followed by GRU layers to process short- and long-term dependencies across time. Figure i without the need for any explicit word or character alignment information in y.Using the CTC-loss, the STT model is trained by means of a differentiable loss functioni in the batch. Though minimizing the same loss function as in the case of end-to-end lip-reading (maximizing the probability of the target sentence given a frame sequence), the substantial difference lies in the AFE already holding knowledge about the relationship between audio and video. Although the audio feature predictions are erroneous, they serve as a low-dimensional feature set encoding substantial information needed to perform a text transcription. By fine-tuning, the STT is given a chance to adapt to the errors introduced by the imperfect AFE predictions that also occur at runtime, thus improving the performance of the final, combined visual speech recognition system.Besides training AFE and STT models individually and afterward evaluating them together at runtime, where the AFE output is fed to the STT model, the STT model can be fine-tuned by training on the AFE predictions instead of the ground-truth audio features of the video to obtain the modified CTC loss function1-gram) language model, performing a word-for-word correction of the source sentence based on the total dataset’s corpus. For each word w in the decoded sentence, the model obtains a set of word candidates w by inserting, deleting or replacing characters. At the same time, the maximum number of modifications (edit distance) is limited to Sentences produced by the CTC-decoder of the STT module undergo a spell correction based on a simple bag-of-words  by adding a constant value to the image array.All of the previous augmentation steps are applied randomly with a probability N, 75, 50, 100, 1) and the corresponding audio features are of shape N is the batch size used for training and After preprocessing, input frame sequences are of shape  and MFCCs . We further conduct all of the experiments with and without augmentation applied to the AFEs input frames. In the case of augmentation, the input frames to the AFE, when fine-tuning the STT model, are augmented as well.LipNet22, which is known as the first end-to-end deep neural network model for lip-reading on the GRID dataset. We see this model suited to be trained on the VSRICU dataset, as it shows good performance on the mid-sized GRID dataset while more recent end-to-end models rely on huge audio-visual resources24. We train LipNet as described in the original paper.To compare our model against an end-to-end model from literature, we further implement An informed consent for participating in collecting the dataset and publishing both the full dataset and identifying images from the dataset was obtained from all subjects and/or their legal guardian(s) in both open access and none open access publications before participating in the recording sessions of Visual Speech Recognition for Intensive Care (VSRICU) dataset. All methods were carried out in accordance with relevant guidelines and regulations. This is not a medical study, and it did not involve any patients, nor did it compare, assess or evaluate any kind of human behavior.Thus the need for ethics approval is deemed unnecessary according statutes of the ethics committee of the faculty of medicine of the RWTH Aachen version dated 20.03.2008, and declared in the official notice number 2020/103. The procedure of data collection, access, deletion, storing and processing has been acknowledged by University Hospital Aachen legal department.The results in Table LipNet22 on the VSRICU dataset results in a CER/WER of 7.2%/12.7% (with language model), which is significantly worse, while the performance on GRID is reported to be similar with a CER/WER of 1.9%/4.8%.Table From the text predictions on both datasets, it becomes obvious that short words (and sentences in the case of VSRICU) are among the more commonly misspelled. The prediction errors on the GRID dataset are limited to almost exclusively the single letters present in every GRID command. This type of error is unfeasible to correct for a word-level language model (and due to the structure of the GRID corpus for any other language model), thus explaining why the effect of using a language model here is not visible in the evaluation metrics. Further we avoided the use of a higher order language model  due to the limited corpus available for training it on the VSRICU dataset. Table 26 or25, we back-propagate class decision boundaries to the network input to visualize the region of interest in the input signal. The colored regions in the input lip frames prove the model to work and perform visual speech recognition. Looking at the intermediate audio feature signal, we can see that lower-order features contribute more to the character sequence output than the higher-order features. This can be expected due to the lower-order features carrying more information after the DCT is performed to obtain MFCCs. The lower part of the figure shows the sequence of character probabilities before being processed by the CTC decoder.Figure This work introduces a two-stage approach to Visual Speech Recognition in an intensive care environment using low-dimensional audio features as an intermediate prediction target. The fully trained system achieves a word error rate of 6.3%, which we assume to be sufficient to restore the essential patient-to-clinician communication for speech-impaired patients. The experimental evaluation in this work suggests that both careful data preparation (incl. augmentation) and postprocessing of text predictions play a vital role in the final system’s performance.The recorded sentences in our ICU dataset are closer to the context of a patient-clinician communication than existing datasets. Nevertheless, the linguistic diversity of the corpus is still limited to the essential needs of the patient. The limited context of the dataset introduces a bias towards these sentences in models trained using the recorded videos. Although this leads to higher accuracy for sentences originating from the dataset, it might worsen the predictions if the linguistic context changes, e.g., in a conversation between the patient and his/her family. Further, though planned to be expanded, our dataset still suffers from its small sample size of only a few thousand videos. This limits the capacity of the model trained using this dataset, even using techniques like image augmentation to artificially increase the dataset’s size. Finally, the reality gap is increased by the fact that all subjects in the VSRICU dataset are young and healthy adults, which implies that patients in need of the proposed system do not have problems expressing a regular lip movement. Although younger people might suffer from respiratory failure (and other diagnostic findings blocking speech utterance), the majority of patients in need of the system will not be able to move their lips vividly like a young and healthy person due to other ailments, weaknesses, or simply, age.27 might facilitate more realistic results, using only a small amount of sample videos from the target group to modify the existing dataset.The limitations mentioned above can be addressed both from the data and the model perspective. On the data side, a flexible and accurate medical lip reading solution requires the extension of our VSRICU dataset both quantitatively  and qualitatively (number of speakers and variety of the linguistic context). On the model side, the two-stage approach allows for using other publicly available datasets to increase the prediction quality, e.g., by pre-training the second stage of the model with an extensive, established speech recognition dataset for the application in a non-clinical conversation of the patient. As for the aforementioned problem of elderly or ill people, the trivial solution of recording a dataset featuring this specific group with the desired sample size and diversity of people is not feasible for ethical reasons. Nevertheless, this problem might be addressed in a data-driven approach as well by artificially worsening the visual speaker dynamics, which mainly involves slowing down lip movement and reducing the maximum opening of the mouth to represent the lip movement and speed of elderly, weak or ill patients. While this can be implemented by resampling the video and warping frames based on facial landmarks, applying recent advances in generative models such as neural network based style transfer28 framework for video processing. Considering the quality of predictions on the other end of the system, we are working on a more sophisticated and adaptive language model tailored to the use with intensive care patients. The very specific application allows for a precise prediction of the most important sentences and commands uttered by the patient as reflected in our dataset. At the same time, state-of-the-art general language models can help to improve text inference in a less restricted conversation context. All of the above-mentioned points, such as the robustness of predictions, the real-time capability of the system, and the use of adaptive, state-of-the-art language models, are subject to our current continuation of the research presented in this document and will ultimately facilitate the implementation of our theoretical work in the intensive care unit.Considering future work, we are currently developing a hardware prototype that will allow us to test our theoretical results in a real-world intensive care setting. This will help us understand the limitations of our approach and identify any reality gaps between our theoretical work and the actual conditions inside the intensive care unit. Besides the above-mentioned problem of the VSRICU dataset not being entirely representative of the clinical target group, working on a hardware implementation will introduce additional challenges to overcome. One of them is the real-time capability of the system, which might be improved by using state-of-the-art solutions such as the MediaPipe


The COVID-19 pandemic has produced social and economic changes that are still affecting our lives. The coronavirus is proinflammatory, it is replicating, and it is quickly spreading. The most affected organ is the lung, and the evolution of the disease can degenerate very rapidly from the early phase, also known as mild to moderate and even severe stages, where the percentage of recovered patients is very low. Therefore, a fast and automatic method to detect the disease stages for patients who underwent a computer tomography investigation can improve the clinical protocol. Transfer learning is used do tackle this issue, mainly by decreasing the computational time. The dataset is composed of images from public databases from 118 patients and new data from 55 patients collected during the COVID-19 spread in Romania in the spring of 2020. Even if the disease detection by the computerized tomography scans was studied using deep learning algorithms, to our knowledge, there are no studies related to the multiclass classification of the images into pulmonary damage stages. This could be helpful for physicians to automatically establish the disease severity and decide on the proper treatment for patients and any special surveillance, if needed. An evaluation study was completed by considering six different pre-trained CNNs. The results are encouraging, assuring an accuracy of around 87%. The clinical impact is still huge, even if the disease spread and severity are currently diminished.  SARS-COV2 (severe acute respiratory syndrome coronavirus 2) in humans was first reported in Wuhan, China, on 31 December 2019; it is a single-strand ribonucleic acid virus from the Coronaviridae family . The models training options are presented in The external validation was completed by splitting the dataset with the ratio 80:20 for training and testing . For assuring that there is not any mixing of images of the same patient into these two groups and for a good balance regarding the number of images and patients, respectively, this splitting was completed manually by a medical expert, after careful attention. For internal validation, the training dataset was further randomly split into 80:20 for adjusting the network weights and in-training validation (used only to check the metrics of the model after each epoch) with a fixed random seed. This last step was repeated 5 times to perform the 5-fold cross validation.In addition to well-known evaluation metrics, the Cohen’s kappa coefficient and Matthew’s correlation coefficient (MCC) are computed. The kappa score measures the degree of agreement between the true values and the predicted values: a value of 1 means perfect agreement, and 0 means chance agreement. A value less than 0 means a disagreement, so the classifier is incorrect.MCC applied for binary classification identifies the ineffectiveness of the classifier in classifying, especially for the negative class samples, and it has high values if the classifier is doing well. However, F1 score is highly influenced by which class is labeled as positive. Still, F1 score is different from MCC in respect to its magnitude, as the minority class is usually labeled as negative.®2022a with an academic license. Considering the speeding up, the running time of the computational environment for these tests was completed on a supported GPU device, an NVIDIA GeForce 1650 with 4 GB RAM.The algorithm was implemented in a copy of MATLABThe dataset distribution into classes for the training, validation, and testing is organized in The global evaluation metrics, when considering five-fold cross validation, re given in The plots for the convergence of accuracy and loss for both training and validation are presented in For comparison, model 2 from  was consdevelopment of a new classification algorithm for the suggestive aspect of COVID-19 lung damage into mild, medium, or severe stages, for the axial view of computed tomography images using deep neural networks;enhancement of the five different online databases with new images collected from 55 patients;manual selection of the open lung phase images and their labeling into four classes.The main contributions of the presented paper are as follows:The main challenge was the poor quality of the available online datasets. There were cases of sample duplication when the data were merged. Moreover, some images were labeled with different landmarks or text indications. An imagistic expert was needed to revise, to identify the images belonging to each patient, and to label them into four classes. Due to restrictions on acquiring new medical data and to speed up the computational time, transfer learning via fine-tuning was used for 7300 images belonging to 173 patients. An evaluation study was completed by considering six different pre-trained CNNs. The results were compared regarding the precision, accuracy, recall, etc. The best model was obtained with Resnet50, by considering 30 epochs and 32 mini-batch size. The model performance was evaluated considering its accuracy of more than 86%. The Cohen’s kappa coefficient of 0.65 approached 1, which means a good degree of agreement between the true values and the predicted values. The Matthews’ correlation coefficient of 0.76 means an effectiveness in the classification of the four classes. The best matching class was the normal one, which is explainable considering the lung aspect, while the worst case was the mild class, which may be due to the rather small lung damages.The main pitfall of deep learning is collecting more and more data for improving the models. A bigger dataset will not always produce better results, due to a lack of attention or expertise regarding the quality of the input data . Hence, one of the limitations of this study is the significance of the database of images used to instruct the network . Our future work will include a more robust instruction of the AI tools, with the type of data distribution and a possible dimension reduction like in .Discovering ways to enhance the process of diagnosing and treating COVID-19 patients is a worldwide emulation. The proposed technical solution will impact our society in the battle against COVID-19 disease, in terms of the faster screening of patients. The medical protocol for diagnosing patients is a physical exam and PCR testing. Since the result of the PCR test can take almost 24 h, physicians can choose to examine the patients’ lung damage using CT scans. Having a powerful computing tool with an AI-enhanced diagnosis will assist medical staff, will improve clinical management, and will even improve treatment of COVID-19 disease.


This paper describes the process of developing a classification model for the effective detection of malignant melanoma, an aggressive type of cancer in skin lesions. Primary focus is given on fine-tuning and improving a state-of-the-art convolutional neural network (CNN) to obtain the optimal ROC-AUC score. The study investigates a variety of artificial intelligence (AI) clustering techniques to train the developed models on a combined dataset of images across data from the 2019 and 2020 IIM-ISIC Melanoma Classification Challenges. The models were evaluated using varying cross-fold validations, with the highest ROC-AUC reaching a score of 99.48%. Cancer is considered as one of the most alarming diseases nowadays, killing nearly 10 M persons each year according to the World Health Organization. Although it may come in many forms . HoweverDespite the decisive role of prompt detection, early stage diagnosis of melanoma is an arduous task due to its similarities with common moles or benign skin tumors. Neglect of such skin lesions may lead to deep invasion and wide spread of melanoma cells through the lymphatic or circulatory system. Typically, skin cancer is verified through a biopsy, using extracted samples of the suspicious skin lesion for lab analysis in a demanding process that is often time-consuming. Therefore, providing automated assistance to healthcare professionals, or supporting self-diagnosis directly to patients is of critical importance towards the early detection of melanoma. Artificial intelligence (AI), in particular machine learning (ML) or deep learning (DL) algorithms, can facilitate the development of such automated decision-support systems to enable non-invasive diagnosis procedures.Machine learning techniques are widely used to detect meaningful patterns in varying types of data representations. Utilising specific feature selection approaches and transfer learning methods, ML algorithms have been successfully used in several medical imaging applications . Due to Research in the domain of skin lesion classification through AI is broad and includes algorithms such as support vector machine (SVM) , decisioThis paper focuses in dermatoscopy image analysis (DIA) through state-of-the-art CNNs and aims to improve the results of the winning solution in the (2020) IIM-ISIC Melanoma Classification Challenge , in termThe rest of this paper is organized as follows. Convolutional neural networks are specialised deep architectures that target visual data analysis. Typically, CNNs are characterised by the number of filters applied to the input image, where a higher number allows more subtle details and features extracted from the input. Another characteristic of their architecture is the number of layers, which affects how the CNN will be able to fit to the corresponding training data. CNNs demonstrate increased efficiency in image classification tasks due to their ability to automate the learning of useful features, as opposed to requiring human-engineered filters in other approaches.The study in  demonstrAn ensemble of CNN models was employed to efficiently identify melanomas using skin images along with patient metadata in , providiSVMs belong to the class of supervised machine learning algorithms that specialise in data analysis for classification problems. Based on the concept of hyperplanes, SVMs target to select the support vector, which is the hyperplane that maximises the distance to the nearest data point of the related classes. SVMs perform well in linear classification tasks, but are also suitable for non-linear data by mapping the inputs to high-dimensional spaces through various kernel functions.A nonlinear SVM classifier was used to detect melanomas of affected skin lesions in  by extraIn  the typeIn the context of machine learning, decision trees are supervised learning algorithms that primarily target classification problems. These classifiers utilise a tree structure, whereby each node represents a feature of the dataset. Leaf nodes are used to represent the class labels, and branches represent the decision conditions. Typically, such decision trees are built using the CART algorithm.A skin cancer detection system based on a decision tree classifier is presented in . Input iUsing the ID3 algorithm, a decision tree is constructed in  to classWhile several methods presented in the literature have proven their effectiveness in melanoma classification, automated detection systems built around them still face limitations due to limited data availability that often causes racial bias with respect to certain skin types. This limited data availability hinders the ability to further improve current state-of-the-art models. These limitations are expected to be resolved when technologies such as infrared, optical coherence tomography, and confocal microscopy  are utilThe dataset employed in this work includes images from both the ISIC 2020 and the 2019 melanoma classification datasets. After the removal of duplicate records, the combined dataset includes 48,007 images from approximately 2000 patients of all age groups. Each record belongs to one of nine classes, as outlined in The base model uses the EfficientNet-B6 architecture, originally pretrained on ImageNet and then trained on the employed dataset, due to its exceptional performance in transfer learning tasks, while keeping the number of trained parameters significantly low. The architecture comprises 667 layers built from the modules shown in The loss function utilised for the training is cross entropy loss, given by Equation :(1)L=−∑kEquation :(2)σ(z)iThe primary focus of this work lies in the identification of melanoma in the provided dataset. This gives rise to the possibility of grouping the rest of the data in different ways to create various clustering categories in an effort to increase the melanoma classification performance. To obtain useful clustering information, the images were passed through the base model, retrieving the values of the penultimate layer which includes 2304 features. Several clustering techniques were employed, as outlined in Dimensionality reduction was performed on the data using principal component analysis (PCA), to reduce the amount of the 2304 retrieved features. The aim of using PCA was to capture the components that describe the largest explanations of variation and list them in order of importance. As showcased later in the evaluation, this process improves the time efficiency of the clusterisation, but also results in higher performance metrics. Overall, PCA helps in alleviating the effect of the “Curse of Dimensionality”. By creating a cumulative total of the explained variance, the graph in The primary focus of this work is to improve the melanoma classification rate through AI clustering methods. However, it was deemed necessary to examine a non-algorithmic clustering approach as well to obtain useful insight about the potential categories. This clusterisation was designed based on the relationship between each class and the priority of seeking medical expertise, as described in . The obtm points inside n dimensions into k clusters so that the sum of squares in each cluster is minimised. In essence, through k-means the points are categorised based on their proximity in the Euclidean space. To determine the suitable number of clusters, the measure of within-cluster sum of squares, called inertia, is calculated and plotted for a varying number, as demonstrated in K-means clustering aims to categorise Two methods of hierarchical clustering were utilised: single-linkage and complete-linkage. The first performs clustering based upon the minimum distance between any point in that cluster and the data point being examined. In contrast, complete linkage performs clustering based upon the minimisation of the maximum distance between any point in the cluster and the data point being examined. Hierarchical clustering may lead to unequal cluster sizes, but since the classes in the employed dataset are not balanced, this does not affect this work.The approach of single-linkage is demonstrated as a dendrogram in T-distributed stochastic neighbour embedding (TSNE) is a method of visualising multi-dimensional space in two dimensions. This approach is of high interest in this work, as it can help highlight clusters of samples within the melanoma class. The TSNE approach was also examined on the PCA data with 95% explained variance, utilising the top 46 PCA features. Finally, the k-means clustering was also applied to the TSNE data after reducing them with PCA and after using the 95% explained variance features as well.To validate the performance of the study, evaluation metrics were calculated on the experiments conducted in the training and classification steps, using a subset of 5024 images taken from the combined dataset.Following the dimensionality reduction using PCA, the clustering study proposed a variety of class groupings, as illustrated in The performance is assessed using the AUC and ROC-AUC metrics. The ROC curve is plotted based on the recall versus precision, where the first indicates the true positive rate, calculated as To calculate the performance of the model across all classification thresholds, AUC is used to calculate the Area under the ROC curve, which effectively measures the probability that a positive sample will be ranked higher than a negative sample. This metric is selected due to its invariant nature both in terms of scaling and in terms of classification threshold. As observed in the table, model T_22 was able to achieve the highest AUC score and therefore was retrained using different learning rates. A further increase was therefore achieved with the final score reaching 0.9962, thus a 0.1% improvement over the base model.In addition to the previous experiments, the evaluation studied quadratic discriminant analysis (QDA) and linear discriminant analysis (LDA) as alternatives to the final layer of the base model to investigate a non-linear and a linear boundary between the classifiers, respectively. As demonstrated in the graphs of The conclusive evaluation was conducted by performing k-fold validations, calculating the final ROC-AUC score for selected models. Each test image was evaluated in eight different transposes/flips, and the average result was obtained. The final scores are outlined in The top models were able to slightly outperform the base model, achieving scores of 99.48%. While further improvements to the base model were expected to yield diminishing returns. Since it is based on state-of-the-art designs and data augmentation, this study was able to achieve a baseline increase of 0.05% in the ROC-AUC score when compared to . Figure These results are more clearly illustrated by the confusion matrix presented in As the incidence rate of skin cancer increases, non-invasive diagnosis approaches are becoming more essential for prompt detection and treatment of malignant melanomas. Towards this goal, this study developed a highly effective classification model based on a state-of-the-art CNN architecture. Based on a comparative analysis of varying clustering methods, the base neural network was retrained through the mapping of the target labels as the result of the feature output clustering. Evaluation experiments demonstrated how this approach was able to improve the performance of the winning model in the 2020 IIM-ISIC Melanoma Classification Challenge in terms of ROC-AUC, reaching a value of 99.48%. Additional improvements were observed with respect to precision and F1-score, with values of 95.19% and 92.72%, respectively.As of the writing of this paper, the use of CNNs for medical diagnosis is generally not permitted in health institutions. This is also encouraged by current GDPR regulations which state that all patients have a right to explanation with respect to processing of their personal data, therefore conflicting with such automated diagnosis systems. Considering the above, the melanoma classification model developed in this work is not supposed to be a substitute for a dermatologist or further analytical procedures deemed necessary, rather it is designed to alert experts and provide valuable insight regarding potential cases, which can assist in prioritisation and early detection of suspicious lesions. As a result, the outcomes of this study outline the potential of developing AI-assisted decision support systems as fast and reliable tools to augment dermatologists during the diagnosis procedure.


Breast Cancer Detection being an area of importance in detecting it in early stages and to assist the Oncologists in analyzing the stage and mode of further treatment, it gains equal importance to all types of cancer detection. Using the proposed optimization based machine learning model in detection process is supposed to help the patients and the oncologists in deciding further process.In recent years, breast cancer detection is an important area of concentration towards curative image dispensation and exploration. Detection of a disease at an early stage is an important factor in taking it to the next level of treatment. Accuracy plays an important role in the detection of disease. COA-T2FCM (Chimp Optimization Algorithm Based Type-2 Intuitionistic Fuzzy C-Means Clustering) is constructed for detection of such malignancy with the highest accuracy in this paper. The proposed detection process is designed with the combination of type-2 intuitionistic fuzzy c-means clustering in addition to oppositional function. In the type-2 intuitionistic fuzzy c-means clustering, the efficient cluster center can be preferred using the chimp optimization algorithm. Initially, the objective function of the type-2 intuitionistic fuzzy c-means clustering is considered. The chimp optimization algorithm is utilized to optimize the cluster center and fuzzifier in the clustering method. The projected technique is implemented, and in addition, performance metrics such as specificity, sensitivity, accuracy, Jaccard Similarity Index (JSI), and Dice Similarity Coefficient (DSC) are assessed. The projected technique is compared with the conventional technique such as fuzzy c means clustering and k mean clustering methods. The resulting method was also compared with existing methods to ensure the accuracy in the proposed method. The proposed algorithm is tested for its effectiveness on the mammogram images of the three different datasets collected from the Mini–Mammographic Image Analysis Society (Mini–MIAS), the Digital Database for Screening Mammography (DDSM), and Inbreast. The accuracy and Jaccard index score are generally used to measure the similarity between the proposed output and the actual cancer affected regions from the image considered. On an average the proposed method achieved an accuracy of 97.29% and JSI of 95% Breast cancer is the world’s most deadly disease affecting women. It starts with unregulated cell division, which leads to the growth of a tumor in the breast . AnomaliImage detection in breast cancer is utilized to organize the disease or is mass pixels that can be used to identify a mammogram, careful orchestration, and image-aided suggestions. In order to filter out suspicious breast images from a whole sum of data collected, artificial intelligence model developed was used . In anotIn a fuzzy C-means clustering system, the MRI image detection approach to K-means clusters the exhibition. To give proper cerebral breast cancer disease detection, thresholding and level set partition locations are followed. For the picture detection of the K-means group, this method can obtain sections of unbounded computing time. Similar to ambiguous C-references in precision pieces, it can attain the best conditions. The adaptive controlled strip-based MRI image cerebral mammogram segment obscure C-marker boxing system demonstrated potential improvements for local system adaptation . It limiTo monitor complex queries in methodology and procedures, AI computations have received a lot of attraction and responded with suitable and accurate responses. AI-based techniques have shown remarkable performance in a variety of image recognition problems . The empNumerous such techniques are adopted to work on breast cancer images towards detection of malignant status in order to facilitate further diagnostic and treatment procedures. A framework is created  based onAnother transfer learning method developed  distinguA breast cancer detection system is completely automated . First, Breast cancer sample classification using an MLP-based homogeneous ensemble technique is proposed . The goaThe method has been designed for the best detection of breast cancer disorders. Based on the studies conducted and considering the stages of implementation of all the previous methods, the workflow of the proposed work is given as follows.The proposed detection method incorporates type-2 intuitionistic fuzzy c-means clustering for clustering suspicious region from the mammogram. Initially, the type-2 intuitionistic fuzzy c-means clustering objective function is considered with the consideration of intuitionistic fuzzy data extracted from the MRI images.The chimp optimization approach can be used to identify the most efficient cluster center in type-2 intuitionistic fuzzy c-means clustering. Chimp optimization algorithm is utilized to optimize the cluster center and fuzzifier from the clustering method.The projected technique is executed in Python to evaluate the performance and similarity measure of the proposed system. The projected technique is compared with the conventional techniques such as fuzzy c means clustering and k mean clustering methods.Breast cancer detection from MRI images is a difficult operation due to irregularity in cohort of disease images in addition to its complex structure. Hence, in this paper, COA-T2FCM is developed to achieve optimal breast cancer detection. The proposed method contains two main phases: a pre-processing stage and the detection phase. In the primary phase, the pre-processing stage eliminates the unwanted noise present in the images, which is achieved by intensity normalization, orientation correction, and pectoral muscle removal. In the second phase, the detection phase detects the lump from the MRI images. Usually lumps in MRI images occur by means of diseases or abnormalities. The proposed method’s entire structural design is depicted in The dataset is essential for analyzing the proposed methodology, so it is gathered from the open-source system. The Mini–Mammographic Image Analysis Society (Mini–MIAS) dataset , the DigSome sample images taken up to show the stages of proposed method is as shown in During the stage of pre-processing, the unwanted information from the MRI image is removed from the images, which improves the quality of the images. The pre-processing methods of the proposed methodology include removing noise like cropping borders and unwanted information, Min-Max normalization is performed to take up intensity normalization; the pectoral muscle is removed from the resulting image. Since images of two varying views, such as the mediolateral oblique (MLO) view and cranial caudal (CC) view, are considered, the preprocessing steps may slightly differ, as removing pectoral muscles does not happen with cranial caudal view images. (a)Intensity normalizationI can be described as a primary intensity value, and σ can be described as the standard deviation of an entire scan. Based on this process, the detection of the MRI image can be achieved in the range of , without any important information loss.In the breast cancer MRI image, the main drawback of the images is that similar kinds of tissue do not have an exact concentration. Various MRI arrangements provide various intensity parameters for the similar tissue type level inside a similar object. This kind of intensity change gives difficult operation of image analysis and detection process. Hence, intensity normalization is the required portion of MRI analysis . Here, G(b)Pectoral Muscle removalFlipping the image depends on the angle of scan image. CC/MLO decides if the image flip is required. Since breast images are always intended to be in the lower left quadrant of the frame, pixels in the top half are compared with that of the bottom half to determine if the flip is actually needed. For CC images, flipping is conducted if the sum of the top half of image is greater than that of the bottom half. For MLO images, the location of pectoral muscles are first determined since the location of pectoral muscle can change if the orientation of image changes. If the pectoral muscle is located in the top left or right corners, then the image is right side up; otherwise if the pectoral muscle is located in the bottom left or right corners, then the image needs to be flipped vertically.jth cluster center and ith pattern; ith pattern related to the jth cluster; C can be described as several clusters; and N can be represented by several data The clustering method  is utilijth cluster center, the Karnik mendal iterative algorithm is utilized. While the iterative algorithm is performed, the right memberships and left memberships are calculated related to each feature for a pattern which is formulated as follows:M can be described as the number of features of a pattern; jth cluster center; jth cluster center; To validate the minimum value in addition to the maximum value of the With the help of proposed clustering methods, breast cancer images are detected from the MRI images. The cluster center is an essential parameter for achieving the best cluster of the detection process. The optimal cluster center is selected based on the COA progression. The suggested COA-based cluster center selection is described in full in the section below.InspirationThe COA is used in this proposed technique to choose the best cluster center. The mathematical formulation of the COA is presented as follows:Usually, the separating collective community is the common people of a sim. It is considered one of the social orders; the composition of the community may be the work of time variation. Furthermore, in the eyes of the public, each area has a distinct mission and capacity that may alter over time. From this point of view, the point of autonomous ideas is formed in this calculation. As a result, each flock of chimpanzees attempts independently to observe the hunting ground with its specific characteristics planned with a focus on open duty. On a larger scale, four types of chips are introduced: attacker, chaser, border, and driver. Considering these types, the practices of the chips in the hunting system are changed for the production chase action. The drivers collected the prey without resorting to hunting, according to the chimp’s estimations. Boundaries form themselves in vegetation to divert a dam into a prey leak path . Prey isTrack and dynamic the preyA, M, and C. COA status vectors are recorded under the condition,Prey can be chased in the COA through exploitation and exploration stages. Furthermore, the numerical scheme of driving the prey is described as follows:Investigation phaseThe attack behavior of the numerical model of the chimp is planned as follows: to the right of the bat, the chimps can give the area of prey; besides, they can circle it. For a long time, predators were often maintained by invaders. In most chases, the chaser, border, and driver are all involved. There was no information about the prey’s ideal condition during the initial inspection at the examination stage. The attacker’s location should be used to update the chaser, square, and drive positions. As a result, the four best arrangements may be saved, and the various chimps can be stopped to update the levels associated with the best chimps’ components. The following is a numerical introduction to this work:The search agent position is updated at the trial site depending on one more location of a chimpanzee. As a result, the chimp’s final level, which is portrayed as drivers, chasers, obstacles, and attacker levels, is optionally positioned in a circle.Development phasef is specifically defined in the chimp assault system. A vector is similarly decreased in the vector way f and, furthermore, is a self-determined variable at intervals of Before the specified time, the chimps will chase the victim through the attack progression even leaving the prey successively away. The value of A phase of exploitation with the help of a social incentivef coefficients with the thought of its collection technique. The best prey area differs in the importance of the driver, chaser, range, and attacker. Later, at that time, the levels are updated, separating from the prey, and there is better tuning of m and c. Additionally, the value of f can be changed from 2.5 to 0, which activates the false trend. In the long run, the level of individuality and emphasis is checked, which gives the best results to deal with the consequences.In COA, chimps’ social motivation and community were linked to meat hunting. The chimp can limit its hunting interactions in the final stage of the chimp hunting process. As a result, they are frantically trying to obtain prey meat for social forces. These attributes of the chaotic map are programmed with the chaotic maps created as follows:This section evaluates and supports the proposed technique’s presentation. In this division, proposed process exhibitions are approved through implementation and comparison testing. To recognize the presence of a projected MRI image detection, the proposed strategy is implemented on an Intel Core i5-2450 M CPU 2.50 GHz PC and 6 GB RAM. This strategy is implemented in Python programming. To approve the interpretation of the proposed strategy, data are collected from open-source system. The proposed technique is implemented and approved using performance metrics such as DSC, JSI, Accuracy, specificity, and sensitivity. The proposed cluster technique is used to distinguish breast cancer disease from MRI images.The confusion matrix is processed according to the limitations that come with it.The malignant regions are introduced and shown as detected which is termed as False Negative (FN).An actual malignant region is not detected, which is called a False Positive (FP).An actual malignant region is not detected, which is called an undiagnosed true negative (TN).An actual malignant region is detected as occurred and called as True Positive (TP).Given the improvement of the confusion matrix rules, the proposed process implementation is evaluated by measurements, which are defined as:Accuracy: It is classified into precision detected information examples at all times. The equation of accuracy is introduced as follows:Sensitivity: It can be categorized as a ratio of precisely detected positive examples, including positive events of the following:Specificity: It tends to be characterized as the proportion of accurately sectioned negative occasions to add up to negative examples which are formed as follows:DSC: It is defined as a similarity index computed based on the below equation:The proposed methodology is validated by similarity measurements of DSC and JSI, respectively. From In recent years, breast cancer detection is an important task to empower disease prediction as well as classification. In this paper, COA-T2FCM has been designed for detection. The suggested detection method employs a mix of type-2 intuitionistic fuzzy c-means clustering and an oppositional function. The efficient cluster center was selected with the use of the chimp optimization algorithm in type-2 intuitionistic fuzzy c-means clustering. Initially, the type-2 intuitionistic fuzzy c-means clustering objective function has been considered with the consideration of intuitionistic fuzzy information collected from the MRI images. After that, a chimp optimization algorithm is utilized to optimize the cluster center and fuzzifier from the clustering method. The projected technique is implemented in Python as shown in