The use of optical fiber probe in two-phase flow measurements is very frequently encountered, especially in the applications of chemical engineering and petroleum industries. In this work, the influence of bubble piercing signals caused by bubble deformation is studied experimentally using a laboratory-prepared wedge-shaped fiber probe in a lab-scale gas–liquid flow generator. A three-dimensional simulation model is established to study the influence of bubble deformation on the piercing signals. A theoretical analysis of the characteristics of the pre-signal influenced by the bubble deformations is undertaken for a wide range of different modeled bubble shapes. Combining the experimental and simulation results, a promising analytical method to estimate the bubble shapes by analyzing the characteristics of pre-signals is proposed. The results of this investigation demonstrate that it is possible to estimate the bubble shapes before the fiber probe contacts the bubble surface. The method developed in this investigation is therefore highly promising for reducing errors caused by deformation during the probe piercing process. Two-phase flow environment widely exists in chemical, petrochemical and biochemical industries. The accurate analysis of the movement of bubbles in liquid has become a significant challenge when making measurements in gas–liquid two-phase flows ,2,3. TheThe use of optical fiber probes for the measurement of bubble velocity, bubble size distribution and bubble void fraction has become increasingly widespread and popular for use in two-phase flow detection ,12,13,14In the study reported in this article, the experimentally observed pre-signal characteristics are established using a fully integrated wedge-shaped fiber probe for piercing different shapes of bubbles. The experimental results reported in this article show that when the fiber probe pierces a suborbicular air bubble and an approximate ellipse air bubble, the pre-signals are different. The measurements results show that the appearance of different pre-signals immediately prior to bubble piercing is strongly influenced by the bubble shape. However, in bubble flow measurements, the local interaction between fiber probes and gas–liquid interfaces is a transient phenomenon. Even if the fiber probe was fixed and relatively motionless, the uncertainties during the experiments introduced by the relatively random process of bubble evolution render it impossible to perform a quantitative analysis of the luminous intensity of the reflected light from the bubble surface for varying bubble shapes. Thus, further investigations were conducted using a ray tracing simulation method. A three-dimensional simulation model was established to study the influence of bubble deformation on the piercing signals by changing the radii ratio of bubble minor semi-axis, as well as the bubble aspect ratio. Based on an optical theoretical analysis of the simulation results, different characteristics of the pre-signals were determined by considering the bubble rear surface shapes which is a significant factor in the generation of the energy concentration area in front of the front bubble surface. In the authors’ previous work, the mechanism of energy concentration area appeared in front of the bubble surface before the bubble piercing process was discussed in detail . This stThe optical fiber sensing system employed in this study is the same as the one utilized in the author’s previous work  which deDue to the transient hydrodynamic process of the fiber probe bubble piercing, it was difficult to extract and analyze the pre-signal characteristics varying the bubble shapes purely from the experimental results. Thus, for the purpose of further analysis of the relationship between the bubble shapes and the measurement signal properties, a ray-tracing simulation method was implemented. A three-dimensional optical simulation model was established to simulate the process of different shapes of bubbles piercing in a cartesian coordinate system. The framework of the bubble piercing model is outlined in z-axis.The cube geometry (e) represents the water column. Inside the cube (e), an ellipsoid (a) represents the air bubble, the fiber probe comprises the fiber core (b) and fiber cladding (c), the illuminant (d) released from the end face of the fiber probe is represented as a conical beam of random rays with uniform density and distribution over the incident angle range from 0 to 20 deg along the direction of the negative z-axis. The algorithm for the optical calculation was described in detail in a previous article by the authors of this article [The wedged angle of the fiber probe was set to 50 deg which is the same as the experimental probe utilized in this investigation. In this model, no hydrodynamic characteristics of the bubble surface were included in the computation, but instead relied on a quasi-static optical modeling of the fiber position with respect to the bubble axis. The position of the fiber probe apex relative to the bubble was controlled by shifting the coordinates of bubble frontal surface using a step length of 0.1 mm in the  article .φ = b/a., the value of φ changed by varying the upper and lower radii of bubble minor axis, which correspond to the different shapes visualized in the experimental investigations.A modeled bubble was created in order to offer different bubble shapes for the purpose of making a quantitative analysis of the pre-signal characteristics varying with the bubble shapes. The axial plane of the modeled bubble is shown in A wedge-shaped Poly-Methyl Methacrylate (PMMA) optical fiber tip with a wedge angle of approximately 50 degrees was utilized to detect the bubble flow signal under the conditions of different shapes of bubbles in the laboratory-scale gas–liquid flow generator described in In the visualization experiments, the bubble deformation trend was observed and ranged from a near perfect sphere to oblate spheroid from the instant the bubbles escape the air bleeder and rise in the water. The fiber probe was located at different locations above the exhaust capillary in order to obtain the measurement signals of different shaped bubbles. The experiment was undertaken in a homogeneous continuous bubble flow where the fiber probe was placed approximately 10 mm above the stainless capillary as previously described ; Figure When the fiber probe was located at a higher position, being 15 mm above the air bleeder, there are clear differences in the time resolved experimental results. The captured bubble image is shown in Work reported previously by the authors of this article demonstrated that the appearance of the pre-signal is related to the reflected light generated at the air–liquid surface (front bubble surface) . Thus, tThe geometry and optical parameters of the modeled wedged fiber tip were the same as the experimental fiber probe used in this investigation. The horizontal axis of the simulation graphs represents the bubble frontal surface position relative to the fiber tip (corresponding to elapsed time), and the vertical axis corresponds to the calculated reflected optical output power signal from the fiber probe. The position that bubble frontal surface is just in contact with the fiber probe apex corresponds to the origin of the coordinate system.φ changed correspondingly. R1 and R2 were set equal. The graphs of φ. When the bubble shape was approximated to an oblate spheroid, a two peak pre-signal was generated which occurred just before the main bubble piercing rectangular pulse signal. With the bubble shape trending to a sphere, the first spike peak of pre-signal disappeared. These results are consistent with the experimental results presented in φ was investigated, including the reflection signals generated from the bubble rear and frontal surface, and the results of this simulation are presented in φ increasing, the bubble shapes approximate to a sphere, the fist spike peak signal approaches the second spike peak signal till the two spike peak signals superimpose, which makes the one-spike-peak pre-signal  are presented in The detailed bubble deformation effect on the reflection signals generated on both the bubble rear and frontal surface required further computational analyses and these were conducted by separately changing the value of the upper and lower radii of bubble minor axis, R1 and R2. The simulation of the single bubble piercing at different φ increased correspondingly and the results are consistent with those obtained in φ values increased correspondingly and are consistent with those of The results in It can be assumed that the modeled bubble in this simulation acts as a non-standard convex lens, the energy concentration area acts as an equivalent focus point, and the optic fiber probe acts as a moving point light source, the light emitted from the fiber tip is approximated as directional light. When the fiber probe was located at the equivalent focus point of the bubble (which was explained as the energy concentration area in the author’s previous work ), the fiThe appearance of the second peak of the pre-signal generated from the bubble fontal surface can be explained using the standard theory of light ray reflection ,28. As sThe experimental and simulation results combined indicate that when the bubble shapes trend to an oblate spheroid, the focal length of the equivalent focus point increased, and the first spike peak appears. Conversely, when the bubble shapes approximate to a sphere, the focal length of the equivalent focus point decreased, and the two spike peak signals superimpose which manifests as a single spike peak pre-signal. The discovery of this phenomenon and its explanation provides a highly promising analytical method to distinguish several different bubble shapes through obtaining the measured time resolved characteristics of the signals. In particular, by analyzing the peaks of pre-signal, the basic bubble shape before the bubble piercing can be determined, offering a promising improvement in measurement accuracy in both industrial and general chemical application fields.A wedge shaped PMMA optical fiber tip with a wedge angle of approximately 50 degrees was utilized to experimentally measure the time resolved bubble flow for a wide range of different bubble shapes in a laboratory-scale gas–liquid flow generator. In the visualization experiments, the bubble deformation trend was observed to evolve from near spherical to an oblate spheroid following their generation from the air bleeder and subsequent rise through the water column. Different pre-signal characteristics were observed in the bubble flow measurement when the fiber probe was placed at different locations above the exhaust capillary. In order to investigate the influence of bubble deformation on the characteristics of the pre-signal and discover the regular connection between the bubble shapes and the measurement signal, a ray-tracing based three-dimensional model was established to simulate the piercing process for different bubble shapes. Assuming the modeled bubble in this simulation acts as a non-standard convex lens, the simulation showed that changing shapes caused the focal distance of the equivalent focus point, which in turn leads to the changing of the maximum value and position of appearance of the peak caused by the bubble’s rear surface. The experimental and simulation results both indicate that when the bubble shapes tend to an oblate spheroid, the focal length of the equivalent focus point increased, and the first spike peak appears. Conversely, when the bubble shapes approximate to a sphere, the focal length of the equivalent focus point decreased and the two spike peak signals superimpose, which accounts for the appearance of the one-spike-peak pre-signal. Using this analytical method, it is possible to assess the bubble roundness in two-phase flow measurement, which is conducive to reducing the errors potentially caused by deformation by the piercing fiber probe.


We are developing a system for long term Semi-Automated Rehabilitation At the Home (SARAH) that relies on low-cost and unobtrusive video-based sensing. We present a cyber-human methodology used by the SARAH system for automated assessment of upper extremity stroke rehabilitation at the home. We propose a hierarchical model for automatically segmenting stroke survivor's movements and generating training task performance assessment scores during rehabilitation. The hierarchical model fuses expert therapist knowledge-based approaches with data-driven techniques. The expert knowledge is more observable in the higher layers of the hierarchy (task and segment) and therefore more accessible to algorithms incorporating high level constraints relating to activity structure . We utilize an HMM and a Decision Tree model to connect these high level priors to data driven analysis. The lower layers (RGB images and raw kinematics) need to be addressed primarily through data driven techniques. We use a transformer based architecture operating on low-level action features  and a Multi-Stage Temporal Convolutional Network(MS-TCN) operating on raw RGB images. We develop a sequence combining these complimentary algorithms effectively, thus encoding the information from different layers of the movement hierarchy. Through this combination, we produce a robust segmentation and task assessment results on noisy, variable and limited data, which is characteristic of low cost video capture of rehabilitation at the home. Our proposed approach achieves 85% accuracy in per-frame labeling, 99% accuracy in segment classification and 93% accuracy in task completion assessment. Although the methodology proposed in this paper applies to upper extremity rehabilitation using the SARAH system, it can potentially be used, with minor alterations, to assist automation in many other movement rehabilitation contexts . As the US and global populations age, we observe an increasing need for effective and accessible rehabilitation services for survivable debilitating illnesses and injuries, such as stroke and degenerative arthritis , 2. EffeTelemedicine and telehealth are gaining significance as viable approaches for delivering health and wellness at the home and in the community at scale . ApplyinBanks of video camera arrays can seem intrusive in the home and lead patients and/or their families to feel as if they are under surveillance . More prin situ and real-time, or when later rating videos of performance  system. The SARAH system comprises two video cameras, a tablet computer, a flexible activity mat, and eight custom-designed 3D printed objects, as shown in The system can be easily installed on a regular kitchen or living room table. The two cameras initiate and record only during training, and they are activated and controlled by the patient using a custom-designed application on the tablet computer. The system aims to integrate expert knowledge with data driven algorithms to realize coarse real-time automated assessment of movement of stroke survivors during therapy at the home . This asThe SARAH system is optimized for daily at home therapy of stroke survivors with moderate and moderate-to-mild impairment. Stroke survivors who score above 30 on the Fugl Meyer test and can initiate even minimal movement into extension of the elbow, wrist and digits when discharged from the clinic post stroke, can show significant improvement in function through repetitive training lasting 2 weeks to 6 months –29. CurrThis paper focuses on our hybrid knowledge-based data-driven approach to automated assessment of human movement in the home. Our approach leverages expert rubrics for standardized rating of overall task performance to inform automated rating of movement performance based on low cost, limited, noisy, and variable kinematic data. The assessment process and outcomes need to be compatible with therapist assessment approaches so as to assist remote therapists in using summaries of the computational assessment when remotely monitoring progress and structuring therapy at the home. Our approach has two components: (i) making the expert raters process as observable as possible; and (ii) leveraging the expert rating process to inform the structuring and improved performance of computational algorithms. In previous publications, we have presented in detail our research and development activities for the first component , 30, 31.not standardized. We worked with expert therapists to standardize the segment vocabulary into a state machine that can produce all 15 tasks of the SARAH system , and probabilistic composite observation of relevant features . This process is not well standardized as the filtering and compositing activities are based on individual experience and training. We further worked with expert therapists to define a consensus-limited set of composite movement features that are important when assessing the performance of each segment in our model . For exaBased on the background discussed above, we propose a cyber-human movement assessment hierarchy for upper extremity rehabilitation. The five layers of the hierarchy (listed top to bottom) are: overall task rating, segment rating, composite feature assessment, raw kinematics, and raw RGB images see also . As we mIn the following sections, we show how we leverage the observable expert knowledge of the higher levels of the hierarchy to improve the performance of computational algorithms using raw kinematics and visual features for automated task segmentation, segment classification, and task performance assessment at the home. As the patient selects the task they want to attempt using the tablet computer, the system knows the desired sequence of segments for satisfactory performance of the task. The expected topology of each segment of each task within our standardized activity space is also specified. We can thus utilize a feature-matching approach to measure the “distance” between the observed sequence of segments by the system and the expected sequence of states for the performed task to determine whether the task was successfully performed.The state-time characteristics of our hierarchy  motivates our use of automated segmentation though a hidden Markov model (HMM). We also leverage recent advances in machine-learning models for processing video data. Specifically, we tested the use of transformers  operatinWe used an earlier version of the SARAH system  to recorAll patients could follow the instructions given by the SARAH system and use their unimpaired limb to control the system. All patients could engage all objects of the SARAH system , 26 at lThe movement of the patients in this study was recorded using one consumer grade video camera (we used the side view camera of the SARAH system to capture the profile of the body and impaired arm as this is the preferred viewing point of the therapist). The camera placement instructions were relatively high-level to ensure that they could be implemented quickly and without interfering with the patient's therapy session. The research assistant collecting the videos was asked to place the camera on the side of the impaired limb of the patient. The camera had to be far enough so as to not interfere with the performance of the tasks and be able to capture the full upper body of the patient. No specific distance, height or viewing angle were given for camera placement. As expected, this process provided minimal interference and could be realized very quickly but produced high variability in the captured videos in terms of location of the patient, activity space in the image, lighting, camera height, and viewing angle.All collected videos were rated by expert therapists using a custom developed SARAH rating interface that allows therapists to rate the task, segment and composite movement features of each video recording in an integrated manner . TherefoOur segmentation and classification framework is illustrated in The transformer, the HMM, MSTCN++, and the rule-based decision tree pipelines have a feature extractor as demonstrated in x, y)-pixel coordinates of the skeleton joints as shown in We use OpenPose , an openIt was also imperative to obtain high quality object identification and location data for every frame, as the type of object being used and the relationship between the objects and the keypoints extracted from the subjects play an important role in determining segment classification. To this effect, we fine-tuned an object detection algorithm which classifies the objects in the frame, places a bounding box around the classified object, and provides the object/bounding box location relative to the overall frame. We considered a Faster-RCNN model  pre-traiAs stated in section 2.1, there is a high variability in the training data, owing to issues such as use of a single camera view, non-standardized camera angles, variability in the patient's movement, unconstrained ways of grasping the objects, errors in transportation etc. As a consequence, the object detection model misses certain objects in certain frames due to factors such as occlusions or dropped objects, and sometimes even misclassifies certain objects due to partial visibility. However, as described in section 1, the task being attempted in each captured video is known to the algorithm and each task is associated with specific objects. We use this knowledge to filter out misclassifications. To further improve the smoothness of the trajectories of the objects, we use a simple Kalman filter based tracking algorithm SORT . Given tThe activity space in different videos has high variability. As the camera angle and patient position are different for each captured session, the ratio of the activity space to the frame resolution changes. Low camera resolution and low frame-rate , variable focus (cameras were set up by non-experts without standardized instructions) and varied lighting conditions , create challenges for pose-estimation methods from the raw frames. As a result, the keypoint detection and bounding box estimation accuracy can suffer, sometimes even completely missing all keypoints and not detecting any bounding box in a frame. To reduce the missing data problem we first detect the outliers. We calculate the z-score for each sample using,xi is the keypoints or object locations for the ith frame . If the z-score value is higher than a threshold, we consider the sample as an outlier. For our case, using a threshold of 2.5 gives good performance. Then we use spline interpolation to fill the missing values or replace the outliers.where Denoising and normalizing the input data is a vital step in achieving good performance through machine learning. We experimented with three different techniques to find the best techniques for normalizing both the patient keypoints data and the object locations—(i) global normalization with respect to the image frame, (ii) normalization with respect to the activity mat area, and (iii) normalization using a computed homography matrix.Global-Norm: Here, we just divide the raw patient keypoints and object location data with the image frame's width and height, thus mapping the data to a  range. In practice due to the high amount of noise in the data, and variability in inter-patient performance, this global normalization fails to achieve good results and was therefore not applied (see Table 6).Mat-Norm: Here, we normalize the data with respect to the ratio of the area of the activity mat to the total image area. This is done by labeling the activity mat in one of the frames per subject per capture session. For every capture session the camera position is set at the beginning. We could thus precompute the area of the mat for all recorded tasks of each capture session. This normalization with regards to the mat area removes a good degree of variability and thus performs better than global normalization.Homography-Norm: Since the camera angles across subjects and across attempts vary considerably,it is important to reduce the impact of this variation. To achieve this, we apply a simple homography-based re-projection of the object and keypoint data. First, a homography matrix is pre-computed that transforms data from image frame to the coordinate frame defined by the activity mat. This transformation matrix is then applied to the object and keypoint data. Even though this achieves high invariance, it also removes discriminative features and thus there is not any observably significant improvement in the performance of the transformer models.Using the denoised keypoints and object locations, we calculate 20 kinematic features that, based on our prior work , can be S = S1, S2, …, SN as the N states. Evolution between states is modeled by the transition probability conditioned on the previous state. The transition probability between states is represented by a matrix, A = aij; where aij = Pr(Sj(t + 1)|Sj(t)). For a given observation, O = O1, O2, …, Ok, the emission probability distribution is represented by a matrix B = bj(k); where bj(k) is the probability of generating observation Ok when the current state is Sj, where k is the total number of observation symbols.A hidden Markov model (HMM)  consistsHMMs have been used in movement quality assessment because of their good performance in detecting subtle inconsistencies in the movement –47. HoweS = {IPT, MTR1, MTR2, R&R}. Each state is initialized with a normal distribution of mean, μ = 0.5, and standard deviation, σ = 0.1. We choose the normal distribution because it produced the lowest chi-square error after intensive experimentation with different distributions like beta, exponential, gamma, log-normal, normal, Pearson3, triangular, uniform, and Weibull. The distribution parameters can be randomly initialized, and we find that the model is not sensitive to random initialization. The training of the HMM tunes the parameters which are then compactly represented as,To train the HMM, we first need to define the initialization for each state in the transition matrix. For example, to initialize an HMM with T4 transition matrix, we need four states: T and F refer to different transition matrices and features and π is the initial state distribution and is defined as π = πi where πi is the probability of state Si being the initial state. Based on the tasks, π = . Since all the tasks start with the state IPT, the first state probability is always 1 followed by zeros for other states present in the transition matrix. This is another strong design prior we leverage, in the form of supervision of the labels with state names. In this case, maximum likelihood estimation (MLE) estimates the emissions from data partitioned by the labels and the transition matrix is calculated directly from the adjacency of labels. Then, various transition matrices and their initialization are further tuned with patient data. As we provide more training data, the final models are expected to become more generalizable across different types of tasks. Another prior we use for structuring our HMM relates to the use of distributions of individual kinematic features. Therapists establish through experience an expected distribution per feature per segment . Therefore, to learn the distribution parameters for each of the kinematic features, it is a requisite to train separate HMMs for each of those features.Here, the subscript We use the open source toolbox Pomegranate  for the p is the predicted state probabilities. N is the total number of observations in the training set, Tmax is the maximum observable frame number, F represents the total number of features, and w is the weight for optimization. We optimize the objective function by adding two constraints on the weights. Firstly, restricting the range of weights w ∈  and secondly, constraining the weights to unit sum: where,  (SLSQP)  due to iN is the number of videos, F is the total number of feature and Tmax is the maximum observable frame number including the zero-padding, (ii) partition the data X into training Xtr, validation Xval, and test Xte sets randomly, with a constraint that no patient-task pair should be repeated in train and test sets, (iii) train the transformer with the objective to find a function g that maps Xtr to Y, where Ytr ∈ ℝTs1 ×  is the per-frame label vector, by minimizing cross entropy loss between f(gtr) and Ytr. We train the network with different augmentations on the input data such as random drift, Gaussian noise etc using   are showWi is the weight assigned to each model's prediction Pi. To determine the weights Wi, we have conducted a grid search for each of them in the range of . Thus, for a three model ensemble, we run the evaluation 10 × 10 × 10 = 1, 000 times for each combination of weights. We experimented with four ensemble schemes to find the combination that provides the most robust results: (i) HMM and Transformer, (ii) HMM and MSTCN++, (iii) Transformer and MSTCN++, and (iv)HMM, Transformer, and MSTCN++.Finally, we fuse the outputs of the data driven models with the expert knowledge constrained models. The HMM algorithm uses kinematic features and design priors that are constrained by the expert knowledge. The predictions from the HMM encode information about the segments and task level of the movement analysis hierarchy shown in To perform task completion assessment we need to organize the per-frame segment labels into segment blocks. We again leverage therapist expertise for grouping segment samples into segment blocks (as embedded in the SARAH system design), to automatically extract segment blocks and assess the order, duration, and continuity of the blocks. To calculate segment blocks, we first denoise the ensemble predictions. The per-frame segment predictions resulting from the ensemble model have two types of noise: false transitions and missing transitions.Gs, where μs and σs denotes to the mean duration and standard deviation of each segment, s calculated from the training data. For a test data of length, N, we calculate the posterior distribution and use a 75% confidence cutoff to get the upper and lower bounds for each segment. We only accept a predicted transition, if it falls within the bounds. In addition, we pose a 20 frame window constraint on the predictions since all the task segments in the SARAH system require more than 1 s to complete. Therefore, any transitions within 20 frames is discarded. This filtering technique reduces the majority of the false predictions.In the SARAH system, the performance of all tasks is realized through a sequence of a small number of segments. These segments are performed with a defined set of objects, and within a defined activity space. Further, they are executed at a speed that is constrained by human biomechanics and potentially constrained by stroke induced impairment. We can filter some of the false predictions through a Gaussian filter for each transition model that implements these priors. In To correct the remaining false predictions and find the missing segment transitions, we use a rule-based binary decision tree (RBBDT) to encode the process that the experts (therapists) use to segment the patient movement. As observed during the development of the SARAH system with therapists, instead of classifying each frame of movement, therapists utilize a few key events to find transition point candidates between segments. These events are primarily based on three relationships: (i) patient–object interaction, (ii) patient–activity space relationship, and (iii) object–activity space relation. People execute functional tasks differently . These differences are much more pronounced for stroke survivors as they have different types of impairment. Therefore, therapists organize the patient activity space into a few generalizable regions that are robust to variation. Standardized stroke rehabilitation assessment tests  rely on such generalizable regions and the regions have accordingly been adopted in the design of the SARAH system (see section 1).To model relationships between the object(s), patient, and the generalizable activity regions, we recreate the activity regions as eight bounding boxes on each frame of the video. These bounding boxes are illustrated in We use previously calculated object and end point position and velocity to mark the object and limb end point stopping time. We can then calculate multiple distance features like limb-bounding box distance, object-bounding box distance, and limb-object distance. These simple distance features do not exhibit a simple statistical pattern  across different patients for each type of segment and are therefore not easily amenable to training the HMM segmentation algorithm. However, under the RBBDT framing they did indeed become usable for detecting segments blocks because they encode the features used by experts to define segment blocks as shown in In our previous work , we usedLet's consider the previous example of exercise three and four where the patient transports the object close to their mouth to simulate drinking action. To get the candidates for the IPT-MTR1 transition, the highest observable feature is the location of the object and limb as a phase change occurs at the beginning of MTR1. The next most observable feature is the directionality of the object's movement in relation to the bounding boxes. The process for calculating the confidence value of the candidates for the above IPT-MTR1 transition is illustrated in For a given task, the segment blocks will inform us about the continuity and order of the segments. In addition, we can calculate the duration of the completed task. Once we have calculated the segment blocks and duration for each task, we can compare that calculation to the expected sequence of segments and duration for each task for a coarse assessment of task completion. The algorithm checks for three conditions:Is the task completed with all the necessary segment blocks?Is the order of the segment blocks correct?Is the task performed within allowable time?If a task satisfies all three conditions, the algorithm gives a score of 3. If the first two are satisfied then it generates a score of 2. If either of the first two are not satisfied then it generates a score of 1. Finally, a score of 0 is assigned if the task is not attempted. Although these scores are similar to the scores assigned by therapists they do not take into account issues of movement quality. Therapists may assign a score of 2 to a task that is completed within the allowable time because of issues of movement quality. Since our methodology connects different types of movement features to assessment of segment and task execution, in the future we plan to use actual therapists ratings of patient videos to further train our algorithms so they can automatically assess task completion and movement quality.Our initial dataset consists of 610 captured videos. The experimental dataset includes 404 videos after exclusion of videos where severe limb impairment caused multiple object dropping or multiple segment occurrence leading to an incomplete exercise. Including these videos would severely skew the training sets at this early stage. However, some of these videos were included in the test sets for task completion and in the future these videos can be included in the training sets too. In our dataset each patient performed each task multiple times in a single session. Therefore, we design random split experiments based on three factors: patient ID, session ID and task number. For any task number, the same patient ID with the same session ID is included in either of the training or test set but not in both. We choose different random seeds and create 5 random experiments for fairness based on this selection method. The average training and test size of the 5 splits are 370 and 34, respectively. We implement the deep learning experiments on two NVIDIA RTX 2080 Ti GPUs. To evaluate the segmentation performance of the experimented models, we calculate frame-wise accuracy, precision, and recall. The calculation formula for the matrices are given in the appendix section. In the We experimented with the proposed HMM using various numbers of features. The feature selection is an important factor as the combination of correct features captures the unique patterns of different transitions. To understand the effect of feature selection, we perform three experiments: (i) 13 kinematic features; (ii) six kinematic features including four derivatives; and (iii) one composite feature. To get the composite feature from 13 kinematic features, we experimented with different dimensionality reduction techniques like PCA, NMF, LDA, and RP –61. BaseThe proposed HMM model uses five transition matrices. We also experimented with one transition matrix model to understand the capability of the HMM to learn different variations of segment transitions across all types of tasks. A 6 × 6 transition matrix is necessary to represent all the segment transitions across all performed tasks in our data set. In We also experimented with time dependent models and wanted to exploit the duration of segments to correct the predictions from the HMM. To model time dependency in the HMM predictions, we use the gamma correction  on the pwhere, α and β is calculated per segment using mean (μ) and standard deviation (σ) of the segments in the training set. Since the dataset has a very high standard deviation, the gamma function [Γ(α)] results in a large value and thus Equation (4) generates probabilities < 0.01. Therefore, the effect on the HMM predictions are negligible. In Like the HMM, the transformer was also tested on five random splits. In We experimented with different number of keypoints while always using the object features as input to the transformer based pipeline. We decreased the number of keypoints from 8, 4, 2, and then to 1. We selected the keypoints for each reduction based on our prior work  that defWe experimented with reducing the amount of training data to the transformers to better study the generalization properties of our pipeline in low data scenarios. This is a practical setting, as the amount of time required to label the data is very high. However, as transformers contain large number of parameters, it is important to study the over fitting trends as well. We report average accuracies on the same test data for fair comparison while changing the size of input training data. As can be seen from the The results for the five split experiment using the MS-TCN++ is shown in We experiment with three combinations of model ensembles. In In In In In this paper, we propose a hierarchical model for automatically segmenting stroke survivor's movements and generating task performance assessment scores during rehabilitation training. The hierarchical model fuses expert knowledge-biased approaches with data-driven techniques. The expert knowledge is more observable in the higher layers of the hierarchy (task and segment) and therefore more accessible to algorithms incorporating high level constraints relating to activity structure . The lower layers need to be addressed primarily through data driven techniques. By developing a sequence for combining complimentary algorithms that effectively encode the information from the different layers, we produce robust segmentation and task assessment results driven by noisy, variable, and limited data.The MSTCN++  is a datThe Transformer utilizes raw kinematic features of the torso, upper limb, and object. We experimented with 8, 4, 2, and 1 keypoints from open pose combined with object movement features. We selected the keypoints representing joints that have been show, in our previous work, to have prominence in characterizing upper limb impaired movement , 34. It The hidden Markov model (HMM) utilizes kinematic features that are prominent in characterizing upper limb functional movement  and alsoR&R. Since this prior is incorporated into the HMM, the predicted order of segment blocks are correct and the per-frame segment label prediction accuracy increases. As is evident from For all the splits, the ensemble model with three algorithms performs the best in the frame level classification. This proves that a fusion of algorithms that have different sensitivities to different layers of the hierarchy outperforms individual models. Without the HMM, the ensemble model predictions have a higher misclassification rate and incorrect sequence orders. For example, without the HMM a prediction can start with MTR or end with IPT. But based on our design prior, we know that all exercises start with IPT and end with To automatically assess the completion of a task, we need to assess the completion and order of segment blocks. This cannot be done solely through the ensemble model as the ensemble model has many false positive blocks and missed transitions. The connection of the segmentation layer to the task assessment layer requires an algorithm that is heavily biased by expert knowledge toward behaviors characterizing transition between segments, as well as the effect of stroke related impairment on these behaviors. The RBBDT algorithm successfully incorporates key unimpaired and impaired movement priors for segment block transition points in one integrated decision sequence. For example, a phase transition is expected for both object and limb end point for a simulated drinking motion (task three) when the object gets close to the head. However, not all stroke survivors are able to lift the object all the way to their mouth, so the phase transition can sometimes happen away from the head. Thus, the phase transition becomes the first decision node with the place of the transition in the activity space the second node. The RBBDT is highly sensitive to transition point features but not sensitive to per-frame data patterns across the whole task. Thus, the RBBDT produces low quality performance in per-frame classification but higher quality performance in identifying transition candidates. When integrated with the ensemble model, the correct identification of segment blocks rises above 90%.To perform the task completion assessment, we use a very simple algorithm relying only on comparison of the type and order of the segment blocks given by the integrated ensemble and RBBT predictions, with the expected type and order of segments in tasks established by the expert therapists who designed the SARAH tasks. By adding this codification of prior task knowledge to our analysis, we are able to correctly classify completed and uncompleted tasks over 90% of the time. We are also able to classify completed tasks that took longer than the performance time allowed by the therapist for impaired performance. These results further enhance the validity of the use of a hierarchical model of automated analysis combining algorithms with different sensitivities to the different layers of the hierarchy. This includes layers primarily incorporating observable expert knowledge (higher layers), layers primarily incorporating computer analyzed data (lower layers) and layers integrating data and expert knowledge , or move to another simpler or harder exercise (if they have repeated the exercise enough times or are not making progress on the current exercise).To realize this type of interactive training at the home the system needs to have the capacity to assess each performed task within 10 s from its completion. During these 10 s patients rest and self-asses their performance of the previous task. Then the system provides the automated assessment and the prompt for the performance of the next task. The results of our run-time comparison (section 5.5) show that with some further optimization of our algorithms we can potentially automatically segment video recorded tasks and assess task and segments completion within 10 s from completion of each performed task.Our team is currently working to develop a hybrid approach where we will combine the vision-based and IMU based techniques to fully capture movement patterns of stroke survivors. We will use four IMU sensors that will record accelerometer data of both hands, one index finger, and the pelvis. The ring sensor on the finger will capture the hand openness that will differentiate between different grips and release patterns. Our team is currently developing techniques to reconstruct 3D skeleton from accelerometer data using the other three sensors.Our computational hierarchy, since it is based on computational data, realizes the hierarchical analysis in a bottom up manner . Therapists utilize a similar hierarchical analysis in a top down manner  so as to leverage their heuristics about functional task performance. Therefore, the SARAH system can send a daily summary of training results to a remote therapist. Since the computational results will be highly compatible with the therapist assessment heuristics, the therapist will be able to quickly use this summary to structure the next day's therapy and send a message to the patient guiding their training the next day. The therapist will also be able to select any analysis result and review the corresponding video thus further informing their assessment and therapy adaptation. The feasibility of low-cost semi-automated rehabilitation at the home using the SARAH system will allow the collection of many more videos used to further train our algorithms. We are currently planing to use the SARAH system in a pilot study in the homes of stroke survivors in the Spring of 2022.Our team also developed an intuitive assessment interface allowing expert therapists to rate videos of therapy tasks in a top down hierarchical manner, complimenting their assessment approach: rate the overall task, then the segments, then the composite features per segment, and then return to a final assessment of the task) . As the The datasets presented in this article are not readily available because as per instruction from the institutional review board (IRB), the authors are not allowed to disclose the raw data. The processed features supporting the conclusions of this article will be made available on request to the corresponding author.The studies involving human participants were reviewed and approved by The Emory University IRB. The patients/participants provided their written informed consent to participate in this study.All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


Aiming at the shortcomings of traditional recommendation algorithms in dealing with large-scale music data, such as low accuracy and poor real-time performance, a personalized recommendation algorithm based on the Spark platform is proposed. The algorithm is based on the Spark platform. The K-means clustering model between users and music is constructed using an AFSA  to optimize the initial centroids of K-means to improve the clustering effect. Based on the scoring relationship between users and users and users and music attributes, the collaborative filtering algorithm is applied to calculate the correlation between users to achieve accurate recommendations. Finally, the performance of the designed recommendation model is validated by deploying the recommendation model on the Spark platform using the Yahoo Music dataset and online music platform dataset. The experimental results show that the use of improved AFSA can complete the optimization of K-means clustering centroids with good clustering results; combined with the distributed fast computing capability of Spark platform with multiple nodes, the recommendation accuracy has better performance than traditional recommendation algorithms; especially when dealing with large-scale music data, the recommendation accuracy and real-time performance are higher, which meet the current demand of personalized music recommendation. With the continuous development of network and information technology, music works have exploded and gradually formed a vast music work library. Personalized recommendation has become a problem that needs to be solved by powerful music platforms , 2. The Based on this, some scholars have proposed improvements and optimization based on the original algorithm models such as collaborative filtering, content, and model and have achieved certain achievements. Lampropoulou et al.  proposedAlthough the above recommendation algorithm can accomplish music recommendations well, there is still a large room for improvement. Therefore, this paper proposes a K-means clustering music recommendation method based on AFSA optimization based on the Spark platform. The improved AFSA is used to solve the global optimization-seeking and local extremum problems of the K-means algorithm to improve the accuracy of multicategory clustering. The collaborative filtering algorithm is also used to deal with the similarity between users and improve the clustering effect and recommendation efficiency by the parallel computing capability of multiple nodes of the Spark platform.Apache Spark is a distributed computing engine specialized in processing large-scale data, with the advantages of fast data processing, strong stability, and scalability and support for multinode collaboration, which can complete data stream processing, interactive query, and batch processing in real time , 10. As Spark platform uses DAG (directed acyclic graph) job scheduling scheme. The dependencies between different data are specified by building a model of data relationships in RDDs (resilient distributed datasets). To facilitate the iterative computation of data, the results of the distributed analysis are usually stored in memory. The RDD is a read-only, partitioned, and immutable dataset. Since Spark uses the RDD computation model to do most of the data computation, the model calls more methods and does not need to focus on the underlying scheduling details . This fuC = {ci, i = 1, 2,…, m} with m sample points; each sample has an N data dimension feature description  is the increment of that state.where The optimized foraging behavior discards the phenomenon of artificial fish randomly advancing and provides a better possibility of running for fish movement, which can effectively avoid artificial fish falling into local optimum and can further improve the accuracy of the search results.(2)When the clustering behavior or tail-chasing behavior fails, the artificial fish no longer performs the foraging operation but executes the random swimming behavior, and the optimized unexpected moving behavior is defined and expressed asIn the foraging behavior of AFSA, By optimizing the forward rule after the failure of clustering and tailing behavior, we can reduce the invalid brash behavior of artificial fish, thus effectively reducing the algorithm running time and fundamentally reducing the computational complexity.σ is the attenuation factor with forwarding direction to constrain the step length, the field of view, and forward movement of the artificial fish, σ ∈ . Then, the equations of adaptive step length and adaptive decay field of view can be expressed asThe two parameters of the artificial fish's field of view and step length determine the current search range and the subsequent moving speed. When artificial fish are searching far away from the extreme point, they should have a larger field of view and step to improve search speed. Also, when the fish in the area nearer to the extreme value point are searched, the values of the field of view and step length should be small to ensure the accuracy of the search. Considering the global fixed values of the field of view and step length in the original algorithm, it is easy to fall into the dire situation of blind search , 20. In The iterative process of AFSA is a continuous process of artificial fish aggregation to the global extremum. By introducing a decay factor, the fish can dynamically adjust the field of view and step size according to the external environmental information during the search process. In the early stage of the algorithm's operation, the artificial fish use a larger field of view and step size to conduct a coarse search in a wide range near the optimal value, thus speeding up the individual search speed and making the algorithm converge quickly. After several iterations, the artificial fish gradually approaches the extreme value. At the end of the iteration, the field of view and step size is adjusted adaptively by the attenuation factor to enhance the local search ability and reduce the oscillation range. The artificial fish can quickly find extreme global points with high accuracy. Especially in the case of multidimensional data, the algorithm of finding the optimal effect with the introduction of the decay factor is more prominent.I represents the set of shared items jointly rated by users uu and uv, ruc and rvc represent the ratings of items ic by users uu and uv, respectively, and uu and uv, respectively.After clustering by the AFSA-optimized K-means algorithm, the user's rating matrix of all resources and services to be recommended can be obtained. The collaborative filtering algorithm can find users with the same rating items in this rating matrix and calculate the similarity to complete practical personalized music recommendations. There are many kinds of filtering algorithms, but PPMCC (Pearson product-moment correlation coefficient) solves the difference of each dimension, and the similarity calculation process focuses on the whole so that it can complete the multidimensional examination of the same user . So, thiThe recommendation process of K-means clustering mining based on AFSA optimization is shown in To verify the performance of the AFSA-optimized K-means clustering personalized music recommendation algorithm on the Spark platform, public and proprietary datasets are tested separately. The public dataset is Yahoo Music, a classical dataset for testing recommendation systems which can well validate the recommendation performance of clustering mining. The dataset contains large-scale user and music resource data to validate the recommendation system's performance adequately. The experimental dataset is shown in pb denotes the predicted score, qb denotes the actual score, and N represents the number of items. Smaller RMSE values indicate better algorithm performance.The clustering center K value is more sensitive to the music scoring matrix and has a more significant impact on the stability of collaborative filtering recommendations. Therefore, this paper differentiates the K values and uses RMSE (root mean squared error) as an evaluation index to verify the recommendation accuracy under different K values . The expThe data from the Yahoo Music dataset were randomly taken and formed into three sample sets of different sizes YM-Data1 (13.82 MB), YM-Data2 (43.39 MB), and YM-Data3 (140.24 MB), and then experiments were conducted with the three sample data, respectively, to derive different K-value recommendation accuracy RMSE values, as shown in K value is 20, 20, and 22. When increasing the K values, the RMSE values gradually become more extensive, and the recommendation stability worsens. Therefore, in the subsequent training process, the range of K values is set to .As shown in The K value is set to 20, and the designed A-optimized K-means algorithm is used for clustering mining. The recommendation algorithm model is used for training the Yahoo Music sample set. All master nodes and work nodes are involved in the computation during training. The recommended performance of the three sample sets is shown in As shown in To further validate the recommended performance of this method, a dataset from an online music platform is used for testing. First, four sets of sample sets with different capacities based on music platform are built: KW-Data1 (10.75 MB), KW-Data2 (474.84 MB), KW-Data3 (1.62 GB), and KW-Data4 (6.65 GB). Then, cluster mining is applied to each data sample to get the score between all users and music resource attributes. Finally, the recommended results are obtained by using the collaborative filtering algorithm. KW-Data2 is used as an example to visualize its clustering results to facilitate the observation of the clustering effect, and the result is shown in To make the results comparable, the algorithm performance was experimented with differentially setting the number of clustering centers for K taking values of 18, 20, and 22. The obtained results are shown in As can be seen from T1 denotes the recommended time for a single node and Tρ denotes the time used for multinode parallel computation under the Spark platform. The speedup performance at the different number of nodes is shown in The higher the speedup ratio is, the more parallelized the platform is. The acceleration ratio expression isFrom Five hundred samples were randomly selected from each online music platform sample set. A new selection set KW-Data5 was constructed to test the SVM classification algorithm proposed in , the XG-As can be seen from To address some shortcomings of the traditional music recommendation algorithm, the improved AFSA is applied to solve the global optimization and local extremum problems of the K-means algorithm, thus improving the effect of multicategory clustering, completing the personalized recommendation of music by collaborative filtering algorithm, and improving the speed of large-scale data processing by using the parallelized computing feature of Spark distributed platform. The experimental results show that the AFSA-optimized K-Means clustering mining algorithm has good recommendation capability, and the recommendation accuracy is improved by 6.77% and 15.32% compared with the commonly used SVM and XG-Boost methods; the algorithm iterates faster to meet the requirements of real-time recommendation, which is enough to improve the recommendation efficiency of personalized music effectively and can be used as a reference for music, video, and similar fields, with good guiding significance.


Accurate remaining useful life (RUL) prediction of bearings is the key to effective decision-making for predictive maintenance (PdM) of rotating machinery. However, the individual heterogeneity and different working conditions of bearings make the degradation trajectories of bearings different, resulting in the mismatch between the RUL prediction model established by the full-life training bearing and the testing bearings. To address this challenge, this paper proposes a novel RUL prediction method for roller bearings that considers the difference and similarity of degradation trajectories. In this method, a feature extraction method based on continuous wavelet transform (CWT) and convolutional autoencoder (CAE) is proposed to extract the deep features associated with bearing performance degradation before the degradation indicator (DI) is obtained by applying the self-organizing maps (SOM) method. Next, a dynamic time warping (DTW) based method is applied to perform the similarity matching of degradation trajectories of the training and testing bearings. Driven by the historical DIs of the given bearing, the grey forecasting model with full-order time power terms (FOTP-GM) is applied to model the degradation trajectory using a parameter optimization method. Then, the failure threshold of the given testing bearing can be determined using a data-driven method without manual intervention. Finally, the RUL of the given testing bearing can be estimated using the preset failure threshold and the optimized degradation trajectory model of the given testing bearing. The experimental results show that the proposed method retains the individual differences of bearing degradation trend, realizes the independent and reasonable bearing failure threshold setting, and improves the prediction accuracy of RUL. Accurate remaining useful life (RUL) estimation of bearings is a significant challenge in the prognostics and health management (PHM) system for rotating machinery to improve the equipment reliability and reduce equipment failures as well as maintenance costs. In the literature, the prognostic approaches in a PHM framework can be implemented in three different ways: physics-based approaches, data-driven approaches, and hybrid approaches (a combination of data-driven and physics-based approaches) . By solvWithout considering the complex degradation mechanism of the system, the data-driven-based prognostic approaches can reduce the dependence on the amount of prior knowledge and have the advantages of high prediction accuracy and strong applicability. From the perspective of mathematical modeling, the data-driven prognostic methods can be further divided into statistical methods and artificial intelligence (AI) based methods . StatistThe main way to perform an AI-based framework for bearing RUL prediction is machine learning (ML) based prognostics, including artificial neural networks (ANN) , supportThe deep-learning-based prognostic methods can learn the deep information of original data and accurately evaluate the degradation status of bearings. The deep-learning-based methods applied to predict the bearing RUL mainly include: deep belief networks (DBN) , integraTo address the mismatch between the pretrained RUL prediction model and the test bearing, one approach is to map the pretrained model or the testing data by employing transfer learning (TL) approaches to adapt the prediction model to the test bearing domain . To thisBuilding a degradation model for the given bearing using practical features that can characterize the performance degradation of bearingSetting practicable failure thresholds for the testing bearings with different performance degradation patternsEstablishing a prediction framework with a good generalization ability that can update the model parameters dynamically according to bearing test informationAnother approach for cross-domain RUL prediction problems is the degradation indicator extrapolation method. The DI extrapolation method sets a reasonable failure threshold according to the data characteristics of the given test bearing, then establishes a fitted degradation model of the test bearing according to its historical DI curve, and finally realizes the RUL prediction of the given testing bearing based on the preset failure threshold and fitted the DI curve. Compared to the black-box transfer learning-based methods, the DI extrapolation method has good physical interpretability. The challenges in this method are how to fit the bearing historical DI curve quickly and accurately and how to determine the failure threshold automatically with no human intervention, in which three issues should be addressed carefully:As an approximate exponential model, the grey forecasting model (GM) can analyze the internal law of the grey system with fuzzy structure and incomplete or uncertain exponential data, showing a high forecasting precision and robust performance. K. Peng et al.  proposedThe accuracy of bearing RUL prediction is not only affected by the bearing degradation model but also closely related to the failure threshold. A reasonable failure threshold can make the facility managers carry out more effective maintenance depending on the bearing health condition. In the existing literature on data-driven based bearing RUL prediction, there are few discussions on the setting of failure threshold, and most of them are set manually, which mainly includes four setting methods: manual empirical method , vibratiT et al. [Some researches have been done to fill this gap. Wang T et al.  proposedT et al.  proposedT et al. , which cGiven the challenges and discussion above, this work proposes a bearing RUL prediction approach that considers the difference and similarity of bearing degradation trajectories. In the proposed method, the DTW is used to measure the similarities between degradation trajectories of the training and testing bearings, and the FOTO-GM  model is utilized to fit the degradation trajectory of the given bearing with high simulative precision. Firstly, the time-frequency diagrams of the training and testing bearings are generated by CWT before the deep features associated with the performance degradation are extracted by inputting such time-frequency representations into the CAE network. Secondly, the degradation trajectories of the training and testing bearings can be obtained by inputting such hidden features into the pretrained SOM networks. Then, the fitted degradation trajectories models of the training and testing bearings are obtained by the FOTO-GM  model, and the failure thresholds of testing bearings can be determined using the fitted degradation curves of the training bearings and the degradation trend distances between the training and testing bearings measured by DTW. Finally, the RUL prediction of the testing bearings can be realized using the preset failure thresholds and fitted degradation curves of testing bearings.A CWT-CAE-SOM-based bearing degradation indicator construction method is proposed without manual steps of signal feature extraction, selection, and fusion. This method extracts the hidden deep representations driven by the monitoring data of the given bearing to meet the consideration of the difference and independence between the full-life training bearing and test bearings. Then the DI can be obtained using the SOM network pretrained by the normal state deep features without considering the influence of the insufficient degradation data of the test bearing.A DTW-based bearing degradation trend similarity matching method is proposed. This method calculates the DTW distances between historical degradation trajectories of the training and testing bearings, and the training bearing with the minimum DTW distance to the given test bearing can be selected as the reference bearing of the given test bearing.A parameter optimization method for the FOTP-GM  model is proposed to determine the optimal order of time power terms. In this method, the fitting error between the fitted degradation curve and the original degradation trajectory is selected as the evaluation metrics for the training phase, and the distance between the fitted degradation curves of the given testing bearing and corresponding reference training bearing is considered as the selection criteria in the testing phase, which fully considered the difference and similarity of the degradation trajectory of the training bearing and the test bearingA data-driven-based bearing failure threshold setting method is proposed. This method adaptively determines the bearing failure threshold of the given testing using the DTW distance between the given testing and reference training bearing and the life endpoint value of the fitted degradation curve of the reference training bearing, which avoids the blindness of the artificial subjective of the failure threshold setting.The proposed framework is evaluated on the IEEE PHM 2012 Challenge data sets  and the The remainder of this paper is organized as follows. ψ ∈ L2(R) is usually a function with zero average and finite length, where L2(R) is the space of square-integrable complex functions [a > 0 is the scale factor for dilating or contracting the wavelet and b is the shifting factor for transitioning the wavelet along the time axis. For the given bearing signal x(t), the CWT operation decomposes the signal x(t) into wavelet coefficients according to the following integral:ψ∗ is the complex conjugate of mother wavelet ψ. The CWT is useful for obtaining the frequency components at different time scales and resolutions. For small scales (a > 1), ψa,b(t) will be short and of high frequency, while for large scales (a < 1), ψa,b(t) will be long and of low frequency.The time-frequency representation provides the joint distribution information of the time domain and frequency domain, and the continuous wavelet transform method is applied to extract the time-frequency distributions of the monitoring bearing signal effectively. The CWT can decompose the given bearing signal into a time-scale plane representation by scaling and shifting the mother wavelet. A mother wavelet unctions . The famIn view of the individual variability of bearing degradation trends, the CAE is used to automatically extract degradation characteristics from bearing monitoring data sets and realize the effective extraction of bearing performance degradation features without prior knowledge of bearing RUL. Compared with the traditional autoencoder, CAE, as an unsupervised deep learning method, uses convolutional operation for the encoding and decoding part instead of slicing and stacking the data, which significantly improves the performance of training parameter optimization and feature extraction .TFi denotes the time-frequency map of the given bearing data Xi at time i, where TFi ∈ RL1×L1×D, and the H represents the potential latent space representations of the given bearing time-frequency data, where H ∈ RL2×L2×K. The kth feature map in encoder output H can be expressed as follows:σ(·) is the nonlinear activation function, ∗ represents 2-D convolution, and ωk and bk denote the weights and bias value of the kth convolution kernel of the encoder, respectively. Then the kth encoded hidden representation can be decoded to reconstruct the input data TFi using deconvolution operation as follows:E denotes the mean-square-error (MSE) distortion between the original input image TFi and reconstructed image E can get an optimal hidden space representation of the input bearing time-frequency data TFi, which can be used as a deep feature of bearing performance degradation at that time i.The convolutional network in the CAE encoder encodes the input data into a set of hidden space representations, and then the decoder reconstructs the input data using deconvolution operation. As shown in To improve the training efficiency and generalization capability of the CAE network, multiconvolution layers are usually adopted, and each convolutional layer is followed by a batch normalization layer to make sure that the inputs and outputs of each layer have the same amplitude distribution with input data, which make the CAE can use a larger learning rate for training, accelerate the training speed, and overcome the influence of covariance offset.Due to the different sensitivities to track the degradation trend of bearing performance, the multiscale depth representations extracted by CAE cannot reflect the hidden information of bearings in the degradation process in a unified manner, and it is necessary to map such depth features into a unified performance degradation indicator. To obtain an accurate DI curve of the given bearing, the SOM is used to fuse the multiscale depth representations extracted by CAE into a nondimensional DI.Hi denotes the n-dimensional hidden representations of the given bearing time-frequency data TFi output by encoder, Hi=, where i=1,2,…, N, N is the number of bearing training samples. Wj= is the weights of the j th neuron in the SOM network, where j=1,2, ⋯, M, M is the neuron number of the SOM network. The construction process of DI is illustrated as follows:H1 is firstly input into the SOM model, and the winning neuron c is selected according to the minimum Euclidean distance standard: H1 − Wc=minH1 − Wj. Then the weight vectors, learning rate, and neighborhood radius are updated according to the method mentioned in reference [The normalized eference .H2 ~ HN, are input into the SOM network in turn to do the same operation, and when all training samples participate in SOM training, one iteration ends. When the iteration times reach the set threshold, the SOM model established by normal feature samples can be obtained.The training samples, Ht at time t and the weight vector W:All the historical time-frequency diagrams of the given bearing are input into the trained SOM model, and the DI trajectory can be obtained using the minimum value of Euclidean distance between hidden feature Let DI(t) measures the deviation degree of the bearing hidden representations between degradation conditions and the normal condition in kernel space performed by SOM. A higher DI value represents a more severe degeneration in bearing performance.The Dynamic time warping, which can effectively cluster time series with noise and time distortion, is an effective pattern dissimilarity measurement technique. DTW-based bearing degradation trend matching is a nonlinear regularization method that combines bearing operation time regularization with degradation trajectory distance calculation. It takes the historical degradation curve of the given testing bearing as the reference template, compares the full-life degradation trajectories of the training bearings with the reference template one by one, and finds the training bearing with minimum trajectory distance as the degradation trend matched training bearing of the given test bearing.DItrain denotes the full-life degradation trajectory of a training bearing, and the notation DItest represents the degradation trajectory of a testing bearing, where DItrain ∈ RNtrain1× and DItest ∈ RNtest1×. Define a matrix d ∈ RNtrain×Ntest, where d represents the distance between point DItrain(i) and point DItest(j) to find the most fitting path that passes through a number of grid points of this matrix grid, ensuring the final total distance between the two degradation curves is the shortest. Such a path can be expressed as follows:p represents the regularity degree of the two trajectories, pk=, 1 ≤ k ≤ Npath; 1 ≤ i ≤ Ntrain;  1 ≤ j ≤ Ntest and max ≤ Npath ≤ Ntrain+Ntest. The grid points through which the path passes are the points on which the two sequences are aligned for calculation. To ensure that each point of DItrain and DItest appears in the path p, the i and j in pk= must be monotonically increasing. The obtained regularized path is required to satisfy the shortest distance path rule:D is the cumulative distance of the two sequences from p1= to pk=, that is, the sum of the Euclidean distance d that represents the distance between the point DIlife(i) and point DItest(j), namely, the cumulative distance of the smallest neighboring element that can reach the point . The value of the last point in matrix D is the total cumulative distance of the two sequences and can be considered as the similarity of the two sequences.The notation Bearings with similar degradation trajectories should have a similar degradation pattern. The failure threshold of the reference training bearing with degradation trend matched with the given testing bearing can be used to determine the failure threshold of the given testing bearing.a is the development index,  bii=1,2,…, h is the grey actuating quantity, and h is termed as the order of time power terms bith−i.The traditional grey model is only suitable for fitting the time series of pure exponential change law, while the performance degradation of rolling bearings is affected by many complex factors, such as steady disturbance, constant speed disturbance, and acceleration disturbance, and its performance degradation trajectory will not strictly follow the pure exponential change law . FOTP-GMX(0), a 1-AGO sequence X(1) can be generated by the following:Based on the given degradation curve a and bii=1,2,…, h of the FOTP-GM  model can be estimated by the least-squares method, and the corresponding time response function can be obtained as follows:c is a constant that can be optimized to acquire the minimum error of simulation. The reconstructed sequence can be obtained as follows:Then the parameter sequence The structural parameters of the FOTP-GM  model can be adjusted adaptively with the dynamic changes of the actual time sequence, which can fit the homogeneous exponential sequence accurately and approximate the nonhomogeneous exponential sequence without error.Due to the nonlinearity and uncertainty of the bearing degradation process, it is difficult to predict the RUL of the given bearing accurately. The essential step to this problem is to establish a believable and reasonable mathematical model to characterize such a nonlinear degradation process. Based on the theories mentioned in Firstly, the time-frequency diagrams of the vibration data of the training and testing bearings are input into the CAE network to extract the deep hidden representations, and the degradation trajectories of the training and testing bearings can be generated from such deep hidden representations using the SOM network. Secondly, the trend similarity between the degradation trajectories of the training and testing bearings is evaluated using DTW, and the degradation trend matching results can be determined using the minimum value in DTW distances. Then, the degradation trajectory models of the training and testing bearings can be obtained by applying FOTO-GM  model using an order optimization method. Finally, the RUL prediction of the testing bearing can be realized using the preset failure thresholds and fitted degradation curves of testing bearings.Generating time-frequency diagrams of the given bearing data using equation  given inExtracting the deep hidden features of obtained time-frequency diagrams using the CAE method mentioned in t, DI(t), can be obtained using the minimum value of Euclidean distance between hidden feature Ht and the optimized weight vector W of the SOM network.Mapping the degradation indicator by inputting the normalized depth hidden features into the pretrained SOM network. The DI curve of the given bearing at time DI(t) sequence of the given bearing using the FOTP-GM  model with different time power terms orders, then using the method given in h.Fitting the An accurate bearing degradation model can reduce the nonlinearity and uncertainty of the bearing RUL prediction. Based on the theories mentioned in bith−i. A fitted degradation trajectory model with a suitable h can accurately reflect the process of bearing performance degradation. So an appropriate time power terms order is needed to be selected.Since the curve reconstructed by the FOTP-GM  model is sensitive to the order of time power terms DIfitted, can be described as follows:u, p, q, r, and s are the fitting parameters.The FOTP-GM  model is used to fit data points in the historical degradation trajectory of the given bearing and the corresponding fitted time series,e_train represents the root-mean-square value of fitting error between the origin degradation trajectory DItrain and the fitted degradation curve DItrain_fitted, h ∈ N∗ is the order of time power terms, and N is the length of the origin degradation trajectory DIlife. The smaller the value of e_train, the smaller the fitting error between the fitted curve and the original curve. To prevent overfitting, the h corresponding to the trend transition point in the e_train curve can be selected as the optimal order of time power terms for the given training bearing.For the training phase, the fitting performance of different fitting order of time power terms can be expressed as follows:DItrain_fitted and DItest_fitted are the fitted degradation curves of the reference training bearing and the given testing bearing, respectively. The h corresponding to the minimum point in the e_test curve and making the fitted curve monotonically increasing can be selected as the optimal time power terms order for the given testing bearing.For the testing phase, the optimal order of time power terms can be determined using the DTW distance between the fitted degradation curves of the given testing bearing and the corresponding reference training bearing with degradation trend matched:The discrepancy and similarity of degradation trajectories are considered to determine the failure threshold of the given testing bearing. The degradation trend of the training bearing, which has a similar degradation trend to the given testing bearing, is selected as the reference performance degradation pattern, and the DI value at the life endpoint is considered to set the testing bearing failure threshold. However, the degradation trajectories of training and testing bearings cannot be the same, so the DTW distance between these two curves, which indicates the similarity of the degradation trend, is considered to reduce the difference between these two degradation curves.DItrain and DItest_fitted represent, respectively, the original DI trajectory and the fitted degradation curve reconstructed by the FOTP-GM  model of the given reference training bearing, as shown in DItest and DItest_fitted denote, respectively, the original DI trajectory and the fitted degradation curve reconstructed by the FOTP-GM  model of the given testing bearing, as shown in DItrain_fitted(N) at the life endpoint is considered as the failure threshold of the given training bearing. The failure threshold of the given testing bearing with a similar degradation trend to the training bearing can be determined as follows:Let The RUL prediction method of rolling bearings based on FOTP-GM  can dynamically optimize the model structure according to the performance decline trend of rolling bearings to obtain the smallest simulation error of the original time sequence as much as possible, with higher prediction accuracy and stronger generalization ability.DItest(m), , to model and solve the parameters with equation  have different abilities to predict the bearing RUL. Using the method proposed in In the fitting error curves of the FOTP-GM  model for training bearings, there exists a trend transition point after which the fitting error is getting smaller fluctuation, as shown in To predict the bearing RUL accurately, a reasonable failure threshold for each testing bearing is needed. The similarity matching analysis is performed between the DI curves of testing bearings and the lifecycle degradation trajectories of the training bearings to classify the testing bearing degradation trends based on the degradation trends of training bearing, and the corresponding training bearing degradation model can be selected as the reference degradation model of the selected test bearing. Using the method mentioned in Based on Tables etest curves, the optimal fitted degradation curves, and the RUL prediction results of bearings 1_3, 2_7, and 3_3 are shown in Based on the fitted DI curves and the preset failure thresholds of the test bearings, FOTP-GM  model is adopted to predict the RUL of the test bearings using an optimal time power terms order selection method introduced in As shown in To further verify that whether the vibration amplitudes of bearings 1_5 and 1_6 have reached the preset failure threshold, the whole life vibration peak amplitude curves of these two bearings are presented in The performance of the RUL prediction results of testing bearings can be evaluated using three metrics: root-mean-square error (RMSE), symmetric mean absolute percentage error (SMAPE), and scoring function. The RMSE and SMAE are defined as follows:i∈ [Ei is the percent prediction error on testing bearing i, and Ai is the score of accuracy of RUL estimates for testing bearing i. The Score represents the overall accuracy of testing bearing RUL prediction, and higher value in Score, higher overall RUL prediction accuracy.The scoring function has been adopted by many researchers and IEEE PHM 2012 Prognostic Challenge . By conswhere i∈  states fTo verify the performance of the proposed method, several state-of-the-art RUL prediction methods are compared, including Sutrisno's vibration frequency signature anomaly detection and survival time ratio , Hong's The experimental lifecycle data sets of rolling bearings provided by the Institute of Design Science and Basic Component at Xi'an Jiaotong University  are analThe Morlet-based CWT method is applied to obtain the time-frequency diagrams from the vibration data sets of the six training bearings, as shown in Taking bearing 2_1 as an example, the deep hidden features of its time-frequency diagrams is extracted by the CAE model proposed in Case 1, and four typical feature curves changing with the operation time are shown in Figures DIlife and the fitted degradation curve DIfit. The fitted degradation curves with different order of time power terms of FOTP-GM  have different abilities to predict the bearing RUL. Using the method proposed in Similar to Case 1, there exists a trend transition point after which the fitting error is getting smaller fluctuation in the fitting error curves of the FOTP-GM  model for training bearings, as shown in To verify the predictive capability of the proposed method, the data sets of testing bearings are segmented with different prediction starting points to perform 10 subtasks, in which the division ratio includes 50%, 55%, 60%, 65%, 70%, 75%, 80%, 85%, 90%, and 95%, as shown in Taking subtask #6 with the division ratio of 75% as an example, the similarity matching analysis is performed between the historical DI trajectories of testing bearings and the lifecycle degradation trajectories of the training bearings to match the testing bearing degradation trends with the degradation trends of training bearing, and the corresponding training bearing degradation model can be selected as the reference degradation model of the selected test bearing. Using the method presented in Based on Tables The RUL of the given testing bearing can be estimated using the optimal time power terms order and the preset failure threshold. The RUL prediction results of subtask #6 are listed in The predicted whole lives of the testing bearings are shown in The results of RUL predictions are shown in The CAE-based feature extraction method can adaptively extract the deep features associated with the bearing performance degradation. The SOM-based DI construction method can effectively handle the deep features extracted by CAE into a practical degradation indicator.The FOTO-GM  model parameter optimization method can determine the optimal fitting order based on the fitting errors. The optimized FOTO-GM  method can construct the fitting degradation model of the given bearing based on the degradation information.The failure thresholds setting method based on DTW and optimized FOTO-GM  can adaptively adjust the failure threshold of the given bearing according to the accumulation of test bearing information without human intervention.This paper proposes a new method that involves the difference and the similarity between the degradation trajectories of training and testing bearing to predict the RUL of roller bearings. This method mainly focuses on two aspects of bearing RUL prediction: data-driven based accurate failure threshold setting and optimal mathematical degradation model construction using deep features. From the experimental results, we have the following conclusions:The proposed bearing RUL prediction model can adaptively update its parameters when new monitoring data are available. The comparison between the experimental results of this method and the existing methods shows that this method can effectively improve the prediction accuracy, reduce the uncertainty of prediction, and has better engineering practicability.