Reservoir computing (RC) is an attractive paradigm of a recurrent neural network (RNN) architecture, owning to the ease of training and existing neuromorphic implementation. Its simulated performance matches other digital algorithms on a series of benchmarking tasks, such as prediction tasks and classification tasks. In this article, we propose a novel RC structure based on the coupled MEMS resonators with the enhanced dynamic richness to optimize the performance of the RC system both on the system level and data set level. Moreover, we first put forward that the dynamic richness of RC comprises linear dynamic richness and nonlinear dynamic richness, which can be enhanced by adding delayed feedbacks and nonlinear nodes, respectively. In order to set forth this point, we compare three typical RC structures, a single-nonlinearity RC structure with single-feedback, a single-nonlinearity RC structure with double-feedbacks, and the couple-nonlinearity RC structure with double-feedbacks. Specifically, four different tasks are enumerated to verify the performance of the three RC structures, and the results show the enhanced dynamic richness by adding delayed feedbacks and nonlinear nodes. These results prove that coupled MEMS resonators offer an interesting platform to implement a complex computing paradigm leveraging their rich dynamical features. Artificial neural networks (ANN) have played an important role in the current boom in artificial intelligence (AI), especially with the invention of the Internet of Things (IoT) and ubiquitous sensing . ANN hasReservoir computing (RC) is a brain-inspired computational framework suited for temporal data processing, owing to its derivation from the recurrent neural network (RNN) ,7. The mIn this work, we present a couple-nonlinearity RC structure with double-feedbacks (CRSD) based on the coupled MEMS resonators with the enhanced dynamic richness to promote the performance of the RC. Moreover, we first put forward that the dynamic richness of RC comprises linear dynamic richness and nonlinear dynamic richness, which can be enhanced by adding delayed feedbacks and nonlinear nodes, respectively. In order to elaborate this point in detail, we compare three typical RC structures, a single-nonlinearity RC structure with a single-feedback (SRSS), a single-nonlinearity RC structure with double-feedbacks (SRSD), and the couple-nonlinearity RC structure with double-feedbacks (CRSD). In addition, we describe the evolution for the promotion of the performance of the three structures. Furthermore, four different tasks—the parity benchmark task, nonlinear autoregressive moving average (NARMA) task, isolated spoken digits recognition, and human action recognition—are investigated to prove the feasibility of these RC structures. The results show the enhanced dynamic richness by adding delayed feedbacks and nonlinear nodes.The rest of this paper is organized as follows. In A general single nonlinear node RC structure with a single-feedback (SRSS) can be conceptually decomposed into three layers, which are shown schematically in nt state . Input gThe dynamic reservoir layer is the key to determine the performance of the RC because of its ability to nonlinearly transform input data into high-dimensional space. In addition, the reservoir is fixed and only the readout is trained with a simple method such as linear regression and classification. Therefore the reservoir is sensitive to its optimizable architecture properties, and attention should be paid to the optimization for the reservoir layer rather than the optimization of the input and output layers. The dynamic richness describes the response of the reservoir, which provides memory and nonlinear computing capabilities of the system. Based on our analysis, the dynamic richness consists of linear richness and nonlinear richness. The linear richness represents the linear combination of the current virtual node state and various previous nodes information, which are determined by the delay feedback loop. The general SRSS has only one delay feedback loop with a delay feedback gain  of time . Thus, tThe nonlinear transforming of injected time-dependent data into a higher dimensional state is one of the most crucial properties of RC . This trIn this paper, we propose the coupled nonlinear mapping nodes to enhance the transforming property of CRSD, as illustrated in itude is :(4)kc=−∂After processed by mask signal and input gain, and modulated by the drive voltage, the input signal is injected into the drive electrode of the coupled resonators. The coupled resonators produce two different current output signals. The current signals are converted to voltage signals by the custom transimpedance amplifier (TIA) and then through the band pass filter to remove the noise. The amplitude modulation signals restore the amplitude information through the envelope (ENV) and then enter the long delay feedback loop and short feedback loop. Finally, two feedback signals are injected into the input signal to form a closed loop. The values of the virtual node from the short feedback loop are used as the processing data in the postprocessing algorithm.The memory of the previous input is very significant in the RC system. Insufficient memory can degrade performance, thereby depriving the reservoir of all prediction or processing ability . Linear In order to study the effect of the delay feedback loop on memory ability in more detail, we used memory capacity (MC) to make a quantitative analysis. Jaeger proposed a test for system memory capacity in 2002 . The inpObviously, SRSD with two delay feedback loops has better memory capacity than SRSS with only one delay feedback loop. That is to say, due to the addition of a delayed feedback loop, the RC system enhances the linear richness. Since the coupled resonators have two output signals, we do not add more (three or more) feedback loops in the simulation.As noted above, the reservoir has the ability to nonlinear transform the time-dependent input signal into a higher dimensional state space. The higher dimensional input signal greatly simplifies the difficulty of linear regression in postprocessing. Compared with SRSD, the CRSD uses the coupled nonlinear resonators to replace the single resonators, which improves the nonlinear richness of the RC system. The coupled resonators have two output signals from each resonator, and they connect a long delay feedback loop and a short delay feedback loop, which is the same as SRSD.Because the value of the virtual node directly determines the quality of the classification, analyzing the value of the virtual node can more intuitively show the nonlinear mapping ability of the RC system. In brief, CRSD has a richer node status than the other two systems. That is to say, the system enhances its nonlinear richness because of the coupled nonlinear resonators. Compared with the general structure SRSS, the CRSD increases the linear richness through another delay feedback loop and nonlinear richness through the coupled MEME resonators, which we call dynamic richness of the system.The performance of the system lies in the following two aspects, prediction ability and classification ability, which are assessed by four typical data sets to prove that the coupled nonlinear resonators improve the dynamic richness of the RC system. The prediction ability is numerically investigated via the parity benchmark prediction task and NARMA task, while the classification ability is discussed via the TI-46 isolated word task and human action recognition task.The prediction ability of the system is first assessed by means of the parity benchmark task. The parity function is considered as a benchmark, as it requires both memory ability and nonlinear computational capability . For thi predict . As n>1,orted in . The metorted in  and willThe result of this prediction task shows SRSD with another delay feedback loop can improve the success rate of the RC system by increasing its linear richness.The nonlinear autoregressive moving average (NARMA) prediction task is one of the most popular tasks in the RC community and it is also a more difficult task than the parity benchmark. In this prediction task, the RC system is trained to predict the behaviors of the system, for instance, a nonlinear autoregressive moving average (NARMA) of order The results of this complicated prediction task show CRSD with two delay feedback loops and the coupled resonators can decrease NMSE by increasing its linear richness and nonlinear richness.Through the isolated spoken word task, the classification performance of reservoir computing when inputting complex data is evaluated. This task is the classification of isolated audio sequences from a subset of the National Institute of Standards and Technology Texas Instrument-46 Corpus (NIST TI-46 Corpus) . The datIn this task, we use three RC systems  to test the performance, and the same parameters were set as In order to visualize how the time-dependent data separation occurs and understand the recognition capabilities of the different RC structures, the t-distributed stochastic neighbor embedding (t-SNE) technique  is used The results of this isolated word classification task show that CRSD can decrease the WER by increasing its dynamic richness.Recognizing human actions in video streams is a difficult task in computer vision. This task was first applied to RC systems in 2019  as a staIn this recognition task, we also use three RC systems  to test the performance, and the same parameters were set as N = 3000 in this task.In addition, we test the effect of the number of virtual nodes N=16,384 . In our We propose the coupled MEMS resonators can optimize the performance of the RC system based on the enhanced dynamic richness that is composed of linear richness and nonlinear richness. In order to set forth this point, we compare three typical RC structures with a single-nonlinearity RC structure with single-feedback, a single-nonlinearity RC structure with double-feedbacks that is treated as a transition structure, and the couple-nonlinearity RC structure with double-feedbacks. We also describe the evolution for the promotion of the performance of the three structures. At the system level, we use the delay characteristic diagram and memory capacity to illustrate that the delayed feedback loop increases the linear richness, and use the output signal timing diagram and the histogram of the virtual nodes to illustrate that the coupled resonator increases the nonlinear richness of the system. At the data set level, four different tasks, the parity benchmark task, nonlinear autoregressive moving average (NARMA) task, isolated spoken digits recognition, and human action recognition, are investigated to prove that CRSD can improve the accuracy of the system to a certain extent. These results have significant implications with respect to hardware implementation of the reservoir and open new future possibilities for exploring real-world applications of coupling resonators in the RC system.


To our knowledge, MTBAN is the first variant effect prediction tool based on a deep generative model that provides a user-friendly web server for the prediction of deleteriousness of variants.The development of an accurate and reliable variant effect prediction tool is important for research in human genetic diseases. A large number of predictors have been developed towards this goal, yet many of these predictors suffer from the problem of data circularity. Here we present MTBAN , a method for predicting the deleteriousness of variants. We apply a form of knowledge distillation technique known as the Born-Again Networks (BAN) to a previously developed deep autoregressive generative model, mutationTCN, to achieve an improved performance in variant effect prediction. As the model is fully unsupervised and trained only on the evolutionarily related sequences of a protein, it does not suffer from the problem of data circularity which is common across supervised predictors. When evaluated on a test dataset consisting of deleterious and benign human protein variants, MTBAN shows an outstanding predictive ability compared to other well-known variant effect predictors. We also offer a user-friendly web server to predict variant effects using MTBAN, freely accessible at  The type II circularity occurs when all variants in a given gene are labelled either all deleterious or all benign, which results in the model predicting the same label for all variants in that gene. Previous studies3 have suggested that this problem of data circularity can result in an inflation of the reported performances of many supervised predictors. On the other hand, unsupervised methods do not learn from labelled variant data and learn solely from the evolutionary information contained in multiple sequence alignments. A recent study which carried out an extensive comparison of variant effect predictors claimed that a class of unsupervised models, namely the deep generative model, is a promising area of research for variant effect prediction3.While recent sequencing technologies have resulted in a tremendous amount of sequence variant data, the identification of deleterious variants is still a difficult problem. Development of a reliable computational tool to predict the effects of sequence variants would aid in the treatment of many human genetic diseases. To achieve this goal, many predictors have been developed based on different approaches. Among these methods, supervised methods learn from labelled variant data consisting of known deleterious and benign variants, and many of them show good predictive ability. However, many supervised methods face the problem of data circularity, which can be divided into two types according to Grimm et al4 based on a deep autoregressive generative model, and showed that it demonstrates state-of-the-art performances on the prediction of functional effects of variants. In this work, we apply a knowledge distillation technique called the Born-Again Networks (BAN)5 to the mutationTCN model and develop an improved model called MTBAN. In machine learning, knowledge distillation is a process involving the transfer of knowledge learned from one machine learning model to another. Using the Born-Again Networks allows the student network to achieve an improved predictive power compared to the teacher network. When evaluated on human variant datasets with deleterious and benign variants, MTBAN shows superior predictive performances compared to other variant effect predictors. Our model is fully unsupervised and is not dependent on labelled data for training. This gives the model advantage over supervised predictors, for which data circularity is an inherent problem. We also offer a freely accessible web server for using MTBAN for variant effect prediction.Here, we introduce MTBAN , an enhanced method to predict the deleteriousness of single amino acid variants. We previously developed a method called mutationTCN4. For each protein variant, the model is trained on the multiple sequence alignment of the corresponding protein. As it is a generative model, it is trained by maximizing the likelihood of the training data, which is equivalent to minimizing the negative log likelihood between the input sequence and the predicted output. After training, the model can predict the probability of observing a given protein sequence under the parameters of the trained model. The deep autoregressive generative model is implemented using the temporal convolutional network architecture6. Each sequence from the input multiple sequence alignment is encoded by representing each amino acid in the sequence as a distinct integer. The input is passed through an embedding layer, followed by a series of dilated causal convolution layers, an attention layer, and a fully connected layer label loss, which refers to the cross entropy loss between the input sequence and the softmax output distribution of the teacher network. In the next step, only the student network is trained, with the loss being the sum of the label loss and the teacher loss. Here, the label loss refers to the cross entropy loss between the input sequence and the softmax output of the student network. The teacher loss refers to the cross entropy loss between the softmax output of the student network and the softmax output of the teacher network. The softmax output distribution The model structure of MTBAN is shown in Fig. 7. Using higher temperatures leads to more “softened” output distributions. According to Hinton et al.7, these softened output distributions contain the “dark knowledge,” which is the hidden knowledge learned by the teacher network. In BAN, the transfer of this “dark knowledge” from the teacher to the student contributes to the improved performance of the student network. In our implementation, we used a temperature of 4. We trained both teacher and student networks for 500,000 iterations using the mini-batches with the size of 128. For both teacher and student networks, the learning rate is set to 0.001 when the number of training iterations is smaller than 3000, and 0.0001 when it is greater than 3000.where .8 These pre-computed predictions on the Hopf dataset were used for evaluating the model on the test set. According to Hopf et al., their alignment generation protocol involves multiple iterations of profile HMM homology search in an attempt to ensure that there are enough sequences in the alignment and that the alignment coverage of the target protein sequence domain is sufficient8. This allows us to obtain an alignment that contains as much evolutionary information as possible.We computed the predictions of MTBAN for a total of 1605 human protein alignments provided by Hopf et al4. The smaller the score, the more likely the variant has a deleterious effect. Second, the z-score is computed by normalizing the distribution of log probability scores for all possible missense variants against the target protein sequence. This normalization process is done due to the variations in the score distributions across different proteins. Third, the probability of deleteriousness for each variant, ranging from 0 to 1, is computed. This is determined from the set of variants in the Humsavar database (release 03/2020)9 which overlap with our pre-computed model predictions for the Hopf dataset, which are 1221 deleterious and 1221 benign variants. We obtained the z-score distribution for this set of variants, divided the distribution into equal-length z-score intervals, and calculated the proportion of deleterious variants in each z-score interval. Finally, using the same z-score intervals, we determined a z-score cutoff which maximizes the classification accuracy , which aims to perform an unbiased assessment of variant effect predictors, uses data from deep mutational scanning experiments as part of their benchmark dataset10. Therefore, we use the functionally determined variant data from Mahmood et al. in addition to the disease-related variant data for comparing MTBAN with other predictors.To evaluate the ability of the model to classify human protein variants as deleterious or benign, we created a test dataset by combining the variants from the datasets used by Grimm et al.11, PolyPhen-212, MutationAssessor13, fathmm-MKL14, MPC15, GenoCanyon16, phastCons17, DANN18, GERP++19, and phyloP20. The predictions of the commonly used predictors on the test dataset were obtained from dbNSFP21 via the Ensembl variant effect predictor22. Since the score cutoffs for phyloP, DANN, phastCons, GERP++, MPC, and GenoCanyon were not provided by dbNSFP, we computed the cutoffs for each predictor using the Humsavar database (release 03/2021) as described in “We compared the performance of our model with mutationTCN and other commonly used variant effect predictors, SIFT. and Mahmood et al. for which MTBAN predictions exist in the pre-computed Hopf dataset, and used those variants for comparison with other methods. Since the number of deleterious variants was significantly larger than that of benign variants, we randomly selected variants from the deleterious variant data to match the data size of the deleterious variants and the benign variants. This resulted in a balanced test set consisting of 1709 deleterious and 1709 benign variants in total.We found variants among the datasets from Grimm et alThe following metrics were used for evaluating the classification ability of the variant effect predictors: ROC-AUC (Receiver Operating Characteristic Area Under Curve), PR-AUC , accuracy, Matthews Correlation Coefficient (MCC), precision, specificity, sensitivity, F-score, and Negative Predictive Value (NPV). For MTBAN, ROC-AUC and PR-AUC were calculated using z-scores, and other evaluation metrics were calculated using the predicted label. The following equations were used for computing the evaluation metrics:.1 and functionally determined variants from Mahmood et al.2, resulting in a total of 1709 deleterious and 1709 benign variants. When compared with 11 other variant effect predictors in terms of ROC-AUC and PR-AUC, our model outperformed all other predictors, achieving a ROC-AUC of 0.883 and a PR-AUC of 0.878  further improves the predictive power of the model, by transferring the knowledge learned by the model to the second model of the same capacity. We conducted an assessment using the test set combining the disease-related variants from Grimm et al.1 and the functionally determined variants from Mahmood et al.2, and further assessment using each of the two variant sets. In all cases, MTBAN consistently shows outstanding predictive ability compared to other prediction tools. The results indicate that MTBAN is a reliable method for predicting the deleteriousness of human protein variants.Here, we have introduced MTBAN, an improved method for predicting the deleteriousness of single amino acid variants. As demonstrated in our previous work3 have pointed out concerns regarding the problem of data circularity in many supervised predictors, which can lead to an inflation of the reported performances of these tools. Due to the fully unsupervised nature of MTBAN, it is not hindered by the problem of data circularity and can be considered to have higher generality compared to supervised models. Moreover, while we only considered human protein variants in this work, it is possible to predict the effects of protein variants in any other species if a multiple sequence alignment is available.Previous worksAs previously mentioned, the BAN involves the transfer of the “dark knowledge” hidden in the softened output distribution of the teacher network to the student network. We speculate that due to the large size and the high complexity of the training set used in this study, the student equipped with the teacher’s knowledge can better model the distribution of the training data, compared to the teacher alone. In other scenarios where the model is of high capacity and the training data is limited in size, the student network may possibly perform worse due to overfitting.One potential limitation of MTBAN and mutationTCN is that they can only make predictions for variants which correspond to the conserved positions in the multiple sequence alignment of a protein. However, when we analyzed all of the 9935 human protein multiple sequence alignments in the Hopf dataset, approximately 88% of the target sequences were conserved, which is a considerably large proportion. Another potential limitation of MTBAN is that the training time is longer compared to mutationTCN alone for prediction. Although MTBAN takes a longer time to train, it shows a higher predictive performance compared to the previous model.The results of our work show that the deep generative model is a powerful tool for predicting the effects of sequence variations. We expect that deep generative models will continue to play an important role in discovering the effects of genetic variants. In addition, to our knowledge, MTBAN is the first variant effect prediction tool based on a deep generative model that provides a user-friendly web server for the prediction of deleteriousness of variants. This method is expected to be a useful tool for the prioritization and identification of variants involved in human genetic diseases.Supplementary Information.


Infrared sensing technology is more and more widely used in the construction of power Internet of Things. However, due to cost constraints, it is difficult to achieve the large-scale installation of high-precision infrared sensors. Therefore, we propose a blind super-resolution method for infrared images of power equipment to improve the imaging quality of low-cost infrared sensors. If the blur kernel estimation and non-blind super-resolution are performed at the same time, it is easy to produce sub-optimal results, so we chose to divide the blind super-resolution into two parts. First, we propose a blur kernel estimation method based on compressed sensing theory, which accurately estimates the blur kernel through low-resolution images. After estimating the blur kernel, we propose an adaptive regularization non-blind super-resolution method to achieve the high-quality reconstruction of high-resolution infrared images. According to the final experimental demonstration, the blind super-resolution method we proposed can effectively reconstruct low-resolution infrared images of power equipment. The reconstructed image has richer details and better visual effects, which can provide better conditions for the infrared diagnosis of the power system. The concept of the Internet of Things is proposed, and the era of the Internet of Everything is coming. One of the keys to building the power Internet of Things is to realize online monitoring and analysis and evaluation of the operating status of various power equipment. Among various monitoring technologies, infrared monitoring technology has the characteristics of being long distance and non-contact and having high levels of accuracy and speed ,2. The eSuper-resolution (SR) aims to reconstruct a high-quality image rement Y . SR is aThe SISR method can be divided into three categories according to the principle. The first type is interpolation methods, which are often simple and easy to implement, but usually excessively smooth the high-frequency details of the image, resulting in poor visual quality of the reconstructed image ,5,6. TheIn addition, the SISR method can also be divided into three categories according to the different problems it solves. The first type of method ignores the general blurring in the LR image formation process. These methods consider the LR image to be absolutely clear and only improve its resolution ,8,11,12.In order to improve the quality of SR reconstructed images, to meet the actual needs of the power industry, we propose a blind SR method. Since the joint restoration of the blur kernel and HR image may produce sub-optimal reconstruction results, we chose to estimate the blur kernel first, and then reconstruct the LR infrared image through the non-blind SR method. For the blur kernel estimation, we improved the basic SR model of compressed sensing and introduced the image Extreme Channels Prior to the model; thus, we propose an LR image blur kernel estimation method based on the compressed sensing theory. For the non-blind reconstruction after blur kernel estimation, we propose an adaptive non-blind SR reconstruction algorithm. The algorithm uses adaptive control of the intensity coefficient of the regular term in the reconstruction process to suppress the generation of artifact ringing and improve the quality of the reconstructed image. The final experimental results show that our proposed blind SR reconstruction method for infrared images of power equipment can effectively reconstruct LR infrared images through successive blur kernel estimation and non-blind reconstruction. The reconstructed image has richer details and better visual effects, which can provide better conditions for the infrared diagnosis of the power system.Our blur kernel estimation model is improved from the basic SR model of compressed sensing. So, we first briefly introduce it, and its model is:. Donoho  pointed The most sparse Obviously, due to the undetermined nature of the problem, the estimated value of ioned in  as a conIn this section, we introduce the image Extreme Channels Prior. There are many types of prior information that can be used to estimate the image blur kernel, such as L0-regularized gradient prior, dark channel prior and Extreme Channels Prior. The reasons why we chose Extreme Channels Prior are as follows: In the infrared image of power equipment, the faulty heating part and the edge texture part of the equipment contain the most important information. The faulty heating part is the focus of infrared diagnosis, and it often appears as the brightest part in the image. The edge texture part is the main basis for image segmentation and target recognition, which is often the darkest part. The Extreme Channels Prior used in this article is calculated based on the local brightest and darkest pixel values in the infrared image before and after blurring. Therefore, Extreme Channels Prior pays more attention to the processing of local brightness extremes in the image and can use this as the focus to estimate the blur kernel, thereby improving the visual quality of the faulty heating part and the edge texture part of the final reconstructed image. This feature can make the infrared image of the power equipment reconstructed by the method more suitable for the needs of practical applications.It should be noted that this paper does not directly process the infrared image in RGB mode, but converts it into YCbCr mode. Y is the luminance component, Cb is the blue chrominance component, and Cr is the red chrominance component. Since the human eye is more sensitive to the Y component, the visual difference caused by the subtle changes of the other two components is extremely small, so the following image Extreme Channels Prior statistical analysis will take the Y component as an example. Specifically, Extreme Channels includes a bright channel component and a dark channel component, which are, respectively, arranged by the local maximum and minimum values in the Y component according to a certain rule. For image terature ,26. FiguIn this section, we will propose a blur kernel estimation model based on maximum a posterior probability (MAP) framework and the basic SR method of compressed sensing. The MAP framework is:Combined with the SR model of compressed sensing, the objective function of blur kernel estimation can be constructed:After completing the estimation of Equation (11) does not contain any prior information except for the sparse constraint. It is difficult to achieve the effect of deconvolution and deblurring due to the ill-posedness of the problem using the image reconstructed directly from Equation (11). Therefore, it is necessary to introduce other prior information as constraints according to the law of image statistics.Currently, the best deblurring method generally uses a set of filter output edge statistics to match the edge statistics of the clear image as a priori constraint for the deblurring problem. The Gaussian distribution, Laplacian distribution and Hyper-Laplacian distribution are commonly used to fit the image gradient. Assuming that the edge distribution is Gaussian, the deblurring problem has an analytical solution in the frequency domain, and the image can be restored efficiently through fast Fourier transform. However, a clear infrared image usually has a non-Gaussian edge, as shown in Therefore, it can be seen from the above discussion that a prior constraint of image edge distribution needs to be introduced for Equation (11). No matter what kind of distribution is used to fit the image edge, it can be written as The non-blind super-resolution objective function is determined in It can be seen from The adaptive regular term intensity adjustment method we adopt is to use different priors as constraints to reconstruct the image twice according to the model of Equation (13). In the first reconstruction, the model uses a Gaussian prior which is easy to solve, extracts the significant edges of the reconstructed image, and generates the label image. The secondary reconstruction adopts the Hyper-Laplacian prior as the constraint, and adaptively adjusts the regularization intensity of different pixels according to the label image. The specific steps in Step 1: Let Step 2: Use filter bank Step 3: Perform threshold shrinkage on edge images; the shrinking method is:Step 4: Solve Step 5: Set the elements smaller than Step 6: Perform mathematical morphological processing on the binarized image. The opening and closing operations are performed once to remove the binarized image noise, and the final label image After Let In Equations (16) and (17) both contain variables Equation (18) is a typical least squares problem, which can be solved by the gradient descent method. The derivative of Equation (18) with respect to The number of iterations and step length are determined by the one-step steepest descent scheme introduced in . We solvIn addition to the common variables, Equations (16) and (17) also contain their own unique variables, and their solutions are introduced separately below. The variables Their solutions are:The variables In order to maintain the consistency between The function of the The solution of (30) can be obtained by FFT:We estimate the blur kernel For the subproblem accurate ; therefoThe solution of (33) can be obtained by FFT:Algorithm 1: Blur Kernel Estimation Algorithmgenerate the initial value of each variable repeat solve for repeatsolve for repeat solve for repeat solve for until until  until  solve for  end for Output: blur kernelInput: Blurred imageSince the blur kernel ested by ,26,29, wFinally, only the solution of The purpose of initial reconstruction using Gaussian prior is only to extract the significant edges of the image. Therefore, WhenEquation (38) can be divided into the The solution is the same as (35). After obtaining We set the derivative of Equation (41) to be 0:We further transform Equation (42) into:The root of Equation (43) is r. According to , when r er speed .We, respectively, give the solution of each unknown variable in the objective function of blur kernel estimation and non-blind SR. When Our test environment parameters were as follows: Intel(R) Core(TM)i5-9300H CPU @2.40 GHz; memory: 16.00 GB; operating system: Windows 10; MATLAB R2019a. We obtained the following fixed parameters through repeated experiments and adjustments: In order to make the experimental results more convincing, in addition to comparing our method with the classical blind SR method, we also design comparative experiments for the blur kernel estimation and non-blind SR reconstruction in our method. In the blind SR comparison experiment, we compared our method with the methods proposed by Keys , Shao 2, MichaelEntropy represents the uniformity of a system in physics. The more uniformly a system is distributed, the greater its information entropy is. The concept of image information entropy is derived from this, which can be defined as follows:In addition, in order to prove the effectiveness of the blur kernel estimation method in this paper, we compared the blur kernels estimated by our method and the algorithms proposed in ,23,32. WFinally, in order to verify the performance of the non-blind SR reconstruction method in this paper, we use the known blur kernel to process the HR and clear infrared image according to Equation (1). Taking the synthetic infrared image and the known blur kernel as input, we compare our method with the existing non-blind SR methods. We select the methods proposed by Keys , GlasnerFirst, we reconstruct the LR infrared images of the power equipment that are actually collected to verify the effectiveness of the blind SR method in the practical application. For the experiment, we used eleven infrared images taken on site with a resolution of 128 × 128 for SR reconstruction. As shown in In order to prove the effectiveness of the blur kernel estimation method in this paper, we used the six blur kernels shown in In this section, we use the six blur kernels in It can be seen from In this paper, the blur kernel estimation model involves many parameters. In this section, we analyze the influence of its value. The blur kernel estimation model involves five main parameters In order to reflect the superiority of this algorithm in practical applications, this section selects the advanced methods of deep learning algorithms  and our In order to improve the quality of SR reconstructed images, so as to facilitate the status monitoring and fault analysis of power equipment, we propose a blind SR method of compressed sensing. Our method is divided into two parts: blur kernel estimation and non-blind SR reconstruction. For the blur kernel estimation part, we improved the basic SR model of compressed sensing and get the basic blur kernel estimation model. In order to improve the estimation accuracy of the blur kernel, we introduce Extreme Channels Prior based on the color characteristics of the infrared image. For the non-blind SR reconstruction method, we propose an adaptive non-blind SR reconstruction algorithm. It controls the intensity coefficient of the regular term adaptively during the reconstruction process to suppress the generation of artifact ringing and improve the quality of the reconstructed image. The above blur kernel estimation method and the non-blind SR method are combined to form our blind SR method. In the experimental part, we compare the two parts of the blind SR method with their corresponding existing classical methods to illustrate the superiority of the performance of our method. According to the experimental results, it can be seen that our method can estimate the blur kernel more accurately, which can complete the non-blind SR reconstruction of LR infrared images with higher quality. The HR infrared image reconstructed by our method has more detailed textures and better visual effects, which can provide better conditions for the application of power system infrared diagnosis. Under the current background of the power industry actively carrying out the construction of the IoT, our method provides a feasible way to reduce the hardware cost of its construction. We think it enjoys broad application prospects to use the mode of front-end using low-cost sensors to collect information and back-end using algorithms to recover the high-quality collected images. This method can effectively reduce the construction cost of IoT and the cost of data transmission and storage. Because the construction of power IoT is still in its infancy, we do not choose to use the data-driven method for infrared image super-resolution reconstruction. When the data acquisition and storage system become more standardized and mature in the future, we think that it will also be an interesting idea to train dictionaries according to the types of power equipment and update them online to achieve sparse representation of different images, which may be able to achieve a better reconstruction effect on the premise of ensuring the accuracy of image information.


This study proposes a fully automated gearbox fault diagnosis approach that does not require knowledge about the specific gearbox construction and its load. The proposed approach is based on evaluating an adaptive filter’s prediction error. The obtained prediction error’s standard deviation is further processed with a support-vector machine to classify the gearbox’s condition. The proposed method was cross-validated on a public dataset, segmented into 1760 test samples, against two other reference methods. The accuracy achieved by the proposed method was better than the accuracies of the reference methods. The accuracy of the proposed method was on average 9% higher compared to both reference methods for different support vector settings. The gearbox fault detection problem has become an intensively studied topic in the last few decades. Early detection of possible gearbox faults  can increase the operational safety of a device, as it can reduce the costs of maintenance and prevent total failure. To detect a possible fault of a gearbox, the vibrations of the gearbox are typically measured, often by using multiple sensors at once. The damaged teeth in the gearbox produce force impulses that are usually reflected in the vibration signal. However, evaluating of the vibration signal is difficult. We can view the underlying process of generating the vibration signal as strongly non-linear, non-stationary, and non-Gaussian. The faults produce a signal with energy distributed over various frequencies, which makes successful detection even more difficult. Another issue that arises with the analysis of gearbox vibration signals is that every single gearbox produces a unique signal, so the approach and settings that are successful with some gearboxes may completely fail for different ones. That is probably the reason many other published methods were not validated and evaluated with appropriate datasets. Due to the popularity of this topic, many methods utilizing different approaches have been developed.As was already mentioned, the signal of a faulty gearbox or rotating machinery has a different spectrum than the faultless one. The fast Fourier transform  (FFT) isThere are approaches that are focused on residual signal analysis, such as the utilization of the auto-regressive model  or an auIn some studies, only empirical mode decomposition (EMD)  without O(nlogn) .Various methods based on adaptive filtering have been developed over decades. In , the autA unique approach based on the estimation of the cointegration factor of a vibration signal to detect the fault of a gearbox was presented in article . QuantitRecently, multiple methods using deep learning have emerged. In  the authMore information about gearbox fault detection approaches can be found in the following review papers: ,54,55.Most of the studies mentioned above were mainly qualitative, and some of their methods require expert opinions to conclude on the gearbox’s condition. However, in our study, we focused on a simple, robust, and fully automated solution of gearbox fault detection without prior knowledge of the operation or measurement details, and without a need for a large training dataset. Those factors make it different to many of the other deep learning-based methods. The proposed method features a multiscale approach and can utilize a custom number of parallel sensors attached to the gearbox. The evaluation criteria were assessed using measurement data, and the proposed approach was cross-validated.Our proposed method is based on utilizing the error prediction of an adaptive filter. The description of the adaptive filter is presented in A publicly available dataset already presented in the paper  was usedThe original whole data According to our results, every sensor provides a different amount of information about the health status of the gearbox. Furthermore, the information from the independent sensors is not fully correlated, and thus the sensors can be used in a complementary fashion.i-th adaptive weight, The scheme that depicts the adaptive filtration problem is in given byy^(k)=∑i=lgorithm  with eveZ-transform notation).The block schema of the filter is depicted in The proposed method is based on the error evaluation of a normalized-last-mean-square (NLMS) adaptive filter  in predin directly represents the number of historical values used for prediction. In this study, the filter size was set to The NLMS algorithm is a modification of the LMS (stochastic gradient descent). The LMS weight adaptation is given as follows.To detect a gearbox fault, we propose a descriptor based on the prediction error of the adaptive filter adapted via NLMS algorithm. This data segment E stands for the expected value. This descriptor represents the actual effort of the adaptive filter made to follow the target signal. This effort can be understood as a novelty in the data, or irregularity in the measured signal. Relation between irregularity in the data and the health condition of the gearbox is the main idea behind the proposed method. Note that in our study we processed only nonoverlapping segments.The resulting descriptor for the whole segment of filter error epj  is evaluTo present a suitable challenge for the proposed method, two other reference methods were used to evaluate the health status of the gearbox.The first reference method is based on evaluation of the raw measured data standard deviation as the direct feature for the gearbox health assessment:1.Create upper and lower envelope—connect local minima/maxima by a cubic spline line.2.Get the first component   is used j) is determined via the stoppage criterion. There are four known stoppage criteria for the sifting process: the standard deviation criterion; the number criterion; the threshold method; energy difference tracking. The implementation of EMD used in this study utilizes the standard deviation criterion. The sifting process stops when the standard deviation is smaller than a pre-supplied value. The standard deviation of the sifting step is determined as [The number of sifting steps (mined as :(22)SDi=In this work, the EMD was used to filter out the slower frequencies from the data to enhance the standard deviation change present in the original data. Then, the standard deviation was evaluated utilizing the first intrinsic mode function (IMF):k points was formed asAll studied methods (proposed and reference ones) extract a single scalar (standard deviation) as the descriptor of the whole sensor measurement. However, the measured data contain four different sensor measurements. Every sensor contains slightly different information. To make a prediction on gearbox health state from those 1–4 available descriptors, the SVM  is used.Note that entation .Exhaustive 10-fold cross-validation was used to obtain the accuracies of the proposed and reference methods. Before splitting the data into ten same size chunks, the segments were shuffled to balance the healthy vs. broken segment ratio in every chunk. In every iteration of cross-validation, one chunk was used for testing and nine chunks were used for training. The average accuracy was determined over all testing chunks. The scheme of the cross-validation process is depicted in The core of the proposed method is the approach of extracting features from the raw data. The extracted features can be evaluated with any classifier. In this study, the well-known support vector machine (SVM)  was usedIn this study, we compare the proposed method with two reference methods. The data shapes and organization of the methods in the testing framework are displayed in The full cross-validation results are shown in Furthermore, a method’s consistency, shown in Moreover, note that a higher number of sensors increases the accuracy.A novel and time-effective method of gearbox fault detection was proposed in this paper. The proposed method is superior in comparison to the reference methods: raw data statistics and frequency analysis-based feature extraction. Furthermore, the proposed method does not need any information about the measurement setup or the gearbox. The fault detection is based purely on machine learning. The classification criteria are determined according to the data. In other words, the proposed approach can adapt the gearbox diagnosis criteria according to the particular gearbox without human expert input or supervision.The ultimate goal of this paper was to show that the adaptive filtration error can enhance the information in raw data (the first reference method) in a greater way than the other often-used method of feature extraction—EMD (the second reference method). Therefore, we used a straightforward, noncomplex approach of standard deviation estimation and SVM classification for simple and robust evaluation of the methods. More complex evaluation methods can lead to misinterpreting the feature extraction methods.The possible utilization of a more advanced classifier  would be a suitable future direction for the research of the proposed feature extraction approach. Another attractive future study might be on applicability research regarding different fault detection tasks, including rotating machinery.


The quantum phases of the electron paths driven by an ultrafast laser in high harmonic generation in an atomic gas depends linearly on the instantaneous cycle-averaged laser intensity. Using high laser intensities, a complete single ionisation of the atomic gas may occur before the laser pulse peak. Therefore, high harmonic generation could be localised only in a temporal window at the leading edge of laser pulse envelope. Varying the laser frequency chirp of an intense ultrafast laser pulse, the centre, and the width of the temporal window, that the high harmonic generation phenomenon occurs, could be controlled with high accuracy. This way, both the duration and the phase of the electron trajectories, that generate efficiently high harmonics, is fully controlled. A method of spectral control and selection of the high harmonic extreme ultraviolet light from distinct quantum paths is experimentally demonstrated. Furthermore, a phenomenological numerical model enlightens the physical processes that take place. This novel approach of the electron quantum path selection via laser chirp is a simple and versatile way of controlling the time-spectral characteristics of the coherent extreme ultraviolet light with applications in the fields of attosecond pulses and soft x-ray nano-imaging. Laser pulse peak intensities of the orders of 3, tunnelling ionisation4, over the barrier ionisation6, as well as high harmonic generation (HHG)7 allowing for coherent spectral sources in the XUV region. In such intense EM fields, the interaction between the laser field and the atom is governed primarily by the laser pulse characteristics  as well as the depth of the atomic Coulombic well. Consequently, semiclassical theories have been developed to describe such interactions, the most renowned being the three-step model for the HHG process8.The rapid evolution in science and technology of femtosecond laser pulses during the last three decades has led to the development of intense and coherent electromagnetic fields (EM)In the three-step model picture, the electrons are initially ionised via tunnelling or even over the barrier ionisation. Then, they are driven by the strong alternating laser electric field back and forth the atomic nuclei. There is a non-zero probability for the oscillating electron to recombine with the parent nucleus emitting the excess energy as high frequency radiation. The majority of the electrons, however, are emitted into the continuum resulting in a partially ionised medium. The above recollision process is repeated every half-cycle of the laser electric field as long as the medium is not fully ionised. This periodicity is manifested in the frequency spectrum of the emitted photons as a comb of odd order harmonics. Specifically, as the harmonic order increases, there is initially a region in the frequency spectrum where the harmonics signal exhibits a decrease (lower order harmonics), followed by an extended region of harmonics with approximately the same efficiency (plateau), and finally a region where the harmonics signal rapidly decreases (cut-off).short and long trajectories, respectively. Specifically, the short trajectory duration is shorter than 0.65T, where T is the laser pulse period, while the long trajectory duration is longer than 0.65T, as shown in the inset (b) of Fig. 7 depends on the time spent in the continuum while it also depends linearly on the intensity of the driving laser pulse9. The total quantum phase of the qth-order harmonic is I(t) is the cycle-averaged laser pulse time-dependent intensity, while the parameters qth-order harmonic signal results from the coherent summation of the aforementioned quantum paths and corresponds to an energy of There is an infinite number of quantum paths that the recolliding electron can follow. However, for the same electron return energy, there are only two surviving paths that result in a recollision with the parent ion during the first laser cycle. For these two paths, both ionisation and recombination time instants are different. One of the paths lasts shorter time than the other and the paths are consequently termed 12. However, a significant obstacle in generalising these attempts was that the cut-off harmonics inherently suffer from low efficiency.The multi-cycle temporal characteristics of the plateau harmonics depend strongly on the phase difference between the long and short trajectories which is time-dependent, since the temporal intensity varies for every laser field cycle. Moreover, for the plateau harmonics, the time duration of the short and long quantum paths is markedly different, thus 14 and coherent diffraction imaging18, where the geometrical and phase characteristics of the harmonics beam play a major role. The main distinctive footprint of the two trajectories in each plateau harmonic is the difference in their divergence. The long trajectory has a larger divergence than the short, due to the fact that the electron travels longer distances in the partially ionised medium during the recollision process19. In addition, for laser intensities where the ionisation of the medium is relatively high, the central wavelength of the two trajectories is also slightly different. This phenomenon is attributed to the different time dependence of the two trajectory phases, which leads to coherent summation for each trajectory at different wavelengths (double-peaking phenomenon)20.It becomes evident from the above discussion that controlling the relevant contribution of long and short trajectories of the plateau harmonics, is of fundamental significance, not only for the detailed understanding of these phenomena, but also for applications such as attosecond pulses generation23, while attempts to separate and selectively control the two trajectories have also been reported. Bellini et al. were the first to experimentally demonstrate the different divergence distribution of the two trajectories using interferometric methods24. Their results were soon adopted by the attoscience community to filter the long trajectory and achieve high quality attosecond pulse trains14. Willner et al. used a multi-jet arrangement of two gases showing a clear spectral separation of the two trajectories26. Ishii et al. used a phase-locked two-colour  19: (i) 19. In addition, the high harmonic phase offset 19. Finally, it is a common observation in our HHG spectra, for the conditions where the high harmonics generated by the short and long trajectories are spectrally separated, that short trajectory high harmonic exhibit a higher signal than the long trajectory high harmonic. We experimentally estimate an overall acceptable ratio of Using Eq. , presentIn Fig. Additionally, in Fig. Our experimental results and theoretical treatment clearly show that the frequency chirp of the ultrafast laser pulses is a crucial parameter for the control of the electron quantum paths in the HHG from atomic gases. As a consequence, this control affects not only the HHG spectra, by separating the footprint of the short and long trajectories in the plateau harmonics, but also significantly affects their relative generation efficiency. We clearly demonstrate in this work that for certain experimental conditions, easily achieved with nowadays HHG installations, the HHG efficiency of the long trajectory is dramatically reduced compared to that of the short trajectory, simply by adjusting the frequency chirp of the driving laser pulse to an appropriate value. This is attributed to the interplay of the inherent quantum phases for each quantum path, controlled by the fine adjustment of the HHG temporal window within the laser pulse envelope, the slope of the temporal intensity at this window, and the temporal frequency of the laser pulse. The above conditions are met for driving laser intensities strong enough to cause a significant degree of ionisation in the generating medium, that defines the temporal window for HHG. It should be noted here that for the laser pulse characteristics and varying the Ar gas pressure, the suppression of the HHG from the long trajectory routinely appears for small positive chirp values and corresponding pulse durations in the range of +30 to +40 fs. This is fully supported by our spectral model calculations since small variations in the values of the physical parameters of the model do not appreciably alter the findings.29, the simplicity of our experimental method, that is based upon the appropriate selection of (a) the intensity of a single driving laser field, (b) the atomic target, and (c) the laser frequency chirp, easily controlled by detuning the ultrafast laser compressor gratings, signifies the applicability of our method as well as its potentiality for applications. In addition, since our method utilizes a semi-infinite cell, the generating medium can be easily kept at the same conditions. Thus, our method is attractive for the new generation high average power MHz amplified fs laser systems, which is the new trend for high averaged power XUV high harmonic sources, favourable for imaging applications and for basic research in storage ring accelerators.Compared to previously reported methods29. Furthermore, high efficiency plateau harmonics generated by only the short trajectory, achieved by our method, could be used in the attosecond pulse formation, since the produced attosecond pulses exhibit single sign frequency chirp, and thus could be compressed to their FTL value. Note that these attosecond pulses exhibit low divergence and thus achieve higher intensities with higher-quality foci favouring their use.We especially emphasise on our finding that, under certain experimental conditions, plateau harmonics are fully attributed to the short electron trajectory. The geometrical characteristics of such coherent harmonic radiation significantly favours coherent diffraction imaging applications in the scale of few-tenths of nanometersThe HGG experiments were performed using the experimental setup shown in Fig. mentclass2pt{minim19. The extension refers to the addition of the temporal dependence of the Ar gas ionisation and of the laser propagation in the generating medium that induces spectral variations owed to self-phase modulation (SPM) induced by the optical Kerr effect and by the plasma. This extension is necessary for our HHG conditions since the laser peak intensity is in most of the cases higher than the intensity inducing considerable Ar gas ionisation and thus the higher order harmonics are generated at the different temporal intervals of the leading edge of the laser pulse as seen in Fig. The developed spectral model for the high harmonics treats the problem as single atom response only temporally, omitting the spatial contributions like spatial phase matching, laser focusing, and self-phase modulation outside of the laser k-vector. It is an extension of the phenomenological model described by Carlstrom et al.qth-order XUV harmonic radiation, corresponding to the short and long trajectories, symbolised with the subscripts s and l, respectively, can be written as19:n is a non-linearity parameter for the HHG conversion. 39 and 41:c is the speed of light, The electric field of the qth-order XUV harmonic radiation obtained in Eq. (P(t) is the tunnelling ionisation probability up to the moment t, described as:4:l is the orbital quantum number, m is the magnetic quantum number, The HH spectral intensity is calculated as the absolute square of the Fourier transform of the coherent summation of the E-fields over the participating harmonics. For each harmonic the E-field of the short and long trajectory is considered. Thus, the spectral intensity equation reads:


The accurate segmentation of retinal vascular is of great significance for the diagnosis of diseases such as diabetes, hypertension, microaneurysms and arteriosclerosis. In order to segment more deep and small blood vessels and provide more information to doctors, a multi-scale joint optimization strategy for retinal vascular segmentation is presented in this paper. Firstly, the Multi-Scale Retinex (MSR) algorithm is used to improve the uneven illumination of fundus images. Then, the multi-scale Gaussian matched filtering method is used to enhance the contrast of the retinal images. Optimized by the Particle Swarm Optimization (PSO) algorithm, Otsu algorithm (OTSU) multi-threshold segmentation is utilized to segment the retinal image extracted by the multi-scale matched filtering method. Finally, the image is post-processed, including binarization, morphological operation and edge-contour removal. The test experiments are implemented on the DRIVE and STARE datasets to evaluate the effectiveness and practicability of the proposed method. Compared with other existing methods, it can be concluded that the proposed method can segment more small blood vessels while ensuring the integrity of vascular structure and has a higher performance. The proposed method has more obvious targets, a higher contrast, more plentiful detailed information, and local features. The qualitative and quantitative analysis results show that the presented method is superior to the other advanced methods. Retinal vascular image segmentation is an important topic in medical image research, which can effectively assist doctors in the clinical diagnosis and treatment of rapid cardiovascular diseases, diabetes and other diseases. In recent years, many scholars have studied retinal vascular image segmentation and achieved some results. However, due to the complexity of retinal images and the influence of noise and light factors in the image acquisition process, accurate retinal vascular image segmentation is still a challenging task ,2,3. TwoAt present, retinal vascular segmentation methods are mainly divided into supervised learning and unsupervised methods. Among the supervised segmentation methods, deep learning-based algorithms are currently a hot research topic. Reference  proposedUnsupervised segmentation methods can be divided into filter-based methods, clustering based methods, tracking based methods, morphology-based methods and threshold-based methods. Tolias et al.  first prMost supervised segmentation methods utilize manually designed features to model the retinal vasculature. However, manual design of features is a heuristic and laborious process that relies heavily on experience and skills. In addition, the parameters used in the algorithm usually need to be carefully tuned in order to address pathology, image noise, and other complex situations. The unsupervised segmentation methods have a fast operation time and low cost, and can obtain better segmentation performance. But these methods are susceptible to noise and cannot capture small blood vessels which provide important information for the detection of diseases. In this paper, a new unsupervised method is presented. We design a multi-scale joint optimization strategy for retinal vascular segmentation, which can segment more small blood vessels and improve the segmentation accuracy. The segmentation results are beneficial for the diagnosis, screening and treatment of cardiovascular and ophthalmologic diseases. The main contributions of this paper are as follows: (1) the MSR algorithm is used to adjust the brightness of the image and reduce noise. (2) A multi-scale Gaussian matched filtering method is proposed to enhance the contrast of the images. (3) The PSO algorithm is used to optimize OTSU three threshold, accelerate the speed and improve the accuracy.The remainder of this paper is arranged as follows. In The flow chart of the algorithm for blood vessel segmentation in this paper is shown in The basic idea of Retinex  is that When As shown in The red-green-blue three-channel extraction experiment is carried out on the images processed by MSR, and it can be found that the contrast between the target and the background of the red channel is low and the noise of the blue channel is large. The green channel image has a balanced brightness, high contrast and uniform gray distribution. Therefore, the green channel image is selected for subsequent processing.In this paper, we use Gaussian matching filter to extract the features of the retinal blood vessel. The Gaussian matching filter was first proposed by Chaudhuri et al. . The GauSuppose G represents the filtered image and R represents the input image.The filtered image is obtained by convolution of the input image with the two-dimensional Gaussian kernel. The mathematical expressions are as follows:Because the length, width, branch and angle of retinal blood vessels are different, it is difficult to accurately extract the vascular feature information in a single scale. Therefore, this paper selects a multi-scale matching filter to extract the characteristics of vascular images. When the macroscale image is selected for filtering, the coarsest blood vessel is mainly extracted. When the small scale is selected to filter the image, the smallest blood vessel is extracted. After many experiments, whenThe effect of vascular information extraction at scales σ of 1.9, 0.5 and 0.13 is shown in Multi-scale matched filtering can obtain most of retinal vascular information features information at different scales. In order to effectively enhance the contrast between the target blood vessel and the background and obtain better retinal vascular images, the results of each scale matched filtering are fused. The fusion calculation is given as follows:The fusion results are shown in When we use the large scale ofThe OTSU algorithm was first proposed in 1979 ; it seleThe given image is divided into The following two relationships expressed by Equation (12) can be easily verified:The objective function of the OTSU method can be defined asWhenPSO algorithm  is a swaThe background, target and noise of the pre-processed image are at different gray levels. In order to obtain the best segmentation effects, we use multi-threshold to segment the image, and the image can be divided into multiple regions with a multi-gray level. However, it would take too much time to search an optimal threshold combination in the full gray range. To simplify the calculation and improve the operation speed, we used the PSO algorithm to search the optimal threshold combination. It was found experimentally that when the number of the segmentation threshold combination is 3, a better segmentation effects can be achieved. Since the expert segmentation results in the retinal image data set are all binary images, in order to ensure the accuracy of the evaluation index calculation, it is necessary to use the OTSU single threshold to transform the retinal vascular image obtained after the OTSU-PSO algorithm into a binary image, and then the final result can be obtained after image post-processing (see (1).the median filter is used to denoise the image and connect the broken blood vessels.(2).the morphological processing is used to connect domain area and remove the large noise.(3).the mask image is extracted from the source retinal image, and the difference image between the source retinal image and the mask image is obtained.(4).The difference image is binarized by the OTSU algorithm, and then the binary image is expanded by the morphological processing.(5).The segmented vascular image is subtracted from the expanded edge image to get the final output image.The image obtained by the OTSU-PSO algorithm is re-segmented to get the segmented retinal vascular image. The segmented retinal vascular image has the following problems: (1) Some blood vessels are broken; (2) The field edge of fundus camera with false segmentation exists; (3) Noise is also enhanced when detailed features are extracted. In order to solve the problems and compare with the expert segmentation results of retinal image dataset, we post-process the image. The specific operation steps are as follows:Randomly selected images on the Drive dataset are used to test the PSO-based OTSU three-threshold segmentation results. The effects of the multi-threshold and single-threshold segmentation methods are shown in All the experiments are implemented in Matlab2016a  on 2.30 GHz processor with 3.8 GB RAM. We use two publicly available datasets, DRIVE dataset  to evaluIn order to better judge the segmentation effect of the model, it is necessary to compare the segmentation results with the ground truths manually marked by experts. Three most common evaluation metrics, Accuracy , the linear tracking morphological (M2) and cap transformation (M3), as shown in Compared with the unsupervised based methods, for DRIVE dataset, the In this paper, we present a multiscale joint optimization strategy for retinal vascular segmentation. The use of the multi-scale matching filtering method can enhance the contrast between the target blood vessels and the background. The optimization strategy utilized PSO can get the optimal segmentation threshold combination. In order to evaluate the effectiveness and applicability of the proposed method, the experiments are implemented on the DRIVE and STARE datasets. The qualitive and quantitative analysis demonstrates that the proposed method outperforms other existed methods and has strong robustness. The segmented images of the presented method have more small blood vessels and better integrity of vascular structure, which is beneficial for the diagnosis of diseases. The main purpose of retinal vessel segmentation proposed in this paper is to assist doctors in the diagnosis of cardiovascular and cerebrovascular diseases. In the future, we plan to classify fundus related diseases based on the segmented retinal vessels, such as glaucoma, senile macular edema and so on. Limited by the currently available datasets, in the future, we will build a new dataset that contains retinal images from patients of diabetic retinopathy, glaucoma and other ophthalmic diseases, and the new dataset can be used to evaluate the capability of the algorithms in handling pathological images.


Enriched environments and tools are believed to promote grasp rehabilitation after stroke. We designed S2, an interactive grasp rehabilitation system consisting of smart objects, custom orthoses for selective grasp constraining, and an electrode array system for forearm NMES. Motor improvements and perceived usability of a new enriched upper limb training system for sub-acute stroke patients was assessed in this interim analysis. Inclusion criteria: sub-acute stroke patients with  Stroke is the leading cause of disability in developed countries. Impairment affects the large majority of stroke survivors and most of them require rehabilitation. Effective treatment is timely required to avoid the learned non-use of the affected arm ,2. IncreUpper-limb paresis is the most common of a sequelae of impairments following a stroke, yet an effective treatment able to counter sensation of pain, feeling of foreignness, muscle weakness, and spasticity is still an open issue. NeuroMuscular Electrical Stimulation (NMES) is one of the treatments used to revert the learned non-use of the affected limb, and to improve grasp capabilities  in severWe designed an improved version of our device to comply with updated clinical and technological requirements, and to support unsupervised task-driven clinical tests. A preliminary test with a cohort of sub-acute stroke survivors was performed in two clinical centres.Historically, NMES systems have been tested primarily with chronic patients whereas studies on patients in the acute and sub-acute phases are more limited ,14. The In the next sections we describe the design process, the technological outcome, and clinical outcome of the interim analysis. Specifically, in Materials and Methods, we first describe the methodology for the iterative design and validation with clinicians, caregivers, and patients. The interaction of the central controller with orthoses, electrodes, stimulator, and environmental sensing elements is described, alongside with the modality of operations of each subcomponent. The description of a training session is then introduced. Finally, we analyse the interim data both in terms of patients’ outcome, and through a usability analysis of the whole assembly. One of the clinics involved in previous trials ,15 is taThe neuroprosthetic system was designed to rehabilitate hand functions through passive and active components. Passive wearables allow selective wrist and finger motion, and Electrode Arrays (EAs) induce controlled hand opening and closing via NMES. User-centred methodologies were used throughout the design phase; usability requirements were updated and refined during each development and verification phase. Physicians and neurologists, directly involved in the project, provided high-level clinical specifications; the bioengineering unit of one clinic further refined technical specifications. Two clinical focus groups—one with physicians not directly involved in the project, and one with patients and caregivers—were provided with an overview of the proposed system via a presentation and demonstration of looks-like prototypes of the sensorized orthoses. Questions raised during the demonstration were further addressed by the research technical staff and the hospital technical staff provided the updated requirement lists. In further steps, therapists suggested further changes. The resulting sub-components—Wearable NMES Orthosis, Graphical User Interface (GUI) for Virtual Electrodes (VE), Control Interface (CI), smart objects for environmental interactivity, Embedded Control System (ECS), and stimulation apparatus—are detailed in the following sections.The wearable NMES system uses topographically mapped multiple EAs, and acts as a modular tool usable for grasp rehabilitation. Three independent EAs are positioned on the forearm target extrinsic grasp muscles. An external stimulator with demultiplexers delivers stimulation patterns that can be manually tuned to elicit functional grasp, to obtain whole muscle conditioning, and to produce open-loop or closed-loop grasp control. Custom orthoses selectively constrain fingers and wrist, grasp force and hand kinematics are estimated through force and inertial sensors. The system interacts with daily-life objects able to supply information  to drive their usage. The objects are equipped with radiofrequency sensitive passive tags, and a reader embedded in the system processes the associated information in order to run predefined sequences of actions. A devoted processing of the signal strength received from the tags and environmental constraints allows the recognition of the selected objects among others. The device control software is implemented in two main elements: (i) a master CI operates on a windows tablet, and (ii) a real-time compliant ECS implementing a deterministic state machine aimed at the control of the stimulation apparatus and of the wearable NMES hand orthosis. This concept is exploited to drive rehabilitation exercises to obtain deeper knowledge on the recovery of the patient. We updated a previous prototype  by incluPost-stroke flaccidity, appearance of spasticity, and onsets of pathological synergies  affect tA GUI, hosted by the CI, represents sets of VEs associated with task-specific stimulation maps. Single VEs and complete stimulation maps can be personalised and tested in calibration mode. Location and intensity of stimulation to each VE is set on a touch responsive grid  mimickinlearning . PatientThe stimulator RehaMove Pro  is a minThe stimulator RehaMove Pro  is a minThe RFID module acknowledges the CI that a certain object or a position in the space has been reached by the patient. The system is composed by three elements: a radio-frequency emitter-receiver  with an external antenna positioned on the hand orthosis, objects/positions equipped with three passive tags each, and a back-end to manage the operations and elaborate the signal. For each tag, identified through its Unique IDentifier (UID), the Received Signal Strength Indicator (RSSI) measures the power of the backscattered signal. Functional elements in the physical workspace were labelled with programmable RFID tags, so that the signal received from a tag UID is associated with a specific smart object. The RSSI of each target tag is used as an estimator of the relative proximity between a smart object and the patient’s hand on which we mounted the antenna. Tags available in the operational range of the reader are retrieved and contextually filtered. For each instant, the RSSI value of the matching tags is extrapolated and weighted to provide spatial filtering. For each detectable object the five most recent frames of RFID information, approximately equal to one second, is averaged. The target is labelled as reached when the signal strength exceeds the corresponding threshold.The CI controls the modules not requiring real time constraints. It comprises a GUI, a database and a finite state machine to perform training sessions. The GUI is designed as a step-by-step wizard, guiding therapist and patient through the steps required by the training session: configuration (adapting the system to the patient’s needs), donning, parametrization (stimulation and state machine settings), training execution. The GUI is designed to be touch-compliant and to operate on a tablet PC . The ECS controls all the modules requiring real time constraints, such as the stimulator, the NMES controller and the IMU and force sensors. To keep the CI and the ECS synchronised, a strict master–slave concept using a custom-made communication protocol based on User Datagram Protocol (UDP) was implemented. The ECS is powered with one single medical power supply and all sensors of the orthosis are connected via one DSUB 25 plug. Inside the ECS a Beaglebone Black runs a standard Debian Linux kernel. A customised embedded real-time target for the Simulink Coder  is used to generate the real-time code for the experiment. Transitions between states are triggered upon reception of new sensor data , but only executed if predefined conditions are met.Each patient of the experimental group was assigned with a personal kit of electrode arrays, and of orthoses of the appropriate size, which could be further shaped as needed. A typical training session consists of four main phases: the setting, donning and parametrization of the system, and the training which consists of a sequence of selectable exercises. The GUI supports the therapist and the patient through all the phases.In the setting phase, the therapists created a new user or selected an existing one, and selected the desired exercise and its duration, while in the donning phase they positioned the electrode arrays on the patient’s forearm.During the donning phase of the first session, rigid orthoses of appropriate size were heated and shaped to match the patient characteristics. If the exercise were aimed at the wrist motion consolidation, the Type-B orthosis was donned. For the exercises requiring fingers motion consolidation, or for grasp exercises with the objects, the Type-A orthosis was required. In this case, after checking the electrode positioning, the rigid forearm–hand part was first stabilised with Velcro on the hand palm, then on the wrist, and finally at the proximal end.In the calibration phase, stimulation maps set through a GUI walkthrough. Each map, containing the stimulation location and intensity for the VEs, was updated as needed at the beginning of each session. Similarly, the desired kinematic thresholds in flexion and extension for the fingers, or for the wrist, and for the desired grasp force for each object were set. The touch interface allowed dragging of the VEs and the immediate application of the new configuration, therefore the therapist not only verified the efficacy of the stimulation, but also its perceptual acceptability. This approach also allowed the patient to self-calibrate the stimulation in an easy and understandable fashion. The most recent calibration parameters for the sensors and for the stimulation maps were stored on the GUI as templates. This approach allowed us to speed up the setup in the following sessions if the motor response did not differ significantly.Training consisted of exercises involving the reaching, hand opening, grasping, moving objects  in a plane, and releasing of the objects. Exercises could include the above-mentioned tasks, or a subset. Moreover, exercises were stratified to provide incremental complexity. A GUI controlled the exercises’ execution, and allowed the therapists to supervise the single tasks both visually and with audio.This interim clinical test is part of a larger multicentre RCT of the RETRAINER project, in which a lightweight arm exoskeleton (RETRAINER S1) and the hand orthosis (RETRAINER S2) were tested. The S1 and S2 subsystems are considered for a future integration, and thus tested on populations sharing the same inclusion criteria. This set of preliminary tests of the open-label parallel two-arms trial specifically refers to the RETRAINER S2 prototype, tested at Asklepios Neurologische Klinik Falkenstein, Königstein im Taunus, Germany, and at Clinica Villa Beretta, Centro Complesso di Neuroriabilitazione, Ospedale Valduce, Costa Masnaga, Italy. Participants were recruited directly by the clinicians from the inpatient population. In Italy the study was approved by “Comitato etico interaziendale delle provincie di Lecco, Como, e Sondrio”, protocol number 0019737/16U, and by the Italian Ministry of Health (protocol number 0022261 27/04/2016 DGDMF COD UO-P). In Germany the study was approved by Ethik-Kommission bei der Landesärztekammer Hessen, tracking number III/1/sja/bog FF 29/2016. The participants provided written informed consent before participation and consented to the publishing of their collected data.Patients who have suffered a first stroke with major unilateral functional impairment, aged between 18 and 85 years, male or female, can be enrolled in the study between two weeks and nine months after the acute event. The Motricity Index  of the aMRC>0) , and sufMMSE>20) . LimitatMMSE>20)   flexion and extension of fingers; (ii) grasp and release cylindrical objects with different sizes and weights; (iii) grasp, move, and release objects on a plane; (iv) grasp, move at shoulder level, and release objects in space. In exercises (iii) and (iv) the subject is seated in front of a desk with the height to have the elbow at 90° of flexion and no compensation of shoulder in frontal plane; three positions are chosen within the workspace reachable by the patient; an object has to be reached, grasped, and moved to the chosen position and released each time; after each drag and drop sequence the subject returns in the rest position. Counter gravity lifting is required in the most complex scenario. The mix of exercises is chosen by the supervising clinician according to the residual functional ability of each subject and the rehabilitation goals, and can be adapted in order to follow the evolution of the subject.It is particularly important to notice that central fatigue and perceived peripheral fatigue play a central role in the willingness to focus on targeted exercises execution, as well as specific dissociations between motor control, body representation, and sensorial afferences. The supplemental treatment with S2 may allow exploring combinations of treatments, or intensities of treatment not obtainable with the conventional mix of technologies. Hence the exercise selection may compensate for “feeling of early exhaustion” and “aversion to effort”  and alloThe experiment consists of twenty-seven treatments over nine weeks. Each treatment provided to the patients includes one hour of conventional treatment. The conventional treatment can include a combination of the following treatments: upper limb passive motion, arm cycle ergometer with or without NMES, NMES on non-target segments, upper limb exercises using augmented-reality or virtual-reality, occupational therapy, constraint induced movement therapy, upper limb active movement , repetitive task training, mirror therapy, writing training, and chemodenervation therapy. S2 orthoses and clasps are not used for the conventional treatment. Participants assigned to the control group receive an additional treatment of 30 min per session, whereas participants assigned to the experimental group perform exercises with the above described prototype. A test was conducted to validate the functionality of the system, verify its usability in a clinical environment, and to have an interim assessment of the potential outcome of the treatment. The functionality of the whole arm  were collected during the assessments at inclusion , end of treatment , and follow up . ARAT score and sub-features were compared between the two groups of patients with a linear model to predict score outcome using initial conditions and treatments as fixed factors and patients as random factors. Model acceptance, n 0.12.2 .The system here described is a prototype system to evaluate the technical concept, which of course can not provide the usability of a commercial device. However, Likert questionnaires were used to benchmark the perceived usability of the system. Patients training with the prototype evaluated the whole system at the training completion (T1). The assessment of the prototype was performed in a two steps process. Participants were asked to complete the System Usability Scale questionnaire (SUS) ,28. For Between 11 April 2016 and 1 December 2018, 24 patients were enrolled; in each arm 4 patients were unable to continue the treatment as outpatients for logistic issues and 16 patients completed the training. The interim analysis presented here focuses on 11 patients who met the supplemental conditions of ARAT score at inclusion not subject to ceiling effects  of the total ARAT score in chronic stroke patients is approximated to  to Lang , in sub-t-test for independent samples, Between the two groups, differences in age, delay of intervention after stroke onset, and ARAT level at inclusion were not significant (p < 0.0001), but not associated to specific groups. Grip score displayed a trend to significance  for in-treatment proportional variations . Gross movements score improved during the in-treatment phase (T1 vs. T0, In general patients improved after the treatment for both groups. Assessment results for ARAT scale and sub-scales are summarised in We proceeded assessing descriptive statistics and correlation of SUS and TAM scores at the end of treatment, and of the ARAT scores at inclusion. The SUS score, detailed in Our neuroprosthetic system has been developed taking into account clinical suggestions and tested in a clinical proof of concept with two clinical centres. The results confirm the feasibility of using our system for the hand rehabilitation after stroke. All the participants were able to use the system, and through the SUS and TAM assessment judged positively the usability of the device. During the first training sessions, clinicians needed to understand how to effectively calibrate the sensors and to set up the system. SUS scores improved over time, suggesting that an improved confidence of the clinicians with the device perceived by the participants. The main concept of this experimental treatment is to help the subject in performing the grasping of the interactive objects of the setup, and to perform exercises mimicking ADLs. Therefore, the target group consists of stroke patients with limited grasp capabilities, and with a variable degree of wrist-hand support. The capability to control accurately the extended reaching of an object—that is to stabilize correctly the shoulder and control the elbow—is only required for a subset of the exercises, and the workspace is adaptable. Our system seems able to complement the conventional treatments. As the overall ARAT score improvement is higher in the experimental group than in the conventional treatment, at least for patients with moderate to severe grasp deficiencies, it would be worth assessing how a higher quantity of treatment, or the interaction with other enriched treatments would balance the apparent benefits.The system here presented has some technological limitations. First, the electrode arrays could benefit from a more flexible material to have better localized adhesion with the skin, and longer tails acting as mechanical stress absorbers; the gel is also mechanically cut from the commercial roll, and the overall packaging design could be improved to ease the removal of the plastic layers used to prevent gel dehydration. Second, the stimulation of intrinsic hand muscles would allow training a variety of pinch grasps, now not possible. Third, ECS required ultra-low capacitance cables to communicate with the IMUs, requiring stiff cables. An ECS supporting reliably wireless sensors communication would avoid this problem, and improve overall lightness, usability, and reliability of the wearable. Additionally, the stimulator’s demux is not always able to sustain the required intensity of stimulation; sample cases are subjects with excessive adiposity—which acts as a high impedance—bad skin preparation, or badly connected wearables. Last, the engagement in the exercises is improvable with smoother environmental interactivity; since the RFID antenna should not obstruct the grasp, nor the user’s view, the current size of the RF antenna obliges to position it on the dorsal side of the hand; the hand thickness, in contrast, may shield the RF emission and response leading to a variable efficacy in object detection.The framework of reactive exercises with interactive objects seem to offer adequately challenging training to patients with limited grasp capabilities. Results emerging from the interim analysis suggest that a larger recovery may be conditioned to the use of the described system as a supplement to the conventional treatment. One source of clinical disparity lies in the fact that each clinic used its own standard best set of treatments as conventional treatment, consequently potentially masking or enhancing the effect of this enriched treatment. The sample size was low and thus the reported results are not generalizable to the target population of sub-acute stroke survivors with upper-limb motor impairments. However, it is important to notice that a clinical study is ongoing, and the complete dataset of 68 patients is expected to provide more robust insights on the treatment effects. Future work to translate the research findings herein reported into clinical practice will require the deployment of the proposed system in the home setting and a rigorous evaluation of its robustness, usability, and clinical effectiveness.