Monitoring mosquito density to predict the risk of transmission of the virus and develop a response in advance is an important part of prevention efforts. This paper aims to estimate accurately the mosquito swarm count from a given image. To this end, we proposed an attention-based multi-scale mosquito swarm counting model that consists of the feature extraction network (FEN) and attention based multi-scale regression network (AMRN). The FEN uses VGG-16 network to extract low-level features of mosquitoes. The AMRN adopts a multi-scale convolutional neural network, and with a squeeze and excitation channel attention module in the branch with a 7 × 7 convolution kernel to extract high-level features, map the feature map to the mosquito swarm density map and estimate mosquitoes count. We collected and labelled a data set that includes 391 mosquito swarm images with 15,466 mosquitoes. Experiments show that our method performs well on the data set and achieves mean absolute error (MAE) of 1.810 and root mean square error (RMSE) of 3.467. It has become a global public health problem3. Surveillance and control of mosquitoes is an important way to stop the spread of mosquito-borne diseases. Scientific and effective mosquitoes surveillance can evaluate the risk of outbreaks to provide early warning and take the best mosquitoes measure to control. It also provides a scientific basis for handling mosquito-borne disease outbreaks and effectively prevents the further spread of the epidemic. However, the existing mosquito counting methods, such as the human landing catch method and labor hour method, mainly rely on manual counting5, which have a large workload and are low efficiency. In addition, operators are easy to be bitten, so there is a certain risk of infection, and it is not easy to count high-density mosquito swarm. So it is of great practical significance to study a fast automatic counting method with a low error rate.As carbon dioxide emissions rise and global warming intensifies, mosquitoes become more active, and the risk of contracting mosquito-borne diseases increases. Mosquito-borne infectious diseases transmit pathogens through mosquito bites. Current climate change, ecological change, globalization, urbanization and other factors, as well as macro policies around the world, will have an important impact on the epidemic of this diseaseWith the advancement of science and technology, many techniques have made great breakthroughs in the fields of computer vision and pattern recognition recently, and they play an important role in many applications of different fields. Mosquito swarm counting based on image processing is a non-contact technique with high efficiency and high accuracy, and it can greatly reduce labor costs. Many counting methods based on image processing have been developed. Existing counting methods can be classified into detection-based methods, segmentation-based methods, and regression-based methods.7. The key step is feature extraction and refinement. Early research focused on extracting global features such as grayscale histogram8, color9, and texture of images10. Although such features are easy to extract, the representation based on global features does not work well in complex backgrounds. Therefore, some researchers extract various local features and fuse them as the final feature. However, these methods are not good for mosquito swarm counting especially when the mosquito swarm density is high and some mosquitoes touch each other.Traditional detection-based counting methods consist of steps including image acquisition, preprocessing, image feature extraction and refinement, pattern recognition, and counting12. These methods have achieved some good results. A limitation of these methods is that the segmentation results depend on image segmentation methods, which greatly affects the subsequent counting. Meanwhile the segmentation process is influenced by the image quality, adhesions between objects, isolated noise, etc., resulting in poor counting results.Segmentation-based counting methods segment objects from the background image and counts them14. This class method typically works by segmenting the foreground, extracting various features from the foreground, and utilizing a regression function such as ridge regression or Gaussian process regression estimating the number of objects15. So foreground segmentation results have great influence on the subsequent counting. Regression for objects density map estimates the density map by a regression function. Unlike the former class method, this class method is to learn the mapping between the features in the local area and their density map and indirectly estimate the objects’ density map using algorithms such as random forest16, then obtain objects’ count by integrating based on the density map. With the development of deep learning, the methods of density map estimation based on convolutional neural networks (CNNs)18 have been gradually favored by researchers. The methods can output the density map whose integration is the number of objects when the network inputs the image, which simplifies the estimation steps while achieving a good performance.With the development of the research, regression-based methods provide a new solution to the counting problem. These methods can be classified into two classes, regression for objects counting and regression for objects density map. The main idea of regression for objects counting is to learn a mapping between features and the number of objectsForeground segmentation is essential and indispensable in most computer vision tasks. However, in our task, we have to estimate the mosquitoes count without segmenting the foreground because it requires pixel-by-pixel labelling and inaccurate segmentation results will have adverse effects on the subsequent counting, especially there is no definitive information about scene geometry or motion, mosquito swarm size in the dataset, and even the view point of the image can be arbitrary. It is almost impossible to accurately segment the mosquito swarm from its background.Since the scale of mosquito can vary significantly in the images, it is difficult to hand-craft features for all different scales. We need to obtain features of different scales by automatic learning effective features to accurately estimate mosquito swarm count in different images.The density and distribution of mosquito swarm in the data set are significantly different, and even there are occlusion and adhesion among mosquitoes in some images. In addition, fewer pixels of mosquitoes in the image makes the task more difficult.Among the above counting methods there is no literature report on mosquito swarm counting. In this paper, we aim to perform accurate mosquito swarm counting from any still image, with any camera perspective and crowd density. It is a quite challenging task, since we need to overcome a number of challenges:Use the first 10 layers of VGG-16 network as the feature extraction network, which deepens the network and enables the subsequent ARMN to extract more complex high-level features of different sizes. It improves the density map accuracy of the network.Multi-branch architecture is adopted in the AMRN to extract features of different scales. The three different kernel sizes in the three branches correspond to receptive fields of three different sizes so that the learned features in three branches of CNNs are adaptive to large variation in mosquito size and details due to different perspectives or image resolutions.A channel attention module is introduced into the AMRN to increase the sensitivity of the network to the informative features and to increase the asymmetry of the AMRN structure to learn more different features, which helps the network to preserve the more information and improve the count estimation accuracy.The built network has no restrictions on input image size. The input image to the model can be arbitrary size, and the output is an estimate of the density of the mosquito swarm from which a swarm count can be derived.We build a new dataset for evaluation of mosquito swarm counting methods. Mosquito swarm datasets have not been reported in the literatures. In this work, we build a new mosquito swarm dataset of 391 mosquito swarm images with 14,566 accurately labeled mosquitoes. As far as we know, it is the first mosquito swarm counting dataset.To conquer the above challenges, we propose an attention-based multi-scale convolutional neural network (CNN) for mosquito swarm counting in any still image. It includes the feature extraction network (FEN) that consists of the first 10 layers of VGG-16 network and the attention based multi-scale regression network (AMRN) which consists of three parallel CNNs that have local receptive fields of different sizes. And a squeeze-and-excitation (SE) channel attention module is introduced into the AMRN to emphasize informative features. The output feature maps of all CNNs are concatenated and mapped to the density map whose integration is the mosquito swarm count. The main contributions of this paper are as follows:In this paper, we estimate the density map by an attention-based multi-scale CNN. To solve the problems that effectively extract features of mosquitoes at different scales, we deploy the FEN to extract the low-level features and deepen the network, and the AMRN with an SE module to extract more complex high-level features of different scales.19. The pre-trained model parameters from classification tasks are loaded into the proposed model, and then it is fine-tuned to convergence. The structure of the proposed network is shown in Fig. Deep networks require massive training data. However, it is not feasible to collect so much data with their corresponding labels in mosquito swarm counting. So it is not a good way to train a deep network from scratch, which will easily lead to overfitting the network. It is a desirable method to use transfer learning between task domains22, VGG-16 network is used to build the FEN by comparing different pre-trained model structures. We removed the fully connected layers of VGG-16 network to make the model adaptable to variable size inputs. VGG-16 model has 5 max pooling layers, and its output resolution will be reduced to 1/32 of the original size for each image. Due to the small size of mosquitoes, they occupy fewer pixels in the entire image. Too many pooling layers will cause serious loss of mosquitoes information, resulting in a low-quality density map and large counting error. Besides too deep FEN will increase the number of model parameters, making the training time be longer. We use the first ten layers of VGG-16 model as the FEN to extract low-level features of mosquitoes. The FEN uses the three max-pooling layers of VGG-16 network so that the output feature map is reduced to 1/8 of the original size for each image, which can reduce the number of parameters and speed up training while ensuring good performance. And max-pooling layer helps to preserve the texture information of the image, which makes the model focus on the texture features of mosquitoes as much as possible.Borrowed from the literatures17. Inspired by the literature23, we proposed a multi-scale feature fused network using convolution kernels of the local receptive field with different sizes to achieve the feature maps of mosquitoes with different scales. The AMRN has three different sizes of receptive fields to extract features. It consists of three parallel CNNs with the convolution kernel sizes The AMRN further extracts more complex high-level features of different sizes and estimates pixel-level density map by a regression network. Convolution kernels of the same size receptive field are unlikely to capture features of mosquitoes with different scales24 to increase the sensitivity of the network to the informative features and to suppress the unuseful ones. SE attention module contains SE operations as shown in Fig. Due to different parameters in the convolution kernels, different branches extract different features. We used an SE channel attention moduleTo verify the effectiveness of the proposed model, we evaluated experiments on the built mosquito swarm dataset. We conducted comparative experiments on the proposed network with different settings, including the different pre-trained networks in the FEN, different scales in the AMRN, the location of the SE module, and whether SE modules exist or not.The images of mosquito swarm in the data set were collected from the Zhejiang Provincial Center for Disease Control and Prevention. The images were captured and the size of each image is We used the proposed network to estimate the mosquito swarm density map from an input image, and the ground truth density map quality in the training data largely determines the performance of our method. To effectively extract features of mosquitoes with different scales, we empirically classified the images into 8 classes according to the size of the mosquitoes, as shown in Fig. Considering the fact that the data set contains mosquitoes of different sizes, we can not accurately get the sizes of mosquitoes to determine the sizes of Gaussian kernels. But we found that it is effective to empirically classify the mosquito sizes into 8 classes with their corresponding mentclass2pt{minimith input image. Euclidean distance is used to measure the difference between the estimated density map and the ground truth density map. The loss function is defined as Eq. :2\documeith image. ith image.We evaluated the proposed method using mean absolute error (MAE) and root mean square error (RMSE). MAE indicates the accuracy of the predicted values, and RMSE indicates the robustness of the predicted values. The smaller of MAE and RMSE, the better the accuracy of the prediction model. MAE and RMSE are defined as Eqs.  and 4)  4) respen as Eq. , where \All experiments are performed on a computer with NVIDIA 1060 GPU, 16 GB RAM running Linux operating system under Anaconda Python3 environment. The back propagation time and stochastic gradient descent (SGD) are adopted for network training. The batch size is 2. The learning rate (LR) is In the process of network training, MAE and RMSE in the test dataset are measured after each epoch. We showed the curves for MAE and RMSE in Fig. We tested the network on the test dataset, and achieved the best result with MAE of 1.810 and RMSE of 3.467. Figure We compared the results of different pre-trained networks in the FEN. VGG-16, ResNet-18 and InceptionV3 are the three common pretrained networks. We tried to deploy them as the FEN to extract low-level features in the proposed network respectively, and their results are shown in Table We also compared the results of different numbers of convolution Layers in the FEN. The backbone of the FEN is VGG-16 network. We conducted experiments on the proposed network on the training set with different numbers of convolution layers in the VGG-16 network, and their corresponding MAE and RMSE are shown in Table We also compared the results on the single-scale and multi-scale AMRN. And Table In order to study the influence of the SE channel attention module in the model, we evaluated its experimental results on three branches 3 × 3, 5 × 5, and 7 × 7 respectively. And the experimental results are shown in Table 17 and the CSRNet method18 on the complete test set. These two methods are classical density map based regression methods in the field of crowd counting. MCNN model contains three parallel CNNs and each parallel includes four columns CNNs. Different parallel CNNs have different sizes for modeling the density maps corresponding to objects of different scales. MCNN stacks the output of all CNNs and maps them to a density map. CSRNet method uses the first 10 layers of VGG-16 as the front-end network and dilated convolutional layers as the back-end network to enlarge receptive fields and extract deeper features. Compared with the other two methods, the performances of MCNN and CSRNet method are inferior to our method. Unlike the proposed network, MCNN model lacks of an FEN and different parallel CNNs have the same architectures except for the sizes and numbers of filters. So, on the one hand, our network is a deeper network, which can extract more complex features than MCNN. On the other hand, we introduced the SE module that boosts informative features of the mosquitoes, into the regression network to increase the asymmetry of the network structure, so that the network can learn more different features than MCNN. For CSRNet, it can extract deeper features and obtain smaller MAE than the MCNN, but it is the network with sing-scale and could not adapt to mosquitoes of different sizes. Meanwhile, the introduction of SE module in our regression network makes CSRNet further disadvantaged. The experimental results are shown in Table To highlight the performance of the proposed model, we compared the proposed method with the MCNN methodWe have successfully applied the deep learning and channel attention module in mosquito swarm counting. We first used the modified VGG-16 network to build the feature extraction network to extract low-level features of mosquitoes, then adopted a multi-scale regression network with SE channel attention module to extract more complex high-level features and mapped feature maps to mosquito swarm density maps. To better evaluate the performance of the proposed method, we collected and labelled a data set including about 391 mosquito swarm images with 14,566 mosquitoes. Our method performs well on the data set and achieves mean absolute error (MAE) of 1.810 and root mean square error (RMSE) of 3.467. In the future work, we will study mosquito swarm counting methods applicable to mobile terminals, especially the deep learning model based on lightweight network.


This study seeks to assist small and medium enterprises break free of the constraints of the conventional financing model and lessen the supply chain finance risks they face. First, the supply chain financial business model and credit risk are analyzed, followed by a discussion of the application principle of blockchain in the control of supply chain financial credit risk. The next topic up for discussion is the emancipation of individuals and the application of financial technology toward the management of financial risk in supply chains. In the final stage of the development of the computerized risk assessment model, the Fuzzy Support Vector Machine (FSVM) is optimized, and the effectiveness and efficiency of risk classification are enhanced by introducing a variable penalty factor C. To test the efficacy of the C-FSVM risk assessment model, the Chinese auto sector is used as the study’s object. According to the results of the study, the C-FSVM model has a classification accuracy of 96.35% for the entire sample, 96.45% for credible firms, and 95.34% for default enterprises. The training time of the C-FSVM model is 473.9s, which is far lower than the SVM and FSVM models’ training times of 1631.6s and 1870.2s. In summary, the C-FSVM supply chain financial risk assessment model is effective and has great application value in banking practice. There is a network chain structure made up of SMEs that collaborate upstream and downstream, linked by the risk platform of supply chain finance and the cooperation platform of the industrial supply chain. Supply chain finance in this instance evaluates the upstream and downstream of the supply chain based on the structure of supply chain transactions and the comprehension of operational risks, with the aid of the credits of core firms or the self-compensation of online merchants. True to its name, risk management is fundamental to supply chain finance –3. Its bThe overall financial infrastructure for the dynamic expansion of the technology-enabled supply chain must seek to enter industries based on strategic needs. It is not viable to construct a large, all-encompassing, cross-functional platform in the business-to-business sector. It is possible to fine-tune and efficiently manage the supply chain structure by breaking it down into smaller parts based on the high-level design ,5. SupplA growing number of fintech firms are specializing in the supply chain industry, and they’re doing so by offering businesses digital SCFSs and technological solutions based on blockchain –12. CompLarge and medium-sized businesses are frequently well-established, financially stable, and operate on a large scale. When it comes to funding, commercial banks and other financial institutions typically favor large and medium-sized businesses, ignoring the requirements of SMEs. By using the credit of core firms to reactivate corporate inventories and accounts receivable, the SCFR management model helps SMEs resolve their financing problems, lowers the cost of financing for corporations, and improves the operating efficiency of funds ,14.Accounting receivables are pledged as collateral for business loans so that companies can fund their operations. Buyers farther down the supply chain can use confirmed warehouse finance to apply for loans on the platform to pay upstream core suppliers in advance of future products deliveries. Concurrently, the supplier agrees to repurchase any items that aren’t picked up, and will transfer the right to do so to a finance model managed by banks. Financing warehouses are a type of inventory financing in which a company uses its stock as collateral in a loan arrangement with a bank. Therefore, the finance warehouse service may provide high-level logistics services for companies, resolve financing issues for SMEs, close the cash flow gap in business operations, and boost supply chain efficiency –17. The The current paradigm for managing the financial risks inherent in supply chains is predicated mostly on the management and monitoring of key firms. Unfortunately, there are two major flaws with this model. On the one hand, the lack of thorough pre-loan investigation, loan supervision, and post-loan management might lead central enterprises to blindly give credit guarantees for upstream and downstream enterprises based on various social networks. On the other hand, fraud involves a conspiracy between businesses to fake supply contracts. There is a significant increase in supply chain risk because of the existing absence of appropriate tools to deal with information asymmetry among chain companies and collusion fraud among chain businesses . It is aThe data in a blockchain cannot be altered and verification is straightforward. The information from one data block is used to construct the next block. Checking the final block’s function value is sufficient for confirming all accounting. Under the current technical framework, blockchain may be broken down into three distinct types: public chain, alliance chain, and private chain. The consortium technique is used by the blockchain-based supply chain financial system in all related situations. Users include financial institutions like financing banks, core merchants, and third-party logistics providers, and supply chain members including upstream and downstream shops acting as nodes in the network. Safe, efficient, and consistent with standard business practice are only some of the features of the six-tiered blockchain technology that characterizes the alliance link between the public and private chains. Alliance chains in supply chain finance use contracts and other techniques to create a blockchain that facilitates trust and consensus amongst participating enterprises and institutions. Credit penetration of the supply chain financial system is made possible by blockchain technology, which also alleviates the burden of high financing costs for intermediaries. Two distinct functions are fulfilled by the blockchain. In order to resolve the credit financing issues faced by secondary suppliers, it is necessary to first check the rights of the core enterprise, which includes verifying and certifying the authenticity and validity of the complete bill.To effectively transmit credit along the supply chain, reduce cooperation costs, and increase performance efficiency, this ecology of trust allows the credit of core enterprises  to be converted into digital warrants, while smart contracts are used to prevent performance risks . When diCommercial banks get access to the supply chain in terms of financial financing thanks to the trust endorsement connection between the upstream core firms and the downstream SMEs, using the accounts receivable financing model as an example. Insight into how a large company C earns the trust of a small company S can be gained by applying evolutionary game theory to this process. If the core enterprise agrees to guarantee the financing of SMEs, its expected income can be written as A company’s predicted profit is given by Then, Functions representing the income expectations of small and medium-sized enterprises (SMEs) considering financing, non-financing, and mixed strategies are as follows.A corresponding replication dynamic equation for SMEs to select financing is given by Combine Eqs  and 8)  8) to foFurther solution can be obtained according to Eqs  and 11)11).Ultimately, five game behaviors can be obtained, namely 0, A, B, C, and D. Supply chain finance works best in sectors with wide market spaces, dispersed upstream and downstream markets, standardized production goods, constant demand, and modest price changes. As a result, commercial banks place a premium on sectors with established markets and high levels of maturity, such as the petrochemical, coal, electricity, and non-ferrous metals sectors, because of the resource features of these sectors. Supply chain financing has expanded into new areas, propelled by the proliferation of Internet-enabled devices and services, such as in the computer communications, medical, agricultural, light industrial, retail, and other sectors.The growth of supply chain financing generally and the promotion of digitization and capitalization through increased transparency and efficiency are all factors that contribute to the emergence of digital supply chain finance ,23. ConnBy establishing an accurate risk control and early warning system, businesses can fortify client institutions to supply financial institutions with more authentic and dependable data during the pre-loan approval process and guarantee the approval effect. The team will execute sensor fusion processing on the aforementioned multi-channel data and link enterprise data with generic risk control indicators using foundational technologies like smart IoT and edge computing to boost the indicators’ credibility, reach, and responsiveness. The direct coupling of systems like enterprise resource planning and software-as-a-service, as well as the availability of data interfaces, ensures the data’s veracity against the backdrop of monetary technology’s emancipatory potential; real-time monitoring and timely warning are conducted based on the comprehensive credit granting model for transaction data and external data; the IoT enables the linking of digital information with physical objects, hence lowering the price tag associated with product monitoring .In supply chain finance, the credit risk of SMEs comes from both objective breach of contract and moral hazard and the objective breach of contract and moral hazard of their counterparties (leading enterprises in the supply chain). On the one hand, debt repayment capacity is influenced by the financial health and management of SMEs and their counterparties; on the other hand, the supply chain’s future prospects and competitive climate. Stability in the cooperative relationship between SMEs and leading firms and the credit status  of both parties should have an impact on the moral hazard of both. As a result, the financial position and credit level of supply chain counterparties, the number of supply chain partnerships, and the level of supply chain development are all factors that influence SMEs’ credit risk. In addition, the emergence of third-party credit complicates the assessment of credit risk in supply chain finance. If commercial banks wish to evaluate the credit risk of certain SMBs, they must analyze the supply chain, which includes the SMBs and their partners .d12 between them is the classification interval, which can be written as The neural network and SVM-based classification model have a remarkable capacity for identifying credit risk. Consequently, this study introduces the support vector machine model for assessing the credit risk of SMBs in the supply chain. SVM, a machine learning algorithm based on kernels, excels at addressing nonlinear, separable classification problems. SVM’s learning technique, by constructing the multidimensional decision-making optimal classification hyperplane, may accomplish the ideal separation of two categories of data with reduced empirical risk . Hyperplω stands for the feature space dimension, and b refers to the undetermined scalar parameter.In When assigning credit to newly collected data, SVM often uses the most recently classified samples as the default, reliable ones. It is challenging to execute an exact credit risk evaluation via SVM in the supply chain financing mode because the target financing firm is not entirely credible or defaulting, and there are frequently various indications used for evaluation. The fuzzy link between credible samples and default samples is described by the membership degree determination function and the fuzzy relationship matrix in the FSVM theory, which are used in the FSVM state analysis method. This process continues until the membership degree matrix has been set. For fuzzy reasoning, a membership function is chosen. The search for the best solution to the objective function can be seen as a way to solve the optimal classification surface problem.C signifies the penalty factor. The optimal discriminant function of FSVM can be written as In In light of the fact that the bank should punish defaulting customers more severely, knowing which customers are likely to go into default is more crucial than knowing which customers are reliable. This study builds on the FSVM model by introducing a new variable penalty component for model optimization. Finally, the empirical risk and confidence interval can be balanced with the help of the non-negative penalty factor C, which allows SVM model classification to incur the least amount of structural risk possible. Therefore, the value of the penalty factor should satisfy After incorporating the penalty variable, the quadratic programming issue is stated as C1, as shown in The ideal classification function can be derived by picking the proper kernel function and variable penalty factor This study chooses China’s automobile manufacturing industry as its research object in order to precisely analyze the C-FSVM risk assessment model developed here. There are 82 different businesses involved in the auto industry; ten are core firms that produce cars in the center of the supply chain, and the remaining 78 are SMEs that focus on sales and provide things like engines and software for cars. Shenzhen Stock Exchange provides annual reports and information disclosure data for each firm, while the "China Enterprise Financial Information Analysis Database" supplies the quantitative data for the sample.Principal component analysis on the SCFR assessment indices is performed using the Factor procedure in SPSS software. Parameters are optimized using a 5-fold cross-check procedure, which yielded C1 = 20 and C2 = 750 for the penalty factors of credible and default samples, respectively. Classification outcomes are recorded and compared with preexisting classification data, and both the training and test samples are randomly picked for testing a total of five times.The main component f is input for model training in Matlab, and the classification results for the training samples are acquired. Next, the trained model is put to the test by inserting the test samples into it. Rates of type I error (for credible users) and type II error (for default customers) are displayed in Classification accuracy and model training time are two areas where the C-FSVM model stands out from its competitors, the SVM, the FSVM, and the LSFSVM. Smart money management has entered the scene. Despite technological advancements, fundamental facets of finance remain unchanged; this includes the insidious nature of hidden, contagious, and unexpected financial dangers as well as the inability to control their spread. Financial risk assessments look at the supply chain as a whole and not just at any one link in it to determine risk. Financial institutions no longer solely base their risk assessment on the industry, firm size, and guarantee procedures associated with the applying SMEs. Supply chain finance, as a financial service for the entire supply chain, necessitates an evaluation of the overall industry risk faced by SMEs.This study investigates the potential of technology to facilitate the creation of a model for assessing the financial risks inherent in the supply chain. A new, flexible penalty component is introduced to fine-tune the rating system based on the FSVM model. Finally, the empirical risk and confidence interval can be balanced with the help of the non-negative penalty factor C, which allows SVM model classification to incur the least amount of structural risk possible. The effectiveness of the C-FSVM risk assessment model is evaluated in this study by applying it to the Chinese automobile industrial sector. The total sample classification accuracy of the C-FSVM model is found to be 96.35%, while the identification accuracy rates for credible firms are 96.45% and for default enterprises they are 95.34%. It is evident that the C-FSVM risk assessment model developed here can be widely applied in commercial bank risk assessment practice. This study is not without its flaws. In this case, for instance, the research object is the automotive sector, which imposes constraints. There is a need for additional validation of the risk assessment model made possible by blockchain technology.S1 Data(ZIP)Click here for additional data file.


Electrical treeing is one of the main degradation mechanisms in high-voltage polymeric insulation. Epoxy resin is used as insulating material in power equipment such as rotating machines, power transformers, gas-insulated switchgears, and insulators, among others. Electrical trees grow under the effect of partial discharges (PDs) that progressively degrade the polymer until the tree crosses the bulk insulation, then causing the failure of power equipment and the outage of the energy supply. This work studies electrical trees in epoxy resin through different PD analysis techniques, evaluating and comparing their ability to identify tree bulk-insulation crossing, the precursor of failure. Two PD measurement systems were used simultaneously—one to capture the sequence of PD pulses and another to acquire PD pulse waveforms—and four PD analysis techniques were deployed. Phase-resolved PD (PRPD) and pulse sequence analysis (PSA) identified tree crossing; however, they were more sensible to the AC excitation voltage amplitude and frequency. Nonlinear time series analysis (NLTSA) characteristics were evaluated through the correlation dimension, showing a reduction from pre- to post-crossing, and thus representing a change to a less complex dynamical system. The PD pulse waveform parameters had the best performance; they could identify tree crossing in epoxy resin material independently of the applied AC voltage amplitude and frequency, making them more robust for a broader range of situations, and thus, they can be exploited as a diagnostic tool for the asset management of high-voltage polymeric insulation. Electrical energy is generated, transmitted, and distributed through power system networks, and the utilization of that energy is through many electrical apparatuses. One important aspect affecting the reliability of power systems and apparatuses is the dielectric (equipment insulation) reliability. Insulation systems are key elements in the safe and reliable operation of not only high-voltage equipment, but also power electronic devices that work at high electric fields, stressing dielectric materials . Epoxy rThe main mechanism of degradation and long-term failure in solid polymeric insulation  is the formation of electrical trees ,8. They PDs in electrical trees have been analyzed using more traditional techniques such as phase-resolved partial discharge (PRPD)  and pulsSamples were prepared in epoxy resin Mepox 1124(cl)/2262 (DGEBA type of transparent resin), with the resin being manufactured by Kumho P&B Chemicals and the hardener by Yunteh Industries. The samples’ geometry was conventional needle-plane configuration with a separation of 2 ± 0.3 mm between the energized metallic needle and the bottom grounded-plane, as shown in a) were measured by simultaneously using two PD measuring systems: the commercial equipment Omicron MPD600 and the combination of an oscilloscope (Keysight Infiniium S DSOS104A) and a High-Frequency Current Transformer . The image monitoring and recording of tree progression were carried out with a Canon T6 reflex camera with a Canon MP-E-65 mm f/2.8 1–5× macro lens. For more details on the measurement system, see [For the growth of electrical trees in the specimens, the experiments were carried out under monitored laboratory conditions using the test circuit shown in tem, see .As shown in MW), which characterizes the frequency spectrum of PD signals where their energy content is more relevant. With these two parameters, a 2D graph (map) is plotted [As the data analyzed for this paper were obtained from two measurement systems, MPD 600 and an oscilloscope, different procedures were applied in each case. The analysis intervals from both measurements were the same, and their selection was dependent on the oscilloscope measurements, which were not continuous recordings as for Omicron. The data that came from the Omicron measuring system contained, for each PD pulse, the phase angle, time of occurrence, and the apparent charge to construct PRDP and PSA patterns. The data were filtered to reduce the presence of noise in the measurement; this was achieved by discarding all the discharges with apparent charges below a threshold defined by a test, which in this case varied from 3 to 5 pC. The other PD measuring system was the oscilloscope. A signal source separation process was applied to the measured waveform data to eliminate signals associated with noise, following the procedure described in . This se plotted . The resTo visualize the evolution of the parameters, graphs such as the one in Among the main advantages of using the PRPD analysis technique is the extensive amount of previous research and accumulated experience, as well as the relative ease of calculation it represents. However, it also has disadvantages, such as availability problems relating to pure sinusoidal HV sources for PD tests (diagram distortion with sources with harmonic contamination) or measurement problems in noise-contaminated environments. In PRPD analysis, each PD event is represented by its phase of occurrence and charge amplitude. PD information is measured during several cycles of the external voltage and then graphed in a 2D plot, usually known as PRPD diagram or ‘pattern’, such as the one shown in The PRPD analysis used here has two approaches: qualitative and quantitative. The qualitative approach uses the visualization of PRPD patterns, using the applied voltage as a reference signal and plotting the phase of occurrence of the pulses and their apparent charges. This representation allows the creation of patterns associated with different PD sources and, therefore, can be used to identify PD sources. However, when using just PRPD patterns, the classification is not easy, and it usually requires highly trained human experts . On the  consult ,20.Partial discharge in solid dielectrics relates not only to the externally applied voltage but also to the local electric field in the cavity; however, it is not easy to measure the local electric field which produces the discharge. Thus, pulse sequence analysis (PSA) was developed to obtain information about the changes in the local electric field from variables that could be measured externally. To represent each PD pulse, PSA could use the data of the charge amplitude  2D plot ,21.PSA focuses on a qualitative approach to the study of PD patterns for which the visualization of graphs is essential. This tool considers the relationship of each pulse with its neighbors to avoid the loss of information on the physical phenomenon associated with the variation in the electric field. PSA proposes various patterns to analyze, but in the case of this work, the main analyzed pattern was rization ,23.In order to simplify the comparison between tests at different voltage and frequency levels, the patterns were normalized utilizing Equation (1), obtained from .(3)ΔUnΔ consult .N measurements of a single variable x; therefore, we have a time series NLTSA is a set of tools derived from nonlinear dynamic system theory usually related to mathematic chaos. This paper used the correlation dimension loyed in . The algc system .In this context, the correlation dimension can be interpreted as an estimation of the fractal dimension of the state space trajectory of the reconstructed dynamical systems. Furthermore, provided some formal requirements are met, Takens’ method ensures that the reconstructed space’s correlation dimension is approximately equal to that in the actual state space. This is key in practical applications of correlation dimension, such as PD analysis, because it allows for the characterization of PD dynamics through correlation dimension N is the total number of PD, In this equation, Applying NLTSA tools is of interest in electrical treeing since this phenomenon can be analyzed as a nonlinear system . In thisThis method of discharge analysis is based on characterizing the waveform of each pulse by means of parameters in the time or frequency domain which are taken to graphic schemes to search for patterns or relationships that provide relevant information. These have been mainly used for partial discharge classification and noise filtering. Among these techniques, we can find the time frequency maps  and the These parameters were proposed in  and seleThe technique of time frequency maps  is descrThese parameters are derived from the calculation of the spectral power of the PD signals in two frequency bands  and thenThe analysis in this section focuses on evaluating the progress of electrical trees in epoxy resin using PRPD, PSA, and NLTSA techniques and comparing them to the results obtained using the PD waveform. The evolution of a variety of parameters derived from the waveform and others from the PRPD analysis is shown in sults in , and theIt can be observed that considerable changes occur in all the parameters in the crossing to the counter-electrode, which is coincident with the findings in .Given the importance of crossing in the development of an electrical tree, it is relevant to visualize the changes that occur in PRPDs before and after the tree crosses the insulation gap. In order to quantitatively visualize the variations in PRPD, the statistical moments associated with the skewness, kurtosis, mean value, and variance of the data should be compared with respect to the charge, positive cycle C(+), and negative cycle C(−), as shown in 2 to 2000 pC2, which in terms of the standard deviation is a variation from 4 pC to 44 pC, evidencing change in the order of magnitude.An increase in the mean value of charge can be seen in Since the results shown in The PSA patterns in the pre- and post-crossing intervals were analyzed to evaluate the ability of the PSA technique to identify the insulation crossing of the electrical tree and compare with the PRPD technique. In general, The previous analysis shows that the parameters calculated from the PRPDs could present problems as predictors of the development stages of trees due to their high variability between samples; so, considering the possibility of applying machine learning to identifying stages of development could complicate its application. However, this would not occur with the waveform parameters, since it was observed that they present more stable and robust behavior when dealing with changes in the applied voltage and frequency, especially in the post-crossing stage.We compared techniques for characterizing electrical tree growth in epoxy resin to evaluate their ability to identify tree bulk-insulation crossing, a key event that is a precursor to the catastrophic failure of high-voltage solid polymeric insulation. We deployed traditional techniques such as phase-resolved partial discharge (PRPD) and pulse sequence analysis (PSA), and more recent techniques such as the tools of nonlinear time series analysis (NLTSA) and PD pulse waveform analysis. Both traditional partial discharge analysis techniques (PRPD and PSA) had the ability to identify the crossing of the electrical tree to the counter-electrode. The PRPD analysis showed sensitivity in the charge histogram parameters, displaying a distribution that tends to positive asymmetry and the leptokurtic type in the post-crossing; the parameters of mean value (The waveform parameters show variation in the tree bulk-insulation crossing event, thus identifying the moment when this stage is reached. In general, PDs tend to be shorter and faster pulses in post-crossing; that is, the temporal parameters (Overall, it is concluded that techniques based on the information that the PD pulse waveform provides are the most appropriate methods for assessing the aging state of epoxy resins due to electrical tree propagation. While the focus of this study was on evaluating the ability of these techniques to identify tree insulation crossing, future research can expand on these findings to characterize the type and propagation of electrical trees. By developing more advanced criteria and intelligent systems for insulation diagnosis, researchers can enhance the accuracy and efficiency of the analysis and gain a more detailed understanding of the aging process. The insights gained from this study have the potential to inform more effective maintenance and replacement strategies, thus contributing to the development of more reliable and sustainable power systems.


Human action recognition is an essential process in surveillance video analysis, which is used to understand the behavior of people to ensure safety. Most of the existing methods for HAR use computationally heavy networks such as 3D CNN and two-stream networks. To alleviate the challenges in the implementation and training of 3D deep learning networks, which have more parameters, a customized lightweight directed acyclic graph-based residual 2D CNN with fewer parameters was designed from scratch and named HARNet. A novel pipeline for the construction of spatial motion data from raw video input is presented for the latent representation learning of human actions. The constructed input is fed to the network for simultaneous operation over spatial and motion information in a single stream, and the latent representation learned at the fully connected layer is extracted and fed to the conventional machine learning classifiers for action recognition. The proposed work was empirically verified, and the experimental results were compared with those for existing methods. The results show that the proposed method outperforms state-of-the-art (SOTA) methods with a percentage improvement of 2.75% on UCF101, 10.94% on HMDB51, and 0.18% on the KTH dataset. Recognizing human actions is a vital process in most computer vision applications, such as violence detection ,2, surveThe proposed work is motivated by the information bottleneck theory , which aThe major contributions are as follows:The fusion of spatial and temporal cues, represented by intensity and optical flow vectors, respectively.The proposal of a single-stream shallow network—HARNet Architecture—for extracting deep learned action features.The proposal of a KNN-based machine learning framework for classifying up to 101 human actions.Experimentation and comparison with SOTA (state-of-the-art techniques) on a benchmark dataset.The organization of the paper is as follows. Recently, deep learning networks have been widely used for analyzing events in videos. One such model was designed using ResNet50 to extract the important features of each frame of the input, followed by a recurrent neural network (ConvLSTM) for detecting any abnormal events . A methoPedestrian attribute recognition (PAR) is very important in the computer vision techniques applicable for video surveillance systems. PAR-based methods have been implemented to provide a comparison between deep learning and conventional algorithms . In ordeThe proposed spatial motion feature learning framework comprises three steps, namely preprocessing, the design of the neural network, and representation learning for HAR, as depicted in The input video data are segmented into frames, which are subsampled to reduce the redundancy. The motion vector is computed by finding the optical flow between the frames using the Horn and Schunk approach . The intThe proposed network is constructed by stacking five stages of convolution layers, as shown in The convolution layers in the following stages of the network have an increasing number of channels with a scaling factor of 2, but the size of the kernel is fixed at v denotes preprocessed video data about the random output variable l denotes the label of the output action category. Given their transition probabilities The information bottleneck principle is based on information theory, which is used to extract more significant contents contained by the random input variable X and Y is assumed. As shown in Y that is lost in one layer cannot be regained in the succeeding layers. According to the information theoretic learning principle for deep neural networks (DNNs), each layer in a DNN processes only inputs from previous layers. There is a loss of information in the consecutive layers Y. The equality in the above expression can be achieved if, and only if, each layer gives sufficient statistics for its input. Hence, it is necessary to obtain not only the most pertinent representation at each layer, but also the most compact representation of its input. Hence, a massive layered network may result in an information bottleneck. This limitation can be overcome by designing a shallow network. Each layer needs to try to enhance Significant average information is given by mutual information asThe features learned in a fully connected layer are extracted to train the machine learning classifiers. The efficacy of the proposed network in latent representation learning has been analyzed with various ML classifiers, such as support vector machine (SVM), decision tree (DT), k-nearest neighbor (KNN), linear discriminant analysis (LDA), naïve Bayes (NB), and ensemble.k-Nearest Neighbor classifier: k-NN is practically applicable for recognizing patterns of human actions, as it is non-parametric, which means that it does not assume the distribution of data. Hence, it works well in our proposed approach. The k-NN classifier stores all cases of training data and tries to classify test data based on similarity measures. Euclidean distance is considered as a similarity measure for finding the neighbors in our experiment. The number of neighbors included for classifying the test sample is one.Support Vector Machine: Complex data transformation is performed by SVM based on the chosen kernel function. With the help of those transformations, the separation boundary between data is maximized.Decision Tree: The decision tree is a systematic approach used for multi-class classification problems. A set of queries in relation to the features of a dataset is posed by DT. It is visualized using a binary tree. Data on the root node are again split into two different records that have different attributes. The leaves represent the classes of the dataset.Naïve Bayes: Bayes theorem is the basis of the naïve Bayes classification method. The naïve Bayes method is used because of the assumption that there is independence between every pair of features in the data.Linear Discriminant Analysis: The linear discriminant analysis (LDA) classification method is used in our experiments. It assumes that data of different classes are based on different Gaussian distributions. LDA uses the estimated probability that a piece of test data belongs to a particular class for classifying it. The class with the highest probability is predicted as the output class of the given sample.Ensemble: The adaptive boosting multi-class classification method is used as an ensemble aggregation approach for our experimentation. The number of learning cycles used in our experiments is 100, with the same learning rate for shrinkage.In order to train the classifiers, the latent representations are retrieved from the training dataset. The classes of representations that were learned for the test dataset are then predicted using the trained models. The hyperparameters of the k-NN classifier are tuned while training machine learning classifier models with KTH, as shown in The overall training regime of the HARNet is described in this section. Video sequences are preprocessed to obtain the fusion of spatial and motion information. The input to the network is downsized to Three standard HAR datasets considered for experimentally evaluating the performance of the proposed work are HMDB51, UCF101, and KTH. The various features of the datasets, such as the number of video clips, frame rate, number of action categories, challenges, and variations in data capture are listed in HMDB51  is a larTo quantitatively assess the performance of the proposed network model, the standard evaluation metrics for assessing the accuracy of human action recognition are calculated for the overall model using Equation .(4)AccuThe other performance evaluation metrics of precision, recall, specificity, and F1-score are evaluated in order to compare the performance of different classification models, as well as to analyze the performance of the same model by varying different parameters . The perBy taking the macro average, which is the arithmetic mean of the performance metric over each class, the evaluation metrics are computed for multiple classes.The proposed network model was trained with preprocessed UCF101 video clips for categorizing 101 action classes. After learning the weights and latent representation of data, the features were extracted from the fully connected layer of the proposed network. The extracted features were used to train conventional ML classifiers for evaluating the efficacy of the network in learning latent representation. Human actions in the test data were predicted by the trained ML models with an average classification accuracy of 98.33% with LDA, 93.73% with NB, 93.04% with ensemble, 99.56% with DT, 99.98% with SVM, and 99.99% with KNN. A detailed classification report listing all of the standard multi-class performance metrics is given in Standard metrics such as accuracy, precision, recall, F1-score and specificity for the HMDB51 dataset are listed in The performance of the proposed approach had been experimentally evaluated on the untrimmed KTH dataset, which has six action classes. As this dataset contains frames with and without humans in the foreground, a preprocessing step of detecting people with the Gaussian mixture model was used. After detecting humans in the foreground, the frame was converted to a spatial motion cue, which was further processed by the network model for learning latent representation. The learned representation was used to train ML classifier models and then the test dataset was used to assess the prediction accuracy, precision, recall, specificity, and F1-score. The results shown in The ablation study was conducted to determine the importance of each layer in influencing the performance of the proposed network model. It was implemented by removing certain layers from the proposed model; the experimental results are listed in The efficacy of the proposed network in representation learning has been analyzed and compared with other existing methods on the three benchmark datasets of UCF101, HMDB51, and KTH. Since state-of-the-art (SOTA) methods often employ a data split of 70:30, the proposed method also uses this split in order to compare with SOTA techniques. The results of action recognition experiments using the proposed network features classified by k-NN are compared with earlier studies, and are listed in g method , this isIn this work, a representation learning model of HARLearning based on the information bottleneck principle is proposed. It was implemented by designing a novel directed acyclic graph-based residual CNN named HARNet. It was built by stacking convolution layers followed by a batch normalization layer and ReLU with max pooling layers after residual connection via (


Identifying human actions from video data is an important problem in the fields of intelligent rehabilitation assessment. Motion feature extraction and pattern recognition are the two key procedures to achieve such goals. Traditional action recognition models are usually based on the geometric features manually extracted from video frames, which are however difficult to adapt to complex scenarios and cannot achieve high-precision recognition and robustness. We investigate a motion recognition model and apply it to recognize the sequence of complicated actions of a traditional Chinese exercise . We first developed a combined convolutional neural network (CNN) and long short-term memory (LSTM) model for recognizing the sequence of actions captured in video frames, and applied it to recognize the actions of Baduanjin. Moreover, this method has been compared with the traditional action recognition model based on geometric motion features in which Openpose is used to identify the joint positions in the skeletons. Its performance of high recognition accuracy has been verified on the testing video dataset, containing the video clips from 18 different practicers. The CNN-LSTM recognition model achieved 96.43% accuracy on the testing set; while those manually extracted features in the traditional action recognition model were only able to achieve 66.07% classification accuracy on the testing video dataset. The abstract image features extracted by the CNN module are more effective on improving the classification accuracy of the LSTM model. The proposed CNN-LSTM based method can be a useful tool in recognizing the complicated actions. I.Identifying human activities from raw observation data has long been a question of great interest in intelligent rehabilitation, health surveillance, medical caregivers, training recovery and skill assessment Along this track, efforts have recently been made to recognize the actions in Yoga Traditionally, rehabilitation exercise therapy for chronic diseases is usually carried out in a formal rehabilitation center or clinical environment, and requires the direct supervision of professional physiotherapists In this literature, some human activity recognition systems based on wearable sensors have been developed Feature extraction and classification are two main procedures in video action recognition. Klishkovskaia et. al. On the other hand, thanks to the excellent performance of CNN in the field of image recognition and the outstanding results of long short-term memory network (LSTM) in modeling time series data, efforts have been paid to combine these two algorithms to build a video-based action recognition model In light of this, we propose in this work a motion recognition model based on LSTM and a popular CNN model, the VGG16 (VGG ConvNet configuration D) architecture proposed by the Visual Geometry Group In summary, there are two main contributions in this work. First, a video-based human action recognition model is proposed and especially trained to recognize Baduanjin actions, using the VGG16 and LSTM network. Second, the performance of the Baduanjin action recognition model based on the traditional manual feature extraction procedures is also established, and compared with that of the newly developed CNN-LSTM method.II.A.The experimental setup to collect and recognize Baduanjin practicers’ actions is shown in On the other hand, the traditional manual feature extraction based on the geometry of the detected skeletons is also considered for comparison with the CNN-based feature extraction. To this end, Openpose is used to identify the joint positions in the skeletons B.Traditional motion recognition technology mainly depends on manual feature extraction. The selection of features generally depends on the domain knowledge of experts More specifically, from the 18 key point coordinates tracked by Openpose, the key node positions and joint angles in each video frame are extracted and calculated. The detailed Baduanjin poses are described in Specifically, the initial skeleton data are preprocessed for the spatial alignment between a skeleton instance and its template counterpart. The coordinate system is established with the center of gravity of the human skeleton as the origin. The coordinates of each joint point in the human skeleton are calibrated. Denote the coordinates of the center of gravity of the human skeleton as Furthermore, through the geometric analysis of the poses of the human body with open arms facing the camera, the minimum circumscribed rectangle of the skeleton in a video frame can be calculated as follows:Then, the width Let Here, In summary, nine features including joint point positions or angles as shown in C.The proposed Baduanjin action recognition algorithm firstly uses the pre-trained VGG16 network to extract the feature vector sequence from every video frame collected by the RGB camera. The feature vector sequence is then sent to the trained LSTM model to realize the recognition of Baduanjin actions.More specifically, since the CNN does not support video inputs, each video clip is first intercepted and saved in the original frame rate by sparse down sampling. To see this, let a video clip be denoted as A CNN is generally composed of convolution layers, pooling layers, full connection layers and softmax layers. Likewise, VGG16 is built by repeatedly adding 3-by-3 convolution kernels and 2-by-2 pooling kernels to achieve feature extraction. The network can not only simulate the effect of larger receptive field and increase the number of feature map channels, but also achieve better extraction D.The movements of Baduanjin are dynamically arranged in a temporal order. To process these time series data, LSTM is a better option than general neural networks, due to its recurrent connections. More specifically, LSTM introduces memory units to expand a cyclic convolution network, controls the state and behavior of each memory unit with the concept of gate, and can thus establish the dependency between related information on a longer time scale. Therefore, the dependency of the sequential Baduanjin movements on each other can be captured by a LSTM network.The LSTM controls the memory unit through the input gate and forgetting gate, and combines the output gate to describe the long-distance dependence more effectively. The mathematical operations of the input gate, forgetting gate and output gate can be respectively written as follows:III.A.Since there is no public image dataset of Baduanjin for pattern recognition yet, we collected the videos of 18 people (11 men and 7 women) performing the entire routines of the eight poses individually. The original videos were in the MP4 format, with a resolution of The performed poses in the datasets varied greatly from person to person even for a same pose, due to the following reasons. First, the participants performed the same movements at different speeds in different repetitions. Second, the participants were of different heights and shapes. Third, the participants performed the routines in different scenes. These variations made the video data more complicated, and necessitated the robustness of the recognition method.B.The accuracy of the recognition method was calculated as C.In the experiment, an initial learning rate of 0.001 and a dropout rate of 0.5 were set to keep the training process relatively stable. After training for 200 epochs with 8 iterations per epoch, the Baduanjin action recognition model achieved 98.31% accuracy on the training data and 96.43% accuracy on the testing set. In On the other hand, for the method based on manually extracted motion features, as introduced in The classification performances on the testing set of the two Baduanjin action recognition models respectively based on the traditional manual feature extraction and CNN-LSTM are finally compared in IV.To deal with the complicated motions captured in a video format, we have proposed and tested in this work a combined CNN and LSTM model structure. The VGG16 convolution network is applied to extract the features of a pose snapshotted in every single picture. The time series of the pose features are then fed into the LSTM module, which models the dynamics of the poses in the video. The proposed CNN-LSTM method can thus model the complicated actions in the Baduanjin exercise.By training and testing the method on the video data set (3:1 division at the video level) recorded from 18 Baduanjin practicers, the proposed model managed to recognize the Baduanjin actions with the average success rate of respectively 98.31% on the training data and 96.43% on the testing set. The results have therefore verified the effectiveness of this method.The proposed CNN-LSTM method has been compared with the traditional method based on geometric feature extraction. For fair comparison, the time series of the multidimensional geometric features extracted from every picture in the video, including nine typical joint angles and position trajectories, are also processed by a LSTM module. However, it turned out in the testing that the method based on these manually extracted features was only able to achieve an average success rate of 66.07% on the testing video dataset, and always performed worse than the CNN-LSTM based method in recognizing all the eight different actions in Baduanjin. Through this comparison, the effectiveness of the VGG16 net in extracting the abstract motion features to uniquely represent the different poses in the video has been clearly verified.The proposed method still contains a few limitations. First, the convergence rate is not high. In the training experiment, with the initial learning rate of 0.001 and the dropout rate of 0.5, the accuracy and loss function converged after approximately 1500 iterations. Second, the method does not perform well for the different actions containing similar poses, indicating the sensitivity of the method needs to be further improved. The typical example is the action of “Touching the toes and bending backward”, which can be misclassified as “Propping up the heavens” or “The wise owl gazes backward”, because all these three actions first require rising arms up while standing still in the same fashion.On the other hand, a major advantage of convolutional neural networks over traditional methods is that they can automatically learn multi-scale information. However, the capability of representing multi-scale information by a VGG network is relatively limited compared with other recently improved CNN models, which may lead to the aforementioned limitation in the sensitivity of the proposed method. The examples of the improved CNN models include Res2Net and DenseNet, which can further enhance multi-scale representation through skip layer linking, and can thus outperform VGG networks in multi-scale feature expression V.In this work, a combined VGG16 type of CNN and LSTM model has been developed and investigated for identifying human actions from video data, which has a wide applicability in many newly emerging fields to improve human wellbeing, such as intelligent rehabilitation, health surveillance and sports skill assessment. More specifically, the method has been applied on recognizing the sequence of complicated actions performed by practicers of a traditional Chinese exercise . Its performance has been verified by the high classification accuracy on the testing video dataset. Moreover, this method has been compared with the LSTM model, which takes as inputs the positional and angular motion features extracted from the detected skeleton data via Openpose. The results have clearly demonstrated that the image features extracted by the CNN module are more effective than the geometric features on improving the final classification accuracy of the LSTM model. Therefore, the proposed CNN-LSTM based method can be a useful tool in recognizing the actions of Baduanjin and other similar exercises, and can whereby help the practicers improve their skills.A potential extension of the current work is to implement and test the proposed CNN-LSTM based method in other exercises with even faster movements, e.g., boxing and badminton. For these exercises, faster imaging solutions should be developed. Another direction of the future work is to further improve the sensitivity of the proposed model by incorporating the multi-scale representation functionalities of Res2Net, or to implement more advanced models, such as TS-LSTM.


Federated learning (FL) is an effective method when a single client cannot provide enough samples for multiple condition fault diagnosis of bearings since it can combine the information provided by multiple clients. However, some of the client’s working conditions are different; for example, different clients are in different stages of the whole life cycle, and different clients have different loads. At this point, the status of each client is not equal, and the traditional FL approach will lead to some clients’ useful information being ignored. The purpose of this paper is to investigate a multiscale recursive FL framework that makes the server more focused on the useful information provided by the clients to ensure the effectiveness of FL. The proposed FL method can build reliable multiple working condition fault diagnosis models due to the increased focus on useful information in the FL process and the full utilization of server information through local multiscale feature fusion. The validity of the proposed method was verified with the Case Western Reserve University benchmark dataset. With less local client training data and complex fault types, the proposed method improves the accuracy of fault diagnosis by 23.21% over the existing FL fault diagnosis. As a critical component of the motor, rolling bearings are susceptible to failure due to overload, aging, complex working conditions, and other factors ,2,3. WheAs an effective data feature extraction tool, deep learning can extract features from raw data collected by sensors without relying on information about the remaining lifetime of critical components, and achieve end-to-end fault diagnosis by classifiers, and without the constraints of an exact physical model. Therefore, deep learning-based fault diagnosis research has received attention ,7,8,9,10The amount of high-quality labeled data is a key factor limiting the effectiveness of deep learning fault diagnosis, but high-quality labeled data is difficult to obtain in the industry. When the operating load of the equipment changes, the statistical characteristics of the data collected by the monitoring sensors changes significantly, and this situation where the statistical characteristics change with the load is called multiple working conditions. The training samples collected from multiple working conditions violate the basic assumption of Independently Identically Distribution (i.i.d) required by the deep learning training mechanism, which affects the effectiveness of deep learning-based fault diagnosis and leads to unreliable fault diagnosis results. Therefore, it is necessary to carry out research on multiple working conditions fault diagnosis based on deep learning.In fact, it is difficult to obtain multiple working condition samples, and the multiple working condition samples from a single client are not enough to build an accurate deep learning multiple working condition fault diagnosis model since the effectiveness of deep learning depends on the amount of data. On the other hand, the working conditions of the data may be different from one client to another, and the data of the client is usually confidential. Therefore, it is a worthwhile research problem to develop an effective multiple working condition fault diagnosis method jointly with multiple clients without sharing client data directly.FL has received much attention from experts in the field of fault diagnosis in recent years due to its ability to accomplish collaborative training of multiple clients while ensuring data privacy. FL-based fault diagnosis aims to obtain a global model with satisfactory performance using information provided by multiple clients ,12,13,14FL does initial feature mining on private data through local clients before sharing the mined information to the server for aggregation. The model parameters, gradients, and features of the client can be uploaded to the server as information for aggregation. Traditional FL-based fault diagnosis methods aggregate the information uploaded by the client to the server by averaging ,16,17,18Remark 1.Useful information from the client refers to information that contributes positively to the fault diagnosis results, and useless information refers to information that contributes negatively to the fault diagnosis results.This paper designs a multiscale recursive attention gate federation method for multiple working conditions fault diagnosis from the perspective of inter-client federation aggregation and local client feature extraction, including inter-client multiscale recursive federation strategy, and intra-client multiscale recursive fusion strategy. When the local model of the client receives new information, it continues training on the original model, which makes the feature representation of the neural network more powerful and thus makes the network output features more distinguishable. The purpose of optimizing local client feature extraction by using features from other clients is thus achieved. The method proposed in this paper can obtain a more powerful multiple working condition fault diagnosis model since useful information is better focused on both local client feature extraction and federal feature aggregation.A multiscale recursive attention-gate federation for multicondition fault diagnosis is designed, which can improve the accuracy of the inter-client federation and optimize the local multiple working conditions fault diagnosis effect by making full use of other client features.A multiscale recursive federation strategy for inter-client and a multiscale recursive fusion strategy for local clients are designed. A multiscale recursive federation policy located at the server can focus more on useful information uploaded by the client and improve the accuracy of the global features of the server. The locally located multiscale recursive fusion strategy can make the local client features and global features fully fused to achieve the purpose of optimizing the local multiple working condition fault diagnosis effect by making full use of the information from other clients.When there are differences in working conditions between data from different clients, using the method designed in this paper, satisfactory accuracy of multiple working conditions fault diagnosis can be achieved.The contribution of this paper is as follows:Deep learning has received a lot of attention from experts in the field of fault diagnosis due to its ability to extract features from the raw signals monitored by sensors. Refs. ,7 pointeFL is a distributed machine learning approach where clients use private data to train local models and then aggregate them at the model level without involving data sharing. In this way, FL combines the information of multiple clients without sharing data, which protects the privacy of local client data. The schematic diagram of FL is shown in The above research method achieves collaborative optimization of multiple clients through different federation approaches, and the federation centers all use traditional federation averaging, giving the same weight to different clients. However, there are differences in the useful information provided by each client due to local data work and model performance, and Federated Averaging, (FedAvg) ignoring such differences will not fully exploit the advantages of FL. Therefore, this paper designs a multiscale recursive federation strategy among clients, which makes the server’s aggregation more focused on the useful information provided by the clients, and thus makes full use of the data from the clients. At the same time, the client local multiscale recursive fusion strategy is designed to fuse the global information into the local client feature extraction process, so that the local client features and global features are fully fused to achieve the purpose of using other client information to optimize the current client fault diagnosis effect.The purpose of FL-based fault diagnosis is to federate multiple clients to train a powerful global model, but the information uploaded to the server contains both useful and useless information when the local clients have different data conditions and model performance. The traditional FedAvg method ignores the differences in client information and assigns the same weight to different clients, which inevitably affects the fault diagnosis effect of all clients participating in the federation. Therefore, this section designs a multiscale recursive attention gate federation fault diagnosis method, which uses attention gates to give more attention to the useful information uploaded by the client, thus improving the accuracy of the client’s multiple working conditions fault diagnosis.Neural networks abstract features to higher scales through layer-by-layer feature representation, where features at different layers represent features at different scales. The feature representation of the same set of signals at different layers is different, and the distinguishability of features is also different.On the other hand, it is difficult to obtain multiple working conditions data from local clients, so it is especially important to develop a multiple working conditions fault diagnosis method jointly with multiple clients. The simplest way to jointly develop multi-client fault diagnosis models is to share data, but sharing data involves the privacy information of clients. Therefore, the FL approach is that the clients locally do preliminary feature mining on private data first and then share the mined information, thus enabling multiple clients to jointly develop fault diagnosis models for multiple operating conditions. However, existing FL approaches do not take into account the differences in information uploaded by clients due to differences in local data and model performance. Therefore, a multiscale recursive attentional gate FL model (MAGFL) is designed in this section to improve the accuracy of multicondition fault diagnosis for the federation’s later client. The algorithm steps are as follows:Remark 2.Shallow scale information is more comprehensive but coarser and less distinguishable. Deep-scale features are more accurate, but there is information loss. Therefore, the comprehensive use of multiscale features can improve the fault diagnosis accuracy of rolling bearings.Step 1:Designing a multiscale recursive FL framework between clients.The traditional FL method gives the same weight to the information uploaded by the client, which leads to the waste of useful information and thus affects the effectiveness of FL-based fault diagnosis. Therefore, a multiscale recursive FL framework among clients is designed as shown in n denotes the number of network layers. j-th layer. For multiple clients Then, Equation .(3)Ff,iEquation .(4)Fg,2n-th scale are obtained, and Equation .(5)Ff,iStep 2:Multiscale recursive fusion within the client.The local layer-by-layer recursive use of global features provided by the server can make the global features better serve the current client. The information flow relationship of the proposed method is shown in Remark 3.The features are essentially the outputs of the neurons in the hidden layer of the neural network. The mapping of information from data space to feature space is actually the transformation of the neural network from input to output. The reason is that the distinguishability of fault information in data space is not strong, and the purpose of transforming it to feature space by the neural network is to train the parameters of the neural network to make the distinguishability of fault information in feature space stronger.Remark 4.The client uses private data to train a local neural network model without requiring the sequential order of time series. In the process of FL, if the working conditions of each client are the same, then even the traditional FL method can obtain good fault diagnosis results. However, when it faces the problem of multiple working conditions, the fault diagnosis effect of the traditional FL method is not guaranteed. This paper focuses on fusing multiscale features of the client using a multiscale recursive federation approach, with attention gates used to focus on information useful to the client, thus solving the problem of multiple working conditions.t-th round of federation, the features For the By local multiscale recursive fusion of Then, Equation .(8)Ff,iIn such a way, the client locally performs n times of multiscale recursive fusion to obtain the top-level fused features In fact, different types of failures may occur when the client is working under different working conditions. The role of the attention gate is to selectively utilize the multiple working conditions information according to the local needs of the client. When different clients do not learn the same fault type information, the local client’s attention gate selects the information that is useful for their own fault diagnosis. The performance of the models differs from client to client when the length of the training set varies, which requires more attention to useful information, so attention gates are crucial to ensure the effectiveness of federal learning.When the clients participating in the federation are running in different working conditions, they receive different useful information from the client to the server due to the difference in the quality of the client data. Traditional FedAvg methods ignore this difference in information and unselective averaging can lead to the propagation of useless information through the federation process, which does not guarantee the validity of the FL. The MAGFL-based fault diagnosis method is given in this section, and its flowchart is shown in Remark 5.Traditional criteria for evaluating data quality include whether the data contains noise, whether there is an imbalance in the data, and whether the data is under network attack, etc. In deep learning, data quality is good if the data are conforming to independent homogeneous distribution. Because the data conform to independent homogeneous distribution is a prerequisite for the effectiveness of deep learning, and the effective features can be extracted by neural networks. If the data does not conform to the independent homogeneous distribution, then the quality of the data is poor.As shown in This section validates the effectiveness of the proposed method through experimental analysis using a benchmark dataset from Case Western Reserve University (CWRU) .The CWRU vibration data used in this section is collected by an accelerometer mounted on the drive side of the motor with loads including 0, 1, 2, and 3 HP. The EDM technique was used to reintroduce single-point faults on the test bearings with fault sizes of 0.007, 0.014, and 0.021 inches. Vibration data are collected by an accelerometer, which is fixed to the drive and fan side of the motor housing using a magnetic base. The fault types and labels for the data used in this section are listed in In order to verify the effectiveness of the proposed method, this section designs multiple working conditions fault diagnosis experiments, as listed in Remark 6.This paper starts the experimental verification with four clients, each containing one kind of load as an example. In fact, the equipment is in different stages of the life cycle, when the normal characteristics of the equipment will change, which is also a kind of multiple working conditions, and in addition, when the load or the working environment changes, the statistical distribution characteristics of the data will also change.This section uses a sliding window to process the raw vibration data to obtain the samples used in the experiment, with a window length of 900 and a sliding step size of 20. The sampling frequency of the data is 12 khz and the sampling period is 1/12,000 s. The length of the sliding window is 900 sample periods for 3/40 s. The sliding step is 20 sample periods for 1/600 s. If the sampling period is too small, the variability between two adjacent samples is small and can almost be regarded as the same sample, which contains less distinguishable information and will affect the results of fault diagnosis. This paper uses a DNN model with 3 hidden layers as the backbone model, the number of hidden layer neurons is 1200/1000/500, and the learning rate is 0.001. Local training is completed when the local training loss is less than the threshold, and the features are uploaded to the server with a loss threshold is 0.00001. The number of rounds in the federation is 100 rounds.The running environment for the experiments is Python 3.7.11, Tensorflow 2.3.0 GPU. The computer is configured with an 11th generation Intel(R) Core(TM) i9-11900K 3.50 GHz, and it is manufactured by Intel Corporation, United States. and a NVIDIA GeForce RTX3090 GPU manufactured by NVIDIA in the United States. The operating system is Windows 10 Home.Different quality of data from clients provides different useful information to the server, and focusing on the useful information provided by clients will affect the effectiveness of federal learning fault diagnosis. The experimental results are listed in Comparing the mean fault diagnosis accuracies of DNN and MCNN in Comparing FedAvg and BNCNN in However, the above approach does not take into account that the information provided by the client contains both useful and useless information, and the direct average aggregation approach does not take into account the propagation of useless information. At the same time, the client in the above method directly inherits the information of the global model as the initial value of the local optimization, without fully integrating the global information with the local information. The proposed method locally uses multiscale recursive feature fusion to utilize global information and fuses global features into the local client feature extraction process, which makes more full use of global information, so the multiscale recursive federation between clients and the local multiscale recursive feature fusion makes the proposed method achieve 99.09% fault diagnosis accuracy, and the fault diagnosis accuracy of each client is higher than existing methods.Since the proposed approach allows the federation model to focus more on the useful information provided by the client, Experiment 2 and Experiment 3 were designed with a small amount of data from the client, and the results are listed in Comparing This helps us to conclude that the proposed approach is clearly superior to existing methods, especially when the available data on the client side is relatively small, as the utilization of useful information from limited data becomes particularly important at this point. In order to improve the readability of the experimental results, the confusion matrix of the experimental results of Experiment 3 is given in this section as shown in Comparing To verify the effectiveness of the proposed model when the fault diagnosis task is more complex, experiments 4–6 were designed by increasing the fault types, and the fault diagnosis results when the fault types are ten classes and the number of training samples for each class is 1000 are listed in For a certain client, the complexity of the fault diagnosis task means that the types of faults to be diagnosed increase, but the number of samples in the training set remains the same, which will lead to a more difficult task of fault diagnosis. Therefore, comparing Comparing Comparing This paper takes Experiment 1 as an example to analyze the variance and standard deviation of the fault diagnosis accuracy of each model. Each model performs fault diagnosis 20 times, and the variance and standard deviation of the fault diagnosis results are calculated from the fault diagnosis results 20 times. The experimental results are shown in From The purpose of FL is to accomplish joint optimization of multiple clients without sharing data directly, and the information uploaded to the server by each client is given the same aggregated weight. However, the information uploaded to the server by each client is not equal due to the change in working conditions, and directly assigning equal weights to all clients’ information will ignore the attention to useful information and affect the effect of federal learning fault diagnosis for multiple working conditions. Therefore, the multiscale recursive FL framework aims to make the server pay more attention to the useful information provided by the model and ignore the useless information through attention gates to ensure that the information provided by the client is fully utilized during each federation. On the other hand, for the global information aggregated by the server, the client uses the designed multiscale recursive fusion, which can effectively fuse the global information into the local client feature extraction process and achieve the effect of using other client information to optimize the current client information. The designed multiscale recursive federation among clients and the multiscale recursive fusion method locally by clients can complete the joint optimization of each client in the case of obvious differences in client work conditions, and make full use of the useful information provided by clients. The experimental results show that the proposed method is 23.21% higher than the existing FL fault diagnosis method when the client faces a complex multiple working condition fault diagnosis task and the available training data is relatively small. The experimental results show that the use of the proposed method can give more attention to the useful information provided by the client in the process of federation.This study used a small amount of labeled data from a single customer and did not consider the large amount of unlabeled data collected during the operation of the device. Therefore, future work should design a semi-supervised federal learning mechanism for the case of small amounts of labeled data.