Deep learning algorithms have the advantages of a powerful time series prediction ability and the real-time processing of massive samples of big data. Herein, a new roller fault distance estimation method is proposed to address the problems of the simple structure and long conveying distance of belt conveyors. In this method, a diagonal double rectangular microphone array is used as the acquisition device, minimum variance distortionless response (MVDR) and long short-term memory network (LSTM) are used as the processing models, and the roller fault distance data are classified to complete the estimation of the idler fault distance. The experimental results showed that this method could achieve high-accuracy fault distance identification in a noisy environment and had better accuracy than the conventional beamforming algorithm (CBF)-LSTM and functional beamforming algorithm (FBF)-LSTM. In addition, this method could also be applied to other industrial testing fields and has a wide range of application prospects. In recent years, due to the increasing emphasis on energy conservation and emission reduction, straw power plants have developed rapidly. As an important piece of equipment for conveying materials, the safety of idlers in belt conveyors has a crucial impact on the smooth operation of the entire conveying system. As the key support and rotating component of the belt conveyor, the chain reaction caused by the failure of the idler poses a safety threat to the entire straw power station. The long-term operation, substantial conveying distances, and huge weight of idlers in harsh working environments necessitate daily inspections and regular maintenance that not only entail significant time and money costs, but also allow for some seriously degraded parts to be overdue for replacement. Therefore, it is of great significance to design non-contact acoustic signal measurement and fault distance estimation methods suitable for belt conveyor idlers in power stations, so as to reasonably arrange maintenance activities; reduce time, manpower, and material costs; and ensure the reliability and safety of mine belt conveyor operation.In the field of idler bearing fault diagnosis, a method of energy collection was proposed in  that wasWith the acceleration of industrial processes, the safety of machinery and equipment has become the top priority, and methods for the fault diagnosis and condition detection of machinery and equipment are also constantly improving and upgrading. In recent years, the use of deep learning to diagnose bearings has also become a popular research direction. Zhou proposed a novel combination of discrete probabilistic entropy-based health indicators (HIs) and long short-term memory (LSTM)-based methods to predict bearing health. The LSTM model included in the proposed method could accurately predict bearing health . AljemelBeamforming suppresses signals in non-selected directions and enhances signals in selected directions by merging the signals collected by the microphone array; in addition, it can realize focused pickup in the specified direction, which can effectively improve the signal-to-noise ratio of the received signal. As beamforming algorithms have become more popular, they have also played an increasingly important role in the field of sound source identification ,9,10,11.Efficient fault feature extraction: The MVDR algorithm can effectively extract fault signal features. Due to its distortion-free nature, the MVDR algorithm can minimize noise and interference while retaining the fault signal, thereby improving the accuracy of fault detection.Excellent spatial resolution: In terms of fault source location, the high spatial resolution of the MVDR algorithm can provide more accurate fault source location information.Adaptability: In complex industrial environments, troubleshooting conditions can change. As an adaptive algorithm, MVDR can automatically adjust parameters according to changes in the environment, so as to effectively diagnose faults under various conditions.Robustness: The MVDR algorithm has good robustness for independent and uniformly distributed noise signals, which allows it to maintain good fault diagnosis performance in various noise environments.Subramanian et al. used a deep learning neural network to identify and locate multiple sound sources on the basis of direction of arrival estimation, and experimental results showed that the proposed method could classify DOAs with high resolution . ConsideThe combination of deep learning and beamforming algorithms can better solve some tricky problems and has unique advantages in accelerating fault diagnosis efficiency and improving fault diagnosis accuracy, which not only reduces manual operation, but also makes mechanical equipment run more safely and smoothly. In order to solve the problem of the harsh working environment and serious noise interference of idlers, an MVDR-LSTM fault distance estimation model with a diagonal double rectangular array is proposed. This model can greatly suppress the interference of noise and non-coherent sound sources and realize the accurate estimation of the failure distance of idlers. At the same time, compared with the CBF-LSTM and FBF-LSTM models, our model has better robustness.This article will first explore the relevant theory and historical background of the MVDR-LSTM roller fault distance estimation model. Secondly, the methods and processes adopted in this study will be introduced, including the specific steps of data collection and analysis. Then, we will elaborate on our research results and attempt to provide a deeper understanding of the MVDR-LSTM idler fault distance estimation model by comparing and interpreting these results. Finally, we will summarize our findings and explore their potential impact on the field of fault detection, as well as future research directions.A microphone array is an advanced technology used to capture sound signals that consists of multiple tiny microphones and can receive sound signals in different directions. Microphone arrays provide a more accurate and clear sound signal than a single microphone, because they suppress ambient noise and echo and can focus on receiving sound in different directions. Microphone arrays are widely used in many fields, such as video conferencing, speech recognition, and audio signal processing. In video conferencing, microphone arrays can provide clear voice signals, improving the efficiency and quality of meetings. In speech recognition, microphone arrays can improve the accuracy of speech recognition, because they reduce the impact of ambient noise. In audio signal processing, microphone arrays can realize functions such as spatial filtering and sound source localization to improve the quality of audio signals. As a result, microphone arrays have become an important part of modern communication and audio technology, and they are constantly improving our daily lives and work experiences .The formation of a microphone array refers to the arrangement of microphones, and different formations can be adapted to different application scenarios. The common forms of microphone arrays are shown in Line array: Multiple microphones arranged in a straight line. This array is suitable for scenarios wherein the sound source is moving horizontally.Cross-shaped array: An array that positions multiple microphones in a cross-shaped manner, which is suitable for sound source localization and speech recognition scenarios.Rectangular array: Multiple microphones arranged in a rectangle. This is suitable for scenarios wherein sound sources move horizontally and vertically.U-shaped array: The microphones are arranged in a special U-shape, suitable for high-quality language recognition. For example, Translator 4.0 developed by iFlytek was paired with a U-shaped microphone array, which helped improve the accuracy and quality of sound capture. L-Array: Multiple microphones arranged in an L-shape. This array is suitable for scenarios wherein sound needs to be captured within a specific area.The circular microphone array positions multiple microphones in a circle and has the following benefits: (1) 360-degree omnidirectionality. The circular microphone array can achieve 360-degree omnidirectionality in the horizontal direction and can evenly receive sound from all directions without mechanical rotation or adjusting the orientation of the microphone. This is useful for applications that need to capture the sounds from the whole environment, such as speech recognition, voice conferencing, and voice commands. (2) Eliminating noise and echo. The circular microphone array can use the delay difference and amplitude difference between multiple microphones in the array to achieve spatial sound source localization and sound source separation, so that the influence of noise and echo can be effectively eliminated. This is very helpful for collecting clear speech signals in noisy environments, thus improving the performance of speech recognition and speech processing ,18.The diagonal double rectangular array proposed in this article consists of two rectangles. The distance between the adjacent microphones is equal, and the centers of the two rectangles are staggered from each other, forming a diagonal line. The advantages of diagonal double rectangular arrays include: (1) The improved accuracy of sound source positioning. Since the microphones in a diagonal double rectangular array are staggered, more sound direction information can be captured, thereby improving the accuracy of sound source positioning. (2) Reduction in echo and noise interference. The diagonal double rectangular array can use the signals of multiple microphones for noise reduction processing, thereby reducing the interference of echo and noise. This is useful for speech recognition and sound source localization in some noisy environments. (3) Supporting spatial filtering. The diagonal double rectangular array can use the signals of multiple microphones for spatial filtering processing, thereby improving the quality of voice signals. (4) Supporting multi-sound source separation. The diagonal double rectangular array can realize the separation of multiple sound sources by analyzing the time domain and frequency domain characteristics of multiple microphone signals. This is of great benefit for idlers working in environments with complex sounds. In general, the diagonal double rectangular array has unique advantages in improving the positioning accuracy of sound sources, reducing echo and noise interference, supporting spatial filtering, and supporting multi-source separation. Hence, the diagonal double rectangular microphone array was selected as the acquisition array for idler fault distance signals.0f according to the frequency band division method of 1/3 octave bandwidth, the upper frequency of the 1/3 octave band Beamforming technology and signal space wavenumber spectrum estimation are the two main research directions of free space signal array processing. In order to maximize the array gain, Capon proposed the MVDR beamformer, also known as the Capon beamformer, in 1969. MVDR is an adaptive beamforming algorithm based on the maximum signal-to-noise ratio (SNR) criterion. The MVDR algorithm can adaptively minimize the power in the desired direction of the array output and maximize the signal-to-drying ratio. It can greatly improve the resolution and suppress noise in spatial spectral estimation ,21. Thisnth element signal at the receiving end isThe specific operation steps of MVDR are as follows. Suppose the desired target location P’ in space is In this formula, the direction of the receiving guide vector When the constraints are also satisfied, in order to minimize noise, the result of the objective function optimization isThen, the MVDR weight optimization problem can be expressed asThe essence of the MVDR beamformer is to solve the weight coefficients of each array. Then, using the Lagrange multiplier yieldsAfter deriving Equation (6), let it be 0, which can be obtained asThen, it is solved as follows:According to the MVDR criterion, the optimal value of the array weight is obtained:t, Recurrent neural networks (RNNs) are robust deep neural networks with high performance that process serial data of varying lengths for end-to-end distribution . Long shf is a nonlinear activation function; m is the sequence length; l is the hidden size.In this formula, t of the storage unit tc. The efficiency of the storage unit is determined by three gates: the input gate ti, output gate to, and forgetting gate tf. The upgraded equation can be expressed as:sigm is an S-type function; tanh is the hyperbolic cut function; and operator A diagram of the LSTM unit is shown in t and comprise the following three steps: (1) From the new input functions tx and early hidden state t−1h, one can obtain the forgetting gate tf, whose values range from 0 to 1. When the value of tf reaches 1, the last memory cell t−1c of incoming data is strongly preserved. On the other hand, when the value of tf reaches 0, the incoming data are distributed and cannot be retained. (2) Input gates can be removed from the new input functions tx and early hidden state th and added to the memory cell, specifying tc. (3) Therefore, the output gate must select an item from the memory cell that can be processed to form a new hidden state th [The main operating points of the LSTM unit are defined according to each phase of the time state ht ,27,28,29h, c also begins to be processed, forming a complete LSTM model.The overall LSTM model is shown in The MVDR algorithm was simulated by MATLAB, and the parameters set are shown in The simulation results of the microphone array are shown in The LSTM model consisted of one input layer, two hidden layers, one fully connected layer, and one output layer. When the hidden layer was set to one layer, the effect achieved was not ideal. After increasing it to two layers, the demand was already met. When increasing it to three layers, the calculation time of the model increased. Therefore, two hidden layers were used. The accuracy of the model with different numbers of hidden layers is shown in The diagonal double rectangular microphone array was used as a sound signal acquisition device. The sampling frequency was 16,000 Hz, the number of sampling points was 1024, and the device consisted of 8 MEMS microphones and microphone holders. In the experiment, five positions were selected as sound source points. The distance between each sound source points was 0.5 m, and the microphone array was 1 m away from the ground and 1 m away from the central sound source, as shown in the schematic diagram in Step 1—The microphone array was used to collect the idler fault audio signal.Step 2—MVDR processing was performed on the collected signal to generate idler fault distance data.Step 3—We processed the data again to build the dataset.Step 4—We divided the data into the training set (90%) and testing set (10%).Step 5—We trained and tested the model to draw conclusions.The flow chart of the experiment is shown in White noise was used as background noise to collect the sound signal of the idler, and the waveform diagram obtained is shown in After beamforming the acquired sound signal, the processed data were passed to the LSTM model. The data samples were processed 100 times, and the number of data (samples) passed to the program for training at a time (batch_size) was 16. The accuracy and loss rates of the training set and the test set were observed as the model performance, and the accuracy of the training and test sets of the MVDR-LSTM model gradually converged after 65 iterations, with the accuracy remaining at 100% for both. The loss rate remained steady after 65 iterations as the number of iterations increased, and the results are shown in Under the same experimental conditions, the background noise was replaced with irregular impact interference noise. The obtained waveform is shown in Similarly, 100 iterations were performed on the newly generated data samples, and the number of data (samples) passed to the program for training (batch_size) was 16. The accuracy and loss rates of the training and test sets are shown in In order to verify the superiority of the proposed model, the diagonal double rectangular array was used as the acquisition device, and the MVDR-LSTM, CBF-LSTM, and FBF-LSTM models were used as the comparison models. We set the number of data (samples) passed to the program for training at a time (batch_size) as 16, the activation function was Softmax, and the optimizer was Adam. Under background noise interference, the accuracy of the three models, the confusion matrix classification results, and KPCA dimensionality reduction analysis were used as evaluation criteria. The accuracy results obtained over 10 experiments are shown in We used a confusion matrix to compare the classification results with the actual measured values and displayed the accuracy of the classification results in the same confusion matrix. As shown in The KPCA dimensionality reduction diagram of the idler fault distance data sample is shown in The MVDR-LSTM, CBF-LSTM, and FBF-LSTM models in the circular array were selected as comparison models. We set the number of data (samples) passed to the program for training at a time (batch_size) as 16, the activation function was Softmax, and the optimizer was Adam. Under background noise interference, the accuracy of the three models, the confusion matrix classification results, and KPCA dimensionality reduction analysis were used as evaluation criteria. The accuracy results obtained over 10 experiments are shown in The confusion matrix of the roller failure distance data sample of the circular array is shown in The KPCA dimensionality reduction diagram of the idler fault distance data sample obtained by the circular array acquisition signal is shown in Through the comparative analysis of the above methods, we observed that the effect of acquiring signals using a diagonal double rectangular array in a noisy interference environment was better than when using a circular array, and the MVDR-LSTM model presented substantial advantages in all aspects compared with the other five models, so the MVDR-LSTM model was selected as the fault distance estimation model.Due to the special structure of an idler, the human diagnosis steps are cumbersome and time is wasted. Based on deep learning, our model removed the tedious step of manually extracting idler fault features, which could greatly shorten the time required and improve the efficiency of idler fault diagnosis.Through simulation experiments, we found that the diagonal dual rectangular microphone array had higher resolution and noise resistance compared to the other microphone arrays and could be beneficial to the estimation of roller fault distance.After adjusting the model structure and parameters, we trained the generated idler fault distance samples. The accuracy of the proposed model was 100%, with better performance than the other models and results closer to the true values.Five fault locations during idler operation were analyzed, and the experimental results showed that the proposed model had the ability to estimate the fault distance and provide better robustness. It could be used as a standard for judging the fault distance of idlers and provide ideas for combining beamforming algorithms and deep learning with idler fault diagnosis.In this paper, an MVDR-LSTM model for idler fault distance estimation was proposed. The model was tested in two different interference environments, and the results were compared with five other models to reach the following conclusions:The main advantage of LSTM is its ability to learn and remember long-term sequence dependencies. However, compared to CNNs, LSTM models have higher computational complexity because they contain gating mechanisms and time-step dependencies, resulting in greater computational costs in the training and inference process, which may require a longer training time and higher computing resources. Thus, it is necessary to continue to optimize the model in future research to reduce the computational costs and shorten the computing time.


Super-resolving the coarse outputs of global climate simulations, termed downscaling, is crucial in making political and social decisions on systems requiring long-term climate change projections. Existing fast super-resolution techniques, however, have yet to preserve the spatially correlated nature of climatological data, which is particularly important when we address systems with spatial expanse, such as the development of transportation infrastructure. Herein, we show an adversarial network-based machine learning enables us to correctly reconstruct the inter-regional spatial correlations in downscaling with high magnification of up to 50 while maintaining pixel-wise statistical consistency. Direct comparison with the measured meteorological data of temperature and precipitation distributions reveals that integrating climatologically important physical information improves the downscaling performance, which prompts us to call this approach  To prevent an irreversible collapse of the current ecosystem and resulting impoverishment of human lives, many countries have set specific medium- and long-term goals for the reduction of greenhouse gas emissions, and similar paradigm shifts in decision making have occurred even at the private sector level.The increase of greenhouse gases in the air composition due to human activities is now believed to have led to the rise in the frequency of unusual disasters5. Such future projections with high accuracies rely on the overall consideration of the global atmospheric and oceanic circulation 12, and thus, the horizontal spatial resolution is sacrificed by the required computational costs; the typical resolution of the GCMs is only down to the order of 117, such as transportation infrastructure development and sustainable energy networks, future urbanization, and agricultural intensification.Numerical approaches are regarded as the most powerful and reliable scientific option at the moment in quantitatively evaluating the efficacy of political or management plans that aim to tackle climatological issues. The Global Climate Model (GCM) is the prime example, which has accurately reproduced past and current climate changes, and its reliability of quantitative future estimates is sufficiently highdownscaling (DS) methods in meteorology and climatology, have been developed25. They are categorized roughly into two groups: dynamical21 and statistical DS methods25. The dynamical downscaling method is based on physical footings: several coupled differential equations are numerically integrated with the results of the GCM (or any other crude-resolution simulation results) being used as the boundary conditions. However, the computational cost again creates a trade-off between the accuracy and the feasibility. In contrast, in the statistical approaches, we turn a blind eye to the physical laws behind the data. Instead, empirical links between the large- and local-scale climates are identified and applied to the crude-resolution climate model outputs. Since the systematic errors of the naively interpolated GCM output (referred to as the bias) are locally corrected such that the statistical properties are precisely reproduced, the spatial correlation, i.e., the information on the events occurring at distant places, is discarded28. The statistical downscaling methods overcoming the latter problem remain to be developed.A variety of techniques to super-resolve GCM outcomes, which are referred to as the 32, our method is based on the generative adversarial network (GAN) approach, which has been proven to be a very powerful downscaling tool through several previous studies34. To accurately reproduce the physical nature, we use auxiliary but climatologically important data, sea-level pressure distribution and topographic information, in addition to the target variables, temperature and precipitation distributions  as the basic machine learning algorithm, which was proven to have potential in DS with a scale factor up to 5042. Among them, we employed the sea-level pressure, one of the fundamental hydrodynamic (or aerodynamic) variables on which the various quantities of sub-models of GCMs are based, as a piece of key auxiliary information. Also, this variable is described with fewer assumptions in the models than other meteorological variables such as humidity. In the literature, strong links between synoptic-scale horizontal circulation and vertical motion are discussed in terms of the sea-level pressure45. In the first variant, we incorporate the low-resolution pressure field as an auxiliary physical information  data from 1980 to 2018We use the data from 1980 to 2018 . These data are split into training, validation, and test datasets in a time-series manner as summarized in Table We first present typical qualitative visualizations for the temperature and the precipitation fields of one day in Fig. 17 are recognized to be advantageous in reproducing such fine structures. The maximum precipitation values of the DS results are all very close to that of the GT. Please refer to Fig. The difference in the downscaled temperature from the ground truth is not very large . The results are shown in Fig. We first measure the probability distribution functions (PDFs) of the precipitation data at 12 representative sites, p. Moreover, surprisingly, even Overall, Fig. To investigate the difference in the performance of oka Fig. D where tl:As another meteorologically important statistical point of view, we further measure the statistics over all sites: the PDFs of the mean Remarkably, regarding the statistics of pixelwise average over all dates in the test dataset 17, e.g., in the context of impact assessment of climate change. However, conventional DS methods such as CDFDM have proven to overestimate the correlation even though the statistical consistency with the GT is maintained28. Such a tendency is actually seen in the qualitative visualizations in Fig. l and k-th sample at site l from its reference average value M indicates that the average is taken over the data of month M, the superscript M. Since the distribution of the correlation coefficients is known to have features specific to each month, we measure the monthly values of the coefficients. Below, we focus on the results for 28.Next, we examine in detail the spatial correlation of the downscaled results. The importance of the spatial correlation of the meteorological variables, i.e., the relation between two distant sites, has been realized very recentlyl (the locations of the reference points are marked by the star symbols). The correlations measured for the CDFDM are too high compared to the GT at almost all sites, as shown in Fig. Figure l is the reference site and 33. If we further compare the results of SRGAN-based methods, although P(p)) while maintaining the expression ability of the natural spatial expanse. Such a strong downscaling ability highlights the applicability to local-scale and interregional assessments of climate change.To quantify the accuracy of 35 on the basis of the SRGAN-based method, and specifically, we developed a framework to use the proper auxiliary physical information along with the low-resolution input to attain large improvements in the DS performance as summarized in Fig. 28. Since the importance of the multiregional spatial correlation has recently been recognized17, the present method is a promising new-generation alternative to conventional statistical DS methods, particularly in situations where the integration of the multiregional information is necessary.We have developed a machine learning-based statistical downscaling (DS) method with a large scale-factor of 50, while maintaining both the basic statistical properties and the spatial correlation. We employed a physics-informed type approachinter alia in the context of climate change assessments. The methods including the present 35. If we succeeded in directly incorporating some part of the governing equations into the learning process while maintaining the computational efficiency, local phenomena such as heavy rains would be predicted with high reliability. Another direction is to take measures to reform the basic machine learning architecture itself. Following the GAN-based approach, flow-based and diffusion model-based methods have attracted public attentions as powerful next-generation tools for general super-resolution tasks47. The main feature of these approaches is to generate multiple image candidates from a single input. Therefore, probabilistic information is expected to be drawn from the multiple super-resolved images, which would enable us to tackle the rare event predictions.The detection and prediction of rare events are vital issues 48 the Gaussian process is used to reduce the computational burden of multiscale simulation for polymeric liquid to achieve a reduction by a factor of 30-100 without loss of accuracy. Breakthroughs driven by similar approaches are expected once the complexity of the governing equations for the climate models is overcome.Another perspective concerns the use of machine learning techniques to improve the efficiency of dynamical downscaling, i.e., developing a high-speed machine-learning-based solver for the governing equations of climate models. Here we refer to an example of a speed up of multiscale simulations; in Ref.Finally, we refer to the generalization ability of SRGAN. Here, we have selected SRGAN instead of Among a variety of statistical methods, we use, as a reference, the cumulative distribution function-based downscaling method (CDFDM) with quantile mapping that is in actual use.38.If we simply map the low-resolution GCM simulation results onto the point at which the observations are available, we generally see a systematic difference, defined as bias, which comes from the systematic error of the model prediction and/or from the interpolation error. Removing this inherent bias is especially important in applying the downscaling results to the impact assessments. In the CDFDM, bias is corrected via an empirical transfer function constructed in advance using measured data of distributional variables and the corresponding simulation results. The detailed procedure of constructing the transfer function is described as follows38, to more sensitively capture the seasonal trend50.The crude low-resolution data obtained from the GCM are first mapped onto a 2 km mesh using simple bilinear interpolation. At each mesh point, an empirical cumulative distribution function (CDF) is then constructed using the interpolated data of the variable of interest over a specified time window. The transfer function is defined as a map of a variable onto the one at which the corresponding CDF of the observation falls within the same quantile level. This preconstructed transfer function is applied under the assumption that the error-percentile relation is conserved over time. In the present study, the time window of a month is employed, while the original time window is over a half-year52.Note that while this CDFDM is a nonparametric method, the corrected CDF perfectly matches the corresponding CDF of the observation (for the training data); the statistical properties of the downscaling results are expected to reproduce the observation well. The bias-corrected climate scenario obtained with this method has been widely used in climate change impact studies53. The terminology super-resolution  refers to a method of restoring a high-resolution image from the corresponding low-resolution data and is the counterpart of the downscaling in the realm of the general image processing. The GAN-based methods are capable of generating realistic images by pitting a discriminator network against a generator network that generates samples  method as the basic machine learning architecture, which is called Super-Resolution Generative Adversarial networks (SRGAN)see Fig. A. The di33, the authors compared the performances of SRGAN-based downscaling methods with previous methods . Although the SRCNN-based method appeared to be superior in evaluating the performance in terms of the simple pixelwise MSE, the SRGAN-based method provided realistic results satisfying the important physical requirements, e.g., the energy spectrum of the wind velocity field satisfied the Kolmogorov 5/3 scaling law39 with remarkable accuracy. The network architecture in our 53, although the batch normalization layers are removed obeying Ref.33: the explanation of the precise architecture is presented in Such network-based super-resolution techniques have recently been used for the DS tasks of climatological data. In a representative report by Stengel and coworkers, Ref.Supplementary Information.


Fast convergence routing is a critical issue for Low Earth Orbit (LEO) constellation networks because these networks have dynamic topology changes, and transmission requirements can vary over time. However, most of the previous research has focused on the Open Shortest Path First (OSPF) routing algorithm, which is not well-suited to handle the frequent changes in the link state of the LEO satellite network. In this regard, we propose a Fast-Convergence Reinforcement Learning Satellite Routing Algorithm (FRL–SR) for LEO satellite networks, where the satellite can quickly obtain the network link status and adjust its routing strategy accordingly. In FRL–SR, each satellite node is considered an agent, and the agent selects the appropriate port for packet forwarding based on its routing policy. Whenever the satellite network state changes, the agent sends “hello” packets to the neighboring nodes to update their routing policy. Compared to traditional reinforcement learning algorithms, FRL–SR can perceive network information faster and converge faster. Additionally, FRL–SR can mask the dynamics of the satellite network topology and adaptively adjust the forwarding strategy based on the link state. The experimental results demonstrate that the proposed FRL–SR algorithm outperforms the Dijkstra algorithm in the performance of average delay, packet arriving ratio, and network load balance. Terrestrial communications already meet much of our daily communication needs, but for users in remote areas and at sea, their communications needs cannot be met. Unlike terrestrial communication, satellite communication has a longer range and better communication quality . It meetBased on the distance of the orbital plane from the Earth’s surface, satellite systems can be classified as geostationary Earth orbit (GEO), medium Earth orbit (MEO), and low Earth orbit (LEO) satellite systems. Unlike GEO and MEO, LEO satellites are usually located in the orbital plane, at an altitude of 500 to 2000 km [Typically, the routing problem of the LEO satellite is mainly divided into two parts: how to obtain the network topology, and how to generate the route paths based on the network topology . Mauger There has been a significant amount of research on the application of Software Defined Network (SDN) technology to address satellite routing challenges. For example, a study conducted in  investigThe above approaches mainly consider how to shield network dynamics and then run traditional routing algorithms for static network topology. These methods take up a certain amount of storage space and do not yield accurate satellite network topology. In recent years, a lot of research has started to use machine learning methods to solve routing problems. The difference between machine learning methods and traditional methods is that the former are data-driven, while the latter are model-driven. If the model does not describe the problem accurately, the performance of the model-driven method will be poor. Recently, machine learning has been applied in network areas, such as regulating congestion at the transport layer , optimizThe most suitable machine learning method for solving routing problems is the reinforcement learning technique. In , they usWe propose a distributed reinforcement learning method named FRL–SR; it has lower end-to-end latency and lower signaling overhead than traditional algorithms;We propose a learning mechanism that allows the algorithm to converge faster in the face of changes in the network link state;We compare the impact of different reward functions on the performance of reinforcement learning algorithms. The experimental results show that the design of reward functions is crucial for reinforcement learning algorithms.Even though many researchers have applied reinforcement learning to routing problems , few of The remainder of this paper is organized as follows. In In this section, we first give a model of the LEO satellite constellation and its characteristics, based on which research scenario is depicted. After that, we describe the traditional Q learning algorithm and its application in routing problems. The definition of the routing problem is given in the last sub-section.Currently, LEO satellite constellations can be divided into two types based on the number of satellite orbital planes: single-layer and multi-layer constellations. The Iridium system is representative of single-layer constellations, while Starlink is a multi-layer constellation with satellite orbital planes mainly distributed between 300 and 600 km. To facilitate the analysis, this paper uses the Iridium system as the satellite environment.As shown in Reinforcement learning is one of the areas in machine learning . Inspires represents the current state of the agent, a is the action performed, r is the reward value by performing action a under state s, A is the action space in state Q value of state Q learning is a traditional reinforcement learning algorithm; it provides the method by which the intelligence chooses actions in a given state by maintaining a Q-table. Each Q value in the Q-table represents the total benefit of taking a certain action in a certain state. The update of the Q value is mainly realized through the Bellman Equation:i, j represent the index of different nodes. This equation is the essence of agent learning.Q routing is the application of the Q learning algorithm to the routing problem. In the Q routing algorithm, each communication node is treated as an agent which can independently learn the forwarding policy and forward packets to the next port . As showi, j) to represent the position of a satellite, where i represents the satellite’s orbit number and j represents the satellite’s number in orbit , where V represents the set of satellite nodes and E represents the set of satellite links. Consider an Iridium-like system with M number of orbits and N satellites per orbit; we use  whose final state occurs when the packets have arrived at the destination node. We use  to represent a state of the MDP, where S is the state of the current system, A is the action space, P is the probability of state transition, and R is the reward. Each satellite node only forwards packets to its neighbor nodes, which means that the action space is up to four. Therefore, we chose reinforcement learning rather than deep reinforcement learning to achieve this.p-th node for p = 1 to p = P. In multi-agent reinforcement learning, each agent observes a different state, and they make independent routing decisions based on the current state;States: Each state Actions: The action  to four ;Reward: The reward function is designed according to the goal we want to achieve, such as the minimum time delay and the maximum packet arrival rate. In this paper, the transmission delay consists of two parts, propagation time and waiting time, so the reward function consists of the following three parts:–Propagation delay. In satellite networks, the star link often fails and becomes unavailable, which requires frequent inter-satellite reconnections. For convenience, we consider both reconnection time and propagation time as propagation time delay. –Queue length. Each satellite node maintains the receiving queue –Load growth. To avoid packets being forwarded centrally to individual satellite nodes, causing network congestion, we record the receiving queue length in the previous stage as the load growth of the satellite, which is recorded as When using reinforcement learning to solve problems, it is crucial to design state, action, and reward functions . For difN, which ensures that the packets are transmitted to the destination node as soon as possible.Equation  gives thAs shown in Since the topology of the satellite network changes dynamically and the links between satellites are unstable, the neighbor nodes of the satellite change. Therefore, agents must periodically check the neighbor nodes so that they know where to forward packets when they arrive.Satellite nodes receive information about their neighbors by receiving ‘hello’ packets from neighboring nodes . In addiThe algorithm proposed in this paper needs offline training before being applied to satellite nodes. The purpose of this process is to reduce the training time of the algorithm online and improve its initial performance. The network To simulate the random nature of packet generation, the initial and destination nodes of packets are randomly selected from the set of satellite nodes. To reduce training time, if a packet is forwarded to its destination node, a new packet is generated, and its initial node and destination node are also randomly generated.To better explore the unknown space, the Equation , the ageAlgorithm 1 is the process of offline training. First we need to initialize the training parameters: Equation . OtherwiIn the online training phase, the pre-trained parameter in Unlike offline training, agents in online training do not make decisions according to the We simplify the satellite network routing problem to finding the smallest delay path. The delay of a packet consists of two parts: propagation delay and queuing delay. The propagation delay depends on the inter-satellite link distance and link status, and the queuing delay depends on the available resources. According to Equation , we knowAs shown in Equation , we use Equation , we can If we suppose that the link state of a satellite changes, it is obtained first by the two satellite nodes of this link, followed by the neighboring nodes of the two satellite nodes. Therefore, the state of links is serial propagation, which causes certain difficulties for the convergence of the reinforcement learning algorithm. Especially, the state of the satellite network is prone to change; it is possible that the convergence of the previous stage is not yet complete and the link state has changed again.As shown in Algorithm 2 is the pseudocode of the online learning process of the algorithm. It differs from the traditional reinforcement learning algorithm in that this paper proposes an empty packet convergence method, which accelerates the obtaining of network status and is more suitable for the state-variant satellite network environment.    In this paper, we compare FRL–SR with the Dijkstra algorithm in terms of algorithm complexity. The time complexity of the Dijkstra algorithm is Another important indicator is the communication overhead during the execution of the algorithm. In the Dijkstra algorithm, each satellite node must send its own status information in a flood so that each satellite node can receive the global information. The communication overhead is high and also has some negative impact on the network load. In the FRL–SR algorithm, each satellite node only needs to send the status information to its neighboring nodes without forwarding it to the whole network. Even if we increase the number of communications between neighbor nodes to improve the convergence speed, it is much lower than the communication overhead of the Dijkstra algorithm.In this section, we present the simulation results of the FRL–SR algorithm. The experiment was mainly divided into two parts: First, we simulated different reward functions in reinforcement learning and found out which one could minimize the delay for our subsequent experiments. Then, we compared the performance differences between the FRL–SR algorithm proposed in this paper and the traditional Dijkstra algorithm, both running in the same network environment and with the same user data requirements. We compared these two algorithms in terms of average delay, packet arriving ratio, and network load balance, and explained the reasons for their performance differences. To make the experimental results more accurate, we repeated all the experiments three times and took the average of the results as the final result.In this paper, we chose the Dijkstra algorithm as the comparison algorithm because it is the most mature and widely used algorithm in satellite routing algorithm, and through experiments, we verified that the proposed algorithm can effectively improve the satellite routing strategy and better support communication services .The parameters of the experiment are given in i and node j.The performance of different rewards is shown in reward2 function was inversely relative to the queue length of the node and the distance between the two nodes.The Based on the simulation results, we can conclude that the design of the reward function has a great influence on the performance of the algorithm. The goal of the algorithm was to obtain the path with the least delay, so we chose the second reward function for subsequent simulations.Both The FRL–SR algorithm outperforms the Dijkstra algorithm in terms of average delay in the same network environment. This is because the FRL–SR algorithm adjusts its forwarding policy based on the current state of the network and then forwards the packets to the optimal port, while the Dijkstra algorithm makes forwarding decisions based on the previously collected global information. The link state of the satellite network changes rapidly, and the information collected by Dijkstra cannot accurately reflect the state of the satellite network at that time. As can be observed from the above graph, the average latency increases slowly with time because the inter-star link is prone to failure, resulting in packet loss. The algorithm in this paper does not have a data retransmission function, which means the delay of lost packets will keep increasing, resulting in a rising average delay.The relationship between cumulative number of packets and time is shown in Considering Based on the above simulation results, we conclude that the FRL–SR algorithm has lower network latency and higher data throughput, which is more suitable for satellite networks with variable network status. In addition, the FRL–SR algorithm also has good load balancing characteristics, and it considers both satellite link status and satellite load when making data forwarding decisions, avoiding network congestion. The Dijkstra algorithm only blindly forwards packets to ports with good link status, causing congestion on some network links.In this paper, we proposed a fast-convergence reinforcement learning algorithm to construct the routing issue in the LEO constellation. Aiming at addressing the characteristics of large dynamic satellite network status and unstable inter-satellite links, we designed a routing method named the fast-convergence reinforcement learning satellite routing (FRL–SR) for online decision-making. This method is always aware of the network link status and dynamically adjusts its routing strategy. The FRL–SR algorithm includes three parts: neighbor node discovery, offline training, and online training. By shortening the cycle time of agent obtaining network states, we accelerated the convergence speed of the algorithm, so that the routing decision was more suitable for the current network state. In addition, we also performed a complexity analysis, and the FRL–SR algorithm was superior to the Dijkstra algorithm in both time complexity and spatial complexity.The simulation results showed that the FRL–SR algorithm had a lower average delay and higher packet arriving ratio compared with the Dijkstra algorithm. In addition, the FRL–SR algorithm also had a good performance with respect to load balancing. It made full use of the resources of each node and reduced the probability of network congestion. Multi-agent cooperation is a promising method to solve the problem of large-scale satellite network routing. In future work, we will continue to work on multi-agent reinforcement learning algorithms to better solve the problem of satellite network routing.


It is estimated that around 28 million surgeries will be postponed or canceled worldwide as a result of this pandemic, causing a delay in the diagnosis and treatment of more than 2 million cancer cases. In Brazil, both the National Health Agency (ANS) and National Health Surveillance Agency (ANVISA) advised the postponement of elective and non-essential surgeries, causing a considerable impact on the number of surgical procedures that decreased by 33.4% in this period. However, some women need treatment for various gynecological diseases that cannot be postponed. The purpose of this article is to present recommendations on surgical treatment during the COVID-19 pandemic. The first human coronaviruses were isolated for the first time in 1937. In 1965, the virus was described as coronavirus because of its profile under microscopy that resembles a crown. Coronavirus is a family of viruses that cause respiratory infections.Since the beginning of February, the World Health Organization (WHO) has officially called the disease caused by the new coronavirus as Covid-19. COVID stands for COrona VIrus Disease, while ‘19’ refers to 2019, when the first cases in Wuhan, China, were publicly released by the Chinese government in late December.The disease pandemic (COVID-19) is a global health emergency and governments around the world are trying to curb the rate of infection by using complete or partial population isolation for the reduction of people's mobility, thereby slowing the spread of the disease. At this moment, Brazil is one of the countries most affected by the virus. The plan is that different phases of the lockdown will contain the circulation of the virus and prevent the complete socioeconomic collapse. The extent of population isolation depends on the situation of the epidemic, with three different phases: (1) total lockdown in the emergency period during the peak of the epidemic; (2) an intermediate period started when the contagion curve begins to fall; and (3) progressive resumption of all normal activities. In Brazil, we are starting to resume activities in some regions, while others are still applying total social isolation.In many cases, COVID-19 pneumonia requires hospitalization and intensive treatment. Currently, more than 7 million people have already been diagnosed with the disease in the world and about 400 thousand people have lost their lives as a result, with a lethality rate of more than 5%. In Brazil, more than 700 thousand people were diagnosed with the disease and about 37 thousand people died. As the surgery is a high-risk situation for the transmission of respiratory infections,It is estimated that around 28 million surgeries are postponed or canceled worldwide due to this pandemic, causing a delay in the diagnosis and treatment of more than 2 million cancer cases. In Brazil, both the National Health Agency (Portuguese acronym: ANS) and the National Health Surveillance Agency (Portuguese acronym: ANVISA) advised the postponement of elective and non-essential surgeries. This had a considerable impact on the number of surgical procedures, which decreased by 33.4% in the country in this period.However, some women need treatment for various gynecological diseases that cannot be postponed. Conservative non-surgical treatment, including pharmacological and alternative therapies, can be implemented even if there is no high-level scientific evidence recommending it. Performing surgery on potential patients with COVID-19 poses a high-risk challenge and several international societies have recommended a non-surgical approach when possible.In China, a group of asymptomatic patients infected with SARS-COV-2 who underwent a surgical procedure had an unfavorable evolution; 100% of patients developed symptoms, 44.1% required intensive care and 20.5% died.Guidelines on the surgical treatment of gynecological diseases at this time are scarce and mostly based on expert opinion, current information available and recommendations from international scientific societies. The purpose of this article is to present recommendations on surgical treatment during the COVID-19 pandemic.The virus that causes COVID-19 is transmitted by saliva and respiratory droplets. Airborne transmission is possible because the virus remains in the air for long periods of time and can be transmitted to others at distances up to 1 meter,The SARS-CoV-2 is present in several cavities of the body of infected patients therefore, during surgery, it can be disseminated through the spray generated by surgical instruments, because the aerosol may contain the virus or parts of it. Some authors suggest the virus remains viable in the aerosol for at least three hours.However, there is no evidence available from the current pandemic or from previous epidemics to conclude that respiratory viruses can be transmitted from patients to health professionals in the operating room via the abdominal route.The European Society of Gynecological Endoscopy (ESGE)All surgical procedures must be considered high risk, because asymptomatic patients may be carrying the virus. Even in vaginal or laparotomy approaches with the use of electrosurgical instruments, there may be formation of aerosol containing viral particles. Thus the importance of implementing measures to minimize these risks, such as performing dissection and vascular control using non-electrosurgical techniques whenever possible; using electrosurgical and ultrasonic devices when needed, in order to minimize aerosol production and avoid long drying times; using suction devices; and minimizing the spread of droplets of blood or fluids.However, in choosing the surgical route, the presence of some patient comorbidity that results in greater morbidity due to laparotomic procedures, longer hospitalization period and higher risk of nosocomial infection should be considered.Careful anamnesis with search for signs or symptoms of the disease  and a history of recent contact with people with COVID-19 or flu-like symptoms, as well as detailed physical examination seem to be effective in detecting patients at risk.Testing for SARS-CoV-2 (RT-PCR) or serological tests before elective surgery are essential to protect patients and healthcare professionals.Currently, several types of tests can be used for virus detection or to check the individual has already had contact with it.RT-PCR: considered the gold standard in the diagnosis of COVID-19. Confirmation through the detection of SARS-CoV-2 RNA in the analyzed sample, preferably obtained from nasopharyngeal scraping (increased sensitivity 10-15% in relation to the oropharynx) performed between the third and tenth day of symptom onset.Serological tests: serology, unlike RT-PCR, checks the body's immune response to the virus. This is done by detecting IgM and IgG antibodies in people who have been exposed to the SARS-CoV-2. In this case, the test is performed on a patient's blood sample. Some laboratories offer the dosage of IgA, but its sensitivity is lower than that of IgG and IgM.Rapid tests: two types of rapid tests are available: antigen tests (detect proteins from the virus at the stage of infection activity) and antibodies tests (identify an immune response of the body in relation to the virus). The advantage of these tests would be their rapid results, although the existing ones have very low sensitivity and specificity in comparison to other methodologies.Many laboratories can take up to 48 hours to perform the exams, and this fact must always be taken into account .We will divide patients into asymptomatic and symptomatic for a better understanding of all and to help establish the procedure with a patient who needs elective, urgent or emergency surgery throughout this period:Asymptomatic patient: asymptomatic patient, afebrile, without a history of fever, without current respiratory symptoms or in the last 14 days. The patient will not always report these complaints actively and the professional assisting her should seek the information. This patient could be released for hospitalization and should undergo RT-PCR testing before the surgical procedure as a guarantee for her and the health team assisting her. Remember the sensitivity limitations of this test in asymptomatic patients. If the test is positive, the decision made in agreement with the patient may be to proceed with the surgery and assume greater risks of serious complications in the postoperative period, or postpone the procedure to a safer time, if possible. If RT-PCR is negative, the serological test can be performed. If IgM and IgG are negative, the patient probably has not yet had contact with the virus and could have her surgical procedure performed with the necessary care for a safe surgery. If IgM is positive, the patient is probably in the acute phase of infection, and procedure should be the same as that with a patient with positive RT-PCR. If IgM is negative and IgG is positive, the patient has already had contact with the virus and established immunity. However, it is not yet known whether this immunity is temporary or definitive and if the patient could have her surgical procedure performed.Symptomatic patient: patients with a respiratory symptom or fever should be referred to a hospital area reserved for suspected COVID. In this location, a detailed clinical evaluation and a new classification of the patient's health impairment will be performed by repeating the measurement of temperature, pulse oximetry and respiratory rate, as well as RT-PCR testing. If the test is positive, the decision may be to proceed with the surgery (made in agreement with the patient) and assume greater risks of serious complications in the postoperative period, or to postpone the procedure to a safer time, if possible. If the test is negative, discussing with the patient on the real need for the procedure is interesting, since she is symptomatic. If surgery cannot be postponed, all PPE should be used and conditions for safe surgery adopted.The recommendation is that anyone working in the operating room wears full PPE, which includes shoe covers, waterproof scrubs, surgical masks, head protection, gloves and eye protection. Limiting the number of people inside the operating room as much as possible and reducing entry and exit movements are also important measures.In patients confirmed with COVID-19 or highly suspicious cases, elective surgery should be postponed until the patient has fully recovered. When surgery cannot be postponed, an exclusive operating room should be used.Another point to be considered by the medical team when performing gynecological surgical procedures at this time concerns the anesthetic act. It is a known fact that proximity to the patient at this moment for support and the support during the anesthetic block , as well as manipulation of the upper airways  can generate and release droplets and aerosols containing the virus in the surgical environment.Use electrosurgical and ultrasound devices in order to minimize the production of plume, with low power setting and avoid long dissection times;Whenever available, use a closed smoke evacuation/filtration system with capacity to hold small/ultra-small particles;Use laparoscopic suction to remove surgical plume in a closed system before deflating the abdominal cavity; do not vent pneumoperitoneum in the room;Use low intra-abdominal pressure (10–12 mmHg), if possible;Avoid rapid deflation or loss of pneumoperitoneum, particularly when changing instruments or extracting samples;Tissue extraction must be performed with minimal CO2 escape ;Minimize the spray or spread of blood/fluid droplets;Minimize the leakage of CO2 from trocars .Perform dissection and bleeding control using non-electrosurgical techniques whenever possible;Employ electrosurgical and ultrasonic devices in order to minimize the production of plume, with low power setting and avoid long dissection times;Use smoke evacuators with a filter element to absorb small particles whenever possible;Use a suction device to remove surgical plume as it is produced;Minimize the spray or spread of blood/fluid droplets.The risk of COVID-19 transmission at the time of hysteroscopy with bipolar electrosurgical devices and normal saline as the infusion medium is unknown, but theoretically low;Recommendations for the use of PPE follow what has been exposed previously;Outpatient surgery should be recommended whenever possible;The risks related to laser vaporization and conization procedures are also unknown, and the above recommendations on minimization and suction of surgical plume also apply to these procedures.Surgery for gynecological patients during the COVID-19 pandemic should be evaluated on a case-by-case basis, taking into account patient risk factors and comorbidities and local resources.Minimally invasive and vaginal approaches are associated with lower morbidity for the patient in many cases and to shorter length of hospital stay.Data on the risk of surgical plume exposure and transmission of COVID-19 are limited.There are strategies for all surgical approaches that can help mitigate the risk of exposing teams in the operating room.Every patient who will undergo surgery should be tested for SARS-CoV-2;If testing is not possible before the surgery, every patient should be considered potentially contaminated and advised on signs and symptoms of SARS-CoV-2 that may appear in the postoperative period;Personal Protective Equipment should be offered to the patient and the entire care team for the maximum safety and reduced contagion during all times of hospitalization;Every health team should be tested with RT-PCR collection from the nasopharynx or serological tests.


The energy harvesting capability of a graded metamaterial is maximised via reinforcement learning (RL) under realistic excitations at the microscale. The metamaterial consists of a waveguide with a set of beam-like resonators of variable length, with piezoelectric patches, attached to it. The piezo-mechanical system is modelled through equivalent lumped parameters determined via a general impedance analysis. Realistic conditions are mimicked by considering either magnetic loading or random excitations, the latter scenario requiring the enhancement of the harvesting capability for a class of forcing terms with similar but different frequency content. The RL-based optimisation is empowered by using the physical understanding of wave propagation in a such local resonance system to constrain the state representation and the action space. The procedure outcomes are compared against grading rules optimised through genetic algorithms. While genetic algorithms are more effective in the deterministic setting featuring the application of magnetic loading, the proposed RL-based proves superior in the inherently stochastic setting of the random excitation scenario. Several solutions have been developed for harvesting energy from elastic and acoustic waves, one of the most ubiquitous and accessible energy sources. Acoustic and elastic energy is transduced into electricity through electromagnetic, electrostatic and piezoelectric mechanisms3 or, alternatively, through magnetostriction and electroactive polymers5. Among these transduction mechanisms, piezoelectric materials are usually preferred thanks to their large power densities and ease of application6. The basic mechanism of piezoelectric energy harvesting is the direct piezoelectric effect converting the deformation of the host structure due to mechanical or acoustic vibrations to an electrical potential7.The reduced power requirements of recent small electronic components9. They often rely on local resonance for sub-wavelength control of waves, enabling the design of compact devices10. Since such systems already contain a collection of resonators, the inclusion of vibrational energy harvesters is straightforward leading to truly multifunctional meta-structures combining vibration insulation with harvesting13. To reduce wave scattering induced by impedance mismatch and simultaneously increase the broadband capabilities, graded designs have been proposed. The term graded is adopted to denote a smooth variation of a particular parameter of the local resonators along space , creating local band-gaps to control wave propagation. Graded metamaterials are thus able to manipulate waves by confinement over some spatial region along the structure, enabling wideband vibration attenuation and simultaneous energy harvesting17. In Fig. In an attempt to enhance energy harvesting capabilities, metamaterial based structures are increasing in popularity, thanks to their advanced performance in wave control and manipulation18, spatially programmable trapping or mode conversion20. High frequency homogenisation was employed to set the gradient function in graded metamaterials21. A general procedure for grading optimization (both in terms of frequency and spacing) has also been proposed22. The procedure, based on reinforcement learning (RL), has been developed to maximise the mechanical energy confinement of a graded metamaterial under monochromatic input. Whilst the solution obtained provides relevant insights on the physics of graded metamaterials, the relation between grading and energy harvesting efficiency remains an unsolved problem. Moreover, most of the studies on metamaterials consider ideal input sources, neglecting the effect of realistic noisy signals. In this paper, we define a RL procedure for the optimisation of graded metamaterials for energy harvesting under general loading conditions, here exemplified for the common cases of magnetic loading or random vibrations. Other works treating design optimisation as a Markov decision process (MDP) solvable via RL were proposed for phononic crystals for matching24 or maximising band-gaps25, and for acoustic metamaterials to minimise wave scattering26. The combination of MDP and RL was also employed in structural optimisation28, material design30, optics31, and chemistry32. Attempts to improve the computational efficiency of RL-based design optimisation were done by exploiting transfer learning to accelerate the design synthesis starting from data regarding existing bio-structures33; and by using convolutional neural network to generate optimisation initiation points closer to the final outcome26.Different grading profiles have been studied for energy harvesting34, and because RL does not require the functional relation between the design parameters and the objective function which is hard to obtain when there are discontinuities with respect to the design parameters. Alternative optimisation methods such as particle swarm usually suffer from the difficulty of imposing constrains on the design parameters35. To assess the performance of the proposed RL-based procedure, a comparison has been made with genetic algorithms (GA)36, a state of the art metaheuristic method for nonlinear optimisation. When the optimisation problem is defined in a deterministic setting, GA have proved to be more effective and less computational demanding than the RL. However, when a stochastic setting is considered, the proposed RL-based procedure has notably outperformed GA.A RL-based approach has been preferred to gradient-based optimisation because RL does not require continuity and low-modality throughout the design space37, the combination of physical knowledge and RL has only been explored recently. For example, physical considerations were exploited to provide meaningful and synthetic representations of the studied system38, and governing equations were used to inform policy search and model learning39. Here, the design space is constrained on the basis of the physical understanding of wave propagation in a locally resonant system. Specifically, the dimensionality of the design space is reduced to get configurations possibly leveraging the rainbow effect41, i.e. spatial signal separation depending on frequency.While physics-informed machine learning (ML) has emerged as an effective and elegant approach to solve regression problems22, we modified the interpolation rules, used to reduced the dimensionality of the state of the system, on the basis of considerations on the learning of the RL agent, in this way greatly benefiting its training. Such considerations are general and can be applied to other design optimisation problems. On the application side, the optimisation of graded metamaterials for energy harvesting under a realistic input has been considered for the first time. The design we propose is fully compatible with micro fabrication paving a path towards the development of ultra-wide band (UWB) micro-electro-mechanical systems (MEMS)42 for energy harvesting using metamaterials.The innovative aspects of this paper concern both the methodology and the application. On the methodological side, the knowledge of the physics of the problem has been exploited to constrain the state representation of the system and the action space. At variance with what proposed in44. For this reason, frequency up-conversion (FuC) techniques have been developed to enable MEMS excitation at low frequencies45. Among different FuC techniques, a privileged position is occupied by the magnetic interaction via permanent magnets46, consisting of exploiting a magnetic field to generate an impulsive phenomenon on a piezoelectric transducer without any contact. An optimisation of the graded metamaterial for this kind of input is addressed here. As the theoretical evaluation of the magnetic force is a complex operation, out of the scope of the present paper, we use the simplified formulation developed by Akoun and Yonnet47, which demonstrated good experimental prediction for cubic magnets in Neodymium–Iron–Boron alloy46. In particular, the definition of the magnetic force is done by evaluating47: the magnetisation vectors; the magnetic permeability of vacuum; a few coefficients whose definition depends on the dimension of the two magnets and on the force direction. The magnetic loading is depicted in Fig. Dealing with energy harvesting at the microscale, one of the greatest limitations is the enormous mismatch between the frequencies at which the energy content of the environment is available and the MEMS natural frequency17. Both the waveguide and the resonators are made from silicon with Young’s modulus 48. Alternative choices are not investigated due to the reduced impact of spacing on the dispersive properties of rainbow-based metamaterials22.In Fig. 17. From the mechanical side, a certain mass 49. The steps of the method are briefly described here; additional details can be found in50. First, the piezoelectric coupling term The mechanical response of the system is simulated through the finite element (FE) method by employing 304 Euler–Bernoulli beam elements to discretise the waveguide, and by associating a single degree of freedom (dof) to each resonator pair. Specifically, an equivalent lumped mechanical parameter system is employed to simultaneously model the mechanical response of the beam-like resonators and the effect of the attached circuit51, thus minimising the operative conditions in which, typically, reflections at the waveguide extremities cannot be exploited due to the large substrate to which the device is attached. In the laboratory, such absorbing properties are achievable through acoustic black holes53. The length of the absorbing layers is set to 2.5 mm as it must be at least 4 or 5 times longer than the wavelength to be effective. Consequently, the exciting force, applied at Two absorbing layers are used to avoid wave reflections22. Hence, the analysis is ended only if the harvested energy does not modify meaningfully for a sufficiently long time interval. The cumulative harvested energy is computed as in Eq.  is 54 ruled by the Euler–Bernoulli beam theory. The extended proof and its numerical validation are reported in the supplementary material. This denotes an intrinsic energy decrease at higher frequencies for the same applied force. For the linear grading the upper limit of 2 MHz is exceeded by the resonance of the first three resonator pairs with which the propagating waves interact.For white noise loading, the performance of the RL based optimisation has been evaluated again by comparing it to a linear grading and a GA optimised grading. The white noise spectrum always spans the frequency range between 0.1 and 2 MHz but the energy carried by each frequency components is randomized for a total of 128 different loads. Moreover, given that the input to the numerical analyses is a force that follows the white noise function, the energy component of each frequency is not constant with frequency. The relation between the work inserted in the system by an applied harmonic force The outcome of the RL optimisation  algorithm56, is used to maximise the optimisation target, here the amount of harvested energy. The RL agent takes actions according to a stochastic policy parametrised by a neural network (NN). The NN plays the role of a component of the agent termed actor. The policy is updated according to a second NN, modelling another component of the agent called critic. In turn, the critic is updated on the basis of the expected return computed in the environment, here the FE setting in which the response of the piezo-mechanical system is modelled. By assuming a finite number 56.The workflow of operations of the RL-based design optimisation is detailed in Fig. entclass1pt{minima55. Algorithms handling continuous action spaces have been considered. Looking for the best performance, we have tested the following algorithms on the white noise loading case: the deep deterministic policy gradient57 (DDPG), the trust region policy optimisation58 (TRPO), the proximal policy optimisation (PPO)59, and SAC. The latter has recorded the best performance. It is worth to mention that SAC enjoys the advantage of not requiring the tuning of the learning rate.The use of SAC with respect to other RL algorithms is now motivated. Employing policy gradient methods is suggested to handle the stochastic optimisation setting required by the white noise load case. The actor-critic paradigm reduces the variance error in estimating the optimal policy42) leads to an exploding number of possible designs. Consequently, a continuous action space and a continuous system state have been preferred, and the use of function approximators, namely NNs, is required instead of tabular approaches.We have pointed out that the selected algorithm can handle continuous action space. In principle, a discrete action space could have been set by modifying the length of each resonator according to the tolerances of the manufacturing process. However, considering 30 resonators and a fabrication process tolerance of 0.15 As the automatic extraction through NNs of low-dimensional features informative of the system state is challenging in a RL setting, we have decided to provide an already dimensionality reduced vector as input in alternative of a vector detailing the lengths of each resonator. For this goal, we have introduced the interpolation rule reported in Fig. 22 in which B-spline interpolation was exploited. However, B-spline interpolation makes extremely challenging for the agent to foresee the effect of its decisions. As shown in Fig. To reduce the dimensionality of the system state and of the action space, we have exploited the physical understanding of wave propagation in a local resonant system to exploit the rainbow effect. Specifically, resonators’ lengths are set according to an envelope defined by interpolating a reduced number of points, as done in our previous proposalIn conclusion, we have demonstrated potential advantages of RL-based optimisation in designing graded arrays of resonators. Superior performances with respect to conventional optimisation procedures, e.g. genetic algorithms, have been demonstrated for inherently stochastic setting of random excitations. More in general, the greatest potential of the procedure is the ability to deal with stochastic optimisation. Possible limitations come from the large number of required model evaluations, implying a three times longer optimisation process with respect to genetic algorithms for the problem at hand. Such drawback becomes more severe by scaling the dimensionality of the problem. For this reason, a critical aspect of the method is represented by the definition of the physical-based constraint on the state representation and action space to avoid evaluating less promising designs, still allowing for unexpected outcomes as seen in the proposed applications.Future work will act to refine the RL-based optimisation with additional interpolation points, as well as to include asymmetric configurations with further types of waves, e.g. longitudinal or torsional. Extensive experimental activity will be also carried out to validate the obtained numerical results. The proposed method could be applied to 2D or even 3D wave propagation problems for the optimisation of elastic lattices or frames. Finally, further extensions come from the ability to cope with not simultaneous, possibly competing, design goals thanks to the adopted MDP formalisation of the optimisation problem.Supplementary Information.


The discussers would like to thank the authors for investigating the efficiency of machine learning methods in predicting splitting tensile strength of high-performance concrete (HPC). In the discussed study . I could understand the meaning after carefully reading research methodology section. In the 4th paragraph of the introduction section, the authors define the ANN, extreme learning machine (ELM), SVR, decision tree, random forest, gradient boosting and adaptive boosting as machine learning (ML) algorithms. This is also not correct because all these are ML methods and training algorithms are used to obtain their model parameters. Such wrong statements are also seen in the last paragraph of the introduction section .In the abstract of Wu and Zhou , the autx1, x2, …, x12 to be able to better follow the difference between each parameter.In the caption of Fig. 6, the authors mention the rough and fine selection but they did not give any information for these in the text. Which methods did they use for these selections? There is no equation and reference for the GA-ANN definition part while GS-SVR has several equations and references. It would be better to calibrate both SVR and ANN using same algorithm, GA or GS. The other issue worth to mention here is that the abbreviations of the input parameters could be used in Table


This study aims to critically analyse the workflow of the in situ bioprinting procedure, presenting a simulated neurosurgical case study, based on a real traumatic event, for collecting quantitative data in support of this innovative approach. After a traumatic event involving the head, bone fragments may have to be removed and a replacement implant placed through a highly demanding surgical procedure in terms of surgeon dexterity. A promising alternative to the current surgical technique is the use of a robotic arm to deposit the biomaterials directly onto the damaged site of the patient following a planned curved surface, which can be designed pre-operatively. Here we achieved an accurate planning-patient registration through pre-operative fiducial markers positioned around the surgical area, reconstructed starting from computed tomography images. Exploiting the availability of multiple degrees of freedom for the regeneration of complex and also overhanging parts typical of anatomical defects, in this work the robotic platform IMAGObot was used to regenerate a cranial defect on a patient-specific phantom. The in situ bioprinting process was then successfully performed showing the great potential of this innovative technology in the field of cranial surgery. In particular, the accuracy of the deposition process was quantified, as well as the duration of the whole procedure was compared to a standard surgical practice. Further investigations include a biological characterisation over time of the printed construct as well as an in vitro and in vivo analysis of the proposed approach, to better analyse the biomaterial performances in terms of osteo-integration with the native tissue. Bone repair is a biological process consisting of the self-regeneration of the tissue that heals itself without a fibrous scar ,2,3. Nevdura mater and other surrounding tissue [Considering the case of cranial bone defects, the gold standard for the treatment is autologous bone substitutes for their great properties of biocompatibility and osteointegration, despite their clinical use being restricted by donor-site morbidities and because autograft tissue is limited . To avoig tissue . Polymerg tissue ,18. Moreg tissue .In recent years, scaffold-based solutions for cranial reconstruction have been investigated. The scaffold can be made of different biomaterials both synthetic and natural ; composite materials combining two or more of the previously cited biomaterials) and may also include bone growth factors or stem cells . The useThe in situ bioprinting approach consists of the layer-by-layer fabrication of the tissue substitute directly on the patient’s defect site: different biomaterials or bioinks can be deposited in the damaged area to form complex architectures which attempt to replicate extracellular matrix and improve cell adhesion and their capacity to regenerate the tissue ,22. The In the literature, there are several attempts at using in situ bioprinting for bone tissue regeneration with different types of printers and biomaterials. For example, Keriquel et al.  used lasTo date in situ bioprinting for cranial defects can represent a good method for cranial fractures. There are some critical aspects related to this procedure that are still in the research phase. Some of these defects necessitate the replacement of multiple tissue layers for the reconstruction of the tissue’s original anatomy. Moncal et al.  tested aThe main aspect that may or may not lead to the success of the experiment is the choice of the appropriate bioink/biomaterial ink. Biomaterials, including those mentioned above, can be printed on the defect but some researchers are looking for different solutions. The ideal bioink/biomaterial ink must be compatible, and processable with a suitable bioprinting technology and has to maintain the shape and have the right mechanical properties at physiological conditions ,29. For In this study we present the application of in situ bioprinting to a simulated scenario, inspired by a real clinical case study, using a robotic-based in situ bioprinting approach proving the great potential of this technology. The term bioprinting is applied in this study following the definition of Moroni et al. , i.e., bAcquisition of the 3D digital model of the damaged area: the geometry of the anatomical portion to be reconstructed can be obtained with medical imaging techniques , magnetic resonance imaging (MRI)) to ensure high accuracy and resolution of the model;® is used);Printing path planning: based on the acquired geometry the printing pattern to restore the defect is planned. Depending on the slope of the surface on which the material has to be deposited, it is possible to use two different approaches to planning the trajectories: (i) with slopes greater than 45°, the extruder must be kept perpendicular to the surface in order to achieve an optimal deposition; (ii) on the other hand, it is sufficient to keep the extruder vertical and deposit the filler following the profile of the surface. The coordinates of the points of the planned path are referred to the computer-aided design/computed-aided manufacturing (CAD/CAM) software reference frame ;Registration of the printing path in the robot workspace: the path planning is generally defined in the pre-operative scanner reference frame, which does not coexist in space and time with the robot one, which includes the patient. The transformation matrix between the two reference frames can be computed by acquiring anatomical or artificial landmarks in both reference frames .A standardised robotic-based in situ bioprinting workflow can be schematised in four main steps ,34 Figu:Acquisit® R2020a application [Projection method: a single layer pattern, loaded as g-code, can be projected on the substrate mesh and the end-effector path is computed constraining its orientation to be perpendicular to the printing surface;® application, and the end-effector path is computed always maintaining the vertical orientation.Non-planar method: a single or multi-layer pattern is generated in a non-planar slicer software ,38, uploThe experimental part was carried out using IMAGObot, a robotic biofabrication platform developed in a previous study ,33,35. Ilication ,33. Thishttps://github.com/CentroEPiaggio/IMAGObot, accessed on 7 May 2023) and is constantly updated.IMAGObot LinuxCNC source code has been released as an open-source project on the GitHub platform  was selected. No skin wounds were present, nor underlying brain contusions. Consciousness status was quite normal, according to Glasgow Coma Scale (GCS) . IndicatITK-SNAP , to geneThen, the skull model was exported as a .stl file, and the open-source software Blender  was used®, thus simulating the surgical procedure  by taking into account only half of the skull with the injured portion. This allowed an easier positioning on the print bed and a reduced amount of support , as well as a shorter total print time (around 12 h). The model was 3D printed with the Creality Ender 5 3D printer using ivory poly-lactic acid (PLA) b,c. Bonerocedure d,e. Baserocedure f and fabrocedure g by castra mater ) into a dura mater) is repeated in height to reconstruct the missing bone tissue, thus filling the whole defect. The mesh file was uploaded on the CAD software Autodesk® Fusion 360 and the defect portion was extracted . Only two of the semi-spherical artificial markers on the phantom a were avw/v Pluronic acid F127, Sigma Aldrich, Milan, Italy) were bioprinted onto the soft tissues in order not to deposit the bone substitute directly onto the Dura mater. This gives a two-fold advantage: (i) It allows a little gap to be left during printing between the bone tissue and the underlying tissue, enabling better cross-linking of the former without the risk of damaging the dura mater; (ii) it provides a homogeneous layer for the deposition of the upper layers, favouring their adhesion. The benefit of using pluronic acid is that it is easily removable after cross-linking the overlying tissue. The bone phase was then obtained through the deposition of ten layers of an osteoinductive hydrogel  in 10% w/v gelatin  cross-linked with 0.2% w/v genipin  [https://rheo.herokuapp.com/form, accessed on 7 May 2023). As optimisation inputs, constrained layer height was set to 0.6 mm. Considering an in situ bioprinting procedure in vivo, the sterilisation of the biomaterial inks used should be taken into account. The treatment with ethylene oxide seems to be the most promising since it does not affect the biochemical and rheological properties of the hydrogels, thus not influencing the printability and osteo-inductive activity of the used materials [Two layers of sacrificial support material (20%  Taiwan) ). All th Taiwan) . Briefly Taiwan)  via rheoaterials .®.The quality of the printed pattern was quantitatively assessed by measuring the mean Euclidean distance between each point of the planned trajectory and the actually printed path. Data related to the real trajectory were acquired by storing joint angles values from LinuxCNC encoders variables in a .txt file. These values were obtained for each line of the g-code file during the in situ bioprinting step a. Using Dura mater) was demoulded after the complete polymerisation of the Eco-flex 00-10 silicone (about 12 h) and the phantom was assembled as shown in The skull phantom was successfully 3D printed as can be seen in Slic3r software , using both anatomical and artificial markers. Given our experience, the availability of implanted artificial markers increases the possibility to find clearly distinguishable points to perform an accurate registration, but some of them can be removed during the initial steps of the intervention if not visible in the intra-operative scanner image. For this reason, a redundant number of artificial markers should be implanted.Starting from the digital model of the segmented internal portion of the damaged area the non-planar path was computed using the software b. The deThe registered printing pattern was used to compute the joint angles in the IMAGObot app. The obtained pattern b was simDura Mater  can take up to 3 h. Moreover, the additive manufacturing of metal cranial implants, mostly used in the current practice, can take up to 10–15 h, not considering the design phase. The timings of the various steps of the whole operational workflow and their accuracy are reported in The duration of whole printing procedure for the reconstruction of the 40 cm® . In addition, the biomaterial used for bone regeneration would promote better integration with the residual bone, thus leading to a faster recovery by the patient. In fact, as reported by Chiesa et al. , the useFinally, comparing the workflow proposed in this study with that proposed by Fortunato et al. , the regIn this study, we presented the application of in situ bioprinting to a simulated clinical case study. This technology was used to simulate the restoring of the bone in an injured head, showing a standardised robotic-based workflow onto a purposely designed phantom. The successful regeneration of the cranial defect, in terms of damaged area filling, proved the great potential that the robotic approach can have for in situ bioprinting applications in cranial surgery. Furthermore, looking at the whole procedure, this approach is also promising for more advanced clinical applications, such as remote robotic surgery, where the surgeon can acquire data about the patient and, after programming the operating phase, remotely control the robot to regenerate the damaged site. Future investigations will be focused on the testing of the proposed approach both in vitro and in vivo, to analyse the osteo-integration of the in situ bioprinted implant with the native tissue.