Appropriate wound management shortens the healing times and reduces the management costs, benefiting the patient in physical terms and potentially reducing the healthcare system’s economic burden. Among the instrumental measurement methods, the image analysis of a wound area is becoming one of the cornerstones of chronic ulcer management. Our study aim is to develop a solid AI method based on a convolutional neural network to segment the wounds efficiently to make the work of the physician more efficient, and subsequently, to lay the foundations for the further development of more in-depth analyses of ulcer characteristics. In this work, we introduce a fully automated model for identifying and segmenting wound areas which can completely automatize the clinical wound severity assessment starting from images acquired from smartphones. This method is based on an active semi-supervised learning training of a convolutional neural network model. In our work, we tested the robustness of our method against a wide range of natural images acquired in different light conditions and image expositions. We collected the images using an ad hoc developed app and saved them in a database which we then used for AI training. We then tested different CNN architectures to develop a balanced model, which we finally validated with a public dataset. We used a dataset of images acquired during clinical practice and built an annotated wound image dataset consisting of 1564 ulcer images from 474 patients. Only a small part of this large amount of data was manually annotated by experts (ground truth). A multi-step, active, semi-supervised training procedure was applied to improve the segmentation performances of the model. The developed training strategy mimics a continuous learning approach and provides a viable alternative for further medical applications. We tested the efficiency of our model against other public datasets, proving its robustness. The efficiency of the transfer learning showed that after less than 50 epochs, the model achieved a stable DSC that was greater than 0.95. The proposed active semi-supervised learning strategy could allow us to obtain an efficient segmentation method, thereby facilitating the work of the clinician by reducing their working times to achieve the measurements. Finally, the robustness of our pipeline confirms its possible usage in clinical practice as a reliable decision support system for clinicians. Wound healing is a complex process where many factors, whether they are physical, chemical, or biological, work in balance to allow the repair of damaged tissue. Evaluating the impairments that may affect these factors is fundamental to ensure the greatest chances of healing acute and chronic ulcers in clinical settings .A holistic approach to the patient’s situation is recommended, considering social conditions such as the possibility of obtaining access to care, their age, the caregivers’ presence, and a complete clinical and instrumental examination, which make it possible to recognize any comorbidities or deficits limiting the healing ability. Once an all-inclusive assessment of the wound has been performed and the correct therapy has been set, methodical and instrumental-assisted continuous monitoring becomes essential to ascertain whether the healing process is proceeding correctly or not .A clinical wound follow-up involves the observation of different features in the ulcer site and leads physicians to infer data that can be used to determine prognosis and correct treatment. These parameters include the recognition of the wound margins, the bottom, the amount of exudate, the peri-wound skin, and its color, and finally, the shape and size of it . Of all Given the high inter- and intra-operator variability in collecting the physical characteristics ,12, variThe automatic segmentation of wounds is becoming an increasingly investigated field by researchers. Related studies on digital wound measurement mainly involve using video-assistive software or the use of artificial intelligence. The first ones used software coded to recognize specific image characteristics of the wounds  and provide a numeric value. On the other hand, the latter ones offer the possibility of obtaining measurements using different models, such as those ranging from machine learning supervised by humans and based on various classification methods  to unsupervised black-box-type models.Deepskin, represents one of the largest sets of chronic wounds used for the training of deep learning models. Therefore, the Deepskin dataset constitutes a novel starting point for deep learning applications in the analysis of chronic wounds and a robust benchmark for future quantitative results on this topic. Our study rationale is to develop an AI method, based on a convolutional neural network (CNN), enabling highly reproducible results thanks to its data-driven, highly representative, hierarchical image features [While classical software video-assisted classification methods provide a sufficient overall accuracy , AI sensfeatures . Our futfeatures .Deepskin dataset, we compared the results obtained by our deep learning model also on public dataset [In this work, we started with a core set of 145 (less than 10% of the available samples) manually annotated images  with our U-Net model, incrementing the number of training images/masks at each round. We started with a set of 145 images, validating the model using the remaining samples. The training rounds that were performed with the related number of samples used at each round are reported in The model was trained with the same set of hyper-parameters, resetting the initial weights in each round. We trained the model for 150 epochs, monitoring the scores described in The results obtained by our training procedure at the end of the fourth round of training are shown in The results also prove the efficiency of the transfer learning procedure which was guaranteed by the EfficientNetb-3 backbone used in the U-Net model: after less than 50 epochs, the model achieved a stable DSC that was greater than 0.95. A transfer learning procedure was also proposed by Wang et al., but the model used in their work required more than 1000 training epochs, drastically increasing the computational time of the model training.We tested our trained model on the FUSC public dataset without model re-training. In this way, we aimed to monitor the generalization capability of our model and its robustness against different images. The results obtained on these datasets are reported in Deepskin dataset, allowing a direct comparison to be made between the two models and the robustness of the datasets used for their training. We report in Deepskin and FUSC), which are expressed in terms of the same metrics used above.The same dataset was also used by Wang et al. for training a MobileNetV2 model. We re-trained the same model (which is public available in ) on the The active learning strategy implemented in this work for the semi-automated annotation of a large set of images produced remarkable results. Starting with a relatively small core set of manually annotated samples, in only four rounds of training, we were able to obtain annotations for more than 90% of the available images. This procedure could drastically improve the availability of huge, annotated datasets with minimum timeframes and costs as required by expert clinicians.We remark that the “classical” active learning involves a continuous interaction between the learning algorithm and the user. During training, the user is queried to manually label new data samples according to statistical considerations of the results proposed by the model. In our framework, we re-interpreted this procedure, requiring only a binary evaluation of the results by the user: if the generated segmentation did not satisfy the pre-determined set of criteria, it was excluded from the next training set, and vice versa. In this way, we can optimize the expert clinicians’ effort, minimizing the manual annotation requirement.Deepskin dataset introduced in this work constitutes one of the largest benchmarks available in the literature for the wound segmentation task. The heterogeneity of the wound images, combined with their high resolution and image quality, guarantees the robustness of the models trained on them. All of the images were acquired using the same smartphone photo camera, providing a robust standardization of the dataset in terms of image resolution and putative artifacts. At the same time, this characteristic poses the main limitation of this dataset, since batch effects could arise in terms of the generalization capability of the model. A robust data augmentation and deep learning model is crucial to overcome this kind of issue. This limit is a direct consequence of the single-center nature of our work, which reduce the possible heterogeneity of the images, i.e., the different conditions in picture acquisition and the operators available.The Deepskin images were acquired for clinical purposes. Therefore, the focus of each image is the estimation of the clinical wound scores using the photo. In this way, the amount of background information was limited as much as possible, and the center of each image was occupied by the wound area. This was not true when we used other public datasets, such as the FUSC one used in this work, for which a pre-processing was necessary [Furthermore, we would stress that all of the ecessary .Deepskin dataset arises from the geographical location of the Dermatology Unit which collected the data. The IRCCS Sant’Orsola-Malpighi Hospital of the University of Bologna is an Italian hospital with a strong prevalence of Caucasian people. Most of the wound segmentation studies involve US hospitals with a greater heterogeneity of ethnicities. An artificial model trained on a dataset including an unbalancing number of Caucasian samples could introduce ethnicity artifacts and biases. In conclusion, a model trained on the only Deepskin dataset must take care of this limitation before any practical usage.Another limitation of the Deepskin dataset is related to the intrinsic definition of the wound area and the boundary. For both the initial manual annotation and consequent validation of the generated masks, we forced the model to learn that the wound area is the portion of the image that includes the core of the wound, excluding the peri-wound areas. Since there is not a standardized set of criteria for the wound area definition, its specification is made according to the clinical needs.A third limitation of the results that we reported using the Deepskin dataset, as much as the robustness of our U-Net model which was trained on it. An equivalent result was obtained by comparing our U-Net model to the benchmark MobileNetV2.All of the above points must be considered when one is analyzing the results obtained by our model of the FUSC public dataset. The FUSC dataset includes low-quality images, without an evident focus on the wound area and with annotations based on different criteria. Furthermore, the dataset includes only foot ulcer wounds, which were acquired by US hospitals, with heterogeneous patient ethnicity. Nevertheless, the results showed in Despite the unfair comparison of our deeper U-Net model with the lighter MobileNetV2 one, indeed, the generalization capability obtained by our architecture confirms the complexity of the wound segmentation task and the need for a more sophisticated architecture to address it.The analyzed images were obtained during routine dermatological examinations in the Dermatology Unit at IRCCS Sant’Orsola-Malpighi University Hospital of Bologna. The images were retrieved from subjects who gave their voluntary consent to the research. The study was approved by the Local Ethics Committee, and it was carried out in accordance with the Declaration of Helsinki. The data acquisition protocol was approved by the Local Ethics Committee (protocol n° 4342/2020 approved on 10 December 2020) according to the Helsinki Declaration.Deepskin dataset). A Smartphone digital camera  acquired the raw images under uncontrolled illumination conditions, various backgrounds, and image expositions for clinical usage. The selected patients represent a heterogeneous population, and thus, the dataset includes samples with ulcers at different healing stages and in different anatomical positions.We collected 474 patient records over 2 years (from March 2019 to September 2021) at the center, with a total of 1564 wound images .Two trained clinicians performed the manual annotation of a randomly chosen subset of images. The annotation was performed by one expert and reviewed by the second one to improve the data reliability. The manual annotation set includes a binary mask of the original image, in which only the wound area is highlighted; we intentionally did not try to define the peri-wound areas since it is not well confined, and thus, it is not representable with a binary mask. This set of image masks was used as the ground truth for our deep learning model.Pixel-wise annotations are hard to achieve also for expert clinicians, and they are particularly time consuming. For this reason, we have chosen to minimize the number of manual annotations. This small core set of manual annotations was used as a kick starter for an active semi-supervised training procedure via a deep learning segmentation model. The initial set of segmentation masks was relatively rough, and it mostly consisting of polygonal shapes. This did not affect the following re-training procedure as it has already been observed that neural network models are able to generalize even from rough manual segmentation .Several machine learning and deep learning models have been proposed to address the automated wound segmentation problem in the literature ,15,19,24In this work, we propose a combination of active learning ,26,27,28We tried several CNN architectures that are commonly used in segmentation tasks during the research exploration, starting with the lighter U-Net  variantsAll of the predicted images were carefully evaluated by the experts of the Dermatological research group of the IRCCS Sant’Orsola-Malpighi University Hospital Dermatology Unit. Their agreement, jointly with the training numerical performances, led us to choose a U-Net-like  model as−5) and a batch size of 8 images.We implemented the U-Net-like model using the Tensorflow Python library . The modTP, FP, and FN are the True Positive, False Positives, and False Negative scores, respectively.For each epoch we monitored the following metrics:As a loss function, we used a combination of Dice score coefficient (DSC) and Binary Focal (BF) loss functions, i.e.,We performed an intensive data augmentation procedure to mimic the possible variabilities of the validation set. We provided possible vertical/horizontal flips, random rotation, and random shift with reflection for each image.®, with 8 cores, and a GeForce RTX 2070 SUPER NVIDIA GPU).All of the simulations were performed using a 64-bit workstation machine  and  zero-padded to a shape of 512 × 512. The same dataset was used also for the Foot Ulcer Segmentation Challenge (FUSC) at MICCAI 2021, and it constitutes a robust benchmark for our model.The main difference between our dataset and the FUSC one is related to the heterogeneity of wound types. In our dataset, we collected wound images sampled in several anatomical regions (including feet), while the FUSC dataset is focused only on foot ulcer wounds. Moreover, also the image quality changes: the FUSC photos were sampled at more than 2× lower resolution with a different setup, but they were stored in a lossless format. For these reasons, the FUSC dataset represents a valid benchmark for the generalization capability of our model. We performed the evaluation of the entire FUSC dataset using our model, providing the same metrics that were used by Wang et al. for a direct comparison of the results.Deepskin dataset. We remark that with a minimal effort by the expert clinicians, the proposed active semi-supervised learning strategy allowed us to obtain an efficient segmentation method and a valid benchmark dataset at the same time.In this work, we introduced a fully automated pipeline for the identification and segmentation of wounds in images that were acquired using a smartphone camera. We proved the efficiency of the training strategy discussed for the creation of the Deepskin dataset, confirming the robustness of the dataset. The results obtained in this work confirms the possible usage of the proposed pipeline in a clinical practice as a viable decision support system for dermatologists. The proposed pipeline is currently being used in the Dermatology Unit of IRCCS Sant’Orsola-Malpighi University Hospital of Bologna in Italy, and we are currently working on overcoming the issues pointed out in this work. These improvements will be the subject of future works.We proved the segmentation efficiency of a CNN model trained on the


Glass ceramic materials have multiple applications in various prosthetic fields. Despite the many advantages of these materials, they still have limitations such as fragility and surface machining and ease of repairing. Crack propagation has been a typical concern in fullceramic crowns, for which many successful numerical simulations have been carried out using the extended finite element method (XFEM). However, XFEM cannot correctly predict a primary crack growth direction under dynamic loading on the implant crown.  In this work, the dental implant crown and abutment were modeled in CATIA V5R19 software using a CT-scan technique based on the human first molar. The crown was approximated with 39514 spherical particles to reach a reasonable convergence in the results. In the present work, glass ceramic was considered the crown material on a titanium abutment. The simulation was performed for an impactor with an initial velocity of 25 m/s in the implant-abutment axis direction. We took advantage of smooth particle hydrodynamics (SPH) such that the burden of defining a primary crack growth direction was suppressed.  The simulation results demonstrated that the micro-crack onset due to the impact wave in the ceramic crown first began from the crown incisal edge and then extended to the margin due to increased stress concentration near the contact region. At 23.36 µs, the crack growth was observed in two different directions based on the crown geometry, and at the end of the simulation, some micro-cracks were also initiated from the crown margin. Moreover, the results showed that the SPH algorithm could be considered an alternative robust tool to predict crack propagation in brittle materials, particularly for the implant crown under dynamic loading.  The main achievement of the present study was that the SPH algorithm is a helpful tool to predict the crack growth pattern in brittle materials, especially for ceramic crowns under dynamic loading. The predicted crack direction showed that the initial crack was divided into two branches after its impact, leading to the crown fracture. The micro-crack initiated from the crown incisal edge and then extended to the crown margin due to the stress concentration near the contact area. Numerical simulation is a strong tool for predicting the fracture pattern of engineered materials and analyzing their damage behavior. Smooth particle hydrodynamics (SPH) is one of the most robust computational methods employed to simulate solid mechanics and fluid flows developed by Gingold, Monaghan. The classical SPH Lagrangian formulation is well-known to suffer from some numerical drawbacks. However, Lee et al introduced a total Lagrangian upwind algorithm for large strain explicit solid dynamics to improve the stability and accuracy of the SPH analysis.Predictable mechanical efficacy is one of the essential requirements for manufacturing prosthetic crowns. Despite the many advantages of monolithic ceramic crowns, they still have limitations such as fragility and surface machining and ease of repairing. investigated the resistance of functionally graded cementitious composite against repeated penetration. The penetration depth, fracture pattern, and penetration damage of different concrete targets have been numerically studied using the SPH method. They found that SPH could predict the dynamic fracture pattern with reasonable accuracy in comparison with the experiment. Based on the literature survey, relevant and reasonable studies on SPH methods to obtain the mechanical behavior of restorative ceramic materials considering these material’s features are lacking. In the present work, crack propagation under dynamic loading of an implant crown was investigated using the SPH method.Lai et alThe SPH method discretizes a continuum into a set of particles. These particles interact through a kernel interpolation function with a characteristic radius known as the “smoothing length” (h). It implies that any particle’s physical properties can be obtained by summing the relevant properties of all the particles that lie within the compact support of the kernel. This can be understood in two steps. First, an arbitrary field A is written as a convolution with W:Secondly, the integral is approximated using a Riemann summation over the particles:j is the volume, Aj is the value of a quantity, and r denotes the position. For example, the density ρi of particle i can be expressed as:where the summation over j includes all the particles lying within the compact support of the kernel, Vmj =pjVi denotes the particle mass, and ρi is the particle density and Wij = Wji is a short notation for.W The quality of SPH approximation relies on different characteristics such as smoothing length, particle distribution, and particle size, to name a few. In this work, we considered a constant smoothing length and regular particle distribution.Where The geometry used in the present study is shown in AUTODYN explicit hydrodynamic code was employed to construct the numerical model for predicting the implant crown fracture pattern based on the SPH formulation. The numerical model is depicted in  The simulation was performed for an impactor with 25 m/s initial velocity in the implant-abutment axis direction. The base region of the abutment was fixed in all directions as the model boundary condition.Moreover, self-interaction was activated for the crown. The material model constants for all parts were assigned to the numerical model as reported in reference.The induced reaction force of the impactor was decreased due to the dissipated energy during the crack initiation and crack propagation. Up to 0.1 ms, all reaction forces were converged to zero with reasonable accuracy.In the SPH method, there was no need to define the crack path. This is one of the most important benefits of the SPH method compared with FEM. At 2.336E-2 ms, the crack growth was observed in two different directions based on the crown geometry. As  Studies on the methods to obtain the mechanical behavior of restorative ceramic materials considering the features of these materials are lacking. Hence in this study, the SPH method has been employed to investigate the fracture pattern during material cracking. For a better presentation of the fracture pattern in the implant-supported single crown, a lateral cross-section is provided in Ceramic restorations have become popular because of their favorable mechanical properties, excellent biocompatibility, and pleasing esthetics. However, they are physically weak, and the stress state applied in the oral cavity is complex. It is desirable to evaluate the loads a restoration must endure during function over time. It is more appropriate to assess analogous tooth-shaped restorations under conditions similar to the oral cavity.As Moreover, as seen in The main limitation of the present work is the variability of the crown geometry per person/patient. Moreover, the effect of temperature change is an interesting topic that could be studied in the future by coupling mechanical and thermal material properties. Besides, although feldspathic porcelain has been shown to have a low strength compared with other commonly used materials in dental restoration, in this research, the ability of SPH to predict the crack growth pattern of a feldspathic crown as a case study under the dynamic loading has been investigated. We hope that the present study opens up new horizons to prove the ability of the SPH algorithm to predict the fracture pattern of the single-crown implant restorations. Since an accurate SPH simulation requires several mechanical properties supported by particular material characterization experiments, it is suggested for future studies to conduct high-velocity experiments on the recently common materials, such as monolithic zirconia or zirconia layered with porcelain.In the present paper, the crack growth pattern of ceramic crowns undergoing dynamic loading was investigated using the SPH algorithm. The main achievements of the present study could be summarized as follows:The SPH algorithm is a useful tool to predict the crack growth pattern in the brittle material, especially for ceramic crowns under dynamic loading.The predicted crack direction shows that the initial crack is divided into two branches after the impactor’s impact, leading to crown fracture.The simulation results demonstrated that the micro-crack based on the impact wave in the ceramic crowns initiated from the crown incisal edge and then extended to the crown margin due to the stress concentration near the contact region.MK: Conceptualization, writing the original draft, simulation review, and editing. PF: Review, literature review, and editing. AG: Review, SHP reviewing, and English editing. PP: Simulation, manuscript preparation, and literature review. FSG: Review, literature review, and editing. PK: Review and editing.No humans participated in this study. The study did not have any ethical registrations.The authors declare that they have no competing interests related to authorship and/or publication of this work.


A challenging task for the biological neural signal-based human-exoskeleton interface is to achieve accurate lower limb movement prediction of patients with hemiplegia in rehabilitation training scenarios. The human-exoskeleton interface based on single-modal biological signals such as electroencephalogram (EEG) is currently not mature in predicting movements, due to its unreliability. The multimodal human-exoskeleton interface is a very novel solution to this problem. This kind of interface normally combines the EEG signal with surface electromyography (sEMG) signal. However, their use for the lower limb movement prediction is still limited—the connection between sEMG and EEG signals and the deep feature fusion between them are ignored. In this article, a Dense con-attention mechanism-based Multimodal Enhance Fusion Network (DMEFNet) is proposed for predicting lower limb movement of patients with hemiplegia. The DMEFNet introduces the con-attention structure to extract the common attention between sEMG and EEG signal features. To verify the effectiveness of DMEFNet, an sEMG and EEG data acquisition experiment and an incomplete asynchronous data collection paradigm are designed. The experimental results show that DMEFNet has a good movement prediction performance in both within-subject and cross-subject situations, reaching an accuracy of 82.96 and 88.44%, respectively. Neurological diseases after spinal cord injury are one of the major causes of ipsilateral limb locomotion impairments, leading to functional disability and loss of independence in daily living activities for patients with hemiplegia  is the core to enhance the interaction between the user and the robotic exoskeleton, and its main function is to predict the movement of exoskeleton wearers. Traditionally, HEI based on the physical signal, such as inertial measurement unit signal, is certainly one of the earliest used interfaces to predict the movements of the exoskeleton wearer . Because of the strong correlation between sEMG signal and movement, the HEI based on sEMG signal has been earlier and most applied to robotic to help predict the movements of exoskeleton wearers  mechanism to achieve the mapping and deep fusion between sEMG and EEG signal features. The main contributions are as follows:An experimental paradigm for the incomplete asynchronous acquisition of EEG and sEMG signals is designed, which solves the problem of large interference to EEG signals from the synchronous acquisition of EEG and sEMG signals of the hemiplegic lower limbs.Aiming at the high accuracy movement prediction of the lower limbs of hemiplegia, a DCA mechanism based Multimodal Enhance Fusion Networks (DMEFNet) is proposed, which realizes the mapping and deep fusion between the sEMG signal and the EEG signal feature.In the two cases of within-subject and cross-subject, the lower limb movement prediction performance of DMEFNet is compared with six traditional machine learning-based movement prediction models and two deep learning-based movement prediction models.Human-exoskeleton interface is a kind of Human-robot interface (HRI). According to different biological signals, HRI can be divided into HRI based on sEMG signal (MHRI) and HRI based on EEG (BCI).As the biological signal most relevant to exercise, sEMG has been applied to HRI for a long time, and the research on MHRI is particularly rich. An entire MHRI includes three main processing stages data collection and processing, feature extraction, and classification. Traditional MHRI mainly extracts some manual normative time-domain, frequency-domain, and time-frequency domain features, and then uses machine learning methods to construct the mapping between features and different movements  enables direct communication with a robot Recent research has explored the application of deep learning in BCI . Zhu et al.  used theet al. used a multi-dimensional feature fusion network framework to detect muscle fatigue, which used a time-domain attention network and a frequency-domain attention network to extract important time-domain features and frequency-domain features in sEMG signals, respectively, and then Feature fusion is performed, and the results show that the proposed framework can effectively improve the detection performance of muscle fatigue , the input of the model is divided into two parts: the first part is the asynchronous  collected motor imagery EEG (MI-EEG) data . The third module of MCSNet is a Depthwise CNN layer, which is used to combine and optimize the time-frequency features between different channels. Namely, it is used to extract the synergy features between the sEMG signal channels. Same as above, the feature extracted from XsEMG data can be expressed by Equation 2.For the synchronously collected sEMG data Dense Co-Attention Symmetric Network (DCAN) was originally proposed to solve the visual question answering (VQA) problems. It presents an architecture that enables dense and bidirectional interactions between two modalities and contributes to boosting the prediction accuracy of answers , and the EEG signal feature after the reshape operation, E0, is sized to (d×ne). M0 [M0 = reshape(FsEMG)] and E0 [E0 = reshape(FEEG2)] are fed into the DCA layer directly.Since Stacked DCA Layers requires the input of the two modal data to have the same length in the feature representation dimension, we performed the reshape operation on the simultaneously collected sEMG and EEG signal features after feature extraction. The sEMG signal feature after the reshape operation, i)-st layer, the output from the previous layer Mi−1 and Ei−1 are first interacted by multiplication with a learnable weight matrix Wi and get the affinity matrix of sEMG and EEG signal (refer to the Equation 3).The DCA layer does not change the dimensions of the input and output. For the DCA -st layer, Mi and Ei are computed after the interaction by concatenating Mi−1 to PEi, Ei−1 to PMi (refer to the Equations 5 and 6). The features of sEMG and EEG have been further integrated.Next, WMi, WEi∈ℜ(d×2d) and bMi, bMi∈ℜ(d×1) are learnable weight matrix and bias.where FEEG1, and they are followed by a softmax operation. Finally, we can get the predicted lower limb movement. It can be expressed by Equation 7.After the calculation through the stacked DCA layers, we flatten the outputs of the DCA layer and the asynchronously collected EEG signal feature Labelpredicted is the label of lower-limb movement predicted, N is the number of DCA layers.where In terms of movement prediction models based on traditional machine learning, we compare the performance of DMEFNet with six traditional movement prediction models based on handcrafted features. In the selection of sEMG signal features, referring to the research conclusions of time domain and frequency domain features in the literature  have an average age of 24 years, a height between 160 and 180 Before the experiment, record the relevant physical parameters of the subject, and inform the experimental procedure to the subject. Then let the subject wear an EEG cap, apply electrode gel, and paste sEMG acquisition electrodes on the 10 muscles of the left and right legs . During the whole experiment, myoMUSCLE  and waveguard  are used to record the subject's lower-limb sEMG signal and MI EEG signal data.Hz, 10 channels) collects the sEMG signal data of each lower limb movement of the subject, and waveguard  collects the EEG signal data of each MI of the subject. After obtaining the sEMG data, a 50 Hz notch filter is used to remove the power frequency interference of the current, and a 10–450 Hz bandpass filter is used to retain the effective information of the sEMG signal. As for the EEG data, a 0.3–30 Hz band-pass filter is used to retain the effective information of the MI EEG signal. Since the application of this article is aimed at the prediction of lower extremity hemiplegia rehabilitation training , this article only uses the EEG signal data during the period of 5.5−8.5s and the sEMG signal during the period of 7.5−8.5s. Considering the availability of biological nerve signal data on the affected side of the hemiplegia patient, we only take the sEMG signal channel and EEG signal channel of the subject's one-side body  to train the proposed model. In addition, for the sEMG signal, this article uses 200 ms (including 300-time series data) as a time window to segment the sEMG signal, and the movement step of the time window is 100-time series data  are set to , and all LSTM's output and hidden unit are of dimension 300. Exponential linear units (ELU)  is used to test the results statistically  as the response variable).We compare the performance of traditional machine learning-based movement prediction models  with DMEFNet. Within-subject results across all models are shown in P>0.05).In general, the DMEFNet proposed in this paper can obtain a better lower limb movement prediction performance in the within-subject situation, and its performance is not statistically different from the performance of other contrast movement prediction models . This means that the DMEFNet proposed in this article can more effectively find the co-attention and mutual mapping relationship between sEMG and EEG signal features, and achieve more accurate lower limb movement prediction.Cross-subject prediction results across all models are shown in In this article, a DCA mechanism based HEI is proposed for lower limb movement prediction in hemiplegia rehabilitation training. The interface constructs a DMEFNet, it uses EEGNet and MCSNet structure to extract the features of EEG and sEMG signals and introduces the DCA mechanism to enhance the common attention between sEMG and EEG signal features, and achieves the deep fusion of multi-modal signal features. An sEMG and EEG data acquisition experiment and an experimental paradigm for the incomplete asynchronous acquisition of EEG and sEMG signals are designed to verify the effectiveness of the DMEFNet model, which solves the problem of large interference to EEG signals when the synchronous acquisition of sEMG and EEG signals of the hemiplegic lower limbs. The experiment results show that DMEFNet has a good movement prediction performance in both within-subject and cross-subject situations. In the future, we consider applying the proposed HEI to an actual exoskeleton platform for rehabilitation training. In addition, We will focus on the study of HEI that can be rapidly adaptive for cross-subject.The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.The studies involving human participants were reviewed and approved by Ethics Committee of University of Electronic Science and Technology of China. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.KS designed the movement prediction model, performed the experiments, and drafted the manuscript. RH, FM, and CZ participated in the design of the movement prediction model and assisted in the manuscript writing. KH and ZP participated in doing the experiment and implemented the designed algorithm. XY and HC guided the writing of the manuscripts and carrying out experiments. All authors contributed to the article and approved the submitted version.This study was supported by the National Key Research and Development Program of China (no. 2018AAA0102504), the National Natural Science Foundation of China (NSFC) (no. 62003073), the Sichuan Science and Technology Program , the China Postdoctoral Science Foundation under Grant (nos. 2018M643452 and 2021M700695) and the Research Foundation of Sichuan Provincial People's Hospital (no. 2021LY12).The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


During the past decade, Virtual reality (VR)-enhanced education has been adopted as a pedagogically new learning approach to smooth the learning progress. With the rise of VR-enhanced activities, investigating the effect of personality types of EFL learners on their writing performance to use VR-based instruction for learning may be a crucial factor influencing their achievement. This study was an attempt to research the impact of VR- enhanced classes on paragraph writing of extrovert and introvert English foreign language (EFL) Learners. To achieve the purpose of the study, first, the Preliminary English Test (PET) was administered for ensuring the homogeneity of the sample. Once the homogeneity was established, a total of 52 EFL intermediate students were selected and placed into two comparative and two control groups, with 13 participants in each group. Four groups took part in the study, with two groups undergoing treatment with the non-VR teaching approach of paragraph writing and two groups undergoing treatment of VR-enhanced education. Among these four groups, two consisted of introverts and two extroverts that were identified through Eysenck Personality Inventory. The two comparative groups received 12 sessions of VR-enhanced education in online classes , but the control group received learning writing paragraphs through using instructor’s instructional materials. After 6-weeks of treatment sessions, all participants of the study took the post-test. According to the results, the VR-enhanced education was more effective than the non-VR teaching approach in developing paragraph writing of both introverts and extroverts. Moreover, the results of the research showed no significant differences between the performance of the introverts and extroverts, meaning that both had the same performance. The results of this study may pave the way for teachers to use VR-based technology in online and traditional classes without worrying about how learners with different personality traits respond. The application of educational technology tools has become a crucial part of the learning process in and out of classes. In the words of Su and Zou , the useThe constant development and improvement of technology have revolutionized the education system and have made it possible for students to use different tools. One of these technologies which is incorporated recently in English foreign language (EFL) teaching and learning is virtual reality (VR) which has attracted the attention of many scholars Chen, . VR usesWriting involves complex process skills and students can benefit from using appropriate technology to help them master these skills , but there has not been a parallel uptake of VR activities in formal language learning settings (state universities and public schools), and it remains unknown in schools and universities. Although different studies accepted affordance, a full-scale search that discussed how they can be applied to EFL learning seemed to be missing in the literature  was administered and 52 learners at the intermediate level of proficiency that ranged in age from 18 to 24 years old were selected. The researcher selected those whose scores were around the mean. It should be mentioned that to identify the participants’ personality types, first, the Eysenck Personality Inventory (EPI) questionnaire was distributed among them and based on the scores of the questionnaire and PET, the selected participants were randomly divided into four groups; two groups (13 introverts and 13 extroverts) undergoing treatment with the non-VR teaching approach of paragraph writing and two groups undergoing treatment of VR-enhanced education (13 introverts and 13 extroverts). They were informed in advance that their identities would be kept anonymous during all phases of the study. They were also informed that the issue of confidentiality would be observed.Three instruments were used in this study as follows:To check the homogeneity of the participants, the English language proficiency test called Preliminary English Test (PET) which is one of the standardized tests  formula. The computed Pearson correlation coefficient for coding the writing; such as Advanced writing book  = 0.88, which represents a large effect size, p = 0.00] and post-test ., Advanced Writing .The Table In order to answer the first research question whether there were any significant differences between the effects of VR- enhanced education and the non-VR teaching approach on EFL learners’ paragraph writing; a two- way ANOVA was performed. At first, the results of the pretest were investigated to see whether the groups were similar before the study or not. Descriptive statistics for the scores of the participants on the pretest are summarized in the following table:As is evident from Table According to Table To investigate the effect of the treatment and to find the differences between the groups at the end of the study, descriptive statistics and Two- Way ANOVA was run on the post-test scores of all groups.The Table The results of Table To address the second research question whether there were any significant differences between the effects of VR- enhanced education and non-VR teaching approach on extrovert learners’ paragraph writing, an ANCOVA was run on the post-test scores of extrovert learners in both groups.Table To address the third research question whether there were any significant differences between the effects of VR- enhanced education and non-VR teaching approach on introvert learners’ paragraph writing, an ANCOVA was run on the post-test scores of the introvert participants in both the comparative and the control groups.Table The first aim of this study was to determine to what extent VR-based approach could contribute to the development of the paragraph writing of intermediate EFL learners. The mean scores from the learning result of the participants who were treated with VR-approach (18.87 for the introverts and 18.93 for the extroverts) were higher than the mean scores of those treated with non-VR teaching methods (16.45 for the introverts and 16.63 for the extroverts). In other words, the participants who were taught by VR-enhanced education significantly outperformed the participants who were taught by the traditional teacher-centered methods of teaching paragraph writing. Regarding the second aim of the present research, both personality groups had an equal performance, and the interaction between personality type and treatment approach was significant, meaning that both personality types responded similarly in each group. The results showed that the process paragraph writing achievement of the participants in the VR-enhanced education was higher than the process paragraph writing achievement of the participants who were taught with the non-VR teaching approach, although the comparative and control groups received the same amount of time and instruction. Their different performance can be explained by the fact that VR-based environment involved the comparative participants directly in the activity as experiential learning. The VR app used in the present study, gave the participants the opportunities to practice process paragraph writing in a relaxed and pleasurable manner as they visualized real-world through the realistic simulation of the actual world and interacted within that environment. The VR app was a 3D technology that simulated the sequential steps in a procedure leading to its accomplishment. It helped the participants’ visualization of how to do something  which provided them with both interactivity and visual representation. The participants of comparative groups were asked to play the VR app and got enough information required to complete the process and write a process paragraph when they got the guidance for the how-to-do-the process and share it in their group. It enabled them to interact directly with the concepts and made them fully immerse themselves in the context created by the VR app. The results of the present study support the findings of other studies that showed VR-enhanced education increased the participants’ engagement, visual and physical immersion , suggesting the superiority of VR- enhanced education over non-VR teaching approaches to learning.Like the second research question, the third research question showed a difference between the paragraph writing result of the introverted participants in the comparative and the control groups, with the comparative group performing better than the control group, evidence from their mean scores in the post-test . The extrovert and introvert participants in the comparative group that received the treatment through a VR- enhanced education had better performance and made more progress. In the literature, mixed results are achieved regarding the superiority of extroverts over introverts and vice versa, and therefore, making a distinction between them is difficult. According to Ellis , second Although no special studies are conducted on the role of technology in general, and VR- enhanced education in particular in developing process paragraph writing of extroverts and introverts, the results gained about these two personality traits confirm the results of the studies which found no or even negative difference between the performance of the extroverts and the introverts (e.g., Chamorro-Premuzic & Furnham, This study explored the impact of VR-enhanced education on students’ personality traits in their paragraph writing. Based on the results, the significant role of the VR-enhanced education in improving the paragraph writing of the introvert and extrovert participants was stated. It can be concluded that there was an interaction between VR-enhanced education and personality in the participants’ paragraph writing. Taking into account the fact that using the VR-enhanced education can be considered as helpful instructional material in teaching paragraph writing to EFL learners, it can be stated that the proper use of VR technology would significantly assist instructors to promote EFL learners’ paragraph writing. The findings for VR- enhanced education and traditional classrooms revealed no significant differences in process paragraph writing based on introversion/ extroversion. Therefore, teachers can employ this specific technology without worrying about what their class is dominantly with, the extrovert or the introvert learners.Like any other study, the present one has its limitations. Firstly, the sample size was small; as the participants were divided into four groups, each group had just 13 students, and as a consequence, the findings of the study cannot be generalized to other language learners. Secondly, it was conducted on just university learners. The study might be replicated on a larger scale and if possible learners at another level of proficiency.The study has important pedagogical implications. Firstly, well-planned and designed VR apps can promote EFL learners’ paragraph writing. The results of the current study can help EFL teachers to embed VR technologies in their classrooms to improve EFL learners’ writing. Also, using VR apps in EFL classes can enhance EFL learners’ motivation, enjoyment and also their team-work skills. Teachers need training in using VR apps that is easily available and they should be trained on how to adopt technology in their teaching practices.


Begonia sp. and complex 2.5 D photonic crystal slabs from the diatom Coscinodiscus granii, we demonstrate how the setup can be used for measuring self-adapting photonic-photosynthetic systems and photonic properties on single-cell scales. We suggest that the setup is well-suited for the determination of photonic–photosynthetic systems in a diversity of organisms, facilitating the cellular, temporal, spectral and angular resolution of both light distribution and combined chlorophyll fluorescence determination. As the catalogue of photonic structure from photosynthetic organisms is rich and diverse in examples, a deepened study could inspire the design of novel optical- and light-harvesting technologies.Natural photonic structures are common across the biological kingdoms, serving a diversity of functionalities. The study of implications of photonic structures in plants and other phototrophic organisms is still hampered by missing methodologies for determining in situ photonic properties, particularly in the context of constantly adapting photosynthetic systems controlled by acclimation mechanisms on the cellular scale. We describe an innovative approach to determining spatial and spectral photonic properties and photosynthesis activity, employing micro-Fourier Image Spectroscopy and Pulse Amplitude Modulated Chlorophyll Fluorimetry in a combined microscope setup. Using two examples from the photosynthetic realm, the dynamic Bragg-stack-like thylakoid structures of  Natural photonics, the study of light–matter interactions in nanostructured biological systems, is a growing and diversifying field. Research has focused on the fundamentals underlying those natural photonic systems, as well as on their implications in nature and how to utilize them as inspiration in biomimicry. There is a vast amount of work on understanding the photonic properties of nano-scale structures found in a variety of animals, spanning from insects ,2,3,4 toThe biological function of photonic structures in many photosynthetic organisms is widely unknown. It is often speculative due to the lack of methodologies that allow measurement of the dynamic nature of photosynthesis, where light absorption and energy transfer constantly adapt to environmental triggers for an optimized process . DifferePhotosynthesis activity can be measured at different steps along the process: Biomass accumulation and growth can be regarded as the net photosynthesis activity, where energy quenching and dissipation are not considered. Some studies determine oxygen evolution arising from water-splitting at photosystem II as a proxy for this process. Such methods are commonly performed on bulk culture samples with Clarke-type electrodes or with optical sensing techniques usable down to the cellular scale ,20. AnotBeyond the molecular aspects of photosynthesis, it was recently demonstrated that the thylakoid arrangement in the chloroplast might have intrinsic nanophotonic properties . The strCoscinodiscus granii open photonic bandgaps and guided modes [λ ≈ 560 nm) when immersed in water. Between different individuals of that species, the photonic properties are highly conserved for natural refractive index conditions in water, giving rise to questions regarding potential biological functionalities, such as implications in light harvesting and photosynthesis.Another advanced natural photonic system has been recently described in diatoms, some of the most abundant microalgae on Earth. Diatoms form silicon dioxide shells (frustules) made of valves and girdle bands as subordinate components. Recent experiments showed that the complex 3D porous networks in the girdle band of the species ed modes , reminised modes . When meThe current paper will describe our recently developed methodology for measurements of local photosynthesis activity on the organelle scale using PAM techniques, combined with simultaneous determination of cellular structures' spectral light attenuation and scattering profiles. We demonstrate this methodology's wide range of application and reliability, using two diverse photosynthetic examples as model systems, both of which form photonic nanostructures within or close proximity to the photosynthetically active organelles. Furthermore, unlike many natural photonic structures in animals, photosynthetic organisms show photo-adaptive behaviours, which offer many opportunities for biomimetic design inspiration, but also some complications regarding their analysis; primarily, how to optically analyse a self-adapting system, sensitive to light application during the measurement? This paper aims to present a method that allows the analysis of both the dynamic photonic behaviour and measuring the photosynthetic characteristics of phototrophs with photonic structures.The optical system described here relies on the combined implementation of Fourier Image Spectroscopy (FIS) and PAM, two well-established microscopy techniques. This section will describe the technical implementation of the individual optical paths necessary for each technique. Next, we will show how they are combined in the same setup to allow simultaneous FIS and PAM characterization. Finally, we evaluate specific technical problems of the combined FIS-PAM setup when compared to other systems that provide both types of measurements separately.λ = 380 to 800 nm. Illumination from the PAM system is provided via RGB LEDs with narrow-band spectral ranges. If the LEDS are combined, the resulting illumination is referred to as LED white light. The term photosynthetic active radiation (PAR) is used when the illumination spectrum and intensities are expected to induce photochemical quenching [As different illumination sources are implemented in the setup, we defined the nomenclature used throughout the manuscript. White light, as used in the setup, refers to a halogen Tungsten lamp as the illumination source, irradiating the sample with a broad spectral range from ca. uenching .FIS is a technique originally developed to analyse light scattering by a wide family of nanophotonic structures from plasmonics and photonic crystal structures . In the One means of obtaining these measurements is to place a sample on a rotation stage in front of a long working distance objective lens and acquire a series of measurements for different rotation angles. In that implementation, the measurement spot, of several to tens of microns, will change position as the angle is varied except at the very centre of rotation of the goniometric system, which imposes limitations on the measurement of organelles within cells, for example, as translation across the sample is no longer an option . Other iAdditionally, if the back focal image is projected into the slit of a spectrograph  or is scFollowing the description above, to correlate the optical response of photosynthetic unicellular organisms or intracellular photosynthetic organelles, it is necessary to combine FIS and PAM to enable simultaneous measurement with both techniques. Given that PAM is a fluorescence technique it is possible to implement it in optical microscopes, and even commercial solutions  integratIlluminating samples for measuring both Fourier spectroscopy and PAM fluorescence measurements requires careful consideration. Effectively, two light sources are required to achieve reliable measurements from both systems. The fundamental requirements are that (a) the Fourier system requires a continuous spectrum source to produce reliable spectra across the wavelength range of interest and (b) the PAM measurement requires pulsed light at specific, predefined wavelengths, commonly achieved by LED illumination. In practice, the former requires a continuous white light source and the latter an array of LEDs of specific wavelengths with timing control. This means that both sources are required to produce results to a high standard; the LED white light from turning on all the PAM LEDs is not continuous, so it cannot be used for Fourier spectroscopy  and directed towards the top input port of the microscope. A polariser is placed in this beam to control the polarisation of the incoming light. This then strikes a beam splitter mounted in the microscope body and a fraction of the light is sent to the objective lens, which focuses the light onto the sample. In contrast to the FIS source, the PAM source is an array of three different coloured LEDs. By selecting either one colour of LED, or all three, either three independent wavelengths or ersatz LED white light can be produced. These are coupled to a liquid light guide (4 mm diameter), and the output is collimated and projected into the second port on the microscope and directed into the sample by a second beamsplitter.In terms of optical alignment, switching between real and Fourier-plane imaging is a simple process in optical systems, relying on the 4f system to flip between one and the other. TL from the back focal plane of the objective lens by the design of the body of the microscope. A primary image is formed outside the microscope body by TL, at a distance fTL. An extra lens FL1 with fFL1 is added at a distance fFL1 from the primary real image , which forms a 4f system formed by TL and FL1 that, if the focal distances are matched, will project the back focal plane image of the objective lens at a distance 4f from the sample.A schematic diagram of the microscope system is shown in 1) is used to mount a variable iris aperture, which allows spatial filtering of the image if required. In the space between the imaging plane and FL1, a beam splitter is mounted to split the beam into two, with one path going to the Fourier measurement system and the other to the PAM camera. This second path will be described below. FL1 is confocal to the objective's back focal plane and fFL1 = fTL, generating a Fourier image at a distance of 4f from the back focal plane of the objective lens. 200 mm doublet lenses (Thorlabs L200D) are used throughout. In the system being described, it is at this point that the analyser polariser is placed, providing high polarisation quality as the beam is collimated at this point. If required, this could be moved a short distance and/or replaced with a Fourier filter to, for instance, remove a strong central maximum from a laser if one were being used. From here, the second pair of 200 mm lenses produce a Fourier image plane on the slit of a spectrograph . Two additional side paths, which are accessed by flip-mirror mounted beamsplitters, allow complementary measurement. Firstly, a fibre-coupled spectrometer , allowing the measurement of spectra either from the entire Fourier pattern simultaneously  or from a small region of the Fourier image by removing the lens and placing or scanning the fibre across the Fourier image. A flip mirror beamsplitter placed between lenses RL2 and FL2 allows the beam to be directed to an imaging camera  to allow imaging of the beam and photograph collection.In our configuration, these lenses are a Nikon objective (switchable for different magnifications) and tube lens pair mounted in a Nikon Ti2-Eclipse microscope. The imaging plane at the camera mounting position . In this case, we are interested in collecting the real image into the PAM CCD detector. Therefore, a two-lens system is used to project a magnified image form plane RPCoscinodiscus granii bearing photonic crystal structures in the girdle band and (2) the iridoplast found in certain Begonia sp. forming 2D multilayer structures by dynamic thylakoid stacking.To demonstrate the capabilities of this system design, we investigated two photosynthetic systems known for presenting well-defined natural photonic structures; (1) the diatom species Previous studies have tested waveguiding in valve structures of that same species by focusing a laser beam to a few µm on a live cell . SimultaIt has been suggested that diatoms featuring photonic crystal-like structures in their frustules guide light around the cell, allowing chlorophyll on all sides of the cell to photosynthesise efficiently, even when in the shadow of the rest of the cell contents. The experiment proposed to test such a phenomenon was to illuminate a living diatom on one edge with a high magnification, tightly confined spot and to monitor the photosynthetic behaviour of chloroplasts across the cell. To ensure that the illumination was well-aligned onto the diatom girdle, the component featuring the photonic crystal structure in this species, the reflected spectrum was monitored, with the girdle band reflection presenting a strong, dispersive band corresponding to the band gap of the crystal, which prevents the transmission of light of certain wavelengths at specific angles. The silicon dioxide girdle-band structure opens highly reproducible bandgap properties at low refractive index conditions . HoweverCoscinodiscus granii were grown following a well-documented procedure [C. granii with one edge pumped with a white light supercontinuum source had shown that the light appeared to be redistributed across the cell [For the experiments presented here, the rocedure . This prthe cell , but thev/Fm, can be calculated, yielding a value of 0.54 , avoiding major veins to allow the sample to lie flat on a coverslip. This leaf section was then placed upside down onto a coverslip along with a small drop of water. The focus was found on one edge of the sample using white light. Once found, the white light was blocked, and the measurement’s PAM light source was used to track the focus across the sample while not triggering photosynthesis. A new area, unexposed to white light, was chosen and focus on the iridoplast was performed using the PAM camera and pulsed light, again preventing the onset of photosynthesis. A photographed example of the cells under white light illumination, when imaged on the PAM camera, is shown in This approach fulfils two purposes. Firstly, it reduces the number of photons per area at this stage of the preparation for measurement in order to ensure that the white light triggers no PS activity at this stage. Secondly, it allows for a wider field of view, which helps understand the sample under inspection in the future measurement area. After the image had been taken, the beam was again reduced to the original spot diameter suitable for the microscatterimetry measurements described in the first part of this manuscript. One iridoplast was selected to be illuminated with white light and was positioned at the site of the white light beam, which is known from the calibration of the system, and thus no white illumination was performed at this time. PAM measurements were started, and allowed to run for several minutes to ensure the stability of the response. After several measurements were made, the white light was turned on. There is no need for the LED illumination to be activated on the PAM system as the white light here acts as the PAR and the software performs the measurements without this setting needing to be activated. The PAM measurements are continued, with Fourier spectra being taken simultaneously from this point forward. The simultaneously measured PAM data and angle vs. wavelength dispersions are shown in The angle-resolved plots in We demonstrated an experimental system design that allows simultaneous Fourier plane spectroscopy and PAM fluorescence measurements for photosynthetic organisms. This system allows for assessing dynamic photosynthetic systems and how they adapt or adjust to changes in environmental light levels, both spectrally and for photosynthetic activity. There are many organisms where the simultaneous measurement of photosynthetic and spectral properties may provide key information about the function of photonic structures or even non-photonic, but dynamic behaviours.We believe that the system described here presents a system that offers new insights into the subcellular and single-organelle behaviour of phototrophic organisms, including in the case of the dynamic behaviour of organelles as a response to changes in the light environment. In addition, the system design is highly compatible with standard microscopy setups distributed in most life science and physics laboratories around the world, making this technique a prime candidate for an easily implemented tool for the research in-vivo of light–matter interactions in photosynthetic organisms. This is particularly relevant in the field of natural photonics, where the strategies to achieve high photosynthetic yields via natural photonic nanostructures is a promising area to tackle important global challenges such as increasing food security by enhanced crop yield.


Physics-informed neural networks (PINNs) have enabled significant improvements in modelling physical processes described by partial differential equations (PDEs) and are in principle capable of modeling a large variety of differential equations. PINNs are based on simple architectures, and learn the behavior of complex physical systems by optimizing the network parameters to minimize the residual of the underlying PDE. Current network architectures share some of the limitations of classical numerical discretization schemes when applied to non-linear differential equations in continuum mechanics. A paradigmatic example is the solution of hyperbolic conservation laws that develop highly localized nonlinear shock waves. Learning solutions of PDEs with dominant hyperbolic character is a challenge for current PINN approaches, which rely, like most grid-based numerical schemes, on adding artificial dissipation. Here, we address the fundamental question of which network architectures are best suited to learn the complex behavior of non-linear PDEs. We focus on network architecture rather than on residual regularization. Our new methodology, called physics-informed attention-based neural networks (PIANNs), is a combination of recurrent neural networks and attention mechanisms. The attention mechanism adapts the behavior of the deep neural network to the non-linear features of the solution, and break the current limitations of PINNs. We find that PIANNs effectively capture the shock front in a hyperbolic model problem, and are capable of providing high-quality solutions inside the convex hull of the training set. The range of applications is extraordinary, from natural language processing3, image analysis4 and autonomous driving5, to earthquake forecasting6, hydrological modeling8, playing videogames10, generating game content11, and, more recently, numerical differentiation12 and scientific computing14.Deep neural networks (DNNs) have achieved enormous success in recent years because they have significantly expanded the scope of possible tasks that they can perform, given sufficiently large datasets16, in particular high-dimensional partial differential equations (PDEs)18. One of the most promising approaches to efficiently solve non-linear PDEs is physics-informed neural networks (PINNs)21. PINNs are trained to solve supervised learning tasks constrained by PDEs, such as the conservation laws in continuum theories of fluid and solid mechanics24. In addition to fluid and solid mechanism, PINNs have been used to solve a big amount of applications governed by differential equations such as radioactive transfer26, gas dynamics28, water dynamics29, Euler equation31, numerical integration32, chemical kinetics34 and optimal control36.Neural networks can approximate the solution of differential equations19. PINNs have the potential to serve as on-demand, efficient simulators for physical processes described by differential equations (the forward problem)37.Once they have been trained, PINNs can potentially run faster and yield better predictions than standard numerical simulators of complex real-world phenomena21. PINNs may also be used to assimilate data and observations into numerical models, or be used in parameter identification (the inverse problem)38 and uncertainty quantification42.The idea behind PINNs is to train the network using automatic differentiation (AD) by calculating and minimizing the residual, usually constrained by initial and boundary conditions, and possibly observed dataupwind discretizations that yield smooth and accurate solutions near the shock fronts. Methods that use centered approximations, without numerical dissipation, such as standard finite differences or the Galerkin finite element method, lead to solutions that are polluted by spurious oscillations around the shock front, often leading to numerical instabilities. PINN might present similar issues if the same schemes are used to calculate residuals.Learning solutions of nonlinear PDEs using current network architectures presents some of the same limitations of classical numerical discretization schemes. For instance, one of the main limitations of classical numerical methods  is the need to devise suitable 43. Hyperbolic PDEs are challenging to solve numerically using classical discretization schemes, because they tend to form self-sharpening, highly-localized, nonlinear shock waves that require specific approximation strategies and fine meshes44. Solving hyperbolic PDEs seems to be challenging for neural networks as well, as the ability of current PINNs to learn PDEs with a dominant hyperbolic character relies on adding artificial dissipation48, or on using a priori knowledge to increase the number of training points along the shock trajectories39 or adaptive activation functions50.A paradigmatic example is the solution of hyperbolic PDEs. Hyperbolic conservation laws describe a plethora of physical systems in gas dynamics, acoustics, elastodynamics, optics, geophysics, and biomechanicsIn this work, we propose a new perspective on solving hyperbolic PDEs and traditional limitations of classical numerical methods using deep learning. The proposed method relies in two core ideas: (1) A modified PINN architecture can provide a more general method for solving hyperbolic conservation law problems without a priori knowledge or residual regularization. (2) Relating network architecture with the physics encapsulated in a given PDE is possible and has a beneficial impact. Our hypothesis is that sophisticated, physics-specific network architectures  may be more effectively trained and easier understood than standard feed-forward multilayer perceptrons.attention mechanisms to automatically detect shocks in the solution of hyperbolic PDEs. Our use of attention mechanisms to enrich PINN network architectures is inspired by recent advances in deep learning for language processing and translation52. The resulting network is a combination of gated recurrent units (GRUs) and attention mechanisms; we call this a physics-informed attention-based neural network (PIANN). The combination of both elements in the architecture allows for determination of the most relevant information  to adapt the behavior of the deep neural network to approximate sharp shocks without the necessity of residual regularization or a priori knowledge (attention mechanism).The proposed new architecture uses 53, introduced initial and boundary conditions in the formulation as a penalty term in the objective function. The main drawback of this approach is, if this term was not exactly zero after training, the boundary condition is not completely satisfied. Other authors as Lagaris et al.54 or more recently, Schiassi et al.56 make use of the Extreme Theory of Functional Connections57 to enforce the initial and boundary conditions in the solution. As in these works, we also enforce the initial and boundary conditions as hard constraints.Previous works as58. The Buckley–Leverett equation with non-convex flux function is an excellent benchmark to test the overall potential of PIANNs in solving hyperbolic PDEs. We find that PIANNs effectively capture the shock front propagation and are capable of providing high quality solutions for mobility ratios inside the convex hull of training set. Remarkably, PIANNs are able to provide smooth, accurate shock fronts without explicitly introducing additional constraints or dissipation in the residual, through an artificial diffusion term or upwinding the spatial derivatives.We test our PIANN approach by solving a classical hyperbolic model problem, namely the Buckley–Leverett equationThe rest of the paper is organized as follows. The mathematical formulation of the problem, describing incompressible two-phase flow in porous media is introduced in “Section 58 describes the evolution in time and space of the wetting-phase (water) saturation. Let M is the mobility ratio of the two fluid phases. We use the subscript M to indicate that, once the form of the constitutive relations is specified, the solutions of problem  equationmentclass2pt{minim44.This first-order hyperbolic equation is of interest as its solution can display both smooth solutions (rarefactions) and sharp fronts (shocks). Although the solution to this problem can be obtained analytically for simple one-dimensional settings, the precise and stable resolution of these shocks poses well-known challenges for numerical methods45 who report good performance for concave fractional flow functions. In addition, Fraces and Tchelepi59 provide an accurate solution introducing two physical constrains and subsequently modifying the fractional flux equation by a piece-wise form which differentiable form allow to capture the shock. However, these constraints are problem dependent and do not provide a general solution for hyperbolic equations in heterogeneous media. Whether the solution of the BL problem with non-convex flux function can be learnt by deep neural networks without the aid of artificial physical constraints remains an open question.Physics-informed neural networks (PINNs) have been tested on this problem by Fuks and Tchelepi44. Note that equation  and the output is a vector where the i-th component is the solution evaluated in t is a variable of the vector function Let For the sake of readability, we introduce the architecture of The parameters of the PIANN are estimated according to the physics-informed learning approach, which states that mentclass2pt{minimtion Eq. , or in ot since we know the functional form of After utilizing the information provided by the initial and boundary conditions enforcing mentclass2pt{minimi-th location x is included as a fixed parameter in our architecture.The second term of Eq. , the derThe loss function to estimate the parameters of the PINN is given as53. The main drawback of this approach is, if this penalty term is not zero after training, the initial and boundary conditions are not totally satisfied by the solution. To solve this problem, we directly enforce the initial and boundary conditions in the architecture of our PIANN. Thus, we follow the path of the second line in PINN literature56 that enforces initial and boundary conditions as hard constraints. With respect to the first line, this provides several advantages. First, we enforce a stronger constraint that does not allow any error on the initial and boundary conditions. Second, the PIANN does not need to learn these conditions by itself, and it can instead concentrate exclusively on learning the parameters that minimize the residuals of the BL equation. Third, there are no weights to be tuned to control the effect of the initial and boundary conditions in the final solution.There are two main lines in PINNs literature to deal with the initial and boundary conditions. The first one is to include them as a penalty term in the objective functionAdam optimizer60 to minimize Eq. (The parameters of the PIANN are estimated using the mize Eq.  with res62). For that reason, we have proposed a new architecture, inspired by51, to solve non-linear PDEs with discontinuities under two assumptions.Although it has been demonstrated that neural networks are universal function approximators, the proper choice of architecture generally aids learning and certain challenging problems (e.g. solving non-linear PDEs) may require more specific architectures to capture all their properties  Instead of using just neighboring cells’ information to calculate t, M) to a single fully connected layer. Thus, we obtain N GRU blocks (yellow). Each of them corresponds to a spatial coordinate Figure This generates a set of vectors i-th block 51, we introduce an attention mechanism between both GRU block sequences. Our attention mechanism is a single fully connected layer, a, that learns the relationship between each component of 46 since is able to capture the discontinuity without specific prior information or the regularization term of the residual.In addition to the hidden state This is the first paper to use attention mechanisms to solve PDEs. Furthermore, we show that attention mechanisms are particularly effective in capturing the challenging nonlinear features of hyperbolic PDEs with discontinuous solution.To test the proposed PIANN methodology, we perform a set of numerical experiments where we train our network to learn the solutions of the Buckley–Leverett problem –3) for  for 3) fAdam optimizer60 for 200 epochs with a learning rate of 0.001.The training set is given by a grid Figure mentclass2pt{minimM used during training. Top, middle and bottom rows correspond to x coordinates, the first region on the left is one where the water saturation varies smoothly following the rarefaction part of the solution. The second region is a sharp saturation change that corresponds to the shock in the solution and the third region is ahead of the shock, with undisturbed water saturation values that are still at zero. For all cases, we observe that the PIANN properly learns the correct rarefaction behavior of the first region and approximates the analytical solution extremely well. In the third region, the PIANN also fits to the analytical solution perfectly and displays an undisturbed water saturation at zero. As for any classical numerical methods, the shock region is the most challenging to resolve.Figure Around the shock front, the PIANN seems unable to perfectly propagate a strict discontinuity, and the represented behavior is a smeared shock that is smoothed over and displays small non-monotonic artifacts upstream and downstream of the front. The location of the shock is, however, well captured, suggesting that the network has correctly learnt the conservative nature of the BL equation. The non-monotonic behavior is reminiscent of the behavior observed in higher-order finite difference or finite volume methods, where slope-limiters are often used to correct for the non-physical oscillations.51 for every timestep. These maps visualize the attention weight The PIANN needs to learn where the shock is located in order to fit differently to both sides of it. This is the role of the attention mechanism of our architecture. On top of each figure we have visualized the attention map introduced byM: We test the behavior of our methodology to provide solutions for BL at points within the training range of M outside the training set: M is far from the original training set. We observe that our method predicts the behavior of the shock and accurate solution for M. It is important to note that the results show that our method is stable and converges for the different cases.In addition to interpolation, we have studied the behavior of PIANN for extrapolation, in other words, to solve BL for mobility ratios outside the initial training set, We test how the neural residual error progresses based on different mentclass2pt{minimFinally, we have compared the results with central and upwind finite difference schemes for the term of the vector of fluxes. The first-order upwind difference introduces a dissipation error when applied to the residual of the Buckley–Leverett equation, which is equivalent to regularizing the problem via artificial diffusion. Figure In this work, we have introduced a new method to solve hyperbolic PDEs. We propose a new perspective by focusing on network architectures rather than on residual regularization. We call our new architecture a physics informed attention neural network (PIANN).PIANN’s novel architecture is based on two assumptions. First, correlations between values of the solution at all the spatial locations must be exploited, and second, the architecture has to be flexible enough to identify the shock and capture different behaviors of the solution at different regions of the domain. We have proposed an encoder-decoder GRU-based network to use the most relevant information of the fully encoded information, combined with the use of an attention mechanism. The attention mechanism is responsible for identifying the shock location and adapting the behavior of the PIANN model.The loss function of PIANNs is based solely on the residuals of the PDE, and the initial and boundary conditions are introduced as hard constraints in the architecture. As a result, PIANN’s training aims only at minimizing the residual of the PDE; no hyperparameters are needed to control the effect of initial and boundary conditions on the solution.M inside the range of training set, as well as to extrapolate when the value of M is outside the range. However, we observe that if M is too far away from range of the training set, the quality of the solution decreases. In that case, retraining of the network is recommended. (iv) We observe that the residuals decrease when the resolution of the training set increases. However, the change in the residuals is not highly significant. This is advantageous with respect to traditional numerical methods where small values of We have applied the proposed methodology to the non-concave flux Buckley–Leverett problem, which has hitherto been an open problem for PINNs. The experimental results support the validity of the proposed methodology and conclude that: (i) during training, the residuals of the equation decrease quickly to values smaller than In conclusion, the proposed methodology transcends the current limitations of deep learning for solving hyperbolic PDEs with shock waves, and opens the door to applying these techniques to real-world problems, such as challenging reservoir simulations or carbon sequestration. However, the scaling up the proposed method for 3D problems will require to compute attention mechanism very intensive for memory, attention mechanism scale quadratically. A possible approach to overcoming this problem is to use a sparse representation of neural network, use attention mechanism to determine the most relevant time sequences or use simpler attention mechanism that has a linear dependency respect to number of cells. Finally, it is plausible that this method could be applied to model many processes in other domains which are described by non-linear PDEs with shock waves. The extent to which PIANNs could be applied in the more general space of differential equations is left to future research.