The challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this problem well due to the handcrafted fusion operations and the extraction of features only from a single scale. In this work, we solve the problems of insufficient information extraction and fusion from another perspective to overcome the difficulties in lacking textures and unhighlighted targets in fused images. We propose a multi-scale feature extraction (MFE) and joint attention fusion (JAF) based end-to-end method using a generative adversarial network (MJ-GAN) framework for the aim of IVIF. The MFE modules are embedded in the two-stream structure-based generator in a densely connected manner to comprehensively extract multi-grained deep features from the source image pairs and reuse them during reconstruction. Moreover, an improved self-attention structure is introduced into the MFEs to enhance the pertinence among multi-grained features. The merging procedure for salient and important features is conducted via the JAF network in a feature recalibration manner, which also produces the fused image in a reasonable manner. Eventually, we can reconstruct a primary fused image with the major infrared radiometric information and a small amount of visible texture information via a single decoder network. The dual discriminator with strong discriminative power can add more texture and contrast information to the final fused image. Extensive experiments on four publicly available datasets show that the proposed method ultimately achieves phenomenal performance in both visual quality and quantitative assessment compared with nine leading algorithms. Multimodality image fusion is the synthesis of multiple original images of objects in the same scene captured simultaneously from different sensors into a single new image that is enriched with information through image processing and computer technology, and provides a more intuitive understanding with the human eye system. As a typical heterogeneous sensor image fusion, infrared and visible image fusion (IVIF) has already been applied in various fields, like military reconnaissance, video surveillance, vehicle night navigation, target detection and identification  and muchIn the past few years, research on IVIF approaches has drawn extensive attention. In ,6, reseaIn view of the above disadvantages, one research focus is to design IVIF models in an end-to-end fashion. In particular, the end-to-end methods completely circumvent the shortcomings of the traditional and combinatorial-based methods. For instance, DenseFuse  and TSFNGenerally speaking, feature extraction and fusion of the source images are two key steps in the design of IVIF algorithms. On the basis of all previous comments, the motivation for our paper consists of two folds. In the first place, the key to image fusion is to design a more comprehensive feature extraction strategy based on neural networks. This is also the fundamental goal of training models for most IVIF algorithms, that is, to train a network with strong feature extraction capabilities. However, all of the above models only focus on single scale features in the sources. For example, Refs. ,21,23 alTo solve the problems mentioned above, we propose a novel IVIF method using GAN with multi-scale feature extraction (MFE) and joint attention fusion (JAF), called MJ-GAN. On one hand, multi-scale information in multimodality images is considered. More specifically, the highlighted objects in infrared images and the textures in visible images are automatically captured via more MFE modules. Additionally, an improved self-attention structure, which can achieve contextual information mining and attention learning, is introduced into MFEs to enhance the pertinence among multi-grained features. On the other hand, there is compelling evidence that the human visual system (HVS) automatically pays more attention to some salient features or areas rather than the whole. Therefore, we design a JAF network based on the channel attention and spatial attention to strengthen the attention to salient and important features in the source images during the feature fusion stage. Consequently, the fused images will be more consistent with human visual perception. Besides, it is also a well-known phenomenon that the stronger the discriminative ability of discriminators, the better the fused images produced by the generator. For this purpose, the loss functions of the dual discriminator are designed based on the idea of SCD loss function  to improTo visually exhibit the superiorities of our method, we select some representative end-to-end methods, including DenseFuse , CSR 1717, FusioThe contributions and characteristics of this work can be generalized as follows.To adequately preserve the global information, multi-scale feature extraction (MFE) modules are introduced into the two-stream structure-based generator to extract source image features of different scales for fusion.To focus more on the important and salient features during the fusion step, we select and merge significant features via a joint attention fusion (JAF) network.To improve the discriminative ability of the discriminator, a dual adversarial mechanism between the source images and their contributions is designed, which will drive the generator to transfer more original information into the final fused images.The rest of our paper is organized as follows. We introduce some works in Currently, image fusion methods designed in an end-to-end manner have achieved fusion performance far exceeding traditional and combinatorial based methods. These methods mainly include CNN-based, AE-based, and GAN-based architectures. Especially for CNN- and GAN-based methods, the key steps in image fusion, namely feature extraction, feature fusion, and feature reconstruction, are implemented in an implicit fashion.For the CNN-based approaches, infrared and visible image fusion (IVIF) are fulfilled by specially designed metrics of unsupervised learning. For instance, Long et al.  designedGAN has been a significant success in IVIF by virtue of the characteristics of its unsupervised adversarial learning manner, and over the last couple years with a number of effective algorithms being proposed . In 2017Different from the methods based on CNN and GAN, the feature extraction and fused image rebuild in the AE-based methods are accomplished by pre-training an autoencoder model. But the feature fusion is implemented by applying some rough fusion strategies such as addition and L1-norm. For example, Li et al.  incorporConsidering the above limitations, we accomplished IVIF using a novel GAN with multi-scale feature extraction (MFE) and joint attention fusion (JAF), together with two specifically stronger discriminators, which can achieve phenomenal fusion performance. Unlike previous approaches, the novelty of the proposed method lies in three points. First, we designed a multi-scale feature extraction module that is dedicated to extracting more comprehensive representations from the source images. Second, we designed a joint attention fusion module based on the spatial and channel attention mechanisms to recalibrate the extracted features and use them for fused image reconstruction. Third, according to the adversarial principle, a discriminator with better judgment can further force the generator to produce more realistic fusion results. Therefore, we designed two discriminators based on the principle of SCD , aiming at enhancing the discriminative ability of the proposed discriminators.The attention mechanism (AM) has been widely used in various speech recognition , naturalFusionGAN  was firsThe loss function of G in FusionGAN is formulated as:The loss function of D in FusionGAN is formulated as follows:In the training phase, the aim is to train a generator that can generate the preliminary fused image to fool the dual discriminator. The framework of our proposed method is schematically shown in In the testing phase, only the generator is working. Each image pair with arbitrary size as a whole from the test datasets is fed directly into the well-trained generator to produce the fused images with prominent thermal targets and perceptually pleasing backgrounds.Based on the requirement that achieving the leading fusion performance in the IVIF task should not only extract multi-scale features from the source inputs, but also choose important and salient features for fusion, the generator incorporates multi-scale feature extraction (MFE) modules and a joint attention fusion (JAF) network to achieve salient feature extraction and fusion. Also, the dual-encoder-single-decoder structure is adopted as the generator to achieve the cross-modality united representation and extraction of different information. In the process of feature extraction, the MFEs are employed to extract multi-grained features from the source images. As we all know, the infrared image contains main contrast information and auxiliary texture information, while the primary texture information and secondary contrast information come from the visible image. The inputs obtained after concatenation are passed through two encoders separately to extract discriminative features at different scales from the source images. Then, in each path, an independent convolution block is first applied to roughly extract the shallow features from the input. Next, we can obtain the multi-scale features from the adjacent MFE block. Here, the combination of an independent convolution block and its neighboring MFE block is defined as a multi-scale block. In each encoding branch, four multi-scale blocks are arranged in turn to extract the deep features. To circumvent the problem of gradient vanishing caused by the deeper network designed in our model, the DenseNet structure is applied to each encoder. Benefiting from the dense connection, it can not only enhance the information flow between different multi-scale blocks, but also take full advantage of the multi-scale features extracted from the middle blocks. Finally, we can obtain hierarchical synthetic features for each modality image.In the process of feature fusion, a JAF network based on spatial attention and channel attention is devised to select important features extracted by encoders for merging. Four simple but effective convolution blocks are employed in the reconstruction of the preliminary fused image. The structures of MFE and JAF will be described in detail in In the generator, two encoders share the same structure, which includes four Efficient feature fusion depends on extracting comprehensive discriminative features from images of different modalities. As the single scale features extracted from the source inputs cannot represent the overall spatial properties of large targets in the original images, multiple convolution kernels with different sizes are used to extract comprehensive information from the sources. The input features are fed into three separate branches containing The application of multiple small convolution kernels can bring many benefits. On one hand, under the circumstance of an identical receptive field, the deeper the network, the more nonlinearity of the model. On the other hand, stacking multiple small filters can increase the receptive field, which also means that more global and intrinsic features of the target can be extracted. Not only that, but the parameters of the network will be greatly reduced. Therefore, instead of the To address the redundancy and noise in multi-scale representations of source images, self-attention modules (SAM) are introduced in each branch, aiming to enhance important features. The major advantage of the SAM over other attention networks is that it can trigger interactions between different spatial locations to capture the intrinsic correlations of the input data. Hence, driven by , we propThe purpose of the other path is to extract dynamic feature information. The original inputs To deepen the interaction between features, we also concatenate Hence, the feature represented by global dynamic information . The AM works by accessing all input sequences to calculate weights, then combining the weights with the inputs to selectively strengthen the attention more toward discriminant parts of the input images. Therefore, motivated by the success of AM in IVIF , a jointIn the channel dimension, we mainly focus on which features of the input are meaningful. In general, the infrared radiation information is mainly represented by the low-frequency information captured by a global average pooling operation. Therefore, the inputs F1 with In the spatial dimension, the purpose is to concentrate on which parts of the output are rich in effective information, which can bridge the shortage of only using channel attention to some extent. We utilize a simple yet effective To sum up, the purpose of selecting salient features to reconstruct a fused image can be achieved by re-mixing channel-wise fused features and spatial fused features:In our work, the two enhanced discriminators have the same structure. To guide model training in an even better fashion, the loss function of the generator considers structural information, complementary information  of multi-mode images, and the adversarial losses between a generator and two discriminators. We formalize it as follows:pired by , LcontenTo measure the loss of structural integrity as well as luminance consistency, the structural similarity loss The adversarial loss Indeed, according to the loss function of the generator designed in this article, the fusion results with both the information of heat radiation and visible textures can be obtained in the absence of a discriminator. But that is far from enough. Therefore, the adversarial architecture is adopted to keep more information of the sources in the fusion results. Generally speaking, the stronger the discriminative ability of the discriminator, the better the implementation of the fused image produced by the generator. We thus designed two discriminators with the same simple and naïve structure to respectively distinguish one of the source images and its contribution . SpecifiThe loss function of the dual discriminator can be denoted as:In this section, we will firstly introduce four publicly available datasets in detail for training and testing the proposed model. Then, the details of training and testing are described. Thirdly, we choose nine state-of-the-art algorithms for comparison with our method. Finally, we introduce eight commonly used metrics for quantitative evaluation of image fusion performance.https://figshare.com/articles/dataset/TNO_Image_Fusion_Dataset/1008029) (accessed on 5 February 2023), OSU dataset (available at http://vcipl-okstate.org/pbvs/bench) (accessed on 5 February 2023) and RoadScene dataset (available at https://github.com/hanna-xu/RoadScene) (accessed on 5 February 2023). The TNO dataset mainly takes military scenes as background and collects infrared and visible image pairs of soldiers, vehicles, and buildings in different environments under the same scene for image fusion research. The OSU dataset includes infrared grayscale images and color visible image sequences. As we only studied grayscale image fusion, we needed to convert color images to grayscale images in advance. The RoadScene dataset is dominated by traffic scenes, including different scenes formed by the combination of roads, cars, pedestrians, buildings, and other elements. Compared with the TNO dataset, it has higher spatial resolution. The above three datasets all contain pairs of normally exposed infrared and visible images. It is well known that the goal of infrared and visible image fusion is to generate fused images with significant targets and rich textures in extreme environments. Therefore, we further tested the robustness of the proposed algorithm on the Multi-Spectral Road Scenarios (MSRS) dataset (available at https://github.com/Linfeng-Tang/MSRS) (accessed on 5 February 2023) containing both night and daytime scene images. The MSRS dataset contains 1444 pairs of aligned infrared and visible images of high quality. Each dataset and its role are clarified below.We selected three commonly used IVIF datasets to train or test our method, including the TNO dataset  three times (t = 3) before alternating the training of the generator (G) and dual discriminator once per batch. As the training process continued, more infrared intensity information and visible texture information were gradually added to the fusion result when the dual discriminator could hardly distinguish the source images from its contributions simultaneously. In other words, we obtained a generator with strong capability that could produce realistic fused images. The entire training details are presented in Algorithm 1. During the testing stage, only the G was valid. Each image pair from the training dataset or testing datasets was fed into G as a whole instead of image patches to directly generate the fusion result. The proposed network was programmed on TensorFlow.Parameter settings: The learning rates of G and D were the same as As described in To guarantee the fairness of the evaluation results, all competing methods were run based on the publicly available code of the corresponding author or a well-known toolbox, and the settings refer to corresponding original papers. The above methods were run on the same GPU, an NVIDIA GeForce RTX 3070.Trivial differences among fusion results will bring about a challenge for precise subjective assessment. Generally, it is a reasonable matter to adopt multiple image quality metrics for overall evaluation. Over the past few years, all kinds of quantitative assessment indexes for IVIF algorithms have been presented . These m(1)SCD1, S2) can be formulated as:SCD can be adopted to measure how much of the fused image comprises complementary information from the two source images. The differences between the fused image (F) and two sources (SThe SCD can be expressed as:(2)MIMI estimates the amount of information transferred from the two source images to the fused image. The definition of MI in infrared and visible image fusion is:(3)CCCC can measure the degree of linear correlation between the sources and the fused image. It is mathematically expressed as:(4)SDSD can express the contrast of the fused image. The definition of the SD is:(5)SFSF can reflect the texture details of the fused image according to gradient distribution and is defined by spatial row frequency and column frequency:(6)VIFFVIFF is used to calculate the fidelity of the fused image based on human visual perception and is formulated as:In practice, the calculation steps include: (1) filter and divide the source images and the fused image into different blocks; (2) evaluate the visual information of each block; (3) calculate the VIF for each sub-band; (4) calculate the overall index.(7)MSSIMSSIM is used to model loss and distortion between the sources and fused image based on their similarities in light, contrast, and structure information. Mathematically, MSSIM can be defined as follows:(8)ENEN is used to measure the amount of information in the fused image. The mathematical formula of EN is expressed as:What is noteworthy is that the higher the above metrics, the better the fused image. Moreover, we used the codes provided by the author or a well-known third party to perform the calculations of all image quality indicators through MATLAB.At first, four infrared and visible image pairs from the TNO dataset were selected to implement fusion operations using different methods. As shown in From We selected 37 image pairs from the TNO dataset to objectively assess the fusion performance of our method. The first and second ranks on EN and MI denote that the fusion results of our method contained the maximum amount of information. The best and second-best SD and SF demonstrated that our fusion results for the highest contrast contained more edges and texture details. The best MSSIM showed that the fusion results of our method had the highest structural similarity to the two source images, and implied the least loss and anamorphosis in our fused images. The best VIFF indicated that the fusion results for the proposed method were more in line with the HVS. The best SCD and CC values indicated that our fused images were highly correlated with the source images. All in all, the performance of our method was competitive on all eight metrics. Among the quantitative evaluation metrics of the fusion results obtained by our method, few were lower than those of the other comparison algorithms. This is because our method achieved multi-scale representation of the source images and selected significant features to reconstruct the fused images.We validated the generalization ability of the proposed method on the OSU dataset, and The RoadScene dataset is also commonly used in IVIF tasks. Therefore, 20 image pairs from the RoadScene dataset were selected to implement the test of generalization ability. In order to further verify the generalization ability of our method, 20 image pairs from the RoadScene dataset were calculated by eight evaluation metrics to analyze quantitatively. The source images on the MSRS dataset contained diverse scenarios and illumination variations. We implemented testing experiments on 20 daytime infrared and visible image pairs from the MSRS dataset. Additionally, 20 pairs of nighttime infrared and visible images from the MSRS dataset were selected for evaluation testing. In the nighttime scenario, the quality of the visible images was degraded by insufficient illumination. Hence, the fused images should have retained more texture details of the infrared images to enhance the description of the night scene. Similarly, we also calculated the objective evaluation values for 20 pairs of nighttime infrared and visible images on the MSRS dataset. The evaluation results are shown in We further performed ablation experiments on the TNO dataset to illustrate the necessity of the multi-scale feature extraction (MFE) and joint attention fusion (JAF) networks. The details are described below.At first, the related ablation experiments were conducted using qualitative and quantitative approaches to validate the effect of the proposed multi-scale feature extraction (MFE) network. Specifically, a model termed “without-MFE” was retrained on the TNO dataset, and the others were retained. The third row of Subsequently, the importance of the joint attention fusion (JAF) network was also demonstrated qualitatively and quantitatively. Instead of the JAF, we trained a model that obtained the fused features in a concatenation manner (termed as “without-JAF”) and compared their fusion performance. The fourth row of Finally, the impact of both the MFE and JAF networks on the fusion results was also tested. We removed the MFE and JAF structures simultaneously (termed as “without-MFE and JAF”) and kept the others the same as above. The fifth row in Comparing the above ablation results with ours from the viewpoint of intuitive assessment, it was found that they all achieved good fusion performance. However, intuitively, the presence of texture detail loss and artifacts in the ablation results were still observed. We attribute the dip in performance to insufficient feature extraction and fusion. The complete model could reduce the likelihood of unnecessary artifacts by combining MFE and JAF.Due to imperceptible differences in the ablation results, it was necessary to analyze them from the perspective of a quantitative evaluation. Due to traditional algorithms included in the baseline methods run on the CPU, we only compared the time and space complexity among the various deep learning-based algorithms in We designed a GAN-based end-to-end method with multi-scale feature extraction (MFE) and joint attention fusion (JAF) networks (named as MJ-GAN) together with two specific, stronger discriminators that can achieve more promising fusion performance in IVIF tasks. The inventiveness of our method is that the generator implements feature extraction at different scales and utilizes the attention mechanism (AM) to fuse features in a salient way. Therefore, the difficulties of heuristic design faced by combinatorial-based and conventionally based fusion algorithms can be surmounted. Furthermore, the dual discriminator with strong discriminative ability adds more information to the fused image based on the adversarial relationships between two kinds of nets. Importantly, a hybrid loss function will guide the fusion direction and the preservation of information types from the source inputs in the final fused image. As a result, extensive experiments demonstrated the superiority of our proposed method over other representative and state-of-the-art algorithms in terms of both subjective visual quality and objective evaluation metrics.Although the proposed image fusion method achieves competitive performance in infrared and visible image fusion tasks, there are still several issues that deserve to be highlighted. First, the proposed method is mainly aimed at grayscale image fusion, and its practical applications are limited. Second, the designed loss function only focuses on retention of the primary and secondary information of the source image, but neglects the improvement in the visual perception quality of the fused image. Third, there is still room for improvement in extracting and fusing useful features from source images. Therefore, in the future, we will try to extend the application fields and conditions for our method, such as nighttime infrared and visible image fusion, multi-focus image fusion, and multi-exposure image fusion.


The simple model of an ionic current flowing through a single channel in a biological membrane is used to depict the complexity of the corresponding empirical data underlying different internal constraints and thermal fluctuations. The residence times of the channel in the open and closed states are drawn from the exponential distributions to mimic the characteristics of the real channel system. In the selected state, the dynamics are modeled by the overdamped Brownian particle moving in the quadratic potential. The simulated data allow us to directly track the effects of temperature  and the channel’s energetic landscape for conformational changes on the ionic currents’ complexity, which are hardly controllable in the experimental case. To accurately describe the randomness, we employed four quantifiers, i.e., Shannon, spectral, sample, and slope entropies. We have found that the Shannon entropy predicts the anticipated reaction to the imposed modification of randomness by raising the temperature (an increase of entropy) or strengthening the localization (reduction of entropy). Other complexity quantifiers behave unpredictably, sometimes resulting in non-monotonic behaviour. Thus, their applicability in the analysis of the experimental time series of single-channel currents can be limited. Ion channels are transmembrane proteins that fluctuate between variant open (ion-conducting) and closed (non-conducting) conformational states. These conformational fluctuations are called channel gating. The primary experimental source of information about the channel gating phenomena is a patch clamp method . It alloThe Markovian diagrams represent the standard kinetic model of these conformational changes with specific interconnections between the particular states and the respective transition probabilities ,4,5. In Quantifying the physical system complexity is a tricky task. Ludwig Boltzmann historically introduced a measure of the number of possible states for the microscopic world. He connected it to Rudolf Clausius’ concept of entropy and the irreversibility of macroscopic processes. Entropy lies at the heart of the second law of thermodynamics, which states that entropy cannot decrease with time for isolated systems, which implies its maximum for the equilibrium state. Entropy is a thermodynamic property that can measure the amount of thermal energy in a system that is unavailable for doing useful work. It is now associated with disorder, randomness, or uncertainty, rather than the actual state-counting process. In such a case, it can serve as a measure of complexity—the more complex the system is, the higher entropy possesses. In 1948, Claude Shannon introduced the measure for missing information as an analog to the thermodynamic entropy . The difThe ion channel signal reflects the complexity of the channel’s switching among its available states. Only a few reports have addressed the entropy-based patch clamp data analysis ,15,16. IDeterministic forces and stochastic thermal fluctuations shape the single-channel activity. In the mentioned works, entropy values were based only on experimental ion channel recordings. The complexity description gave combined characteristics of the signal, being both shaped by the deterministic interactions associated with protein–protein, protein–ligand, or protein–lipid interactions, as well as the thermal fluctuations of the helices forming the channel and its membrane surroundings. It is challenging to extract information about the relative impact of deterministic forces and stochastic thermal fluctuations on channel gating and signal entropy from the experimental data. Therefore, implementing the information entropy for the patch clamp recordings analysis has been based only on finding the differences in signal complexity with appropriately changed experimental conditions. In this work, we decided to go a step further and provide a more detailed description of how the properties of conformational space and the number of possible states of the channel influence entropy.To that aim, we use data obtained in a simulation of a relatively simple Markovian channel gating model and investigate the effects of the changes in conformational diffusion space parameters on signal complexity. The investigated parameters will refer to the channel’s energetic landscape that governs the open–closed fluctuations in a confined space. Moreover, we decided to provide a signal description using different entropy measures. Our choice is concentrated on two leading groups of entropy features. We consider the standard Shannon information entropy and its frequency-based analog, in the form of spectral entropy. These two measures are based on probability distributions or power spectral density functions, and the interpretation of their values is limited to the statistical description of data. For the comprehensive characterization of ion channel activity, we decided to also select the specific kinds of entropy that can investigate the information rate considering the system’s dynamics. Sample entropy (SampEn) is one such measure, with proven effectiveness in biological signal classification  and the Regarding the effectiveness of these techniques, we decided to set the slope and sample entropy results together to test the possible scenarios related to the potential change of information loss in such a complex ion channel system. In particular, we will test the response to the changes in the energetic landscape of conformational states (corresponding to the stability of available channels’ open and closed conformations) or the relative noise intensity (temperature).The original Shannon idea of the signal description by its information content was later extended to new concepts of complexity measures, such as Rényi , spectraX is related to the spectrum of the k potential states. For the probability distribution, k–th state, the Shannon entropy On average, the entropy of a signal Spectral entropy (S) is the frequency-domain analog of the time-domain characteristics presented above. The power spectrum density (PSD) plays the role of the probability F stands for the number of all frequency components. PSD is usually estimated with a standard (fast) Fourier transform. Because the frequency components are linearly independent, this measure is not sensitive to signal nonlinearities.The methods for estimating information loss based on probability distribution or PSD are easy to apply, although they lose some critical features of usual complex and nonlinear dynamics. For instance, timation ,36. The timation . It quansample entropy (SampEn) [N data points, a set of vectors m consecutive values of a series, starting with the m consecutive vector elements is basically zero for the experimental data. To confirm the similarity between two sample vectors r heuristically. Usually, it is taken between The weakness of the original formulation of the algorithm for obtaining metric entropy (inclusion of self-matches) was corrected later, in terms of (SampEn) . For thel values . If the i–th template vector is close to any other is given byj–vectors f(t)For single-channel kinetics, we can estimate the average voltage rather precisely, although the intrinsic value will always fluctuate. The statistical characteristics of such fluctuations are well-known .The actual dynamics can and is usually much more complex. We can often distinguish several open  states . The trat. Thermal fluctuations are modeled by the zero-mean T is the temperature, and k symbolizes the Boltzmann constant. Without the loss of generality, we will keep the friction coefficient equal to one  can be observed in the form of a varying range of the single-channel current amplitudes, which correspond to the open and closed states, at a fixed noise level (determined by the D parameter). Thus, the model is anticipated to enable the imitation of the experimental ionic current distributions after appropriate parameter optimizations.The representative trajectories obtained by model simulation are presented in The following results of the numerical simulations are presented for the most straightforward possible setup with only two allowed states—one open versus D will remain constant for any value of D.Calculating the information entropy for white Gaussian noise, for which all the possible states are allowed, would result in the maximum entropy value . In termIf, however, one would reduce the ability of travel for the particle by placing it inside the entropic potential, the whole picture would change. The Brownian particle cannot pass through the barriers of the proposed quadratic potential and remains located around the potential minimum, regardless of the open or closed state. The increasing temperature will increase the thermal energy available for the particle. This, in turn, would enable the particle to travel further against the gradient and visit distant locations in the potential well, causing global complexity to rise.versus the noise intensity D. In this scenario, the particle deals with the energy barrier, which causes the particle to sit around the potential minima separately for the open and closed states. The value of Shannon entropy is constantly rising with increasing thermal energy D, to the point where it saturates for the intensity D. The former is slightly more sensitive to temperature change and increases around D. Initially,  can be incorporated to make the model more universal. Including additional open and closed states of different characteristic conductance could increase the randomness of the model, which can better imitate the recorded currents flowing through some ion channel types in the biological membranes. This can lead to a better understanding of ion transport processes and how they contribute to the functioning of biological membranes.


Three-dimensional reconstruction of the left myocardium is of great significance for the diagnosis and treatment of cardiac diseases. This paper proposes a personalized 3D reconstruction algorithm for the left myocardium using cardiac MR images by incorporating a residual graph convolutional neural network. The accuracy of the mesh, reconstructed using the model-based algorithm, is largely affected by the similarity between the target object and the average model. The initial triangular mesh is obtained directly from the segmentation result of the left myocardium. The mesh is then deformed using an iterated residual graph convolutional neural network. A vertex feature learning module is also built to assist the mesh deformation by adopting an encoder–decoder neural network to represent the skeleton of the left myocardium at different receptive fields. In this way, the shape and local relationships of the left myocardium are used to guide the mesh deformation. Qualitative and quantitative comparative experiments were conducted on cardiac MR images, and the results verified the rationale and competitiveness of the proposed method compared to related state-of-the-art approaches. As one of the most important organs of the human body, the heart plays an essential role in the entire blood circulatory system. Among the various tissues of the heart, the left myocardium is the most important because it pumps blood sent to the entire body via the left ventricle. In addition, many cardiac diseases, such as cardiac hypertrophy, directly manifest as structural abnormalities of the left myocardium. The study of the left myocardium, therefore, holds important pathological significance . Three-dFor 3D modeling of the left ventricle, traditional methods can be roughly divided into two categories. The first is the use of professional modeling software, such as Maya, AutoCAD, etc., to perform 3D modeling of the heart. However, these methods require modeling personnel to have professional knowledge of computer graphics. In addition, the shape parameters of some irregular soft tissues are too small to be captured, and the accuracy of the model largely depends on the professional experience of the operator. The second method is to perform 3D modeling on acquired two-dimensional cardiac slices  using a specifically designed reconstruction algorithm. These algorithm-based modeling methods can be further subdivided into model-based and non-model-based approaches. Model-based algorithms use a prior model to deform and approximate the shape of the target object. The model is generally a statistical model generated from many acquired data sets. The typical representative methods of this division include the active shape model (ASM)  and the On the contrary, non-model-based methods do not rely on any prior shape information; the reconstructed 3D model is completely dependent on the medical image data. Representative algorithms include contour reconstruction, finite element, and marching cubes (MC). For example, Gonzales et al.  used a fIn recent years, the powerful representation and learning ability of deep learning techniques have attracted researchers to apply them in the field of 3D reconstruction. Eigen et al.  applied According to the geometric representation of the reconstructed model, deep-learning-based 3D reconstruction can be roughly divided into three categories: voxel-based, point-cloud-based, and triangular-mesh-based. Voxel-based reconstruction algorithms directly extend the correlation operations from 2D images to 3D voxels. For example, Choy et al.  proposedAs another representation of the reconstruction results, triangular meshes have the advantages of being lightweight and preserving topology, in contrast to voxels. Furthermore, compared with point clouds, the points in triangular meshes are connected by edges, so the relationships among points are more compact, thus they can better represent the surface information of the object. For example, Wang et al.  used an Compared with traditional methods, a model constructed by deep learning usually needs a large data set and complex training processes. Some methods tried to combine two above categories. For example, Bhalodia et al.  proposedBased on the above analysis, a non-model-based left myocardium 3D reconstruction algorithm is proposed in this paper by designing an iterated residual graph convolution neural networks. The initial triangular mesh is obtained directly from a coarse result generated based on 3D segmentation images. The obtained mesh is then deformed using a residual graph convolutional neural network. Meanwhile, a point feature extraction module is proposed to extract the corresponding point cloud of initial triangular mesh to obtain the characteristic point cloud. Feature fusion with the points in the mesh is then performed to assist the mesh deformation. In an iterative manner, the triangular mesh is adjusted for a better representation of the left myocardium.Specifically, the main contributions of the proposed method are as follows.(1) A deep-learning-based approach is applied to the left myocardium 3D reconstruction from MRI slices. The specific residual graph convolution neural network is designed towards mesh deformation with iteration. In 3D reconstruction tasks, the mainstream is based on explicit approaches such as marching cubes or implicit approaches such as Poisson-based reconstruction. However, these methods do not incorporate the point’s feature to guide the mesh deformation. The proposed model focuses on the deep feature learning approach to drive the mesh deformation, making the reconstructed result robust to a variety of shapes and sizes of the left myocardium across different subjects.(2) A two-stage mesh generation is designed to build a personalized surface of the left myocardium. In the coarse deformation step, the Cubify algorithm  is adopt(3) A vertex feature learning module is built to assist the mesh deformation of the left myocardium. The characteristics of the initial points are learned by an encoder–decoder neural network to represent the skeleton of the left myocardium at different receptive fields. In this way, the shape and local relationship of the left myocardium expressed by the deep feature is helpful for subsequent mesh deformation. After that, the corresponding deep feature associated with the vertices on the initial triangular mesh is fused with those on the input triangular mesh of each module in the residual graph convolutional neural network to guide the mesh deformation.The remainder of this paper is organized as follows. The overall framework of the proposed 3D reconstruction of the left myocardium is shown in (1) Initial mesh generation module: It builds an initial mesh of the left myocardium using an UNet++ 2.5D segmentation and the Cubify algorithm on the stacked 2D segmented images.(2) Vertex joint feature learning module: The encoder–decoder neural network is designed to extract the joint deep features of the point cloud and multiple CNN feature maps corresponding to the initial triangular mesh to obtain a skeleton of vertices.(3) Mesh deformation module: An iterated residual graph convolutional neural network is designed to gradually deform the initial triangular mesh. Meanwhile, the hybrid deep features obtained in the vertex joint feature learning module are added to enrich the features of vertices in each intermediate triangular mesh for obtaining a better reconstruction. As a result, the mesh is rapidly refined with each iteration.In the following subsections, the details of these modules are presented.The geometric structure of the left myocardium varies widely across different people. Even in the same subject at diverse time points of cardiac motion, the shape of the left myocardium muscle changes remarkably. To handle this challenge, the structure of triangular meshes instead of voxels from the 2.5D segmentation results of the left myocardium was suitable to represent the surface.Typical segmentation methods include traditional multi-atlas label fusion or deep-learning-based methods. Since our work focuses on the 3D reconstruction of the left myocardium, we directly applied the well-known UNet++ based algorithm  to produThe initial triangular mesh obtained by the Cubify algorithm can interpret the shape characteristics of the object; however, it is difficult to accurately recreate the surface of the left myocardium because the mesh contains many abrupt edges and corners. To solve this problem, we propose to optimize the initial surface using a mesh deformation technique based on a graph convolutional neural network. To deform the shape of triangular mesh in the correct direction, it is essential to learn the approximate three-dimensional structural features of the object from its corresponding point cloud of the initial triangular mesh.Compared to data with regular structure such as images, point cloud data have some undesirable characteristics: irregularity, disorder, and permutation invariance ,17,18. TOn the other hand, Lawin et al.  used a vTo overcome the limitations above, we introduce an encoder–decoder neural network to learn the features in point cloud. The flowchart is depicted in Our vertex feature learning is similar to the encoding method of PointNet++ , which iThe sampling layer selects a set of points from input points, which defines the centroids of local regions. Considering the geometric character of the left myocardium, we use geometric subsampling instead of uniform sampling (such as iterative farthest point sampling) to choose a subset of points. The key idea of geometric subsampling is to pay more attention on the region with larger curvature. It consists of three steps. The first step is the pseudo-curvature estimation. The kNN neighborhood is built for each point and the angle between the center point and its neighbor is calculated. The larger the curvature in a region, the bigger the angles, so the angles can be taken as a coarse representative of curvatures in this region. The second step is region separation. Each point in the whole region is divided into two parts according to their angles. If the angles are larger than a threshold (such as 5 degrees), the corresponding points are put into a geometric region; otherwise, the points are put into a flat region. The third step is the respective uniform sampling in these regions. In comparison to iterative farthest point sampling, our proposed geometric subsampling can offer more points in the regions with significant geometric structure, while uniform sampling can be performed in local regions according to the requisite of the sampling number. Additionally, this sampling shows high stability and computational efficiency in comparison to pure curvature estimation and sampling.K is the number of points in the neighborhood of a centroid point; after grouping, the size enlarges by K times, while each group corresponds to a local region.The grouping layer intends to construct local region sets by adding local supportive features around each point in the subsampling set. For example, suppose The PointNet layer adopts a mini-PointNet to encode local region patterns into feature vectors. In this layer, each local region in the output is abstracted by its centroid and local feature that encodes the centroid’s neighborhood. In practice, the point-to-point relationship in the local region is captured by using relative coordinates together with point features. This kind of relationship can be used to guide the mesh deformation of the left myocardium.After this hierarchical point set abstraction processing, the network extracts deep features on the sampling points with better coveragence of the entire point set given the same number of centroids   and Philips  . The image size ranged from 160× 288 to 512 × 512, and the number of slices along the z-axis of each subject ranged from 256 to 512. For each subject, the left ventricular myocardium tissue was labeled from images by two trained students. Their results were cross-validated by a clinical expert. The total labeling process requires nearly 50 h. In our experiments, 10 cases were randomly selected as the training data set and augmented to 80 cases by geometric operations such transformation, scaling and rotation. The augmented data set was then divided as follows: 60 were used for training, and the remaining 20 were used for validation. After these 10 subjects were randomly selected from the original data set, the remaining 10 subjects were used as the test data set.The experimental platform was an Ubuntu 64-bit PC with a 2.0 GHz Intel CPU and a 48 GB RAM. The GPU was NVIDIA RTX 2080Ti with 11 GB of memory. The programming environment included Anaconda 5.0.1 (Python 3.6), TensorFlow1.4 and PyTorch1.2.0.Because it is difficult to directly compare the difference between meshes, 90,000 points are sampled on the reconstructed triangular mesh and mesh labels. The difference between the two obtained point clouds is then used to represent the mesh difference. The point cloud sampled on the reconstructed triangular grid is called the predicted point cloud, while those sampled on the corresponding grid label are called the label point cloud. The evaluation metrics used in the experiments include the Chamfer distance (Chamfer), the normal consistency  and the 1τ score .To calculate the score, the precision The implementation of the proposed model basically followed the backbone of the PointNet++  and PoinBesides the parameters needed to be learned during the training, there are two key parameters  reconstruction algorithm ,31, the The Poisson surface reconstruction algorithm is a famous implicit-function-induced reconstruction approach that solves for an approximate indicator formulation of the inferred solid whose gradient best matches the input normals. The output scalar function, represented in an adaptive octree, is then iso-contoured using adaptive marching cubes. It is a representative of fitting-based 3D reconstruction methods in our experiments.The HRBF surface reconstruction algorithm is another well-known implicit-function-based method for reconstructing surfaces from scattered Hermite data points. A recent improved version is a closed-form formulation to construct HRBF-based implicitly by a quasi-solution to approximate the exact solution . It is tt-test at a 95% significance level. The reconstruction effect of the proposed method is better than that of MC and Poisson models in the five metrics. It closes to that of Poisson in terms of the normal consistency , and HRBF in terms of the Chamfer distance (Chamfer), The proposed method includes multiple functional modules such as deformation, point deep feature extraction and residual graph convolution networks. To verify the rationality and functionality of each module, three variant methods were built by removing the deformation, deep feature learning and residual connection in ResGCNet modules independently. Three ablation experiments were then carried out under identical experimental conditions.The reconstruction results by removing residual connections are similar to those obtained from the full model, but there are still some differences. When deepening the network structure, the use of residual connections can prevent exploding gradients. In image data processing with regular structure, deepening the network structure can increase the receptive field of a pixel and make the underlying features mapped by the pixel more abundant so as to obtain better results. Here, by deepening the network structure, the features represented by each vertex in the triangular mesh are more abundant, and more structural features of the underlying triangular mesh are mapped to obtain better reconstruction results. Meanwhile, a graph residual convolution module is added to the network structure to avoid gradient explosion.The proposed method can be coarsely regarded as an improvement of initial iso-surface mesh by designing an iterated residual graph convolutional neural network to drive the mesh deformation. To investigate the rationale of the proposed mesh deformation, a qualitative experiment was further conducted to evaluate the difference between the initial triangular mesh and the final mesh. Considering that the ground truth of reconstruction is difficult to obtain, their distances were directly computed as a form of histogram and displayed as different colors on the final mesh.The proposed method consists of several parts such as segmentation, Cubify, iterated deformation and different parts that require different time to implement. During the experiment, the computational cost of each part is performed under the same experimental conditions as much as possible. The results of average test time by compared methods are reported in As can be seen from the table above, the reconstruction speed of the proposed algorithm achieved a leading advantage with 2614 ms and the iterated deformation taking about 56.5% of the time. In particular, the Cubify initial reconstruction takes less than one second than the improved MC algorithm that takes about 6200 ms. In traditional methods, HRBF achieves more accurate performance than MC and Poisson, but with a computational cost of nearly 59,000 ms, significantly longer than MC’s 6200 ms and Poisson’s 2800 ms. The main reason for this expensive time cost is due to the HRBF interpolant. The performance can be improved by using the compactly-supported RBFs that change the global interpolation effect of the interpolant . In contIn the vertex feature learning module, a feature extraction structure similar to PointNet++ is employed. Due to the difference in the structure of the left myocardium, there may be uneven density distribution for the point cloud formed by the vertices of the input triangle mesh, which causes the important features of some key points to be lost during the processes of sampling and grouping. In the future, adaptive density learning strategy could be introduced to assign more suitable scale neighborhoods to key points in the grouping process so as to achieve better feature extraction results. This will be beneficial in guiding the deformation of the triangular mesh in subsequent steps to achieve better 3D reconstruction results of the left myocardium.This paper presents a deep learning method for personalized 3D reconstruction of the left myocardium from MRI images. To better handle the variety across individuals, the algorithm directly obtains the corresponding initial triangular mesh from the 3D segmentation results of left myocardium images. Then, a residual graph convolutional neural network is designed to refine the initial triangular mesh in an iterated manner. The learned vertex features that encode the local region are incorporated to the mesh deformation network at each iteration so as to assist the deformation towards the right direction. As a result, the reconstructed surface is a geometrically valid model meeting the requirements of personalized left myocardium. Experimental results validate the performance of the proposed approach with quantitative and qualitative comparison to some state-of-the-art algorithms.It should be noted that the quality of the left myocardial segmentation results can directly affect the performance of the final reconstruction. Although the powerful deep-learning-based segmentation approach is employed, the segmentation and reconstruction are separated processing steps in the proposed model. In future study, the 3D segmentation and 3D reconstruction of the left myocardium will be integrated as a whole to improve the reconstruction results.


EEG decoding based on motor imagery is an important part of brain–computer interface technology and is an important indicator that determines the overall performance of the brain–computer interface. Due to the complexity of motor imagery EEG feature analysis, traditional classification models rely heavily on the signal preprocessing and feature design stages. End-to-end neural networks in deep learning have been applied to the classification task processing of motor imagery EEG and have shown good results. This study uses a combination of a convolutional neural network (CNN) and a long short-term memory (LSTM) network to obtain spatial information and temporal correlation from EEG signals. The use of cross-layer connectivity reduces the network gradient dispersion problem and enhances the overall network model stability. The effectiveness of this network model is demonstrated on the BCI Competition IV dataset 2a by integrating CNN, BiLSTM and ResNet  to decode motor imagery EEG. The network model combining CNN and BiLSTM achieved 87.0% accuracy in classifying motor imagery patterns in four classes. The network stability is enhanced by adding ResNet for cross-layer connectivity, which further improved the accuracy by 2.0% to achieve 89.0% classification accuracy. The experimental results show that CLRNet has good performance in decoding the motor imagery EEG dataset. This study provides a better solution for motor imagery EEG decoding in brain–computer interface technology research. Brain–computer interfaces (BCIs) are systems that enable communication between the brain and a device by decoding the electroencephalogram (EEG) signals captured from the brain during mental activities and then generating commands for the device ,2. TheseElectroencephalogram (EEG) is a time-varying bioelectrical signal that exhibits characteristics such as temporal variability, low amplitude and high randomness, with a frequency spectrum ranging from 0.5 to 47 Hz. When there is brain activity, electrode recordings can capture the potential changes generated by a large number of neurons. These potential fluctuations over time are commonly referred to as EEG traces. EEG traces reflect the electrical activity of cortical cells in the brain and exhibit complexity and diversity, representing different brain functional states. Motor imagery EEG (MI-EEG) refers to the brain signals generated when an individual imagines body part movements without actually performing them. Compared to traditional EEG signal studies, MI-EEG is better able to reflect people’s intentions and behaviors, and further research on MI-EEG is of significant importance for exploring the neural basis of motor imagery and understanding the underlying neural mechanisms of motor imagery.The detection of MI-EEG relies on highly sensitive EEG devices and corresponding data analysis techniques. In recent years, there have been significant advancements in the detection and application research of MI-EEG signals, although certain aspects still present challenges. MI-EEG has a low signal-to-noise ratio due to the extremely weak electrical signals obtained by the electrodes when measuring brain neural activity. It is also susceptible to interference from other bodily signals, which necessitates the use of filtering techniques to mitigate the effects of unrelated signals . FurtherIn recent years, deep learning algorithms have developed rapidly and competitively in the field of algorithms, with great success in speech sequence modeling, image classification and video tracking . Deep leSome researchers proposed separating the channels of CNN to encode multichannel data and then connecting the encoded features back to the recognition network to perform the final MI-EEG recognition task . It was (1)Improving algorithmic models: The study of feature extraction algorithms and classification algorithms with better noise immunity is one of the goals to achieve the improved classification accuracy of MI-EEG signals. Better feature analysis can be achieved by using automatic feature extraction methods based on neural networks. Meanwhile, traditional classification models such as Bayesian networks and support vector machines require a large number of training samples to obtain more satisfactory classification results. The algorithmic model to improve the classification accuracy of motion images using limited samples is a key research direction at present.(2)Algorithm combination: Each algorithm has its own advantages and disadvantages. Multiple algorithms can be combined  to leverage the advantages of each algorithm to obtain more comprehensive feature information, thus providing more predictive interpretation and improving the accuracy and generalization of classification. Moreover, since different algorithms have different error responses when faced with the same data, the combination of algorithms can improve the overall model stability and facilitate the extension to new dataset applications.For the data processing stage of MI-EEG, many effective methods have been proposed by scholars. However, due to the limitations of the data and the characteristics of the EEG signal itself, bottlenecks have been encountered in the current stage of research, and it is difficult to improve the classification results of MI-EEG signals to a large extent. It has been shown that deep learning methods such as CNN and RNN have good results on MI-EEG classification, and the decoding of MI-EEG signals can be better achieved by fusing multiple deep learning algorithms into one network and taking advantage of each network. In terms of classification, research on MI-EEG signals needs to be further improved in the following aspects:Based on this, this study proposes a combination of CNN and BiLSTM and ResNet, i.e., CLRNet is used for decoding MI-EEG. CLRNet is a hierarchical end-to-end network structure in which a convolutional neural network (CNN) is used to find the most informative linear subspace in the MI-EEG signals by a layered end-to-end network structure. A special recurrent neural network (RNN) with a bi-directional long short-term memory network (BiLSTM) was then developed as a regression algorithm to capture temporal dynamics. For the long sequence feature information captured by the BiLSTM network, a ResNet cross-layer connection was used to purposefully enhance the data processing capability of the network. This study compares the CLRNet model with various other methods on publicly available datasets and finds that the model constructed in this study performs well on the BCI Contest IV dataset 2a.Neural networks have powerful nonlinear fitting and self-learning capabilities. Compared with traditional methods, neural networks have a powerful network structure and are able to omit the step of manual feature extraction. They have excellent performance in the feature extraction part of network training and in handling classification and regression problems. CNNs reduce the complexity of the feedback network and enhance the generalization ability of the network through unique structural designs, such as locally shared weights and neural-network-based pooling. CNN can extract the rich linear subspace of the original signal and update the weights using a back propagation algorithm, which is one of the main advantages of CNN.RNN is used to extract temporal patterns and build a sequential model of the time series. In this model, past data iterations predict future inputs, while future iterations infer past interactions. This type of network is applied in RNN variants of LSTM and gated recurrent neural (GRU) networks. LSTM is an improved version of RNN that increases the unit structure and reduces the weight of unimportant information. It effectively improves the learning efficiency of long sequence data and avoids the gradient disappearance and gradient explosion problems that RNN usually faces. For an input vector In the above equation, Ideally, the fitting power of a neural network increases with the number of network layers. However, in practice, the cumulative multiplicative effect of the backpropagation algorithm leads to problems in error gradient transfer between shallow and deep networks, resulting in reduced gradients that may disappear or explode. As a result, not enough feature information is extracted to obtain better performance. This is the phenomenon of underfitting as well as overfitting in the network. To solve these problems, ResNet was proposed as one of the classical network structures in the field of deep learning. The innovation of ResNet is the introduction of residual blocks, which combine the outputs of different network layers by spanning connections to form a residual mapping. This design ensures that the gradients can be passed efficiently even if the network is deepened, making the network easier to train and optimize. At the same time, the spanning connection also maintains the flow of gradient information from the previous layer, effectively avoiding the problem of gradient disappearance and dispersion. Its basic structure is shown in In The convolutional layer in CNN performs spatial filtering operations on the input MI-EEG signals, optimizing the convolutional effect by adjusting various parameters of the CNN. A pooling layer is added after each convolutional layer to downsample the data, followed by another convolution to obtain deeper features. To mitigate overfitting, dropout regularization is employed. Considering the characteristics of MI-EEG, constructing a CNN structure suitable for processing its specific data format can facilitate feature extraction and dimensionality reduction of EEG. The CNN network structure in this study consists of a total of six layers, including four convolutional layers and two pooling layers. The specific structure is illustrated in The details of each network layer structure in Input layer: The MI-EEG data format of 22 × 288 × 240 is used as the CNN network input.Convolutional layers: The first two convolutional layer filters are set to 128, the convolutional kernel size is set to 3 × 3 and the number of filters in the last two convolutional layers is set to 256. The size of the convolutional kernel is set to 5 × 5 and the step size is 1. The features of the input data are obtained by sliding the convolutional kernel window according to the step size and are filled with the same after each convolution.Pooling layer: The role of the pooling layer is to downsample the input data and reduce the complexity of the overall model computation. Similar to a convolutional layer, a pooling kernel is used to pool the convolutional output of the previous layer, and a maximum pooling approach is taken to compare all data in the pooled region. The output of the pooling area is the largest of these values, the pooling kernel is set to 3 × 3, the step size is 1 and the same padding is used to fill the same after the pooling operation.Dropout layer: The role of this layer is mainly to prevent overfitting when training the network, and a dropout rate of 0.5 is used to improve the overall generalization ability of the network.Fully connected layer: The fully connected layer is the last layer of the CNN, and the role of this layer is to integrate the features obtained from the previous network through multiple convolution and pooling. The extracted features are mapped to the sample data to expand the differentiation of the data, and the purpose of classification is achieved using the softmax function.In CLRNet, the structure and parameters of CNN are introduced. The network parameters and structure of BiLSTM as well as ResNet are shown in In CNN performs well in the space domain feature extraction ability, and LSTM can effectively extract information in the time domain. The network model combining CNN and BiLSTM can decode MI-EEG signals more effectively. When the number of layers of the network increases, gradient disappearance and overfitting problems occur. Therefore, there is still room for improvement in the structural optimization of deep networks. In this case, adding ResNet is a very effective way to improve the performance of the network. Enhancing network generalization by adding direct mapping between the outputs of BiLSTM feature vectors using ResNet’s cross-layer connectivity in BiLSTM further enhances the performance of the network in MI-EEG signal decoding tasks.In the decoding of MI-EEG, preprocessing is necessary to extract the required electroencephalogram data. In this study, we designed a network model that takes preprocessed EEG data as the input for feature analysis and validates four types of motor imagery states, as shown in When analyzing MI-EEG, it is necessary to continuously explore network structures suitable for its decoding. As both CNN and LSTM have shown promising results in MI-EEG processing, this paper makes improvements on their base network architectures to further enhance the decoding performance of MI-EEG. The proposed CLRNet network structure in this study consists of multiple different network layers. The first six layers are CNN layers used to capture the spatial information of MI-EEG signals, while the following five layers are BiLSTM layers used to capture the temporal information of MI-EEG signals. In the BiLSTM layers, ResNet’s cross-layer connections are incorporated for residual processing, enhancing the network’s data processing capability and stability, and improving its performance. The overall network model combines spatial convolutional transformations, time-frequency analysis and feature selection.When comparing different network models, the training and testing datasets used for the network models under study are often different. Differences in datasets may lead to differences in the accuracy of network models, resulting in some network models outperforming others on some datasets. To address this issue, this study uses the motor imagery EEG dataset commonly used by researchers as the BCI Competition IV dataset 2a. The dataset is obtained from experiments performed on nine experimenters with a sampling rate of 250 Hz, which has been processed using a band-pass filter of 0.5–100 Hz. The dataset contains 25 channels (22 EEG and 3 EOG) and is a four-class MI  EEG dataset. The paradigm of the dataset is shown in The first 2 s of the paradigm in the figure is the preparation time. A cue is given at the 2 s mark. From 3 s onwards, subjects performed imaginary movements until the 6 s mark.In the whole MI-EEG signal acquisition experiment, the experimental data are recorded throughout. According to the introduction of the experimental paradigm, the duration of motor imagery is from 3 s to 6 s. Here, 3 s of the motor imagery segments is extracted via segmentation as the data for this experiment. It is filtered from 8–30 Hz and the effects of the oculomotor signals from the ‘EOG-left’, ‘EOG-central’ and ‘EOG-right’ channels are removed at the same time.After the operation of filtering and segment extraction of the MI-EEG data, the MI-EEG signals format of a single subject is 22 × 288 × 750. However, inputting the data into the CLRNet network model to verify the performance of the model revealed that the classification accuracy obtained is low. After exploration and experimental validation, it is determined that the initial MI-EEG signal features are relatively weak. It is difficult to extract effective features directly through the CLRNet network model, so the results obtained after feature extraction and classification are weak. To address this problem, a step of simple wavelet packet decomposition is added to the preprocessing stage and good results are achieved. The specific operation is a five-layer wavelet packet decomposition for a sampling rate of 250 Hz. The band range of each node in layer 5 is 0–7.8125 Hz, and the band ranges of nodes 2 and 3 are 7.8125–15.625 Hz and 15.625–23.4375 Hz, respectively. Node 2 and node 3 associated with the Mu (8–14 Hz) and Beta (16–24 Hz) rhythms of motor imagery are selected for signal reconstruction. The format of the reconstructed signal data is 22 × 288 × 240, where 22 represents the number of channels, 288 represents the number of trials and 240 represents the sampling points after data processing. This data format is used for experimental validation in the following experimental procedures.The CLRNet network model proposed in this study is tested on the BCI Competition IV dataset 2a. The input EEG data format is 22 × 288 × 240. The size of the batch training is set to 8. The number of iterations is set to 200. The Adam optimizer is selected to update the learning weights of the network model, and four categories of motor imagery patterns  are classified. The dataset used in this study is the BCI Competition IV dataset 2a, which contains data on a total of nine subjects. The nine sets of pre-processed data are validated by the model separately, and the accuracy rate is used as the evaluation index of the classification results. The obtained classification results are shown in The final classification accuracy for each subject is given in In the decoding of MI-EEG signals, feature extraction plays a very critical role, and channel selection is a special feature selection method. Channel selection is closely related to the physiological context in which the MI-EEG signal is located. By removing channels that contain noise and redundant information, the amount of data that needs to be processed can be reduced and the data processing capacity of the BCI system can be improved. Also, adding more valuable channels can improve the signal-to-noise ratio of the system and increase the decoding performance of the BCI system. Therefore, the role of channel selection is very important in the MI-EEG signal decoding task. It can improve the signal quality and interpretability by reducing noise and redundant information and optimize the training and testing accuracy of feature extraction and classifier. In turn, the overall performance of the BCI system is improved. C3 and C4 are considered to be the most relevant channels to MI-EEG . In ordeIn addition, the model proposed in this study contains several commonly used networks. In order to verify that the effect of the combination can obtain the advantages of each network, the CLRNet model proposed in this study is subjected to controlled variable experiments using CNN, BiLSTM and CNN-BiLSTM networks for the BCI Contest IV dataset 2a, respectively. The performance of each network is verified, and then ResNet is added to compare and evaluate the performance of each network model, as shown in For non-stationary MI-EEG signals, CNN and BiLSTM have good performance for feature extraction and classification of MI-EEG signals. Their average classification accuracies reached 83.0% and 76.2%, respectively. It proves that spatial and temporal feature information is an important basis for decoding MI-EEG signals. CNN-BiLSTM combines the advantages of both networks and provides excellent performance in processing data containing spatio-temporal information. The average accuracy of the classification is 87.0%, and the overall performance is stronger than that of the individual networks. The CLRNet model proposed in this study combines the features of three networks, CNN, LSTM and ResNet. CNN can effectively extract the spatial features of the signals; BiLSTM can handle sequence information very well. ResNet networks can alleviate the gradient disappearance problem in deep neural networks and reduce the complexity of network training. The experimental results show that the combined approach of the CLRNet model can give full play to the advantages of each network and realize the improvement of classification accuracy of BCI race IV dataset 2a. It is further demonstrated that the CLRNet model proposed in this study is an effective method for MI-EEG signal decoding, and it also shows that the multi-network fusion model is an important research direction for MI-EEG signal processing.To demonstrate the excellent decoding performance of the CLRNet network model for MI-EEG in more depth, this paper is compared with MI-EEG classification methods proposed by other researchers. All comparison trials are conducted on the dataset of each subject in the BCI Competition IV dataset 2a. The same classification accuracy is used as an evaluation metric. The comparison results are shown in The four MI-EEG classification methods included in this paper are compared separately in In this research work, a network consisting of CNN, BiLSTM and ResNet is proposed to decode MI-EEG. The main elements of this study are as follows: A hierarchical network is proposed, combining CNN and BiLSTM with ResNet. CNN is used to obtain the null domain features from the original MI-EEG, and BiLSTM is used to obtain the time domain information of the MI-EEG time series. To avoid network degradation, a residual network is used to perform residual processing on the output of BiLSTM to enhance the data processing capability of the model. The CLRNet model presented in this study is evaluated using a publicly available motor imagery dataset to demonstrate the validity and utility of the model. The experimental results demonstrate the ability of CLRNet to effectively reduce the long-term non-smooth effects of EEG sequences and improve the robustness and accuracy of EEG-based motor imagery classification models. The model may be used to develop MI-BCI applications in the future.For neural networks, the settings of network parameters are not fixed, and different parameters may bring results with large disparities. In order to achieve better MI-EEG decoding, there may be room for improvement for the CLRNet network model architecture parameters proposed in this study. This requires a lot of time for tuning and comparison, and this is the area that needs to be focused on and studied in the future.


An optical multiple-image authentication is suggested using computational ghost imaging and total-variation minimization. Differing from encrypting multiple images into a noise-like ciphertext directly, as described in most conventional authentication methods, the related encoded information is embedded into a cover image to avoid the attention of eavesdroppers. First, multiple images are encoded to form real-valued sequences composed of corresponding bucket values obtained by the aid of computational ghost imaging, and four sub-images are obtained by decomposing the cover image using wavelet transform. Second, measured sequences are embedded into one of the sub-images, and embedding positions are randomly selected using corresponding binary masks. To enhance the security level, a chaotic sequence is produced using logistic map and used to scramble measured intensities. Most importantly, original images with high quality can be directly recovered using total-variation minimization. The validity and robustness of the proposed approach are verified with optical experiments. Among tAs an intriguing technology, computational ghost imaging can provide the potential for optical information security due to its significant property, i.e., two-dimensional signals can be recorded using a bucket detector. Meanwhile, a set of real-valued measurements are considered as the ciphertext, the channel burden of transmitting secret information can be efficiently reduced. Therefore, how to apply computational ghost imaging in cryptographic systems has received more and more attention recently . With thIn the presented article, an optical authentication is suggested for multiple images based on total-variation minimization, where binary masks also are used to choose the necessary information of each image. Instead of using traditional ghost imaging, speckle patterns to illuminate the original image can be generated with a computer in computational ghost imaging, which can efficiently reduce the complexity of optical setup. Differing from embedding the sparse information, multiple original images to be authenticated are first encoded into real-valued sequences composed of measurements using computational ghost imaging in this method, and then these measurements are embedded into one of sub-images decomposed from the cover image using the discrete wavelet transform. In this process, random binary masks are generated to choose the embedding positions in the sub-image. Two kinds of algorithms are used to reconstruct original images. One is to calculate the correlation between measurements and speckle patterns, where recovered images have a lot of noise. Another is to use the optimization such as total variation minimization by augmented Lagrangian and alternating direction algorithms (TVAL3), where measured intensities and corresponding patterns are directly input into optimization process to reconstruct original images. Also, the sub-image containing measured intensities is scrambled using the chaotic sequence produced with logistic map, and random binary masks are adopted as secret keys.The rest of this article is arranged as follows. In Section 2As an efficient approach, space multiplexing is widely used to implement the goal of authentication for multiple original images, where the individual information is integrated into the ciphertext . For an The vulnerability of conventional DRPE is an insurmountable problem. To deal with this problem, Deepan et al.  proposedSpatial multiplexing is widely used in multiple-image encryption and authentication because it can efficiently suppress the silhouette problem, especially for optical methods using interference. Wang et al.  proposedAs it is recognized by more and more researchers, using additional mechanism of information hiding in the encryption and authentication processes can further enhance the security level. Sui et al.  carried 3In this part, comprehensive details are given about the suggested authentication scheme based on total variation optimization. The diagram of the embedding process with the help of the discrete wavelet transform is shown in In the first step, the optical schematic arrangement as shown in In the second step, the cover image is indicated as  decades . For theIn the last step, some coefficients in the suitable sub-image are chosen and fused with all measured intensities of original images to be authenticated. According to most traditional multiple-image authentication methods, the final ciphertext image is noisy. Differing from these method, all measured intensities of original images are embedded into the cover image in the proposed method. The marked cover image has high visual quality, which can avoid being noticed by saboteurs to a certain extent. Therefore, there are two competing requirements to be satisfied in the proposed method. One is the perceptual transparency of measured intensities, where the cover image cannot be destroyed with the embedding of these information. The other is to make the marked cover image robust, where some common attacks such as noise attack can be efficiently resisted. To satisfy the above requirements, some coefficients in the sub-image It should be pointed out that, to improve the security of multiple-image authentication, the sub-image containing the information of measured intensities is confused using the chaotic sequence generated with logistic map, where the related parameters are served as secret keys with high sensitivity. Besides that, a very low number of measurements are embedded into the cover image for enhancing the security. Although no valid information is observed from its reconstruction, the corresponding authentication of this image still can be efficiently implemented. As the amount of measurement increases, the original image will be gradually reconstructed. At this moment, multiple-image encryption can be implemented. Furthermore, to solve the problem that the reconstructed result does not have high visual quality, the optimization algorithm , such asAs an effective approach to reduce the influence of noise in image reconstruction, TVAL3 has been widely used in the field of digital imaging. The isotropic TV model is applied to reconstruct an original image by considering its measured intensities as the observation, which is mathematically expressed as .(7)minuTThe corresponding augmented Lagrangian problem can be described asTo solve the above problem of Eq. essed as (10)T2=[1Once the matrix with the given order is obtained, a two-dimensional structured pattern with (1)For each original image, a total of (2)Applying the Haar wavelet transform, four sub-images as described in Eqs. (3)Randomly generating (4)Denoting Based on the above discussion, the embedding and the authentication process can be summarized in detail. Let (5)Giving the related parameters Obviously, original coefficients of the sub-image are directly substituted with measured intensities in the proposed method. In this way, the fused sub-image is obtained by Eq. Sorting (6) According to (1)The marked cover image (2)Besides the integer (3)Scanning the secret key (4)To reconstruct the original image, one intuitive way is to calculate the correlation of recovered measurements as described in Eq. (5)Furthermore, the nonlinear correlation map between the noisy recovered result and the original image is calculated to implement the authentication, which is mathematically expressed as .As shown in In addition, when the number of recovered measurements is large, the utilization of the Gaussian filter can enhance the reconstruction performance. In the proposed method, the reconstructed image is processed with the filter with the size of 4A series of optical experiments are executed based on the optical setup as shown in To quantitatively evaluate the performance, some important indexes such as peak signal-to-noise ratio (PSNR) and correlation coefficient (CC) are calculated. Among them, the PSNR between the cover image and its marked result is used to analyze the imperceptibility of the information of original images. If the PSNR between the cover image and the marked cover image is higher, the visual quality of the marked cover image is better. Thus, the marked cover image almost is the same as original one and the sparse information embedded will not be noticeable. Usually, it is enough that the PSNR value is larger than 30 dB. Let The CC between an original image and its reconstructed result is used to evaluate the effectiveness of the proposed method, which plays a crucial role in analyzing the visual quality of the recovered image as well as the key sensitivity of the cryptosystem. Usually, the CC value can represent the degree of consistency between the original image and its reconstruction. Due to the limitation of the mechanism of imaging, the reconstructed image will be very noisy for computational ghost imaging, especially in practical experiments. In order to improve the quality, a larger number of measurements are collected. However, it will seriously reduce the efficiency of imaging. Therefore, the CC value can be used to evaluate the noisy reconstruction efficiently. Of course, the higher the CC value, the better the reconstruction. If the reconstruction is the same as original one, the maximum CC value will be 1. The CC value is mathematically calculated asAs shown in To verify whether the goal of multiple-image encryption can be implemented, all 4096 structured speckle patterns are generated to record measurements of all original images, and these measurements are adjusted by the weight factor and randomly embedded into the sub-image According to an optical method, if only a part of measurements is collected, the efficiency of the cryptosystem can be greatly improved. However, the reconstructed result will have seriously deteriorated. Especially, when the number of measured intensities is very low, the reconstructed result will be noisy. At this moment, the original information will be verified using the correlation map calculated with Eq. As described in the previous section, three kinds of secret keys are used in order to improve the security level. First, as shown in In order to verify the influence of binary masks, it is investigated that once how much data in these masks are discarded, the existence of original images cannot be authenticated. After many comparison experiments, it is found that when the number of discarded pixels of a binary mask is equal to or larger than 16, i.e., more than 0.39% of pixels, the authentication will be failure. In Due to their high sensitivity to slight variation, the related parameters of logistic map are used as secret keys, which can potentially enhance the security. As shown in To verify the robustness, the following analysis are used to investigate its resistance against noise and occlusion attacks. Let the marked cover image 5An optical multiple-image authentication method using TVAL3 and wavelet transform has been proposed in this paper. After multiple images are encoded using computational ghost imaging, the resultant measurements are modulated with the weight factor and embedded into one of the sub-images decomposed from the cover image using the Haar wavelet. The marked cover image has good visual quality, and the imperceptibility of the hidden information is high, which can avoid the attention of eavesdroppers. When the sampling ratio of measurements is large, the goal of multiple-image encryption can be implemented. Otherwise, the authentication can be verified using the nonlinear correlation map. Moreover, the original image reconstructed using the optimization algorithm such as TVAL3 is better than that using correlation computation. Because multiple kinds of secret keys are applied, such as random binary masks and the related parameters of the logistic map, the security level is guaranteed. Besides that the feasibility is demonstrated, and the effectiveness of resisting malicious attacks has been verified through optical experiments.National Natural Science Foundation of China (NSFC): 62031023.Yaoling Zhou; Liansheng Sui: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Wrote the paper.Yueer Sun; Mu Yang; Junzhao Hou: Analyzed and interpreted the data.Zhaolin Xiao: Contributed reagents, materials, analysis tools or data; Wrote the paper.Asundi Anand: Analyzed and interpreted the data; Wrote the paper.Data will be made available on request.The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.