Blood coagulation is a complicated dynamic process that maintains the blood’s fluid state and prevents uncontrollable bleeding. The real-time monitoring of coagulation dynamics is critical for blood transfusion guidance, emergency management of trauma-induced coagulopathy, perioperative bleeding, and targeted hemostatic therapy. Here, we utilize optical vortex dynamics to detect the blood coagulation dynamic process in a rapid and non-contact manner. To characterize the temporal changes in viscoelastic properties of blood during coagulation, we track the stochastic motion of optical vortices in the time-varying speckles reflected from 100 blood samples with varied coagulation profiles. The mean square displacement (MSD) of the vortices increases nonlinearly with time lag during blood coagulation reminiscent of the particles in viscoelastic fluids. The MSD curves with coagulation time are similar to the tracings of thromboelastography (TEG) during the blood coagulation. The retrieved coagulation parameters, such as reaction time and activated clotting time measured using the optical vortex method, exhibit a close correlation to those parameters acquired from TEG. These results demonstrate the feasibility of the optical vortex method for monitoring blood coagulation at the point of care. Our method is also applicable to measuring the viscoelasticity of complex fluids and turbid soft matters. The coagulation system is the body’s defense mechanism for maintaining normal blood flow and preventing blood loss ,2,3. ItsHowever, coagulation is a cascade of complex biochemical processes involving interactions between platelets, red blood cells, and over a dozen coagulation factors, as well as two distinct coagulation pathways—endogenous and exogenous ,9,10. CoCurrently, the primary clinical devices that can monitor the coagulation dynamics process are thrombelastography (TEG) and rotational thrombelastography (ROTEM). They evaluate the viscoelastic properties of a blood sample by applying a continuous oscillation to it and bringing it into contact with a metal wire to assess the formation, stabilization, and ultimately dissolution process ,14,15,16Optical microrheological methods are frequently employed to investigate the rheological properties of soft tissues in a non-contactable manner. According to the generalized Stokes–Einstein relation (GSER), the viscoelasticity of a complex fluid can be obtained by calculating the MSD of the probe particles in it. Currently, optical techniques still need to track the motion of probe particles in surrounding environments, either directly or indirectly. In diffusing-wave spectroscopy (DWS) and dynamic light scattering (DLS), the viscoelastic properties are characterized by analyzing the temporal fluctuations of dynamic speckle patterns, y glands ,21,22,23ed, etc. .The laser speckle has a generic structure known as the optical vortex, which exists at an intensity of zero . The reaIn this study, we track the random motion of optical vortices to evaluate the viscoelastic properties of blood samples and thus assess blood coagulation dynamics. We investigate the relationship between the temporal change in the MSD of optical vortices and blood coagulation dynamics and retrieve the coagulation parameters, including reaction time (R-time), activated clotting time (ACT), and maximum amplitude of viscoelastic modulus of the thrombus (MA). To demonstrate the capability of the optical vortex method (OVM) for coagulation dynamics monitoring, we compare OVM and TEG results for more than 100 blood samples with different coagulation statuses. Finally, two-dimensional images of the blood viscoelasticity are obtained by scanning the surface of the blood sample at different stages of coagulation.We prepared over 100 whole blood samples with different coagulation statuses. The citrated whole blood was collected in a plastic bottle with 3.8% buffered sodium citrate with a citrate-to-blood ratio of 1:9. The heparinized whole blood was collected with 1% buffered sodium heparin with a heparin-to-blood ratio of 12.5 U/mL. After incubating the citrated blood for 5 min under 37 °C, 900–1300 µL of blood was transferred into vials pre-loaded with a kaolin buffer solution . After 3 gentle vial inversions, 30–70 µL of 0.2 M calcium chloride  was added to the kaolin vial for activating coagulation. After another 3 gentle vial inversions, ~120 µL of kaolin-activated whole blood was immediately pipetted into an imaging chamber; in the meantime, an additional 360 µL of blood from the same tube was loaded into the sample cup of TEG . The coagulation measurements were taken simultaneously in the OVM and TEG. The porcine blood samples exhibited a broad distribution of coagulation parameters, due to the difference in the type of anticoagulants, the ratio of the amount of blood to the amount of calcium chloride, and the storage time of the blood.The optical experimental setup shown in 2 with a motorized compact linear stage .To demonstrate the capability of the OVM for monitoring micro thrombosis, we performed the spatiotemporal analysis of the blood viscoelasticity during blood coagulation. The blood samples were scanned over an area of 1.7 × 1.7 mmDynamic laser speckles are formed when the incident light is scattered by the moving scatterers in the whole blood, such as red blood cells, leukocytes, platelets, and fibrin monomers. For locating the optical vortex and reducing the complexity of the optical system, the captured speckle sequences are first converted into the complex analytic signal by Laguerre–Gaussian transform to obtain pseudo-phase ϕ^ :(1)I˜:t ≤ 20 min , confirming the accuracy of measurement to clotting time. The OVM parameters R-time and ACT bore a linear relationship with the corresponding TEG values with correlation coefficients, R, of 0.79 and 0.73, respectively. In contrast, for the MA parameter, a significant correlation between the OVM and TEG was not found in the experiment. Many possible factors led to the deviation, one of the most likely being due to differences in the moduli measured for clots formed under quiescent conditions by the OVM, versus agitation conditions of mechanical strain by TEG, as discussed below [In ed below .We also analyzed the spatiotemporal behavior of optical vortices to present the spatial distribution of viscoelastic properties of blood during coagulation. The two-dimensional color maps of MSD in In this study, we have proposed a novel method to monitor the dynamics process of blood coagulation by gauging the viscoelasticity of the blood from the stochastic motion of the optical vortex. The results show that the dynamic behavior of the optical vortex under restrictions during blood coagulation is similar to the scattering particles in a viscoelastic medium. Moreover, comparative experimental results of the OVM and TEG demonstrated the capability of the OVM to evaluate blood coagulation status. Blood coagulation parameters such as R-time and ACT measured by the OVM showed a strong correlation with TEG. The time required for global coagulation profiling by the OVM is significantly less than that required by TEG when blood is activated with the same coagulation activator, as shown in Laser speckle is a granular intensity pattern produced by the interference of coherent light undergoing multiple scattering from the randomly distributed particles . Laser sIt is thus possible in principle to characterize the interaction of the wave through the sample by tracking the displacement of specific features of the speckle pattern in time ,40. As tHere, we investigate the accuracy and sensitivity of the method to estimate the coagulation status of whole blood samples. In unclotted blood, RBCs and platelets are the most dominant scattering particles, with random and rapid motion, maintaining stable optical properties. When coagulation is initiated, soluble fibrinogen is transformed into insoluble fibrin and polymerizes with activated platelets and RBCs, which leads to scattering particles undergoing restricted Brownian excursions. The displacement of scattering particles defined by the MSD is related to the viscoelastic modulus of the surrounding environment via the GSER as follows ,44:(6)|Gsμ, which evolves continuously as coagulation progresses in the blood sample. To avoid omitting many parameters, the MSD of actual scattering particles is represented by the MSD of the optical vortex. As a virtual particle, it makes little sense to measure its average radius. Its MSD can be calculated directly by the method mentioned above and used to characterize the viscoelasticity of the medium, which is verified in the previous study.However, the average radius of the light scattering particles changes continuously with the blood coagulation. It is difficult to obtain the quantified values of the viscoelastic modulus using optical methods based on microrheology. Furthermore, the methods require measurement of the scattering coefficient, To calculate the MSD of the optical vortex accurately, the acquisition frame rate should be selected suitably. Due to the rapid change of laser speckle during blood coagulation, the low acquisition frame rate is unable to capture the trajectories of the optical vortex. In this study, the high frame rate of 800 frames per second was chosen for capturing the rapid speckle fluctuations induced by a cascade of coagulation processes. Although higher frame rates are theoretically more useful for capturing the trajectory of the optical vortex, the acquisition frame rate is limited by some hardware parameters such as spatial resolution and exposure time. Small spatial resolution may cause an insufficient number of the optical vortex to be ensemble averaged, resulting in unreliable MSD results. Exposure time directly affects the contrast of the image, which is important for the location of the optical vortex. While ensuring sufficient contrast in the image, the exposure time can be reduced by increasing the gain value, thus allowing for a higher frame rate, but this causes more noise and reduces the SNR. A significantly reduced SNR could cause trouble in locating and pairing the optical vortex. Thus, the balance between frame rate and other hardware parameters is essential for the accurate calculation of the optical vortex’s MSD.The captured speckle intensity distribution is represented by real values, while the optical vortex is located in the corresponding phase map. It is common practice in physics and engineering to use complex-valued signals to represent the real-valued signals. A true phase map cannot be obtained by using the simple optical device in this study. We have demonstrated that the MSD of the optical vortex calculated by using pseudo-phase has a strong correlation with it by using real phase in the previous simulation experiments. Therefore, the Laguerre–Gaussian transform was subsequently applied to the 2-D speckle pattern, acquiring the pseudo-phase according to Equation (3) in this study. Several reports have shown that the Riesz transform and Hilbert transform could also be used for obtaining the pseudo-phase, but the Laguerre–Gaussian transform with some irreplaceable advantages was chosen eventually for this study. It can readily be seen from Equation (2) that the amplitude of the Laguerre–Gaussian function is doughnut-like, which can suppress high-frequency components of unstable phase singularities, thus serving as a band-pass filter. As the bandwidth of this filter, t = 0.04 s. Moreover, in the initial stage of coagulation reaction (2–3 min), the existence time of the optical vortex trajectories becomes shorter due to faster fluctuations of laser speckle. Here, the MSD values at the moment of 0.02 s were chosen eventually to indicate clot viscoelasticity for the evaluation of the whole coagulation process. The coagulation parameters, R-time and ACT, which were defined by the OVM presented a strong correlation with corresponding TEG values, while the MA parameter showed a weak correlation. It may be explained by the differences in the measurement mechanisms of the two methods. The OVM is an optical method that can measure the blood viscoelasticity without mechanical contact during coagulation, causing less mechanical disruption of the clot. In contrast, in TEG, oscillatory shear stress is applied for measuring the viscoelasticity throughout the whole coagulation process, disrupting the clot structure during polymerization and stabilization, thereby having an obvious impact on MA. Furthermore, the MA parameter is related to stiffness measured for clots formed under quiescent conditions, which is connected with the amount of calcium chloride. In the OVM, t = 0.02 s, corresponding to a high shearing frequency of The annihilation of optical vortices resulting from the collision of positive and negative vortices is a disruption for the statistical calculation of MSD. The random distribution property of the optical vortex originating from the random field of laser speckle could be the main cause of this problem ,46. DuriThe studies described here are mainly limited by the variations of optical properties during blood coagulation. Optical vortex behavior in dynamic laser speckles is sensitive to the random motion of the scattering particles, and therefore related to the viscoelastic properties of the surrounding medium. However, speckle fluctuations are influenced not only by sample mechanics, but also by changes in optical properties of blood, the reduced scattering coefficient rheology ,23. AnotIn the past, devices such as TEG and ROTEM have been investigated for efficient detection of coagulation at the point of care. In these methods, a cylindrical cup containing a whole blood sample oscillates continuously and a pin on a torsion wire is suspended in the blood. The torsion wire is a sensor that can transmit the changes in blood viscoelasticity to an electromagnetic transducer, implementing a coagulation detection function . HoweverIn this study, we validated the capability of the OVM for the non-contact and rapid monitoring of blood coagulation dynamics at different stages of coagulation, using only a few drops of whole blood. The low cost and compact setup that can be easy to operate without special treatment is also a noteworthy advantage for assessing a patient’s blood coagulation status at the bedside. In the future, clinical experiments will be conducted for further correlation studies to validate the potential as a miniaturized medical device for coagulation testing. Our technology extends the optical micro-rheology to turbid fluids like blood by using optical vortices as virtual probes instead of real particles. We believe that our approach can find applications in the viscoelasticity measurement of various complex fluids and biological tissues.


The seriously degraded fogging image affects the further visual tasks. How to obtain a fog-free image is not only challenging, but also important in computer vision. Recently, the vision transformer (ViT) architecture has achieved very efficient performance in several vision areas.In this paper, we propose a new transformer-based progressive residual network. Different from the existing single-stage ViT architecture, we recursively call the progressive residual network with the introduction of swin transformer. Specifically, our progressive residual network consists of three main components: the recurrent block, the transformer codecs and the supervise fusion module. First, the recursive block learns the features of the input image, while connecting the original image features of the original iteration. Then, the encoder introduces the swin transformer block to encode the feature representation of the decomposed block, and continuously reduces the feature mapping resolution to extract remote context features. The decoder recursively selects and fuses image features by combining attention mechanism and dense residual blocks. In addition, we add a channel attention mechanism between codecs to focus on the importance of different features.The experimental results show that the performance of this method outperforms state-of-the-art handcrafted and learning-based methods. Due to the color distortion, blurring and other quality problems of haze images that affect further information capture, single image deblurring has always been a challenging and highly concerned problem. The deblurring method originates from the classical atmospheric scattering model, and the imaging formula is as follows:I(x) is the degraded image, J(x) is the brightness of the scene when it does not propagate through the water, t(x) is the transmissivity of the propagation medium, β(λ) is the attenuation coefficient of different wavelengths of light, λ represents different color channels, d(x) is the distance between the camera and objects, and A is the ambient atmospheric light of the scene. Many deblurring methods based on imaging models  based on swin transformer. PRnet consists of recurrent block, transformer codecs and supervised fusion modules. First, we have a recurrent block that learns shallow features of input images and introduces a long short-term memory (LSTM) network to connect different iterations, ensuring that more of the original image features can be retained over multiple iterations of the model. The transformer codec then learns the sequence representation of the input image through the u-net structure, and effectively extracts the remote context features from multiple scales of the image. The encoder introduces swin transformer block to encode feature representation from the decomposed patch, and continuously reduces the resolution of feature map for local relationship modeling. Decoder decodes hidden features through convolution and upsampling and realizes dimensional transformation to further predict the semantic output of the global context representation. In addition, we connect the encoders through skip connection and add channel attention. this design can effectively avoid the loss of original features and improve the quality of the output image. Finally, the supervised fusion module combines the attention mechanism and dense residual blocks to recursively select and fuse the image features and transfer the attention-guided features to the next stage, which can effectively preserve the original features of the image and prevent the model from over-fitting. In addition, the whole recursive process under the supervision of the original input image can effectively retain the original resolution characteristics of the image, improve the learning efficiency and defogging performance of the network.To validate our approach, we tested it on different data sets. A large number of experiments and qualitative and quantitative evaluations show that our iterative strategy is beneficial to image restoration and is superior to other state-of-the art methods see . In shorWe introduce the swin transformer into the iterative progressive residual network (PRnet), which obtains sufficient contextual semantic information and spatial features by learning multi-scale feature information of the input image.We introduce channel attention between the encoder and decoder, which makes the module focus on extracting significant useful features related to clean image in the input image.We design a supervised fusion module, which combined the dense residual block with attention to conduct recursive supervised fusion of image features under the supervision of ground-truth.In this section, we will conduct a comprehensive review of fog removal methods and vision transformer relevant to our work. We will conduct a comprehensive review of single image defogging algorithms, including traditional image defogging and deep learning-based image defogging methods.By observing and analyzing the imaging process of fog image and its relationship with clean image, the physical model of atmospheric scattering for fog imaging is established. The model-based method tries to estimate the atmospheric light and medium transmission map using the handmade prior knowledge, and then restore the blurred image. Dark channel prior (DCP) is one of the outstanding representatives of priority-based methods. He et al.  assumed In recent years, a large number of methods based on deep learning have flooded with the field of dehazing. Some deep learning methods still combine physical models or prior knowledge to improve the accuracy of fog removal. Kar et al.  takes thIn addition, some methods take into account the morphological differences of fuzzy images at different scales to extract, transfer and fuse multi-scale image features. For example, Yeh et al.  relies oTransformer was first proposed for machine translation Vaswani et al.  and is wIn this section, we first introduce the cross scales supervisory integration mechanism (CSSI) and then introduce our overall architecture of progressive residual networks. As shown in Our analysis shows that if the encoder and decoder are independent from each other, multi-scale features cannot interact with each other, which will greatly reduce the performance of the model . If featEi and Ci represent the output of the encoder layer and channel attention mechanism respectively. Next, the output feature of the encoder layer is fused with the up-sampling and convolution operation results of the previous decoder layer to obtain the input feature of the next decoder layer:where Di−1 and Di represent the features of the previous and next decoder layers.where CSSI explores the relationship between feature maps of different channels through channel attention, adjusts and aggregates different feature maps in the process of feature interaction, and finally transfers them to the decoder layer. On the one hand, CSSI makes the network pay more attention to find the significant useful information related to the current output in the input data, which can effectively avoid the loss of original features and improve the quality of the output image. On the other hand, CSSI can improve the efficiency of feature fusion and interaction between codecs with different resolutions, effectively reducing the network burden.Swin Transformer interacts with the global information of the image, without considering the importance of the content of the image area and the overall structure of the object, and cannot pay better attention to the structure and details of the image. In order to make up for the above defects, we propose a new progressive residual network (PRnet), which solves the problem of fog removal through multiple stages. At the same time, u-transformer encoder-decoder is used in each stage to learn the morphological features of foggy images at different scales. To avoid the increase and over-fitting of network parameters, different from the previous multi-stage, we do not pile up several sub-networks, but use the recursive calculation between stages to share the same network parameters in multiple stages. In addition, while swin transformer avoids the segmentation edge loss problem, the Transformer image is smaller than the original image resolution. Therefore, ground truth is used to supervise the network, which can suppress features with less information in the current stage and only allow useful features to be transmitted to the next stage.xt−1 from the t−1 iteration and the previous hidden state ht−1. The feature graph it is obtained by convolution, which is used to determine which information is important and needs to be retained. Then feature graphs ft and ot controlling forgotten data were obtained through sigmoid activation function, and then forgetting and remembering were carried out according to the following formula:We designed a Recurrent block in PRnet to learn the shallow features of the input image, and introduced the Long Short-Term Memory (LSTM)  module to perform self-attention computation in the window region without overlap. Because of its hierarchical design and generalization, it has proven its effectiveness in several fields such as object detection, semantic segmentation and image denoising. Therefore, we apply Swin transformer directly in encoder to encode the feature representation from the decomposed patch.H×W into RGB image patches, and marked them as the original pixel Mosaic vector with a size of 4 × 4. It is then mapped to a vector of dimension 4C using linear embedding. At this time, the output dimension is set to 2C and the feature size is set to zl−1 enters two consecutive Swin Transformer Blocks for feature transformation. Unlike MSA in ViT, Swin Transformer Block computes self-attention by adding a relative position bias B to the corresponding head, then the output feature zl−1 of layer l can be written as follows:Our encoder generates different number tokens through three layers of encoder layer. The first, second and third layers generate zl represent the output of MLP.where Corresponding to the encoder, a symmetric decoder is constructed based on the swin transformer, forming an encoder-decoder based on the u-net architecture.To recover the spatial order, we use a convolution module and upsampling to form a Decoder layer. In the first layer the hidden features are first decoded by bilinear upsampling of the input features  and structural similarity (SSIM). PSNR is a pixel-level image quality evaluation method used to measure the difference of gray values between two images. The higher the PSNR value,the lower the distortion between the evaluated image and the ground-truth image, and the better the quality; on the contrary, the poorer the quality. SSIM is a measure of covariance to determine the degree of structural similarity between images according to the degree of correlation between image pixels. The higher SSIM value, the more structure or color information the image retains, and the better the effect of the resulting image. What's more, we use the scikit-image library of python to calculation them. In addition, since there is no ground-truth image in real-world datasets, we use Fog Aware Density Evaluator (FADE) . Our method recursively invokes the residual network to gradually recover clean images under ground-truth supervision. First of all, PRnet learns the features of the input images through recurrent block, while taking care of connecting the different stages to ensure that more original image features are retained during the multi-stage feature transfer of the model. We design a codec with u-net structure in combination with swin-transformer, which can ensure that sufficient contextual semantic information and spatial features are obtained during long-distance transmission. In addition, CSSI, which can ensure the synergy and connectivity of the transformer codec. Finally, the supervised fusion module can adaptively select and fuse the image features, and transfer the attention-guided features to the next stage.In addition, we demonstrate the effectiveness of the progressive network through experiments, and our model provides high-quality defogging on multiple data sets. Nonhomogeneous de-hazing is the next topic we would like to explore with our approach, as it is crucial to study complex foggy environments in real scenarios.The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.This research was supported by the National Natural Science Foundation of China  and Shandong Natural Science Foundation of China (ZR2021MF107 and ZR2022MA076).Author ZY was employed by Intgrow Education Technology. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


Among them flavanones, including naringenin (Nar), arise as a promising group of mitoBK channel regulators from a pharmacological point of view. It is well known that in the presence of Nar the open state probability (pop) of mitoBK channels significantly increases. Nevertheless, the molecular mechanism of the mitoBK-Nar interactions remains still unrevealed. It is also not known whether the effects of naringenin administration on conformational dynamics can resemble those which are exerted by the other channel-activating stimuli. In aim to answer this question, we examine whether the dwell-time series of mitoBK channels which were obtained at different voltages and Nar concentrations  are discernible by means of artificial intelligence methods, including k-NN and shapelet learning. The obtained results suggest that the structural complexity of the gating dynamics is shaped both by the interaction of channel gate with the voltage sensor (VSD) and the Nar-binding site. For a majority of data one can observe stimulus-specific patterns of channel gating. Shapelet algorithm allows to obtain better prediction accuracy in most cases. Probably, because it takes into account the complexity of local features of a given signal. About 30% of the analyzed time series do not sufficiently differ to unambiguously distinguish them from each other, which can be interpreted in terms of the existence of the common features of mitoBK channel gating regardless of the type of activating stimulus. There exist long-range mutual interactions between VSD and the Nar-coordination site that are responsible for higher levels of Nar-activation (Δpop) at deeply depolarized membranes. These intra-sensor interactions are anticipated to have an allosteric nature.The large conductance voltage- and Ca 2+-activated K+ channels from the inner mitochondrial membrane (mitoBK) are modulated by a number of factors, including flavanones like naringenin (Nar). In the presence of Nar the open state probability (pop) of mitoBK channels significantly increases, however the molecular mechanism of the mitoBK-Nar interactions remains still unrevealed. It is also not known whether the effects of naringenin administration on channel gating can accurately mimic the dynamical consequences exerted by other channel-activating stimuli. In order to answer this question, we examine whether the dwell-time series describing the mitoBK channel activity obtained at different voltages (Um) and Nar concentrations are discernible by means of artificial intelligence (AI) methods . The key inferences from the current study are as follows:The large conductance voltage- and Capop of highly voltage-activated mitoBK channels.Naringenin most significantly increases the The Nar binding site and voltage sensor may cooperate allosterically.Um- and Nar-specific effects on gating dynamics that allow for a proper AI classification for the majority of input data.There are some The local features of the signal seem to be highly stimulus-specific, which can be inferred from a relatively good performance of the shapelet classification method. Here, mean open state probability pop is used as the criterion for similarity of channel’s activity level. If the data obtained at different [Nar] and Um are unambiguously discernible according to the ML methods, these stimuli should affect the conformational dynamics of the channel in a highly specific way. In those terms, a sole dwell-time subseries allows us for identification at what Um and Nar concentration they were obtained. Accordingly, the number, connectivity and/or stability of the available channel conformations may be independently affected by [Nar] and Um. In opposite case, i.e. when the input signals obtained at different conditions of Um and [Nar] are indiscernible, it will suggest that there exists a set of predefined stable conformations which are characteristic for channel’s activation level regardless of the type of activating stimulus. In those terms, naringenin binding would allow the channel to mimic the conformational kinetics that usually correspond to its highly voltage-activated state, and vice versa.We are convinced that each stimulus should affect the channel’s conformational space in a different manner. Therefore, the relative effects of Nar- and voltage-activation on conformational dynamics will be evaluated by comparing the short excerpts (of 50 points) from the dwell-time series obtained at different different [Nar] and Considering the methodology used in the current work to perform the classification of the input signals, we decided to apply the K-nearest neighbors algorithm (k-NN). The k-NN method was commonly used in Time Series Classification (TSC), also in the case of biological signals such as electrocardiography  or electTo get a better insight into the nature of the analyzed problem we also implemented the shapelet-based ML technique. Shapelet is a subsequence of signal, on the basis of which the similarities within the signal can be identified. Based on specific shapelet features the process of time series classification can be evaluated with high accuracy. Since being presented in 2009 , the shaUm and naringenin concentrations [Nar] are presented in The obtained patch-clamp recordings confirmed that naringenin acts as mitoBK channel activator . To visupop values. Maximal difference of mean pops was assumed to be 0.01. As presented in pop ≈ 0.58 constituted by recordings at ,  and , second group, where pop ≈ 0.60 constituted by recordings at  and , and third group, where pop ≈ 0.66 comprised the recordings obtained at  and .Considering the merit of this study, the preliminary task of its experimental part was to find such values of membrane potential and naringenin concentration to find groups of recordings which have matching For all obtained raw experimental data, the corresponding dwell-time sequences were constructed and further analyzed. The results of the signal classification by the use of k-NN and shapelet learning methods are summarized in Um, regardless of the fact that the investigated signals exhibit comparable pops. Nevertheless, these individual features do not manifest themselves often enough to ensure unambiguous discrimination of each dwell-time subseries corresponding to fixed [Nar] and Um. About 30% of analyzed data had not sufficiently different characteristics to distinguish them from each other unequivocally. One can interpret that in terms of the existence of some common predefined paths of conformational switching regardless of the type of activating stimulus.As one can see, the obtained accuracies indicate that the classification of data is non-random which suggests that some specific features of the dwell-time sequences obtained at different naringenin concentrations and voltages exist.  In the majority of cases (over 60%), these features allow us to distinguish from each other the dwell-time subseries obtained at different [Nar] and pop = 0.12) is administration of naringenin. Here, the compared groups of recordings were obtained at fixed membrane potential Um = 60 mV in absence of naringenin and when this modulator was introduced [Nar] = 10 μM. In the second case, one can observe a similar level of open state promotion (Δpop = 0.11) which was, in turn, related to the raising value of membrane potential (from 20 mV to 60 mV) at fixed [Nar] = 10 μM. According to the results in Um or Nar) affecting efficiency of data classification. There are, however, differences between the performance of separation algorithms for the data referring to the Nar- and voltage-activation. The k-NN results allow for reaching relatively high prediction accuracy for channel activation by Nar coordination. It may suggest that naringenin-activation mostly shapes the large-scale features of the channel gating. In turn, voltage-activation is supposed to exert significant effects on local features of the mitoBK channel gating, according to the better accuracy of the shapelet analysis.To further investigate the problem, we have also performed the ML classification for the experimental data obtained at two different naringenin concentrations and fixed membrane potential, and vice versa. The results are presented in To further inspect the problem of differences in accuracy of the shapelet classification of signals recorded at different naringenin concentrations and membrane potentials, in Figs Um in shapelet–transform space. In μM. In that case 38% of points overlap and consequently may be misclassified. The points corresponding to the different Ums (Um = 20mV and Um = 60mV) are relatively well separated (only 18% of points overlap). Thus, one can infer that at a relatively short observation scale (of the Ns order of magnitude) the Um predefines the channel gating dynamics and leads to occurrence of the voltage-characteristic sequences of dwell-times. Whereas the Nar coordination has only an accessory effect on the short-scale signal’s characteristics.In Figs pop(Um)) toward more negative potentials in presence of naringenin. Moreover, Nar also promotes channel opening in a voltage-dependent manner. Greater increments of pops for raising [Nar] are observed at highly depolarized membranes  and the channel pore opening communicate.The current study allows us to conclude that naringenin activates the mitoBK channels in a voltage-dependent manner  and its embranes . It may pops were near to zero (pop ≤0.04), which were obtained at different naringenin concentrations. Then, we estimated the apparent gating charge according to the formula  = 10 μ M and in absence of this modulator  analysis to compare the effects of membrane depolarization and the increase of Nar concentration on the temporal characteristics of channel’s conformational dynamics. The obtained results suggest that both stimuli affect the structural complexity of the analyzed signal. There exist stimulus-specific features of the signal that allow distinguishing over 60% of analyzed dwell-time sequences obtained at different CO2 atmosphere, at 37 °C. The cells were reseeded every third day. The presence of mitoBK channels in the inner mitochondrial membrane was previously described in .In this study, we aim to verify the performance of ML methods used for the classification of mitoBK channels obtained at various membrane potentials k = 5). The choice of this algorithm is motivated by its well–documented time–series classification efficiency [Our first attempt of the data analysis consists of application of the standard k–NN (k–nearest neighbors) technique with an euclidean distance metric Selection of the number of neighbors Calculation of the euclidean distance between the new data point intended for classification and the training samplesk–nearest training points and assignment to the appropriate category by the majority voting.Choice of the Although k–NN is fast and its application yields in many cases excellent accuracies it does not give much insight into data. In the context of our study, it does not allow us to investigate whether the analyzed ion channels’ activators exert a specific effects on the structure of dwell-time sequences. In order to address this problem, we decided to apply the series shapelet method  which hal (l can be chosen arbitrarily) as potential candidates for a shapelet Mi,k defined in The shapelets fined as :Mi,k=miTo significantly accelerate our computations, instead of searching for the optimal shapelets among all candidates, we use the method proposed in . This teStart with the random shapeletLearning the optimal shapelet by minimizing the classification loss function.i is denoted as yi.This method works as follow. Let us assume, for the sake of simplicity, only two classes of the time series. Then, the corresponding labels can be either 0 for time subsequences belonging to the first class or 1 for the samples attributed to the second class. The label for time sample i is calculated with the logistic regression model as:bi, Wk  are free parameters of the model learnt in the process of training and coefficients Mi,k are defined in The prediction bi, Wk, and indirectly shapelets Sk are found through the minimization of the loss function:The parameters found in .n and their lengths l are chosen arbitraily. We found that the optimal choice for our dataset is n = 10 and l = 20. Furthermore, in order to minimize the loss function given in Adam optimizer with the learning rate lr = 0.01 along with the L2 weight regularizer of value 0.001.Within the below–described approach, the number of shapelets learning–shapelet method) we preprocess it according to the procedure summarized in Before feeding the data into the ML algorithm  of an algorithm is evaluated as the ratio of the number of correctly predicted dwell-time series to all samples in a dataset. In the case of binary classification we use the below–presented formula:TP and TN denote the number of correctly identified samples in the first class (TP) and in the second class (TN), whereas FP and FN stand for the misclassified subseries belonging to the first and second class, respectively. Note that, the extension of this formula to the three groups classification problem is trivial.In both cases, the performance of the ML algorithm is verified using the The overall performance score is an average of the accuracy scores as calculated across 10 test folds.learning–shapelet technique in order to reduce the dimensionality of our samples and visualize them in the 2–dimensional space. For this purpose, we calculate the distances d1 and d2 of each dwell-time subseries (according to s1 and s2 and treat these distances as the new coordinates in the distance–transformed space. Note that, this procedure is applied only for the visualization purposes and does not have any impact on the results presented in Tables Additionally, apart from classification, we use the learning shapelet and k–NN methods were conducted with the use of the tslearn and scikit-learn Python libraries [All calculations concerning ibraries , 62.


Cross-media visual communication is an extremely complex task. In order to solve the problem of segmentation of visual foreground and background, improve the accuracy of visual communication scene reconstruction, and complete the task of visual real-time communication. We propose an improved generative adversarial network. We take the generative adversarial network as the basis and add a combined codec package to the generator, while configuring the generator and discriminator as a cascade structure, preserving the feature upsampling and downsampling convolutional layers of visual scenes with different layers through correspondence. To classify features with different visual scene layers, we add a new auxiliary classifier based on convolutional neural networks. With the help of the auxiliary classifier, similar visual scenes with different feature layers have a more accurate recognition rate. In the experimental part, to better distinguish foreground and background in visual communication, we perform performance tests on foreground and background using separate datasets. The experimental results show that our method has good accuracy in both foreground and background in cross-media communication for real-time visual communication. In addition, we validate the efficiency of our method on Cityscapes, NoW, and Replica datasets, respectively, and experimentally demonstrate that our method performs better than traditional machine learning methods and outperforms deep learning methods of the same type. Influenced by COVID-19, the home office has become a new office situation, and cross-media visual communication technology has become a key technology for telecommuting. Cross-media visual communication technology is also more common in our life. Thanks to the maturity of 5G technology, current visual communication technologies such as teleconferencing, webcasting, and remote manipulation play an important role in long-distance communication. Nevertheless, current software technologies still have some shortcomings that lead to the loss of the physical connection between the camera and the display, hindering the widespread use of cross-media visual communication systems  have triA cross-media visual communication system often consists of a foreground and a background, with the foreground representing the communicator involved in the visual communication and the background representing scenes other than the communicator , 7. A coWe experimentally verify the efficiency of traditional machine learning methods and deep learning methods in visual communication, and the deep learning methods perform better. So, in this study, we propose a visual communication method based on an improved generative adversarial network. The visual scene is layered by layer features, and foreground and background scenes are trained with unused methods and datasets, respectively. Then, we trained and verified the effectiveness of our method on a public dataset. For the scene segmentation problem of foreground and background, we added control experiments to verify the visual communication efficiency of foreground and background separately with specific datasets. The experimental trials demonstrate that our method is accurate and can achieve real-time performance in the visual communication of foreground and background.The rest of this study is organized as follows. Researchers in the literature  have proIn addition to the optimization of the stereo camera matching algorithm, some researchers have discovered a new stereo matching technique, namely, the dynamic planning algorithm –19. ThisFor several years, Markov random fields have been gradually explored by researchers, and their excellent pixel-labeling ability has become a key player in dense stereo matching problems. Some researchers have found in their experiments that the Markov random field is gradually upgraded to a dedicated technique for graph cutting based on the Markov random field in the face of the transformation energy minimization problem, and the main principle of this technique is the confidence propagation minimization law –24. We fResearchers in the literature  found inGenerative adversarial networks have great applications in computer vision. Generative adversarial networks are essentially deep neural networks for generative models, which consist of two parts: a generator and a discriminator. The purpose of the network is to simulate learning training samples and generate samples with similar features to the training samples. During the training process, the generator generates a random fake sample from the input sample, and then, the discriminator directly determines the gap between the fake sample and the real sample, and if the gap is beyond the specified range, the fake sample is returned, and the generator regenerates a new sample based on the returned information until the discriminator passes the sample. The structure of the generative adversarial network is shown in We apply generative adversarial networks to cross-media visual communication. The accuracy of cross-media visual communication is improved by utilizing the mutual game learning approach of this network. Considering the scene complexity and image processing details of our visual communication, adaptive optimization based on generative adversarial networks will be considered to simulate optimal parallax for different visual communication levels. The details of the optimization will be explained in the subsequent subsections.For generators, which are usually simple deconvolutional networks or fully connected neural networks, their main purpose is to generate dummy samples based on the output information, and their principle of action is shown in Ilow transforms different visual scene data features to the hidden layer. Inspired by the literature [s = 2, kernel = 3 × 3 setting, and all the convolutional layers are followed by a normalization operation and then activated nonlinearly by ReLU. For input surfaces with different expression midpoints, we use the Χ conv operator [K input points , the K input points are weighted by a multilayer perceptron, and then, a K × KΧ transformation matrix, Χ=MLP, is performed, followed by an element product summation, and is given to the Χ transform features in as a typical convolution operator treatment. Considering the influence of relative positions between different feature points, we define the Χ conv operator as follows:p represents the feature points, K represents the adaptive convolution kernel, P=T represents the K points in its neighborhood, and F= represents the features of different points. Using the principle of the Χ conv operator, we construct generators representing the parallax lines of visual scenes with different hierarchical structures, as shown in The literature , 34 propterature , we alsooperator . For a gIn the structural design of the discriminator, we also use a combination of a deconvolutional and fully connected neural network designed to distinguish the difference between the fake and real samples generated by the generator. If the difference exceeds a predetermined value, , and if it is trained by fine-tuning the discriminator's samples, it is fed back to the generator by backpropagation. The principle of its action is shown in Gen represents the visual scene features generated by the generator, and Dis represents the visual scene features determined by the discriminator. {Ilow, Ihigh} represents a pair of parallax lines with different feature levels but the same visual scene class. The adversarial loss function equations for generator Gen and discriminator Dis are shown below.N denotes the total number of training samples. In the convergence process, the discriminator converges the features of the fake samples generated by the generator to the features of the real samples by setting the values so that the generator generates rich virtual visual scenes. Similarly, we constructed visual scene discriminators with different hierarchical structures. The hierarchical structure is shown in Low-level visual scene parallax is converted into high-level parallax. We use a generator to filter the underlying data density of high-level visual scene parallax features and a discriminator to separate the visual scene parallax features generated by the generator from the true high-level visual scene parallax features. Inspired by the study in the literature , we use The scene recognition of different layers of features in similar representations is considered. We introduce auxiliary classifiers in generative adversarial networks for retaining feature information under different layers. In real life, each person is capable of generating hundreds of foregrounds and backgrounds with different meanings and hierarchies in different visual scenes. In our study, the generator is made to enable feature reconstruction of real visual scenes while keeping the visual scene category variables stable. We use the adversarial loss to rationally bootstrap the generator and train the auxiliary classifier to extract visual scene features. Perceptual loss is imposed when optimizing the generator and discriminator.ϕ denotes the feature extractor, and the biggest role of the perceptual loss function is to capture the variance distance between high-level sensitive features between similar visual scenes, thus maintaining the hierarchical stability of parallax features of visual scenes. In addition, it can also compensate the parallax lines of different layers of features to achieve a steady state.The auxiliary classifier can efficiently handle highly variable data. To match the adaptive linear fitting neural network, we introduce a convolutional neural network model. The dataset used for classifier training is processed real samples, all of which contain different layers of features. In our experiments, we build an online training method to guarantee its perfect convergence. In the subsequent network phase, we use a feature extraction layer instead of a fully connected layer, and the equation of the perceptual loss function in the feature extraction layer is shown below.Ihigh in the real sample and the synthetic virtual scene representation feature Gen(Ilow) in the fake sample. As also mentioned in the literature [Npixel represents the geometric pixels in the two-dimensional sample and also denotes the three-dimensional sample point cloud data points. Combining the above loss functions, the systematic loss function of our optimized generative adversarial network is formulated as follows:ω1, ω2, and ω3 represent the weighting coefficients. During the training process, we adopt an alternating training approach so that the generator repeatedly iterates to the optimum and finally generates the sample with the most similar physical signs to the real sample.To contribute to the consistency of the image representation, we use point-by-point loss optimization between the high-level scene representation feature terature , the disX conv operator. The features of different forms of virtual visual scenes are extracted and filtered in the joint output, and the real sample features of the auxiliary classifier are used as the guidance to fuse into a more comprehensive visual scene, so as to achieve an overfitting fit state with the real scene and achieve the maximum matching rate with the real scene. The detailed network structure is shown in To enhance the stability and accuracy of cross-media visual communication systems. We propose an improved deep neural network algorithm for visual communication, and the network underlying the method is a generative adversarial network. With our method, the stability of the cross-mediated visual communication system and the matching accuracy between communication messages can be further enhanced, and a visual scene feature grading model is also used to obtain the optimal parallax lines and compensation values. In our model, the generator synthesizes expressive features with high expression levels under the dual guidance of discriminators and auxiliary classifiers. For 2D samples, the convolutional neural network will assist in generating the samples, and for 3D data, they will be synthesized by the To validate the performance of our method, we selected Cityscapes, a public dataset for automatic segmentation of street scenes, NoW, a dataset for 3D face reconstruction, and Replica, a dataset for high-quality 3D reconstruction of indoor scenes, for experimental validation. Before the preprocessing operation on the above datasets, we normalize the visual scene boundaries for all datasets. In the preprocessing stage, the image and video frames are segmented into specified sizes, and different sizes of test methods correspond to different data. We randomly selected 80% of the samples as the training set and 20% as the test set. Next, we describe the three datasets in detail.The dataset Cityscapes is a large-scale 3D video sequence of urban street scenes, divided into two types of strongly and weakly annotated frames. The biggest advantage of this dataset is its ability to automatically understand the semantics of the scenes and implement pixel-level semantic annotation of the full visual scenes The main advantage of this dataset is that it can automatically understand the scene semantics and achieve full visual scene pixel-level semantic annotation. We mainly choose this dataset to train the model for scene semantic segmentation performance. The dataset NoW is a face reconstruction dataset where all samples are independently performed by a 3D scanner. We mainly choose this dataset to obtain the matching accuracy of communicators in foreground reconstruction and to ensure a clear boundary between foreground and background. The dataset Replica is a high-quality indoor scene dataset, which contains 18 high realistic indoor visual scenes, and its biggest advantage is that all visual scenes are combined at the same level using plane mirrors and reflectors. We mainly choose this dataset to obtain the background matching accuracy and to achieve a high visual background reconstruction rate. The detailed information of the above dataset is shown in F), F1 score, and recall (R). The detection results of each metric are fed back to the individual metrics in the dataset, and the performance of text detection recognition can be indicated by the connection between the metrics, and the parallax baseline outputs after passing the visual scene classifier, which can also be labeled with the data to react to the accuracy of the virtual scene. There are two main steps in our cross-media visual communication work; first, we perform the visual scene classification work, and then, we perform the virtual scene reconstruction work. The visual scene classification performance is shown in To demonstrate the high efficiency of our method, we compared three methods, random forest (RF), RCNN, and BiLSTM. To ensure the accuracy of each method, each network is independently kept trained without the detection module during the training and tuning process. They are all trained using the ground truth of the three datasets mentioned in the previous section. The deep networks do not use any visual scenes, and among the evaluation metrics, we mainly look at precision , virtual scene reconstruction rate (VSRR), and real scene distribution rate (RSDR), and the results of the dataset evaluation are shown in To separate the foreground and background of the cross-media visual communication and highlight the detailed features of the characters, we re-edited the NoW dataset by reconstructing the 3D face reconstruction data in the dataset according to a standardized template and then took a smoothing process for each sample frame. In the scene reconstruction of foreground communicators, we mainly examine the metrics of face matching rate (FMR), foreground segmentation rate (FSR), and pixel offset rate (POR). The experimental results are shown in We also re-edited the Replica dataset and divided the indoor 3D scenes in the dataset into three levels, A, B, and C, where A denotes complexity less than 30%, B denotes complexity between 30% and 70%, and C denotes complexity greater than 70%. Then, a smoothing process is taken for each sample frame. In the scene reconstruction of the background, we mainly examine the metrics of background matching rate (BMR), background segmentation rate (HSR), and pixel offset rate (POR). The experimental results are shown in From the experimental results in Tables To improve the accuracy and real-time performance of cross-media communication, we propose an improved generative adversarial network. Considering the cascade structure of the deep adversarial network will be useful for cross-mediated visual communication. Therefore, we choose the generative adversarial network as the basis and add a combined codec packet to the generator, while configuring the generator and discriminator as a cascade structure, preserving the feature upsampling and downsampling convolutional layers of visual scenes with different layers through correspondence. To classify features with different layers of visual scenes, we added a new auxiliary classifier based on convolutional neural networks. With the help of the auxiliary classifier, similar visual scenes with different feature layers have more accurate recognition rates. In the experimental part, we validate the efficiency of our method on Cityscapes, NoW, and Replica datasets, respectively, and experimentally demonstrate that our method performs better than traditional machine learning methods and outperforms deep learning methods of the same type. To better distinguish between foreground and background in visual communication, we perform performance tests on foreground and background with separate datasets. The experimental results demonstrate that our method has excellent accuracy in both the foreground and background of cross-media communication and can achieve real-time visual communication.Cross-media visual communication is an extremely complex task that encompasses scene reconstruction, scene matching, and foreground-background segmentation. Each application scene has thousands of foreground-background combinations. In this study, we choose the simpler indoor scenes as our research point. However, for real cross-media visual communication scenes will be more complex. For complex visual scenes, our approach still poorly performs. In the next study, we will try to use the local perceptual layer of the BiLSTM network as the implicit layer visual scene extraction point to increase the visual scene capture range and improve the coverage of the model for visual scene types.


In recent years, with the development of smart grid, the power systems and other energy systems are gradually forming integrated energy systems. The electric-thermal integrated energy system is a mature integrated energy system at present. The electric-thermal integrated energy system uses modern communication technology to realize the comprehensive regulation of electric energy and thermal energy, which greatly improves the efficiency of energy use. However, this also greatly increases the risk of malicious tampering with the energy dispatch system. In this paper, we study the regulation of electric-thermal integrated energy systems considering false data injection attacks. First, we establish a compromised model of an electric-thermal integrated energy system considering false data injection attacks. Then, we designed vulnerable variable observers for different tampering scenarios to observe the tampered variables. Finally, considering the relationship between the observed data and the measured data, we design a tampering behavior detection method based on relation network. The simulation results verify the effectiveness of the detection method proposed in this paper. The electric-thermal integrated energy system (ETIES) is an important part of the integrated energy system. With the aid of the advanced network information technology and innovative operation and management models, ETIES integrates electrical and thermal energy in the region, realizes operation optimization and coordinated control among various heterogeneous energy sub-networks through energy coupling equipment, and effectively improves energy conversion efficiency and promotes sustainable energy while meeting the diverse energy needs of users –3. HowevThe spread of malicious attacks in the communication network will destroy the environment of network communication, make the economic operation of the system impossible, even destroy the stability of the system , 7. In , the autFalse data injection attack (FDI attack) is a new form of attack that has appeared in recent years to undermine the credibility of the operational data of integrated energy systems , 12. Whel0 norm and l1 norm of the attack vector, an attack evaluation index to measure the attack effect and attack cost is proposed. References [At present, there are two main perspectives in the research on considering the existence of FDI attacks in the system. On the one hand, from the perspective of FDI attackers, researchers design an optimal attack strategy that can improve the probability of successful attack and attack effect. Reference  studies ferences –18 consiferences , 20 desiferences  used traferences  proposedferences  demonstrferences  added a On the other hand, researchers propose defense strategies against network attacks from the perspective of system defense. References , 26 use Similar to the original social power supply, heating and other systems, in the operation process of ETIES, one of the most concerned issues is how to realize the economic scheduling of the system, that is, how to comprehensively allocate the capacity distribution between multiple energy units on the premise of meeting the system security constraints, so as to minimize the economic cost of the system, and then realize the dual guarantee of system operation in terms of security and economy. The economic scheduling method of ETIES can be divided into centralized method and distributed method. Although the centralized method has high efficiency in information processing, it has some problems, such as high communication cost and sensitivity to single point of failure. The distributed method can use the sparse communication network structure to realize the decentralized cooperation of various equipment components of the system, which has less communication burden, stronger robustness and privacy. Therefore, in recent years, experts and scholars at home and abroad have proposed many ETIES economic scheduling methods based on distributed optimization.However, it is worth noting that although the above method can effectively solve the distributed economic scheduling problem of ETIES, its premise is that the system operates in an ideal network communication environment, that is, a large number of interactive measurement and control data can be reliably transmitted on the communication line. However, ETIES based on distributed optimization architecture is an energy system with high integration of information and physics. While the information system of ETIES interacts with a large amount of information and data, it is bound to suffer from a huge threat of network attack. The spread of malicious attacks in the communication network will destroy the bad environment of network communication, make the economic operation of the system impossible, and even destroy the stability of the system, resulting in the paralysis of the energy supply system.ETIES is a large system with electrical-thermal coupling characteristics, and its structure and operation are much more complex than traditional power systems. Therefore, malicious attackers need to adopt more complex and targeted strategies according to system conditions when attacking ETIES. So far, most of the research on the impact of network attacks on system performance is carried out on a single power system, and there is no research on the impact of network attacks on the operational security of ETIES. The distributed scheduling of ETIES depends on the security and reliability of the communication network, and network attacks will inevitably affect the scheduling process of ETIES, thereby affecting the performance of the system.Aiming at this research gap, the motivation of the paper is to enhance the safety and security of the electric-thermal integrated energy system by studying the ETIES model under FDI attacks and designing an attack detection method based on machine learning algorithm.We establish attack templates in the electric-thermal integrated energy system and discuss the impact of false data injection attacks on the integrated energy system.In the electric-thermal integrated energy system under FDI attack, we propose an observer-based method for observing vulnerable variables of the system, so that the compromised variables can be effectively observed.Using the observation data obtained by the observer, we propose a relation network-based attack detection algorithm to detect FDI attacks in integrated energy systems.The main contributions of the paper are three fold:The scope of the paper is shown as follows: first, the compromised model of the electric-thermal integrated energy system is discussed in this paper; Then, based on the variables in the system, a machine-learning-based attack detection method is studied to identify the FDI attacks on ETIES.The remaining part of this paper is organized as follows: in xpP: Incremental cost of power only device.xpC: incremental cost of combined heat and power device.xhC: thermal incremental cost of combined heat and power device.xhH: thermal incremental cost of heat only device.upP: electric power mismatch of power only device.upC: electric power mismatch of combined heat and power device.uhC: thermal power mismatch of combined heat and power device.uhH: thermal power mismatch of heat only device.m(k): attack vector.ypP: electric output power of power only device.ypC: electric output power of combined heat and power device.yhC: thermal output power of combined heat and power device.yhH: thermal output power of heat only device.x: state vector of the system.e: estimation error.dm: data vector in the measured data set.do: data vector in the observed data set.dm): feature vectors of measured data.ℱ(do): feature vectors of observed data.ℱ. DDCA employs two different consensus protocols. One of the consensus protocols is used to calculate the incremental cost corresponding to the optimal solution of the ETIES economic dispatch problem. Another consensus protocol aims to estimate the amount of electrical/thermal local power mismatch for coordinating device output. The two protocols of DDCA use different but strongly coupled consistency variables to calculate the electric/thermal incremental cost, electric/thermal output power and electric/thermal local power mismatch corresponding to the optimal solution of ETIES economic dispatching problem, so as to finally realize the distributed economic dispatching of ETIES. ETIES scheduling depends on the information exchange and local calculation between each unit and its neighbors. Each energy unit contains a distributed controller for operation.A is the consistency algorithm update matrix in DDCA, which is determined by the adjacency relationship between the current energy unit and the surrounding energy unit; B is the algorithm convergence rate adjustment matrix in DDCA; M is the corresponding attack weight matrix.The attacker can attack the incremental cost estimator and the output power decision of the energy unit in DDCA, thereby affecting the output power of the unit in the energy unit. Inspired by reference , the comC is the cost coefficient matrix; N is the corresponding attack weight matrix.The compromised output power decision-maker studied in this paper can be written asIt can be learned that FDI attacks can change the power output of the energy unit by tampering with the state variables of different modules in the ETIES, which has an impact on the power balance of the integrated energy system. In the next section, observers for different attack intrusion locations are designed to observe the FDI attacks.ice can be expressed asIn this part, we focus on the observer for compromised incremental cost estimator. The compromised system ∑m(k − 1) at k − 1 time as an additional state, we can obtain the augmented state vector Taking the attack vector z represents the state vector of the dynamic system ×(n+q)( and V ∈ ℝn+q)×(n+q)×(n+q+m) < 0 and e(k) is convergent. Let W=PL, then inequality equation  as an additional state, we can obtain the augmented state vector Taking the attack vector Similarly, for this augmented system, we can also construct an observer in the form of formula . Conditis can be expressed asωa(k) and ωs(k) are unknown input vectors caused by uncertainties of system; Ea and Es are known constant coefficient matrices with appropriate dimensions. Taking the attack vector as an additional state, we can obtain the augmented state vector In this part, multipoint FDI attacks are considered: the attacker can launch FDI attacks on incremental cost estimator and output power decision-maker simultaneously. The compromised system ∑z represents the state vector of the dynamic system equation Vk=eTγ(k)=0, from equations (V(k) < 0. The error dynamic is asymptotically stable.If quations  and 34)γ(k)=0, fLetWe can deriveBased on equations  and 36)36), we cV(∞) ≥ 0 and V(0)=0, we can deriveIn view of the fact that Based on the proposed observer, we can derive the observed data of the variables and the measured data of those in DDCA. For the defender, it is necessary to identify the similarities between the measured data and the observed data under normal situations and distinguish the differences under the compromised situations.The method of calculating dual source data vector similarity based on traditional Euclidean distance requires too much prior knowledge level of defenders. In this paper, we use an embedding module and a relation module to extract the similarity of the dual source data automatically.Traditional machine learning methods need the distance of data vector in feature space to identify, which means that large scale of training data set is needed. In this paper, we skip the learning of feature distance and directly learn the relationship between dual source data, so as to effectively reduce the demand for the size of data set.In this section, we study the attack detection scheme against FDI attacks based on the observed data of the variables and the measured data of those in DDCA. A relation-based detection network is proposed to extract the similarity of the dual source data. We design the machine-learning-based detection scheme based on the following considerations:As is shown in dm. The data vector in the observed data set is written as do. The embedding module, which consists of full connect layers and rectified linear units (ReLUs), is used to extract the features of samples with a nonlinear function ℰ. Compared with the traditional manual feature extraction method, the feature extraction by full connect layers can reduce the prior knowledge requirements of attack detection network for attack features. Rectified linear units are used to improve the generalization ability of the embedding module. The feature vectors of measured data and observed data generated by the embedding module can be expressed as ℱ(dm) and ℱ(do). To alleviate the over fitting problem of the embedding module, class prototype of each feature vector class is adopted. The prototype 𝒫im of the measured data feature vectors and the prototype 𝒫io of the observed data feature vectors can be expressed asAs to the datasets, the data vectors in each dataset consists of the time series data of target variables in DDCA, including the data of incremental cost and those of output power. The data vector in the measured data set is written as 𝒞 by concatenating the prototypes in depth dimension. The relation module is used to extract the similarity between the concatenations with a nonlinear relation function ℛ. The similarity 𝒮 can be written asWe can derive the class feature vector Lm.To train the attack detection model, mean square error (MSE) is used as the objective function lm ≠ lo and 𝒮 is closed to 0. If the measured data is normal, then lm=lo and 𝒮 is closed to 1.If the measured data is compromised, then Pseudocode for the proposed detection scheme is provided in In this section, simulations are carried out to illustrate the effectiveness of the proposed observer and attack detection network of the variables in DDCA. The Barry Island electricity and heating networks is used as the tested system. The structure and parameters of the system can be found in .In the DDCA system, the coefficient matrices areThe attack vector is xpP. Based on the method proposed in xpP can be obtained. The simulation result of the dual source data is shown in First, we illustrate the performance of observer against false data injection attacks on incremental cost estimator. The attack target variable is ypC. Based on the method proposed in xpC in DDCA can be obtained. The simulation results are shown in Figures Then, the performance of observer against false data injection attacks on output power decision-maker is studied. The attack target variable is ypC in the output power decision-maker makes the measured incremental cost data xpP different from the observed ones. Compared with the FDI attacks on incremental cost estimator, the impact of FDI attacks on output power decision-maker can be reflected by the variables in incremental cost estimator.It can be learned that the FDI attacks on electric output power xhC and yhC are compromised simultaneously. Based on the method proposed in xhC can be obtained. The simulation results are shown in Figures To illustrate the performance of the proposed observer in situations of multiple modules being compromised, we analysis the simulation results considering the situation that In this subsection, we evaluate the performance of the proposed attack detection scheme. In the embedding module, there are three full connect layers and rectified linear units. The batch size of the relation network is chosen as 20. In the measured data set, there are 500 normal sample data and 500 compromised data from the historical database. In the observed data set, 1000 observed data are generated based on the proposed method studied in Section B. The simulations are carried out on a personal computer with Intel processor core i7, cache 3.4 GHz, NVIDIA GTX 2060, and random-access memory (RAM) 32 GB.(1)Accuracy:TP represents the number of true positive detection results; TN represents the number of true negative detection results; FP represents the number of false positive detection results; FN represents the number of false negative detection results.where (2)The probability of detecting correctly:(3)Success ratio:(4)Probability of identifying normal cases:(5)Mdc and Ms:Trade off between 𝒯 is the trade-off coefficient. Details about the performance metrics can be found in [where found in .To evaluate the performance of the relation-based attack detection scheme, the following metrics are used:To illustrate the effectiveness of the proposed detection scheme, six methods are adopted for comparison: (1) The proposed relation-based attack detection scheme (ME1); (2) Attack detection scheme using relation network without prototype module (ME2); (3) Attack detection scheme using multi-layer perception (ME3); (4) Attack detection scheme using signal forecasting method (ME4); (5) Attack detection scheme using support vector machine (ME5); (6) Attack detection scheme using clustering artificial bee colony algorithm (ME6).The simulation results are shown in In order to further investigate the stability of the detection performance of the proposed attack detection scheme, the performance of the attack detection scheme with different proportion of training sets is studied: at an interval of 5%, samples with a proportion from 40% to 80% are selected as the training sets. The simulation results are shown in Considering the insufficient samples of compromised data in practice, we further discuss the reliability and stability under different positive and negative sample ratios. In this section, the ratio of positive samples to negative samples is 1 : 1, 1 : 2, 1 : 5 and 1 : 10 respectively. The specific performance verification effect is shown in Considering that the detection scheme proposed in this paper depends on the real-time data of the sensors, we further study the impact of measurement noise and measurement delay on the attack detection performance in the process of collecting sensor data. We design two metrics, security noise and security delay, to evaluate the detection performance of the proposed attack detection method. Safe noise (delay) refers to the maximum noise (delay) that can be tolerated when the detection accuracy (MA) reaches a specified threshold.Safe noise and safe delay considering different threshold of MA are in In this article, false data injection attacks on distributed controller of electric-thermal integrated energy system and countermeasures are studied. Observers of variables in DDCA are designed to track the compromised data. The proposed observer can achieve the observation considering different attack targets in DDCA. Based on the observed data and the measured data, we proposed a relation-based attack detection scheme to identify the false data injection attacks.The simulation results show that the attack detection scheme has better performance than the current mainstream scheme under multiple evaluation indexes. The better detection performance of the proposed scheme is attributed to its direct judgment of the difference between normal data pairs and compromised data pairs, which reduces the learning of other unnecessary or incorrect features. For the stability of the proposed scheme, compared with other schemes, the proposed scheme can maintain better detection performance with less proportion of training sets.Therefore, we believe that the proposed attack detection scheme can achieve good performance against FDI attacks on ETIES.It can be seen that the limitation of the proposed method used in this paper is that it requires real-time data of the system, which makes the defender have a certain dependence on the real-time sensor communication network. As to practical implementation, the challenge is how to deal with the large-scale destruction of more sensors by attackers. In such a scenario, the trusted data available in this paper will be reduced, and the ability to identify attacks will be reduced.A possible mitigation approach is to stop using the real-time data obtained by the sensors of the system. As an alternative, the defender can use the system model and historical data to generate prediction data for real-time data, and use the predicted data combined with the algorithm proposed in this paper to identify cyber attacks. It can be seen that this research idea further reduces the dependence on real-time sensors, thereby reducing the uncertainty under large-scale attacks.