High-resolution large-eddy simulation (LES) is exploited to study indoor air turbulence and its effect on the dispersion of respiratory virus-laden aerosols and subsequent transmission risks. The LES modeling is carried out with unprecedented accuracy and subsequent analysis with novel mathematical robustness. To substantiate the physical relevance of the LES model under realistic ventilation conditions, a set of experimental aerosol concentration measurements are carried out, and their results are used to successfully validate the LES model results. The obtained LES dispersion results are subjected to pathogen exposure and infection probability analysis in accordance with the Wells–Riley model, which is here mathematically extended to rely on LES-based space- and time-dependent concentration fields. The methodology is applied to assess two dissimilar approaches to reduce transmission risks: a strategy to augment the indoor ventilation capacity with portable air purifiers and a strategy to utilize partitioning by exploiting portable space dividers. The LES results show that use of air purifiers leads to greater reduction in absolute risks compared to the analytical Wells–Riley model, which fails to predict the original risk level. However, the two models do agree on the relative risk reduction. The spatial partitioning strategy is demonstrated to have an undesirable effect when employed without other measures, but may yield desirable outcomes with targeted air purifier units. The study highlights the importance of employing accurate indoor turbulence modeling when evaluating different risk-reduction strategies. I.The global COVID-19 pandemic, caused by SARS-CoV-2 virus and its variants, brought about a debate concerning the relevant transmission modes of the disease. The initially adopted view clung to the paradigm that most probable infection route involves large respiratory droplets which are ballistically sprayed due to respiratory activity either directly into the eyes, nose or mouth of a bystander or onto surfaces from which they are subsequently carried to the body openings with mucous membranes. Thus, keeping a “safe distance” (1–2 m) to others and practicing hand hygiene were identified as sufficient preventive measures against transmission. This picture was brought into question early on as evidence began to accumulate on cases where the only viable transmission mechanisms pointed to airborne transmission or more specifically transmission by inhalation of non-ballistic virus-laden aerosol particles originating from the respiratory activity of the infected person. Despite the prolonged institutional reluctance to accept the relevance of airborne transmission mode, the general consensus has shifted under mounting evidence recognizing now that airborne transmission is the primary mode fueling the global pandemic.et al.It is now recognized that the respiratory particles, exhaled by an infected individual, are ranging from smaller than Indoor air flows are challenging to model by means of computational fluid dynamics (CFD). The flow systems are complex combinations of different turbulent source mechanisms and largely occupied by weak free turbulence. Such flow systems are highly sensitive to modeling errors and ill-suited for Reynolds-averaged Navier–Stokes (RANS) approaches where all the turbulent motion and related effects are modeled instead of resolved.The predictive capacity and physical validity of any numerical modeling approach must be established before the obtained results can be used to justify actions and guide decisions. Due to the absence of indoor-specific dispersion datasets, this work documents a transparent validation study to assess the LES model's performance in the context of aerosol dispersion under realistic indoor conditions. The experimental part involves particulate matter sensors distributed evenly throughout the test room, recording the evolution of aerosol concentration originating from a known source. The level of agreement between the experimental and LES modeled concentration time series is established with a group of performance metrics used in air-quality modeling.In this study, high-resolution LES modeling is exploited to examine a realistic indoor ventilation flow system which features the level of complexity found in spaces used for social gatherings. Although a specific spatial and ventilation configuration is employed, the objective is to gain physically sound insight into the nature of mechanically ventilated indoor flows in general. Moreover, the role of air purifiers augmenting the existing ventilation capacity in mechanically ventilated spaces are studied. The emphasis is to recognize the physical mechanisms and their relevance in indoor air and to develop a reliable LES methodology to evaluate different air hygiene strategies.Having demonstrated that the LES model is able to resolve the principal flow physics governing the evolution of respiratory aerosol concentration field, the analysis is broadened to consider the exposure of other subjects to the viral load emitted by an infected host and determine their probability to contract an infection. A rigorous adaptation of Wells–Riley infection-probability analysis is employed to cast the obtained LES results to relevant context.et al.The widely applied analytical Wells–Riley infection-probability model, based on the early works by WellsHowever, spatially constant concentration is a strong simplification. In reality, the concentration field varies strongly as a function of indoor turbulence and ventilation conditions. We hypothesize that this systematically leads to strong underestimation of the mean concentration and subsequently resulting in over-optimistic infection probabilities which is an unacceptable trait for a risk-assessment tool. Situations where the aerosol source (host) is in the immediate proximity of an outlet vent or an air purifier do constitute exceptions, but the probability of such circumstances is judged insignificant. Moreover, the constant-concentration assumption ignores peak concentrations and thus peak probabilities leading to further underestimation.et al.Zhang Realistic approximations for time- and space-dependent concentration fields can only be obtained by carefully implemented high-resolution CFD modeling. Combining CFD data with the Wells–Riley probability model has been suggested in several studies.et al.,et al.,et al.A number of recent studies have focused on the infection risk due to instantaneous emission events such as coughs or sneezes, e.g., Balachandar The article is organized as follows: Sec. II.The study exploits a real restaurant room located downtown Helsinki, Finland, as a laboratory for the experiments and modeling. The aerosol dispersion experiments were conducted onsite while the computational LES model was constructed in accordance with detailed laser-scanning data sets obtained from the interior of the restaurant room. The room's floor area is approximately A.The restaurant building is from the early 1900s, but it has been augmented with a modern centrally operated mechanical ventilation system (forced flow in both inlets and outlets) such that the room features three inlets feeding fresh, heated air toward the windows and three outlets on the opposite wall from the windows. Under normal circumstances, the room's outlet ventilation capacity is shared by an adjacent room connected via an entrance opening , heating elements  and cooling surfaces (windows during cold seasons). The complexity ensures that the studied indoor flow system is representative of a range of realistic mechanical indoor ventilation scenarios where the flow is driven by a multitude of turbulence generating mechanisms.In general, the most notable deviation separating real and modeled indoor flow systems stem from the natural imperfections found in actual buildings. Buildings, particularly older ones, are not perfectly sealed from the ambient  but leak through windows and other seams and cracks in the structures. Such leaks do manifest in the volume flow rate budget of this study's lab room. The forced ventilation maintains the room at low pressure in relation to the ambient resulting in leakage flows at windows and other small gaps in the structures giving rise to a flow rate imbalance between the inlet and outlet ducts. This non-idealization is accepted and addressed in the context of model evaluation see Sec. .The effect of added indoor air filtration is examined by considering nominal ventilation scenarios in juxtaposition with situations where the room is equipped with two portable UniqAir PROIII.A.8 particles per second.The aerosol particles were generated by Omron Ultrasonic Nebulizer Model NE-U17. It consists of an ultrasonic vibrator  and nine SPS30 particulate matter (PM) sensors . The reference instrument APS is a time-of-flight-based particle size spectrometer, which measures the number and aerodynamic size of particles from 0.5 to 20 y-axis. When conducting an experiment, the row of the three masts were moved between different x-axis coordinates W, C, and D  to capture a total of 27 measurement points. The APS1 and APS2 were used to measure the particle size distributions at different locations within the room, and their location varied from one experiment to another. The measured particle number concentration data used to evaluate LES model relied solely on the data measured with the sensor nodes.An illustration of the measurement arrangement is shown in C.Aerosol particle dispersion experiments were conducted on a generic seating configuration with (GEN+FLT) and without (GEN) air purifiers. Both experiments were divided into three parts in order to cover all 27 aerosol particle sampling points. Each part of an experiment began by ventilating the room so that the current aerosol particle number concentration corresponded to that of the general background concentration. Then, the nebulizer was switched on and the dispersion of aerosol particles was observed using the nine sensor nodes attached to the three masts. After the experiment had lasted 60 min, the nebulizer was switched off and the room was thoroughly ventilated. This process was carried out three times both in GEN and GEN+FLT configurations in order to cover all three mast-row positions W, C, and D, see Contrary to IV.A.U is the unfiltered velocity field, u is the grid-resolved (space-filtered), and The nomenclature conventions used throughout this article are presented herein. All Cartesian vectors are denoted by boldface font and scalars by normal font. For instance, the coordinate vector and its components are denoted as B.g is the gravitational acceleration, z-direction, ρ0 is the constant density of air at ground level, I is the identity matrix, ν being the kinematic viscosity of air at room temperature, and The PALM large-eddy simulation model systemθ is the air temperature, c is the scalar concentration, cD are the corresponding molecular diffusion coefficients.In the energy Eq. et al.θ, concentration c, the SGS turbulent kinetic energy e, and the modified perturbation pressure PALM solves the prognostic equations on a staggered Cartesian Arakawa-C grid. Subgrid-scale (SGS) turbulence is parameterized using a 1.5-order closure after DeardorffThe discretization in time and space is achieved using a third-order Runge–Kutta scheme after WilliamsonThe incompressible flow solver algorithm employs a predictor-corrector approach where the predictor step involves the calculation of a preliminary prognostic velocity from Eq. Parallelization of PALM is achieved by using the message passing interface  which acts as a conventional zero-gradient boundary for velocity terms.Due to limitations of the finite difference formulation in PALM, the mass conservation is not guaranteed in the smallest inlet and outlet duct sections and therefore the continuity is reinforced with locally applied momentum volume source terms in the relevant duct sections. This treatment guarantees The winter time thermal complexity of the room, dictated by the heating radiators within the inlet ducts, cold window surfaces, and the stationary humans occupying the room during the experiments, is captured by the model's thermal boundary conditions. The solid walls are considered to remain at room temperature, while the surface temperature of windows and clothed humans, measured on-site with a thermocouple and an infrared camera, are imposed as solid wall temperatures within the model. i, j, k computational cell within a specified source volume The boundary conditions for the scalar field 1.With the imposed momentum boundary conditions, the effective ventilation rate of the model restaurant room is set to The study employs two air purifiers,2.The numerical simulations were initialized by running the indoor model for at least CFL number was set https://docs.csc.fi/computing/system/#puhti. Each simulation required approximately With the third-order Runge–Kutta time integration approach, the maximum allowed A set of shorter D.user code functionality of the PALM system.For the validation study, the LES model with GEN configuration was utilized and the time-accurate concentration sampling was achieved by exploiting the As the nebulizing unit's generation rate of suspended aerosols could not be experimentally verified, the experimental and LES model results were made comparable by normalizing the concentration values at each sensor location asRNMSE), fractional bias (FB), and correlation factor (R), which are defined for the time series o as in observation) and LES model (subscript m) as follows:The validation is quantified by three different metrics: root-normalized means square error  are most observable closest to the concentration source (sensor masts W1 and C1), and they subdue considerably with distance from the source.•With only few exceptions, the asymptotic levels are well reproduced. The largest deviations occur close to the source where the dispersion outcome is most sensitive to the nebulizer's plume behavior in the experiment which is influenced by positive buoyancy due to the heat generated by the nebulizer and negative buoyancy due to the cooling evaporation process that occurs when the plume is emitted. This thermal complexity of the aerosol concentration source is not replicated in the LES model where the conditions around the source are near neutral.A comparative inspection of the time series depicted in RNMSE values register as acceptable (green or yellow) and 96% of the values for the GEN and 81% for the GEN+FLT configurations register at firmly acceptable level (green). This establishes the overall consistency of the time series, which is further substantiated by the high acceptance rate of the correlation factor R (89% for GEN and 69% for GEN+FLT), which needs to take into account that the shortcomings  are primarily due to the realization-specific turbulent fluctuations near the source . From the perspective of these two metrics, the model validation is firmly supported. The summary of the evaluation metrics in FB) values, on the other hand, reveal that systematic bias dominates the deviation between the experimental and model datasets as nearly half (48%) of the FB values for GEN and 42% for the GEN+FLT configuration fall outside the acceptable range. The deviations in temporal mean values [see Eq. The fractional bias  to central locations within the room significantly energizes the flow, making the units a dominant source of momentum (particularly in the LES modeled flow system). From the flow's perspective, the air purifiers act as fans which add energy into the system. This added energy diminishes the LES flow solution's dependency on the inlet and outlet boundary conditions and their uncertainties, subsequently reducing the lag times It is essential to acknowledge that both experimental and LES model time series each represent a unique realization of a lone dispersion event with a continuous source. The turbulent nature of the flow system dictates that individual realizations will vary, particularly at the beginning of the dispersion event whereas the long-time asymptotic state is not expected to alter between realizations. Thus, the concentration signals are expected to embody a degree of randomness which decreases with time. For most rigorous outcome, a statistically representative ensemble of realizations should have been analyzed, but with the currently available experimental and supercomputing resources, this could not be achieved. Nonetheless, the level of detail and breadth of the validation enterprise is deemed high enough to provide sufficient evidence that the documented LES model is capable of resolving the relevant flow physics governing the spread of pathogen-carrying aerosols indoors.B.Here, the test room's LES flow solutions are examined to gain insight into the physical phenomena, which are relevant to this system specifically and indoor ventilation flows in general. It is acknowledged that implementation details of mechanical indoor ventilation arrangements vary between constructions, but the principal flow physics remain similar across different realizations. Principally, indoor flow systems are primarily taken up by free turbulence, which is maintained by mechanical and thermal momentum sources such as inlet jets and thermal plumes from heating or cooling elements. The ventilation inlets are ideally designed to operate as jets which penetrate into the surrounding air, generating shear layers which diffuse via turbulent mechanisms, mixing freshly supplied air uniformly throughout the living space. The outlets, on the other hand, are ideally placed such that the incoming air has the opportunity to mix into the ambient volume and sweep across the intended portion of the space before reaching the outlets and exiting the system. A ventilation short-circuiting, a process where freshly supplied air is exhausted before it has had the opportunity to mix sufficiently with the volume of air, is considered a major flaw in the system design.As ventilation flows are notoriously difficult to model and optimize, it is reasonable to expect that imperfections are ubiquitous in practice. The sensitivity of ventilation flows to modeling errors demands that the chosen numerical scheme is sufficiently accurate and robust that it is able to capture and uncover any undesired features in the system. This further highlights the importance of relying on high-resolution LES modeling, which is capable of describing even the weaker turbulent structures without risking contamination due to modeling error. In an effort to meet the stringent numerical demands, the LES model documented herein resolves at least 98% of the turbulent kinetic energy within the space dominated by free turbulence.  Thereby, it is evident that vast majority of the turbulent motion within the model is practically fully resolved, as in direct numerical simulation (DNS) approach. This is computationally feasible thanks to the exceptional parallel and cache efficiency of the structured PALM LES solver. Moreover, this percentage of resolved turbulent kinetic energy, 98%, is by far higher than the criterion 80% for well-resolved turbulence suggested by Pope.in situ measurements, but only after the room was allowed to lay dormant The LES results reveal that the flow system of the test room does exhibit design flaws as undesired stratification is observed in both ventilation flow and temperature fields. An instantaneous state of the LES solved velocity magnitude field The introduction of air purifiers, which from the indoor air's perspective act as fans, significantly energizes the flow within the room, particularly in the stagnant lower part. This is visually apparent in ith concentration bin ic and (b) the normalized scalar content  In the context of rapidly released puff of respiratory aerosols (due to a cough or a sneeze) the dilution rate of the aerosol concentration has been shown to scale with the turbulence level of the indoor flow system.The frequency distribution in C.et al.Recently, infection probability has been studied in several works, e.g., by Buonanno V is the volume of the indoor space, qG is the quanta generation rate (quanta/s), and The analytical model for spatial average of the time-dependent quanta concentration Underestimation of the mean concentration predicted by the Wells–Riley theory in this case is clearly seen in Concentration results without (GEN) and with (GEN+FLT) the air purifiers are shown in The time-dependent probability of infection k refers to an individual experiment and N experiments in the ensemble; see We generalize this theory for spatially variable situations in qG: 100, 20, and qG proposed by Buonanno et al.et al.et al.In living zone with z ranging from 0.1 to The left-hand column of D.The effectiveness of the risk-reduction strategies is examined using In the light of these results, it is clear that none of the considered risk-reduction attempts is able to reduce the risk to a safe level in this case. Only a reasonable risk reduction is achieved by using the air purifiers at volume-flow rate, which increases the effective ventilation rate by approximately 60% or more. A drastic increase in the air-purifier volume-flow rate capacity would naturally lead to further risk reduction but the added cost and noise level and possible other disadvantages would probably make this option impractical.VI.A detailed LES study on indoor turbulence and its effect on dispersion of airborne respiratory pathogens under realistic ventilation conditions is reported. The focus lies on the long-distance aerosol transmission and relevant flow mechanisms governing aerosol dispersion indoors.The study involves an aerosol particle dispersion experiment and a LES model validation against the measured concentrations in 27 locations inside the restaurant room. The level of agreement between the experimental and LES modeled concentration time series is established with the set of performance metrics adopting a strict set of evaluation criteria used in air-quality modeling.The role of turbulent mixing in the context of continuous emission event is demonstrated to be an important mechanism in controlling indoor hygiene. Higher turbulence level dilutes and disperses local concentration peaks faster, consequently leading to faster activation of ventilation outlets and added filtration in exhausting pathogens from the indoor space.The aerosol concentration results are further refined to obtain estimates for the infection probabilities. The Wells–Riley model is formally and rigorously extended to rely on CFD-predicted time- and space-dependent realistic concentration fields and to yield time- and space-dependent realistic infection probability fields.In the infection-risk analysis, the original Wells–Riley model gave three to four times smaller mean infection probabilities than the LES-based calculation. This underestimation is due to the perfect and instantaneous mixing assumption embedded in the analytical model, which leads to the omission of local peak concentrations and overestimation of the ventilation exit term. The underestimation stems from the mean concentration, subsequently affecting the inhaled cumulative doses and finally infection probabilities. The extent of underestimation varies depending on the relative positions of the infection source(s) and the ventilation outlet vent(s). The two air purifier units used in the study increased the existing ventilation capacity by 65%. This leads to a relative reduction of 39% in the nearly asymptotic  may, therefore, be very expensive and impractical solution. The second strategy which relies on space partitioning is confirmed not to reduce the infection risk. The partitioning does lead local accumulation of infection-risk, but the general risk level is not reduced. Therefore, it should not be considered a risk-reduction strategy alone. When air purifiers are applied together with space dividers, the situation improves dramatically, but this approach requires that added filtration units should be added to each individual segment. Otherwise, the risk distribution may become highly variable. Thus, the best efficiency from a small number of air purifier units is obtained without obstructing the air flow within the indoor space.


The automatic analysis of endoscopic images to assist endoscopists in accurately identifying the types and locations of esophageal lesions remains a challenge. In this paper, we propose a novel multi-task deep learning model for automatic diagnosis, which does not simply replace the role of endoscopists in decision making, because endoscopists are expected to correct the false results predicted by the diagnosis system if more supporting information is provided. In order to help endoscopists improve the diagnosis accuracy in identifying the types of lesions, an image retrieval module is added in the classification task to provide an additional confidence level of the predicted types of esophageal lesions. In addition, a mutual attention module is added in the segmentation task to improve its performance in determining the locations of esophageal lesions. The proposed model is evaluated and compared with other deep learning models using a dataset of 1003 endoscopic images, including 290 esophageal cancer, 473 esophagitis, and 240 normal. The experimental results show the promising performance of our model with a high accuracy of 96.76% for the classification and a Dice coefficient of 82.47% for the segmentation. Consequently, the proposed multi-task deep learning model can be an effective tool to help endoscopists in judging esophageal lesions. The incidence of esophageal lesions is getting higher and higher due to the continuous growth of the population and the changes in the lifestyles of people. Especially, esophageal cancer is the tenth-most common cancer in the world and has the sixth-highest mortality rate . FortunaAt present, the typical strategies currently used to detect esophageal lesions are gastrointestinal endoscopic screening such as white light imaging (WLI), narrow-band imaging (NBI), capsule endoscopy, and so on. Unfortunately, there are some deficiencies in using gastrointestinal endoscopic to diagnose esophageal diseases. It is difficult to accurately diagnose patients with esophageal cancer based on WLI because of its lower sensitivity and specificity . NBI notRecently, benefiting from the rapid development of deep learning, many advanced techniques based on deep learning have been applied in various medical fields, such as skin lesion segmentation , retinalAlthough deep learning methods have made great contributions, especially for those diseases with a high fatality rate or difficult to diagnose, most of them have a common problem that they only focus on a given task or a certain type of disease, such as esophageal squamous cell carcinoma . TherefoTo solve the above problems, developing a model to achieve multiple tasks is a good strategy by using the shared features between different single-task deep learning models. Multi-task learning is an important paradigm of deep learning. Its goal is to mine common features between different tasks to improve the performance of the model and its better generalization ability . The basTherefore, based on the hard parameter sharing of multi-task learning, we developed a multi-task deep learning model that achieves classification and segmentation for esophageal lesions using endoscopic images at the same time. The classification task no longer only focuses on the prediction of a certain type of disease but can also predict esophagitis, esophageal cancer, or normal images. Additionally, image retrieval in the classification task is used to provide more aid in diagnostic information. For each query image, it can find the five most similar images from the historical patient libraries. When the endoscopists encounter a controversial or difficult-to-diagnose sample, the retrieval can provide more aid information besides classification results. The segmentation task can locate the cancer lesion area. It is better than the methods based on detection, because it avoids the problem of inaccurate positioning but a high confidence level. To achieve a better segmentation performance, we designed a mutual attention module to capture more diverse features in the segmentation task.(1)We proposed a novel multi-task deep learning model for automatic esophageal lesion analysis. It can synchronously achieve multiple tasks, including classification and segmentation for esophageal lesions.(2)To provide endoscopists with more supporting information in classification, we built a retrieval module on the classification branch to assign a confidence for each prediction result. Classification and retrieval can be optimized at the same time without affecting each other.(3)To improve the performance of esophageal cancer segmentation, we designed a mutual attention module in the segmentation task that can generate weight matrices from different features and guide each other to obtain diversified features.(4)The experiments show that the proposed model is better than other similar methods and can effectively help endoscopists improve the accuracy of a diagnosis.In summary, our contributions are mainly the following four:In this section, we discuss three types of works that are most related to our work, including esophagus classification, esophagus segmentation, and a multi-task medical image analysis.A lot of the traditional classification method was proposed to classify esophageal lesions based on color and texture information. For example, Munzenmayer et al.  proposedInstead of the traditional classification methods, the deep learning-based method has been used in the classification of esophageal disease using endoscopic images. Kumagai et al.  construcMany efforts are devoted to addressing esophagus segmentation by developing effective methods. Before deep learning, most methods used shape or appearance models to guide esophagus segmentation ,28. HoweInspiring by the successful application of deep learning-related methods in esophagus classification, a growing number of deep learning-based approaches have been used in esophagus segmentation. For instance, Mendel et al.  used conAlthough the above-mentioned deep learning-related methods have achieved significant results in esophagus classification or segmentation, they can only achieve a given task. There are few methods that simultaneously realize classification and segmentation for esophageal endoscopic images. Wu et al.  proposedIn this section, we will first introduce the multi-task deep learning model as a whole and then describe each subtask in detail separately.In order to provide richer and more effective diagnostic information, we used hard parameter sharing to develop a novel multi-task deep learning model that realizes two tasks. The first task is to distinguish whether this sample is cancer, esophagitis, or normal. Additionally, based on the features of classification, image retrieval was used to find a group of images that are the most similar to the input images. The second task is to determine the lesion area when the image is cancer. It can be observed from At last, in order to reduce the negative impact caused by the imbalance of sample categories, the classification task adopts Focal loss  as the lFor the segmentation task, the cross-entropy loss function is used as the loss function. It is given by:The goal of the first task proposed is to determine the type of input image . This can be obtained through the classification branch. To help endoscopists make a more accurate diagnosis, we introduced a deep retrieval module  to provik hit candidates to average all n candidates as the confidence level of the prediction. The confidence level can be defined as follows:qC indicates the confidence level of the prediction. kS means the similarity of the thk hit candidate.In the training set, the number of samples for each category is different. When we obtain the predicted category and ranking of the query through the system, we take the top-n features from the ranking as candidates. n refers to the number of samples of the predicted category. Then, the missed candidates whose category is different from the predicted category of the query are removed from the candidates, leaving k hit candidates. Finally, we use the similarity of When it was determined that the input image was a cancer lesion, the proposed model can mark where the lesion area was by the segmentation task. Compared with other detection-based methods that can only mark the approximate area of the lesion, Transformer  showed eThe dataset used in this study contains 1003 upper gastrointestinal endoscopy images from Kiang Wu Hospital. All images can be categorized into three classes . Among them, the training set has 805 images , and the testing set has 198 images . All images have pathology reports, and the lesion areas are marked by experienced endoscopists. For data augmentation, we adopt random crop, random rotation between 45 and 135 degrees, horizontal flip, and vertical flip for the training set. This comprehensive data augmentation scheme makes the network converge better. The processes of training and testing the proposed model using the dataset are shown in To quantitatively analyze the performance of the proposed models, we employed the following three different metrics for two tasks.C is the number of types of esophageal lesions. TP (True Positives) means the number of positive samples is correctly classified. TN (True Negatives) means the number of negative samples is correctly classified. FP  means the number of negative samples is wrongly classified as positive. FN  means the number of positive samples is wrongly classified as negative.For the classification task, we calculated the Accuracy Precision, Sensitivity, Specificity, Negative Predicted Value (NPV), and F1-score to evaluate the performance. They are defined as:q, we obtained a ranking of each training set image using Hamming distance as the similarity measure. The precision of the query q in the top k rankings can be defined as:hit(i) refers to whether the query q is consistent with the thi image label in the ranking.To evaluate the image retrieval module, we adopted a ranking criterion to evaluate the retrieval performance. Given a query X represents the ground truth, which is masked by endoscopists, and Y is the segmentation region of the proposed model.For the segmentation task, we adopted the most commonly used Dice coefficient and Intersection Over Union (IoU) as the evaluation metrics. They are defined as:The results of classification and retrieval are shown in Additionally, in order to ensure the effectiveness and robustness of the proposed model, we compared five different CNN architectures, i.e., VGG-16 , ResNet-Next, to better verify the performance of the proposed model, we conducted confrontation ablation experiments on whether endoscopists refer to the results provided by the proposed model with the confidence of the predicted category. The endoscopists who participated in the testing included a senior (endoscopy experience > 10 years) and junior (endoscopy experience < 10 years), and the ratio was approximately 1:1. It is shown in In Consequently, the proposed model can not only help endoscopists improve the accuracy of diagnosis, but the additional information provided by the retrieval module can further help endoscopists make a more accurate diagnosis. This demonstrates that the proposed model can be applied to daily clinical diagnoses.Six common segmentation models were compared, including U-Net , PSPNet Furthermore, the segmentation results of the proposed model and other models are shown in In this work, to solve the challenge of endoscopists in diagnosing esophageal lesions , we propOn the other hand, the proposed model can mask the esophageal cancer region with a high Dice coefficient (71.27%) and IoU (82.47%). Although previous studies have applied deep learning to classify or segment esophageal diseases, deep learning models can seldom classify and segment esophageal lesions at the same time. To the best of our knowledge, this is the first such multi-task deep learning model developed in Macau.The proposed multi-task deep learning model not only achieved high accuracy in esophageal lesion classification but also output the mask of esophageal cancer, thereby reminding the endoscopists to pay attention to the location of the suspicious lesion. We hope that the proposed model can be used in the following situations: during the examination, it finds and masks a suspicious area under WLI; this will prompt the endoscopist to use the NBI mode and perform a biopsy. We are currently developing the multi-task deep learning model based on WLI and NBI images to establish a more subjective method that combines the current white light algorithm with the NBI algorithm.Our work has several limitations. First, since our datasets only come from Macau Kiang Wu Hospital, the sample size  was small. Therefore, we plan to collect more images of different esophageal types from different regions and invite more endoscopists to participate in our research. This will make up for the flaws of the imperfect data in this type of research. Second, our work only focused on cancer, esophagitis, and normal images and did not include other esophageal diseases such as esophageal polyps, esophageal leiomyoma, and ectopia of gastric mucosa. In the future, we will persistently collect these esophageal lesions and use them in the proposed model. Finally, we considered improving the robustness of the multi-task deep learning model for poor-quality images. The robustness of the multi-task deep learning model can be obtained by using poor-quality images during the training process. However, low-quality images will impair the convergence of the model and are not easily recognized by the model. Another feasible method is that we can use a model with a stronger learning ability to weaken the negative impact of low-quality images. We can also use a model with a stronger learning ability to weaken the negative impact of low-quality images. For example, we can consider the transformer  model thIn this paper, we constructed a multi-task deep learning model consisting of share layers and task-specific layers to achieve the classification and segmentation of esophageal lesions. The classification task determines the lesion type of the input image. Based on the classification task, image retrieval was used to provide more supporting information to endoscopists by finding a few samples that were the most similar in the input image. If the input image is cancer, the location of the cancer is further determined by cancerous area segmentation. To ensure the effectiveness and stability of the segmentation task, we developed an attention mechanism. The proposed model was evaluated on the testing set. The experimental results demonstrated that it was able to show a favorable diagnostic performance for classifying esophageal lesions with high accuracy and could achieve a high Dice coefficient and IoU for esophageal cancer segmentation. Furthermore, we invited endoscopists to compete with our model. The results showed our model achieved a classification accuracy of 96.76%. The accuracy of the endoscopists was 83.84%. Based on these promising results, the proposed multi-task deep learning model could become a potential assistant to help endoscopists in judging esophageal lesions.


With the application of advanced composite materials in High-Aspect-Ratio wings (HARW), the randomness of structural parameters, such as elastic modulus and Poisson's ratio, is enhanced. Hence, in order to explore the whole picture of aeroelastic problems, it is of great significance to study the role of random structural parameters in aeroelastic problems. In this paper, the dynamic response of flexible HARW considering random structural parameters is analyzed. An aeroelastic model of a one-dimensional cantilevered Euler–Bernoulli beam considering aerodynamic forces acting on the wing is established based on Hamilton's principle. Adopted the idea of simplifying calculation, the effect of random structural parameters is analyzed. Then, considering the elastic modulus and torsional stiffness as continuously one-dimensional random field functions, and discretized by local method. The first and second order recursive stochastic nonlinear finite element equations of wing are derived by using perturbation method. Based on it, statistical expression of aeroelastic effects of the wing is derived. Monte Carlo method is adopted to verify the effectiveness of the method. Numerical simulations indicate that the method proposed can well mirror the statistical characteristics of aeroelastic response. These wings have the characteristics of lightweight and great flexibility. The flexibility of long-span wings makes it possible to generate large deflections during in flight. It is unlikely to get the accurate aeroelastic results by the conventional linear aeroelastic analysis method. Studies has been shown that the aeroelastic results of the nonlinear analysis are different from those of linear analysis4. Due to the development of modeling technology of complex nonlinear systems and the development of mathematical calculation tools, nonlinear aeroelastic problems have been greatly developed in the past two decades5.Aeroelastic problems of flexible HARW are of great concern in developing High-Altitude Long-Endurance aircraft, solar-powered aircraft and heavy rotorcraft6 established a complete aircraft model of high altitude long endurance aircraft and studied its nonlinear aeroelasticity. Significant changes can occur in the wing’s natural frequencies because the tip displacement function is very closely related to the flutter speed. Tang7 carried out theoretical analysis and wind tunnel experimental study on geometric nonlinearity’s influence on aeroelastic characteristics of flexible HARW. The effects of the geometric structural nonlinearity are determined by the ratio of the flap and chord wise bending stiffness. When the ratio is relatively small, the boundary of flutter instability has a little change due to the structural nonlinearity and preflutter static deformation. Garcia8 analyzed the flexible HARW’s aeroelastic characteristics at transonic speeds. The results show that the nonlinear static aeroelastic characteristics of flexible wing with high aspect ratio are different from those of linear aeroelastic response in transonic flow. These differences are owing to the coupling of large transverse bending deflections with drag force, and the kinematic effects of bent wing structures at transonic speeds. Tang9 studied the effect of geometric nonlinearity on the limit cycle oscillations (LCOS) of flexible HARW by both experiment and theoretical calculation. The research shows that the LCO hysteretic response is usually dependent on a delicate balance between stall aerodynamics and the structural nonlinear forces. Shearer10 presented a six-degree-of-freedom coupled vehicle dynamics model, which is an upgraded version of the high-aspect ratio lifting surface structure model based on low-order nonlinear strain. The dynamic responses of nonlinear rigid body coupled nonlinear rigid body and fully nonlinear aeroelastic aircraft in pitch, roll and yaw flight are analyzed by using this model. The simulation results show that the modeling of nonlinear structure is of great significance for high altitude long-endurance flexible aircraft. Su12 used the finite element framework based on low-order nonlinear strain to model the high-flexible wings and analyzed the nonlinear aeroelastic coupling model.Patil13. Physical properties of structure materials, such as Young's modulus, Poisson's ratio, etc., structural properties or even nonlinearities15 can be considered as random factors, which can be represented as random field distribution functions. Due to these factors, the response of structure under excitation will not be linear, but appears as time and field functions of random variables.With the development of aerospace technology, many advanced concepts and new materials and structures have been applied to the next generation of aircraft’s design and manufacture. But at the same time, many new problems have arisen. One of them is the use of composite materials, which leads to many random structural parameters in the aircraft manufacturing process18. However, in reality, the aeroelastic response of wing will be affected by random structural parameters. It becomes even more problematic when nonlinearities are involved because some non-negligible nonlinear factors will cause serious instability. Based on classical finite element method, stochastic finite element method (SFEM) is an extension based on the stochastic framework, which is used to solve stochastic problems with finite elements with random properties19. It can deal with both the structural parameters stochastic problems, and the input stochastic problems21. By analyzing the microstructure model of composites, Stefanou22 obtained the statistical information of the probability distribution and correlation of the composite properties at the micro scale. Based on SFEM, the macro and micro mechanical responses of the composite structure were calculated, and the influence of the micro properties on the response was studied. Sepahvand23 regards the damping coefficient of fiber reinforced composites as a random variable, describes the random characteristics of structural damping by generalized polynomial chaotic expansion method, and analyzes the damping vibration by spectral stochastic finite element method. Zhou24 combined the homogenization method and stochastic finite element method, considered the micro and macro uncertainties of material properties and ply angle, proposed a multi-scale method to analyze the response and reliability of composite structures, and used Monte Carlo Simulation (MCS) simulation to verify the method. Chang25 established a bridge-vehicle model based on SFEM, considering the random material properties and moving loads, analyzed the statistical dynamics. Based on SFEM concept Huh26 proposed an efficient algorithm to evaluate the seismic risk of nonlinear structures, seismic excitation and other uncertainty parameters are incorporated for modeling. Examples showed the validity of the method for real nonlinear structures evaluation subjected to seismic loadings. Stochastic seismic analysis of long-span bridges with Carbon fibre reinforced polymer cables (CFRP) are presented based on SFEM and MCS by vdar27, geometric nonlinear effects and material uncertainties are considered, the feasibility of using CFRP cables in long-span bridge under seismic excitation is discussed. Niranjan28 modeled diverse material properties of functionally graded materials and piezoelectric materials with volume fraction exponent as independent random variables. SFEM was used to obtain the statistics of nonlinear natural frequency response of second-order. Xu29 uses SFEM to model and verify the aeroelasticity of slender aircraft with stochastic factors. The results show that the established model has good statistical characteristics of aeroelastic response.In traditional way, aeroelastic analysis is carried out under the condition that the random structural parameters are regarded as determining valuesSFEM is adopted to study the structural reliability, disaster evaluation and composite properties. And the aeroelastic response of HARW with geometric nonlinearity or SFEM are studied by some existing literatures, but most of these articles consider the random parameters as determining value. The aeroelasticity considering random parameters of HARW has not been reported.On the basis of the previous study, SFEM is used to study the effect of the random structural parameters on the aeroelastic response of HARW considering geometric nonlinearity in this paper. Firstly, the dynamics model of the cantilevered wing is established under the action of the unsteady aerodynamic force. For the purpose of simplifying the calculation, the elastic modulus as well as torsional stiffness are selected as random structural parameters. Then, these two parameters are considered as random field functions, using local average method to discretize these functions in order to obtain the results of aeroelastic response, the SFEM is adopted as the solver for the model, and the comparisons between the SFEM and Monte Carlo method are given. Finally, under the random factors, the aeroelastic response of the wings is studied.4. The associated coordinate system is defined as shown in Fig. HARW in air vehicle are subject to gravity, aerodynamic force, and these forces will result in multiple elastic deformation forms of the structure. Among them, bending and twisting are the most basic forms. In order to obtain a simplified model, we regard the flexible HARW as a cantilevered Euler–Bernoulli beamThe wing’s elastic deformation is defined in the coordinate system The elastic potential energy can be expressed asHere, A0A2 and A0A5 are taken in the cross-section. The length is As shown in Fig. It is assumed that the length of the micro-segment after deformation remains unchanged. Taylor expansion of the above formula yields:From Fig. After Taylor's expansion,Assuming torsional deformation of the micro-segment is small, the displacement at any point The relationship between strain and displacement can be expressed as follows:The kinetic energy generated by the deformed beam is as follows:Here, The work Substitute Eqs. , 8), an, an8), aAlthough the quasi-steady approximate method is simple, and its accuracy is poor in dynamic aeroelastic analysis of HARWs. However, the aerodynamic model with higher precision would increase the difficulty of obtaining partial derivatives of aerodynamic force to structural random paramaters. At the same time, more accuracy aerodynamic model would bring in more uncertainty and nonlinearity, which makes it more difficult to study the influence of structural random factors on the model, and further makes the study of the influence of structural random factors on the aeroelastic response more complicated. Since the research focus of this paper is mainly on the influence of structural random factors, in order to simplify the calculation, we adopt quasi-steady approximate method to carry out the research while ensuring the certain accuracy of the aerodynamic model.30 proposed the theory of unsteady aerodynamics for a thin airfoil undergoing small oscillations in incompressible flow. This method is used to study the amplitude and phase problems of sinusoidal unsteady aerodynamic force at different reduce frequencies. Peters31 proposed a finite state aerodynamic theory for a two-dimensional thin airfoil operating in inviscid and incompressible flow. This theory is derived directly from potential flow theory with no assumptions on the time history of airfoil motions. The aerodynamic states are the coefficients of a set of induced-flow expansions. Based on30, for a particular reduced frequency, the lift and moment per unit span for a wing about its aerodynamic center are given byTaking the quasi-steady assumption, the reduced frequency Here, 32, so Eq. 15) are tAfter establishing the model of a flexible HARW under unsteady aerodynamics, the SFEM will be used to solve the model. Firstly, the Hermite element is used to discretize the dynamic equations above. The transverse displacement within the element 34, 35, In this study, take the elastic modulus and the torsional stiffness as random fields The variance reduction function From Eqs. –21, The The form of the first partial derivative of the stiffness matrix with respect to the random parameters isSubstitute Eqs. , 23 intoZeroth-order:First-order:The expression of the displacement’s mathematical expectation and the displacement covariance can be described asThe aeroelastic displacement expectation and covariance with first order accuracy can be obtained by substituting Eq.  into Eq.From Eqs. –28, the cy, Eqs. –26 are eAs seen in Eq. , due to s in Eq.  have thes in Eq. , the secZeroth-order:First-order:whereSecond-order:After solving  by Eqs. –35, the To check the effectiveness of the results, in the following section, we introduce the classical Monte Carlo method for comparison. Furthermore, the influence of random factors on aeroelasticity is analyzed.36, scholars often use MCS to check the validity of SFEM38. In Sect. In order to confirm the effectiveness of the theory derived in the study above, and to analyze the effect of the random factors on aeroelasticity, numerical simulations are carried out in this part. In Sect. 40 are used for comparing. The simulation data5 are shown in Table In order to verify the fidelity of the model in this paper, the methods of literatures39, Fig. 40.Based on these simulation data, Static deflections of the clamped cantilevered wing is simulated and compared with references. Figure Simulation results show that the model presented in this paper can describe the dynamic behavior of the wing well in a nonlinear large deformation case, and the model shows good agreement with literature results.Simulation data is shown in Table Considering In Fig. To further check the validity of the SEFM aeroelastic model, Fig. Figure After further comparing the calculation results of displacement expectation and variance, it is found that the calculation error of variance is greater than expectation. The reason for this is that the variance results derived in this paper have only first order accuracy.The curves of expectations and variances of wing torsion angle can be seen in Fig. The above simulation is used to compare the displacement expectations with second-order accuracy. However, from the expression of the second-order recursive equation and the simulation calculation, it can be concluded that solving the displacement expectation with higher accuracy needs a large amount of calculation. In particular, it is necessary to calculate the second-order partial derivatives of damping matrix and stiffness matrix for random variables. In contrast, the first order recursive equation has a relatively small amount of calculation. In order to explore the accuracy of the calculation results of the first-order recursive equation, Table It can be seen from Table This section studies the impacts of random variables with diverse correlations on aeroelastic response based on the parameters of Table In Fig. Let δu. Since When the correlation distance From Fig. Figure The reason for this phenomenon is that the random field correlation and variance reduction function will not change all the time. As Simulations indicate that the proposed method could estimate the statistical characteristics of the wing’s aeroelastic effects accurately.Compared with the second order recursive equation, when the coefficient of variation is small, the first order recursive equation not only ensures good accuracy, but also has better computational efficiency.The correlation of random structural parameters will affect the aeroelastic displacement response of wing. The stronger the correlation, the worse the accuracy of the first order recursive equation.In this paper, the aeroelasticity of the flexible High-Aspect-Ratio wing is studied by considering elastic modulus and torsional stiffness as random parameters. The wing is simplified as a cantilevered Eula-Bernoulli beam, and an aeroelastic model is established. Then, the first order and second order recursive stochastic nonlinear finite element equations of wing considering the influence of structural random parameters are derived by using local averaging and perturbation methods. The expression of the numerical characteristics of the aeroelastic response of the wing is obtained. After that, Monte Carlo method is adopted to verify the effectiveness of the method. Conclusions of this paper mainly include:The elastic modulus and torsional stiffness of the wing are chosen as the random factors in this study, but there are many random factors in actual structures. In the future work, other random factors  will also be considered. On the other hand, how to select the first or second order recursive equation according to the coefficient of variation and the correlation of random structural parameters is worth further study.


In this paper, we propose a method to enhance the performance of segmentation models for medical images. The method is based on convolutional neural networks that learn the global structure information, which corresponds to anatomical structures in medical images. Specifically, the proposed method is designed to learn the global boundary structures via an autoencoder and constrain a segmentation network through a loss function. In this manner, the segmentation model performs the prediction in the learned anatomical feature space. Unlike previous studies that considered anatomical priors by using a pre-trained autoencoder to train segmentation networks, we propose a single-stage approach in which the segmentation network and autoencoder are jointly learned. To verify the effectiveness of the proposed method, the segmentation performance is evaluated in terms of both the overlap and distance metrics on the lung area and spinal cord segmentation tasks. The experimental results demonstrate that the proposed method can enhance not only the segmentation performance but also the robustness against domain shifts. Medical image segmentation is aimed at distinguishing the boundaries of lesions or organs in medical images acquired through X-ray, computed tomography (CT), magnetic resonance imaging (MRI), and other techniques. Segmentation results have been used to accomplish valuable clinical objectives in multiple practices, such as tumor detection to enable precise diagnosis and volume analysis to enable treatment planning .Early studies on medical image segmentation adopted the methods such as edge detection, template matching, statistical shape models, and active contours . The rapNevertheless, although CNN-based segmentation networks demonstrate a high prediction performance, these networks produce anatomically abnormal segmentation results in medical images in certain cases . CompareConsidering these aspects, this study is aimed at establishing a method of learning anatomical structures using an autoencoder such that the prediction of a segmentation network is performed in the learned anatomical feature space. This framework can help enhance the segmentation performance for the region of interest by reflecting the anatomical information in the learning process of the segmentation network. The proposed method differs from the existing approaches that use the anatomical features provided by pre-trained autoencoders ,13. SpecThe proposed method was evaluated using the overlap measures including the intersection over union (IOU) and dice similarity coefficient (DSC), and distance measures such as the average contour distance (ACD) and average surface distance (ASD). To verify the effectiveness of the proposed method, we evaluated the performance of lung segmentation on chest X-rays in the widely used public benchmark datasets of the Japanese Society of Radiation Technology (JSRT) and Montgomery County (MC) ,15,16 anThe remaining paper is organized as follows. The existing CNN-based segmentation networks that incorporate anatomical information into the learning process are introduced in Fully convolutional networks (FCN) represent one of the early deep learning networks for semantic segmentation . In an FFurthermore, U-Net is one of the most popular networks based on the encoder–decoder structure for segmentation tasks in the field of medical imaging . This neMany researchers have proposed novel architectures to enhance the segmentation performance; however, such architectures do not consider the common structural information of input images as the objective is usually to perform segmentation in natural images. However, in the case of medical images, the degree of standardization is usually relatively large. If a common structure exists in the full image, it is desirable to learn this structural information to obtain more robust and high performance segmentation networks.In the medical imaging field, a key factor to be considered to enhance the segmentation performance is that a segmentation model must learn the prior knowledge regarding the anatomical structure, such as an organ’s shape or placement . To thisIn addition, certain researchers used autoencoders to learn the low-dimensional features for anatomical structures. Oktay et al.  extracteNotably, the previous studies that reflected the anatomical structure information in CNN-based segmentation networks focused only on enhancing the segmentation performance and did not consider the robustness to domain shifts. In addition, although the approaches involving the autoencoder could effectively enhance the segmentation performance, two-stage methods were required to be used. Considering these aspects, this paper proposes a single-stage method in which the autoencoder and segmentation network are jointly trained to enhance not only the segmentation performance, but also the domain generalization capability.This section describes the proposed method, which includes a denoising convolutional autoencoder (DAE) and a segmentation network. The DAE is trained to learn the anatomical information by using segmentation labels , and the segmentation network is trained to perform accurate segmentation. To reflect the anatomical information obtained by training the DAE in the segmentation network, the output of the last encoder layer of the segmentation network and corresponding output of the DAE are constrained through a loss function. Unlike previous studies that relied on a pre-training autoencoder, the proposed approach is a single-stage method in which the segmentation network and autoencoder are simultaneously trained. First, the details regarding the DAE employed in the proposed method are introduced, and subsequently, the overall framework of the proposed method is described.The DAE is one of the dimensionality reduction methods based on deep neural networks, which can be used to extract the anatomical information in the data. In contrast to the conventional autoencoder, the DAE is trained to reconstruct the noise-added inputs to the original input data . Gaussiay be a segmentation label  used as an input to the DAE. The encoder y. The loss function z is obtained as L. As the training proceeds, the output z of the encoder contains the abstracted anatomical structure information learned by the data. In other words, the encoder performs an embedding that maps the input space to the anatomical structure space. Therefore, the feature space modeled by the encoder can be used to constrain the embedding space of the segmentation network.The goal of the segmentation network is to learn how to produce segmentation results from the anatomical structure space modeled by the DAE. To accomplish this goal, the output features from the segmentation encoder and DAE encoder should be tightly coupled. The proposed strategy is to apply the constraint by using a loss function to transfer the anatomical information being learned using the DAE to the segmentation network. In the proposed method, the U-Net architecture is used as a segmentation network.x as an input, and outputs a segmentation result h, and then the decoder h through consecutive convolutional layers to produce a segmentation result. This network is trained using a loss function The overall training scheme of the proposed method is illustrated in h and z denoted as The key concept of the proposed method is to impose a constraint on the feature spaces constructed using the segmentation encoder In general, both the model parameters of refer to .The total loss function This section describes the datasets and performance metrics considered to evaluate the proposed method. In addition, the results of experiments conducted on two segmentation tasks, lung segmentation in chest X-rays (CXR) and spinal cord segmentation in MRI, are presented, which demonstrate that the segmentation network trained using the proposed method exhibits a high performance and domain robustness. In addition, we present the results of an ablation study performed to validate the design choices in our proposed method.For lung segmentation, we used two public CXR datasets: JSRT  and MC . JSRT issite1), Polytechnique Montreal (site2), University of Zurich (site3), and Vanderbilt University (site4). Specifically, the dataset consists of 80 MRI images corresponding to 20 cases from each site. The data from these sites exhibit individual visual characteristics mainly due to the imaging equipment from different vendors being used. Therefore, in the evaluation of the domain robustness of the segmentation networks, the images from each site were considered to correspond to a single domain.For the spinal cord segmentation task with the MRI images, we used the dataset employed in the spinal cord gray matter challenge . The datS is the predicted region, and G is the ground-truth. The IOU can be defined as the ratio of the intersection and union between G and S:Intersection over union (IOU): the IOU is a measure of the degree of overlap between the region of the ground-truth and region predicted by the segmentation network. In Equation , S is thS and G, as indicated in Equation (Dice similarity coefficient (DSC): the DSC can be used to evaluate the overlap performance, similar to the IOU. This indicator also measures the degree of overlap between Equation . Note thS and G, respectively. S to G. The ACD and ASD can be computed as follows:Average contour distance (ACD) and average surface distance (ASD): the ACD and ASD indicate the extent of separation of the ground-truth and predicted region. Notably, the overlap measures such as the IOU and DSC do not consider whether the false positive pixels are near or far from the ground-truth. Let To evaluate the segmentation performance, multiple performance metrics including overlap and distance measures were adopted.h and z should be the same size to compute the embedding loss U-net  was utilIn general, to apply the pre-trained anatomical information to the segmentation network, the ACNN performs lower-dimensional projections of both the segmentation predictions and ground-truths based on the pre-trained autoencoder, and computes the shape regularization loss between these projections. In this experiment, we adopted the binary cross-entropy and mean squared error as the segmentation loss and shape regularization loss, respectively, as in the original study. The weight of the shape regularization loss was set as 0.01 according to the validation process. The SRM  is a varEquation  as 1.0.All the methods were trained using the Adam optimizer  with a lIn addition, we observed that the methods in which the anatomical information was reflected during training, ACNN and SRM, outperformed the baselines, U-Net and U-Net w/o aug. Specifically, the ACNN and SRM exhibited an enhanced performance in terms of the distance metrics and all metrics in the case of the JSRT and MC datasets, respectively. Nevertheless, the proposed method outperformed all the methods in terms of all overlap and distance metrics on both the datasets. The comparison results indicated that in terms of the ASD, the proposed method exhibited an enhancement of 5.6% and 5.4% over the JSRT and MC datasets, respectively, against the second-best performing model SRM. This result demonstrated that the proposed method can help the segmentation network learn the global anatomical structure to be segmented by producing segmentation outputs from the anatomical feature space modeled by the autoencoder.The visualization results are presented in To gain further insight into the proposed method, the reconstruction results from the trained DAE  and tested the trained model by using images from other sources. For example, the JSRT dataset was used for training, and the trained model’s performance was evaluated using the MC dataset in the case of the lung segmentation task. In general, if a network exhibits a high performance on datasets from unseen domains, the network is considered to be robust to domain shifts. We conducted similar experiments using the spinal cord dataset: images from each site, The domain robustness  performance of the segmentation models in the lung segmentation task is summarized in The visualizations of several segmentation results are presented in site2→others, site3→others, and site4→others. For models trained with each source domain, we evaluated the segmentation performance on the images from all other domains. The proposed method exhibited a higher generalization capability on unseen domains, as presented in site3, the proposed method achieved better segmentation results on other domains, especially in terms of the distance metrics. For example, when the source domain was site4, the ACD value averaged over other domains including site1, site2, and site3 was improved by 14.8% compared to the second-best model SRM. According to the averaged performance over three experimental settings, the baseline U-Net, ACNN, and the proposed method showed a similar segmentation performance in terms of the overlap measures, although the distance performances of the proposed method were significantly improved. Specifically, the average ACD value for the proposed method was 0.667, corresponding to a 9.7% improvement over the baseline U-Net.For the spinal cord segmentation task, three experimental settings were considered: site2, site3, and site4 as the source domains, respectively. Similar conclusions as in the previous experiment could be derived. The comparison methods produced several false positive predictions, whereas the proposed method could accurately predict the target area. The quantitative and qualitative results indicated that valuable structural information such as a global shape or location in common across multiple domains can be learned using the proposed method.In this paper, we propose a method to learn global anatomical structures in medical images by using a denoising convolutional autoencoder and constraining a segmentation network through a loss function such that the prediction of the segmentation model is performed in the learned anatomical feature space. Unlike previous studies in which anatomical priors are considered using a pre-trained autoencoder, we propose a single-stage approach in which the segmentation network and autoencoder are jointly learned. To demonstrate the advantages of the proposed method, extensive experiments were conducted on two medical image segmentation tasks: lung segmentation in CXRs and spinal cord segmentation in MRI images. The experimental results indicate that learning anatomical priors using the proposed method can help enhance the segmentation performance. In addition, to demonstrate the additional benefits of learning the anatomical structures, we investigated the domain robustness of the proposed method. The results indicate that the proposed method can enhance the robustness of segmentation networks against domain shifts. This domain robustness property will be particularly useful for other medical applications such as cranial implant design  or preci


Prediction of pedestrian crossing behavior is an important issue faced by the realization of autonomous driving. The current research on pedestrian crossing behavior prediction is mainly based on vehicle camera. However, the sight line of vehicle camera may be blocked by other vehicles or the road environment, making it difficult to obtain key information in the scene. Pedestrian crossing behavior prediction based on surveillance video can be used in key road sections or accident-prone areas to provide supplementary information for vehicle decision-making, thereby reducing the risk of accidents. To this end, we propose a pedestrian crossing behavior prediction network for surveillance video. The network integrates pedestrian posture, local context and global context features through a new cross-stacked gated recurrence unit (GRU) structure to achieve accurate prediction of pedestrian crossing behavior. Applied onto the surveillance video dataset from the University of California, Berkeley to predict the pedestrian crossing behavior, our model achieves the best results regarding accuracy, F1 parameter, etc. In addition, we conducted experiments to study the effects of time to prediction and pedestrian speed on the prediction accuracy. This paper proves the feasibility of pedestrian crossing behavior prediction based on surveillance video. It provides a reference for the application of edge computing in the safety guarantee of automatic driving. Pedestrians are one of the main participants in the transportation system in urban environments. They become the most defenseless road users due to the lack of protection measures and vulnerable to the threats to life from traffic accidents. Therefore, to ensure the safe operation of autonomous vehicles, automated systems require the ability to predict pedestrian behaviors, especially at the point of crossing. Knowing as soon as possible if a detected pedestrian has the intention of intersecting the ego-vehicle path (expecting the vehicle slowing down or braking) is essential for performing safe and comfortable maneuvers preventing a crash, as well as having vehicles showing a more respectful behavior with pedestrians .In the behavioral science literature, the Theory of Planned Behavior (TPB) asserts that behavior extends from intent, which in turn is a product of social-psychological attitudes, subjective norms, and perceived behavioral control. The later integrated model simplified these parts into a path, “attitude/perception-behaviors” . PedestrThe current research on pedestrian crossing behavior prediction is mainly based on vehicle-mounted videos. Due to the limitation of the line of sight, there are some hidden dangers in actual scenes. For example, when pedestrians are blocked by other vehicles or other road elements, it is difficult for the vehicle-mounted camera to observe the pedestrians who will be crossing the street. Therefore, the vehicle cannot timely detect the pedestrian crossing behavior. This is also a common cause of pedestrian crashes. With the development of Cooperative Vehicle Infrastructure System (CVIS), it is possible to use information obtained from surveillance video for autonomous driving. Internet of vehicles communication technology (V2X) makes future vehicle driving no longer rely on vehicle perception alone, but organically connect “people, vehicles, roads, clouds” and other traffic elements . ConnectThis paper focuses on the prediction of pedestrian crossing behavior based on surveillance video. The purpose is to predict whether the target pedestrian will cross the road in a period of time from the monitoring perspective. The dataset used is from a surveillance video camera installed in the University of California, Berkeley, which records a busy street on the campus. The dataset labels the pedestrian’s pose feature points. On this basis, we label the target pedestrian’s crossing behavior to make a dataset for pedestrian crossing behavior prediction under surveillance video. Meanwhile, a novel spatio-temporal feature fusion network is proposed to accurately predict pedestrian crossing behavior from the perspective of monitoring. The network integrates pedestrian posture features, local context features and global context features. Posture features include pedestrian body posture and head posture. The local context features are scene features in a certain range around the target pedestrian, which are extracted from the local scene through a convolutional neural network. The global context is the whole picture, and the optical flow field image processed by optical flow method is fused. First, perspective transformation is used to preprocess the entire image. Then convolutional neural network is used to extract the original image and optical flow field image features, respectively. In this paper, VGG19   network The research results of this paper prove the effectiveness of pedestrian crossing behavior prediction through surveillance video. Through the acquisition and fusion of pedestrian posture, local context and global context in the video, satisfactory prediction results are achieved. This also means that it is feasible to realize the early warning of pedestrians crossing the street in automatic driving through edge calculation. The main contributions of this work can be summarized as follows: (1) Based on the surveillance dataset of the University of California, we have annotated the pedestrian crossing behavior of the target pedestrian, and produced a dataset for the prediction of pedestrian crossing behavior based on surveillance video. The dataset contains 300 video clips of pedestrians crossing the street. (2) A new spatio-temporal feature fusion network is proposed. The algorithm predicts pedestrian crossing behavior by fusing pedestrian posture, local context and global context features. Comparing multiple baseline methods, the model achieves the best results. (3) Through experiments, the effects of TTP and pedestrian speed on the prediction accuracy of each model are studied.Spatial-temporal features. Since the prediction task needs to consider temporal information, most studies use continuous image sequences as the input of the prediction model, which requires the model to accurately extract temporal and spatial information. Spatio-temporal modeling can extract the visual features of each frame through 2D CNN  or graphTrajectory prediction. It is a common way to predict pedestrian crossing behavior through pedestrian trajectory. For example, Lee  proposedPosture features. Pedestrian posture features are direct expressions of pedestrian intentions, such as waving, walking, observing road conditions, etc. Therefore, many researchers use pedestrian posture features to predict pedestrian crossing behavior. For example, Fang  used humFeature fusion. Some other methods focused on novel fusion architecture. Rasouli  proposedCompared with vehicle-mounted videos, surveillance video has a wider perspective and richer extraction of context information especially pedestrian surrounding context and vehicles on the road. The full extraction of this information can make the prediction of pedestrian crossing behavior more accurate.Referring to the benchmark proposed in , we defiThe dataset used in this article is recorded on a busy street on the campus of University of California, Berkeley as shown in The dataset contains 300 videos from the monitoring perspective. Each video has a target pedestrian who will cross the street. Every target pedestrian is marked when it first appears in the image boundary, and the video ends when the crossing is completed, and the pedestrian leaves the image. Each video is about 20 s long at 15 frames per second. The resolution of the video is 1920 × 1080. These videos are collected from different periods of the day. The dataset annotates the target pedestrian’s head and body posture. The head posture is represented by two points, which represent the head position and head direction, respectively. Body posture is represented by 5 body keys. Therefore, as shown in We propose a new multi-source feature fusion model, as shown in We use GRU  for recuIn the formula: Meanwhile, inspired by ,13, we iBefore crossing the road, pedestrians usually walk, wave and wave their hands. In addition, the pedestrian’s head posture also reflects the pedestrian’s intention to cross the street . PedestrThe dataset used in this article has already annotated the key points of pedestrian posture. The extraction of pedestrian posture is not the focus of this article, so we directly use the ground truth pedestrian posture as the input. Pedestrian posture includes body posture and head posture. The head pose is a 4D vector, including head coordinate points and coordinate points representing the direction of the head. The vector composed of these two points can represent the direction of the head. The body posture is a 10D vector, including the horizontal and vertical coordinates of the five points of the pedestrian’s left shoulder, right shoulder, waist center, left heel, and right heel.Pedestrian crossing behavior is usually affected by the surrounding context, such as zebra crossings, intersection signs, etc. In addition, when pedestrians cross the street together, the crossing intention will be greatly affected by the crossing behavior of surrounding pedestrians. Therefore, the understanding of the local context around pedestrians is helpful to predict pedestrian crossing behavior.To define the local context, we take the waist center of the target pedestrian as the center and select the RGB image of 224 × 224 pixels around the center to form the local scene. Then we apply the pre-trained VGG19  model onThe information in the global scene, mainly the traffic information, will have an important impact on pedestrians crossing behavior. The vehicle information on the road, including the distance to the pedestrian, vehicle speed and speed change, must be considered when pedestrians cross the street.In order to highlight the target pedestrian in the image, we use two line segments with a width of 60 pixels to represent the target pedestrian. One indicates the pedestrian’s body position and the other indicates the pedestrian’s direction. This can not only connect the road context with the only target pedestrian, but also more directly judge the target pedestrian’s understanding of the current road environment through the pedestrian head direction, such as whether the pedestrian pays attention to the approaching vehicle. At the same time, since the surveillance video camera has an unchanged viewing angle, we perform a fixed perspective transformation on the input image. In this way, the near end and the far end of the camera can be at the same scale, avoiding the problem that the size and speed of the target at the far end of the camera are too small due to the viewing angle. In addition, in order to focus on the moving vehicles on the road, we use the dense optical flow method to obtain the optical flow field of the picture. Since the road background is basically static in the scene under surveillance video, the information of moving targets can be easily extracted by optical flow method. The processing results are shown in the According to the benchmark proposed in , the folTP represents the number of samples with positive label and positive prediction result. TN represents the number of samples with negative label and negative prediction result. FP represents the number of samples with negative label and positive prediction result. FN represents the number of samples with positive label and negative prediction result.Accuracy represents the proportion of correct data predicted. Precision represents the correct proportion of those data whose prediction is positive. Recall rate indicates the correct proportion predicted by positive samples. Their calculation formula is shown in Equations (7)–(9).Ideally, the higher precision and recall, the better, but the actual situation is that the two affect each other: the pursuit of high accuracy rate will lead to low recall rate; the pursuit of high recall rate will usually reduce the accuracy rate. In order to balance the accuracy and recall rates, the F1 parameter is introduced, and its calculation formula is shown in Equation (10).M is the number of positive samples, N is the number of negative samples, In the case of binary event anticipation, AUC reflects the balanced accuracy of the algorithms.We compare the proposed algorithm with the following four benchmarks to evaluate the performance of our algorithm.Single RNN . First, Multi RNN . Each inSF RNN , a stackPCPA . The attAccording to the summary in , in the The experimental results are shown in In We counted the number of crossing samples and not crossing samples correctly predicted by all models, some models or no models. Correspondingly, the samples are divided into simple samples, medium samples and difficult samples. At the same time, we divide pedestrians into three categories according to their moving speed: fast moving, medium moving and slow moving. The relationship between pedestrian moving speed and sample difficulty level is studied. The moving speed of pedestrians is calculated according to Equation (12) where, The proportion of pedestrian samples at the three speed categories after classification is about 1:1:1. Then, we count the number of simple, medium and difficult pedestrian samples at different speeds. The results are shown in the In order to explore the performance of the model under different TTP we carried out further experiments by changing the TTP. We fixed the observation length to 8 frames (about 0.53 s), increased the TTP from 0.4 s to 2 s, and the step size was 0.2 s. In each group of experiments, the last frame of the positive sample observation sequence starts from t frame and takes 5 consecutive sequences at intervals of 2 frames, that is, the end frames of each observation sequence are t, t + 2, t + 4, t + 6 and t + 8, respectively. According to the experimental results, we selected two indicators of accuracy and F1 parameters to show the prediction performance. As shown in We also conducted some case studies to analyze the behavior types of the pedestrians’ crossing. The cases are displayed in In case 1, the pedestrian will cross but the prediction results of all models are wrong. The reason may be that the pedestrian waited too long on the street, the current situation of road vehicles is complex, and the road environment changes rapidly. This makes the model unable to accurately predict the behavior over a long period of time. In case 2, the pedestrian will not cross but all model’s predictions are wrong. It can be seen from the observation sequence pictures that this is caused by the sudden change of pedestrian’s intention and trajectory. This is also a situation that cannot be accurately predicted by models or even human drivers. In case 3, the pedestrian will cross the street. The prediction of our model is correct, but the other models are wrong. It can be seen from the picture sequence that the pedestrian stayed on the roadside for a long time due to the complex road environment, but he shows behaviors of leg lifting which shows that he is eager to cross the street. Before the vehicle has completely passed, pedestrians cross the street obliquely in the vertical direction, which is a common way for pedestrians to cross the street when they encounter passing vehicles in their daily life. In case 4, the target pedestrian walked towards the road, but then his speed suddenly slowed down, resulting in some model prediction errors. In case 5, the pedestrians will cross the street, and all models predict it correctly. At that time, the target pedestrian was already standing next to the road and was ready to cross the street. The companion next to him was also preparing to cross the street. The pedestrian’s movement was coherent, so all models could predict it correctly.This paper presents a pedestrian crossing behavior prediction model based on surveillance video. Compared with traditional vehicle-based video, surveillance video can capture richer road and vehicle information, which will have a large impact on pedestrian cross-street behavior. In addition, we propose a new feature fusion method, which improves the prediction performance of the model, and obtains higher accuracy, F1 parameters, etc. than the baseline method. When TTP is less than 1.6 s, the accuracy and F1 score of the model can reach more than 80%. This study can be used in the assistant system of auto-driving to warn pedestrian crossing behavior through edge calculation, so as to enhance the safety performance of auto-driving.(1)The surveillance video does not capture all the information on the road, especially the information on the right side of the camera. Pedestrians can observe farther road information than cameras. Therefore, many vehicles that affect pedestrian crossing behavior do not appear in the video.(2)The rules of pedestrian-vehicle interaction when pedestrians cross the street are complex and changeable. The amount of data in the current dataset is difficult to make the model fully learn these rules. Our positive and negative samples are from different stages of pedestrians who will cross. There is a lack of samples of pedestrians who won’t cross. This makes the model less robust and reduces a certain accuracy. These reasons lead to the rapid reduction of model accuracy with the increase of TTP.(3)The proposed method relies on the labeling of key points to encode human posture, which restricts the practical use of the proposed method.However, due to the limited ability and energy, some aspects of the algorithm need to be further improved:Future research will start from these aspects, consider using multiple cameras to broaden the observation field of vision, and increase the number and integrity of datasets to improve the robustness and accuracy of the model. In addition, we will study the detection of human posture key points in surveillance video to realize an end-to-end pedestrian crossing behavior prediction model. At the same time, we also believe that the representation of pedestrian posture is not necessarily the key points of posture. In the future, we will also try different inputs that can contain pedestrian posture information.This paper focuses on pedestrian crossing behavior prediction based on surveillance video. A new spatio-temporal feature fusion network based on stacked GRU is proposed. The algorithm predicts the pedestrian crossing behavior by fusing the features of pedestrian posture, local context and global context. Quantitative and qualitative experiments are carried out using the pedestrian crossing behavior prediction dataset under surveillance video. The results show that our method has the best performance compared with other baseline methods. Then we counted the proportions of simple, medium, and difficult samples in pedestrian samples with different speeds. The results show that the slower the pedestrian movement, the more difficult the sample prediction. We also demonstrated the performance of each model at different prediction times. Experiments show that when the prediction time is short, the accuracy of each model is close. With the increase of prediction time, the performance of all models decreases. However, the performance gap between the models gradually widens with different decline speeds. The research of this paper proves the feasibility of pedestrian crossing behavior prediction based on surveillance video. It can provide a reference for the application of edge computing in the safety guarantee of automatic driving.