This paper proposes a novel hybrid optimization algorithm named GPOFWA, which integrates political optimizer (PO) with fireworks algorithm (FWA) to solve numerical and engineering optimization problems. The original PO uses subgroup optimal solutions such as party leaders and constituency winners to guide the movement of the search agent. However, the number of such subgroup optimal solutions is limited, which leads to insufficient global exploration capabilities of PO. In addition, the recent past-based position updating strategy (RPPUS) of PO lacks effective verification of the updated candidate solutions, which reduces the convergence speed of the algorithm. The proposed hybrid algorithm uses the spark explosion mechanism in FWA to perform explosion spark and Gauss explosion spark operations on the subgroup optimal solutions (party leader and constituency winner) respectively based on the greedy strategy, which optimizes the subgroup optimal solution and enhances the exploitative ability of the algorithm. Moreover, Gaussian explosion sparks are also used to correct the candidate solutions after RPPUS, which makes up for the shortcomings of the original PO. In addition, a new subgroup optimal solution called the Converged Mobile Center (CMC) based on two-way consideration is designed to guide the movement of search agents and maintain the population diversity. We test the presented hybrid algorithm on 30 well-known benchmark functions, CEC2019 benchmark functions and three engineering optimization problems. The experimental results show that GPOFWA is superior to many statE−of-thE−art methods in terms of the quality of the resulting solution. Optimization problems are inevitable in many real-world applications, and these problems usually contain non-linear objective functions and constraints with multiple local optimum, and low feasible regions2. These complex features make it difficult for traditional mathematical programming methods such as conjugate gradient, sequential quadratic programming, Newton's method, and quasi-Newton's method to find optimum3. Meta-heuristic algorithms (MAs) have become prevalent in many applied disciplines in recent decades because of higher performance and lower required computing capacity and time than deterministic algorithms in various optimization problems12. As a branch of random optimization, meta-heuristic algorithms can find a near-optimal solution by using available resources, although it is not always guaranteed to find the global optimum. Most MAs are inspired by human intelligence, the social nature of biological groups, and the laws of natural phenomena. Some classic representatives of MAs, such as genetic algorithm (GA)13, particle swarm optimization (PSO)14, differential evolution (DE)15, grey wolf optimizer (GWO)16, Harris hawks optimizer (HHO)17, bat algorithm (BA)18, whale optimization algorithm (WOA)19, salp swarm algorithm (SSA)20, sine cosine algorithm (SCA)21, water cycle algorithm (WCA)22, and so on, have been successfully used to solve some complex optimization problems.Optimization is a numerical process used to determine the decision variables for minimizing or maximizing the objective function value while satisfying the constraints of decision-space23, which means an algorithm is suitable for a given optimization problem, but may not be suitable for another optimization problem with different characteristics. Therefore, further research on MAs is needed to deal with different optimization problems. The research directions of MAs include proposing new algorithms, improving existing algorithms, and hybridizing different algorithms. Hybridizing different algorithms has drawn attention because it can highlight their respective advantages and make the algorithms have better performance. Various hybrid algorithms have achieved good results, such as hybridizing particle swarm optimization with differential evolution proposed by Wang et al.24, hybridizing sine–cosine algorithm with differential evolution proposed by Li et al.25, hybridizing particle swarm with grey wolf optimizer presented by Zhang et al.26. Fireworks algorithm (FWA) was a newly developed swarm intelligence optimization algorithm, which was put forward by simulating the process of real fireworks explosion and generating a large number of sparks in 201027. When the fireworks explode, the sparks are everywhere. The explosion process of the fireworks can be regarded as the search behavior of the search agent in the local space. The main idea of FWA is to use fireworks and sparks as different kinds of solutions to search the feasible space of the optimization function. As an excellent algorithm, FWA has been used in hybridization with many other algorithms in recent years. Zhu et al.28 hybridized the firework algorithm with the particle swarm algorithm to form DFWPSO, which performed competitively and effectively in numerical optimization problems. Yue et al.29 proposed a new hybrid algorithm called FWGWO based on gray wolf optimizer and firework algorithm and achieved excellent results in global optimization. Guo et al.30 added the differential evolution operator to the firework algorithm and proposed a hybrid fireworks algorithm with differential evolution operator (HFWA_DE) in 2019. Zhang et al.31 introduced the migration operator of biogeography-based optimization into fireworks algorithm to enhance information sharing among populations and presented a hybrid biogeography-based optimization and fireworks algorithm for global optimization.However, the No Free Lunch (NFL) theorem states that it is impossible to solve all optimization problems by a specific algorithm32. Compared with traditional optimization algorithms, PO shows better competitiveness. Therefore, lots of researchers have applied it in different scientific fields since the PO was proposed. Askari et al.33 employed PO for the training of feedforward neural networks to solve the classification and regression problems, and made a good achievement. Durmus et al. used PO to improve radiation properties of concentric circular antenna arrays (CCAAs) in the far-field such as wireless communication of smart grids and the Internet of things and reached a lower sidelobe level (SLL) value than other optimization methods34. Manita et al.35 proposed a binary version of PO to solve feature selection problems using gene expression data. Elsheikh et al.36 presented a novel optimized predictive model based on PO for eco-friendly MQL-turning of AISI 4340 alloy with nano-lubricants. Moreover, some scholars have made improvements to the shortcomings of PO. Askari et al.37 modified each stage of PO to improve the exploration ability and balance of the algorithm because it is found PO prematurely converges for complex problems. Zhu et al.38 also found that PO has the problem of poor global exploration capabilities, and they integrated PO with quadratic interpolation, advanced quadratic interpolation, cubic interpolation, Lagrange interpolation, Newton interpolation, and refraction learning, and proposed a sequence of novel PO variants.Political Optimizer (PO) is a new meta-heuristic algorithm based on human behavior inspired by the multi-stage political process. PO simulates all important steps in politics, such as party formation, party vote, constituency distribution, election campaigns, and party transitions, inter-party elections, and parliamentary affairs after the government is formed. In addition, PO has introduced a new position update strategy, called the recent past-based position updating strategy (RPPUS). The latter represents the behavior that politicians learned from the last electionWe propose a new hybrid optimization algorithm named GPOFWA, which integrates the Political Optimizer (PO) and the Fireworks Algorithm (FWA). Using the spark explosion mechanism in FWA, GPOFWA performs explosion spark and Gaussian explosion spark operations on party leaders and constituency winners based on greedy strategy, which enhances exploitation capability of GPOFWA. At the same time, the Gaussian explosion spark mechanism of the firework algorithm is used to explore areas with better fitness to ensure the effectiveness of RPPUS.We adopt a new method called Converged Mobility Center with bi-directional consideration to generate the subgroup optimal solution of the current population, which enhances the exploration ability and maintains the population diversity.We investigate the performance of the proposed algorithm in solving 30 basic benchmark functions in multiple dimensions (30 and 500), CEC2019 benchmark functions and three engineering optimization problems. To verify the feasibility and effectiveness of this scheme and the accuracy of the results from different aspects, we use experimental and statistical analysis, such as qualitative analysis, quantitative analysis, convergence preference, pairwise comparative analysis (Wilcoxon signed-rank test), computational complexity, and sensitivity analysis of parameters.As a novel swarm intelligence algorithm just proposed, PO still has many areas worth improving. It can be found that the main idea of PO is to guide the movement of the search agent through subgroup optimal solutions. However, the number of subgroup optimal solutions such as party leaders and constituency winners used by PO is limited, because the number of initial populations directly determines the number of party leaders and constituency winners. This leads to insufficient global exploration capabilities of PO. In addition, the recent past-based position updating strategy (RPPUS) of PO lacks effective verification of the updated candidate solutions, which reduces the convergence speed of the algorithm. Moreover, a new local leader called the Converged Mobile Center (CMC) based on two-way consideration is designed to guide the movement of search agents, which enhances the exploration ability and maintains the population diversity. Combining the above ideas, we propose a novel hybrid greedy political optimizer with fireworks algorithm named GPOFWA and verify its effectiveness and superiority through a well-studied set of diverse benchmark functions and three engineering optimization problems. In summary, the main contributions of this research are as follows:The remainder of this research is organized as follows: Section 2 reviews the basic political optimizer and fireworks algorithm. Section 3 proposes a novel hybrid greedy political optimizer with fireworks algorithm. Section 4 discusses the experiment results of different swarm intelligence optimization algorithms on basic benchmark functions and CEC2019 functions. Section 5 applies the algorithm to three different engineering optimization problems. Section 6 presents the conclusions of this work and directions for future work.Political optimizer and firework algorithm are novel algorithms with excellent performance proposed in recent years, which are inspired by different social natural phenomena and can effectively solve optimization problems. The hybrid algorithm proposed in this paper takes the political optimization algorithm as the starting point, and the explosion spark and Gaussian mutation spark mechanism of the firework algorithm are added to the search process of the political optimization algorithm to enhance the performance of the algorithm. This section will briefly introduce these two algorithms.The political optimizer (PO) is a novel intelligent optimization algorithm inspired by the political election process of human society. In PO, each party member can be viewed as a candidate solution, and the election behavior of party members can be seen as an evaluation function. In addition, the votes obtained by party members are mapped to the fitness value of the candidate solution. Unlike traditional algorithms based on political elections, PO considers the complete process of political elections, including five phases of party formation and constituency allocation, election campaign, party switching, inter-party election, and parliamentary affairs. PO seeks the optimal solution through a multi-stage iterative process, and its main algorithm flow is shown in Fig. n parties, and there are n members (candidate solution) in each party. In addition, each party member also plays the role of an election candidate, that is, one member from each party is selected to form a constituency. As is depicted in Fig. n political parties as shown in Eq. ..\documenThe party switching phase is mainly to balance exploration and exploitation, which introduces an adaptive parameter At this stage, the fitness of each candidate solution is determined and the party leaders and constituency winners are updated by Eqs.  and 11)11).10\doThe party switching phase is aimed at the change of the party's perspective, and the parliamentary affairs phase is the change of the constituency's perspective. The constituency winners interact with each other to improve their fitness. Each constituency winner uses the following equation to update its position relative to any other randomly selected constituency. It should be noted that the movement will only be applied if the fitness of The firework algorithm (FWA) is a swarm intelligence optimization algorithm proposed in recent years, which is inspired by the explosion of fireworks. We usually celebrate with fireworks. When the fireworks explode, the sparks are everywhere. The explosion process of the fireworks can be regarded as the search behavior of the search agent in the local space. The firework algorithm is based on this idea, and the flowchart of the firework algorithm is shown in Fig. It should be emphasized that fireworks of different qualities will produce different sparks when they explode. High-quality fireworks will produce countless sparks when they explode. The explosion of the fireworks forms a circle, and the sparks are concentrated in the center of the explosion. Conversely, a bad firework will produce fewer sparks when it explodes, and the sparks will spread out to form irregular shapes. From the perspective of swarm intelligence algorithm, a firework is regarded as a candidate solution. A good firework means that a candidate solution is located in a promising area and is close to the global optimal solution. Therefore, more sparks can be generated near good fireworks to find the global optimal solution, and the search radius is as small as possible. A bad firework means that the position of the candidate solution is not ideal, so the search radius should be larger, and the number of sparks generated will be reduced accordingly.As mentioned earlier, good fireworks should produce more sparks, while bad fireworks produce fewer sparks. The calculation of the number of sparks produced by each firework is shown in Eq. . Good fiWhat should be noted is FA design two ways of generating sparks, one is explosion sparks for normal search, its algorithm is shown in Algorithm 1. The other is Gaussian spark, which is a mutation mechanism, and its algorithm is shown in Algorithm 2.The main idea of PO is to guide the movement of the search agent through the subgroup optimal solution. The number of subgroup optimal solutions such as party leader and constituency winner are limited because the number of initial populations directly determines the number of party leaders and constituency winners, which leads to insufficient global exploration capabilities of PO.In RPPUS, member positions are updated based on the positions of members of the previous generation, the positions of party leaders or constituency winners, and the current positions of members. Considering the numerical relationship between these three indicators, it effectively predicts the favorable area of the member's next move, but this is the future movement trend predicted based on only three indicators, and its accuracy needs to be improved. Moreover, after the update is completed, it is not verified whether the fitness has improved.In the position update process, to consider the influence of the party leader and the constituency winner on the position of the members, the members are successively moved around the two subgroup optimal solutions. If the two subgroup optimal solutions themselves are relatively close, the difference between updating twice and updating once is not large, and updating twice also means that all dimensions of each member must be updated twice, which adds a lot of time consumption.The original PO assigns dual roles to each agent and uses RPPUS to make the algorithm have excellent performance, but through careful observation, we can find that the algorithm still has a lot of room for improvement. There are the following several points:The proposed algorithm puts forward corresponding solutions based on the above points, and finally forms GPOFWA. For the first point, using the spark explosion mechanism in FWA, GPOFWA performs explosion spark and Gauss explosion spark operations on party leader and constituency winner respectively based on greedy strategy, thereby optimizing the subgroup optimal solution. For the second point, GPOFWA uses the Gaussian explosion spark mechanism of the firework algorithm to explore areas with better adaptability to ensure the effectiveness of RPPUS. Regarding the third point, this article proposes a new subgroup optimal solution, called Converged Mobility Center (CMC) with bi-directional consideration, which not only considers the advantages of the party leader and the constituency winner but also maintains the population diversity.The most distinctive feature of FWA is that the firework explosion operator truly simulates the search process of the search agent. Generating a large number of sparks means that a large number of candidate solutions are generated. PO updates the position of search agent around subgroup optimal solutions, but the number of subgroup optimal solutions is limited by the size of the initial population. At the same time, the individuals performing the explosion operation in FWA are selected optimally from the entire population, and subgroup optimal solution of PO has been screened out, which can be used for the explosion operation. Moreover, the two explosion methods of FWA correspond to the two subgroup optimal solutions of PO, and they complement each other. Here, the party leaders conduct the explosion spark operation, and the constituency winners conduct the Gaussian spark operation. The detailed process of their explosive operation is shown in Fig. k is a parameter controlling the total number of sparks generated by party leaders or constituency winners, N party leaders, N constituency winners, and Similar to the FWA, the calculation of the number of sparks generated by subgroup optimal solution is shown in Eqs.  and 15)15). The R denotes the maximum explosion range, N party leaders.Since the party leaders conduct the explosion spark operation, it is necessary to calculate the explosion range. The calculation formula is shown as Eq. .16\documIt should be noted that after the party leaders and the constituency winners perform the explosion operation, based on the greedy strategy, they will update themselves if the sparks they generate have better fitness than themselves. This process is carried out after party formation and constituency allocation, whose pseudo-code is shown in Algorithm 3.As mentioned earlier, RPPUS only predicts the favorable area where the search agent moves and lacks correctness verification after the update. In some cases, the fitness of the candidate solution after the update is worse than the fitness before the update. As shown in Fig. In this paper, the Gaussian spark in the FWA is used to correct the candidate solution whose fitness becomes worse after the update. The specific method is to generate three sparks around the candidate solution and judge whether there is a better solution than the candidate solution before the update among the three sparks, if there is, choose the best spark as the new candidate solution. If the fitness of all sparks is worse than that of the candidate solution before the update, the candidate solution before the update will be inherited and no change will be made. It should be noted that the Gaussian spark here is slightly different from the original firework algorithm because we stipulate that the number of sparks generated by the “problematic” solution is three. The pseudo-code of this process is shown in Algorithm 4.In PO, the party leader and the constituency winner are successively regarded as the center on which the position of the member is moved. If the two centers themselves are relatively close, it is not necessary to update twice. In response to this situation, we propose a new method to generate a new subgroup optimal solution as a mobile center—Converged Mobility Center with Bi-directional Consideration (CMC), which not only uses the advantages of both the party leader and the constituency winner but also maintains the population diversity.kth dimension of the party leader kth dimension of the party leader In order to improve their performance in the election, candidates not only refer to the advantages of party leaders but also compare and analyze with the constituency winners. This action should be carried out at the same time, not one after the other. The higher the ranking of the party leader of the candidate’s party among all party leaders, the more the candidate wants to be close to the party leader. In the same way, the better the constituency winner of the candidate's constituency ranks among all constituency winners, the candidate will prefer the constituency winner. CMC is proposed based on this consideration. As shown in Fig. n in Eq. .17\documThe party weighting factor PF and the constituency weighting factor CF are calculated as follows:The time complexity of the population initialization phase is The fitness value of each candidate is evaluated initially, and the time complexity is The main loop of the algorithm is the main time consumption. The time complexity of the election campaign stage is Time complexity is a key criterion for judging the quality of an algorithm. To demonstrate the computational efficiency of GPOFWA, this section analyzes the computational complexity of PO and GPOFWA. The time complexity analysis of PO mainly includes three parts:In contrast, GPOFWA introduced the search strategy of the fireworks algorithm and adopted the Converged Mobility Center with bi-directional consideration. The time complexity of these two algorithms is different in the main loop. GPOFWA performs explosion spark and Gaussian explosion spark operations on party leaders and constituency winners to optimize subgroup optimal solutions. The time complexity of this process is We can conclude from the detailed analysis that they are of the same order of magnitude.The performance of GPOFWA is evaluated on 30 basic benchmark functions in multiple dimensions (30 and 500), CEC2019 benchmark functions and three engineering optimization problems against a good combination of some advanced swarm intelligence algorithms. These test cases include various types  of objective functions with the different number of decision variables and a range of types , and the number of constraints. All simulation experiments are conducted on a computer with a Win10 operating system and Intel(R) Core (TM) i7-10750H GHz with 16 GB RAM. The proposed algorithm is coded in MATLAB R2020a.To verify the good performance of GPOFWA, we first used thirty benchmark functions for testing which are equally divided into two groups: unimodal function and multimodal function. The unimodal function (F1–F15) with the unique global optimal solution can reveal the exploitative capabilities of different algorithms, while the multimodal function (F16–F30) can be used to test the ability of the algorithm to avoid falling into the local optimal solution. It should be noted that the multimodal function test set also contains some fixed-dimensional functions, which show some optimization problems in the real world.The detailed information of the unimodal function is shown in Table First of all, we tested the performance of all selected algorithms on F1–F15. And used three different statistics to start the first step of the evaluation. These statistics are the best fitness value (Best), the average fitness value (Mean), and the standard deviation (Std). Table By testing the unimodal function F1–F15, we can find the powerful exploitative capability of GPOFWA. To evaluate the exploration capability of GPOFWA, we used the multimodal function set F16–F30 for testing. As with the unimodal function test, we also use the best fitness value (Best), the average fitness value (Mean), and the standard deviation (Std) three statistics to illustrate the experimental results. The experimental results are shown in Table To test the performance of the GPOFWA algorithm on high-dimensional problems, we tested unimodal and multimodal functions of 500 dimensions. It should be noted that the test function used in 4.1 contains some fixed dimension functions, so we chose F1–F10, F16–F25 for testing. For each function, the parameters are the same as those mentioned above. Figure 39, PPSO40, CLPPSO40, HHOHGSO41, DE15 and CMA-ES42. The algorithms used for comparison and their parameter settings are based on the parameters used by the original author or the parameters widely used by various researchers. To ensure the fairness of the experiment, we compare the performance of the algorithms after running each experiment independently 30 times. Figure By testing 30 classic benchmark functions in low and high dimensions, we can already find the excellent performance of GPOFWA. To further explore the effectiveness of the proposed method, we also use the CEC2019 benchmark function for testing. The CEC2019 benchmark function contains a number of shifted rotated functions to test the stability of the algorithm against function shifts. It is worth mentioning that the comparison algorithms we use in this section are some advanced and hybrid algorithms, not the basic algorithm used above. These algorithms are FWHHOTo evaluate the proposed algorithm fairly and accurately, we perform statistical tests on the experimental results. To better determine whether the optimization results of GPOFWA were significantly different from those of other algorithms, a Wilcoxon nonparametric test was performed at a significance level of 0.05. A significance level 32. CPOFWA adds many mechanisms on the basis of PO to enhance the performance of the algorithm. First, GPOFWA performs explosion spark and Gaussian explosion spark operations on party leaders and constituency winners based on greedy strategy, and the Gaussian explosion spark mechanism of the firework algorithm is used to explore areas with better fitness to ensure the effectiveness of RPPUS. The greedy strategy enhances exploitation capability of GPOFWA, and Gaussian spark for verification of RPPUS prevents excluding good solutions. In addition, Converged Mobility Center with bi-directional consideration enhances the exploitation ability and maintains the population diversity, avoiding local optima. We can also analyze the convergence of GPOFWA by observing the convergence curves of numerous test functions. It can be observed that GPOFWA has a faster convergence rate to produce accurate solutions in most cases compared to the comparison algorithms.In original PO, the balance between the exploration and exploitation is attained through party switching, which uses a parameter λ to control the diversity, and the interaction between the constituency winners in the phase of parliamentary affairs ensures the convergence of POThe GPOFWA mainly includes 4 parameters, which are the parameter In this section, we apply GPOFWA to three well- known constrained engineering problems: welded beam design problem, spring design problem and three bar truss problem to demonstrate its performance in solving practical problems. For the fairness and rationality of the experiment, each experiment is independently run 30 times and the number of iterations is 500. These engineering problems are abstracted from various scenes in the real world, which are composed of an objective function and multiple constraints. Therefore, we need a suitable method to deal with these constraint conditions in these engineering problems. In this section, we employ the penalty function method. In this approach, solutions which violate any of the constraints are penalized by a large fitness value (in case of minimization). The penalty function is defined as follows:h), length (l), weld thickness (t) and thickness (b). Its constraints consist of shear , coil (D) and the number of the active coil (N). The Spring design problem is mathematically formulated as follows:This constrained engineering problem is to design a tension/compression spring with minimum weight, the structure of which is shown in Fig. Decision variable interval values:We also compare the statistical results of 30 independent executions of GPOFWA with some other excellent algorithms, and show the values of the design variables obtained, the mean, best value and variance of the optimal solution in Table The threE−bar truss design problem is a classic design problem in the field of engineering structure. The optimization goal of this design problem is to design a truss as light as possible, which must meet the three constraints of stress, deflection and buckling. This problem aims to minimize the volume of the truss structure subject to 3 stress constraints. The structural model and parameters of the threE−bar truss design problem are shown in the Fig. Decision variable interval values:We compare the statistical results of 30 independent executions of GPOFWA with other excellent algorithms, and show the values of the design variables obtained, the mean, best value and variance of the optimal solution in Table As an emerging swarm intelligence algorithm, PO has good exploration capability, exploration capability, and convergence speed, but the subgroup optimal solution used by the original PO is limited, and PO’s recent past-based position updating strategy (RPPUS) has loopholes. The explosion search mechanism of the firework algorithm has certain potential and unique advantages. In this paper, the explosion search mechanism of the firework algorithm is used to expand and optimize the subgroup optimal solution in the political optimization algorithm. At the same time, the Gaussian explosion spark of the firework algorithm is used to make up for some of the shortcomings of RPPUS. In addition, a new local leader called Converged Mobile Center (CMC) based on two-way consideration was designed to guide the movement of search agents.Based on these, a hybrid algorithm called GPOFWA is obtained. In order to verify the good performance of GPOFWA, we conducted a two-part experiment. In the first part, we selected a set of well-researched different benchmark functions and compared them with new swarm intelligence optimization algorithms including the original HHO, GWO, SCA, SSA, WCA, WOA, LSA, PO. Compared with PO, this algorithm has significantly improved accuracy, convergence curve, stability, and robustness when solving functions that are unimodal or multimodal. Compared with other methods, GPOFWA also shows significant advantages. In the second part, we apply GPOFWA to three constrained engineering problems, because of the improvement of the explosion search mechanism, GPOFWA can achieve the best results in all engineering design problems. The results show that GPOFWA has excellent performance for engineering design problems, and it is believed that GPOFWA can expect the same performance for other more complex engineering problems.In addition to the qualities mentioned above, PO has some limitations that need to be highlighted. The limitations of GPOFWA are as follows: Due to the addition of the explosive search mechanism, the algorithm time overhead has increased, although CMC has reduced this newly added time overhead as much as possible. Second, the algorithm has a total of 4 parameters, which is relatively complex and needs to be improved in the future. In future work, the GPOFWA algorithm can also consider a binary version to solve discrete practical problems, such as antenna design, feature selection, etc. At the same time, we can also combine CMC with other swarm optimization algorithms to further test its performance.


In solid tumors, elevated fluid pressure and inadequate blood perfusion resulting from unbalanced angiogenesis are the prominent reasons for the ineffective drug delivery inside tumors. To normalize the heterogeneous and tortuous tumor vessel structure, antiangiogenic treatment is an effective approach. Additionally, the combined therapy of antiangiogenic agents and chemotherapy drugs has shown promising effects on enhanced drug delivery. However, the need to find the appropriate scheduling and dosages of the combination therapy is one of the main problems in anticancer therapy. Our study aims to generate a realistic response to the treatment schedule, making it possible for future works to use these patient-specific responses to decide on the optimal starting time and dosages of cytotoxic drug treatment. Our dataset is based on our previous in-silico model with a framework for the tumor microenvironment, consisting of a tumor layer, vasculature network, interstitial fluid pressure, and drug diffusion maps. In this regard, the chemotherapy response prediction problem is discussed in the study, putting forth a proof of concept for deep learning models to capture the tumor growth and drug response behaviors simultaneously. The proposed model utilizes multiple convolutional neural network submodels to predict future tumor microenvironment maps considering the effects of ongoing treatment. Since the model has the task of predicting future tumor microenvironment maps, we use two image quality evaluation metrics, which are structural similarity and peak signal-to-noise ratio, to evaluate model performance. We track tumor cell density values of ground truth and predicted tumor microenvironments. The model predicts tumor microenvironment maps seven days ahead with the average structural similarity score of 0.973 and the average peak signal ratio of 35.41 in the test set. It also predicts tumor cell density at the end day of 7 with the mean absolute percentage error of  Due to tumor-induced angiogenesis, the newly developed vessels have a leaky and disorganized structure accompanying a microenvironment identified by hypoxia, acidosis, and increased fluid pressure2. As a result, this structurally and functionally abnormal tumor vascular network leads to heterogeneous and inadequate drug distribution inside tumors.Tumors need their blood supplies to grow beyond the size of 1–2 4. IFP is uniform across the tumor almost equal to microvascular pressure (MVP), but it drops precipitously at the tumor boundary. Therefore, the absence of a pressure gradient along the vessels hampers the penetration of cytotoxic drugs to the interior parts of tumors transported by convection3.The chaotic architecture, high vascular permeability of tumor vessels, and lack of functional lymphatics lead to elevated interstitial fluid pressure (IFP), creating a barrier for the transport of therapeutic agents and nanoparticles5. Vascular normalization improves convective transport of drug particles with a decrease in IFP, thereby inducing pressure gradients across vessel walls6.Antiangiogenic agents are mainly utilized to prevent tumor growth in size or metastases to another organ by depriving the tumor of the blood supplies it needs. Normalization of tumor vasculature by using antiangiogenic agents is a widely used treatment modality in cancer therapy. Antiangiogenic agents maintain the balance between proangiogenic and antiangiogenic factors, similar to healthy tissues. Furthermore, these agents normalize the structure and function of the tumor vascular network transiently by inducing reduced vessel diameter and decreased vessel wall permeability. This process triggers tumor vessels to be more useful for the delivery of drugs as well as oxygen and nutrients to the targeted cancer cells7; also, several preclinical and clinical studies show that the application of antiangiogenic agents together with chemotherapy drugs provides beneficial results with increased therapeutic outcomes10. The combination therapy can enhance the delivery of therapeutic agents to the interior parts of tumors6. Normalization is a transient process meaning there is a time window for vessel normalization to occur. For this reason, chemotherapy drugs should be administered carefully within this window to benefit from improved vascular conditions11. Moreover, the excessive application of antiangiogenic agents for more extended periods brings about vessel pruning, which decreases the outcome of combined therapies13. Therefore, the appropriate timing and dosing of the agents should be carefully adjusted to improve the functionality of blood vessels, as well as the delivery of anticancer drugs to tumor cells.Antiangiogenic agents can act directly on the tumor vasculature17. In addition, in 2D and 3D blood flow models, the use of antiangiogenic agents and their effects on tumor response has been studied by Stephanou et al.18. Normalization is also simulated to investigate its effects on blood flow and the combination of antiangiogenic agents with cytotoxic drugs21. In addition to mathematical models, angiogenesis imaging is largely studied to examine tumor growth, characterization of tumor vasculature, and the response of the therapies. Employing various imaging modalities such as computer tomography (CT) and magnetic resonance imaging (MRI) gives an insight into the progression of tumors and vascular structures23. Additionally, photoacoustic imaging is exploited to monitor tumor angiogenesis due to its higher contrast and spatial resolution26. Besides, it is feasible to detect early tumor growth and vascularization and to monitor the progress of antiangiogenic treatments28.In cancer therapies, mathematical models are commonly studied to simulate the delivery of cytotoxic drugs and to understand the relations between tumor microenvironment and drug delivery. Different approaches to the modeling of tumor vasculature and angiogenesis have been suggested to study the applications of antiangiogenic agents combined with chemotherapy drugs. By using discrete vasculature models, the delivery of chemotherapy drugs to tumors and treatment response have been extensively investigated31. These models are capable of modeling the tumor response to an extent, but they are generally deterministic models driven by a limited number of parameters that might constrain the model to comprise the inherent tumor growth patterns. However, several studies in deep learning outperform the traditional approaches in various tasks such as tissue classification, tumor segmentation, and tumor growth prediction35. A convolutional neural network (CNN) is developed by Urban et al. to classify the images of vascular networks taken before and after various antiangiogenic drug applications in vitro36. In another study led by Ha et al., a CNN algorithm is used to determine the chemotherapy response prediction in patients with breast cancer. The breast MRI dataset is utilized as a baseline, and the treatment response before the application of chemotherapy drugs is predicted37. Positron emission tomography (PET) and CT images from different types of cancer are benefited in recent studies to develop CNN models that have the potential to predict the response of chemotherapy with high sensitivity39.Early tumor response prediction, a measure of the effectiveness of treatment, is crucial in anticancer therapies to determine appropriate treatment schedules, apply the optimal drug dosages to patients, and increase patient survival. In this regard, to develop successful models for tumor response prediction, it is vital to monitor the distribution of cytotoxic drugs and to evaluate the tumor response to therapies in advance. Response prediction problems are often formulated by developing mathematical models benefiting from reaction–diffusion frameworks40. For instance, a feed-forward neural network is a function approximation that maps input data to output data. The function is formed by composing simpler non-linear functions where each function provides a new representation of input data41. After extracting the information out of the input data, the model amplifies the essential features for a given task such as classification, segmentation, and regression. Convolutional neural networks, a neural network type, perform convolutions over the input by using the given number of filters, which can learn the spatial and temporal dynamics in the image data40. They are robust to overfitting due to the convolution operation properties, which reduces the full connection of the network. These models have applications in image42 and video recognition43, medical image analysis, and image processing44. By employing CNNs, models can be built for spatial and temporal forecasting problems in an end-to-end manner in the presence of sufficient data.Deep learning allows us to build computational models to learn representations of data with multiple levels of abstraction for the given taskF, takes tumor microenvironment maps t and predicts future tumor microenvironment maps Tumor response prediction can be formulated as a spatio-temporal forecasting problem under the effects of interventions. Interventions can be described as drug injections. Therefore, the forecasting problem is equivalent to modeling the tumor microenvironment during the ongoing treatment. The tumor microenvironment can be described with tumor density, vasculature, interstitial fluid pressure (IFP), antiangiogenic treatment, chemotherapy maps, and drug dosages from the ongoing treatment. The model, denoted by 21. Therefore, the proposed deep learning model F encapsulates the non-linear partial differential equations (PDEs) that govern the spatio-temporal dynamics of the tumor microenvironment. The model consists of multiple CNNs. Our motivation to use CNNs is based upon two key reasons; CNNs can extract coupled spatial features from multichannel inputs and utilize the spatial features to predict future microenvironment maps. In the end, our model uses tumor microenvironment channels and drug scalars as inputs and predicts future tumor microenvironment maps that may assist to find appropriate scheduling and dosages of the combination therapy for patients.In this study, we develop a deep learning model, which aims to capture the tumor response behavior conditioned by the ongoing treatment schedule. Our study suggests that deep learning models can be helpful in assessing tumor growth and drug response in the scheduling of cytotoxic drugs. Since the required input data such as tumor density, IFP, vasculature, and drug maps is hard to collect from clinical patients for various reasons, we use the synthetic data from the mathematical model built in our previous paperOur deep learning model aims to predict chemotherapy response by outputting future tumor microenvironment maps. The proposed model, Diffuser-Elapser Network (DENT), consists of seven submodels where each of which is a CNN. Model specifications and training procedures can be found in the Methods section.45. PSNR represents a measure of the quality of prediction by calculating the ratio between the square of maximum fluctuation among pixels in the ground truth image and MSE between ground truth and predicted images. SSIM measures the perceptual difference between two images in terms of structural information in the images. In this method, ground truth images are considered as references that have reference quality, and the quality of another image is measured by comparing it with the initial one. SSIM is mostly applied to improve or track the perceptual metrics based on the structural information; on the other hand, PSNR relies on estimating the pixel-wise error. Both metrics are unitless quantities. The PSNR score approaches infinity as the MSE approaches zero. The range for an SSIM score is the interval of The DENT model takes input tensors describing the current tumor microenvironment state, consisting of five channels, namely tumor density, vasculature, IFP, antiangiogenic drug, and chemotherapy drug maps accompanied by chemotherapy and antiangiogenic drug dosages, and predicts future tumor microenvironment maps. The model can use its predictions as its next inputs. Since the model predicts future tumor microenvironment maps, we use two image quality evaluation metrics, which are peak signal-to-noise ratio (PSNR) and structural similarity (SSIM)21, which were limited to simulating 8 days starting from the initiation of therapy. The patient receives treatment between day 21 and day 28. The application of the antiangiogenic drug is at the end of days 21, 23, 25, and 27, and the chemotherapy drug is at the end of days 23, 25, and 27. Our model takes the tumor microenvironment maps just before the initiation of therapy as input and predicts the future tumor microenvironment maps by taking its outputs as new inputs. The predicted tumor microenvironment maps show the effects of ongoing treatment on the tumor microenvironment. A qualitative assessment of Fig. In the study, we first investigate the performance of the model on predicting future tumor microenvironment maps. To give an insight, we pick a synthetically generated case that contains a patient’s tumor microenvironment maps during treatment. The tumor microenvironment maps consist of five channels correspond to tumor cell density, vasculature, IFP, antiangiogenic drug diffusion, and chemotherapy drug diffusion maps. The input, ground truth, and predicted tumor microenvironment maps during the treatment are presented in Fig. 21. The consecutive frames correspond to time steps from 0 to 7 and are separated by a day. The five channels represent tumor cell density, vasculature, IFP, antiangiogenic drug diffusion, and chemotherapy drug diffusion maps. Drug scalars indicate the dosage for drug insertion in each time step if there is. The insertion schedule is the same for all cases; the antiangiogenic drug is inserted at the end of time steps 0, 2, 4, and 6, and the chemotherapy drug is inserted at the end of time steps 2, 4, and 6.In the experiments, we use 135 test cases from three different patients. The cases included five different therapy initiation days and nine different dosage combinations of antiangiogenic and chemotherapy drugs. Therefore, the cases differ according to the days of initiation of treatment, applied drug dosages and patient characteristics. A case corresponds to a tensor of 8 days-long tumor microenvironment maps accompanied by synthetically generated drug scalars using a mathematical modelBy using completely new cases to test our model, we assure that the model has not see the test data before. Each forward step of the DENT model is trained to predict tumor maps in the next time step, which is a day ahead. The average PSNR and SSIM scores are shown in Fig. C. Tumor cell density over the whole tumor shows the effectiveness of ongoing treatment. It is calculated by averaging the pixel values after thresholding the pixels of the tumor density map as performed in the previous study21. We compare the tumor cell density of ground truths, We next examine if the model encapsulates the tumor growth and shrink patterns during the treatment by calculating a dimensionless metric called tumor cell density, 21. The parameters used in this mathematical model are calibrated and the simulation results are validated by experimental studies in the literature. In order to adjust the parameters related to the kinetics of tumor cell growth and vascularization, the experiments of Winkler et al.8 studying glioblastoma xenografts that grow in the mouse brain are utilized. The model has a dynamic vascular structure that shows the properties of tumor vessels and inherent vessels in the tissue. The resulting vessel density is compatible with the study of46, with decreasing vascular density towards the tumor center due to increased tumor cell density in that region. Moreover, it is found that IFP increases up to the MVP levels throughout the tumor and it drops sharply around the tumor rim very similar to various experimental studies (47). In the model, antiangiogenic agents are administered in various treatment regimens in combination with large-sized chemotherapy drugs, the distribution of which is mainly dependent on convection. Drug accumulation is mostly observed in the tumor periphery in the simulations, as there are low IFP levels in this region. The results are validated by various experimental studies examining the drug distribution of large drugs, showing that large drug accumulation is mainly observed in peripheral areas51. Here, the proposed deep learning model can predict chemotherapy response to tumor structure over a flexible period via iterating for a chosen number of times. Our model operates on a total of five channels: the maps of tumor cell density, vasculature, interstitial fluid pressure, and antiangiogenic and chemotherapy drug diffusions. The predicted maps show the spatial and temporal effects of ongoing treatment on tumor microenvironment maps. The model can take its outputs as inputs, go further in time, and predict the following state of the tumor by iterating this process as many times as required. In these iterations, there is an option of drug injection fed into the model as an input, if it is in the treatment schedule. To include the injection step in the model, the drug diffusion maps on the tumor structure are predicted, then these maps are used for the next tumor state prediction in combination with the other inputs such as tumor cell density, vasculature, and interstitial fluid pressure maps. To evaluate the predictive capabilities of our model, we track PSNR and SSIM scores calculated between the ground truth and predicted tumor microenvironment maps during the treatment. In a study by Teuwen et al.52, a deep learning model is utilized to synthesize digital breast tomosynthesis, which can be used to detect early breast cancers. When the model uses breast CT images, it has an SSIM score of 0.93. Chaudhari et al.53 use 3D convolutional neural networks to perform the resolution-enhancement of MRI scans. The outputs of the proposed model are compared with the other models using SSIM and PSNR metrics. A study by Zeng et al.54 utilizes CNNs to reconstruct super-resolution single and multi-contrast brain MRI scans. Their proposed model for multi-contrast super-resolution cases has the highest average PSNR and SSIM scores relative to the other methods. Uzunova et al.55 use patch-based generative adversarial networks to generate high-resolution 2D and 3D medical images. Their model achieves SSIM scores of 0.733 and 0.711 for thorax CT and X-ray datasets, respectively. Our model can predict future tumor microenvironment maps up to 7 days with the average SSIM score of 0.973 and the average PSNR score of 35.41 at the end of day 7. We have assessed the tumor cell density among the entire predicted tumor density 21, merely averaging the pixel values that are above a threshold value. We use the percentage error between In this study, we build a deep learning model designed to solve the chemotherapy response prediction problem in a proof of concept setting. The model is trained with simulation data that reflects the biological aspects of the tumor microenvironment, including tumor growth and angiogenesis56 focuses on the prediction of lung cancer treatment response for patients treated with chemoradiation. By analyzing time-series CT images of patients, the survival probability and cancer-specific outcomes have been predicted. In another study by Ha et al.37, it is aimed to predict the effectiveness of neoadjuvant chemotherapy (NAC) response by using a breast MRI tumor dataset. Patients are classified into three groups as complete, partial, and no response, based on their response to NAC treatment. Although these classification-based models predict chemotherapy response to some extent, their predictions lack the spatial and temporal information that provides the necessary explanations behind the predictions. For instance, providing maps of future tumor microenvironment states after the treatment can give clinicians an insight to reschedule the treatment planning process. Thus, there is a challenge in building models that provide detailed information in the chemotherapy response prediction task, which leads us to the modeling of the tumor microenvironment.Chemotherapy response prediction aims to successfully predict whether the patient will respond to treatment, depending on the tumor characteristics of the patient. Current approaches to chemotherapy response prediction are mainly based on classification models. A recent study21 presents a mathematical model to simulate the tumor state according to the previous tumor state and the drug diffusion inside the tumor. In each iteration of the model, a predetermined theoretically thirty-minute gap period proceeds and the subsequent tumor state is calculated. This allows the model to output the tumor states with a time resolution of thirty minutes, which is suitable for comprehensively studying tumor shrinkage or growth under the effects of ongoing treatment. The model is deterministic, which means differential equations, predetermined biophysical factors, and constraints are utilized. In potential clinical use, this may prevent the model from making correct interpolations since these factors and constraints may vary from patient to patient as opposed to the fixed values used in the simulation. Capturing these factors and constraints from patient data is essential to predict patient-specific chemotherapy responses. The calculations in the simulation are at the pixel level, and the model cannot capture the detailed physical structure of the tumor and recognize any special conditions, such as detecting the tumor type or distinguishing smaller parts of the tumor. Since the coefficients used in the differential equations are constant and non-learnable, a deterministic model cannot learn from clinical data and be precise enough for clinical use, making it impossible to generalize over thousands of patients.On the other hand, mathematical models rely on nonlinear PDEs that simulate the dynamics of the tumor microenvironment while considering the effects of treatments. These models enable us to observe the diffusion of the chemotherapy drug to tumor cells via the normalized vasculature network, and the effectiveness of the applied treatment by producing maps of the future tumor microenvironment states. This approach may eliminate the main drawback of the class-based chemotherapy response prediction models, which is the lack of insight behind the predictions. A recent studyIn the wake of these limitations in the literature, we propose a deep convolutional model. The model simulates tumor microenvironment maps that encapsulate spatio-temporal effects of the ongoing treatment such as, tumor growth and shrinkage patterns, along with drug diffusion maps. The model outputs future tumor microenvironment maps that fill the descriptive information gap in classification-based response prediction models. The sequence of the predicted maps indicates the response to the therapy in the daily regimen. The model predictions reveal insights about the effectiveness of the therapy schedule and drug dosages on patients before the application of the therapy. This might allow for better adjustment of schedules, use of the potent schedules, and elimination of the ineffective ones. Although mathematical models can simulate tumor microenvironment, they tend to neglect patient-specific tumor response patterns due to a limited number of parameters, constant coefficients, and deterministic rules governing the model. Since the proposed model is built with end-to-end learning, it can learn patient-specific tumor response patterns in the presence of sufficient data. The model may improve the personalization of treatment by extracting specific clinical features of a patient and making predictions based on them.57 or 3D scanning methods such as CT are the primary candidates of data sources to solve the incomplete environment representation problem. In addition to the third spatial dimension, tumors also have a continuous structure over the time dimension, that is, a continuous course of growth and shrinkage. It may be necessary to model the time dependencies between the sequential states of the tumor, assuming that the pattern of future tumor growth or shrinkage follows its past trend under constant conditions. The input should consist of temporal information of the tumor microenvironment maps to encapsulate the effects of time dependencies on a tumor microenvironment. The model can utilize the input sequence to reveal the features of patient-specific tumor dynamics. These features can be critical to include patient-specific tumor growth patterns and tumor aggressiveness in model predictions. Although our CNN-based model captures spatio-temporal dependencies to some extent, it is preferable to use long short-term memory (LSTM)58 networks with sequential inputs combined with CNNs. Convolutional LSTMs59 can be utilized to extract both spatial and temporal dynamic changes in a single network as they are already exploited in the tumor growth prediction problem60. The main challenge of this approach would be to acquire regularly timed and spatially aligned scans to generate sequential data.Our model for the chemotherapy response prediction can provide insights into the treatment outcomes in clinical settings; however, some limitations should be eliminated with further study when it comes to clinical use. A limitation of the proposed model is to use 2D tumor microenvironment maps rather than 3D maps, which can better exploit spatial information of the tumor microenvironment. When the tumor microenvironment maps miss the third spatial dimension, the true nature of the tumor cannot be reflected in detail, leading to an incomplete representation of the tumor microenvironment within the model. Therefore, 3D mathematical models of tumorsThe simulated cases used in our study differ by therapy initiation day, drug combinations used in the therapy, and unique patients imposed by the randomly initialized vasculature maps. However, all simulated cases share the same insertion schedule in which antiangiogenic drug therapy is initiated at the beginning of the therapy, and chemotherapy is started with a two-day delay. Both drugs are inserted every other day. Since our cases lack the variations in the insertion schedules, the model may be limited in encapsulating the effects of longer or shorter time intervals between the initiation of antiangiogenic and chemotherapy treatments, or non-overlapping insertion days of the two types of drugs. A possible solution is incorporating various realistic insertion scheduling scenarios in the cases.61, IFP62 and cytotoxic drug distributions64. Various scanning techniques should be combined to feed the model’s input to accurately represent the tumor, thus introducing as many aspects of the tumor as possible. This variety of scan types would provide a complete representation of the tumor within the model, boosting prediction and simulation capabilities.Obtaining the training data from simulations is beneficial in a proof of concept study; however, the use of clinical training data is inevitable for a distinguished and genuinely working model. To ensure that a deep learning model can simulate and generalize the chemotherapy response in a real scenario, it is necessary to train a model with a reasonably large amount of clinical data with sufficient variations of treatment schedules. A crucial feature of a clinically applicable response prediction model is that it can make predictions in the same domain with clinical data such as MRI, CT, and PET scans. Each scanning technique captures a unique kind of information about the tumor microenvironment, highlighting a particular aspect of the biological structure. By using these scanning techniques, it is possible to obtain the maps of tumor cells, tumor vasculature21 is used to obtain the training and testing data sets. Non-linear PDEs are written in dimensionless form and utilized in the model to mimic the tumor and its microenvironment biologically.In this study, a mathematical model of tumor built in our previous studyx,t) and m, respectively. In Eq. (The model incorporates tumor cell density and vasculature as well as their interplay, and consists of tumor cell density, vasculature, IFP, antiangiogenic agent and chemotherapy drug. The reaction–diffusion equations are used to describe the spatio-temporal distribution of tumor cell density and vasculature see Eqs.  and 2. T. In Eq. , the firmentclass2pt{minimP is the interstitial fluid pressure, To create vessel islands, a course-grained model is utilized in the model. The term entclass1pt{minimaentclass1pt{minima the Eq.  is used:The transport of the antiangiogenic agents is represented by the following diffusion equation . Here, t the Eq. , the fir21 which mathematically models the tumor microenvironment, as described in the previous section. The simulation results are consistent with experimental studies, allowing them to be used for data generation65. Two datasets are used in the training of the proposed deep learning model DENT. In the training and validation sets, cases from five unique patients are used. A case corresponds to tumor microenvironment maps during eight days of combination therapy. The cases contain tumor microenvironment maps with a time resolution of 30 minutes and have five different therapy initiation days in the range of day 14 to day 21. The combinations of three different antiangiogenic and chemotherapy drug dosages are used in treatments. The insertion schedule is the same in all treatments. The antiangiogenic drug is inserted at the end of time steps The datasets used in this study are synthetically generated with a MATLAB simulationThe first dataset, employed by Diffuser, contains drug map changes in the presence of drug insertion, a combination of antiangiogenic and chemotherapy drugs Fig. . This daThe second dataset, used by Elapser, contains the changes in the tumor microenvironment in the absence of drug insertion between the input and target tensors Fig. . The datWe aim to eliminate any numerical instabilities between the pixel values of the input channels and the drug scalars. All input images and scalars in the dataset are scaled with min–max scaling using the following function:t and x is arbitrary and indicates the number of prediction steps.The proposed model takes the tumor microenvironment maps The DENT model has the task of predicting future tumor microenvironment maps considering the effects of ongoing treatment. The model consists of multiple CNN submodels to extract spatial features from multichannel tumor microenvironment inputs. The model utilizes these features to predict future tumor microenvironment maps, which is equivalent to learning the spatio-temporal dynamics of the tumor microenvironment. The model has two main tasks; the first task is to insert and diffuse the administered drug dosages in the given tumor microenvironment, and the second task is to predict future tumor microenvironment maps. Since these tasks are not overlapping, we build two submodels called Diffuser and Elapser networks that work independently of each other. These two models are integrated to obtain the DENT network which can iterate by taking its outputs as inputs. It can check if there is any injection and interfere with the iteration process in case of necessity.The first part of the proposed model, the Diffuser network, contains two independent CNN submodels that insert and diffuse the scalar drug dosages given together with the input tumor microenvironment maps, generating new drug diffusion maps. It is a conditional network that is initiated if scalars are different from zero. In the case of zero drug dosage, the Diffuser network outputs the relevant drug map without any changes. The second part, the Elapser model, includes five independent CNN submodels that take tumor microenvironment maps as input tensor and predict future tumor microenvironment maps in a channel-wise manner. The Elapser model encapsulates the changes in each channel so that it can extrapolate to the future.By creating a pipeline with the Diffuser and Elapser networks as shown in Fig. As previously mentioned, the proposed pipeline consists of two submodels. The first submodel, Diffuser network, injects and diffuses the scalar drug dosage given the input tensor, generating new antiangiogenic and chemotherapy drug diffusions maps. An input tensor for the Diffuser network contains tumor microenvironment maps accompanied by two dosage scalars, as shown in Fig. 66 optimizer with a batch size of 10 and initial learning rate of 67 backend and trained on a single NVIDIA Tesla K80 GPU.All CNN submodels are trained separately as their tasks differ from each other. In the training, we use 225 synthetically generated cases from five unique patients. We perform a 5-fold cross-validation among the five patients. Therefore, there is not any spatial or temporal dependency between the training and validation sets. The submodels are trained with MSE loss for 200 epochs with early-stopping to avoid over-fitting. The activation function is ReLu and kernel size fixed as Supplementary Information.


Monitoring and treatment of severely ill COVID-19 patients in the ICU poses many challenges. The effort to understand the pathophysiology and progress of the disease requires high-quality annotated multi-parameter databases. We present CoCross, a platform that enables the monitoring and fusion of clinical information from in-ICU COVID-19 patients into an annotated database. CoCross consists of three components: (1) The CoCross4Pros native android application, a modular application, managing the interaction with portable medical devices, (2) the cloud-based data management services built-upon HL7 FHIR and ontologies, (3) the web-based application for intensivists, providing real-time review and analytics of the acquired measurements and auscultations. The platform has been successfully deployed since June 2020 in two ICUs in Greece resulting in a dynamic unified annotated database integrating clinical information with chest sounds and diagnostic imaging. Until today multisource data from 176 ICU patients were acquired and imported in the CoCross database, corresponding to a five-day average monitoring period including a dataset with 3477 distinct auscultations. The platform is well accepted and positively rated by the users regarding the overall experience. In the media briefing on March 11, 2020, the World Health Organization (WHO) declared a state of a pandemic for COVID-19. Patients with a SARS-CoV-2 infection can develop coronavirus disease 2019 (COVID-19) . COVID-1One of the biggest issues in treating COVID-19 positive subjects is how to achieve early detection of the symptoms that indicate deterioration of the disease status and requirement for hospitalization and/or early treatment interventions . Such syIt is evident that there is a need for an easily accessible and simple-to-use set of devices that could provide this assessment in an effective and continuous manner. This applies to both people at the community level with the need for remote monitoring  and healthcare professionals dealing with the COVID-19 pandemic in the ICU or other clinical settings. Additionally, databases containing auscultation data from ICUs from patients with COVID-19 can help better understand the lung and heart pathophysiology changes due to the disease . The intTherefore, there is a need to implement an alternative method of bedside monitoring to patients with COVID-19 who are treated in the ICU, ideally combining different diagnostic techniques and transmitting the recordings wirelessly as fast as possible for further assessment by the ICU team of physicians. There is also a certain need for efficient and reliable bedside diagnostic and imaging modalities that also give insight to respiratory dynamics and cardiorespiratory interactions, to enable intensivists to evaluate the effect of the ventilator settings on the cardiorespiratory dynamics, regardless of the underlying cause for the patient’s critical condition and the need for mechanical ventilation .The CoCross Platform is a complete system for monitoring the evolution of lung and heart sounds and diagnostic imaging of patients with COVID-19 who are treated in the ICU, enhancing collaboration of experts, and overcoming the difficulties that are imposed by the a) use of protective equipment and b) restrictions on hardware transfer in-and-out of the ICU due to COVID-19. The CoCross Platform consists of the following components, (1) the CoCross services back-end, hosting, the authorization and authentication service, the data management service, and the web application for intensivists, (2) the CoCross tablet hosting the CoCross4Pros application along with the device-specific add-on modules for monitoring auscultation and vital signs, and (3) the commercially available, medical devices for the acquisition of auscultation and vital signs. It enables almost real-time hearing of annotated, with regards to the location of the auscultation, lung and heart sounds outside the ICU along with accompanying data such as contextual information  and oximeter measurements. The data are securely stored and can be easily shared among intensivists retaining authorization and access control measures. The CoCross platform is secure, lightweight, and scalable and provides isolated control of data per clinic/ICU while also being easily expansible to support additional biomedical devices for the acquisition of additional vital signs if needed, without rebuilding/updating the CoCross4Pros application. Biomedical devices can be a spirometer, a scale, a glucose monitor, a blood pressure monitor or an ECG monitor. Data acquired using CoCross are used for building a new multisource annotated database that would enable the development of advanced digital diagnostics for COVID-19 patients.Given the severity and persistence of the COVID-19 infection, which is manifested in heavy lung dysfunction causing dyspnea and increased respiration ineffective efforts due to fibrosis, as well as failure in the heart mechanical functionality and kidney dysfunction, there is an urgent need for the development of eHealth systems for use in the clinical environment, as well as outside the hospital for a continuous monitoring of the evolution of the COVID-19 infection and its comorbidities for the optimal therapeutic and preventive health services to the citizens of the world. Some countries such as South Korea, have already developed new ICT systems that are embedded in the hospital information systems (HIS) to tackle the unique problems introduced from the COVID-19 epidemic . In thisThe South Korea paradigm, as well as the China paradigm, show that the COVID-19 pandemic has proved the need for upscaling and accelerating the industrial information integration (III) process, for the health care system and society to tackle worldwide pandemics and acute healthcare needs . It is tThe availability of a wealth of wearable sensors, as well as advanced technologies for monitoring of biosignals, bioparameters and lung imaging, along with a new generation of ICU equipment such as ventilators with EIT  capacity are providing new tools and measurements  which can shed light in the elucidation of the COVID-19 induced mechanisms and thus contribute in the advancement of lung dysfunction digital diagnostics . The neeAdditionally, a number of COVID-19 mHealth systems have been developed for recording in EHR and PHR anamnesis data, as well as public health-related data for patients who were tested positive with the COVID-19 virus either being at home or in the hospital  . There iFrom the experience until now, about 18 months after the breakout of the COVID-19 pandemic, most of the effort at least in the beginning concentrated on developing methodologies and systems to classify COVID vs non-COVID subjects, for use both in the hospitals, as well as in the general population. A number of advanced technologies from nanomedicine to artificial intelligence (AI) to cloud computing to Medical Informatics to tissue engineering and more are being developed to tackle specific problems dealing with COVID-19 such as prevention, cure, disease management, resource optimization and of course data analytics and biomedical information optimal use ,21. One of the current gaps in dealing with COVID-19 is the lack of a widely accepted protocol in dealing with COVID-19 patients in COVID clinics and in ICUs. The problem is the difficulty in assessing the pathophysiology of the lungs and cardiovascular system dynamics, especially in the presence of comorbidities and aging, which in many cases can lead to worsening of the patient’s condition or death. Thus, the unavailability of annotated databases from ICUs with recordings of biosignals, bioparameters and bioimages related to the functionality of the lungs and the cardiovascular system at large, as well as the relationship of these recording and data with the ICU standard recorded parameters and scores  renders the monitoring, medical decision making and interventions a difficult to tackle the problem in COVID-19 ICUs. The most complete existing open-access databases with some annotations and patient information are the MIMIC II  and MIMIThe aims of CoCross are (1) to provide a user friendly and easy to deploy solution for lung and heart sound monitoring of COVID-19 patients receiving care in an ICU, (2) expand the knowledge gained with the development of the ICBHI respiratory database  with theThe CoCross platform, in terms of software, is deployed in two locations; the first is the server (CoCross back-end) hosting, in distinct virtualized containers, the CoCross services and the second is the CoCross tablet hosting the CoCross4Pros application for healthcare professionals along with the applications/modules that are specific to each medical device. The CoCross platform is depicted in In each tablet inside the ICU, the CoCross4Pros application is installed along with distinct applications for each wirelessly connected device . The CoCross4Pros application is managing user authentication, the selection of specific patients, the initialization and termination of a recording session, and the registration of in-ICU notes for the specific patient by the intensivist in charge. It also manages the manual entry of in-ICU accessible values, such as ventilator parameters , etc.), as well as uploading of DICOM files captured by the in-ICU stationed portable ultrasound device. The per-device module applications manage the connectivity with each specific device, setting-up specific parameters either for the device itself or meta information for the specific measurement/signal acquisition.Outside the ICU, in one or more terminals, using the cross-browser, a web-based application for intensivists, the authorized professionals may access all information that was acquired in the recording sessions performed on a specific patient inside the ICU. They can listen to the auscultation sounds that were recorded only a few seconds ago  inside the ICU without the burden of the protective equipment. Using the application for intensivists, the professionals also have access to automatically uploaded ultrasound acquisitions along with patient demographics information and relevant health data.www.keycloak.org accessed on 20 April 2021), which is a widely accepted open-source identity and access management solution and manages user access on both the application for intensivists and the CoCross4Pros mobile application and the sharing procedure of patient data. The data management service is based on the in-house developed-flexible data management framework for integrated care [www.laravel.com accessed on 20 April 2021) and the Model/View/Controller MVC) software design pattern that separates internal representations of information from the way information is presented to the user. The application for intensivists unifies the information retrieved in all the recording sessions of a specific patient and properly presents the data  with an easy-to-use graphical interface accessible from PCs and tablet devices. The CoCross back-end hosts the three components that support the CoCross platform. The authentication and authorization service is based on Keycloak  and ontologies. The semantic data model, defined as an OWL ontology, based on HL7 FHIR resources and datatypes [CoCross is built on linked data , HL7 FHIatatypes , definesatatypes  leverageData creation and data retrieval API endpoints and implementations have been separately packaged. This command query separation allows us to scale those two different aspects of data management independently, depending on the deployment scenario’s requirements.Authentication and authorization services are now available allowing us to define and enforce authorization access rules for specific data resources.Each service component is now packaged as a separate Docker container, and supports horizontal scaling, allowing our framework to be deployed to a wide variety of on-premises or cloud scenarios.The CoCross services’ implementation represents the latest iteration of our in-house developed data management framework for health data. Compared to previously published work :Data creThe authentication and authorization services, using Keycloak and its OpenID Connect implementation. These services allow us to manage user registration, patient registration and data sharing.The data management services are built and deployed using standard Java EE technologies. Data management primarily refers to creating and querying for health data, including unstructured data such as auscultation recordings. In addition, the system’s data model is also managed via a service, and its development is decoupled from the backend deployment. This is a feature of our framework that has proved extremely useful in the case of CoCross. It allowed us to deploy the system and start collecting data quickly, while at the same being able to react to users’ feedback. We can iterate on existing features and provide new ones without experiencing service unavailability. Structured data are stored as RDF resources using the OpenLink Virtuoso server, while the SPIN framework is used for data integrity and validation. The API endpoints are implemented using JAX-RS and the resulting Java war is deployed using the WildFly application server.The CoCross services are:Any intensivist in the intensivists group that has access to the medical record of a patient may access all the data recorded in the ICU recording sessions after their automatic synchronization. Using the CoCross web application, the intensivists manage the contents of each medical record and may perform user registration and record sharing. Along with the presentation, per patient, of all the ICU acquired data, either per data type  or per recording session, the intensivist may enter numerical, categorical or media files acquired outside of the system. Additionally, ICD10 diagnosis along with basic demographic information  may be entered into the medical record along with DICOM images from imaging modalities . Snapshots from the CoCross web application are shown in The stored numerical measurements are presented either as tables or as graphs, depending on the number of data points. The intensivists may playback the recorded auscultations via the web browser or download the auscultation locally as a sound file. Similar capability (online view or download) is provided also for DICOM image series. An internal messaging system has been developed to allow exchange of messages among intensivists in specific context. Via this messaging system the intensivist, either inside or outside the ICU, may discuss/comment on a specific patient, recording session or acquisition.The web application for intensivists is developed using the Laravel framework and it is accessible both via tablet  and PC . The use of the Laravel framework enhances the maintainability of the application and simplifies the addition of extra features and the collaboration of the development team.The CoCross mobile applications manage the interaction with the intensivist and the acquisition of data from the Bluetooth medical devices inside the ICU. The design of the CoCross mobile applications aims to be as simple and unobtrusive as possible for the end-users, while at the same time aiming to be scalable by design allowing the developers to add support for new medical devices. To that end, extending the support of the platform for an additional device, requires only the development of a lightweight application-specific for each device model, managing the communication parameters of the specific model and type of device. These modules have, where possible, no user interface , this means that acquired measurements are automatically transferred to CoCross4Pros once the measurement procedure is completed or have limited interaction with the user to set up measurement of specific metadata . All module applications communicate with the CoCross4Pros application via the Android inter-process communication (IPC) using the Android interface definition language (AIDL) and they require the existence of the CoCross4Pros application to work properly. Using AIDL the CoCross4Pros defines a set of functions that the modules can call/invoke, to be executed in the context of  the CoCross4Pros application. Doing so, the acquired data are stored linked to the selected patient, and they are, subsequently, synchronized to the specific medical record using the credentials of the responsible intensivist.The CoCross4Pros application, apart from the definition of the AIDL functions and the implementation of their functionality, also manages (a) the interaction with the user and, (b) the secure transfer of the locally stored data to the CoCross back-end via the data management service. Specifically, it manages the authentication of the user requiring full credentials (email/password) and active network connection at least once. Each user may define a PIN stored only in the tablet to assist faster and, if needed, an off-line login sequence. The intensivist can view a list of patients under his/her responsibility and start a recording session . All datThe CoCross4Pros application also enables the exchange of various notes about the patient or the specific recording session. These notes may include communication between the resident intensivist and the expert/consultant regarding possible questions for the patient or about a specific signal. This functionality is particularly important during the COVID-19 pandemic since non-critical observations may be missed due to the inability to easily transfer objects in and out of the ICU. The data of each recording session are stored locally on the tablet once acquired and they are securely transmitted to the CoCross data management service without any user interaction.All data are transmitted and stored at the CoCross back-end as RDF graphs serialized in Turtle (Terse RDF Triple Language). In The CoCross platform was deployed in June 2020 in two of the COVID-19 reference hospitals in Greece: Initially in the 1st ICU of “G. Papanikolaou” hospital in Thessaloniki (May 2020) and shortly after (June 2020) in the ICU of “Evangelismos” hospital in Athens. Upon permission of the respective scientific boards , recordings from patients with COVID-19 receiving care in ICU have been performed and are still ongoing. The protocol included auscultations of the lungs from six locations, namely: the auscultations of the (1) right lung—apex front, (2) right lung—base front, (3) right lung—base back, (4) left lung—apex front, (5) left lung—base front, and (6) left lung—base back, and, from four locations of the heart: the standard auscultations points for the (7) aortic valve, (8) pulmonic valve, (9) mitral valve and, (10) tricuspid valve. In ® Electronic Stethoscope Model 3200) and a portable ultrasound device (ATL Milano Wireless Ultrasound Probe Convex Color Doppler—C05C) were provided and kept inside the ICU at all times. Since this was the initial deployment, to ensure the feasibility of the study no dedicated expensive servers were deployed. The CoCross backend was deployed in two dedicated and isolated virtual machines (VMs)—one per site. The specific setup offered the required isolation of data among the different clinics/hospitals while also stress-tested the system under real-world conditions. Specifically, since the VMs used were low-end , the performance of the system was assessed to identify minimum requirements in terms of possible hardware required for new deployments. In each ICU a 10” tablet, a Bluetooth pulse oximeter (Medisana™ pulse oximeter PM150) and a Bluetooth digital stethoscope . Subsequently, the recordings were available for review via the web application for intensivists. The presentation of the patient dashboard view which includes all the data for the specific patient varies slightly depending on the number of sessions that were performed. For, the current, worst-case scenario, i.e., the patient with the most recording sessions , the dashboard was presented in 20 s, which was acceptable by the intensivists.During the first nine months (June 2020–February 2021) of CoCross deployment in the ICUs, 176 patients, 61 females with average age 66.2 ± 12.4 years and 115 males with average age 64.8 ± 11.7 years, were registered to the system with confirmed COVID-19 and at least one auscultation recording session was performed in 134 patients. The patients were monitored from 1 to 46 days with a median value of 5 days, resulting in 508 distinct recording sessions that include 3477 auscultations of 20 s each.A histogram of the number of auscultation recordings acquired via CoCross per subject is presented in It is no longer sufficient to offer applications of new and powerful functionality. Users also expect that they can learn how to use the application without much effort, solve their tasks fast and efficiently and are able to control the interaction at each point. In addition to these goal-oriented interaction qualities, it is also important that the product catches the user’s attention and interest and that using the system is interesting and stimulating. Consequently, hedonic, not directly goal-oriented interaction qualities have to be considered, as well in order to be successful . AlthougCurrently the long-term effects of COVID-19 on patients’ health and especially the association of the virus with the development of comorbidities has not been fully examined. Moreover, the optimal management of COVID-19 positive subjects who are asymptomatic or present mild symptomatology at the time of their initial evaluation, mainly patients in higher danger of exhibiting severe forms of the disease  is still debatable. Social distancing measures imposed on our societies due to the COVID-19 pandemic add an extreme challenge for providing digital personalized point-of-care health tools for achieving prevention in COVID-19 dispersion, early detection, and risk stratification especially in relation to vulnerable groups .The problems we are facing with COVID-19 at the moment fall in two categories: (I) identify COVID-19 positives, (II) assess the COVID-19 severity and identify therapeutic interventions.In the first category, many methods have been developed since last year. Most of them use imaging analytics largely from X-rays since this modality is easy to use widely in clinical and primary health care settings ,31,32. AIn the second category, there is still a long way to go. The severity assessment is difficult to be derived from X-rays, CT or coughing or single biosignals, even from PCR tests. It is our view that the first environment for assessing the COVID severity is either the COVID clinics or the ICU environment. The reason is that patients who enter the ICU have been definitely diagnosed with health-threatening COVID-19 driven lung dysfunction, which in most cases is combined with heart, kidney and other organs and systems impairments.Our goal in this paper was to present CoCross, a new system used for the creation and development of a dynamic database resulting from multisource/multimodal recordings in the COVID-19 ICUs from two Greek hospitals. These data are aimed to be used for the assessment of the COVID-19 patient status and for the medical decision support in prescribing the necessary interventions for the COVID-19 patients’ treatment and positive ICU outcomes. We believe that systems like CoCross provide necessary and innovative biomedical informatics platforms for the development of COVID-19 annotated ICU databases, which in turn, are expected to trigger methodologies and analytics tools for COVID-19 and at large lung digital diagnostics, as well as better management of multi-morbid patients with lung disease as the primary disease .The accumulated experience from the initial deployment of CoCross inside the ICU on COVID-19 patients has been extremely positive. The system has offered the ICU physicians the unique capability to have a clear view of the auscultation findings of their patients, has helped them identify severe complications at an early stage and guide proper therapeutic interventions accordingly. The storage of the sound files and the ability of a group of intensivists to listen to them was an added value leading to more informed and scientifically robust choices. The evaluation study of the system constitutes one of the first attempts to capture the lung sounds and their variations over time in critically ill COVID-19 patients and combine metrics from X-rays and lung ultrasounds for the more accurate and efficient assessment and diagnosis of COVID-19 patients in the ICU . Future One of the great potentials provided by the CoCross system and the resulting database is the possibility to fuse information from the analytics from lung ICU Ultrasound (LUS)  with X-rThe ongoing study using the CoCross platform will address the need for an alternative method of bedside monitoring to patients with COVID-19 treated in the ICU, ideally combining different diagnostic techniques, and transmitting the recordings wirelessly for further assessment by the ICU team of intensivists. This CoCross functionality can address several COVID-19 ICU organizational and management issues related to the optimal resource management of the ventilators, shortage, and safety of the specialized medical staff .The use of the CoCross system is expected to result in the development of a large, annotated database of lung/heart sounds and lung imaging to be leveraged by machine learning and clinical research protocols and advance the Integration Information Initiative (III) including micro-nanosensors, ICT solutions and health care systems management .In this paper we have presented CoCross, a platform that enables monitoring, recording and fusion of clinical information with chest sounds and imaging of COVID-19 ICU patients. The resulting multisource/multimodal dynamic database is aiming to fuel future research regarding open issues, such as, severity and disease progress prognosis and identification of the optimal treatment. The platform was successfully deployed in two ICUs in Greece and resulted in the acquisition of multisource data from 176 ICU patients, corresponding to an average monitoring period of 5 days, including a dataset with 3477 distinct 20-s auscultations. The platform is well accepted and positively rated by the users, who also exhibited that they could learn how to use the application without much effort and with minimal technical support.CoCross is easy to deploy, requires minimal hardware cost and has proven to be technically robust. It has demonstrated its capabilities during use in ICU, collecting annotated auscultation sounds of lung and heart, based on commercially certified stethoscope, and providing almost real-time assessment of these sounds by the experts in and outside of the ICU. The system also supports several commercial certified devices for biosignal acquisition and health monitoring, as well as support for uploading/review of DICOM imaging from different modalities.As limitations of CoCross we identify the current support for only one stethoscope model, and that the tools developed for monitoring the expansion of the generated database are not integrated into the web application for intensivists.The successful deployment at Papanikolaou and Evangelismos hospitals led to the adoption of the system by additional ICUs and pneumology departments. A CoCross clone was deployed in March 2021 for the ICU of the university hospital of Alexandroupolis, Greece, later that year on the pneumology department of the CHUC—Centro Hospitalar e Universitário de Coimbra  and, on December 2021 CoCross was deployed in the ICU of AHEPA university hospital in Thessaloniki Greece. In parallel with the straightforward task of setting up a system clone, additional concepts to be managed were required by local experts such as additional ventilator parameters, recording of interventional events such as intubation or extubation, etc. The new concepts were added, with minimal effort, to the local ontology and to the application for intensivists. Changes applied by the new requirements were propagated to the existing deployments enhancing the system without altering either the existing data or their semantics.Two are the main pillars related to future efforts linked with CoCross apart from network expansion. The first concerns the aggregation of the data and the preliminary analysis of the data that CoCross is acquiring to assess the quality/completeness of the generated database, and the second is the adoption of new devices for auscultation that provide a better quality of acquired signals.