The objective of this study is to establish a framework for analyzing infrastructure dynamics affecting the long-term steady state, and hence resilience in civil infrastructure systems. To this end, a multi-agent simulation model was created to capture important phenomena affecting the dynamics of coupled human-infrastructure systems and model the long-term performance regimes of infrastructure. The proposed framework captures the following three factors that shape the dynamics of coupled human-infrastructure systems: (i) engineered physical infrastructure; (ii) human actors; and (iii) chronic and acute stressors. A complex system approach was adopted to examine the long-term resilience of infrastructure based on the understanding of performance regimes, as well as tipping points at which shifts in the performance regime of infrastructure occur under the impact of external stressors and/or change in internal dynamics. The application of the proposed framework is demonstrated in a case of urban water distribution infrastructure using the data from a numerical case study network. The developed multi-agent simulation model was then used in examining the system resilience over a 100-year horizon under stressors such as population change and funding constraints. The results identified the effects of internal dynamics and external stressors on the resilience landscape of infrastructure systems. Furthermore, the results also showed the capability of the framework in capturing and simulating the underlying mechanisms affecting human-infrastructure dynamics, as well as long-term regime shifts and tipping point behaviors. Therefore, the integrated framework proposed in this paper enables building complex system-based theories for a more advanced understanding of civil infrastructure resilience. In a bid to cater to these burgeoning developments, the demand for civil infrastructure has grown disproportionately to the supply that is currently made available by civic bodies. For instance, the American Society of Civil Engineers estimates that an additional 3.6 trillion dollars are required, merely to keep up with the increasing rate of demand for infrastructure [Globalization and large-scale urban development in the latter half of the 20tructure .An important factor affecting the demand-supply economics of civil infrastructure is the longevity and utility of said infrastructure over a given period of time. External factors such as inclement weather directly affect the performance of physical assets, a statement which is of greater importance today than ever before owing to climate change. In fact, climate change is expected to affect the performance of physical assets, both directly and indirectly  as evideThe ability of civil infrastructure to cope with the impact of these stressors is characterized as resilience, which is an emerging topic in the field of infrastructure sustainability. Resilience, in this context, is defined as the capacity of civil infrastructure to experience the impact induced by various types of stressors, while retaining the function to provide services required for the socio-economic development and safety of humans . In esseConsidering the pivotal role played by civil infrastructure in sustainable socio-economic development as well as in protecting communities, a better understanding of the long-term transformation of infrastructure under external stressors is a critical step towards enhancing the sustainability and resilience of the communities themselves. Over the last decade, literature pertaining to infrastructure resilience –8 has ceCivil infrastructure systems can be considered as complex systems  composedA complex systems approach holds the key to addressing knowledge gaps in the development of a holistic framework that is capable of conceptualizing and assessing the long-term resilience of civil infrastructure. Under this approach, resilience of civil infrastructure is characterized using three mechanisms: (i) external stressors; (ii) internal dynamics; and (iii) regime shifts.  theorizeThis study utilized a complex system-theoretic approach for the civil infrastructure long-term resilience investment. The literature related to ecological sciences has made significant advancements in adopting a complex systems perspective for understanding the long-term resilience in ecological systems. According to –20, resiUsing a complex systems approach, the long-term resilience landscape of civil infrastructure systems can be investigated based on stable states and regimes that determine the performance behavior of the system . ChangesBased on the existing evidence related to resilience in complex systems ,13,18–22External stressors can be chronic or acute in nature. Chronic stressors are gradual, low impact-high probability events that are caused by changes in the internal feedback processes and the external environment. Acute stressors are abrupt, high impact-low probability events that are mainly caused by extreme events. Both chronic (e.g. accelerated erosion due to climate change) and acute stressors (e.g. flooding) affect the dynamics of the infrastructure and as a consequence, its resilience. For instance, climate change (chronic stressor) is a major driver of changes in the socio-environmental conditions surrounding civil infrastructure ,26. ClimCivil infrastructure as a complex system is composed of various components which are connected by a complex set of direct and indirect interactions and controlled by not one micro-behavior, but by a host of drivers. The observable collective behavior  of civil infrastructure emerges from the underlying internal interactions and feedback among the system’s components , as well as through the impact of external stressors. The internal dynamics of civil infrastructure are determined based on the interaction between human actors (i.e. agencies and users) and physical networks. The dynamics of each component  are determined by two factors: (i) interaction between the components, which is governed by dynamic rules that impose the activity or decision on the human actors and distribute the condition (performance) to the physical networks; and (ii) global fluctuations in the overall performance of the infrastructure system . Fig 2 dThe performance regime of infrastructure is defined as the steady state related to the system’s performance, which is maintained by internal dynamics associated with the interactions between human actors and physical infrastructure affected by different stressors. A regime is a characteristic behavior of a system which is maintained by mutually reinforced processes or internal dynamics . A changIn the next step of this study, a dynamic (time-dependent) multi-agent simulation model was created and validated to capture and explain the complex dynamics of stressor-human-infrastructure nexus. The elements of the multi-agent model are explained in the following section.Further to identifying different components of a complex system, capturing mechanisms that dictate dynamic interactions between them is a necessary step towards assessing the long-term resilience performance of the system as a whole. To this end, a Multi-Agent Simulation (MAS) method was adopted in order to capture the coupled human-infrastructure dynamics. MAS enables modeling complex and real-world systems through the adoption of influential concepts such as adaptation, emergence, and self-organization . This meWater distribution infrastructure is a vital part of urban water supply systems, comprising of expensive and often complex physical assets. When thought of as a complex system, it includes physical pipeline networks delivering water to users, the municipality or private company as an institution that manages the infrastructure and the organizations and individuals who consume the water . In wateThe physical condition of an infrastructure asset denotes its structural capacity and ability to withstand different types of stressors. On the other hand, the functionality of infrastructure indicates its ability to serve its intended function at the desired level of service. The approach to modeling the mechanisms of physical degradation varies for different infrastructure  . HoweverFrom a system perspective, age-based deterioration can be used as a criterion to sort various sections (lengths) of piping in the network into discrete age groups based on pre-defined age ranges. As a consequence, aging sections would move from one age group to the next during the process of deterioration. Thus, modeling degradation as a dynamic process would capture long-term transformation effects in the network which would lend greater accuracy to the assessment of the system’s resilience.In addition to physical network degradation, human actors (users and administrative agencies) play an important role in the long-term dynamics of water distribution infrastructure. For instance, expectation of and demands placed by users determine the actual quantity of water supplied by the network, while municipalities and private institutions (administrative agencies) which manage the decision-making processes related to renewal and restoration, affect the ability of the network to maintain the required supply. Consider the following scenario which illustrates the interplay of human behavior in the network. The demand placed on the network fluctuates with changes to the user population. These fluctuations drive the price of water, which in turn affects the revenue generated by administrative agencies. Since revenue drives renewal decision processes as well as regular maintenance work, a direct consequence of this interplay is the degradation rate of physical assets in the water distribution network.In response to the various factors which affect the decision-making process, administrative agencies often identify strategy targets to pursue when maintaining large infrastructure networks. A strategy target may be defined as that indicator of network performance (e.g. average condition and break frequency) which is deemed acceptable by the administrative agency. Thus, any strategy target is a combination of various factors in the behavior spectrum of its human actors. The proposed model captures various behavioral factors affecting the decision-making process such as capital and operational expenses, revenue, water price and capital improvement funds, which are uniquely combined to create three heuristic renewal strategy targets for the network: (i) controlling the average break frequency (ii) controlling average condition of network and (iii) regular renewal based on age of pipes. When the agency adopts a break-control strategy, the motive is to keep the frequency of breaks below a certain target threshold. The annual pipe renewal process would be maintained until the desired average annual break frequency over a five-year horizon is reached. If the target is average condition control, the agency strives to maintain the condition of the network below the required threshold through the renewal process. In either of these strategies, renewal costs exceeding the base allotment are met through the activation of a surplus called the capital improvement fund. This fund is allocated based on a 5-year budget from which 20% can be used annually towards the renewal process if expected costs are exceeded. In the case of an age-based (regular) renewal strategy, the agency performs maintenance work only on sections of the pipeline which are older than 100 years (100 years is the average service life of water pipes as reported by ). The caIt is apparent that renewal decision-making processes of the utility agency and other adaptive actions affect the dynamic variables of the infrastructure, resulting in different performance regimes. Hence, a long-term resilience assessment of the water distribution infrastructure system would be incomplete in their absence.The impact of stressors on civil infrastructure systems has been highlighted in the previous sections. The impact of chronic stressors in particular, has greater bearing when the long-term dynamics of a water distribution infrastructure are being investigated. The proposed model framework considers the impact of population change and funding fluctuations as external stressors. Accordingly, different levels of capital improvement fund and various rates of population change are accounted for, to investigate its influence on the dynamics of the infrastructure. Together, water demand fluctuations arising from population change, and availability of funds for renewal, produce critical points which lead to shifts in performance regimes of the water distribution network.The following section presents the methodology and estimation procedures required to parameterize the MAS model and develop its computational components for a numerical case study of water distribution infrastructure system.The MAS model in this study was created based on the conceptual logic and principals representing the real-world behaviors of an urban water distribution infrastructure system. The creation of a computation representation for all the input and output parameters of the conceptual model entails constructing mathematical algorithms to match the conceptual logic representing the behaviors of water distribution infrastructure. The computational representation of the MAS model was developed in an object-oriented programming platform . It integrates the institutional actors’ renewal decision-making processes with the physical infrastructure performance in order to assess the long-term resilience behavior of water distribution infrastructure system under different stressors  and various scenarios . A numerical case of a water distribution network composed of 180 miles of pipes with different materials and age categories was used to create the computational simulation model that captures the dynamics of the system to examine its resilience. The population in the service area of this case is 113,000 . This MAS model of the water distribution infrastructure system includes three classes of agents: water pipeline network, water users, and utility agency, each of which is simulated in the model as an object . The relationships among agent classes and their attributes are based on the existing literature and empirical information. i = 20,40,60,80,100). For each group, which represents a certain condition level, there is a defined material . Deterioration of pipes is modeled based on their age. As pipes age, their condition group may change; every year, 5 percent of pipes in each condition group (except condition group 100) moves to the next condition group. Another mechanism that affects the transition of pipes into different condition group states is pipe renewal. Based on the renewal decision-making outcomes of the agency agent (which is explained later), a certain percentage of pipes is renewed and moves from condition group 100 to condition group 20, annually. The pipeline network agent includes different pipe classes with a defined total length (in miles) divided into 5 condition (age) group states  in condition group i and in the network, respectively. The other attributes of the pipeline network agent include breakage, leakage, and service reliability. Breakage is modeled using a stochastic process that determines the frequency of pipe breaks during one year. The frequency of breaks for each mile of pipes is a function of pipe’s material and age. The model uses a Poisson Process with a mean rate of break (λ) (shown in λ) represents the mean rate of break per mile per year [N0 is the initial number of breaks per mile per year in a new pipe, g is the growth rate which is 0.07 for PVC and 0.078 for CIP pipes [i is the pipe’s condition group. The total number of breaks in the network per year (NB) would be sum of the breaks in each mile of the entire network pipes. The annual water loss due to network breaks is calculated using WLB is the annual water loss due to network breaks , NB is the total number of annual breaks, d is the total duration of disruption due to break (day), and d are user-defined input parameters into the model.The aging of the pipes is also reflected in the changes in the percentage of materials for each group. At the beginning of simulation, the model starts with a defined percentage of PVC and CIP pipes for each group. As pipes age, these percentages change . Based on the portion of pipe length in each condition group, the model computes the network average condition , which is a unitless variable, depends on network average condition  and the annual water demand , respectively.Similarly, network leakage is determined as total water volume lost in the network, which is a fraction of annual water demand. Leakage rate (S), which represents the delivered water to users , is calculated using Considering the total water loss in the network due to both breakage and leakage, annual water supply (SR) parameter is defined. This parameter represents the extent to which the supplied water through the network met the demand. To determine SR, in each year (t) the cumulative supply is divided by the cumulative demand and this gives the service reliability until that year (SRt) based on Sj and Dj are delivered water and water demand  in year j, respectively.In order to determine whether the supply of water transported by the pipes can meet the given demand, a service reliability (P) and number of households (Nh) consuming water from the given pipeline network, which determine the amount of water  demanded from the network. In each year, water demand (Dt) is assumed to be the average indoor water demand  which is calculated using The agent of users represents population (Pt) is changed in each year based on the user-defined values of growth/decline rate (r) and base population (P0). Population in each year (t) is computed based on Population (WP0) in the model is user-defined . Water price (WPt) can hike annually over the simulation period with a price hike rate (w) based on Revt) is determined based on that year’s projected water demand  and water price  using The utility agency agent models the decision-making processes of the agency. The renewal decision-making process is affected by revenue, operational and capital expenditures, and capital improvement fund. The initial water price (OpEx), which increases exponentially based on the network average condition (UOpEx is the unit cost (dollar per mile) for operating and maintaining the network .UCapEx is the cost of rehabilitation of one mile of pipes with PVC ($1.2 million), and RL is the length of pipes (mile) with age 100 or more (5% of pipes in condition group 100). If the available funding for capital expenditures is less than the required capital expenditures, the renewal length would be equal to the available funding divided by the unit cost of rehabilitation.The renewal rate depends on the availability of funding for the capital expenditures. The available funding for the pipes renewal equals to the annual revenue minus the operational expenditures. The required funding for capital expenditures of renewing pipes is determined using CIF) in addition to the annual revenues. The CIF investment is made every five years, and thus the renewal due to CIF occurs over five years. Hence, each year, up to twenty percent of CIF is invested for renewal. The annual CIF allocation would be utilized to implement more pipe renewals in the network after spending the revenue as explained before . In the model, CIF ($ Million) is a user-defined input parameter. CIF allows) in order for the network to reach the strategy target . That is, similarly, for the strategy of break control, the model determines to what extent the renewal process should be conducted in order for the network to reach the desired number of breaks.For additional renewal and in order to keep the network performance in terms of break frequency or average condition below the user-defined threshold , the agency would use a capital improvement fund  were employed to verify the data, logic, and computational algorithms related to the simulation model. First, the initial conditions and the ranges of the parameters were compared to the existing empirical data to ensure the reliability of the parameters in the model . For exaAfter using different internal verification and external validation techniques to ensure the model quality, the simulation model was used for building various experiments based on all possible scenarios. Various simulation experiments were conducted through the change of model input parameter values and logics in the computational model. The possible scenarios were established based on different combinations of the input variables (parameters) in the model, shown in Two sets of simulation experiments were conducted to evaluate the long-term resilience of the water distribution infrastructure system. First, the impacts of decision and physical infrastructure attributes were examined in order to explore the significant attributes and their critical threshold value that could lead to the occurrence of tipping point behaviors. To this end, time-series results related to the network average condition were visualized to investigate the existence of non-linear increase or decline trends in the simulated long-term performance measure. Accordingly, the threshold values related to decision and infrastructure attributes , at which the steady state of infrastructure performance is disrupted were examined as tipping points. The results related to regime shifts and tipping points were used to evaluate the long-term resilience of the case study network under different adaptation strategy scenarios . Second, the impacts of population changes and funding fluctuations on the network performance parameters and its ultimate effects on network service reliability over the analysis period  were investigated. To this end, the mean values of the infrastructure service reliability were plotted against different values of population change rate and capital improvement funding level to visually examine the occurrence of shifts in the service reliability performance of the infrastructure system. Accordingly, the occurrence of regime shifts and tipping points were detected and used to analyze the sensitivity of long-term infrastructure system performance to external stressors.The MAS model of water distribution infrastructure system was used to extract, animate and visualize the long-term regimes of different performance and condition measures in the case study water distribution network. For instance, the simulation model was run for a scenario of 1% population growth, 4% water price hike, and regular renewal strategy. These visualized long-term performance regimes of the water distribution infrastructure are utilized to detect the occurrence of regime shifts, visually ; and accordingly, the long-term resilience of the system is evaluated. Thanks to the analysis results, two sets of theoretical constructs related to complex systems approach to infrastructure resilience were examined: (i) the performance regime of infrastructure is shaped by its internal dynamics; and (ii) the effective performance of infrastructure is sensitive to external chronic stressors.The simulation model was used to examine how infrastructure dynamics shape performance regimes. To this end, the effect of renewal strategies (as an element affecting infrastructure dynamics) on the performance regime of the network was evaluated. Two renewal strategies  were considered. For each renewal strategy, different required performance targets and capital improvement funding levels were assessed. In total, 25 scenarios were simulated, where each scenario represents a unique state of infrastructure dynamics in the case study. In this analysis, resilience is determined based on detection of regime shifts in the stable state of performance in the system. Different performance measures could be used for assessing possible regime shifts. In this set of analysis, the network average condition was selected among the network performance measures for two reasons: first, the network average condition collectively represents the physical state of the pipes in the network; second, most of the network performance measures such as leakage, breakage, and service reliability, either directly or indirectly, are affected by the network average condition. Accordingly, resilience of the system is analyzed based on regime shifts in the long-term average condition of the network.CIF). As can see in This change in the performance regime of infrastructure is a significant phenomenon and can be characterized as a critical point or tipping point. Critical or tipping points represent values related to the internal dynamics of infrastructure or external stressor, in which a small change in the parameter value leads to a significant change in the performance of infrastructure. For example, t = 100 in The simulation model was also used to examine the impact of external chronic stressors on the performance regime of the case study infrastructure. To this end, the effect of population changes and funding fluctuations were considered. Various scenarios of population change rates and levels of capital improvement fund were defined; and then for each scenario, one thousand runs of Monte-Carlo experiments were implemented to determine the mean value of the water distribution infrastructure’s service reliability. The service reliablity indicator measures how reliable the network is in successfully delivering the demanded water to users. Figs Since for two other renewal strategies the extent of renewal processes prominently depends on the capital improvement fund, the impact of this external stressor (funding fluctuations) on the system effective performance was examined under different renewal scenarios. This study was conducted with the objective of capturing internal dynamic behaviors which influence the long-term resilience of civil infrastructure systems in the presence of external stressors. The proposed framework characterizes the object of study (a water distribution network) as a complex system and offers a multi-agent simulation (MAS) model to quantify its components, dynamic processes, and external stressors acting on it. The output of the model depicts the performance regime of the system over an extended horizon which enables the detection of regime shifts to evaluate the long-term resilience.Different performance regimes were observed in response to changes in the renewal decision-making process of the administrative agency, which is an expected outcome owing to the fact that availability of funding is a function of revenue, which in turn depends on the demand generated by the user population. The model also identified tipping points in the performance regime when specific renewal strategy targets were adopted by the administrative agency. For instance, when a renewal strategy of average condition controlling at 45 was adopted in conjunction with a capital improvement fund of $15 Million, the infrastructure satisfied the target by exhibiting long-term performance. Any decrease to the fund however, caused a regime shift if the agency continued to maintain the same strategy target. These results show that the performance regime of an infrastructure system is shaped by its internal dynamics, which reinforces the premise that changes in internal dynamics would lead to regime shifts in long-term behavior of the system. The model also adequately captured the long-term response of infrastructure when subjected to external stressors. For instance, a positive correlation between decline in user population and the long-term service reliability of the infrastructure was observed. The model showed that a price hike of 4% annually would maintain the service reliability of the case study water distribution infrastructure system in a stable state, despite changes to its user population.The contributions of this study are threefold: theoretical, computational, and practical contributions. From theoretical perspective, this study proposed a complex system-based framework for infrastructure resilience assessment through a better understanding of internal dynamics and tipping point behaviors. Accordingly, it proves that (i) the performance regimes of infrastructure are shaped by their internal dynamics and (ii) chronic stressors affect the effective performance of infrastructure system. In terms of computational contribution, this study developed a MAS model of a water distribution network that adequately captures and quantifies the dynamic behaviors and performance regimes of the infrastructure system in the presence of external stressors. Practically speaking, through the quantifying the impacts of external stressors, internal dynamics and performance regime shifts, researchers can predict the resilience of civil infrastructure systems with greater accuracy. This in turn would help decision-makers formulate policies  that enhance the sustainability and resilience of these systems.The simulation model presented in this study doesn’t include all the dynamic mechanisms affecting the long-term resilience of a coupled human-infrastructure system. Depending on the objective of a study, additional dynamics can be captured and modeled using the proposed framework. For example, in the analysis shown in this paper, the influence of user behaviors was not within the study objectives. Hence, in this model, the influence of consumer actors was modeled exogenously with a prescribed rate of population growth and water demand. Future studies can examine the dynamics of consumer behaviors in evaluating the long-term resilience of infrastructure systems by evaluating individual consumer’s response to water price  and otheS1 File(DOCX)Click here for additional data file.S1 Fig(TIF)Click here for additional data file.S2 Fig(TIF)Click here for additional data file.


In order to realize the intelligent production of sapphire crystal, it is important to obtain the growth status from the furnace by charge coupled device (CCD). However, a significant challenge is that traditional approaches are often not valid to separate the images of the melting interface well due to the low contrast and uneven brightness from the heater. In this paper, an improved Otsu algorithm based on dynamic particle swarm optimization (DPSO) is proposed to find the exact threshold band as contour of the crystal. In this method, the Otsu method is constructed firstly, then DPSO is used to find the optimal threshold band. Experimental results show that the proposed algorithm can separate the texture of crystal growth images well and has high robustness. Sapphire crystal with excellent thermal properties, optical properties, and mechanical properties has important scientific research value and wide market prospect ,2,3, sucThe growth of artificial crystal is usually finished in the melting furnace through the process of melting, seeding, crystallizing, and cooling , and theThe temperature distribution of the melting interface in the seeding step has an important influence on product quality . The purTo obtain the growth status of sapphire correctly and quickly, image segmentation is no doubt an important topic, and thus it is the emphasis of this paper. Many techniques have been developed for image segmentation. Among them, the principle of similarity has been widely used, which utilizes the similarity among the image objects with pre-defined criteria for partitioning. Thresholding, edge detecting, region splitting, and clustering are similarity-based methods ,9. In reThreshold segmentation of digital images is one of the widely used techniques for image segmentation. It divides a digital image with several or fixed threshold values. In the same kinds of pixels, objects with same gray value are regarded as the same object . The chaEdge detection can realize image segmentation by recognizing and extracting the features of the image boundary and can detect image contours. The image obtained after edge segmentation well retain the image’s morphological features. In the region of the image edge, the gray value changes significantly. On the contrary, in the edge of the area, the image gray value changes little . Thus, aRegion segmentation divides an image into different regions according to the similarity criteria. Methods based on common regional segmentation include the regional growth method, regional division combination method, and watershed method. The key idea of this method is to select the right seed points and determine the right growth rules . The advThe clustering method divides the sample data into different categories according to the similarity of measurement criteria However, there are two problems in this method. One is how to compare the similarity between sample data, and the other is how to divide these data into different categories according to their similarity . The maiConsidering the above existing segmentation methods, thresholding holds the prime position from the view of robustness, simplicity, and accuracy . ThreshoOver the past few years, many thresholding-based segmentation techniques have been reported ,16,17. AEscherichia coli presented in the human intestine. The BF algorithm provides better performance based on the quality of the solution and the speed of convergence than the other existing multilevel thresholding methods. However, the efficiency and the robustness of this algorithm are related to the chemotaxis step size. Large step size accelerates the progress of searching for the optimum position but does not ensure the global optimum. On the other hand, small chemotaxis step size guarantees that bacteria will find the global optimum, but it requires more searching time. Meanwhile, a few state-of-the-art methods have been developed for multilevel thresholding based on bio-inspired computing paradigm, such as the Grey Wolf optimizer Escherichptimizer . Though The image segmentation of a melting interface is essential to decide the status or correct seeding time for intelligent crystal growth equipment. However, strong heat radiation and turbulent flow fields in the high-temperature and high-luminance environment inevitably interfere with the CCD camera signal. This causes the target image to become blurred and the images of the melt interface to have poor texture. Hence, it is difficult to obtain good performance by traditional methods. Inspired by the fact that the grey value of the boundary of different regions is different due to the uneven illumination, and the grey value of the crystal image is continuously distributed, an improved Otsu algorithm based on dynamic particle swarm (DPSO) is presented for image segmentation during crystal growth. Firstly, particles are distributed in different areas, and different areas will undergo local optimization. Secondly, parallel processing of the proposed algorithm reduces the computing time and improves the performance. Then, image enhancement is adopted to enlarge the grey difference between crystal and boundary. This method maximizes the Otsu algorithm’s objective function using DPSO to find the optimal threshold band. After getting the threshold band, the given growth images are subdivided into small segments based on the thresholds.The remainder of the paper is organized as follows: in Otsu is a classical algorithm of image segmentation; an image can be divided into object region (A) and background region (B). To get the optimal threshold, the difference between the object and background needs to be as great as possible. The measurement of that difference is called Otsu .L stand for the number of grey levels of a given image Let Global intensity means And where n = 1, 2, 3, …).The above method can be extended to multilevel thresholding problem, which is defined as While the number of thresholds A PSO algorithm based on the simulation of bird individual hunting behavior can be used for intelligent swarm optimizations. It simulates the behavior of birds where each bird will make its contribution in the searching process depending upon its fitness. The overall fitness of the individuals is used for the identification of the food center which is considered as the best global estimation.Step 1Initialize parameters Step 2Repeat the Step 3 to Step 7 for Step 3Calculate fitness for each particle;Step 4Update the best position of the particle Step 5Update the best position of the whole swarm Step 6Update position Step 7Determines whether meeting the termination conditions.The implementation of the particle swarm optimization is detailed as follows:In PSO, an individual is used to represent the potential solution of the optimization problem. The position and velocity on each dimension are updated using the following two equations:A graphical illustration of the proposed method is given in (a)Initializes the maximum number of iterations (b)Initializes velocity (c)Set the lower boundary and upper boundary of (d)Initializes the width of threshold band.Step I: Parameter initialization (a)Histogram calculation;(b)Histogram equalization.Step II: Image enhancement The basic idea of histogram equalization is to put the histogram of the original image into a form of uniform distribution, thus increasing the pixel grayscale to a dynamic range to obtain the overall image contrast. The occurrence probability of the The distribution of Step III: Fitness function calculationCalculate the fitness of current particles’ position using the improved Otsu objective function.Due to the low contrast of the image and inconspicuous boundary line, a single threshold cannot separate the crystal image well. Therefore, a continuous threshold band is proposed for segmentation. Because the crystal image has the characteristic of continuous grey values, taking the threshold band as the boundary line would classify a small part of the crystal as the boundary line and make the overall division clearer.Equation (1) can be rewritten as:Thus, The band is in the range of [Step IV: Update position and velocity for each particle (using Equations (4) and (5))The inertial coefficient is a key parameter for PSO performance. The fixed inertial parameter will cause the algorithm to converge to the local optimal solution. Moreover, the linear reduction of the inertial parameter will cause the algorithm to miss the global optimal solution. Hence, in the proposed method, the  adopted :(10)ω={ωshown in .Also, in , dynamicStep V:Calculate fitness for each particle after updating.Step VI:If the number of iterations reaches the maximum go to Step VII. Otherwise go to Step IV.Step VII:Select the threshold band associated with the overall best particles.Besides the Otsu objective function, regional consistency is also adopted to judge the effect of image segmentation. The interior of the region formed by image segmentation is similar, and the quality of the segmentation can be described by the degree of uniformity within each region. The In order to verify the performance of the proposed algorithm, a crystal growth image is taken as the object, and the DPSO-based image thresholding algorithm is validated through simulations in MATLAB on a computer with the following configuration:(R) Core(TM) i7-6700 CPU @ 3.40 GHz and 8 GB of memory with Microsoft Windows 10 operating system, produced by Legend Corp., Beijing, China.IntelThe objective values along with the corresponding thresholds and CPU time obtained using the proposed method were compared to already reported multilevel thresholding algorithms, such as the BF, PSO, and GA algorithms. The input images are obtained by CCD as shown in The parameters used in the proposed DPSO algorithm are shown in In this study, to validate the efficiency of the proposed image segmentation algorithm for crystal growth, a set of experiments were conducted on a large set of real crystal growth images from different growth stages obtained from CCD. As shown in In order to enlarge the grayscale difference for segmenting, histogram equalization is used for image enhancement. The images after image enhancement and their histograms are given in In The objective function values obtained by the Otsu-based DPSO, PSO, BF, and GA algorithms are presented in The segmented images  of the Otsu-based and bio-inspired methods are shown in An improved image segmentation algorithm based on dynamic particle swarm optimization (DPSO) is proposed in this paper. Images of the melting interface are always obscure and low contrast due to the uneven and random interference from the heater. This means the separation lines of textures do not have the same gray degree. Hence, intelligent swarm optimization was introduced to solve the segmentation problem. In addition, using a threshold band as the boundary was adopted rather than multilevel thresholding in traditional methods for segmentation. Experimental results show that the proposed method based on DPSO gives better performance compared with several typical methods.


As a multichannel signal processing method based on data-driven, multivariate empirical mode decomposition (MEMD) has attracted much attention due to its potential ability in self-adaption and multi-scale decomposition for multivariate data. Commonly, the uniform projection scheme on a hypersphere is used to estimate the local mean. However, the unbalanced data distribution in high-dimensional space often conflicts with the uniform samples and its performance is sensitive to the noise components. Considering the common fact that the vibration signal is generated by three sensors located in different measuring positions in the domain of the structural health monitoring for the key equipment, thus a novel trivariate empirical mode decomposition via convex optimization was proposed for rolling bearing condition identification in this paper. For the trivariate data matrix, the low-rank matrix approximation via convex optimization was firstly conducted to achieve the denoising. It is worthy to note that the non-convex penalty function as a regularization term is introduced to enhance the performance. Moreover, the non-uniform sample scheme was determined by applying singular value decomposition (SVD) to the obtained low-rank trivariate data and then the approach used in conventional MEMD algorithm was employed to estimate the local mean. Numerical examples of synthetic defined by the fault model and real data generated by the fault rolling bearing on the experimental bench are provided to demonstrate the fruitful applications of the proposed method. With the development of industrial production, the key equipment and structure always have the characteristics of complex structure, variable operation status and continuous online service. Especially in the metallurgical industry, most parts such as rolling bearing are working under an environment of high speed, overloading and high temperatures. As an important approach of structural health monitoring, the damage feature extraction algorithm for rolling bearings based on signal processing technology has important significance ,2,3. NatIn order to improve the signals to noise ratio (SNR) of actual measured vibration signal, many scholars have done a lot of research work ,14,15,16The above-mentioned signal processing method of time-frequency analysis is mainly oriented to the single sensor. The single sensor can only provide limited useful information, and the characteristics of the early equipment running status information are hard to be inspected. On the contrary, the multi-channel signals processing technology can be employed to identify the multichannel correlation, improve the accuracy and reliability of diagnosis. In short, it can provide more scientific evaluation to equipment online service quality . EspeciaThis paper was organized as follows: the basic principles and characteristics of the trivariate empirical mode decomposition via convex optimization were introduced in the second chapter. In the third section, the numerical simulation analysis was conducted to confirm the effectiveness of the algorithm. In the fourth part, the validity of the proposed method was verified by the bearing data of the experimental bench. The conclusions of the study were given in section fifth.Given a trivariate signal It was reported that the non-convex regularization methods outperformed the convex regularization methods on matrix approximation and could enhance sparsity . Thus, iTypical non-convex penalty functions should be 2nd order differentiable, continuous and symmetric. An example of a non-convex penalty function is the rational penalty function, which is defined as:Equation (3) is parameterized by the parameter It is noted that the proximity operator is associated with the non-convex function Then, the alternating direction method of multipliers (ADMM) algorithm in conjunction with variable splitting is employed to solve the trivariate data matrix of the low-rank approximation problem. The ADMM algorithm is applied to Equation (2), and yields the following iterative procedure. Firstly, initialize the model with analysis . The valThen, the singular value decomposition (SVD) of the covariance matrix to sequence . It is aThe direction of the highest curvature is sampled by rotating the ellipsoid Algorithm 1. Trivariate empirical mode decomposition via convex optimizationPerform the low-rank matrix approximation via convex optimization framework to a trivariate signal The projection The time instants Interpolate [Calculate the residual by Cauchy-type convergence condition is chosen as the stopping criterion of the sifting iterative process, which indicates the iterative will stop under the circumstance that the discrepancy between adjacent sifting results is less than a threshold with a range 0.2–0.3. For clarify, Rolling bearing is an important transmission component supporting the rotating part of the mechanical equipment. If a bearing with an outer race that is fixed to the structure, a typical fault rolling bearing model can be simplified as follows :(12)xi, vertical (y) and axial (z) directions should be considered as far as possible, which is aimed to collect the trivariate signal. Suppose there is no relative sliding between the raceway surface and the rolling elements, and the outer ring is fixed. Then, the outer fault frequency 87.01 Hz can be obtained. The detailed experimental parameters are shown in The electric spark machining method is used to carry out pitting treatment on the outer ring of the replaceable bearing to simulate the outer faults. A PCB-352C33 acceleration sensor is used to collect the acceleration signals. The installation position of sensor is located in the horizontal direction, vertical direction and axial direction of the bearing on the right side of the experimental platform. The bearing type is deep groove ball bearing with model number of 6207. The external diameter of removable bearing was It can be seen from The spectral analysis by FFT is applied to the measured trivariate signal in Finally, the proposed method is employed to the measured trivariate vibration signal and the result in time-domain is shown in By comparing the results respectively provided by MEMD and the method proposed in this paper, we can make a conclusion that the proposed method has better performance in trivariate signal mode decomposition and fault feature extraction.Based on the traditional MEMD method, a novel method of trivariate empirical mode decomposition via convex optimization for rolling bearing condition identification is proposed and tested in this paper. The main research works are listed as follows: (1) a low-rank matrix approximation via convex optimization framework is proposed for the trivariate signal denoising. By introducing parameterized non-convex function, its performance is improved; (2) the principal component obtained by SVD operator is employed to indicate the sample directions, thus the non-uniform sample scheme is presented to meet the requirement of inhomogeneity of the data distribution in high-dimensional space; (3) through the analysis of the simulation signal and the measured vibration signal, it is demonstrated that the proposed method is superior to MEMD in multi-scale feature extraction. It can be regarded as powerful tool in trivariate signal processing.


In order to enable the calibration model to be effectively transferred among multiple instruments and correct the differences between the spectra measured by different instruments, a new feature transfer model based on partial least squares regression (PLS) subspace (PLSCT) is proposed in this paper. Firstly, the PLS model of the master instrument is built, meanwhile a PLS subspace is constructed by the feature vectors. Then the master spectra and the slave spectra are projected into the PLS subspace, and the features of the spectra are also extracted at the same time. In the subspace, the pseudo predicted feature of the slave spectra is transferred by the ordinary least squares method so that it matches the predicted feature of the master spectra. Finally, a feature transfer relationship model is constructed through the feature transfer of the PLS subspace. This PLS-based subspace transfer provides an efficient method for performing calibration transfer with only a small number of standard samples. The performance of the PLSCT was compared and assessed with slope and bias correction (SBC), piecewise direct standardization (PDS), calibration transfer method based on canonical correlation analysis (CCACT), generalized least squares (GLSW), multiplicative signal correction (MSC) methods in three real datasets, statistically tested by the Wilcoxon signed rank test. The obtained experimental results indicate that PLSCT method based on the PLS subspace is more stable and can acquire more accurate prediction results. In the past few decades, near-infrared spectroscopy (NIR) has been widely used in various fields, because of its fast speed and the fact that it does not cause damage to sample characteristics. These areas include pharmaceutical ,2,3, bioNumerous relevant calibration transfer methods have been proposed in articles. In general, these methods can be divided into two types: transfer standard and non-standard. The transfer standard requires the same standard samples to be measured on the master instrument and the slave instrument. In this type of method, according to the stages in which the adjustment occurs are further divided into four types.The first type is the method of correcting the slave spectra. In the standard samples, the slave spectra are made as close as possible to the corresponding master spectra by a transfer matrix. The most widely used are direct standardization (DS) and piecewise direct standardization (PDS) methods ,16. In tThe second type is the method of simultaneously correcting the master spectra and the slave spectra. Commonly used is calibration transfer by the generalized least squares (GLSW) method ,18. GLSWThe third type is the method of correcting the predicted values. Mainly the slope and bias correction (SBC) method , this meThe fourth type is the projection method. For example, calibration transfer method based on canonical correlation analysis (CCACT) , which uIn practical applications, it is difficult or even impossible to measure the same samples on two instruments due to the position of the measuring instrument and the stability of the samples, etc. At this time, it is necessary to use a method that does not require measurement of the same standard samples, that is, a non-standard method. These methods are mainly divided into two types.One is the signal preprocessing method, which removes the baseline offset and the linearly sloped baselines by simple mathematical operations of the first derivative and the second derivative. Common methods include multiplicative signal correction (MSC) , finite The other is the projection method. It includes transfer component analysis (TCA)  and kernIn this paper, a novel projection method is proposed, which is a feature transfer model based on PLS subspace (PLSCT). PLSCT establishes the PLS model of the calibration set of the master instrument firstly, constructing a low-dimensional PLS subspace, which is a feature space constructed by the spectral feature vectors. The PLS model is then used to extract the predicted features of the master spectra and the pseudo predicted features of the slave spectra, that is, to project all spectra of the master instrument and slave instrument into this PLS subspace. Then, the ordinary least squares method is used to explore the relationship between the two features in the identical PLS subspace, the relationship will then be resorted to construct a feature transfer relationship model.Notice that the pseudo predicted feature of the slave spectra is acquired by the PLS model established by the master instrument rather than the PLS model of the slave instrument. And PLSCT does not need the response variable corresponding to the standard set. In addition, compared with PDS, PLSCT corrects the feature of the spectra rather than the spectra. In contrast to CCACT, PLSCT uses PLS to find the covariance between the spectra and the response variable, instead of using CCA to find the correlation between the master spectra and the slave spectra.In order to validate the performance of the PLSCT model, we not only compare its prediction results against those of the SBC, PDS, CCACT, GLSW, and MSC methods, but also apply the Wilcoxon signed rank test  to deterFirst of all, The number of the factors for constructing the pseudo predicted feature matrix from the standard set of the slave spectra , PDS(5), PDS(7), SBC, CCACT, GLSW, MSC and Recalibration2 at 95% confidence level. The RMSEP improvement resulting from PLSCT compared with these models were 51.77389%, 54.35396%, 57.02112%, 87.45319%, 42.18862%, 61.34526%, 56.43832% and 69.98222%, respectively shown in Table A2As in the previous cases, the LVs and RMSEP of Calibration, Direct transfer and Recalibration are shown in In the PLSCT model, the number of factors for constructing From Compared with other models, the RMSEP improvement of PLSCT over them can reach up 16.3743%, 15.12146%, 14.35178%, 40.04516%, 16.81376%, 41.83697%, 24.21448%, 23.82937% and 2.908651%, respectively. Furthermore, the differences between PLSCT and other models are all statistically significant at the 95% confidence level shown in Table A2http://www.eigenvector.com/data/Corn/. The dataset is composed of 80 corn samples. Three near-infrared spectrometers were used to measure these samples, with wavelength range from 1100 nm to 2498 nm at 2 nm intervals (700 channels). The property of moisture, oil, protein and starch of corn is contained in the dataset. In this paper, the moisture content was chosen as the property of interest. We choose M5 as ‘master instrument’, MP5 as ‘slave instrument’. The difference between the spectra measured on M5 instrument and MP6 instrument can be observed in The first dataset was corn dataset. We can conveniently access to obtain it at http://www.idrc-chambersburg.org/content.aspx?page_id=22&club_id=409746&module_id=191116. The wavelength range of the manufacturer A was 730 nm–1100 nm and the interval was 0.5 nm. The dataset only provides the reference protein values. In this paper, we take the first instrument of manufacturer A as ‘master instrument’ and the second instrument as ‘slave instrument’. The second dataset was the wheat dataset, which consisted of 248 samples measured by three instruments of manufacturer A. This dataset was the shootout data of the International Diffuse Reflectance Conference (IDRC) in 2016. We can obtain it from http://www.eigenvector.com/data/tablets/index.html. There are three reference values associated with this dataset, but we were only interested in weight content for each sample. The difference between the spectra in the pharmaceutical tablet dataset is shown in The third dataset came from the IDRC shootout 2002, which contains 655 pharmaceutical tablets measured on two spectrometers, with the range from 600 to 1898 nm, and the interval was 2 nm. We can obtain it from We adopt the Kennard and Stone algorithm  to splitIt must be noted that the Kennard and Stone algorithm was applied to the master spectra when splitting the calibration set and test set, while the Kennard and Stone algorithm was applied to the slave spectra when extracting the standard samples.α = 0.05).The number of latent variables used in the PLS model was selected by a 10-fold cross-validation. In order to avoid over-fitting caused by the inclusion of redundant latent variables, the optimal number of latent variables was achieved based on the statistical F-test  . The calculation of RMSEP is as follows:In order to compare the prediction performance difference between the proposed model and other models more directly, Equation (2) was used to calculate the RMSEP improvement of the PLSCT method compared with other methods:p-value between the two prediction errors. If p > 0.05, there is no significant difference between the two methods. Otherwise, there is significant difference.In addition, by comparing prediction error of the different models, the Wilcoxon signed rank test at the 95% confidence level was utilized to point out whether there was a significant difference between PLSCT and other methods. In python, we used the wilcoxon function in the scipy package to directly calculate the In this paper, we define the spectral matrix as PLS is a widely used multivariate calibration technique. PLS applies score vectors model the relationship between The matrix of regression coefficients is:With the regression coefficient matrix In the PLSCT, the PLS model was built on the calibration set of the master instrument to construct the PLS subspace, which is also the feature space constructed by the feature vectors of the spectra of the master instrument calibration set. The number of latent variables (LVs) in the PLS model is determined by cross-validation.On the basis of this PLS model, the predicted feature matrix of standard set in the master instrument Similarly, the pseudo predicted feature matrix of standard set in the slave instrument The two predicted feature matrices obtained are derived from the same PLS model of the master instrument, that is to say, all spectra are projected into the identical PLS subspace constructed by the master instrument. In the identical PLS subspace, there should be a linear relationship between the two feature matrices. So The linear relationship between the two feature matrices can be solved through the ordinary least squares method, by the following equation:Once In this paper, an ingenious calibration transfer method based on PLS subspace is proposed. PLSCT uses the same PLS model to project the spectra into the identical PLS subspace. In the identical subspace, a feature transfer model is constructed by narrowing the differences between the predicted feature of master instrument and the pseudo predicted feature of the slave instrument via an ordinary least squares method. Additional, PLSCT does not need the response variable corresponding to the standard set. As expected, experimental results on three real datasets show that compared with PDS, SBC, CCACT, GLSW, and MSC, the PLSCT model is more stable and can obtain more accurate prediction results. The reason why the PLSCT model can achieve such remarkable results is that while the spectra of the slave instrument are projected into this subspace, some noise effects such as scattering that are unrelated to the response variable will be removed from the spectra, and then the feature transfer in the identical PLS subspace can more accurately narrow the differences between the predicted feature of master instrument and the pseudo predicted feature of slave instrument.


The self-assembly of proteins and peptides into amyloids is a key feature of an increasing number of diseases. Amyloid fibrils display a unique surface reactivity endowing the sequestration of molecules such as MicroRNAs, which can be the active moiety of the toxic action. To test this hypothesis we studied the recognition between a model RNA and two different steric zipper spines using molecular dynamics simulations. We found that the interaction occurs and displays peptide-sequence dependence. Interestingly, interactions with polar zipper surfaces such as the formed by SNQNNF are more stable and favor the formation of β-barrel like complexes resembling the structures of toxic oligomers. These sequence-structure-recognition relationships of the two different assemblies may be exploited for the design of compounds targeting the fibers or competing with RNA-amyloid attachment The self-assembly of peptides and proteins into amyloid aggregates is the common thread of a large number of degenerative diseases including Creutzfeldt-Jakob, Parkinson's, Alzheimer's, Amyotrophic Lateral Sclerosis, type 2 diabetes, among others , miR378a (CUCCUGACUCCAGGUCCUGUGU), and miR344 (AGUCAGGCUCCUGGCUAGAUUCCAGG) released in exosomes of prion-infected cells , followed by a run with conjugate gradient algorithm (7 × 103 steps). After minimization, the production simulations were conducted with isotropic position scaling (ntp = 1) and the temperature was kept constant using a weak-coupling algorithm  method as implemented in Amber 16 . Each basic unit cell was replicated along two directions, while explicit water was added along the third direction providing simulation boxes with the dimensions reported in Table For each of the interaction simulations, four different RNA chains  were randomly placed in the bulk water layer. Interestingly, once docked onto the amyloid the RNA molecules may shift their position on the surface, and undergo conformational changes. In some cases, the formation of dimers was observed Figure . ConsideEach of the representative structures was next used to calculate the energetics of the interaction by means of the MMPBSA approach . Within the limitations of our model  . The most representative conformations of RNA in complex with either peptide sequences are qualitatively similar, and are shown in Figure To verify the potential validity of the observed RNA docking modes on the surfaces, we searched the Protein Data Bank for experimentally characterized structures of RNA-protein complexes. It is interesting to note that our simulation-based observation shown in Figure In terms of drug design, we suggest that pharmacophore models may be designed by keeping into account the relative stereochemical relationships of the functional groups most responsible for binding the amyloid aggregates observed along the trajectories and by using different relative weights to keep into account the information on the persistence, stability and/or variability of the interactions established with the model aggregate. In this case, an ideal pharmacophore could be constituted by a central scaffold decorated with 3 to 4 aromatic polar moieties that mimic the pyrimidine groups Figure . IdeallyWe have studied the recognition between a model RNA stretch and different amyloid fibrils by molecular dynamics simulations. To the best of our knowledge, this is one of the first reports of a possible interference or RNA-trapping mechanism exerted by an ordered amyloid fibril surface. We found that this interaction occurs and that for a common RNA stretch, the recognition is clearly peptide-sequence dependent. Interestingly, in the case of the SNQNNF fibril surface the interactions appear to be more stable and to favor the formation of β-barrel like complexes resembling the structures of toxic oligomers. On the basis of these results, specific amyloid sequences may sequester nucleic acid material on the surfaces their fibrillar and oligomeric assembly and derange the biochemical pathways in which RNA is used as an information carrying molecule. Moreover, we suggest that such initial sequence-structure-recognition relationships of the two different fibers may be a source of information for the design of compounds targeting the fibers or competing with RNA-amyloid attachment.MM designed research, ran simulations, wrote the paper. MG and GC designed research, wrote the paper.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Estimating the number of people in highly clustered crowd scenes is an extremely challenging task on account of serious occlusion and non-uniformity distribution in one crowd image. Traditional works on crowd counting take advantage of different CNN like networks to regress crowd density map, and further predict the count. In contrast, we investigate a simple but valid deep learning model that concentrates on accurately predicting the density map and simultaneously training a density level classifier to relax parameters of the network to prevent dangerous stampede with a smart camera. First, a combination of atrous and fractional stride convolutional neural network (CAFN) is proposed to deliver larger receptive fields and reduce the loss of details during down-sampling by using dilated kernels. Second, the expanded architecture is offered to not only precisely regress the density map, but also classify the density level of the crowd in the meantime . Third, experimental results demonstrated on four datasets (Shanghai Tech A (MAE = 88.1) and B (MAE = 18.8), WorldExpo’10(average MAE = 8.2), NS UCF_CC_50(MAE = 303.2) prove our proposed method can deliver effective performance. The automatic analysis of crowd has been a particular security technique in the current intelligent surveillance literature , which wPeople-Counting technology can be generalized into two kinds of literature: detection methods and regression methods. Traditional approaches for crowd counting from images relied on hand-crafted representations to extract low-level features. These features were then mapped for counting or generating density maps via various regression techniques. Some of the early methods  relied oTo the best of our knowledge, the traditional methods are hampered in dense crowds, and the performance is far from expected. Inspired by the success of Convolutional Neural Networks (CNN) for various computer vision tasks, many types of CNN-based methods have been developed, some new techniques are related to visual understanding ,14, whilMost recently, Sam et al.  propose In this paper, the proposed work is motivated by the idea of multiple task learning and dilated convolution in Cascaded-MTL  and CSRNDilated convolutions make it possible to acquire a larger receptive field, meanwhile applying small convolutions. Yu et al.  first atMultiple task learning can boost the classification and regression performance of a deep network. Eigen et al.  presente(1)A principled way of combining different size dilated convolution layers.(2)Multiple tasks learning framework not only regress the density map, but classify the density level of the crowd.(3)Some results could outperform state-of-the-art methods on several benchmarking datasets.(4)The proposed model has been verified in a camera with an embedded system.However, we address designing an easy-trained CNN-based density map manager in this paper. The dilated convolution layers are deployed as the front-end to enlarge receptive fields and fractional stride convolution layers as the back-end to restore its spatial resolution. By making use of such structure, we significantly reduce parameters which make CAFN trained easily. The proposed MTCAFN extends the CAFN to multiple tasks not only in density map regression but also in density-level classification. Moreover, MTCAFN is developed as an extended network, uniting the shallow features extracted from the shallow sub-net with the broad features obtained from deep sub-net contributes to improving the network effectively via updating parameters. Furthermore, the proposed model uses the multi-task learning method to get the ability of density-level classification; finally, the loss of details will be restored by transposed convolutional layers as much as possible. This new improvement could significantly boost the performance of crowd counting. Besides, we outperform the previous crowd counting solution lower MAE, frequently used benchmark datasets, respectively. To summarize, we make the following contributions:The rest of the paper is stated as follows. The framework proposed in this paper is designed to estimate the crowd density if the number of crowd people exceeds a threshold, defined as representing an abnormality.The proposed system could accurately predict the crowd by using smart systems with a single camera and Nvidia Tx2 boards. First, the crowd scene images from three data sets will be inputted into our proposed model CAFN and MTCAFN on PC for training and testing, where we apply 1/10 of training set as the validation set. The trained model is transferred to the Tx2 board for online computing. After that, the real crowd scene will be captured by a camera and the final result will be predicted by our model. When the output counting is larger than the altering threshold T, the system will send the early warning to the surveillance center. The whole flowchart of the alerting system is shown in In the framework CAFN, the fundamental idea is to deploy a double-column dilated CNN for recording high-level features with larger receptive fields and producing high-quality density maps without obviously expanding network complexity. In this subsection, we firstly introduce the architecture, a network whose input is the image and the output is a density map of the crowd (say how many people per square meter), and then obtain the headcount by integration, then we present the corresponding training method.Inspired by the idea of CSRNet, we utilized dilated convolutions as the front-end of CAFN because of its greater receptive fields, unlike adopting dilated convolution to capture more features when the resolution has been dropped off to a shallow level in CSRNet. Atrous convolutions are primarily made use of in this paper, which intent to gain more image information from an original image, then the transposed convolutional layer is to enlarge the size of image and up-sampling the previous layer’s output to supplement the loss of details.In this paper, for attaining the training dataset, we crop 9 patches from each image at different locations with 1/4 size of the inputting images. The first four patches contain four quarters of the image without overlapping while the other five pieces are randomly cropped from the input image. Based on three branches of MCNN, we add dilation rate to filters to enlarge its receptive fields. To reduce net parameters, we consider four types of double-column association as experimental objects, which will be discussed in detail in The overall structure of our CAFN is illustrated in All convolutional layers use padding to keep the previous size. The convolutional layers’ parameters are denoted as “conv (kernel size) @ (number of filters)”, max-pooling layers are conducted over a 2 × 2 pixel window with stride 2. The fractional stride convolutional layer is denoted as “Conv Transposed (kernel size) @ (number of filters)”, and PReLU is used as a non-linear activation layer.Then Euclidean distance is exploited to measure the difference between the estimated density map and ground truth. The loss function is defined as follows:The network is fetched by an image of arbitrary size and outputs crowd density map. The network has two sections corresponding to the two functions, with the first part learning larger scale features and the second part restoring resolution to perform density map estimation. One of the critical components in our model is the dilated convolutional layer. Systematic dilation supports the exponential expansion of the receptive field without loss of resolution or coverage.The higher the network layer, the more information the original image contains in the unit pixel, that is, the larger the receptive field, however, which is done by pooling and takes the reduction of the resolution and the loss of the information in the original image as a cost. Due to the existence of the pooling layer, the size of the feature map in the back layer will be smaller and smaller. Dilated convolution is a kind of convolution idea that the downsampling will reduce the image resolution and the loss of information for image semantic segmentation. This feature enlarges the receptive field without increasing the number of parameters or the amount of computation. Examples can be found in The second component consists of a transposed convolution layer for up-sampling the previous layer’s output to explain the loss of details due to earlier pooling layer. Convolution arithmetic  is shownP is the set of all points where people are located.We apply a simple method to ensure that the improvements obtained are due to the proposed model and are not dependent on the sophisticated methods for calculating the ground truth density maps. Ground truth density map Since neural networks have complex network structures, training a reliable neural network requires a great deal of training data to update network parameters. However, the public datasets of crowd counting are usually included a few dozens to a few hundred samples, which results in relatively poor robustness and weak generalization of the model in some applications. Multi-task learning is a machine learning method that learns simultaneously from multiple tasks, like learning favorable information about similar tasks while effectively alleviating the problem of sparse data and improving network performance. The application of multi-task learning in deep convolution networks is mainly divided into two categories, one way is hard weight-sharing, and the alternative approach is soft weight-sharing, as seen in Therefore a regression problem is converted into a multi-class classification problem. First, the crowd density was divided into five categories among the training data applied in this paper. Then, image classification is roughly shown as the following steps, (1) the original image is utilized as an input; (2) extracting features via the CNN; and (3) outputting the classification results by employing the Softmax classifier. The classification cross entropy is applied as the loss function in sub-network of density-level classification is demonstrated in Equation (3).The crowd density and the classification loss are merged into a loss for the neural network to train and update the parameters. The function definition is shown in Equation (4):The overall structure of the MTCAFN is shown in In this section, we propose the experimental details and evaluation results on four publicly available datasets: Shanghai Tech Part_A and Part_B , UCF_CC_The proposed models are trained in a DELL PC station with a GPU Titan XP using Torch framework  equippedFirst, four types of different column combinations see  with difIn Shanghai tech dataset contains 1198 annotated images, in which a total of 330,165 people with centers of their heads interpreted. As far as we know, this dataset is the largest one regarding the number of annotated people. This dataset consists of two parts: there are 482 images in Part A, which are randomly crawled from the Internet, and 716 images in Part B, which are taken from the busy streets of metropolitan areas in Shanghai. The crowd density varies significantly between the two subsets, making an accurate estimation of the crowd more challenging than most existing datasets. Both Part A and Part B are divided into training and testing: 300 images of Part A are used for training and the remaining 182 images for testing, and 400 images of Part B are for training and 316 for testing.Results from MTCAFN show that the double-column version achieves higher performance on Shanghai Tech Part A dataset with the lowest MAE, shown in The UCF_CC_50 dataset includes 50 images with different perspective and resolutions. With arriving at an average number of 1280, the number of persons annotated per image varies from 94 to 4543. Five-fold cross-validation is performed following the standard setting in Reference . Result Zhang et al. firstly introduced WorldExpo’10 crowd counting dataset. This dataset contains 1132 annotated video sequences which are captured by 108 surveillance cameras, all from Shanghai 2010 World Expo. The author provided a total of 199,923 annotated pedestrians at the centers of their heads in 3980 frames. 3380 frames are used in training data. The testing dataset includes five different video sequences, and each video sequence contains 120 labeled frames. We train our model following the instructions are given in To validate our system on Nvidia Tx2 visualizing the experimental results, the predicted density map and the number of estimated crowd count. Compared with the ground truth and the number of labeled people. The quality of density maps is measured using two standard metrics: PSNR  and SSIM . The higher the two indicators are, the more representative of the generated density map is. As seen in In this paper, we presented a smart camera-aware crowd counting system for jointly adopting dilated convolutions and fractional stride convolutions. Atrous convolutions are devoted to enlarging receptive fields which is beneficial to incorporate abundant characteristics into the network, which enables the model for learning globally relevant discriminative features thereby accounting for large count variations in the dataset. Additionally, we employed fractional stride convolutional layers as the back-end to restore the loss of details due to max-pooling layers in the earlier stages, therefore allowing us to regress on full resolution density maps. The model structure has moderate complexity and strong generalization ability, which possess satisfactory density estimation performance in densely crowded scenes via the experiments on multiple datasets. MTCAFN is extended where shallow sub-net extracts pixel-level detail features for crowd density map estimation, deep network extracts high-level semantic features for crowd density classification. Multi-task learning is adopted to improve the loss function of crowd counting. Through the experimental comparison, the proposed system achieved better results and verified the feasibility and effectiveness of the method.The proposed work has some limitation as follows. A perfect alerting system focuses not only on accurate crowd counting but also on crowd behavior prediction. Moreover, local information such as sub-group trajectory analysis of the crowd also needs to be investigated. In the future, we will also focus on the compression of the broad network to fit different real-time embedding systems.


GPS jamming is a considerable threat to applications that rely on GPS position, velocity, and time. Jamming detection is the first step in the mitigation process. The direction of arrival (DOA) estimation of jamming signals is affected by resolution. In the presence of multiple jamming sources whose spatial separation is very narrow, an incorrect number of jammers can be detected. Consequently, mitigation will be affected. The ultimate objective of this research is to enhance GPS receivers’ anti-jamming abilities. This research proposes an enhancement to the anti-jamming detection ability of GPS receivers that are equipped with a uniform linear array (ULA) and uniform circular array (UCA). The proposed array processing method utilizes fast orthogonal search (FOS) to target the accurate detection of the DOA of both single and multiple in-band CW jammers. Its performance is compared to the classical method and MUSIC. GPS signals obtained from a Spirent GSS6700 simulator and CW jamming signals were used. The proposed method produces a threefold advantage, higher accuracy DOA estimates, amplitudes, and a correct number of jammers. Therefore, the anti-jamming process can be significantly improved by limiting the erroneous spatial attenuation of GPS signals arriving from an angle close to the jammer. A GPS signal is at a disadvantage because of its low signal power, which is typically between –125 dBm and –130 dBm at the earth’s surface and is therefore highly susceptible to intentional and non-intentional interferences. GPS is relied upon in many critical infrastructure sectors [In order to detect and mitigate interference, many authors have proposed solutions that exploit the time domain, frequency domain, space domains, or combinations of the fields. In the time domain, pulse blanking is commonly used, whereas, in the frequency domain, adaptive filtering is used ,4,5. TimIn the space domain, adaptive antenna arrays (smart antennas) that employ spatial processing are used. Accordingly, several advantageous and unique capabilities are introduced, for example, the direction of arrival (DOA) estimation, beam-steering, and null-steering. Once the DOA estimates are computed, interference mitigation can be achieved using null steering by placing nulls in the detected direction of interference ,6,7,8. OTherefore, smart antennas continue to attract interest in the field of GPS interference detection and mitigation due to their superiority compared to single-antenna receivers, when dealing with intentional and non-intentional interferences. The interference mitigation capability of smart antennas is mainly based on the accuracy of the DOA estimation of interference sources ,10,11. FThis paper aims to propose a new high-resolution DOA estimation method of jammers in order to provide smart antennas with the most accurate estimate of the jammers’ DoAs such that beam-steering and null-steering processors nullify the jammer signals without erroneously nullifying parts of the GPS signal, thereby decreasing its effectiveness. The proposed method was designed to operate in the precorrelation section of GPS receivers, where mitigation is more effective. It is based on the orthogonal search (OS) technique  and creaThe proposed method is compared to classical DOA estimation and MUSIC using GPS L1 signals obtained from a Spirent 6700 simulator; the arrays utilized are uniform linear array (ULA) and uniform circular array (UCA) arrays. The jamming signals were simulated as CW sources originating from different directions. The performance of the investigated methods is evaluated in the case of single and multiple interference signals. Our results showed that the proposed method introduces a significant enhancement to GPS jamming detection and hence can improve overall system performance.Spatial domain detection and mitigation is a mature field where many DOA estimation methods have been proposed. Classical beam-forming, also known as conventional beam-forming,  represenCapon’s beamformer, also known as the minimum variance distortionless response (MVDR), attempts to improve the drawbacks associated with the classical method and yield a significant improvement . The outAdvances in the field of direction-finding lead to the development of high-resolution DOA estimation methods that utilize the signal subspace. By performing Eigenanalysis on the spatial covariance matrix, the signal and noise subspaces are generated. MUSIC is among these subspace techniques . MUSIC sThe estimation of signal parameters via rotational invariance technique (ESPRIT) algorithm was proposed by . It is cGenerally, spatial signal processing using antenna arrays is considered one of the most effective techniques for narrowband interference detection and primarily suppression ,8. AntenVarious DOA estimation algorithms have been developed for array signal processing applications. The selection of the DOA estimation algorithm is a crucial element of adaptive antenna array design. It directly affects null steering and beam-forming processes.The DOA estimation techniques that are predominantly used are those that are based on classical DOA, Capon, and MUSIC algorithms . It has thn sensor at time th source, This research studies the application of FOS in the DOA estimation of jamming signals . FOS is fined by :(2)ym, and phase. In order to estimate the amplitude and phase of each candidate, the fitting process is carried by selecting pairs of sin and cos functions for DOAs. Hence, for L DOAs, we have 2L-1 candidate functions.The following model can represent the received signal at the antenna array:OS creates an orthogonal basis set  for the The set of orthogonal basis vectors The coefficients This formula was derived to minimize the mean square error (MSE) between the observations vector y (n) and the orthogonal model. The total MSE (Equations (5) and (8) indicate that the calculation of This observation was used in ,32,33 tol search ,32,33. FThe correlation between the candidate basis vector Consequently, the coefficients of orthogonal basis vectors The derivation of Equations (11) and (12) are based on the properties of orthogonal basis vectors FOS constructs the model defined by Equation (4) by adding the candidate functions corresponding to DOA one at a time. The selection of candidate functions corresponding to an estimate Once Reaching a predefined maximum number of terms to be fitted. This requires the knowledge of the number of narrowband interference signals impinging upon the antenna array. However, the orthogonal search approaches may be combined with any of several statistical criteria to determine when to stop adding terms to the model, thus providing an estimate of the number of signals .Reaching a predefined threshold for MSE reduction. This criterion is achieved when the ratio of MSE to the mean squared value of the input signal is below a predefined threshold. Accordingly, the limitation of knowing the number of expected interference signals is waived. However, it may lead to an increase in processing time. Thus, this ratio should be carefully selected to avoid excessive processing time.When adding another term to the model, the MSE reduction is less than the MSE reduction that would be gained if white Gaussian noise was added.The FOS algorithm can be stopped using one of the following criteria:FOS necessitates floating-point operations of the order of searched . MoreoveL2 + CML .The process of candidate function selection plays a crucial role in the FOS algorithm. In this research, the selected candidate functions are pairs of sine and cosine functions corresponding to the DOAs of the search domain. Accordingly, the chosen candidate functions for a ULA and UCA using the model shown in Equation (3) are given by :(16)aksk=2In the following sections, the array design for the ULA and UCA arrays will be studied from a perspective that will ultimately serve the goal of DOA estimation. Therefore, in order to adhere to the common literature notations, the total number of elements in an array will be referred to as The ULA is capable of filtering the electromagnetic environment in which it operates, based on the location of the signals in its vicinity. Array geometry has a significant effect on the DOA estimation abilities of antenna arrays. In this study , the ULAFor the general ULA showed in The quantity in parenthesis is known as the array factor (AF). This factoring is regularly termed pattern multiplication. It can be utilized when the identical array elements are oriented in the same direction. Moreover, the radiation pattern of the array is the result of the multiplication of the element radiation pattern and array factor.When the elements of the ULA are all located on the z-axis, For an M-element, ULA let The array factor formula can be simplified using the following identity (23)∑n=0Therefore, the array factor for ULA is given by:After factoring, Equation (24) reduces to:The AF magnitude pattern for ULA is shown in llenging .Other vital parameters of array factors include the beam width and sidelobe level. The beam width is usually defined as a null-to-null or half-power beam width. The null-to-null beam width is the angular distance between the first nulls around the main beam . The halThe planar array geometries can be divided into three other subcategories: circular, rectangular, and square. The circular arrays stand out of the three, as they do not have edge elements. In the absence of edge constraints, the beam pattern of a circular array can be electronically rotated. Moreover, the circular arrays also have the capability to compensate for the effect of mutual coupling by breaking down the array excitation into a series of symmetrical spatial components ,36.The array factor of a circular antenna array, which is shown in The array factor for a circular antenna array is a function of both spherical angles; therefore, it can filter the received signals based on their azimuth and elevation angles. The effect of the array on the received signal as a function of the angle of arrival can be demonstrated by examining the array factor. The magnitude pattern of the circular array factor is shown in The following section demonstrates the results obtained from the different scenarios that were implemented in order to evaluate the performance of the developed high-resolution DOA estimation method for ULA and UCA arrays geometry. The proposed method was tested with single and multiple jamming signals at different jamming-to-signal ratios (JSRs). The GPS L1 signals were obtained from a Spirent GSS6700 simulator and were down-converted and digitized using the Novatel Firehose digital frontend, as shown in TM software are utilized to verify the proposed method. Initially, the desired experimental parameters such as the number of antenna elements, array geometry, GPS signal frequency , and consequently antenna element spacing are defined. Upon designating the zero-phase element’s (reference element) location in terms of latitude, longitude, and height, the remaining elements of the array are mapped according to the predefined experimental configuration. Once the locations of all the antenna elements are set, a simulation scenario is created using the simulator software. The simulator software enables the reiteration of the simulation scenario without changing any parameters except for the location of the antenna element in 3D space. M-element array, where each iteration corresponds to one of the array elements where Referring to , one radThe jamming signals frequencies are relative to the GPS signal’s center frequency obtained from digitization and down-conversion. Jamming signals were simulated by adding sinusoidal signals to the DGA output. The simulated jammer frequencies ranged from 100 to 400 Hz. This range ensures that the introduced jamming signals fall within the bandwidth of the DGA output digitized, down-converted GPS L1 signals, which are baseband signals with 1.023 MHz bandwidth.Single JammerThe results obtained for a single jammer simulated at a frequency of 100 Hz and arriving at an elevation angle of 50° at a JSR of 15 and 45 dB are shown in The main contribution of the proposed method when estimating the DOA of a single jammer is the high accuracy in the detection of jamming signals’ amplitude, as shown in Multiple Jamming SignalsThe performance analysis of FOS versus MUSIC and classical DOA was examined in terms of resolution, tolerance to JSR, and tolerance to the jamming signals coherence. For this purpose, three jamming signals were simulated with frequencies of 100 Hz, 200 Hz, and 300 Hz and arriving at elevations of The results obtained for three jammers arriving at elevations of  JSRs are shown in The results shown in the above figures demonstrate that the proposed FOS-based method outperformed both the MUSIC and classical methods due to its high tolerance to the variation of JSR and jamming signals coherency. MUSIC detected three jammers at a JSR of 45 dB with zero error in estimated DOAs, but its performance degraded at a JSR of 15 dB as it detected two jammers only. On the other hand, our method’s performance was stable in terms of the number of jammers detected at JSRs of 15 dB and 45 dB, as it detected three jammers accurately with zero error in estimated DOAs. It also degraded slightly at a JSR of 15 dB where its power allocation for detected jammers was not equally divided among the three jammers that were simulated with equal power.The proposed method (FOS) performance was examined using a UCA configuration described earlier, with seven elements equally spaced on the circumference of a circle with a radius of Single JammerA single jammer was simulated as a 100 Hz sinusoidal signal arriving at an elevation of 40° and an azimuth of 40°. Multiple Jamming SignalsIn this scenario, three equal power jammers were simulated at frequencies of 100, 200, and 300 Hz with elevations and azimuths separations of 10°. The DoAs of the three jammers were simulated at , , and . This experiment is conducted at JSRs of 15 and 45 dB. The results obtained from the MUSIC and classical DoA estimation methods are shown in The results shown in High-resolution DOA estimations for interference detection using the ULA and UCA configurations were studied. A new procedure was utilized to simulate GPS received signals using a single RF output Spirent GSS 6700 GPS simulator and a single RF input Novatel FireHose frontend. Postprocessing validated the GPS signals used. The main contribution of this research is the application of nonlinear signal modeling techniques for GPS jamming detection using ULAs and UCAs. The method employed for jammer DOA estimation is based on FOS. Its performance was evaluated in terms of DOA estimation accuracy, magnitude estimation, and the number of jammers detected at different JSRs. The proposed method was compared to MUSIC and classical DOA estimation. It demonstrated significant improvement in estimating the magnitude of the spatial spectrum when compared to MUSIC and consistently outperformed it. MUSIC only showed similar performance when more powerful jamming signals were applied. The proposed method was more accurate at detecting the amplitudes of the single jammers. In the case of ULA, as for multiple jammers, it outperformed MUSIC by 33% in terms of the number of jammers detected at relatively low JSR. In the case of UCA, as for multiple jammers, the proposed method detected all three jammers, and MUSIC only detected two at relatively low JSRs. Moreover, the proposed method also demonstrated higher accuracy in the detection of the three sources of jammers with slight variations in their amplitude at a relatively low JSR of 15 dB. In addition, the capabilities of the classical method are limited by its low resolution, and MUSIC is much more sensitive to noise and performs worse at lower signal power. Since the proposed method is not optimized for real-time operation, it is limited by its higher computational complexity compared to the classical approach. Nevertheless, the added complexity is justified by the high gains in accuracy. The proposed method will detect a higher number of closely spaced jammers and achieve higher accuracies as the number of array elements increases.