Deep neural networks (DNNs) have been shown to be effective for single sound source localization in shallow water environments. However, multiple source localization is a more challenging task because of the interactions among multiple acoustic signals. This paper proposes a framework for multiple source localization on underwater horizontal arrays using deep neural networks. The two-stage DNNs are adopted to determine both the directions and ranges of multiple sources successively. A feed-forward neural network is trained for direction finding, while the long short term memory recurrent neural network is used for source ranging. Particularly, in the source ranging stage, we perform subarray beamforming to extract features of sources that are detected by the direction finding stage, because subarray beamforming can enhance the mixed signal to the desired direction while preserving the horizontal-longitudinal correlations of the acoustic field. In this way, a universal model trained in the single-source scenario can be applied to multi-source scenarios with arbitrary numbers of sources. Both simulations and experiments in a range-independent shallow water environment of SWellEx-96 Event S5 are given to demonstrate the effectiveness of the proposed method. Multiple source localization in an ocean waveguide is a challenging task because of the interactions among multiple acoustic signals. Several multiple source localization methods have been proposed for tracking underwater targets in past decades. Matched-field processing (MFP) is a classical approach for underwater source localization by correlating the modeled field and the experimental field ,2,3. TheHowever, the model based methods usually require the environmental parameters to model the acoustic model in advance. Difficulty in obtaining complete knowledge of the real environment may lead to incorrect or inaccurate localization results. To reduce the dependence on environmental information, recently, many data-driven techniques are introduced to source localization in ocean waveguides ,12,13,14To apply DNNs to a multiple source localization task, generally, there are two ideas in previous studies. The first idea is to train a single neural network that detects the locations of multiple sources using the mixed signals emitted from various location combinations directly ,26,27,28The second idea tries to simplify the multi-source localization task to single-source localization task. The most popular methods are based on the sparsity assumption on sound source signal ,30. AlthTo circumvent these problems, a two-stage DNN based method is proposed to determine both the azimuth angles and ranges of multiple sources successively, which includes a feed-forward neural network (FNN) for direction finding and a long short term memory recurrent neural network   , that isThe block diagram of the proposed method is shown in kth hydrophone and the first hydrophone), which is the steering vector of the dth source, the cost function for the broadband case can be expressed asL denotes the the snapshot number and Rstogi et al. proposed using the hopfield network  in direcEach significant peak of vector Note that only in the testing stage is the FNN using BP algorithm trained to find the directions that sound sources may emerge. For each direction candidate, we extract the corresponding features, then the sources’ ranges are estimated by feeding the features into DNN-2 .To avoid the exhaustive training, we aim to train a general and flexible model that is suitable for situations with different source numbers. Thus, how to design an effective feature, which can be used for various scenarios, is a critical problem. For DNN analysis, the more similar the test set is to the training set, the better the testing result will be. However, in our task, the training set is composed by the single-source signals at different locations while only the mixture is available when testing. It is vital to extract a feature that can represent each single source information from the mixture, so that the test signal (or feature) can be matched with the training signals. Beamforming, which can enhance the signal from the desired direction while attenuating others, is ideal to extract the individual signal component from the mixture. Nevertheless, if we perform beamforming using all sensors, the horizontal-longitudinal correlations of the acoustic field, which include the spatial information of source, will lost in the enhanced signal. Therefore, we introduce subarray beamforming to extract the individual source component, meanwhile preserving the horizontal-longitudinal correlations. The SCM of the enhanced signals at all subarrays is used as the feature.B subarrays, dth direction at the bth subarray is obtained by applying CBF to the signals received by the hydrophones in the bth subarray,dth time delay of the kth hydrophone corresponding to the first hydrophone at the bth subarray (the first hydrophone is chosen as the reference), kth hydrophone and the reference hydrophone, bth subarray, c denotes the sound speed and dth source at frequency Beamforming algorithms can be used to track those interested sources and null out the other sources as interference by controlling the beampattern of an array. The simplest beamforming technique is adopted in our framework, which refers to the delay-and-sum beamforming. It delays the multi-channel signals so that all versions of the source signal are time-aligned before they are summed. To preserve the horizontal-longitudinal correlations of the low-frequency acoustic field, this CBF is conducted on each subarray. The hydrophone array is divided to dth source is calculated byThe SCM of the signals enhanced to each source direction is used as the feature, because it contains sufficient information about the individual signal. The SCM of the DNN  is a dat network  (TDNN),  network  overcomeThe deep LSTM-RNN is shown in g and h are the cell input and cell output activation functions that are The associated computations that map the input vector to the output vector are given as follows:Q denotes the sample number. We use the truncated back propagation through time (BPTT) learning algorithm  Hz and the sampling rate was The signal was transformed to the frequency domain by operating fast Fourier transformation (FFT) (Hanning windowed). The frame length was In direction finding, the configuration of FNN was 5 layers (one input layer + three hidden layers + one output layer) with 128 hidden nodes. The rectified linear units  (ReLU), In source ranging, the LSTM-RNN was three layers with 896 nodes. The activation function was ReLU. The initial learning rate was It should be mentioned that all parameters  of FNN or LSTM-RNN were chosen based on experiments. The tensorflow  toolkit n is an integer denoting the number of azimuth period, For direction finding, the detected sources were classified into two categories, namely the correctly detected sources and the incorrectly detected sources. The detection was considered to be correct if the estimated azimuth angle deviated no more than r) and the true ranges  between the estimated ranges . White noise was added to the simulated signals, resulting in SNRs of 15, 5, and  The SNR  reported The CBF  was chosFor source ranging, we compared the performance of LSTM-RNN with FNN. The FNN was five layers with three hidden layers and 896 hidden nodes. From We also evaluated the performance on HLA under different SNRs. The results are summarized in The second simulation evaluated the performance with or without data augmentation in the two-source scenario. The SNR was set to 5 dB and the neural network was LSTM-RNN. The The third simulation was made to investigate the performance of the proposed method when the SLs of two testing sources were different, where the source with the higher SL referred to the dominant source. The SNR of the dominant source was 5 dB. Define The last experiment investigated the spatial resolution of the proposed method. The separations of two sources were set to The proposed method was further evaluated by real experimental data that were recorded by HLA North of SWellEx-96 Event S5. The water depth was 213 m and the HLA North array is a 240 m aperture horizontal array deployed on the seafloor. The source ship (R/V Sproul) started its track south of the array and proceeded northward at a speed of 5 knots. The signals of the deep source were used for processing. The map of the source movement and the location of the hydrophone array were shown in F source frequencies into account. Referring to Doppler Shift theory, the maximum Doppler shift is The experimental data with sample rate Firstly, we investigated the performance of our proposed method using different frequency bins in the two-source scenarios. The two-source signals were the combination of snapshots from J131 23:47 GMT to J131 23:53 GMT and snapshots from J132 00:19 GMT to J132 00:25 GMT, which were six minutes in total. Three frequency bin sets were investigated, which were In direction finding, the ROC curves are plotted in To demonstrate that LSTM-RNN can make full advantage of the long-term temporal contextual information, we compared the FNN with LSTM-RNN for source ranging. The results are also shown in Next, we investigated the influence of the parameters of LSTM-RNN on the performance of source ranging. Thirteen source frequencies were used (39 bins). The hidden layers were changed from 2 to 4, the hidden nodes were set to 512, 896, and 1024, and the learning rates were chosen from Finally, we evaluated the proposed method on the three-source scenario. The three-source signals contained six minutes that were combined by snapshots from J131 23:47 GMT to J131 23:53 GMT, snapshots from J132 00:07 GMT to J132 00:13 GMT and snapshots from J132 00:23 GMT to J132 00:29 GMT. The ROC curves are plotted in This paper presents a two-stage DNN based method for multiple source localization in a shallow water environment using UHA. We attempt to train a general and flexible model using single-source signals that is suitable for source ranging in various scenarios with different source numbers. The subarray beamforming technique is taken as the feature extractor that separate sources at the level of feature and LSTM-RNN is leveraged for source ranging. Since the subarray beamforming requires the direction information to be known beforehand, a FNN model is trained for direction finding, meanwhile determine the source number. Both the simulation and experimental results demonstrate the effectiveness and superiority of the proposed framework. As LSTM-RNN can make full use of long-term temporal contextual information for the current estimation, it is an ideal model for source ranging. Our method can localize arbitrary numbers of sources that overlap in the TF domain. In our future work, we will make further efforts to improve the robustness of the proposed method in the more complex environments with lower SNRs and more sources.


This paper presents a novel multi-objective optimization based sensor selection method for multi-target tracking in sensor networks. The multi-target states are modelled as multi-Bernoulli random finite sets and the multi-Bernoulli filter is used to propagate the multi-target posterior density. The proposed method is designed to select the sensor that provides the most reliable cardinality estimate, since more accurate cardinality estimate indicates more accurate target states. In the multi-Bernoulli filter, the updated multi-target density is a multi-Bernoulli random finite set formed by a union of legacy tracks and measurement-updated tracks. The legacy track and the measurement-updated track have different theoretical and physical meanings, and hence these two kinds of tracks are considered separately in the sensor management problem. Specifically, two objectives are considered: (1) maximizing the mean cardinality of the measurement-updated tracks, (2) minimizing the cardinality variance of the legacy tracks. Considering the conflicting objectives simultaneously is a multi-objective optimization problem. Tradeoff solutions between two conflicting objectives will be derived. Theoretical analysis and examples show that the proposed approach is effective and direct. The performance of the proposed method is demonstrated using two scenarios with different levels of observability of targets in the passive sensor network. Tracking of multi-target in sensor networks is an important but challenging task for radar, sonar and other surveillance systems . Most trIn general, sensor selection is developed based on Bayesian multi-target filtering. Mahler’s Finite set statistics (FISST) provides an elegant Bayesian filtering formulation based on random finite set (RFS) theory and has attracted a large amount of attention in recent years . The proSeveral sensor-management solutions have been recently formulated within the FISST multi-target filtering. In general, an objective function is developed as the sensor-management criterion and is used to quantify the effect of each admissible command. A common approach is to measure the information gained via updating the predicted density and a typical objective function is the Rényi divergence between the predicted and updated density. In , the RénThis paper formulates the sensor management problem as a multi-objective optimization problem with two different objective functions, to improve the tracking performance while maintaining low computation cost. The CBMeMBer recursion is used as the multi-target filter because its SMC implementation accommodates non-linear non-Gaussian models and allows reliable and inexpensive extraction of state estimates. In contrast, the PHD and CPHD recursions require clustering to extract state estimates from the particle population, which is expensive and unreliable. Another advantage of the SMC implementation of the CBMeMBer recursion is that it allows the state-dependent probability of detection, which makes the filter applicable to more general detection profiles. At each time step, the multi-Bernoulli parameter set for the updated multi-target density is formed by the union of the multi-Bernoulli parameter sets for the legacy tracks and measurement-updated tracks. Nevertheless, the legacy track and the measurement-updated track are updated in different manners and have different physical meaning. To the best of the authors’ knowledge, there is no literature considering the legacy tracks and measurement-updated tracks separately in solving the sensor management problem. In this paper, the objective functions are developed as the mean cardinality of the measurement-updated tracks and the cardinality variance of the legacy tracks, both of which can be computed analytically. Maximizing the mean cardinality of the measurement-updated tracks while minimizing the cardinality variance of the legacy tracks is conflicting and is modelled as a multi-objective optimization problem. The multi-objective problem gives a set of sensor selection solutions that reveal different trade-offs between conflicting objectives. The preferred solution–the one most desirable to the decision maker or the designer–is selected from the Pareto optimal set . In the The paper is organized as follows. In k, there are An RFS is intuitively a random  point pattern, in which the number of points is random and the points themselves are random and unordered. In multi-target systems, not only does the number of targets vary randomly in time, but also the measurements are subject to missed detections and false alarms. This means that the set of target states and of measurements are both inherently RFSs. Suppose that at time k. The multi-target Bayes recursion propagates The multi-target filtering problem can be formulated in the Bayesian filtering framework. Let r of being a singleton distributed according to a probability density r, while its probability density is  and the measurement-updated tracks [the second term in (11)]. Moreover, the legacy tracks and measurement-updated tracks are updated in different manners and have different physical meaning. Thus, the sensor selection in this paper is posed as a multi-objective optimization problem which consists of two objective functions corresponding to the legacy tracks and measurement-updated tracks, respectively. Assume that the updated multi-target density for sensor k, the mean cardinality of the measurement-updated tracks for sensor s is defined asThe first objective function is the mean cardinality of the measurement-updated tracks. Maximizing of this function has the effect of maximizing the number of detected targets and also has the side-effect of improving the localization accuracy of such target as time goes on. At time Some methods have been proposed for maximizing the number of detected targets in sensor management using the PHD and CPHD filters. In ,26 the ps at time k is given byOn the other hand, the second objective function, the cardinality variance of the legacy tracks, aims to improve the accuracy of the number of legacy tracks. The cardinality variance of the legacy tracks for sensor The cardinality variance is a measure of the uncertainty embedded in the estimated tracks. Therefore, we propose to use the statistical cardinality variance of the legacy tracks as the objective function, which tends to minimize the error in the number of the legacy tracks. Both the mean cardinality and the cardinality variance can be computed analytically and cheaply. Therefore, these two objective functions can provide an efficient solution for sensor management.The objective functions are developed with parameters of the updated multi-Bernoulli distribution. However, the updated distribution depends on the measurement set, which is a function of the chosen sensor-management command. Theoretically, all possible measurement sets should be used to update the distribution, which is computationally expensive. To reduce the computational complexity, the predicted ideal measurement set (PIMS) approach  is used.k, the mean cardinality of the posterior density (given by (20)) and its variance for sensor s are defined asTo distinguish the proposed objective functions from the mean cardinality of the posterior density and its variance, we also give the definitions and notations for the latter. At time Example 1: For sensors For Example 2: For sensors For k, the objectives for sensor This paper intends to account for both the mean cardinality of the measurement-updated tracks and the cardinality variance of the legacy tracks. Maximizing the mean cardinality The mathematical description of the multi-objective optimization problem is given asThe utopian solution is the solution that is optimal for all objectives. However, it does not generally exist since s convex . In fact, there is a set of solutions for the multi-objective optimization problem, which represent different trade-offs between the objectives. A solution  only if (28)∀i∈{A solution is Pareto optimal if and only if there does not exist another solution that dominates it. The set of all Pareto optimal solutions is called the Pareto optimal set. The corresponding objective vectors form the Pareto front. It is necessary to select one particular solution from the Pareto-front which can yield the desired trade-off between the conflicting objectives. We take a simple strategy and use the Euclidean distance to find the compromise solution . The utoThe point in the Pareto optimal set which is closest to the utopia point is regarded as the compromise solution. The shortest Euclidean distance of a point In this paper, the number of sensors to be selected is assumed known a priori and hence the generation of the Pareto front is straightforward. x–y plane as shown in In this section, we demonstrate the performance of the proposed MVO method with two scenarios and compare it with the Rényi divergence based method of  and the  follows (32)pD and bearing measurements are used to guarantee full observability of the targets. In this case, the measurement is informative and hence only one receiver is selected at each time step. Scenario 2 is used to compare the performance of MVO method with other sensor management methods using the bistatic range only measurement. In this case, more than one receiver is selected at each time step to avoid the “ghost particles” caused by the high nonlinearity of the measurement model. For the selected receives, the posterior density is updated sequentially. For more details of the multi-Bernoulli filter with sequential update scheme, the reader is referred to . The geoAt each time step, distance . If X={xIn this scenario, the non-linear bearing and bistatic range measurements are used to track a time varying number of targets. A nearly constant turn model is considered. The target state is denoted as x and y and meters per second for For the CBMeMBer filter, the birth process is a multi-Bernoulli RFS d method  and the p = 1, c = 300 m) and the cardinality estimate are given in The MC averages of the OSPA distance  of the MVO method, the cardinality variance based method and the Rényi detergency based method for S = 2, 3. Due to the lack of observability of the full states, it is difficult to resolve the targets. To avoid this problem and track the multiple targets, the birth multi-Bernoulli RFSs are generated using the initial target states. Therefore, it can be seen that the OSPA distances of all methods are on a rising trend. k can effectively reduce the OSPA distances. Moreover, it appears that the MVO method outperforms other methods. Compared with Scenario 1, the cardinality variance based method performs better in Scenario 2. Thus, the “observability” of the targets and the target motions can be crucial factors in determining the performance of the sensor management method. We verify the performance of the MVO method using different A sensor-selection solution has been proposed to be employed within a multi-Bernoulli multi-target filter. At each step, the next sensor-selection command is chosen by solving a multi-objective problem. It’s easy to see that each objective function can be computed analytically, enabling a principled and efficient solution for the sensor selection problem. The performance of the proposed method was demonstrated in two challenging scenarios where a passive sensor network with low detection probability is used to track targets with different levels of observability. Simulation results demonstrated that the proposed sensor selection method performs better than the competing methods. Future work will apply the proposed multi-objective methodology to other more advanced filters such as the generalized multi-Bernoulli filter.


The process of direct cell reprogramming, also named transdifferentiation, permits for the conversion of one mature cell type directly into another, without returning to a dedifferentiated state. This makes direct reprogramming a promising approach for the development of several cellular and tissue engineering therapies. To achieve the change in the cell identity, direct reprogramming requires an arsenal of tools that combine experimental and computational techniques. In the recent years, several methods of transdifferentiation have been developed. In this review, we will introduce the concept of direct cell reprogramming and its background, and cover the recent developments in the experimental and computational prediction techniques with their applications. We also discuss the challenges of translating this technology to clinical setting, accompanied with potential solutions. Xenopus laevis that the differentiated cells preserve their complete genetic material and it led to the understanding that the state of a fully differentiated cell could be reverted back to a dedifferentiated state [The epigenetic landscape model proposed by Conrad Waddington in 1957 provided a framework to explain cellular differentiation through epigenetic changes rather than genetic inheritance [ed state . For theed state ,4,5. Howed state .Traditionally, the conversion of somatic cells from one specific type to another relied on a successful conversion to an iPSCs which uses the mechanism of epigenetic regulations to remodel somatic cells by resetting its chromatic structure and the methylation states of histone proteins and DNA . In the The iPSCs possess similar properties of an embryonic stem cells (ES) which can differentiate into any other cell type. This allowed the possibility of using iPSCs as well as ES for various clinical applications including cell-based therapy, tissue repair, degenerative diseases, aging and cancer ,17,18,19Despite their potential for clinical applications, the use of iPSCs have raised various concerns including the cost, low efficiency and the duration of conversion due to complex conversion protocols. In addition, the use of iPSC technology in human cell therapy is tightly regulated due to the risk of genetic abnormalities, tumorigenicity, and immunogenicity in the transplanted cells . These dIn order to address the issues related to iPSCs, direct cell reprogramming methods were developed. They avoid the need for a pluripotent state while converting a functional cell type from one lineage to another lineage . One of Recent advances in the sequencing technologies and the availability of wealth of data on gene expression profiles of various cell types and good quality biological networks have led to the development of computational prediction methods that complement the experimental technique of direct cell reprogramming. Few computational methods have been developed to predict the key transcription factors required to achieve direct cell conversion ,25,26,27Here, we review various experimental and computational techniques used to achieve direct cell reprogramming and we elaborate on their applications in tissue engineering and regenerative medicine. We also discuss the current challenges and solution to move this technology to clinics. Overview of the techniques, advancement and applications of direct cell reprogramming covered in this review are illustrated in In vitro transdifferentiation in mice and humans can be guided both by expressing specific transcription factor(s) which are known to play key roles in the maintenance and regulation of the cell state and functions . In ordeCell transfection is a method that allows for the expression of either exogenic DNA or RNA in a host cell and can be either transient or stable. Transient transfection allows the exogenic DNA or RNA to be expressed without needing to be integrated into the host genome. This can be achieved using various non-viral techniques including liposome and non-liposome mediated transfection and dendrimer-based transfection and electroporation. In liposome-mediated transfection, which as the name states, depends on the transient DNA or RNA to be encapsulated by the liposome for entry into the cell’s cytoplasm . In non-Unlike transient transfection, stable transfection allows for the exogenic material to be inserted into the host genome and thus expressed permanently. To achieve the latter, techniques like microinjection, which works by injecting foreign DNA or RNA in a cell of choice , and virViral vectors can be of two types: Integrating and non-integrating. Upon recognition by the specific receptors on the cell surface, the content of the viral capsid is injected into the cytoplasm of the host cell, from there the viral genetic material can either be translocated into the nucleus to be integrated in the host genome permanently in the case of integrating viruses, or remain in the cytoplasm in the case of non-integrating viral systems . The choAll of the above viral systems are designed to take up the edited viral genome alongside the needed inserts and markers permitting for the monitoring, quantification, and validation of the change of cell fate during a transdifferentiation process. Perhaps the most interesting inserts currently used in the field of direct reprogramming are the genes coding for transcription factors, these will be discussed in the next section of this review.Transcription factors have been shown to be the tool of choice to be overexpressed in the starting cell type to achieve the change of cell fate. In 2012, Heinrich and colleagues succeeded in converting astrocytes into GABAergic or glutamatergic neurons , while KFibroblast are probably the most popular source cell for the process of transdifferentiation. Kajimura and colleagues converted fibroblasts into functional adipocytes , while bUnlike in-vitro transdifferentiation, the in vivo conditions favor the natural environment of the cells and eliminates the complex process of cell culture and cell transplantation. In vivo, it was demonstrated that the expression of Gata4, Mef2c, and Tbx5 can convert mouse cardiac fibroblasts into functional cardiomyocyte-like cells , whereasPancreatic exocrine cells were also converted into β-cells in vivo , Sox9+ cTechniques based on chemical modification and the use of small molecules have revolutionized the cell replacement therapies and drug screening. Compared to other methods, small molecules bring the advantage of structural versatility, relatively cheap to upscale the production and fairly easy to control dosage and timing.Histone deacetylase inhibitors and DNA methyltransferase inhibitors were the first to be used to facilitate the generation of iPSCs . Later, In 2010, Oct4 in combination with a selection of chemical compounds were used to transdifferentiate somatic cells into iPSCs ,73 this The use of small molecules during the process of transdifferentiation reduces the risk of genetic manipulation when used under clinical setting. Small molecules have been shown to regulate the target gene expression profiles using various mechanisms elegantly reviewed by Yu and colleagues .In 2015, the successful transdifferentiation of fibroblasts into murine and human neurons using only chemical compounds have been reported ,76. Li eIn 2015, the direct conversion of fibroblast to cardiomyocytes was done using a combination of only chemical compounds  . In 2016The process of direct cell reprogramming involving various epigenetic mechanisms are yet to be understood. The major epigenetic modification that regulate gene expression profiles includes histone modifications, chromatic remodeling, DNA methylation and ncRNAs . The chaHistone modifications play a major role during cell fate and conversion by regulating the expression of certain key genes and can either act as a barriers or stimulators during the process of transdifferentiation . This reIn 2013, Barneda-Zahonero et al. demonstrated that the expression of histone deacetylase HDAC7 was induced in pre-B cells but reduced during the conversion of these cells into macrophages . This waUsing small molecules targeting histone modification and signaling pathways were also shown to induce the process of transdifferentiation. In 2013, Bramswig and colleagues, demonstrated that controlling the histone methylation levels using an unspecific histone methyltransferase inhibitor Adox was capable of converting pancreatic α cell into β cell .MicroRNAs (miRNAs) are small non-coding RNA molecules composed of 18 to 25 nucleotides known to regulate the expression of several genes . They arIn 2011, Yoo and colleagues transfected fibroblasts with mir-9/9* and mir-124 and observed the expression of neuronal markers 4 weeks after . HoweverIn 2012, Jayawardena and colleagues used a cocktail of miRNAs consisting of mir-1, 133, 208, and 409, to convert fibroblasts into myocardial cells . This waRibosomes are complex biological macromolecule that contain a combination of several ribosomal proteins and RNAs ,95. RiboThe study of lactic acid bacteria (LAB) in the conversion of human dermal fibroblasts (HDFs)  have ledThe idea that cell reprogramming can be promoted using proteins is not new. In theory, cell extracts comprise the nuclear regulatory components which are needed to drive cell fate change.In 2002 and 2003, two group have shown that the transdifferentiation of adult cells was possible by incubating them with extracts of either primary or transformed T-cells, or neuronal progenitor cells ,100,101.C-terminal fused undeca-arginine (11R) have also been routinely used for purpose of transdifferentiation. Erythroblastosis virus E26 oncogene homolog 2 and mesoderm posterior 1 fusion proteins have been shown to successfully reprogram HDFs into cardiac progenitors .The conversion of human adipose-derived stem cells (ADSCs) into corneal endothelia (CE)-like cells was carried out with the help of recombinant TAT-transcription factors and small molecules ,107. TheCell differentiation can be controlled via a small number of ectopic factors including TFs, micro RNAs, epigenetic re-modelers , growth Most methods consider a cell-type specific gene expression profile as a representation of cell identity when predicting which TFs are required to acquire the desirable cell. For example, D’Alessio et al. utilized massive microarray dataset for ~200 cell and tissue types primarily from the Human Body Index collection from GEO , which is one of the largest repositories of gene expression . They hyAlthough gene expression alone provides rich information on cell identity, genes do not work independently. Therefore, the transcriptional impact of TFs should also be estimated in the context of gene regulatory network (GRN). In Mogrify and CellNet algorithms, they used GRN information in addition to the gene expression profile.p-value and log fold change from the comparison between target and background cell-types. The background cell-types were selected based on the FANTOM5 cell-type ontology tree for each target cell. The other type of score was the network score which was computed as the weighted sum of differential expression scores of all interacting genes with a TF in the network. Each differential expression score in the network was weighted according to the number of co-regulated genes  and the distance from the TF of interest (to consider that direct regulation is more effective than indirect). Mogrify outputs the highly ranked TFs after filtering out TFs already expressed in the starting cell type and TFs redundant in their influence in the network to get maximized regulatory coverage [Mogrify analyzed CAGE-seq  data for ~300 cell and tissues from FANTOM5 consortium  and incocoverage .In the case of CellNet, it was originally designed to work with microarray data for ~20 cell-types ,111 but Recent advances in DNA sequencing technology allowed us more complex modelling using diverse genetic and epigenetic information. For example, Ronquist et al. incorporated time-series fibroblast RNA-seq and TAD  information from 3-dimensional DNA interacting capturing (Hi-C seq) data . They coThis sophisticated algorithm is applicable only when the relevant data for cell-type of interest is available such as the synchronized time series expression and DNase-seq data. The more complex data for a wide range of cell-types become available, the more advanced algorithm would be applied to the prediction for many cell-types.In addition, most algorithms so far have been applied to gene expression from bulk cell populations or tissues instead of using homogeneous population of a single cell type. Thus, employing single-cell RNA-seq is needed to solve the problem of tissue heterogeneity. This achievement will not only increase the possibility of discovering new cell conversion, but also provide a deeper insight into how the direct cell reprogramming process is controlled.Direct reprogramming of mature cells from one state to another brings with it a tremendous potential for using this technology in various fields. One of the most important application of transdifferentiation is tissue engineering.In 2011, Huang P et al., have successfully reprogrammed hepatocytes to regenerate the liver in mice . In 2012Aside from tissue engineering, regenerative medicine is the next field to benefit the most of direct reprogramming technologies. The three main barriers of the natural cell that call for the use of transdifferentiation are that (i) the target cell type does not proliferate as fast and efficient as required, (ii) are rare in the body, (iii) or are hard to regenerate. Perhaps neuronal cells are the most attractive cell type to be targeted by transdifferentiation technology as they fit in all the above categories.The use of lentivirus poses the main issue with translating reprogrammed neuronal cells to the clinic. All neurodegenerative diseases are characterized by defects in neuronal or glial cells, which lead to condition like strokes and Parkinson’s diseases. The latter is characterized by the death or defects in dopamine-producing neurons in the brain leading to a drop in the dopamine levels and thus an abnormal activity of the brain ,124.In 2017, Tang and colleagues transdifferentiated Stroli cells into dopaminergic neurons showing high functionality and a therapeutic potential. The transplanted cells in Parkinson’s mouse model showed reduction in the size of the lesion while promoting the recovery of the fine motor and sensory neurons .Despite promising technological advancements in direct reprogramming, several major difficulties and challenges still exist in translating this technology to clinical applications. The most evident issue is the use of lentiviruses that has potential risk of mutagenesis . These mKeeping in mind that the transdifferentiated cells may not be capable of mimicking the morphology of the target cell, their inability to perform their functions remain a major concern. Although in vitro experimentations allow for the study of the major characteristics of the reprogrammed cell, in vivo assays are compulsory to fully characterize the reprogrammed cells in their physiological setting. Tight regulation of in vivo assays along with extensive animal testing is key to the translation of direct cell reprogramming to clinical applications.Finally, the efficiency of transdifferentiation remains a challenge. Low conversion rates mean that a longer period of time is needed before the desired yield of the reprogrammed target cells for any clinical application is reached. Thus, improving the efficiency and the yield are vital for making transdifferentiation favorable in clinical setting which are usually time-sensitive.The above can be overcome via a myriad of methods including the optimization of biochemical , biophys


A fire detection system requires accurate and fast mechanisms to make the right decision in a fire situation. Since most commercial fire detection systems use a simple sensor, their fire recognition accuracy is deficient because of the limitations of the detection capability of the sensor. Existing proposals, which use rule-based algorithms or image-based machine learning can hardly adapt to the changes in the environment because of their static features. Since the legacy fire detection systems and network services do not guarantee data transfer latency, the required need for promptness is unmet. In this paper, we propose a new fire detection system with a multifunctional artificial intelligence framework and a data transfer delay minimization mechanism for the safety of smart cities. The framework includes a set of multiple machine learning algorithms and an adaptive fuzzy algorithm. In addition, Direct-MQTT based on SDN is introduced to solve the traffic concentration problems of the traditional MQTT. We verify the performance of the proposed system in terms of accuracy and delay time and found a fire detection accuracy of over 95%. The end-to-end delay, which comprises the transfer and decision delays, is reduced by an average of 72%. Dependable fire detection systems with high accuracy and promptness are essential for the safety of smart city services. Since there are so many false alarms in legacy fire detection systems, they occasionally make system operators take risky actions such as turning off low-precision fire detection systems . ImproviCommercial fire detection systems generally have a simple sensor with low accuracy and are sensitive to sensor failure and malfunction which makes it hard to detect fires . In addiThe existing fire detection systems use rule-based algorithms with static parameters that make it difficult to accurately detect fire in dynamic situations . Fire deIn general, artificial intelligence (AI) is suitable for identifying situations that are difficult to classify using the features extracted from the sensing data . For exaCombustible materials and a delayed reaction time are the main causes of fire spread, accounting for 40% and 28% of all fires, respectively . Prompt Recent studies on advanced fire detection systems have been made based on IoT middlewares ,18,19, sImproved speed of fire detection by minimizing the delay of data transmission by deleting the queuing delay that occurs in the central broker when using the MQTT protocol.Combined and analyzed complex fire sensor data using multi-functional AI framework to improve fire detection accuracy.In this paper, we propose a dependable fire detection system using a multifunctional AI framework which interworks a novel Direct-MQTT. The main features proposed in this paper are as follows: This paper is divided as follows: in Many studies have been conducted to combine fire detection systems with ML algorithms to analyze the images and sensor data and obtain accurate results. Shen proposed a study of flame detection using the YOLO framework . YOLO isRecently, various algorithms have used statistical techniques for data analysis. Algorithms can have different characteristics depending on the calculation formula, and the data analysis result may vary depending on which algorithm is used. There is an ML algorithm for analyzing data. ML algorithms can be divided into shallow learning, such as Support Vector Machine (SVM), decision trees, and K-Means, as well as deep learning, which uses neural network layers such as a Deep Neural Network (DNN) and CNN ,32,33,34Many types of IoT middleware have been studied for their ability to connect various devices and transmit data in large-scale IoT systems. MQTT, CoAP, HTTP, XMPP are representative IoT middleware protocols ,38,39,40The structure of the proposed system is shown in The fire detection system may be constructed as a large-scale system that collects data from multiple IoT gateways. The IoT gateways send the collected data from heterogeneous sensors, which include temperature, humidity, gas sensors, and cameras to a multifunctional AI framework that identifies the fire. The multifunctional AI framework uses multiple machine learning to analyze the heterogeneous data to calculate the fire probability for each sensor. The multiple ML algorithms include a CNN analyzing the fire image data from the camera, and a DNN processing the time series data from heterogeneous sensors. The fire probabilities for each sensor measured through MLs are combined into a single fire probability by using an adaptive fuzzy algorithm, which modifies the dynamic fuzzy factor according to the multiple ML results to infer the fire probability suitable for the environment where the system is installed. The fire detection system transmits data using the MQTT protocol, which is a broker-based middleware that can create a bottleneck problem and cause data transfer delays. The SDN controller minimizes this data transfer delay by solving the bottleneck problem with the data transmission path distribution mechanism. The SDN controller makes the direct paths between the fire detection server and the IoT gateways.The multifunctional AI framework is designed to provide context adaptability, high accuracy of fire detection, and rapid decision-making by combining various AI technologies. It can accurately analyze a variety of data types, integrate the analyzed data as required by the system, flexibly adapt to changes in time and space. As mentioned above, the multifunctional AI framework consists of a CNN algorithm, a DNN algorithm, and an adaptive fuzzy algorithm. The framework has three blocks: an IoT data collection block, a context preprocessing block, and a context decision block. The overall structure of the multifunctional AI framework is shown in The IoT data collection block subscribes to the MQTT session and continuously gathers heterogeneous data from the IoT gateway. In addition, it classifies and feeds the data to the context preprocessing block along with each data type. All MQTT data is stored in the database and used as input data for the sequential data analysis modules through the data preprocessing module.The context preprocessing block has multiple ML algorithms that can be replaced with others depending on the input data types. It combines the image data analysis module with CNN and the sequential data analysis module with DNN. The multiple ML uses the ensemble method to more effectively analyze heterogeneous sensor data than a single ML would. The context decision block uses an adaptive fuzzy module to fuse the outputs from the context preprocessing block in order to calculate the exact probability of the fire. In addition, it can simultaneously reconfigure the fuzzy factors depending on the context. The dynamic fuzzy factors automatically change to deduce appropriate results for the target environment. We will perform experiments to compare the proposed adaptive fuzzy algorithm and other ML algorithms that may be used in the context decision block instead of the adaptive fuzzy module. The image recognition algorithm using CNN has proved its effectiveness and is used in various industrial fields, including fire detection services ,45. In tThe CNN algorithm learns two classes of fire situations and warning situations. A fire situation is determined when an actual fire is detected in the image, and a warning situation is obtained when a fire risk is recognized in the image. For example, a lighter flame does not mean an actual fire but just the probability of a fire risk. There is a score for each class, and the result is the class with the highest score. If the scores of both classes are all lower, the algorithm judges the situation as normal. To compensate for the blind-spot problem of image sensors, we use a heterogeneous sensor and the DNN algorithm to analyze the sensor data . The inpThese graphs show a temperature change over 1 min. The fire situation is a graphical representation of actual fire data, which shows a sudden change of more than 20 °C per minute. The normal data show almost constant graph movement and the abnormal situation shows abnormal graph movement over a short time. The training number and accuracy of the DNN algorithm are shown in Finally, the DNN algorithm derives the fire, warning, and normal situations. The warning situation shows only a short change in the data. The system decides if this situation is a sensor failure or a spark ignition around the sensor. Since it returns to normal data after a short period, it is not considered a fire situation, and only a warning message is sent for confirmation. The results of the fire situation and the normal situation are used in the fuzzy algorithm to calculate the final fire probability, which is identical to the CNN algorithm.Though we can identify the fire, warning, and normal situations in each CNN and DNN algorithm, we also use a fuzzy algorithm by fusing the result of sensor and image data to obtain a more accurate fire probability . In partThe fuzzy algorithm uses a membership function that corresponds to the fuzzy set to obtain the membership grade of the fuzzy rule. Each member function can be expressed as a graph according to its argument value. The membership function of the general fuzzy algorithm is divided into probability based on the data on the horizontal axis of the graph. The fuzzy algorithm that finds the grade of membership based on a fixed threshold is difficult to apply to a variety of environments . If the In general, the fuzzy factor of the adaptive fuzzy algorithm is updated to adapt to situation change on the basis of the suitable formula . AlthougWhen the fuzzy factor changes, the membership function of each fuzzy set changes according to the change in the target environment. Therefore, accurate fire probabilities can be calculated through dynamic criteria. For example, let us suppose that the fuzzy factor of the temperature membership function is 40. When the average temperature of the environment is 10 °C, the membership function of the existing fuzzy algorithm does not change. However, the adaptive fuzzy algorithm membership function is changed to 10 °C by changing the fuzzy factor. In this case, when a fire occurs and the temperature change suddenly increases to 30 °C, the existing fuzzy algorithm has a low fire probability because the fuzzy factor is 40 °C, but the probability of the adaptive fuzzy algorithm is high. Thus, the adaptive fuzzy algorithm enables precise and speedy fire measurement in a variety of environments.The fire detection delay time can be divided into the decision delay and the transfer delay. The decision delay is the computation time for the calculation of the fire probability depending on the complexity of the fire detection algorithm and the computing power of the fire detection server. The transfer delay includes processing delay, transmission delay, propagation delay and queuing delay on SDN switches. In addition, the queuing delay on a MQTT broker is an important consideration for time critical IoT services. The standard MQTT protocol is based on a broker . All pubtotal_delayT) is calculated as in (1): Since the value of The SDN controller adds an extended MQTT block to distribute the flow of data during the routing process. The standard MQTT protocols transmit data using topics. Publishers and subscribers are connected through the broker and transfer messages. The publisher sends the topic that contains data to the broker, and the subscriber receives the topic from the broker. The broker stores the subscriber’s IP address and information on their topic subscription. The SDN controller collects the information on the subscribers stored in the broker. The SDN controller determines the flow of all MQTT packets based on their information to prevent transfer delay. The transmission process of each packet is shown in Publishers and subscribers are connected through the broker and transfer messages. The subscriber sends a subscribe message to the broker to notify it of the subscription topic. The broker sends the subscription topic and IP address of the subscriber to the SDN controller, which creates the subscriber information table in the topic management module of the extended MQTT block. The publisher sends the collected data to the broker and the SDN switch sends the first incoming data to the SDN controller for routing. The MQTT Packet Receiver Module in the extended MQTT block checks the MQTT packet and sends it to the topic-matching module, which matches the topic of the packet information with the topic on the subscriber information table. After the topic is matched, the emergency data recognition module identifies whether it is emergency data. The dest_ip switch module changes destination address of the packet and the SDN controller sets its path. Finally, the packet setting module sends the data of the set route and the destination address to the SDN switch, which determines the flow of the MQTT packet through the configuration data transmitted from the SDN controller. Since the changed destination address is the subscriber’s IP address, the data sent from the publisher is delivered to the subscriber instead of the broker. As a result, traffic distribution minimizes data transfer delays by eliminating queuing delays in the broker.An example of the traffic flow of the proposed MQTT protocol is shown in In this section, we perform fire detection and data transfer delay tests to verify the MAI-FDS. In order to verify the accuracy and speed of the algorithm, we compare MAI-FDS and EFDS fire probability graphs. The data transfer delay tests measure the latency of standard MQTT, CoAP, and Direct-MQTT in a complex network environment.Actual fire data are required to verify the accuracy and promptness of MAI-FDS. Experiments with real fires are limited because of costs and accident risks. In addition, several environmental variables make it impossible to generate identical fire effects. To solve this problem, we used a virtual environment that repeatedly reproduced the characteristics of an actual fire. The Fire Dynamics Simulator (FDS) , which iIn order to verify the speed of the fire decision algorithms used in MAI-FDS, three algorithms were compared: the temperature threshold algorithm, the CNN-based algorithm, and the early fire detection system (EFDS) . The firThe experiment was carried out over six minutes. The DNN and fuzzy algorithms set an adaptive threshold based on five minutes of data. MAI-FDS calculates the current fire probability by combining newly collected data and accumulated 5-min data. In this paper, we did not consider a situation where a sensor fails or the system is turned off and cannot detect a fire. We first verify the speed of fire detection with the assumption that the sensor and system will operate normally. The fire detection time of the temperature threshold algorithm is the slowest due to limitations of the detection capability of the sensor because it only analyzes temperature data. The CNN algorithm is fast at initial detection, but because of image quality limit, it takes a long time for the probability of fire to rise to a certain level. The EFDS using both temperature, humidity, and gas sensor data is similar to MAI-FDS. However, MAI-FDS recognizes fire by 6 s faster than the EFDS algorithm when the fire detection threshold is set to 40%. By increasing the fire threshold to 90%, MAI-FDS is 16 s faster than the CNN.The difference in fire detection time between the other algorithms and MAI-FDS occurs in the processing of sensor data and adaptive fuzzy algorithms. Unlike algorithms that use static rules, MAI-FDS uses z deep-learning algorithm and can train changes in temperature and change the threshold value of the fire decision. When calculating fire probability using temperature data, the temperature threshold algorithm does not show a high probability until it exceeds 50 °C. MAI-FDS shows a high probability of fire if the temperature sharply increases even before it reaches 50 °C. In In order to verify the fire recognition flexibility of the adaptive fuzzy algorithm, we compare fire probabilities of various algorithms using the restaurant data during cooking activities. The temperature, humidity and gas data of a restaurant steeply fluctuate as the cooking process. When the cooking begins, all sensor data will rise. Sometimes the humidity and gas data show opposite results depending on the item being cooked. The identification of fire in a kitchen can be made by the changing data in a short time. Gas data can increase drastically under cooking, but temperature and humidity data are plotted with a gentle slope. If a fire actually occurs, all the sensing data including temperature, humidity and gas will show corrupt changes in a specific period. The adaptive fuzzy algorithm recognizes this difference, changes the threshold value, and can decide the fire situation according to the changed threshold even when the similar patterns of temperature, humidity, and gas data are received. The input data for the experiments are temperature, humidity and gas data generated in the restaurant for 24 h. The image data are excluded from the input data because it is not affected by the adaptive fuzzy algorithm. The data assume a situation in which cooking takes place from 10:00 to 15:00, 16:00 to 22:00, and a fire occurs at 11:00. Changes in data during the cooking process show a gradual change due to kitchen ventilation. Data in a fire situation is a time when the ventilation system does not work and shows a rapid change due to an explosion of fire. The probability of fire recognition over time is shown in The experiments of the transfer delay time for HTTP, CoAP, the standard MQTT and Direct MQTT are made in terms of scalability, that is to say, the number of publishers and subscribers. The experimental environment is shown in The experiments showed that an increase in the number of publishers is based on a single subscriber. Therefore, if the broker is near the subscriber, the standard MQTT obtains a similar result to the Direct-MQTT. However, in a network configuration where there are multiple subscribers, the location of the broker cannot be set near a specific subscriber. Since the IoT system transmits data between multiple publishers and subscribers, it is necessary to check the transfer delay in multi-subscriber situations. In this experiment, we configured the network topology as the default setting. It consisted of 24 switches, a centrally located broker, and 25 publishers from 10 hosts transmitting data. The congestion level of the network was set to 5 ms. We measured the transfer delay by increasing the number of subscribers from one to seven. We assumed that all publishers transmitted the data to all subscribers.By comparing the Direct-MQTT delay times in As described in Equation (1), the end-to-end delay is the sum of the decision delay and the data transfer delay, or the time between fire occurrence and detection. In order to compare the end-to-end delay, we simulated a fire probability measurement experiment where data was collected from 750 heterogeneous sensors. The network topology of the experiment is shown in The calculated end-to-end delays of each system are similar to the measurement delays, as shown in Fire detection systems should consider two important engineering factors: fire detection probability and transfer delay. Since the existing fire detection systems use rule-based algorithms or simple AI algorithms, there are low detection accuracy problems caused by different environmental conditions, in addition to little usage of heterogeneous sensors. In addition, the legacy systems have difficulty satisfying promptness requirements because they do not consider the transfer delays of fire events caused by traffic congestion on a specific node or a server. In this paper, we proposed the usage of MAI-FDS to solve these problems. MAI-FDS adopts a multifunctional AI framework with an IoT data collection block, a context preprocessing block, and a context decision block. Each functional block in the framework has high flexibility depending on its applications. MAI-FDS showed much higher fire recognition capability than the legacy FDSs based on rules, CNN, and fuzzy methods. In addition, the adaptive fuzzy algorithm reduces the false alarms by calculating the amount of data change. In experiments using restaurant data, there were cases where static algorithms were decided as a fire at the time of cooking. MAI-FDS, which uses an adaptive fuzzy algorithm, did not deduced these as a fire at the time of cooking because the data showed no rapid change. Incorporating Direct-MQTT in MAI-FDS improved the speed of the fire detection system through its path distribution mechanism. The SDN controller supported by Direct-MQTT created direct paths between each publisher and the corresponding subscribers. This direct path mechanism effectively reduced the transfer delays by eliminating the queuing delays in the broker. As a result, MAI-FDS achieved an average fire detection accuracy of above 95% and reduced the end-to-end transfer delay by 67% compared with that of the existing fire detection systems. We believe that the research results of MAI-FDS will be an important system of future smart cities and will help prevent the spread of fires through accurate fire detection. Accurate fire detection and reduced latency of critical alarm events improve the reliability of the disaster mitigation system. In a future smart city where all infrastructures can be intelligent, a reliable disaster mitigation system will effectively support rapid firefighting measures such as optimizing the route to the fire scene. However, MAI-FDS adapts in all environments, making it difficult to accurately determine the fire. Adaptive fuzzy algorithms and multifunctional AI frameworks have been designed for environmentally adaptive fire detection systems, but the ability to detect fires in extreme environments is still weak. If a melting furnace is used in a factory, the temperature is very high and there is a continuing presence of flames, which can increase the false positive rate in MAI-FDS. Therefore, in future research, it is necessary to develop a system that can modify the fire detection algorithm and develop a multifunctional AI framework to adaptively detect the fire in the real environment.


Energy provisioning is always a crucial problem restricting the further development and application of smart industrial wireless sensor networks in smart factories. In this paper, we present that it is necessary to develop smart industrial wireless rechargeable sensor networks (SIWRSNs) in a smart factory environment. Based on the complexity and time-effectiveness of factory operations, we establish a joint optimization framework named J-EPMS to effectively coordinate the charging strategies of wireless sensors and the scheduling plans of machines running. Then, we propose a novel double chains quantum genetic algorithm with Taboo search (DCQGA-TS) for J-EPMS to obtain a suboptimal solution. The simulation results demonstrate that the DCQGA-TS algorithm can maximally ensure the continuous manufacturing and markedly shorten the total completion time of all production tasks. With the rise of Industry 4.0 and smart manufacturing, smart industrial wireless sensor networks (SIWSNs) have become a vitally important part of smart factories. Manufacturing under smart factories gives access to actively acquiring production information, real-time analyzing the production status, and intelligently executing production decisions . In ordeIndustrial practices have proved that the large-scale applications of industrial wireless sensors not only enhance the data collection capabilities of industrial sensor networks, but also accelerate the smart transformation processes of manufacturing plants . EspeciaTo implement energy replenishment in IWSNs, many researchers have made use of wireless charging technology to develop a new kind of sensor network called a Wireless Rechargeable Sensor Network (WRSN), which are considered to ultimately solve the energy provisioning problem of wireless sensor networks . In receAs far as we know, there is little research focusing on how to effectively schedule mobile chargers to replenish energy for smart industrial wireless rechargeable sensor networks (SIWRSNs) in a smart factory environment. In fact, this problem urgently needs to be solved to reliably operate a smart factory. Some scholars have highlighted the importance of the energy replenishment for industrial wireless sensor networks, but they did not extensively study the feasible charging strategies with consideration of the complex factory operations . CompareIn addition, minimizing the completion time of entire production tasks is a very important goal for factory operations in a smart factory . CorrespThe remainder of this paper is organized as follows. The rapid development of wireless charging technology provides an effective solution for the energy provisioning problem of wireless sensor networks. In order to collect the object information for a long time, wireless rechargeable sensor networks are being extensively studied in some fields, such as forest monitoring , coal miFirst of all, quantity of chargers, scheduling algorithms, and movement paths are focused on in the scheduling policies of mobile chargers. Xu et al. proposed using a single charging vehicle with separable charger array to charge a WRSN . Dai et Moreover, the layout design of sensor nodes strongly associates with sensors for transmission and charging. Jiang et al. studied a wireless charger deployment optimization (WCDO) problem . Rao et Next, in some special scenarios, researchers have developed some customized network architectures of WRSNs. Erol-Kantarci et al. proposed a sustainable wireless rechargeable sensor network (SuReSense) for long-term monitoring smart grid . Fu et aLastly, some joint optimization problems in the complex scenarios have been studied. Han et al. proposed a grid-based joint routing and charging algorithm to balance the energy problem and survival rates of sensor nodes in an industrial WRSN . Jia et In addition, manufacturing scheduling problems are extensively studied in factory operations. In a smart factory, production operations are highly dynamic and flexible, and minimizing the total completion time of all the production tasks is a very key operational objective . In ordeIn summary, with smart industrial wireless sensor networks being widely applied in smart factories, wireless charging technology provides a good way to replenish energy for SIWSNs, but how to effectively control the charging process closely depends on the actual situation. At present, most studies focus on the application of WRSNs in an open outdoor environment. In these scenarios, wireless rechargeable sensor networks are laid out in a very large space, the distribution of sensor nodes is relatively sparse, the frequency of data acquisition is relatively low, and the energy consumption of sensor nodes is relatively low. Different from the application of WRSNs in a general environment, the complexity and time-effectiveness of an industrial environment cannot be neglected. In fact, industrial wireless sensors are often densely laid out in a workshop with limited space, the frequency of data acquisition may be at the millisecond level on a working machine, and the energy consumption of sensor nodes is relatively high. At the same time, in the case of a given production plan, the associated wireless sensor failures must be prevented during the machine processing some work stage of one workpiece. Charging for the wireless sensors and processing on the associated machines must be separated from Therefore, only maximizing energy provisioning cannot meet the requirements of factory operations, and a framework of multiple constraints joint optimization is worth studying for SIWRSNs in smart factories.In this section, we briefly present an overview of the problem statement and model assumptions.In a smart factory , the proThe first problem is how to reasonably replenish energy for sensor nodes under the condition of insuring continuous machining and avoiding sensors failure, based on actively predicting the available electricity of wireless sensor nodes.The second problem is how to effectively assign production tasks to the optional machines under the constraint of operational relationships, and further minimize the processing time of the entire production tasks on the basis of separation between nodes charging and machines processing.For the purpose of accurately and reliably monitoring manufacturing equipment  in a smart factory, SIWRSNs need to make use of reasonable charging policies to ensure the high-load data acquisition process and prevent sensor nodes from stopping due to a low battery. At the same time, to avoid spatial intersecting and signal interference, charging for sensor nodes can only occur when relevant machines stop working. As a result, we must address two critical problems:Given the complexity of the practical manufacturing processes, we set up some assumptions about this problem and appropriately refine the model. For SIWRSNs, we suppose the communication is very reliable, and the communication errors and retransmissions will not be considered. In addition, because the nodes’ operation is relatively stable in a smart factory environment , the varIn a smart factory, we suppose the production tasks have some workpieces requiring to be processed on some machines . AccordiFrom the above, the detailed model assumptions are as follows:Each machine can only process one workstage at most in synchronization.Each workstage can only be processed by one machine at most in synchronization.The processing sequence of each workpiece must satisfy the requirements of production, and the different workstages of a workpiece cannot coincide.Once the workstage starts processing, it cannot be interrupted until processing is completed.The workpieces are independent from each other, there is no sequence constraint, and there is no difference between priorities.At the beginning, the initial state of all machines is idle.At the beginning, all the workpieces arrive at the factory to be processed.The transfer time between the different workstages of a workpiece is negligible.The working output current of wireless sensors is constant by default.The charging input current of wireless sensors is constant by default.When the machine is not working, wireless sensors can enter low-power mode, regardless of the energy consumption during this period.At the beginning, all the wireless sensors are at full charge.Charging for wireless sensors starts with the associated machine finishing one workstage and has stopped working.When wireless sensors are charging, the associated machine must shut down.Regardless of the capacity loss of the battery, the maximum battery capacity of the wireless sensor is constant.The wireless sensor adopts greedy charging such that it charges until it is full.The maximum battery capacity of wireless sensors is sufficient to support any workstage.The uncertainty of communication and operations on the sensor nodes is negligible.In this section, we provide the detailed implementations of Joint Energy Provisioning and Manufacturing Scheduling (J-EPMS) in smart industrial wireless rechargeable sensor networks. We first define the symbols and variables required for modeling. Second, we consider how to establish a model of J-EPMS with charging strategies and scheduling plans. Finally, a novel solution algorithm is proposed.These decision variables are defined for optimizing the charging strategies and production plans of J-EPMS as given in With the aim of building a framework of J-EPMS, the battery capacity and charging time of wireless sensors must be taken into consideration in SIWRSNs. Meanwhile, production tasks are restricted by manufacturing techniques and machines’ capabilities in a smart factory . On the First, Sequence Constraints of Workstages are presented as:The Sequence Constraints of Workstages highlight that the processing sequence of each workstage cannot be violated. For each workpiece, the latter workstage only begins after the previous one has been completed.Second, Exclusive Constraints of Workstages are presented as:The Exclusive Constraints of Workstages emphasize that each workstage of the workpieces can only be processed on one machine. In other words, each workstage of all the workpieces must be assigned to one machine.Third, Processing Time Constraints of Machines are presented as:The Processing Time Constraints of Machines highlight that the different workstages assigned to the same machine must be processed by order. The beginning time of the latter workstage must be equal or greater than the finishing time of the previous workstage on the same machine.Fourth, Deployment Constraints of Wireless Sensors are presented as:The Deployment Constraints of Wireless Sensors emphasize that each wireless sensor can only be mounted on one machine. However, a machine can hold multiple wireless sensors. In this paper, we suppose the wireless sensors have been mounted on the machines in advance, and the deployment optimization of wireless sensors will not be considered.Fifth, Battery Capacity Constraints of Wireless Sensors are presented as:The Battery Capacity Constraints of Wireless Sensors highlight that the continuous processing time of each machine cannot be greater than the maximum working time of each associated wireless sensor. The continuous processing time of a machine can be calculated by the time interval between charging for the associated wireless sensors. The maximum working time of wireless sensors can be calculated by dividing the battery capacity by the working output current.Sixth, Charging Time Constraints of Wireless Sensors are presented as:The Charging Time Constraints of Wireless Sensors emphasize that the charging time for wireless sensors must be enough to meet the requirement of greedy charging. In other words, when the wireless sensors are arranged to charge at the end of a workstage, the beginning time of the next workstage on the associated machine cannot be earlier than the finishing time of greedy charging. The time for greedy charging for wireless sensors can be calculated by dividing the remaining battery after the continuous processing by the charging input current.Finally, the objective function was:On account of the charging time for wireless sensors included in the processing time of the associated workpieces, our optimization objective is simply to minimize the maximum completion time of production tasks. In this model, a continuous timeline is considered, and the maximum completion time of production tasks can be represented by calculating the maximum completion time of all the workpieces.From the perspective of modeling, this framework of J-EPMS is essentially the joint optimization of flexible manufacturing scheduling and charging schemes for SIWRSNs. As we know, the flexible manufacturing scheduling problem has been proved to be a NP-hard problem . So the i = 1, 2, 3, …, g; j = 1, 2, 3, …, h; g is the population size; h is the size of quantum bits and equals the total number of workstages of all workpieces; ith double-chromosome; jth quantum angle of First, in DCQGA-TS, we utilized double chains quantum coding to represent the variant feasible solution of J-EPMS, and the coding scheme is described as:jth value of m denotes the total number of machines. Therefore, m.Second, we used a double-chain encoding scheme to map the double-chromosome P. For example, if we have three workpieces and each workpiece has two workstages, {1 2 1 3 2 3} is a feasible processing sequence, where the number denotes the serial number of the workpieces and the frequency of occurrence denotes the workstage of the corresponding workpiece, e.g., the first 1 denotes the first workstage of the first workpiece.For the gene chain S by comparing sine function values with r. If jth value of S is 1; otherwise is 0. On the basis of the parent individual P and binary sequence S, we could generate a new feasible processing sequence as a child individual C. If jth value of the binary sequence S, jth value of the child individual C, and jth value of the parent individual P. We got a whole child individual C by filling in C with the residual values of P in turn. In the meantime, the child individual C was regarded as the parent individual P in the next iteration. For example, if P is {1 2 1 3 2 3} and S is {1 0 0 1 1 0}, then C is {2 1 1 2 3 3}.Then we utilized a random number Third, considering the fact that the processing time of all the workpieces is definitely much greater than the charging time for SIWRSNs and the charging time of wireless sensor is extremely unstable in a smart factory, Taboo search was applied to search the suitable charging windows for low-energy wireless sensors on the basis of the decoded double-chromosome. The search strategy was to preferentially find a discontinuous time window of the workstage completion to charge the wireless sensors before the wireless sensors’ failure.θ represents the angle of rotation.Fourth, we used the quantum rotation gate to update the double-chromosome , and theThe rotation operation is:The rotation angle of each quantum bit Fifth, to increase the diversity of population, we used a Hadamard gate. The Hadamard gate is described as:Finally, to minimize the maximum completion time of production tasks, we utilized the elite retention strategy to retain the optimal double-chromosome in each iteration. The detailed procedures of DCQGA-TS for J-EPMS are given in Algorithm 1.Algorithm 1. Presents the detailed procedures of DCQGA-TS for J-EPMS.Input: the population size g, the maximum number of iterations MaxIteration, the basic rotation angle Step1: using double chains quantum coding to generate g double-chromosomes, as the initial population;Step2: using the double-chain encoding scheme to get the solution space of g double-chromosomes;Step3: using Taboo search to find the suitable charging windows for low-energy wireless sensors on the basis of the decoded individual, and calculating the maximum completion time Step4: comparing the current optimal solution with the global optimal solution, and retaining the smaller as the global optimal solution;Step5: if the number of iterations is equal to MaxIteration, then exit and output the global optimal solution; otherwise, go to Step6;Step6: calculating the rotation angle θ, and using the quantum rotation gate to update the double-chromosome;Step7: using the Hadamard gate to make the mutation of double-chromosome on the basis of the mutation probability Since the computation complexity of J-EPMS grows exponentially, we proposed the double chains quantum genetic algorithm with Taboo search (DCQGA-TS) to obtain a suboptimal solution. Some studies have proved that genetic algorithms can effectively tackle these kinds of NP-hard problems . Other sIn this section, we take a production task on multiple machine tools in a smart workshop, for example, to verify the feasibility and efficiency of the proposed DCQGA-TS for J-EPMS. It is important to be noted that these data have been collected and collated from the actual production environment. Since we mainly focus on the validity of the framework and algorithm, how to get these simulated data will not been further described.For this production task, there were four types of workpieces required to be processed on five machines, and the detailed processing time is given in g was 100; the maximum number of iterations MaxIteration was 2000; the basic rotation angle On the basis of above data, we implemented a MATLAB program for DCQGA on MATLAB 2017b. The population size In order to highlight the advantage of the proposed J-EPMS, we also implemented a MATLAB program without the constraints of wireless sensors under the same parameters. The optimization process without sensor constraints is described as For the above simulation results, we can come to the following two verifications. First, our developed framework of J-EPMS could ensure machines are not unexpectedly interrupted by wireless sensor failures. Second, our proposed algorithm of DCQGA-TS for J-EPMS could effectively shorten the maximum completion time of the entire production task. Though the two simulated scenarios are not very complex, we could verify that the joint optimization framework of J-EPMS was capable of describing the complex relationship between the charging strategies and manufacturing scheduling. At the same time, the comparison between the solved scheduling plans also confirmed that the proposed algorithm was able to obtain a suboptimal solution based on the calculation results of DCQGA-TS for J-EPMS. In order to more effectively deal with different requirements of factory operations, some complex simulations are intensively focused in our future works.In this paper, we were concerned about whether factory operators can keep machines with some wireless sensors undergoing high-intensity work for long periods of time by effectively arranging the processing and replenishing energy sequence for SIWRSNs in a smart factory environment. We developed a joint framework of J-EPMS to jointly optimize charging strategies and manufacturing scheduling for minimizing the total completion time of all the production tasks as much as possible. To verify the feasibility of providing continuous factory operations, we set up some assumptions about this problem and appropriately refined the framework. The simulated results of our proposed algorithm of DCQGA-TS for J-EPMS have demonstrated the correctness and practicability of our idea. However, some important factors are not further discussed in the framework, such as communication errors, uncertain nodes’ operations, and the capacity loss of the battery. In some smart factory contexts, the frequent communication errors and retransmissions lead to rapid power consumption, and the uncertain nodes’ operations cause the unavailability of the initial production plan. A dynamic joint optimization framework will be intensively studied in future works.In summary, we present that it is necessary to apply SIWRSNs in a smart factory environment. In order to effectively replenish energy for SIWRSNs, we have studied how to jointly optimize the charging strategies of wireless sensors and scheduling plans of machines running for minimizing the total completion time of the entire production tasks in a smart factory. We established a collaborative optimization framework of Joint Energy Provisioning and Manufacturing Scheduling (J-EPMS). In this framework, we concluded six types of constraints and an objective function. Since the theoretical optimal solution of J-EPMS is difficult to obtain, we proposed a novel double chains quantum genetic algorithm with Taboo search (DCQGA-TS). The simulation results with DCQGA-TS showed that the framework of J-EPMS can contribute to the reduction of the unpredictable interruptions by wireless sensor failures in production processes. At the same time, DCQGA-TS can decrease the maximum completion time of all the production tasks. In a word, our proposed J-EPMS and DCQGA-TS can contribute to realizing the large-scale applications of smart industrial wireless sensor networks in smart factories.