Recent studies on brain-computer interfaces (BCIs) based on the steady-state visual evoked potential (SSVEP) have demonstrated their use to control objects or generate commands in virtual reality (VR) environments. However, most SSVEP-based BCI studies performed in VR environments have adopted visual stimuli that are typically used in conventional LCD environments without considering the differences in the rendering devices (head-mounted displays (HMDs) used in the VR environments). The proximity between the visual stimuli and the eyes in HMDs can readily cause eyestrain, degrading the overall performance of SSVEP-based BCIs. Therefore, in the present study, we have tested two different types of visual stimuli—pattern-reversal checkerboard stimulus (PRCS) and grow/shrink stimulus (GSS)—on young healthy participants wearing HMDs. Preliminary experiments were conducted to investigate the visual comfort of each participant during the presentation of the visual stimuli. In subsequent online avatar control experiments, we observed considerable differences in the classification accuracy of individual participants based on the type of visual stimuli used to elicit SSVEP. Interestingly, there was a close relationship between the subjective visual comfort score and the online performance of the SSVEP-based BCI: most participants showed better classification accuracy under visual stimulus they were more comfortable with. Our experimental results suggest the importance of an appropriate visual stimulus to enhance the overall performance of the SSVEP-based BCIs in VR environments. In addition, it is expected that the appropriate visual stimulus for a certain user might be readily selected by surveying the user's visual comfort for different visual stimuli, without the need for the actual BCI experiments. Electroencephalography (EEG) has been the most widely used neural signal for brain-computer interfaces (BCIs), whose main aim is to provide the paralyzed or disabled with new means of communication with the external environment . TypicalAlthough most VR devices currently employ head-mounted displays (HMDs), no previous SSVEP-based BCI study has considered the environmental differences between the VR-HMDs and conventional LCD monitors. Since the traditional SSVEP-based BCIs have used an LCD monitor as the rendering device to present visual stimuli for the past decades, a number of studies have already been conducted on the influence of the various parameters of this visual stimulus on the performance of the BCIs; these parameters include spatial frequency , temporaIn the present study, we have used two different types of visual stimuli—PRCS and grow/shrink stimulus (GSS)—both of which are known to effectively elicit SSVEP responses in the LCD monitor environment, on 14 participants wearing HMDs. The performance of the two representative visual stimuli was then investigated in terms of individual classification accuracy and subjective visual comfort scores. After the survey of the visual comfort of the participants in the preliminary offline experiments, the performance of SSVEP-based BCIs was investigated through online avatar control experiments in a VR environment.Sixteen young, healthy people  with normal or corrected-to-normal vision participated in our experiment. All participants were informed of the details of the experiments and had given their written consent. The data of two participants were excluded in further analyses: the first was excluded owing to the frequent blinking of the eyes during the presentation of the visual stimuli  and the second owing to the nonexistence of spectral peaks in the recorded EEG. The eye blinks were identified by visually inspecting vertical electrooculogram (EOG) recorded during the offline experiment. This so-called “BCI-illiteracy” is a well-known issue in EEG-based BCIs . This exTwo different types of visual stimuli were employed to elicit SSVEP responses: a PRCS and a GSS. The PRCS is a traditional visual stimulus, which is used most frequently to elicit SSVEP responses in LCD monitor environments; this stimulus alternately presents two checkerboard patterns with 180° phase difference  Figure . The GSSIn the preliminary offline experiments, each stimulus type was presented in a randomly shuffled order to each participant. In each trial, four visual stimuli with different frequencies were presented for 4 s, as shown in https://youtu.be/TC4QMPhW6y8).In the online experiments, the participants who also participated in the preliminary offline experiments were asked to control a human full body avatar standing on a virtual road in a VR environment. The avatar could move in four directions: top, bottom, left, and right. Four visual stimuli with the frequencies used in the offline experiment were presented at the top, bottom, left, and right of the avatar to indicate the possible movement directions of the avatar . Each paz, PO3, POz, PO4, O1, Oz, and O2) using a commercial biosignal recording system . In addition, a pair of electrodes was attached above and below the right eye to acquire the vertical EOG data. The sampling rate was set at 2,048 Hz. The recorded EEG data were re-referenced to Cz [The EEG data were recorded from seven electrodes  , which hThe Wilcoxon signed-rank test was employed for the statistical analysis because the classification accuracies with respect to the two visual stimulus types did not follow normal distribution as assessed by the Kolmogorov–Smirnov test.N denotes the number of stimuli, P denotes the classification accuracy ranging from 0 to 1, and C denotes the time needed to classify a single trial [p < 0.005 for both classification accuracy and ITR for all window sizes). Although a window size of 1.5 s showed the highest ITR  for all window sizes Figures  and 3; Ile trial . Statisthest ITR , 2 s epop=0.424; Wilcoxon signed-rank test) in the online experiment although the average classification accuracy for the GSS was higher than that for the PRCS. The possible reasons for the difference between the two cases, i.e., the offline and online experiments, will be discussed in Discussion.For further analyses, all participants were divided into three groups based on the subjective visual comfort ratings for the two visual stimulus types that were obtained right after the preliminary offline experiment. The participants who were more comfortable with the PRCS were categorized as Group 1, and those who were more comfortable with the GSS were categorized as Group 2. The participants who rated both stimuli equally were categorized as Group 3 and excluded from further analyses. Interestingly, all three participants  in Group 1 exhibited higher classification accuracies for the PRCS than for the GSS, while most participants in Group 2, with the exception of only one participant , exhibited higher or equivalent classification accuracies for the GSS than for the PRCS. These results suggest that the performance of the SSVEP-based BCIs in VR environments might be potentially improved by selecting the best stimulus type for each individual, which would be readily chosen by inspecting the individual's subjective visual comfort for different visual stimuli types.The performances of the reactive BCI systems are highly dependent on the types of stimuli used to elicit specific EEG responses. Although a series of studies has been performed to find an optimal visual stimulus for the conventional SSVEP-based BCIs in the LCD monitor environment, no study has yet been reported on the influence of visual stimuli on the performance of the SSVEP-based BCIs in VR-HMD environments. We hypothesized that the PRCS, which are widely used in the SSVEP-based BCIs, might not be the optimal visual stimulus in a VR-HMD environment because the images displayed on the HMDs are closer to the eyes than those on the LCD monitors, and thus, the PRCS might be too intense for the eyes. Therefore, in this study, we tested another type of visual stimulus called the GSS that changes both size and luminance in VR environments and compared the BCI performances with the PRCS.In the offline experimental results, the GSS outperformed the PRCS in terms of classification accuracy; however, the difference in the performance was considerably reduced in the online experiments. This phenomenon is thought to originate from several factors: first, the spatial frequency of the PRCS in the offline experiment was different from that in the online experiment. The spatial frequency changed from 0.25 cycle/deg in the offline experiment to 0.5 cycle/deg in the online experiment. According to a previous report , spatialOur online experiments demonstrated that the SSVEP-based BCI with a visual stimulus that was more comfortable for the user generally outperformed that with the other stimulus in VR environment. This finding is not in line with previous reports showing that a visual stimulus evoking stronger SSVEP responses induced the severer visual fatigue –31 when In the offline experiment, four participants rated the same visual comfort score for both PRCS and GSS. Interestingly, they commonly achieved better classification accuracies in the GSS than in the PRCS. Although the limited sample size makes it hard to generalize, selecting GSS rather than PRCS might yield better classification accuracies in cases when there is no difference in the subjective visual comfort ratings. However, further investigations are required to formulate a more generalized rule for selecting the optimal visual stimulus for the SSVEP-based BCIs in VR environments. In addition, in the present study, we tested only two types of visual stimuli; however, more types of visual stimuli need to be developed and tested in VR environments in future studies.To the best of our knowledge, this is the first study that has compared different types of visual stimuli for the SSVEP-based BCIs in VR environments. Our study demonstrated that selection of an optimal visual stimulus for an individual could improve the overall performance of the SSVEP-based BCIs and reduce visual fatigue in VR environment. A close association between the performance of the SSVEP-based BCIs and subjective visual comfort was observed, suggesting that the selection of an appropriate visual stimulus via a simple pre-experimental inspection of the individual's preference toward the visual stimuli might help to enhance the performance of the SSVEP-based BCIs in VR environments.


Large-scale public datasets are vital for algorithm development in the computer vision field. Thanks to the availability of advanced sensors such as cameras, Lidar and Kinect, massive well-designed datasets created by researchers are free to the scientific and academic world. ImageNet  is the pixel coordinates of the event in 2D space and p is the polarity of the event indicating the brightness change. Compared to the conventional frame-based cameras, neuromorphic vision sensors are frameless which take a radically different approach, doing away with images completely. It properly addresses the universal drawbacks of conventional frame-based cameras, such as data redundancy, high latency and low temporal resolution in a fresh new paradigm. This sensor has matured to the point of entering commercial market only in the last decade. As a much younger field, one of the main challenges faced is the lack of neuromorphic vision datasets impeding the progress of the field. We can thus learn from the rapid development and maturation of computer vision.DVS is a novel type of neuromorphic-based vision sensor, developed by Lichtsteiner et al. . The senIt cannot be doubted that neuromorphic vision research will benefit from new datasets similar to those of computer vision. However, the unique difficulty in the datasets arises because neuromorphic vision data differs significantly from conventional camera data and no direct method for converting between two data formats exists. To address this, we introduce the largest neuromorphic vision datasets targeting the three human motion related tasks: pedestrian detection, human action recognition and human fall detection. We hope that those datasets will meet the significant demands of the neuromorphic vision, computer vision and robotic communities. More specifically, the open access of three datasets should stimulate the development of algorithms processing the event-based asynchronous stream input. In addition, to allow for a fair comparison with frame-based computer vision, we also introduce three encoding methods which are used to convert the spatio-temporal data format to conventional frames.Previously, several datasets of neuromorphic vision sensors addressing the problem of detection and classification were proposed    for accumulating each event into frames, based on which we calculated the pixel value using the following range normalization equation inspired by Chen  at pixel  within a given interval, and σ(n) is the value of this pixel in the event frame, the range of which is normalized between 0 and 255 in order to fit a 8-bit image.where SAE (Surface of Active Events)  according to the timestamp t. In this way, an grayscale image frame is acquired according to the timestamp of the most recent event at each pixel:In order to take full advantage of the unique characteristic that neuromorphic vision sensors can record the exact occurring time of incoming events with low latency, the t between the pixel value tp and the initial time t0 for each frame interval T as follows:Moreover, to attain an 8-bit single channel image, numerical mapping is conducted by calculating the ΔLIF (Leaky Integrate-and-Fire) neuron model  as a neuron with its Membrane Potential (MP) and firing counter n. The MP value of a neuron will be influenced by input spikes and time-lapse. In detail, each incoming event at pixel , regardless of polarity, will trigger a step increase of this pixel's MP value. Simultaneously, MP value of each pixel will decay at a fixed rate. When the MP value of a pixel exceeds the preset threshold which is chosen based on the effect of LIF output, a firing spike output will be generated there, and the MP value of this pixel will be reset to 0 with no latency.According to the Burkitt, , we regan). After each interval, the firing spikes counter n of each pixel will be reset to 0. The accumulated grayscale frame can thus be obtained.In a specific time interval, we count the number of the firing spike outputs for each pixel, i.e., the occurrence of events  format and 4,670 annotated frames converted from event streams of pedestrian detection. To make event data for training fit with conventional neural networks, three different event-stream encoding approaches are provided with an open-source code. It is worth noting that the neuromorphic vision sensor is a perfect sensor to solve the privacy problems which always occurs for traditional frame-based cameras . The raw data of the DVS sensor are event streams which only keep the shapes and movements of the subjects. In other words, there is no texture or appearance information recorded in our dataset. Therefore, it is impossible to identify subjects from our dataset which highlights one of the advantages of neuromorphic vision sensor over traditional frame-based cameras. We hope that these datasets can contribute to the advance of algorithms for neuromorphic vision sensor data, and further boost the development of neuromorphic vision.It is noted that a fraction of recordings for pedestrian detection are spotted with dense noisy events caused by raindrops. And illumination changes outside will result in tiny noisy events on the recordings. These phenomena indicate that appropriate approaches for filtering event data are supposed to be adopted according to the purpose of researchers. However, details in this data report as well as images shown above prove that the datasets presented here are of high fidelity and high quality, which benefit from low latency and high temporal revolution of DVS.https://github.com/MSZTY/PAFBenchmark.The datasets released in this study are available in SM, GC, XN, YZ, and KR recorded the data. GC and ZB wrote the paper. AK designed the structure of this data report.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The paper introduces an artificial neural network ensemble for decentralized control of traffic signals based on data from sensor network. According to the decentralized approach, traffic signals at each intersection are controlled independently using real-time data obtained from sensor nodes installed along traffic lanes. In the proposed ensemble, a neural network, which reflects design of signalized intersection, is combined with fully connected neural networks to enable evaluation of signal group priorities. Based on the evaluated priorities, control decisions are taken about switching traffic signals. A neuroevolution strategy is used to optimize configuration of the introduced neural network ensemble. The proposed solution was compared against state-of-the-art decentralized traffic control algorithms during extensive simulation experiments. The experiments confirmed that the proposed solution provides better results in terms of reduced vehicle delay, shorter travel time, and increased average velocity of vehicles. Properly controlled traffic signals at intersections improve utilization of the existing infrastructure, increase capacity of road network, and alleviate the congestion problem . Real-tiState-of-the-art traffic control algorithms ,5 are baAccording to the decentralized approach, traffic signals at each intersection in a road network are controlled independently by using an algorithm, which takes control decisions on the basis of real-time traffic data obtained from local measurements. These input data describe current traffic conditions at road segments connected to an intersection.Thus far, several decentralized traffic control algorithms have been proposed in the literature. The input data of those algorithms consist of parameters that describe the traffic streams passing through an intersection. Output of the algorithm is a control decision that determines which traffic stream (or streams) should get a green signal for a subsequent time interval. The consecutive control decisions are taken in constant time steps. To take control decisions, the decentralized algorithms use various forms of priorities that are assigned to the intersecting traffic streams. The priorities for traffic streams are calculated dynamically based on current traffic parameters. A traffic stream, which gets the green signal, is selected by taking into account the priority levels together with some time constraints that relate to safety and stability requirements.This paper introduces a decentralized traffic signal control strategy, which can utilize data from sensor network to evaluate priorities of traffic streams and take control decisions. The considered sensor network is composed of nodes installed along traffic lanes. Each sensor node detects presence of vehicle in a section of traffic lane, which is called a cell. The cell states can also be provided from vision based sensors, mobile devices in vehicles or its fusion. In this case, several cells can be monitored by one camera. The detection results are reported by sensor nodes to a traffic lights controller at the nearest intersection.According to the proposed approach, the priorities of traffic streams are evaluated by the controller with use of an artificial neural network ensemble. The concept of neuroevolution is adapted for training the introduced ensemble of neural networks. During evolution, the performance of the ensemble is evaluated via traffic simulation by using a fuzzy cellular automata model of signalized intersection. The fuzzy cellular automata model facilitates a fast evaluation of vehicle delay at the intersection. The delays of vehicles are considered as fitness function (cost function), which guides the evolution of neural network ensemble towards an optimal solution. Extensive experiments were performed to compare the performance of the proposed method with the decentralized traffic signal control algorithms that are available in the literature.The paper is organized as follows. In the related literature, several decentralized approaches for traffic signal control have been proposed. These approaches ensure high scalability as they do not require any central controller and do not involve communication between local control units at intersections.A simple decentralized control algorithm for self-organizing traffic lights (SOTL) is introduced in . AccordiLammer and Helbing  introducReal-world experiments reported in  confirm Back-pressure  is a decAnother decentralized control strategy for traffic signals is based on so-called virtual impulses that reflect the impact of signals and preceding cars on desired vehicle speed . This apIn , the IsiIn , a decenA multiagent system for distributed traffic signal control is proposed in . Each agDay and Bullock  investigIn , it is nRecently, a distributed traffic control system is proposed, which is based on so-called digital infochemicals . The conNilsson and Como  introducA distributed traffic control strategy based on cell-transmission model of road traffic is presented in . The autIn this study, the performance of a distributed system was analyzed, which controls the traffic signals at intersections by using ensemble of neural networks. According to the authors’ knowledge, this is the first study focused on designing a neural network ensemble to improve the performance of distributed, self-organizing traffic control in a road network with multiple signalized intersections. The experiments reported in this paper involved a comparison of the traffic control performance achieved for various neural network topologies. Based on the experiments, an ensemble of neural networks with new topology is presented, which can be effectively utilized for making control decisions by the decentralized traffic signal systems. A neuroevolution strategy  is also A new topology of neural network is proposed for aggregation of traffic data collected by sensor network. This topology reflects the design of the considered intersection. It means that the neurons and their connections correspond to configuration of traffic lanes, possible movements at the intersection and signal groups, i.e., predefined subsets of traffic streams that are controlled by identical traffic signal indications. This neural network enables adaptive aggregation of input data that describe counts of vehicles. Therefore, the aggregation scheme can be optimized during training of the neural network. For instance, the weights in neural network can be adjusted to determine that a vehicle close to the intersection has a greater impact on the control decision than a vehicle for which the distance to the intersection is large. It should be noted that the existing decentralized traffic control methods assume that the input data are aggregated in advance and the same predetermined aggregation scheme is used for all intersections.To evaluate the priorities of intersecting traffic streams, the proposed method uses a set of fully-connected neural networks (FCNNs) . The appThe proposed neural network ensemble ensures fast evaluation of the priorities for all signal groups and allows the control decision to be made in a short time interval. Thus, the traffic signals can rapidly respond to changing traffic situations. In the case of the state-of-the-art methods that utilize traffic models to predict future traffic evolution, the decision-making procedure requires long time especially for complex intersections, where many possible combinations of traffic lanes have to be considered when deciding which lanes should get green light.A general structure of the proposed neural network ensemble is presented in Input data of the proposed ensemble include detailed information from sensor network about vehicles that are present in road segments connected to an intersection as well as predicted traffic parameters (predicted occupancy of traffic lanes). Based on these data, the ensemble evaluates priorities of traffic streams.i and j are indices of the traffic lane and the cell, respectively). An example of intersection with traffic lanes divided into cells (i). Output variables of the proposed ensemble are priorities of signal groups.The input data collected by sensor network describe current states of so-called cells, i.e., traffic lane segments of equal length. Sensor nodes detect presence of vehicles in particular cells. A cell can be empty or occupied by a vehicle. These two states are defined as binary variables: The parameters (cell states and predicted occupancy of traffic lanes) were selected based on the results available in the related works, where it was demonstrated that a representation of traffic streams by cellular automaton model is suitable for applications in distributed control of traffic signals at intersections ,13. The The basic approach in state-of-the-art methods that collect data from sensor networks is to calculate the priority for a given signal by taking into account the sum on occupied cells (number of vehicles) in a traffic lane controlled by the considered signal. In contrast, the proposed data aggregation module calculates the priorities as weighted sums of the occupancy, where the weights are adjusted during training of the neural network.The data aggregation module includes a neural network with three layers . Topology of this neural network is determined by the design of the intersection. L is set of traffic lanes, M denotes set of movements, and G is set of signal groups. As mentioned above, each traffic lane (l) is divided into cells: m) are defined by pairs  of cars from inbound traffic lane  occupancy of the traffic lanes that are controlled by one signal group. Such outputs are fed into the priority evaluation module, which includes one FCNN for each signal group. The number of FCNNs implemented in this module equals The FCNN is implemented in the proposed ensemble to represent dependency between the occupancies aggregated for signal groups and priorities of these groups. It should be noted that priority of a given signal group depends not only on the occupancy of traffic lanes controlled by this particular group but also on the occupancies of the remaining traffic lanes. FCNN reproduce the aforementioned dependency between #G aggregated occupancies and #G priorities of signal groups.It should be noted here that the FCNN topology was selected for application in the proposed ensemble based on results of previous works, which have shown that this topology allows the number of neurons (and synaptic weights) to be reduced in comparison with the multilayer perceptron topology ,24. The During neuroevolution, a weight value is selected from a predetermined range for each connection between neurons in the FCNN. The allowed range of weight values for inter-neural connections in the proposed FCNN depends on the type of the connection. In the case of connections between the occupancy layer and the priority layer, the weight range is  that should indicate green signal. In the case when two or more signal groups obtain the same priority, the group that has been inactive for the longest time is selected to be active. If the active group is changed, then a setup time (consisting of inter-green period and minimum green period) has to be introduced due to safety requirements. During setup time, the priority estimation is skipped because the active group cannot be changed. Moreover, the algorithm assumes that each signal group should be activated at least once during period of T time steps. Thus, a signal group, which was inactive for more than The proposed neural network ensemble is used to estimate priorities . To train the proposed ensemble, the neuroevolution strategy described in following section was proposed.A neuroevolution approach was included in the proposed system to train the introduced neural network ensemble. The aim of the neuroevolution is to optimize the connection weights in both the aggregation and the priority evaluation module. Moreover, the neuroevolution procedure optimizes the number of neurons in the priority evaluation module.An overview of the proposed neuroevolution strategy is presented by the flow chart in At the initial stage of neuroevolution, the data describing design of the considered intersection  are utilized to determine the neural network topology as well as to configure the intersection model. The initial topology of neural network ensemble is determined according to the assumptions discussed in The intersection model is necessary in the proposed strategy for evaluation of an objective function during optimization of the synaptic weights. Details of this model, which is based on fuzzy cellular automata, are presented in After the initial topology of the neural network ensemble is determined, optimization of synaptic weights is performed for both the data aggregation module and the priority evaluation module. In this study, different soft computing methods were applied for optimization of the synaptic weights in neural network ensemble. The soft computing methods that were considered include genetic algorithm, evolutionary strategies and particle swarm optimization algorithms. The fuzzy cellular automata model was used for all aforementioned optimization approaches to evaluate the objective function (fitness function), which is defined as mean delay experienced by a vehicle at the intersection.To reduce the amount of weights that have to be optimized, it was assumed that the connections between input neurons and movement neurons in the data aggregation module have the same weights:The initial topology of neural network is stored in the memory together with the optimized synaptic weights. At the next step, topology modifications are introduced in the priority evaluation module. During single iteration of this procedure, one neuron is added in the intermediate layer of each FCNN and the optimization of synaptic weights for the priority evaluation module is executed. It should be noted that weights in the data aggregation module remain unchanged. If current modification provides a lower delay of vehicles than the previously stored configuration of the neural network ensemble, then the modified topology and weights replace the stored ones. The modification procedure ends if the current delay is not lower than the result obtained for previously stored (simpler) neural network or if the maximum number of neurons in intermediate layer is reached.The final result of the neuroevolution includes the last stored topology and synaptic weights of the neural network ensemble.The proposed neuroevolution strategy requires the objective function to be evaluated for a large set of candidate solutions in a number of iterations. In the proposed approach, the objective function is evaluated via simulation. A fuzzy cellular automata model  is used Furthermore, locations and speeds of individual vehicles are described by triangular fuzzy numbers. Location refers to cell occupied by a vehicle. Speed is expressed in cells per time step of the simulation (1 s). The locations and speeds of all vehicles in this model are updated at each time step of the traffic simulation according to rules of the cellular automata.The fuzzy cellular automata allow the objective function to be evaluated in a single simulation run and in the same time estimate the uncertainty of simulation results. This approach eliminates the necessity of time-consuming Monte Carlo simulations that use probabilistic models. The Monte Carlo method would require repeating the traffic simulation multiple times in order to reproduce the uncertainty related to random driver behavior. In contrast, the fuzzy cellular automata incorporate many potential scenarios (driving styles) in a single simulation.ith vehicle at time step t. The charts in right part of F is defined by three real numbers l, m, u) of the fuzzy numbers representing vehicle speed During traffic simulation, different cellular automata rules are applied to update components . After finishing simulation, the three components of D are divided by number of simulated vehicles to obtain the average vehicle delay.The total stop delay of vehicles is determined as triangular fuzzy number As mentioned above, the synaptic weights for the proposed neural network ensemble are optimized by means of the population-based metaheuristic methods. In this study, three different metaheuristic optimization algorithms were applied during the experiments: genetic algorithm (GA) , evolutiFor all of the considered metaheuristics, the candidate solutions play the role of individuals in a population (genomes or particles). These solutions are represented by vectors of the real-valued weights. The weights are subject to predefined lower and upper bounds. Quality of the candidate solutions is determined by objective (cost) function, which has to be minimized. The objective function for a given candidate solution is evaluated as mean stop delay experienced by vehicles at a signalized intersection, where the traffic signals are controlled according to the priorities calculated by the neural network ensemble with the synaptic weights determined by the given solution. The value of the objective function is calculated in a road traffic simulation by using an intersection model, which is based on fuzzy cellular automata. This model allows the mean vehicle delay to be evaluated in form of a fuzzy number, as discussed in A and B are some arbitrary fuzzy numbers with the membership functions given by A, and B. It should be noted that the To explain how the fuzzy numbers are compared, let us assume that A geometrical interpretation of the definition in Equation  is preseEquation  describeA is lower than fuzzy number B:Finally, the following formula is used to calculate the probability that fuzzy number A and B describe values of objective function for candidate solutions As explained. above, in the proposed optimization procedure the fuzzy numbers represent values of objective function for candidate solutions. Let the fuzzy numbers The results of the experiments discussed in this section cover total vehicle delay, average speed, and travel time that were determined for the new proposed approach as well as for the existing decentralized traffic signal control methods from the literature ,8,31. EfThe proposed neuroevolution approach to decentralized traffic signal control was experimentally evaluated in two real-world scenarios. The first scenario takes into account a signalized artery (Francuska Street) with five intersections in the centre of Katowice, Poland. This is a one-way road with multiple inlets a, which During simulations, the vehicles were randomly generated. The simulated traffic intensities were based on real-world daily traffic profiles  obtainedi is calculated at time step t by using the following formula:i predicted at previous time step, and As i explained in moothing  and k-nemoothing ,35. In ti at time step t . Moreover, it took into account the distances between intersection and the limited range of vehicle detection systems. Effectiveness of these settings was verified in preliminary experiments [The occupancy of traffic lane eriments .t and t denotes current time step, g = 1, 2, ..., k) are time points in historical data, where the nearest neighbors  have been found, and t or The second considered prediction method uses the k-NN algorithm with data segmentation. According to this method, the future occupancy of traffic lane is calculated as follows:found in .Effectiveness of the proposed approach was compared for three versions: without occupancy prediction, with occupancy prediction based on exponential smoothing, and with the k-NN occupancy prediction. In the case of the version without prediction, inputs Impact of occupancy prediction on effectiveness of the proposed approach was examined in simulation experiments. Results of these experiments for the signalized arterial road scenario are presented in During experiments, several neural networks with various topologies were used to evaluate priorities of signal groups for the decentralized traffic signal control. The following state-of-the-art topologies were considered: MLP , RBF ANNo). The results obtained for various neural networks (ANNs) are compared in o) was equally divided between those layers. It should be noted that for ANNs with one, three and four layers a lower effectiveness of the traffic control was observed than those achieved by the ANNs with two layers.The vehicle delays in k) and number of neurons (l) was modified to obtain the lowest vehicle delay. The ANN_DW/SW achieved the best results for k = 2 and l = 8. In the case of ANN_DW, the weights of connections between neurons in input layer and the first hidden layer are optimized independently for each traffic lane. ANN_SW uses the same weights vector for all traffic lanes. The ANN_DG topology includes the proposed data aggregation module with movement and group layers. In the case of ANN_DG, the weights of connections between input layer and movement layer are the same for all traffic lanes.Effectiveness of the traffic control was compared for several implementations of the decentralized algorithm that calculate the priorities of signal groups by using the proposed neural network ensemble and other ANNs with different topologies. The other considered ANNs topologies are presented in The effectiveness of traffic control was evaluated by taking into account stop delay, travel time, and average velocity of vehicles. The results presented in The evolution of the proposed neural network ensemble is performed in iterations, as explained in The effectiveness of the proposed neuroevolution-based traffic signal control strategy was compared with the existing decentralized traffic signal control methods from the literature. Three different state-of-the-art methods were taken into account in this study: the method LH proposed by Lammer and Helbing , the bacThe proposed approach proved to give superior results compared to the state-of-the-art methods. When comparing the proposed approach with the back pressure method, it can be observed that the traffic delay was reduced by 40%. In the case of the more sophisticated methods (LH and BP_es), the decrease of delay is also significant . The performance of the proposed approach depends on the type of the metaheuristic algorithm, which is used to optimize the synaptic weights during neuroevolution. The best results were achieved by using the evolutionary strategy and the genetic algorithm. Slightly worse results were obtained for the PSO algorithm.The results presented in this section describe the overall effectiveness of traffic signal control for entire road network with multiple intersections. However, detailed results obtained for particular intersections also confirm the above discussed advantages of the proposed approach. The introduced neural network ensemble and neuroevolution strategy enable appropriate evaluation of signal group priorities for the decentralized traffic control at various intersections.In this paper, a new neural network ensemble is proposed, which consists of data an aggregation module and a priority evaluation module. The priority evaluation is necessary for implementation of the decentralized traffic control strategy, which uses the priorities to decide which traffic streams at an intersection should get the green signal. The proposed ensemble can be used with various sensor networks, i.e., road side sensors or visual detection sensors, providing their data are represented as states (occupancies) of cells. The novelty of the proposed approach lies in both the design of the ANN ensemble and the neuroevolution strategy, which was introduced to optimize topology and synaptic weights of the ANNs included in the ensemble.The decentralized traffic signal control system, based on the proposed ANN ensemble and neuroevolution strategy, was experimentally evaluated in real-world scenarios. Advantages of the proposed approach were confirmed during extensive simulation experiments. The proposed solution was compared against state-of-the-art decentralized traffic control algorithms. The experiments showed that the proposed solution can achieve better results than the state-of-the-art methods in terms of reduced vehicle delay, shorter travel time, and increased average velocity. The experimental results also confirmed that a higher performance of traffic signal control can be achieved when using the predicted occupancies of traffic lanes as additional input data for the ANN ensemble. The level of performance improvement strongly depends on accuracy of the prediction method.It should also be noted that the sensing technologies, which collect information about number of vehicles on the road, would be insufficient to achieve the high performance of traffic signal control. During research, it was observed that information regarding locations of vehicles was necessary to improve the performance of signal control. This fact was confirmed by the worse results of the back-pressure algorithm. In the proposed approach, the locations of vehicles are represented by determining occupied cells of traffic lanes. This approach could be further improved by using the sensor network optimization method proposed in  to decreThe proposed approach is suitable for applications based on mobile edge computing (MEC) and fifth-generation mobile communication technology (5G) ,41. In sFuture research directions will be related to different neuroevolution strategies that could enable on-line tuning of ANN parameters during operation of the traffic control system. Another interesting topic for future investigations is introduction of dynamic synaptic weight changes during the traffic control procedure. Finally, various traffic prediction methods for input data determination in the proposed ensemble can be tested.


The forming limit curve (FLC) is used to model the onset of sheet metal instability during forming processes e.g., in the area of finite element analysis, and is usually determined by evaluation of strain distributions, derived with optical measurement systems during Nakajima tests. Current methods comprise of the standardized DIN EN ISO 12004-2 or time-dependent approaches that heuristically limit the evaluation area to a fraction of the available information and show weaknesses in the context of brittle materials without a pronounced necking phase. To address these limitations, supervised and unsupervised pattern recognition methods were introduced recently. However, these approaches are still dependent on prior knowledge, time, and localization information. This study overcomes these limitations by adopting a Siamese convolutional neural network (CNN), as a feature extractor. Suitable features are automatically learned using the extreme cases of the homogeneous and inhomogeneous forming phase in a supervised setup. Using robust Student’s t mixture models, the learned features are clustered into three distributions in an unsupervised manner that cover the complete forming process. Due to the location and time independency of the method, the knowledge learned from formed specimen up until fracture can be transferred on to other forming processes that were prematurely stopped and assessed using metallographic examinations, enabling probabilistic cluster membership assignments for each frame of the forming sequence. The generalization of the method to unseen materials is evaluated in multiple experiments, and additionally tested on an aluminum alloy AA5182, which is characterized by Portevin-LE Chatlier effects. The forming range of sheet metal is evaluated using the forming limit curve (FLC), whose limits are defined by the major and minor strain pairs at the onset of localized necking. In Europe, the procedure used for the generation of the FLC is summarized in DIN EN ISO 12004-2  setups. Metallographic investigations have shown that both conventional deep drawing steels  and dualTo address these problems and to remove the dependence on expert annotations, an unsupervised classification approach was developed (henceforth referred to as Part 2) in . SeveralOn the other hand, the conditional location dependence, i.e., the choice of the evaluation areas is still dependent on the strain maxima, which can be a disadvantage. Furthermore, the criteria used to determine the local necking are constrained by expert a priori knowledge, namely, the evaluation of HoG features, which are not necessarily applicable to other materials. Additionally, the assumption and partitioning of the forming process into a homogeneous and an inhomogeneous forming area can influence the outcome of the probabilistic FLC, if using another split criterion then approximately 50% of the sequence length.In recent years, convolutional neural networks (CNN) have found widespread use in the field of pattern recognition, and can process a large amount of data in finite time due to the computing capacity that is now available, e.g., due to powerful graphics cards. The main advantage of these approaches is the automated, data-driven learning of representative features that are adapted to the problem, such that the design of handcrafted features is no longer necessary for the evaluation and assessment of classification problems. Different CNN approaches have outperformed conventional pattern recognition methods in various challenges  and areIn contrast to Part 1 and 2, this study uses a CNN for the automatic extraction of characteristic features, for the task of FLC estimation. Here, the extreme cases of the forming process (beginning of the homogeneous and end of the inhomogeneous forming phase) are used to train a Siamese CNN  to optimThe FLC is usually determined experimentally with a Nakajima test setup, coupled with a measurement system that consists of a stereo camera, a clamping unit with an inner die diameter of 110 mm and a hemispherical punch with a diameter of 100 mm cf. . To calcIn this study, the three known materials from Part 1 and 2 are reused, while a challenging aluminum alloy is also evaluated: (1) DX54D, a deep drawing steel, with ductile necking behavior and observable localization on the surface; (2) DP800, a dual phase steel, high-strength material with a matrix of ferrite and martensite precipitations and hence multiple observable local maxima during Nakajima tests; (3) AA6014, a lightweight aluminum alloy of the 6xxx series with multiple maxima during Nakajima tests under plain strain conditions and (4) AA5182, an aluminum alloy of the 5xxx series with several local maxima in the form of shear bands (PLC effect). The principal material properties are summarized in The proposed classification approach mainly follows the typical processing pipeline used in pattern recognition solutions, even though it uses a CNN. The conventional pattern recognition pipeline consists of four steps: data acquisition, preprocessing, feature extraction, and classification. The main difference, introduced with the advent of deep learning techniques, is the data-driven feature extraction and combination with the classification step. Instead of predefining the extracted features and employing these to train and test a classifier as in conventional pattern recognition systems, the feature extraction and classification steps are combined and formulated as optimization problem. Solving this optimization problem enables estimation of a problem-specific optimal solution, e.g., by minimizing a suitable problem-specific loss function. Comparable to the conventional pattern recognition approaches, the data set is subdivided into multiple disjoint data sets. During training of the network, the training data set is used to minimize the loss function, by adjusting the weights of the layers of the CNN using a gradient descent optimizer. After the network has seen all the training data, the separation hypothesis is evaluated using the validation data set to assess if the hypothesis generalizes to unseen data or if it over-fits to the training data. This procedure is repeated until convergence or until the error in the validation data set increases, and subsequently, the performance of the network is assessed using the remaining unseen test data set. There exist supervised and unsupervised deep learning-based techniques, similar to conventional pattern recognition approaches that either require expert annotations or solve the optimization problem without the use of data (class-) labels. The framework proposed in this study employs both types of learning techniques. In the supervised part, the extreme cases of the homogeneous and inhomogeneous forming phase, are used as labels for the data set. The number of images used per material for training are found in In the unsupervised part, the unused data between the beginning of the homogeneous and the end of the inhomogeneous forming phase, as well as the extreme cases are assessed using the trained network to create low-dimensional representations. Using an unsupervised clustering approach, the low-dimensional instances of each forming sequence are then assigned to one of the three clusters: homogeneous, transition, necking. For clarification, the data sets are disjoint, meaning that e.g., the extreme cases of all but one material are used for training, and validation as well as to learn an optimal separation between the two classes and a low-dimensional representation of the data. The actual evaluation and clustering is performed using the held-out material.The study uses the data set of Part 1 & 2 and was acquired using the ARAMIS optical measuring system (v6.3.0-7), and consists of three-channel video sequences comprising of major and minor strain as well as thinning. The difference between two successive images were used as input to remove the correlation with the punch displacement and to emphasize the incremental changes. Since the necking and in general the material instabilities are localized changes in the strain distribution, the incremental changes allow a better evaluation of this phenomenon. The sequences were adjusted by removing the frames that contain the fracture information, identified at the end of the forming process with an increasing amount of defect pixels due to specimen cancellation.Single defect pixels or missing values that occurred occasionally were interpolated linearly using the temporal information, e.g., the strain progression. Other defect pixels, such as static defect pixels that contain no information during the whole forming process were replaced by the average value of a The Siamese network comprises two identical sub-networks, where each consists of the first 3 convolutional blocks of a VGG16 network  that wasWhile other loss functions usually sum over individual samples, this loss function evaluates the input samples Y refers to the extreme cases of the input sequence, where L with respect to W leads to low values of The general loss function is described as,m is a margin parameter that defines a radius around The final loss function, the contrastive loss  is denotSo far the network is trained and optimized to create low-dimensional manifolds that support the optimal separation of the extreme cases of forming sequences. Even though, the network never saw a complete forming sequence, leads to the extraction of discriminative features from complete forming sequences. Hence, it may be used to assess and cluster the individual frames of video sequences using the low-dimensional feature representations. Principal component analysis (PCA) is applied to further reduce the dimensionality of the manifolds to two dimensions, facilitating the unsupervised clustering. The first two components were chosen, across all different geometries and materials, as they cover more than 95% of the variance of the data. In contrast to the training process of the network, where geometries and loading conditions of different materials were combined to increase the amount and variance of the data, to enable the network to learn generalized feature representations that describe necking, the individual geometries and materials are not combined during the assessment of the held-out data. This reduces the available information and number of data points that may be used to cluster the data. An increase in the density of the distributions is achieved by artificially augmenting the data, e.g., by cropping randomly, flipping, translating (by 5 px) and rotating (by 15 deg.) each image. The effect of such transformations is well described and visualized in . For evaMultiple experiments are conducted to analyze generalization of the method from one material to another, and to evaluate the possibility of validating metallographic examination results by categorizing individual images of video sequences without defined localization regions.AA5182 was not considered in this experiment, as its behavior and the appearance of its images are significantly different compared to the other materials and hence would deteriorate learning. During forming AA5182 develops shear bands at random locations that seem comparable to the necking behavior of the remaining materials, as visualized by the examples of the homogeneous and inhomogeneous forming phase of DP800-S050 and AA5182-S050, shown in The data used to train the network is restricted to one geometry and material. Two thirds of the data per geometry are used as training and validation set, while the remaining sequence serves as test set. This way it is possible to assess the loading conditions separately and at the same time assess the generalization to unseen, comparable data.All three video sequences per geometry are included in the training set and hence define a theoretical upper bound of the FLC. Of course, no generalization is assessed following this set up. In general, an overfit to the training data is invalid in pattern recognition approaches as it would not generalize to unseen data, but seems valid in the application of defining a forming limit based on three sequences for the generation of the FLC.This experiment assesses the location independence, as it evaluates multiple video sequences of forming processes of sheet metals that have not been deformed until fracture. The aim of this procedure was to stop the forming process close to the onset of localized necking, and hence to investigate the presence of material fatigue, based on metallographic examinations. The proposed method is used to validate these examination results and to ascertain if it is possible to assign frames to failure clusters without a well-defined localization region. Such an evaluation is not afforded by existing techniques, as they all require a priori definition of a localization area. As a result, of this procedure, strain path evaluations would be rendered unnecessary. Again, the evaluation is conducted using the held-out material.Within all experiments, the 50% intersection of the homogeneous forming phase and transition phase as described in A graphical representation of the localization and transition FLC candidates of DP800 are visualized in This especially occurs, when artifacts as visualized in Despite the availability of a limited amount of data, the proposed method generalizes to more complex materials such as AA5182, as visualized in The “line-fit” method (LF) as well as the unsupervised one-class SVM approach of Part 2 are compared to the method proposed in this study. The first approach focuses on a small heuristically defined evaluation area, and uses the reduction of material thickness to define the onset of necking. The second approach exploits more information, as it analyzes all principal strains simultaneously in multiple square evaluation areas in the vicinity of the maximum strain. Edge orientation information based on HoG features is extracted and classified using the one-class SVM coupled with a Gaussian mixture models (GMM) approach. As a result, the FLC can be expressed in terms of failure quantiles and hence leads to a probabilistic FLC, allowing a comparison of the proposed FLC candidates with different certainty levels. As this method expects a stable necking zone, the comparison is only made for DP800. The probabilistic FLC candidate of the unsupervised one-class SVM approach is depicted by the probability quantiles in The homogeneous/transition intersection lookup for S245 was omitted in all experiments, since it tends to not occur within this geometry, or is very close to the necking lookup time point or even superimposed by the other two phases cf. . For theMetallographic investigations are used to describe the micro-structure of materials qualitatively and quantitatively. Therefore, if forming processes are stopped early, it is possible to investigate the progress and status of formed specimen. Hence, if the forming process was recorded with a measurement system, macroscopic descriptions using the strain distributions of the last frame of the sequence correspond to the microscopic characteristics of the metallographic examination. With a multitude of experiments and small enough steps regarding punch progression it is possible to describe material behavior and ultimately the onset of localized necking in more detail  location dependency, due to evaluation only in the vicinity of the maximum strain area; (2) time-dependency, as it requires specimen to be formed until fracture and hence renders a comparison with prematurely stopped forming processes impossible; (3) predefined features limited to edge information; (4) transfer of knowledge to other materials or forming processes is not possible .The proposed method overcomes these limitations in a two-step approach. First a Siamese CNN is trained in a supervised manner, where only the extreme cases of the homogeneous and inhomogeneous forming phase (begin/end of forming process) are used to train the network. These images are separated optimally by minimizing the contrastive loss function, while augmentation is used to increase the data amount throughout several experiments. In the second step, complete forming sequences, not only the extreme cases, are assessed by the network and transformed into low-dimensional representative manifolds. Using PCA and SMM, these manifolds are clustered in an unsupervised manner into three distributions corresponding to the three phases, homogeneous, transition and localized necking. This procedure overcomes the mentioned limitations: (1) location dependency is reduced as a consequence of using the maximum possible square evaluation area of all materials; (2) the overall framework is now independent of time, by replacing the one-class SVM with SMM, allowing assessment of individual frames of incomplete forming sequences coupled with strain paths; (3) optimal features are learned by the Siamese CNN, and are not limited to edge information; (4) transfer of knowledge from one material to another is possible as shown with the lomocv experiment, enabling on-line supervision of unknown materials during forming processes; and (5) generalization of the method to materials with complex forming behavior such as AA5182, with limited data is possible as shown within the overfit and losocv experiment. However, accommodating measurement artifacts and defect pixels remains challenging, as their presence or absence may bias training of the network or negatively affect clustering of the data.In this study, a semi-supervised classification approach for the detection of the onset of localized necking was proposed. It comprises two steps: (1) a supervised, data-driven, feature learning part using a Siamese CNN, where only the extreme cases of the forming sequences were employed; and (2) an unsupervised clustering part, used to group the remaining frames of the sequence as belonging to the homogeneous, transition, or inhomogeneous forming phase. The results achieved using the proposed approach are consistent with previous approaches. However, while the latter required design and selection of suitable handcrafted features and a priori definition of the evaluation area, the main advantages of the proposed approach are its location and time independency. Consequently, the proposed method enables on-line identification of a distinct time point for the onset of localized necking, and could be used to stop forming processes, using the acquired image information. The question of what exactly is localized necking in terms of the strain distribution remains unanswered. So far, the top 90% of thinning or other heuristics are used to define the region regarded as localized necking. Future work could investigate a data-driven approach to learn segmentations of regions representative of localized necking.


Obesity is considered a chronic disease with an increasing prevalence worldwide during the last decades. Laparoscopic sleeve gastrectomy is the most commonly performed bariatric procedure, due to its relative safety and long-term efficacy. The use of bougie to ensure correct size of the gastric tube is part of the standard operation, usually placed by the anesthesiologist and with a very low rate of complications. We report the first case, to our knowledge, of a cervical esophageal perforation caused by the use of bougie during laparoscopic sleeve gastrectomy.2) and was diagnosed the first post-operative day. She was subsequently treated with an emergency thoracoscopy and evacuation of a mediastinal fluid collection, with additional neck incision for primary closure of the esophageal defect which was reinforced with a sternocleidomastoid muscle flap. The post-operative course was uneventful.The complication occurred in a previously healthy 42-year old female patient who underwent laparoscopic sleeve gastrectomy for class 1 obesity (BMI 31 kg/mWe made a literature review to better understand the options considering the diagnosis and treatment in case of very proximal iatrogenic esophageal perforations. The risks related to the use of bougie during surgery should not be underestimated, and its insertion must be done with extreme caution. Esophageal perforation is still a challenging, life threatening complication where prompt diagnosis and adequate treatment are essential. Obesity is a chronic disease that is constantly increasing in prevalence around the world and is now considered to be a worldwide epidemic. Recent studies reported that in 2015, approximately 108 million children and 604 million adults globally were obese [During the last years, laparoscopic sleeve gastrectomy (LSG) has become the most commonly performed bariatric procedure in the world . This pr2) in a private hospital in October 2018. Her past medical history was unremarkable, except for psoriasis and a previous laparoscopy for extrauterine pregnancy. The operation lasted 37 min and there were no obvious intraoperative complications. Immediately after the operation the patient started complaining about pain in the throat, mild difficulty swallowing and pain when moving her neck. During the night she complained about chest discomfort. Her vital signs were checked regularly and were normal. The next morning the pain was increasing together with swelling of the throat and marked sialorrhea. These symptoms led to further investigation with laryngoscopy, which was without remarks. A few hours later the patient developed subcutaneous emphysema and subsequently underwent an emergency CT scan, which showed free air outside the esophageal lumen, up to the neck and along the entire intrathoracic esophagus  used and combined were: “sleeve gastrectomy”, “esophageal perforation or rupture”, “bougie”. The search was limited to studies published in English. References of the selected articles were checked manually for additional relevant studies. A total of 62 articles were identified, but only two case reports describing iatrogenic perforations caused by the use of a bougie during bariatric surgery were found, both describing more distal lesions located in the middle  and loweBariatric surgery is recognized as the obesity treatment with the most effective long-term results. Among various bariatric procedures, LSG has become the most frequently performed worldwide for several reasons: its simplicity, with a limited alteration of normal anatomy, the excellent weight loss and remission of most obesity-related comorbidities  and its proven low morbidity compared with some of the other bariatric operations –9. The mThe standard technique involves the transoral insertion, usually by the anesthesiologist, of a bougie which is advanced to the pylorus and positioned against the lesser curvature, in order to ensure correct size of the gastric tube. A large variety of bougies exist, sizes ranging from 32– 50Fr, with 36Fr being the most commonly used . In our The mortality following esophageal perforation is high, reaching up to 80%, due to the development of mediastinitis and sepsis. Several parameters influence the clinical outcome, including the etiology of the perforation, the size and location of the defect and the time elapsed between the onset of the perforation and the initiation of adequate treatment . These pThe most important principles of the management of an esophageal perforation include resuscitation of the patient, assessment of the defect and timely decision-making regarding operative or non-operative management. If the patient is hemodynamically stable and the mediastinal or abdominal contamination limited, an endoscopy can be performed to assess the perforation and if appropriate treat it with the placement of an esophageal stent , 15, endIn cases where non-surgical management is not feasible, due to the location or size of the perforation or the extent of the contamination, patients should be treated with emergency surgery. The first choice for the surgeon should be the primary suturing of the defect, sometimes with an intercostal or other muscle flap to reinforce the repair. When this is not possible, due to severe sepsis with hemodynamic instability, large size of the defect or friability of the surrounding tissues, the best choice is a damage control approach, e.g. esophageal diversion, which gives the patient the best chance of survival. In these cases, reconstruction of the esophagus is typically performed 6 months to 1 year following the perforation, pending full recovery . In addiIn our case, the patient was stable but endoscopic treatment of the perforation was not deemed feasible; the defect was large but surrounded by vital tissue, so we decided to perform thoracoscopy in order to wash out the thoracic cavity and repair the defect. Nevertheless, the perforation was located too high to be repaired thoracoscopically and a neck incision was necessary in order to gain access. We performed a primary closure of the defect with sternocleidomastoid muscle flap reinforcement, with drainage of the mediastinum and thoracic cavity and broad-spectrum antibiotic therapy. The post-operative course was uneventful with a successful outcome.In conclusion, this is the first reported case, to our knowledge, of a cervical esophageal perforation caused by the bougie during LSG. We present it in order to underline that the risks inherited in the use of a bougie  during surgery should not be underestimated and that the insertion must be done with extreme caution. Esophageal perforation is still a challenging, life threatening, complication. Once the suspicion of a perforation arises all available tests  should be performed without delay, as prompt diagnosis and initiation of adequate treatment is the key to a favorable outcome.


With the growth of available sequenced datasets, analysis of heterogeneous processed data can answer increasingly relevant biological and clinical questions. Scientists are challenged in performing efficient and reproducible data extraction and analysis pipelines over heterogeneously processed datasets. Available software packages are suitable for analyzing experimental files from such datasets one by one, but do not scale to thousands of experiments. Moreover, they lack proper support for metadata manipulation.We present PyGMQL, a novel software for the manipulation of region-based genomic files and their relative metadata, built on top of the GMQL genomic big data management system. PyGMQL provides a set of expressive functions for the manipulation of region data and their metadata that can scale to arbitrary clusters and implicitly apply to thousands of files, producing millions of regions. PyGMQL provides data interoperability, distribution transparency and query outsourcing. The PyGMQL package integrates scalable data extraction over the Apache Spark engine underlying the GMQL implementation with native Python support for interactive data analysis and visualization. It supports data interoperability, solving the impedance mismatch between executing set-oriented queries and programming in Python. PyGMQL provides distribution transparency (the ability to address a remote dataset) and query outsourcing (the ability to assign processing to a remote service) in an orthogonal way. Outsourced processing can address cloud-based installations of the GMQL engine.PyGMQL is an effective and innovative tool for supporting tertiary data extraction and analysis pipelines. We demonstrate the expressiveness and performance of PyGMQL through a sequence of biological data analysis scenarios of increasing complexity, which highlight reproducibility, expressive power and scalability. By means of fast sequencing technologies, modern genomics promises to assist biological and clinical research by answering complex questions, e.g., how gene expression is deregulated in diseases, how mutations can lead to specific traits, how transcription factors interact to create complexes, how the genome organizes within three-dimensional configurations. To this aim, an impressive amount of sequencing data has been being collected by world-wide consortia as well as private laboratories and hospitals. Personalized medicine is slowly but steadily turning from vision into reality and biological research benefits more and more from bioinformatics approaches. In this scenario, computation approaches and tools are paramount to manage, process and analyze these large and heterogeneous collections of data.Primary analysis can be defined as the machine specific steps needed to call base pairs and compute quality scores for those calls. As these steps generate what are referred to as “reads” of small nucleotide sequences, it’s left up to secondary analysis to reassemble these reads to get a representation of the underlying biology, as well as the detection of signals . Tertiary analysis focuses on the integration of these signals to answer research questions and diverges into a spectrum of various study specific downstream investigations .Most of the effort of the bioinformatics and computational biology community was focused on primary and secondary analysis. Nowadays, the most important challenge is tertiary analysis, concerned with the development of complex models and tools for integrating and analyzing the heterogeneous pieces of information provided by secondary analysis, to the aim of producing novel biological knowledge. For addressing tertiary analysis, we proposed the Genomic Data Model (GDM)  and the Tertiary data management practice requires the intertwining and seamless integration of data extraction and data analysis, so that the data scientist can easily build interactive applications which include use of statistical testing, machine learning and visualization. Python is earning more and more attention as vector language for data scientists. For this reason, we designed and implemented PyGMQL, a Python library which embeds the GMQL engine. PyGMQL combines the highly scalable approach of GMQL with the flexibility of Python, and solves the impedance mismatch between set-oriented execution of data management systems and the procedural nature of scripting languages. PyGMQL provides distribution transparency (the ability to address a remote dataset) and query outsourcing (the ability to assign processing to a remote service) in an orthogonal way. By relying on its Spark implementation, it can scale up from local execution to arbitrary cluster architectures. The ability to scale on parallel and cloud computing environments is the most innovative and distinguishing feature of PyGMQL and allows performing complex queries on large datasets. This, together with the high-level genomic operation definition and the possibility to embed complex data analysis workflows inside Jupyter Notebooks, makes PyGMQL a comprehensive tool for big genomic data exploration, management and integration.Several efforts have been done in the design of libraries or command line suites for genomic region manipulation. BEDTools  and BEDOIn  and 11]11] we prPyGMQL has comparable expressive power in region manipulation with respect to these tools, but it adds an implicit iteration over all experiments of the same dataset. Its design is driven by scalability over thousands of experiments. Moreover, it supports metadata management explicitly, through high level operations for metadata extraction and for expressing predicates, that are seamlessly integrated with region manipulations. We designed the API of the library and its documentation so that biologists and bioinformaticians already competent in these tools can rapidly adapt to PyGMQL. For these reasons, PyGMQL can be considered as an upstream software for data manipulation both at the genomic data and metadata level, as demonstrated by several applications reported in the “The development of next-generation sequencing technologies has been followed by an increasing request of highly scalable software to define bioinformatics workflows . This haPyGMQL effectively complements these tools, by supporting data integration among heterogeneous data sources. We envision using PyGMQL at the end of secondary analysis pipelines, for supporting integrative analysis which also include access to large open repositories (as discussed in our application section). We already integrated both GMQL and PyGMQL with FireCloud and its evolution TerraPyGMQL is part of a larger ecosystem of tools for biological data-driven research, that comprises a data manager (equipped with domain-specific data model and query language) and an open repository providing access to several public datasets as well giving users the possibility to import their private data; briefly reviewed next.PyGMQL adopts the Genomic Data Model  to storeThe GenoMetric Query Language  was deveCurrently, the principal deployment of the GMQL system is through a Web application. For every user logged in the system, the GMQL repository reserves a private space to store his/her private data. In addition to this, the system collects a wide set of curated datasets from several sources which can be accessed by the users for integrative analysis. We integrated genomic metadata from five consolidated sources: The Cancer Genome Atlas  from GenPyGMQL is designed to address the specific needs of biologists and bioinformaticians during both the processes of pipeline design and data exploration . PyGMQL lazy execution model, inspired by the Spark implementation. Therefore, no actual operation is executed until the materialize is applied to the variable.PyGMQL offers a set of methods which wrap and extend the GMQL language operators Additional file Once the query is terminated, its results are loaded in memory and stored in a GDataframe data structure, which holds both regions and metadata in the form of two Pandas DataFramesTo facilitate the integration with the Python ecosystem, PyGMQL enables to import datasets directly from Pandas DataFrames that use a BED or GTF format.The interleaving between Python computation and GMQL execution constitutes a powerful tool for the bioinformatician, which is able to build complex pipelines without leaving the Python environment. By embedding PyGMQL within Jupyter NotebooksThe GMQL system is directly linked to one repository deployment, although many deployment technologies are supported. Instead, PyGMQL enables the users also to locally access their data, like most of the Python libraries. In addition, PyGMQL can also interface with an external GMQL system and login with the user credentials (using the login function) to interact with his/her private datasets or the public repository. During the query composition in PyGMQL, the user can specify if a source dataset comes from his local file system (using the load_from_path function) or from the remote GMQL repository (using the load_from_remote function). Therefore, queries in PyGMQL can be composed of genomic operations acting both on local and remote datasets.local mode) or on the remote (remote mode) system  or in HDFS.YARN master: the back-end of the library is submitted to the Application Master of a YARN cluster and interacts with the front-end through a TCP connection. In this case, the local datasets reside on the Hadoop file-system. This deployment strategy must be adopted to run the library on cloud providers like Google Cloud and Amazon Web Services.When the mode of execution of PyGMQL is set to Figure We demonstrate the flexibility of the PyGMQL library through three data analysis workflows, available in the form of Jupyter Notebooks and scripts both in the Supplementary Materials of this paper and in the PyGMQL GitHub repository. For a progressive introduction to PyGMQL usage, the applications are increasingly complex both for what concerns the biological analysis and the data size Additional file Examples show: (a) the interplay of local and remote GMQL resources, (b) the scalability with datasets of increasing size. A schematic description of the three deployment strategies is shown in Fig. Examples show the interplay between pure Python code and scalable genomic operations, implemented in PyGMQL through the GMQL engine. For this reason, a complete implementation of the examples in GMQL is not possible. While it is possible to implement them by stacking together some of the previously cited tools or even standalone Python code, their parallelization and metadata management would induce a great development overhead for the researcher. The examples highlight the expressiveness, cleanness and ease to use of the library, demonstrating that complex parallel genomic data analysis workflows can be achieved using just PyGMQL.In this first application, genes’ promoters are extracted from a local dataset and a large set of Chip-Seq experiments is selected from a remote repository. Then, for every promoter and for every Chip-seq experiment, the average signal of those Chip-Seq peaks intersecting the promoter is computed. The result is finally visualized as a heatmap, with rows representing promoters and columns representing Chip-Seq experiments.This example shows: (i) the integration of local PyGMQL programs with remote repositories, (ii) the possibility to outsource the execution to an external deployment of (Py)GMQL, (iii) the interplay between PyGMQL data and Python libraries written by third parties. These features allow users to write arbitrary complex queries - whose execution and size of the inputs exceed the capabilities of the local environment - and, at the same time, analyze/visualize the output by means of well-known Python libraries.genestart−2000;genestart+2000]). Note that the “.start” and “.stop” attributes automatically consider the strand of the region.The code begins by loading a local dataset of gene annotations and extracting their promotorial regions .Next, we load the external dataset of Chip-Seq from a remote GMQL Web service. In order to do so, the user has to specify the remote address and login. If the user has already signed to the remote GMQL installation, he/she can use his/her own credentials , otherwise a guest account is automatically created, without requiring the user to do it manually.lazy, therefore no actual data is moved or red at this point. Then the user specifies the select condition: the hms[~experiment_target~] notation enables the user to build predicates on the given metadata attribute. The GMQL engine loads from the dataset only the samples whose metadata satisfy such condition, specifically, only experiments targeting the human H3K9ac marker will be selected.Next, we show how to load the Chip-Seq data of the ENCODE dataset from the remote GMQL repository and select only the experiments of interest. First, the user sets the remote execution mode and imports remote datasets with the load_from_remote function. Such loading is Next, the PyGMQL map operation is used to compute the average of the signal of hms_ac intersecting each promoter: iteration over all samples is implicit. Finally, the materialize method triggers the execution of the query. Since the mode is set to "remote", the dataset stored at ~./genes/~ is sent to the remote service GMQL system that performs the specified operations. The result is loaded into the mappingGDataframe variable which resides on the local machine.At this point, Python libraries for data manipulation, visualization or analysis can be applied to the GDataframe. The following portion of code provides an example of data manipulation of a query result. The to_matrix method transforms the GDataframe into a Pandas matrix, where each row corresponds to a gene and each column to a cell line. Values are the average signal on the promoter of the given gene in the given cell line. Finally, the matrix is visualized as a heatmap.cell-specific, i.e., are active only in a limited number of cell lines or tissues . The interesting aspect of this query is that some of the discovered traits are associated with that cell line, e.g., because they are concerned with a disease affecting the tissue of the cell line.In the following example, we explore the interaction between a Genome-wide Association Study (GWAS) dataset . Note that, in addition to GDM-like datasets, PyGMQL can load data with generic format using the load_from_file function. In order to do so, the user must configure a custom parser for the TSV format, by specifying the position of the chromosome and the start/end positions (in this particular case the 11-th and 12-th columns respectively) and the position of any of the fields he/she wishes to import . The file is then imported as a valid GDM instance. To explore instances, we use the head function, that displays the first rows of both regions and metadata. Its effect is shown in the Supplementary Material. interval to ±1500 bp, resulting in enlarge regions of 3000 base pair length.We next load the ENCODE Broad Peak and select from it the Chip-Seq experiments of H3K27ac, as predictor of enhancer activity. This is done with a selection on the experiment_target metadata attribute. We perform a normalization by extracting the position in the middle of each Chip-Seq region with a reg_project operation and then by extending the mid position of a given base pair interval on both sides, again with a projection. We set First, we group the enlarge regions by the cell line on which they were probed. We also merge overlapping regions of the same cell line which could be due to replicas. This is done with a normal_cover operation adding the groupBy clause on the biosample_term_name metadata attribute, which specifies the cell line of origin for every sample.Next, we use again the normal_cover operation to filter cell-specific enhancer regions, i.e. regions that are present in more than a specified maximum number of cell lines (max_overlapping variable). This parameter can be set to small numbers, such as 1 or 2, to increase specificity.DLE(0)). This results in a dataset with a sample for each cell line containing its set of specific enhancers.Finally, we intersect the set of cell-specific enhancers produced by the second cover operation with the set of enhancers grouped by cell line  are known to act in cooperation, as a functional complex. In previous work, we developed TICA  and an associated Web service , 32 usinThe data-intensive computation has been ported to PyGMQL. The purpose of this Section is not to show the PyGMQL code - as it is rather complex, see Supplementary Material - but rather to show that it scales to very large datasets. In essence, for every cell line the code considers all possible pairs of TFs for which ChIP-seq experiments are available (ranging between 116 in GM12878 and 268 in K562) as candidate complexes. For each such pair, it computes the bindings which are at minimal distance within promotorial regions. Once minimal distance pairs are extracted by PyGMQL, a significance test (written in Python) predicts if the two TFs form a complex.The code reported in the Supplementary Material illustrates the data extraction part. It uses a normal_cover so as to merge replicates of the same experiment and then two join operations, the former for detecting the overlap between each TF region and active promotorial regions, the latter for extracting the pairs of regions of two TFs at minimal distance within such regions. Joins are complex operations, and here they are performed in the three-dimensional full genome space formed by the bindings of the two TFs and of the promotorial regions.m5.2xlarge instances, each of which has 8 virtual cores, 32 GiB of memory and 300 GiB of hard disk capacity. We used 4 different setups, all with 1 single master nodes and 1, 3, 5 and 10 slaves.The ability to scale on parallel and cloud computing environments is the most innovative and distinguishing feature of PyGMQL and allows performing complex queries on large datasets. In this Section, we show the computational performance of PyGMQL on the last application, for three cell lines  with increasing amount of available samples. For our experiments, we used AWS Elastic MapReduce clusters of Table Python is becoming the leading programming language for data science, thanks to its flexibility and ease of use, the embedding within Jupyter notebooks, and the huge number of supporting libraries and packages. Within the scope of genomic computing, PyGMQL is a new Python library for linking domain-specific data extraction to domain-independent tools and environments for data analysis and visualization.PyGMQL fills the gap between the scalable Spark-based data management engine of GMQL and the huge body of Python-based resources. PyGMQL supports data interoperability, solves the impedance mismatch between set-oriented data extraction and imperative programming, provides distribution transparency and query outsourcing to powerful server-based and cloud-based systems. The possibility of supporting both local and remote queries enables the efficient prototyping of data extraction pipelines, which can be locally developed and then deployed to big remote services.Project name: PyGMQLhttps://github.com/DEIB-GECO/PyGMQLProject home page: Operating system(s): Platform independentProgramming language: Python (version 3.4 or higher)Other requirements: Java 1.8 or higherLicense: Apache 2.0Any restrictions to use by non-academics: NoneAdditional file 1 Supplementary materials guide. Document describing how to reproduce the pipelines presented in the manuscript (PDF).Additional file 2 Library documentation. Detailed documentation of the software package (PDF).