This paper presents a traffic sign detection method for signs close to road intersections and roundabouts, such as stop and yield (give way) signs. The proposed method relies on statistical templates built using color information for both segmentation and classification. The segmentation method uses the RGB-normalized (ErEgEb) color space for ROIs (Regions of Interest) generation based on a chromaticity filter, where templates at 10 scales are applied to the entire image. Templates consider the mean and standard deviation of normalized color of the traffic signs to build thresholding intervals where the expected color should lie for a given sign. The classification stage employs the information of the statistical templates over YCbCr and ErEgEb color spaces, for which the background has been previously removed by using a probability function that models the probability that the pixel corresponds to a sign given its chromaticity values. This work includes an analysis of the detection rate as a function of the distance between the vehicle and the sign. Such information is useful to validate the robustness of the approach and is often not included in the existing literature. The detection rates, as a function of distance, are compared to those of the well-known Viola–Jones method. The results show that for distances less than 48 m, the proposed method achieves a detection rate of  Traffic accidents are the primary cause of death for young people between 15 and 29 years old. Between 20 to 50 million people are injured each year, while ountries , due to ountries .Most traffic accidents occur in urban areas, especially at road intersections and roundabouts. Country statistics for traffic accidents show that a significant number happened at road intersections, for instance,  the USA ,8, 58.7%ing 2015 , and 9.2ing 2014 . Thus, ting 2014 , which uing 2014  has not ing 2014  by analyWe propose a traffic sign detection approach based on statistical templates built using normalized color information. The novelty of the approach lies in the probabilistic model of the sign (or object) conditioned over the intensity of the normalized color channels instead of using traditional shape descriptors. The results show that this approach is robust to variations in distance between the car and the traffic sign, as well as variation in illumination. Unlike deep learning techniques, the results show that it is possible to implement the proposed traffic sign detection approach with small datasets of a few hundred images.This paper is organized as follows. First, the state-of-the-art of traffic sign detection algorithms is discussed in Traffic sign detection using visible-spectrum cameras may take different approaches. Some works implement feature classifiers. This means that a sliding-window method is used to compute features on different overlapping regions, which are then fed to the previously trained classifier ,15. The k-means clustering [k-means clustering algorithm, the authors build a classifier that employs the H and S components of the HSV space to train a classifier and implement the color segmentation that yields ROIs. More recently, Han et al. [H component of the In the context of traffic sign detection, blob generation and color analysis are the main techniques employed to segment regions of interest. Special efforts have been placed on making the color-based segmentation robust to large variations in illumination and weather conditions. Greenhalgh et al.  transforustering  to generustering  implemenustering  have useh et al.  use the n et al.  have user et al.  have user et al.  employ tThe recognition stage typically employs feature classifiers, and therefore requires a feature descriptor and a classification algorithm. One of the most popular feature descriptors is the histogram of oriented gradients (HOG) , which pk-NN (k-nearest neighbors) algorithm [Concerning classifiers, most of the recent work in traffic sign detection employs SVM (support vector machine) classifiers ; see forlgorithm  is emplolgorithm . Recentllgorithm , Jung etlgorithm , Zeng etlgorithm , Zhu et lgorithm —which prlgorithm  rely on lgorithm  have alsThe main traffic sign databases correspond to the following countries: Germany ,20,29,32Thus country-specific databases are required for the development of traffic sign detection systems. However, there is a lack of databases with traffic signs in Latin America. Therefore, another goal of this work is to contribute to the development of traffic sign detection systems that can be validated also on traffic signs of the Latin American region.The proposed computational strategy for detecting signs found at road intersections and roundabouts is composed of two parts. The first part generates ROIs in which traffic signs could be found calculating and analyzing color statistics in the normalized RGB space. The second part solves the recognition of signs in ROIs using a statistical template matching strategy using templates also in the normalized RGB space.The detection must be done at the furthest possible distance, so that the driver has enough time to react and to stop in time. Examples of typical testing scenarios for the proposed approach are presented in Under adequate illumination conditions, such as daylight or artificial lighting, the color of traffic signs is a feature that can be used to generate ROIs. Comparing the histograms of traffic signs in four color spaces, RGB, HSV, YCbCr and ErEgEb  ,39, it pThe color space that shows the smaller overlap between the histograms of positives (signs) and negatives (non-sign) is the ErEgEb space, where in particular the c computed over a set of positives (images with traffic signs) according to:c at the pixel location p within the traffic sign of the i-th reference image (positive), c for the i-th image, c across the ensemble of Therefore, the candidate regions of interest can be detected using a chromaticity filter, i.e., a filter that works on the variables that define color hue (dominant wavelength) and color purity or saturation  regardless of luminance (magnitude of the color components vector) or psychological perception of illumination brightness or intensity (as an average of the components of the color vector). In other words, a chromaticity filter only requires two variables that describe dominant wavelengths regardless of the total energy by mapping the components of the thrichromacy color model into a subspace of two normalized values. Assuming a normal distribution of the chromaticity channels l images ,40 is poThe last step for the final proposal of ROIs is to eliminate the overpopulation of candidates. To this end, all the candidates that are contained within or are a sub-window of another candidate are discarded, so that only the largest block remains. Next, windows whose mean value is closest to The second stage of the proposed traffic sign detection approach is responsible for solving the identification of ROIs as traffic signs of a given type. To this end, a set of images is employed to create two statistical models, one with the mean intensity and the other with its standard deviation for each pixel belonging to the traffic sign. Testing on a sliding block for the percentage of pixels that fall within the expected intensity range for a given location provides a discriminator to detect traffic signs from background and non-traffic sign objects. A flow diagram of the proposed method is shown in I,k, I.In order to develop the probabilistic template matching model to recognize traffic signs, it is first convenient to introduce the following notation:O and O and I are random samples [The probability that a window corresponds to a particular object (sign) is: samples ,42. AlsoAlgorithm 1: Traffic sign recognition algorithm based on statistical templatesInput: Output: // Loading pre-trained masksLoadAverageMask;LoadStandardDeviationMask;// Pixel mask discarding corresponding to the background// Minimum and maximum accepted masks// Pixel mask accepted// Final decisionifSumPixels and the statistical templates for the mean and standard deviation To this end, To create a mask for discarding background pixels, an adequate thresholding value for the standard deviation template is By the foregoing, the channels to be used in the recognition stage have the luminance to remove the background and the chromaticity values Er and Eb to confirm the ROIs are of a given traffic sign, ensuring robustness to illumination variations.The vehicle shown in The training dataset contains 2567 negative examples, 122 images containing stops signs and 80 images containing yield signs. The positive samples were randomly rotated, scaled and translated in order to produce 7000 positive examples; see The datasets consider a sequence of signs as the car approaches different intersections in the city of Santiago, Chile. Both datasets contain traffic signs in real driving conditions, under varying illumination and partial occlusion.The database has been made available at RAL  to contrIn this section, the proposed traffic sign detection approach based on statistical templates is tested and compared to the well-know Viola–Jones method .The detection rates of the Viola–Jones approach for the stop and yield signs are summarized in The proposed algorithm was executed employing eight different templates sized from In terms of the computational effort, the Viola–Jones method required 450 ms per frame while the proposed approach required 950 ms per frame. This amount could be decreased for real-time operation using dedicated graphic processing units (GPUs).Finally, This paper presented an approach based on statistical templates computed from chromaticity and luminance values for traffic sign detection near road intersections and roundabouts. The approach is evaluated using a dataset of stop and yield (give way) signs from Chile and compared to the well-known Viola–Jones classification method.Y luminance channel. The luminance channel is employed to create a traffic sign mask from the variance of pixels that allows background regions within a ROI to be discarded. The Er and Eb channels are used to compute a statistical template that provides the sign selection thresholds. A probabilistic model and probability distribution function were derived to construct a recognition process based on Bayesian inference.The proposed approach is divided into two stages. The first stage is a segmentation stage based on the chromaticity filter applied to the Er and Eg channels of the ErEgEb color space . In the second stage, the chromaticity filter returns candidate regions that are responsible for the recognition of traffic signs and which must be tested. The selection of the Er and Eg channels is based on the analysis of the histograms of the color components in four color spaces , which shows that the ErEgEb provides greater capacity to discriminate candidate traffic signs. The segmentation stage employs a selection threshold that can be computed automatically from a reference dataset. The recognition stage is based on a statistical template built with information of the The results obtained show that the proposed approach has higher detection rates than the Viola–Jones method. The experiments considered the evaluation of the detection rate at different distances as the vehicle approaches a traffic sign. On average, the proposed approach exhibits a detection rate of The two main advantages of the proposed approach are summarized in that it does not require a computationally expensive training or calibration stage, and it is not sensitive to changes in illumination, partial occlusion or marks drawn on the traffic sign. Future work will consider extending the proposed approach to the detection of traffic lights at road junctions and crosswalks, as well as to the detection of other signs not necessarily found at road intersections. Ongoing research is considering joint analysis of lane geometry analysis and edge continuity together with traffic sign detection to improve the detection of road intersections. The main limitation of this model is the computing time, which is about 950 ms per frame. As a future work, we will work to reduce this processing time.


P that can be related to an upwelling of deep mantle material likely flowing laterally through a window opened by the complete slab detachment.A detailed 3D image of the Calabro-Ionian subduction system in the central Mediterranean was obtained by means of a seismic tomography, exploiting a large dataset of local earthquakes and computing algorithms able to build a dense grid of measure nodes. Results show that the slab is continuous below the southern sector of the Calabro-Peloritan Arc, but the deformation processes developing at its edges are leading to its progressive narrowing, influencing tectonics and magmatism at the surface, and with possible stress concentration in the tip zones. In the southwest, the deformation occurring at a free slab edge lead to propagation of a vertical lithospheric tear in the overriding plate, which extends along a NW-SE fault system (Aeolian-Tindari-Letojanni) up to about 30 km into the Ionian Sea; further southeast, the lithosphere appears only flexed and not broken yet. In the northeast, the slab seems to break progressively, parallel to the trench. Finally, northwest of Mt. Etna, the tomography highlights low V Moreover, the slab retreating and trench shifting towards the foreland was accommodated by lateral tearing which split adjacent segments of the subducting lithosphere; as a result, the active portion of the system has become progressively narrower4. Currently, active subduction residue of the ancient and wide trench zone is found beneath the Calabro-Peloritan Arc .Subduction processes have a great influence on the tectonic evolution and on the geologic architecture of a region, as well as on its seismicity and magmatism. Accordingly, their understanding and the zone where they take place are major topics in geological research. In the geodynamic context of the African and Eurasian plates convergence, the Mediterranean region provides an outstanding study opportunity, being the result of the long–lasting (35 Ma) NW-dipping subduction of the ancient Ionian oceanic lithosphere. The present-day tectonic framework of the central-western Mediterranean arises from the evolution of this subduction process, which has been characterized by the rapid SE-ward rolling-back of the Ionian slab and by the opening of large back-arc extensional basins . There is a general consensus that the slab is still continuous in-depth along a quite narrow segment of the Calabro-Peloritan Arc; however, details about the structure and the processes currently occurring and how they influence the tectonic regime and magmatism of the region are still elements inadequately constrained and matter of discussion. In particular, a number of studies have dealt, often with discordant conclusions (e.g. refs11), with the slab edges, which are tectonically important features, and with the way they deform. In recent years, interpretation of seismic profiles along the Ionian Sea have allowed to reconstruct the shallow part of the Calabrian accretionary wedge and of its southernmost edge (see refs12). Several km-long structural lineaments have been identified, some of which have been interpreted as the shallow expression of lithospheric-scale tear faults bounding the subducting plate ; however, an in-depth lithospheric image of the subduction system is lacking. Detailed knowledge of the shape of the slab is critical element for modeling the subduction system evolution and the related tectonic processes.Much effort has been made by previous investigators to understand the evolutionary phases and to describe the deformation mechanisms that characterize the Calabro-Ionian subduction system (e.g. refs15).In this regard, seismic tomography is the most effective method to explore the deep structure of subduction zones. Seismicity can reveal the plate interactions and map the Wadati-Benioff zone; moreover, the existence of positive seismic velocity anomalies in the upper mantle can be interpreted as the effect of cold oceanic lithosphere that penetrates a warmer and lower velocity asthenosphere, whereas, low velocity regions are preferably associated with partially molten rocks (e.g. refs16) and to improve the relative position of clustered events (tomoDDPS by ref.17). As a result, we were able to investigate the large scale tectonic structures in unprecedented detail with respect to previous comparable tomographic studies (e.g. refs18).In this paper, we focus on improving the image of the Calabro-Ionian subduction system, by means of a local earthquake tomography (LET), in order to better understand the geodynamics and tectonics of the whole region. For this goal, we considered a large dataset of about 20,100 earthquakes, encompassing both shallow and deep seismic events located in southern Italy and in the central Mediterranean, which was exploited to obtain a reliable 3D velocity structure and accurate hypocentre locations. We took advantage of recently developed computing algorithms, which are able to build an adaptive grid of measure nodes according the seismic rays density  to a deeper seismic level beneath the Calabrian accretionary wedge  about the overall structure of the descending Ionian slab: it is continuous at depth only below the southern Calabro-Peloritan Arc, where its curvature is increased19, specifically between the ATLFS and S. Eufemia Gulf; it is very steep (70°), at least down to 200 km of depth. However, processing of new data has allowed us to better image the 3D architecture of the subduction zone, which proves to be strongly controlled by the deformation processes currently developing at its edges and in particular by large tears leading to progressive narrowing and necking of the slab.The tomographic images and the earthquake locations presented here are in agreement with the results of previous studies . Vertical breakoff can be deduced by the substantial absence of deep seismicity west of Milazzo and by the deep lateral velocity discontinuity. This tear has a clear expression at the crustal level by vertically clustered events (down to 40 km of depth) and discontinuities in the VP model which can be observed in the correspondence of the ATLFS and its prosecution in the Ionian offshore  and confirmed by NW-SE dextral or oblique slip focal mechanisms of events deep down to about 70 km30  and the subducting Ionian (oceanic) lithospheres , this layer may be interpreted as being constituted by weak material (soft sediments) scraped from the subducted oceanic crust and accreted in the inner portion of the Calabrian accretionary wedge.In the southwest, there is evidence for a “free” slab edge, produced by a combination of vertical and horizontal tearing which may be hypothesized according to the model by ref.res Fig. . This deres Fig. , which d34 between sectors of material with different densities. These mechanisms may explain changes in seismic anisotropy and anomalous melting patterns in surface volcanism (see ref.35). In particular, along the Calabro-Ionian subduction region, the direction of the SKS fast wave polarization, usually related to mantle flow, presents an abrupt rotation from trench-parallel to perpendicular near the southwestern slab edge (see refs37). Moreover, the geochemical signatures of fluid enrichment and magma mixing in Mt. Etna lavas have been considered a consequence of the progressive opening of a slab window through time . Deep seated thrusting beneath Mt. Etna  could be seen as the consequence of the wedging of the low velocity body in the area41.Another remarkable feature is the large low velocity volume found NW of Mt. Etna at the southern slab boundary. Its extension down to 80 km of depth could indicate upwelling of warm and less dense mantle material to shallower depth. Indeed, the existence of slab tears may create a window through which asthenospheric mantle can flowP patterns also give hints about the current structure of the subduction zone where the slab is still continuous; in particular, in the section HH’  no evidence for lithospheric tearing were found; the Hyblean-Ionian lithosphere appears only flexed and not broken yet. Previously mapped faults in the Ionian offshore19.On the northern side, the slab has been progressively broken parallel to the trench and the horizontal tear may still propagate southwards. Horizontal tearing affecting both the sides of the slab results into a narrowing of the subduction system with consequent stress concentration at the tip zones and enhanced subsidence due to the gravitational pull along the intact segment of the slab43 (i.e. west and southwest of the Mts Peloritani and northeast of S. Eufemia). This matches the scheme postulated by ref.19.This overall image can be confirmed by the topographic profile along the Calabro-Peloritan Arc, which shows a relatively depressed region corresponding to the area overlying the intact slab and highs at its edgesA lot of the analysed seismicity appears to be located near the slab hinge; in particular it is concentrated along the former backstop of the overriding European crust rather than, as expected, at the subduction interface where, conversely, events appear quite sparse. Deeper earthquakes occur within the high-velocity layer (the descending slab) highlighting the deformation of the Hyblean-Ionian lithosphere beneath the Calabrian crust. In the accretionary wedge, likely made of weak material scraped off the subducted Ionian oceanic crust, seismicity is quite low.In conclusion, slab narrowing by the horizontal breakoff and its necking by gravitational pull are factors that must be considered in assessing the future seismic potential of the Calabro-Ionian subduction system, in addition to its still rupturing southern edge.http://istituto.ingv.it/index.php/it/archivi-e-banche-dati). From these databases, we selected earthquakes recorded between 1981 and 2014 within the area of interest  was carried out by the program LOTOS16, by inverting a total of 236,048 P and 104,616 S arrival times recorded at 770 stations. Before performing the tomography, it has been necessary to define a set of parameters and in particular a preliminary guess for the 1D seismic velocities, which was derived from the model optimized for the same area by ref.44. The average value of VP/VS ratio reported in the literature (1.75) was checked by a Wadati diagram45.Seismic velocity modeling . The inversion solution is further controlled by smoothing parameters which reduce the difference in the final values of neighboring nodes. The ability to build such a dense (quasi continuous) mesh is fundamental for a “local earthquake tomography” in order to reduce the bias of the resulting models due to the grid configuration (i.e. too large spacing between nodes to suitably image earth heterogeneities). Fine-tuning of the inversion was then obtained by discarding events with less than 12 observations and arrival-times with residuals of more than 1.5 s for P and 2.5 s for S rays.The accuracy of the final 3D results was assessed by tests with numerical models. The procedure consists of calculating theoretical travel times of the seismic rays for an idealized velocity model (target model). The synthetic data are then inverted using the same starting model and control parameters as for the real data. By comparing the results of this inversion with the target model, one can detect unrestored zones that correspond to areas where tomographic resolution is low.In the first series of tests, we considered traditional checkerboard models composed of blocks with alternating positive and negative anomalies  we can robustly recover small anomalies (10–15 km of size). Increasing the depth, larger features can be recovered, as the seismic ray coverage gradually reduces, and limitations in resolution emerge in some parts of the study area. However, the imaged structures and the slab are well resolved, at least down to about 200 km.The results of these tests clearly indicate that the VSee the Supplementary material for a complete description of the synthetic tests.17, which has the advantage of using a combination of both absolute and differential arrival time readings. In case of clustered events, additional information is used to improve the relative locations: i.e., since the uncertainty of the velocity model along the station-hypocenter path is the same for all the earthquakes, the travel-time differences found at a common station can be attributed to the spatial offset between the events. This procedure produced better clustering and further reduced the residuals (RMS) of about 55%, with an average of 0.3 sec.Earthquake locations of the whole dataset were further enhanced by the tomoDDPS algorithmhttp://istituto.ingv.it/index.php/it/archivi-e-banche-dati). The LOTOS code can be downloaded from http://www.ivan-art.com/science/LOTOS. The tomoDDPS algorithm was provided by H. Zhang (zhang11@ustc.edu.cn). Data not available on these websites can be provided by the corresponding author on reasonable request.Most of the earthquake parameters are available on the INGV institutional website (Supplementary Information


Hyperspectral imaging (HSI) is increasingly gaining acceptance in the medical field. Up until now, HSI has been used in conjunction with rigid endoscopy to detect cancer in vivo. The logical next step is to pair HSI with flexible endoscopy, since it improves access to hard-to-reach areas. While the flexible endoscope’s fiber optic cables provide the advantage of flexibility, they also introduce an interfering honeycomb-like pattern onto images. Due to the substantial impact this pattern has on locating cancerous tissue, it must be removed before the HS data can be further processed. Thereby, the loss of information is to minimize avoiding the suppression of small-area variations of pixel values. We have developed a system that uses flexible endoscopy to record HS cubes of the larynx and designed a special filtering technique to remove the honeycomb-like pattern with minimal loss of information. We have confirmed its feasibility by comparing it to conventional filtering techniques using an objective metric and by applying unsupervised and supervised classifications to raw and pre-processed HS cubes. Compared to conventional techniques, our method successfully removes the honeycomb-like pattern and considerably improves classification performance, while preserving image details. Squamous cell carcinoma is the most common head and neck cancer and the cause of approximately 350,000 deaths per annum worldwide . Early dPrevious studies on honeycomb-like pattern removal from endoscopic images have either filtered in the spatial  or FouriIn addition to the honeycomb-like pattern, HS data collected with a flexible endoscope faces the same problems as HS data recorded by rigid endoscopy. These issues include false registration from one image to the next within the HS cube due to the patient’s heartbeat, white noise in the lower wavelength spectrum and specular reflections (SR). Image contaminations can, however, be successfully corrected by a pre-processor that was recently developed . The honThis paper is organized as follows: For hyperspectral imaging, a Polychrome V monochromator  was triggered synchronously with a monochromatic high-resolution CCD-camera . A flexible endoscope was attached to the monochromator and the camera using a fiber optic light cable. Both devices were controlled using commercially available software . To reduce movement artifacts, the endoscope was held in place by an endoscope holding system. The system generates HS cubes between 390 and 680 nm in 30 iterative steps with a 10 nm interval. HS images measure 1388 × 1040 pixels with a bit depth of 12. The flexible endoscope is made of thousands of hexagonally arranged fibers. Each optical fiber consists of a core and a cladding, which have a higher and a lower refractive index, respectively, and generates total reflection at the core-cladding interface. Light is transmitted from where it enters the fiber to the other end by following the bends through the core without a loss in intensity. The image shows honeycomb-like structures, consisting of bright, circular-shaped patterns surrounded by dark rings analogous to the arrangement of the fibers and the relation of the core and cladding . PatientThis section provides a detailed description of the method for removing the honeycomb-like pattern as well as additional pre-processing steps and the classification methods. Removing the honeycomb-like pattern prior to processing the image further (such as image registration) is important, since the pattern is superimposed over the other data structures. For additional image improvement, the pre-processor developed by  was applPeriodic and quasi-periodic noise appear as concentric bursts of energy at the position corresponding to the frequencies of periodic interference in the FD . FrequenThe honeycomb-like pattern produces six radial, symmetrically arranged and clearly visible peaks in the lower frequencies around the direct current (DC) . DependiThe FD image was computed from the original SD image using a 2D fast Fourier transform (FFT) algorithm. The procedure is explained in detail in the following sections and depicted in t. A local maximum was not considered as a global maximum if a pixel was reached that was already filled during flooding of another local maximum. Another exclusion criterion for a particular local maximum was the hit of a pixel with an amplitude higher than that local maximum. This approach adapts the global maxima detection algorithm of the open source software ImageJ (1.49) [k peaks.The challenging task of peak detection was solved in the following manner. Local maxima were identified on the Fourier spectrum by comparing each pixel to its neighboring pixels using a moving window of 3 × 3 pixels. If a pixel has a higher value than its neighbors, it is defined as a local maximum. To derive peaks from the local maxima, the latter were used as seed points for an eight-connected flood-fill algorithm. Starting with the local maximum of the highest amplitude, flood filling was successively applied to each local maximum in descending order of amplitude. A pixel was included in the flooded region if its value was less than or equal to the value of the local maximum and greater than or equal to the local maximum minus an empirically determined value J (1.49) . The resn for each filter window, an eight-connected flood-fill algorithm was adapted to identify the extent of the affected components corresponding to each peak. The algorithm incorporates new pixels into a region if their value is less than the value of the current pixel. Peaks were used as starting points for flood filling and the resulting connected region marks the affected neighborhood. Next, the highest distance d between the peak and a pixel within the flooded region was identified. This distance was used to define the area n around the peak a is an experimentally determined scaling factor.To derive an individual size Affected components were filtered using the windowed filtering approach . The frex and y are the coordinates within the filter window and We tested several different filtering techniques , includiribed by . This iss and Rayleigh-based line separation criteria r. The combination of both is given by the weighted quality measure γ is the factor that weights the importance of s compared to r. The quality metrics ranges between 0 and 1. As the value of q increases, the better the performance is in terms of γ. The metric takes into account both the positive effects of removing the honeycomb-like pattern as well as the negative impact of blurring. s measures how well the honeycomb-like pattern is removed by comparing the average standard deviation of a homogeneous area of the filtered image to the unfiltered image, while r measures the detail-preserving quality of the filter, which can be derived from the filtered image of a USAF test chart using an image area along line patterns. The less those line patterns are blurred, the higher the detail resolution of the filtered image is.To quantify the performance of the filter techniques, quality metrics were computed on the basis of the USAF test chart. This is a resolution test pattern that consists of several groups of different-sized bars and was originally developed by the USAF to test the resolution of optical imaging systems. The HS cube of the USAF test chart was recorded under exclusion of external light sources to emulate the light conditions of the larynx. The comparison was made using the quality metrics proposed by , variancIn addition to the honeycomb-like pattern, the HS data exhibit other complications, which were corrected by the pre-processor proposed by . The preThe combination of the anatomy of the larynx and the geometry of light source results in various intensities of light reaching the mucosa surface. If a spectrally homogeneous object is variably illuminated, its spectral signatures are linearly scaled variations of one another. Although the shape of the spectral signature remains the same regardless of the illumination conditions, there is a significant impact on classification performance. To effectively reduce image artifacts caused by non-uniform illumination of the scene, NRI  EquationUnsupervised classification with GMM and supervised classification with spectral similarity measure were applied to the raw and pre-processed HS data to test the performance of the introduced filter technique in terms of HS classification. We compared our method with the super-Gaussian filter, SD-Gaussian filter with a kernel size of 17 × 17 pixels and star-shaped filter.X denotes the feature vector and k is the number of mixture components, namely, the number of clusters. kth component, which is expressed by The GMM is an established unsupervised classification method in HS imaging, wherein each cluster corresponds to a Gaussian distribution. The GMM is a parametric probability density function defined byposed by . EM iterSupervised classification was used to visualize the effect of filtering on the information content of HS images. Therefore, a spectral similarity measure was used to detect the cancerous tissue of #caseA using the mean NRI spectrum of the cancerous tissue of the second case (#caseB). The mean NRI spectrum was calculated from 100 randomly selected pixels within the marked area of cancerous tissue. We used the Pearson’s correlation coefficient as a similarity measure, since it is responsive to the spectral shape but not to changes in brightness. The correlation coefficient ranges between −1 and 1, negative and positive correlation, respectively; 0 indicates no relationship between spectra.s and r, as well as the resulting combined measure q for different weights γ, which are 0.5 and 0.8. In the case of s value. If no filter is applied, the image reaches its highest value for detail preservation r, since smoothing does not occur. s is 0 for the unfiltered image, since it was used as reference.The proposed method for removal of the honeycomb-like pattern was tested for various filters and compared to other methods. The USAF test chart was used to evaluate the method’s efficacy in removing the desired pattern while simultaneously preserving image details and compute quality metrics. s. However, super-Gaussian filtering achieves good results in smoothing as well as detail preservation. The small standard deviation of q demonstrates that there are only slight differences in quality among the images of the same HS cube. Similar to the super-Gaussian filter, Hanning, ideal and smoothed ideal filtering achieve high values for s, but low values for r. In addition, ideal and smoothed ideal filtering leads to ring artifacts on the edges of the image.The evaluation reveals that our method, in combination with the various filters described above, removes the pattern by different degrees. In detail, super-Gaussian filtering delivers the best results for pattern removal . This study is the first of its kind to attach an HS system to a flexible endoscope to detect laryngeal cancer. As a prerequisite for proper detection, we proposed a pre-processing method to improve the HS data for further analysis. This feasibility study was conducted with the overall objective of creating a spectral distinction between cancerous and non-cancerous tissue in HS data by utilizing the advantages of the flexible endoscope. The major disadvantage to this system, however, is that honeycomb-like patterns appear and degrade the quality of the HS images. This can be attributed to the fiber optic cables, which make the endoscope flexible yet prevent successful classification without a correction of the images’ degradation. Therefore, we developed a method specifically for the system in order to overcome this drawback, in which the spectral domain is filtered by a selective window that automatically adjusts to the size of the periodic noise. Our method clearly shows better results in terms of pattern smoothing and preserving detail compared with other methods. Using the pre-processed HS data as an input for an unsupervised and supervised classification instead of the raw HS cubes improves classification performance and the ability to locate the cancerous tissue, as revealed by the visual inspection. However, clinical studies are necessary to optimize cancer detection.


DEEP-SEE framework that jointly exploits computer vision algorithms and deep convolutional neural networks (CNNs) to detect, track and recognize in real time objects encountered during navigation in the outdoor environment. A first feature concerns an object detection technique designed to localize both static and dynamic objects without any a priori knowledge about their position, type or shape. The methodological core of the proposed approach relies on a novel object tracking method based on two convolutional neural networks trained offline. The key principle consists of alternating between tracking using motion information and predicting the object location in time based on visual similarity. The validation of the tracking technique is performed on standard benchmark VOT datasets, and shows that the proposed approach returns state-of-the-art results while minimizing the computational complexity. Then, the DEEP-SEE framework is integrated into a novel assistive device, designed to improve cognition of VI people and to increase their safety when navigating in crowded urban scenes. The validation of our assistive device is performed on a video dataset with 30 elements acquired with the help of VI users. The proposed system shows high accuracy (>90%) and robustness (>90%) scores regardless on the scene dynamics.In this paper, we introduce the so-called  Although object tracking represents a fundamental problem in the computer vision community due to the wide range of related applications , it is still an open issue of research. Despite significant progress achieved in the last decade, tremendous challenges still exists in designing a robust object tracker able to handle important changes in viewpoint, object motion, light variation, pose variation, object occlusion or background clutter.DEEP-SEE, based on computer vision algorithms and offline trained deep convolutional neural networks. The proposed framework is integrated within an assistive device, dedicated to visually impaired (VI) users. The major contributions of the paper are summarized below.In this paper, we propose a novel joint object detection/tracking/recognition framework, called A first feature concern the detection and recognition module, which is based on the YOLO (You Only Look Once)  algorithThe second contribution, which represents the methodological core of the paper, concerns the object tracking approach. The proposed method uses two convolutional neural networks (CNNs) trained offline with both motion and visual patterns. We show how an adaptive visual appearance model can be constructed on-the-fly, without the need of any on-line learning procedures. In addition, we introduce an occlusion detection and handling strategy. In this way, the method is able to handle important or complete object occlusions, object movement or camera drift.DEEP-SEE navigational assistant. The YOLO algorithm has been here extended and trained with novel categories specific for an assistive device dedicated to the VI. We make sure that the system can localize, in real-time, both static obstructions  and dynamic objects . The proposed system is able to acquire information from the environment, process, interpret it and transmit alert messages to the VI user in order to avoid dangerous situations or possible collisions. Let us underline the real-time constraints that are necessary to consider and achieve within the framework of this VI-related application which strongly condition the whole processing chain and impact the technical choices retained.The third contribution is the integration of the object detection/tracking/recognition framework in a novel VI-dedicated assistive device, denoted by DEEP-SEE, and we describe the main steps involved: obstacle detection, tracking and recognition. DEEP-SEE navigational assistant, with the proposed hardware/software architecture, main features and performance evaluation. We show that it is possible to obtain high accuracy and recall rates with a computationally efficient approach that notably achieves real-time performances on wearable devices. Finally, The rest of the paper is organized as follows. In Since more than two decades ago, numerous object detection and tracking algorithms have been proposed. In this section, we focus our attention solely on recent discriminative algorithms that have been introduced in the last couple of years. Modern approaches  dedicateIn , the scaIn , an objeThe TricTRACK  system uIn , the KCFDifferently, a color-based object tracker denoted DAT is introduced in . The sysThe tree-structured CNN (TCNN) tracker recently proposed in  uses mulThe Staple single object tracker introduced in  is basedFinally, let us mention the winner of the international VOT 2016 tracking contest , which iHowever, the neural networks are complex to train, require important processing resources and are very slow in the test phase. Thus, in terms of speed, the above-mentioned trackers range from 0.8 fps to 10 fps, while the top performing algorithm in the evaluation tests runs at 1 fps on GPU. Hence, such trackers are penalized when considering practical applications. To deal with this drawback, in , the GOTIn this paper, we introduce an integrated single object detection, recognition and tracking approach that integrates both visual and motion cues to perform accurate object position estimation. The key principle consists of alternating between tracking using motion information and adjusting the predicted location based on visual similarity.Similar to any tracking procedure, we first need an object detection approach that can provide an initialization of the tracking process. In our work, we have adopted the YOLO approach, described in the following section.We have adopted an initial object detection and recognition procedure based on the YOLO  algorithThe YOLO framework encodes during training and testing the contextual information about the object class and its appearance. Due to the global object representations, the system is less likely to return false alarms or missed detections when applied to novel/unexpected video instances.DEEP-SEE system cannot identify descending stairs or holes in the ground, which are difficult to distinguish when using monoscopic vision. To this purpose, the utilization of depth cameras would be more appropriate, but this issue will not be addressed in the paper. The CNN has been trained by using the same training dataset as the one presented in  interval that is automatically generated by the VOT’2016 toolkit.To be able to evaluate separately the influence of the various conditions that can appear in the videos , we have adopted the globalization protocol considered by the VOT’2016 challenge. Here, the videos of the database are split into smaller videos, with a size of Ns frames, we obtain the expected average overlap By averaging the average overlap measure The obtained results are generated using the toolkit set up by offered by the VOT committee. The overall expected overlap scores for the proposed tracker and for 13 other relevant tracking systems, denoted C-COT , StaplerIn addition, As can be observed, the proposed approach shows the highest robustness when important variations of the light intensity are presented on the scene. When the target is occluded or has an important motion, we obtain equivalent performances with the C-COT method. However, the proposed tracking procedure is more sensitive to important camera motion than state of the art systems such as Staple, C-COT or TCNN.The accuracy-robustness score plot is presented in The major improvement of the proposed method is the processing speed, which is about four times faster than the analyzed trackers that return similar performances scores. This result is explained by the fact that our tracker avoids any on-line learning stage, in contrast with C-COT, where the object appearance model is learned online by a CNN. On the opposite side, when compared with the GOTURN algorithm that also uses only offline trained convolutional networks and works in real-time, our method shows a significant improvement in accuracy (more than 10%). In addition, in terms of the number of failures, our approach shows an improvement with more than 75%.From the experimental results presented in To provide a qualitative insight of the behavior of the proposed tracking approach, the following section presents the results obtained on some challenging video sequences.In In the Racing and Butterfly image sequences, all trackers are able to follow the object of interest. However, because of the relatively important change in object appearance, the state of the art trackers  cannot adaptively modify the object bounding box in order to capture its entire shape. Thanks to the continuous update of the object appearance model and to the intelligent adjustment of the object’s bounding box, our method is able to perform more accurately.For the Bmx video, we can observe that all the state of the art trackers tend to drift because the model is not correctly updated in time. In the Glove sequence the object of interest suffers an important change in shape and undergoes an unnatural motion. In addition, two visually similar objects are present in its vicinity . Because all state of the art CNN trackers learn on-line the object features, when a new, visually similar element enters the scene, the tracker tends to shift to the novel objects.In the case of the Soldier video sequence, the state of the art trackers  are updated with sub-optimal object instances and tend to drift in time. For our method, due to the occlusion identification and handling stage, this behavior rarely occurs.DEEP-SEE visually impaired navigational assistant is detailed in the following section.Let us now describe the application considered in this paper: the Recent statistics, relative to people with visual disabilities, published by the World Health Organization (WHO)  in AugusThe visually impaired people are facing numerous challenges when performing daily activities  which can severely impact the quality of their private and professional life. Even though modern technology has found its way into most aspects of people life, in the context of VI, the traditional white cane or the trained walking dog remains the most common mobility aids .Within this context, the development of an autonomous assistive device designed to facilitate the user cognition over the environment and to increase the VI autonomy and safety when navigating in novel places is a crucial challenge.In the state of the art, various electronic travel aids based dedicated to people with visual disabilities were introduced based on either sensor networks ,28,29,30In this section, we introduce a novel assistive device, which exploits the tracking methodology introduced in processing speed , robustness (the system should not by influenced about the scene dynamics or lighting conditions), coverage distance (the maximum distance between the user and an object for which detection can be performed), object type , portability , and friendliness (the system should be easy to learn without any training).When designing the proposed architecture the following set of requirements were considered: DEEP-SEE system is composed of a regular smartphone device  functioning as a video acquisition device, Bluetooth bone conducting headphones  used to transmit the acoustic warning messages to the VI user, an ultrabook computer  used as backpack processing unit, a smartphone waist belt and a backpack. The total weight of our system is inferior to 2 kg.DEEP-SEE architecture adopts a modular framework, composed of the following three units: object detection and recognition .Finally, regarding the user interaction aspects, The acoustic feedback is designed to improve cognition of the visually impaired people over the outdoor environment by transmitting warning messages regarding the recognized obstacles (either static or dynamic) situated in his/her near surrounding. The major constraint that needs to be taken into account is to transmit warning messages fast enough, so that the VI user can walk normally while avoiding dangerous situations. We decided to use bone conduction headphones, in order to satisfy the hands free condition imposed by the VI people and enable the user to keep hearing other external sounds from the surroundings.Most assistive devices dedicated to VI people existent on the market  try to eDEEP-SEE, the semantic interpretation of the recognized objects and the effective transmission of information to VI people are the key elements. The acoustic feedback module is designed to be intuitive and does not require extensive and laborious training phases to understand the technical functionalities.In the context of The key principle that has been adopted respects the VI user recommendations and fulfills the following two rules: (1) the user should not be overwhelmed with too many and useless alerts; and (2) the system should minimize all interference with other senses such as auditory and touch.We decided to use verbalized messages rather than a beeping strategy (with sound patterns transmitted on different frequencies) so that the user is informed about the presence of an approaching obstacle, its type or degree of danger, time to collision and the physical position of the detected obstruction.For each detected obstacle, we determine its degree of danger by estimating the object’s position relatively to the video camera. We propose to use a trapezium of interest defined in the space of the video frame in order to define the user’s proximity area. The dimensions of the trapezium (in pixels) are presented in E) of 0.8–1 m (which corresponds to the height of the waist of a medium person between 1.60 and 1.80 m height). Then, we estimate the distance Dist between the user and the first visible area on the floor as:The smartphone is attached to the VI using a waist belt at an average elevation  between the user and the obstacles detected in the alert area  as descrWorldDist value corresponds to the distance to the user of an object that is entering in the alert area. In our setting, with the above-considered parameters, the WorldDist value is of approximately 6 m. We consider this value as a reasonable choice to decide that an object becomes too close to the user and needs to be notified as urgent (U). Thus, an object obstacle is marked as urgent if it is situated in the proximity of the blind/visual impaired person inside of the trapezium of interest. For objects situated within the trapezium of interest we estimate an average collision time of 1–2 s. Otherwise, if it is located outside the trapezium, the obstacle is categorized as non-urgent or normal (N). By employing the area of proximity, we can prevent the system to launch acoustic warning messages for all the detected objects existent in the scene. We adopted this strategy in order not to overwhelm the user with too much information. However, within the trapezium of interest multiple static/dynamic obstacles can be encountered. The recognized objects are further analyzed in order to differentiate between the various elements and to determine their degree of danger. To this purpose, we exploit the object recognition process described in The approaches the VI user, even if not located within the trapezium of interest, the system will generate an acoustic warning and will inform the VI user about its presence instead of the existence of a static obstacle or a pedestrian near him/her. Moreover, static obstacles or pedestrians that are not located in the alert area do not generate any warning.As it can be observed from This strategy is illustrated in Finally, to provide information regarding the relative position of the detected object, the acoustic warning messages are encoded in stereo using either right, left or both channels simultaneously. Thus, when the obstacle is situated on the left (right) side of the subject the message is transmitted on the left (right) channel of the bone conducting headphones. For frontal objects, both channels are used simultaneously to transmit the warning. In addition, to avoid confusing the VI user with too much information, the warning messages are sent with a frequency rate inferior to two seconds, regardless of the scene dynamics.DEEP-SEEoverall  navigational assistant has been tested on multiple, complex outdoor environments by using the video dataset of [The taset of  that wasBecause the dataset was recorded with the help of visually impaired people in real urban scenes and not in simulated environments, the videos are trembled, cluttered and include important camera and background movement. In addition, various types of dynamic and static objects are present. In P), Recall (R) and F1 score (F1), defined as:Using the ground truth data, the performance of the obstacle detection and classification modules are evaluated using traditional objective parameters such as Precision (taset of  where thWe have compared the proposed approach with our previous method introduced in , which aThe obtained results highlight that the average precision and recall scores are around 90% which means that the proposed framework is able to detect and classify, with high accuracy, both dynamic and static obstacles. When compared with our previous method in , our sysIn terms of computational speed, when implementing the whole framework on a regular ultrabook computer, running on an Nvidia GTX 1050 GPU, the average processing speed is around 20 fps.In DEEP-SEE prototype is able to correctly detect, track and recognize various object encountered by the VI user during the outdoor navigation. In addition, the system proves to be robust to the scene dynamics or to various changes in the light intensity or object appearance.From the experimental results presented in DEEP-SEE, jointly exploits computer vision algorithms and deep convolutional neural networks and is designed to detect, track and recognize, in real time, both static and dynamic obstacles encountered during navigation in crowded scenes. The system does not require any initial a priori knowledge about the object size, shape or initial position. The semantic interpretation of the recognized objects and the effective transmission of information to a VI people are the key elements. The degree of danger of the detected objects is also evaluated by estimating the object’s relative position with respect to the to the VI user. The user feedback is transmitted as a set of acoustic warnings, through bone conducting headphones.In this paper, we have introduced a novel navigational assistant prototype designed to increase the mobility and safety of visually impaired people when navigating in outdoor environments. The proposed framework, denoted From a methodological point of view, the core of the approach relies on a novel object tracking method that uses two CNNs for taking into account both motion patterns and visual object appearances. We evaluated the proposed system on two video datasets: the Visual Object Tracking 2016 database composed of 60 videos. The experimental results obtained on the challenging VOT 2016 datasets demonstrate that the proposed detection and tracking system is capable to obtain state of the art performances by alternating between two CNNs, respectively, trained with motion and visual patterns. In terms of accuracy and robustness, the tracking system returns similar performances with the winner of the VOT2016 challenge while being four times faster.DEEP-SEE prototype for VI assistance purposes, we have considered the video dataset of [Concerning the evaluation of the taset of  recordedtaset of .For further work and developments, we propose to integrate the proposed framework in a complete platform that allows additional functionalities, such as crossings detector, face recognition, shopping assistance, and guiding system, that may help a VI user to reach a desired destination. Next, we propose extending the proximity region and define two areas of interest, both with a trapezoidal shape, in the near surrounding of a user: one situated on the walking path and the other at the head level. Using this strategy, we can launch acoustic warning messages for obstacles situated at the head level, such as tree branches, hanging signs or other dynamic flying objects, and not solely for object located on the ground.In addition, we envisage testing the system in real-life scenarios with actual visually impaired users. Finally, let us observe that elderly VI users may also present hearing disabilities. In this case, an alternative, haptic encoding scheme should be envisaged instead of the acoustic warning approach.


This requirement is challenging, however, for applications in the plant phenotyping field, where available datasets are often small and the costs associated with generating new data are high. In this work we propose a new method for augmenting plant phenotyping datasets using rendered images of synthetic plants. We demonstrate that the use of high-quality 3D synthetic plants to augment a dataset can improve performance on the leaf counting task. We also show that the ability of the model to generate an arbitrary distribution of phenotypes mitigates the problem of dataset shift when training and testing on different datasets. Finally, we show that real and synthetic plants are significantly interchangeable when training a neural network on the leaf counting task.Deep learning presents many opportunities for image-based plant phenotyping. Here we consider the capability of deep convolutional neural networks to perform the leaf counting task. Deep learning techniques typically require large and diverse datasets to learn generalizable models without providing The online version of this article (10.1186/s13007-018-0273-z) contains supplementary material, which is available to authorized users. Non-destructive, image-based plant phenotyping has emerged as an active area of research in recent years. This is due in part to a gap in capability between genomics and phenomics, as well as the complexity of genotype-to-phenotype mapping . The abiUnlike engineered computer vision pipelines, deep neural networks learn a representation of the data without image parameters specified by hand. This makes them potentially more robust to different types of variations in the image data, as the network can adapt to be invariant to such differences. However, the transition from hand-engineered computer vision pipelines to deep learning is not without limitations. While so-called “deep” networks have the representational capacity to learn complex models of plant phenotypes, the robustness of these representations relies on the quality and quantity of the training data. In most vision-based tasks where deep learning shows a significant advantage over engineered methods, such as image segmentation, classification, and detection and localization of specific objects in a scene, the size of the dataset is typically on the order of tens of thousands to tens of millions of images . This alUnfortunately, datasets of plant images, labeled with corresponding phenotypic data, are not yet available on a large scale due to the considerable expense involved in collecting and annotating this type of data. In addition, any supervised machine learning method, including deep learning, requires that the data used to train the model is representative of the data used at test time. Plant phenotyping tasks are vulnerable to such problems with incomplete training data due to the difficulty of generating a dataset in which a comprehensively wide range of phenotypes are represented.The small size of existing plant phenotyping datasets, the expense of generating new data, and the limitations of naturally-generated datasets motivate the use of an alternative source of data to train deep networks for plant phenotyping tasks. For this purpose we propose the use of synthetic plants—images of computer-generated plant models—to augment datasets of plant images or to be used alone as a large and rich source of training data. Compared to generating new data using real plants, once a model is developed, the generation of new data is essentially without cost. Moreover, models can be parameterized to generate an arbitrary distribution of phenotypes, and ground-truth phenotype labels can be automatically generated without any measurement errors and without any human effort or intervention.Deep learning refers to a broad category of machine learning techniques, which typically involve the learning of features in a hierarchical fashion. Such techniques have been shown to be successful in many types of computer vision tasks, including image classification, multi-instance detection, and segmentation . Deep len and m are the image height and width in pixels, and 3 is the number of color channels. In a convolutional neural network, image features are extracted from a volume by a series of convolutional layers, which learn collections of filters. These filters are applied pixel-wise in strided convolutions (in a sliding window fashion) over the input volume, where the dot product between the filter weights and each spatial location (assuming a stride size of one pixel) in the input volume creates an activation map. Similarly, the output volume of the convolutional layer is an p and q are some spatial extents, and k represents the number of filters in the layer (and therefore the number of filter activation maps). As with regular neural network layers, a non-linear function is applied to the activations.We trained Convolutional Neural Networks (CNNs) using the open-source Deep Plant Phenomics platform  to perfoIn order to construct a hierarchical representation of the data, many convolutional layers are alternated with pooling layers, which downsample the spatial size of the input volume. The output of the final convolutional layer  represents a learned representation of the original input data. This learned representation is used by fully-connected neural network layers to perform classification or regression, and all of the network’s parameters are learned simultaneously during training. A more detailed overview of CNNs for plant scientists is provided in , and reaFor some applications, the construction of large data sets of labeled images can be facilitated by crowd-sourcing images freely available on the Internet . Unfortumisspecified and mis-predict the number of leaves. In technical terms, the learning process infers a conditional model P(y|x): the conditional distribution of the outputs given the inputs. Differences between training and testing data can result in two related problems known as covariate shift, where P(x) changes between training and testing, and dataset shift, a different joint distribution P of the outputs and inputs in the test data, compared to that in the training data. This problem is common in machine learning and can be difficult to mitigate [a priori information typically do not have to model the conditional distribution explicitly. The problem of dataset shift is almost inevitable when using supervised learning for plant phenotyping tasks, due to the limitations of generating new plant phenotyping datasets. It is not possible to specify the domain of phenotypes to be represented in the data, and so this limitation will tend to expose problems of dataset shift when using models of phenotypes learned from this data. We investigate the use of computational plant models to mitigate this problem.Even for large training datasets, the network can still fail to properly recognize phenotypes if the distribution of testing data differs significantly from that of the training data. In the case of leaf counting, the distribution of leaf numbers in the training data must be similar to that of the testing data: if the rosettes used for training have significantly fewer leaves than the rosettes used for testing, the learned model will likely be mitigate . AvailabComputational modeling has become an inherent part of studies of plant physiology, development, architecture, and interactions with the environment. Diverse concepts and techniques exists, applicable to construct models at spatio-temporal scales ranging from individual cells to tissues, plant organs, whole plants, and ecosystems –14. The The use of synthetic or simulation data has been explored in several visual learning contexts, including pose estimation  as well The specialized root system models used by Benoit et al.  and LobeArabidopsis thaliana rosette can be used either in conjunction with real data, or alone as a replacement for a real dataset, to train a deep convolutional neural network to accurately count the number of leaves in a rosette image. We also discuss how the concept of model-based data augmentation may extend to other plants and phenotyping tasks.In the present work, we seek to demonstrate that realistic models of synthetic plants are a sufficient replacement for real data for image-based plant phenotyping tasks. We show that a model of the https://figshare.com/articles/SATLC-28-09-17_zip/5450080.For the images of real plants used in the leaf counting task, we use a publicly available plant phenotyping dataset from the International Plant Phenotyping Network (IPPN),When training on synthetic images and testing on real images  using normally distributed random variables. Specifically, for each leaf along the stem, we multiplied n) representing the apex, the parameter n is the node number. We simulated the development of the plant by a set of rewriting rules, which specify the fate of each module (component) over an increment of time. An apex, for instance, produces a new internode and new leaf at regular time intervals. To account for diversity of rosettes, we generated 1000 images with a random variation. Details of our implementation are given in the Additional file Our model was implemented using parametric L-systems, in which each component of a plant  has a corresponding module with associated parameters . For exaTo validate the use of models with deep learning, we conducted three leaf counting experiments using images of both real and synthetic Arabidopsis rosettes. The mean absolute count difference, and the standard deviation of absolute count difference, were measured in each experiment. The experiments were conducted as follows:This experiment tested the usefulness of synthetic plants in augmenting the Ara2013-Canon dataset of real plants for the leaf counting task. For this purpose, we generated a set of one thousand synthetic rosettes (S2) and added them to the training set. The model’s background was set to a brown color approximating the soil in the real dataset. Using synthetic rosettes to augment the training set, we observed a reduction of approximately 27% in the mean absolute count error . The model was then tested on the Ara2012 dataset (with a range of between 12 and 20 leaves) and the Ara2013-Canon dataset (between 5 and 13 leaves). A synthetic training set which is easy for the network to fit will result in poor generalization due to overfitting; in order to introduce more variance to the synthetic data with the goal of reducing overfitting, the model’s background was set to either a soil color or a random color in RGB space  and testing it on synthetic plants (S2) containing the same range of leaf numbers, or vice versa: training on the set S2 and testing on Ara2013-Canon. A small error value in this experiment signifies that the model is a suitable stand-in for real plants for the leaf counting task. Statistics are provided for both cases Table , as wellots Fig.  show theDeep learning models, including the deep CNNs used in the experiments presented here, have a large capacity for fitting the training data. This is essential to their learning ability, but also makes them susceptible to overfitting in the case of small datasets, or large datasets with an insufficient level of variation. Therefore, it is important to consider how to introduce as much variation as possible into the model and the scene. For example, we found that generalization improved when plants were randomly scaled, with the ratio of the plant diameter to the size of the entire image varying between 1:1 and 1:2. This helped prevent the network from using the number of green pixels as a proxy for the number of leaves, which could be a viable strategy if the model lacked enough variance in leaf size. Other considerations include varying the contrast between background and foreground pixels. Such variations in the model, the scene, as well as secondary image-based augmentations such as modifications of the brightness and contrast all contribute to preventing overfitting.Comparing the counting errors during training and testing, we observed that their difference  is larger for real data than for synthetic data Fig. . This meAnother consequence of overfitting is the network’s tendency to discriminate between different types of data. In tests with both real and synthetic data, if these datasets had different leaf distributions, the network would learn to map each type of data to an individual output distribution, with a detrimental effect on generalization performance. This means that the use of synthetic data in conjunction with real data is only advisable if the distributions of phenotypes of the real and synthetic data overlap. Although this could be seen as a disadvantage, we have also shown that the use of synthetic data alone is sufficient and avoids this effect.We observed that models which are not sufficiently realistic resulted in degraded performance compared to more accurate models. For example, an initial rosette model in which all leaves were assumed to be of the same size showed significantly lower interoperability with the images of real rosettes. Taking into account not only the differences in leaf size, but also in shape as a function of their position , as wellFuture directions for research could further explore the relationship between models trained on real data and those trained on synthetic data, including techniques such as transfer learning. Using a feature extractor learned on synthetic data and re-training a regressor with these features may shed light on differences in learned representations between the two types of data.In summary, the results presented in this paper show promise for the use of models in image-based plant phenotyping tasks. The existing body of work on L-system modeling of plants is extensive, with models available for many different species. These existing models are well positioned to take the results demonstrated here on Arabidopsis forward towards other applications. One potentially important application area is the modeling of entire plots of crops. A simulated plot of plants could potentially make it possible to train algorithms for detecting biologically meaningful traits such as flowering time or response to stress with a reduced number of real (annotated) crop images. Other directions for future work could include augmentation using synthetic data for other supervised learning problems, such as leaf segmentation. Other applications, such as disease detection, would be possible if future plant models were able to model such phenomena.We applied a computer-generated model of the Arabidopsis rosette to improving leaf counting performance with convolutional neural networks. Using synthetic rosettes alongside real training data, we reduced mean absolute count error with respect to results obtained previously using only images of real plants . We also


Bacterial sRNA-mediated regulatory networks has been introduced as a powerful way to analyze the fast rewiring capabilities of a bacteria in response to changing environmental conditions. The identification of mRNA targets of bacterial sRNAs is essential to investigate their functional activities. However, this step remains challenging with the lack of knowledge of the topological and biological constraints behind the formation of sRNA-mRNA duplexes. Even with the most sophisticated bioinformatics target prediction tools, the large proportion of false predictions may be prohibitive for further analyses. To deal with this issue, sRNA target analyses can be carried out from the resulting gene lists given by RNA-SEQ experiments when available. However, the number of resulting target candidates may be still huge and cannot be easily interpreted by domain experts who need to confront various biological features to prioritize the target candidates. Therefore, novel strategies have to be carried out to improve the specificity of computational prediction results, before proposing new candidates for an expensive experimental validation stage.rNAV 2.0, for detecting and filtering bacterial sRNA targets for regulatory networks. rNAV is designed to cope with a variety of biological constraints, including the gene annotations, the conserved regions of interaction or specific patterns of regulation. Depending on the application, these constraints can be variously combined to analyze the target candidates, prioritized for instance by a known conserved interaction region, or because of a common function.To address this issue, we propose a new visualization tool The standalone application implements a set of known algorithms and interaction techniques, and applies them to the new problem of identifying reasonable sRNA target candidates.The online version of this article (doi:10.1186/s12859-017-1598-8) contains supplementary material, which is available to authorized users. Bacterial small RNAs (sRNAs) are crucial regulators that often act to transmit environmental signals when bacteria meet stressful growth conditions –3. TheseFocusing on sRNAs acting as negative or positive post-transcriptional regulators by base-pairing mRNAs, the identification of their targets is challenging and needs a better understanding of the topological and biological constraints behind the formation of sRNA-mRNA interactions. Moreover sRNA can have several targets, and some mRNAs may be hybridized by several sRNAs forming SIM (Single-Input Module) or DOR (Dense Overlapping Regulon) motifs of regulation .In the omics era where the amount of available data is still increasing, one of the challenge facing the use of predictive bioinformatics tools is to be able to make use of the huge number of predicted results. Even with the most sophisticated bioinformatics target prediction tool, the large proportion of false predictions may be prohibitive to enable further analysis. To deal with this issue, sRNA target analysis can be run from the resulting gene lists given by RNA-SEQ experiments when available. However, the number of resulting target candidates may be still huge and cannot be easily interpreted by biologist experts who needs to confront various biological features to prioritize the target candidates.Pathway analysis tools intend to deal with these issues. Accordingly, one of the most famous software, Cytoscape , supportrNAV 2.0 software, the first available release of the prototype presented in [In this context, we introduce the ented in  as a proCombining bioinformatics and visualization help biologists in the analysis of such sRNA-mediated regulatory networks by supporting the exploration and visualization of the huge number of predicted sRNA targets produced by existing bioinformatics tools .Information Visualization has now been established as a fruitful strategy to tackle the problem posed by the abundance of information .rNAV 2.0, has to deal with is the representation of the entire network as it eases the identification of the main trends in the data and therefore to guide the user during his/her exploration and to focus his/her attention on interesting parts.Main aspect a visualization system, such as rNAV 2.0 have been mostly developed in C++ but also in Python and is provided under LGPL. The software is based on Tulip framework [ramework . Tulip iramework , Tulip aModel-View-Controller architecture of Tulip allows rNAV to support multiple and synchronized views. Any interaction on a view (e.g. selection of an element) implies the automatic update of all views displaying this data.The The sRNA-mediated regulation network is modelized by a bipartite graph where nodes represent either sRNAs or mRNAs and interactions are given by edges linking an RNA to another.rNAV integrates two well-known bioinformatics tools: ssearch [To predict interactions between sRNAs (fasta file) and mRNAs (embl/genbank file),  ssearch , by reve ssearch . The fun ssearch , a singlmatrix based diagrams  and node-link diagrams (where nodes are represented by a glyph and an edge by a link between the corresponding nodes). While matrix based diagrams are particularly useful for the visual exploration of large and dense graphs as it allows to better perceive local densities, it does not seem to be the best option in the case of sRNA-mediated regulatory networks. Indeed, it would be difficult to emphasize edge’s attributes  with such a visualization technique. In rNAV, node-link diagrams have therefore been preferred over matrix based diagrams as links between nodes provide a relatively large area to show edge’s attributes .Finally, rNAV integrates four categories of algorithms: Filter, Coloration, Layout and Calculation. While algorithms of the Coloration and Layout categories respectively allow to color RNAs and putative interactions according to some measures and to lay the network out in the plan, Filter and Calculation categories contain algorithms to reduce the scope of the analysis. Algorithms of these categories allow to focus the exploration on sub-parts of the network and therefore facilitate the understanding of biological processes and the investigation of the putative cellular organization. Using Filter algorithms, the user can specify RNAs of interest according to biological knowledge , but also according to topological motifs they belong to . In addition that category includes algorithms to cluster for each RNA its putative interactions. That clustering algorithms group putative interactions according to some distance between them. rNAV actually integrates two distances measuring either how close their interacting regions are or how similar their functional annotations are. Then using the distance, the MCL clustering [To guide the user in his/her exploration, R motifs ). Algoriustering  is perforNAV supports the edition of algorithm pipelines as an ordered sequence of algorithms of any category. In that sequence, the output of an algorithm is the input of the next one. For a reproducibility purpose, our tool also supports saving of algorithms pipelines. These saved pipelines are then reachable from the Action list button in the user interface.To extract interesting candidates from the list of all the putative interactions, the user often needs to perform several and successive filters. rNAV integrates classical zoom and pan and box-selection interaction tools as well as a dedicated functionality that displays on demand detailed informations about an RNA of interest . These rectangle as well as box-selection can also be used to select neighbors of the focused RNA.see Fig. . The latrNAV. First, rNAV integrates an exploration tree view displaying the algorithms that have been run during the analysis . As prerNAV, we focused on these two sRNAs, FnrS and RyhB. In this context, we performed an analytical strategy with the objective to propose new target candidates. In our strategy, we exploit the information that can be collected from the validated targets to prioritize the resulting gene lists given by the high-throughput sequencing results of sRNA mutants.In addition, the regulation can be coordinated by several sRNAs in response to crossed perturbations of the cell. Again, an illustration can be given with the RyhB and FnrS sRNAs which share four mRNA validated targets. These four genes are involved in different metabolism processes, covering the energetic metabolism  and the response to external stress (marA and nagZ). To go further and analyze the sRNA regulation using rNAV, derived from both a multi-fasta file for the sRNA sequences and an embl file for the Escherichia coli genome, using adjustable parameters for the 5’ UTR extraction of each mRNA sequences. Interactions were computed using IntaRNA [rNAV  to then apply different annotation filtering stages.To illustrate the usefulness of rNAV: they correspond to the two first branches in the Fig. The following stages of this analysis are represented by the exploration tree view of At last, considering the intersection of both filtering stages, 15 targets are common and may be of particular interest for designing experimental protocols with the objective to identify new targets that are both involved in the iron storage/absorption and in the aerobic/anaerobic energy metabolism.In a first work dedicated to the study of FnrS , the useFocusing on these data from the beginning of the analysis, the following stages correspond to the third and fourth branches of the exploration tree in the Fig. Escherichia coli with the two sRNAs, FnrS and RyhB.We performed an analytical pipeline based on the sRNA-mediated regulatory network of rNAV a list of 15 targets. Making use of this small list of genes, we have carried out a deeper study to prioritize few genes according to a coordinated regulation hypothesis. For instance: Three targets are components of the NADH:quinone oxidoreductase I  and may required a coordinated regulation (SIM-like motif).Wolinella succinogenes the membrane-bound NapGHF complex and have shown its multi-functional role in the maturation of napA precursor. A common regulation of these proteins may be of interest to demonstrate their related actions with the periplasmic nitrate reductase napA of Escherichia coli which plays an important role during the anaerobic growth in low-nitrate environments [Four other targets are parts of the nap operon related to the periplasmic nitrate reductase . Moreover, Kern et al.  have sturonments .The two Fumarase A and B (fumA and fumB) are also parts of the targets and are two of the three fumarase isozymes participating in the TCA cycle. It has been shown that the expression of each of these two isozymes was maximum in aerobic or anaerobic conditions .First, RyhB, whose expression is induced by a low level of iron, is known to interact with iron-containing enzymes and iron-transporting proteins . From thInterestingly, all of these new targets have also been identified using ribosome profiling (Ribo-seq) in a recent work demonstrating the new great applications of these experimental approaches for the genome-scale identification of sRNA targets . Using trNAV and using the MCL algorithm for the 31 targets given by [Second, FnrS is involved in the regulation of enzymes required in the aerobic-anaerobic transition when the cell is undergoing an oxidative stress . Of partgiven by , 37, thirNAV improved our exploration of the sRNA-mediated networks. Very quickly, rNAV can help to realize a deep study that seems very complex at first. This advantage is illustrated by the very small number of steps that were designed in rNAV Additional file 2Data description. Data description and availability, and parameter settings used in this study. (PDF 101 kb)