As an emerging type of Internet of Things (IoT), multimedia IoT (MIoT) has been widely used in the domains of healthcare, smart buildings/homes, transportation and surveillance. In the mobile surveillance system for vehicle tracking, multiple mobile camera nodes capture and upload videos to a cloud server to track the target. Due to the random distribution and mobility of camera nodes, wireless networks are chosen for video transmission. However, the tracking precision can be decreased because of degradation of video quality caused by limited wireless transmission resources and transmission errors. In this paper, we propose a joint source and channel rate allocation scheme to optimize the performance of vehicle tracking in cloud servers. The proposed scheme considers the video content features that impact tracking precision for optimal rate allocation. To improve the reliability of data transmission and the real-time video communication, forward error correction is adopted in the application layer. Extensive experiments are conducted on videos from the Object Tracking Benchmark using the H.264/AVC standard and a kernelized correlation filter tracking scheme. The results show that the proposed scheme can allocate rates efficiently and provide high quality tracking service under the total transmission rate constraints. The Internet of Things (IoT) is a global infrastructure for the information society, enabling the advanced services by interconnecting physical and virtual things based on the existing and evolving interoperable information and communication technologies . The IoTA lot of existing research has considered video coding and transmission schemes to improve the human visual quality ,13,14,15To improve the service quality of video analytics in surveillance networks, plenty of researchers have worked on improving the performance of video analysis schemes in the computer vision field. Detection, recognition and tracking of human/vehicles are the main analysis tasks of smart surveillance systems ,18,19,20The main challenges for the schemes which chose to upload videos and analyze videos on a server, are the limited transmission resources and network state fluctuations. In ,27, videOnly a few researchers have studied the effect of video compression and transmission parameters on video analysis performance. In , a salieIn a realistic surveillance network for vehicle tracking, all surveillance videos are uploaded to server for video analysis purposes. When video sequences are uploaded via a wireless network, multiple camera nodes compete for the transmission resources which are limited by the bandwidth. Therefore, considering the effect of video content and video quality degradation caused by compression and transmission error, a joint source and channel rate allocation scheme is proposed in this paper to optimize performance of KCF tracking scheme in server. The main contributions of this paper can be summarized as follows:(1) Content-aware tracking precision prediction model. The features of the video content, which are not considered in existing prediction models, have a great impact on tracking precision. Based on the effect of video compression and video content on KCF tracking scheme, we take the bits per pixel, luminance values and complexity of the video content as key factors in the process of HOG extraction and object matching in KCF scheme. Then, based on the obtained relationship between these factors and the tracking precision, we establish a content-aware tracking precision prediction model by using curve-fitting method. (2) Tracking-precision-driven joint source and channel rate allocation scheme. Due to the limited wireless resources and the unreliability of wireless networks, it is important to design a joint source and channel rate allocation scheme to avoid degradation of tracking precision caused by video compression or transmission errors. We optimize the rate allocation for camera sensors by a tracking-precision-driven optimization problem. Based on our content-aware tracking precision prediction model, the optimization problem is formulated to maximize the product of tracking precision of each surveillance video in MIoT server under the constraint of the total transmission rate in the wireless network. To improve the reliability of data transmission and the real-time of the video communication, we adopt forward error correction (FEC) in the application layer. The rest of this paper is organized as follows: The proposed structure of the MIoT system for vehicle tracking, which consists of three main components, namely a mobile camera node, wireless network and MIoT server, is shown in Mobile camera node: This can be a mobile sensor device like a car recorder, smart phone, or an unmanned aerial vehicle, etc. In every group of picture (GOP) period, the camera node has two tasks: (1) The camera node calculates and uploads video content features of current uncoded GOP to a rate allocation module in the server; (2) the camera node encodes and uploads the GOP according to the rate allocation policy from the MIoT server. The H.264 encoder and Raptor  FEC encoWireless Network: A wireless wide area network (WWAN) is chosen for video transmission due to the random distribution and mobility of camera nodes. When uploading videos to the server via a WWAN, multiple camera nodes compete for transmission resources which are limited by the bandwidth. Due to the inherent transmission characteristics of wireless networks, packet loss occurs in the transmission network. In this paper, we assume that the packet loss rate of each camera node is perfectly known by the MIoT server.MIoT server: In the MIoT server, a joint source and channel rate allocation module optimizes the source and FEC rates for each camera node according to the content features and the current channel packet loss rates of GOPs in all camera nodes. Then the target source and redundancy rates of all camera nodes are sent back to each camera node for current GOP encoding. Besides, the MIoP server stores or performs vehicle tracking on received videos according to the actual demand. Since the tracking tasks usually are assigned with a delay in realistic scenarios, there is no real-time feedback about the tracking results in the MIoT server.K source packets of video stream, (N−K) redundant data packets are produced by the FEC encoder and are sent with source packets of the source block. As long as the receiver, which is a MIoT server in our structure, receives at least any K of the N packets, it can recover all the source packets [The FEC encoder shown in  packets .The KCF tracking scheme is proposed by Henriques et al. in 2015 . Since KAs shown in ppB) can be calculated as follows:r, Since HOG features can be affected by the artifacts created by video encoder at different compression rates , we emplL) to represent the effect of lighting conditions on HOG computation.When a camera is capturing video, the luminance value of pixels can be greatly affected by the lighting conditions at that time. Under the circumstance of weak lighting conditions, cameras are prone to lose detailed information of objects because of the lack of reflected light from objects and the interference of noise, and HOG features are affected consequently. We employ luminance level , in the covered field frames Based on the luminance levels of classified frames, the video scenarios can be detected and the video luminance level is obtained consequently. According to the previous description about scenarios, the video that has several consecutive covered field frames is regarded as a covered field scenario. Therefore, a video with more than five consecutive covered frames is considered a covered field video. The other videos are considered as open field videos. Due to the error propagation caused by the tracking scheme, different luminance level calculation methods are designed for different scenarios. In the tracking scheme, once a wrong target is determined in a frame due to the loss of HOG information caused by a low luminance level, the right target is hard to choose in all following frames even when the following frames have high luminance levels. Therefore, the open field frames are excluded when calculating the luminance level of video that has daytime covered field scenarios, as shown in Equation (5):f-th frame, N is the total frames in the video sequence. cN is the total covered field frames in the video sequence. Hence, f-th frame in video which can be obtained by Equation (7):f-th frame. In other words, The other videos, which have open field scenarios in daytime or at night have a different luminance level calculation method, that is:V) to represent the effect of video content complexity on tracking precision.In the KCF tracking scheme, the correlation results from kernelized correlation filter depend on the shapes of objects in the frame. In a video with complex content, the target object is surrounded by other background objects and the target object is hard to separate from the surroundings, which decreases the matching precision. We employ the video adjacent block difference  is the total number of foreground MBs in the f-th frame, and clstN(f) is the total cluster number of the f-th frame, which can be obtained based on the cluster results of foreground macroblocks (MBs) in the f-th frame. Here, the foreground MBs refers to the MBs which have more edges than other MBs in the frame. As the analysis in minG is the minimum of meanG is the mean magnitudes of intensity gradients in a frame. Then foreground MBs are detected based on the gradients magnitudes in the frame. Objects, such as vehicles, trees and pedestrians, always have more edges than background like road and sky. Then, we define the foreground set The adjacent block difference of a video sequence is defined as the maximum frame adjacent block difference just as follows:fmbN is defined as the total number of MBs in Then, clstN is then obtained based on clustering of foreground MBs. As shown in c) is the maximum cluster number when the c) + 1 indicates that the The total cluster number clstN is defined as the maximum c in the frame. In one cluster, magnitudes of gradients between adjacent MBs are prone to be small, which makes object matching difficult. Thus a smaller clstN indicates more complex content.Then, We select six videos which take vehicles as the target objects from the Online Object Tracking Benchmark (OOTB) . Among tppB to be our main tracking precision prediction parameter. According to the relationship between tracking precision and ppB shown in P is the tracking precision, and L and V of the video as follows:We use The finally obtained model coefficients are shown in k and n are the Mk represents the source packets of mobile camera node m. The element n represents the FEC packets of mobile camera node m, which contains the source packets and the protect packets. The proposed scheme allocates the appropriate source and FEC packets for GOP in each camera node, such that the product of the tracking precision of all GOPs is maximized. The joint source and channel rate allocation scheme can be formulated as follows:m. S is the packet size and m in application layer [In the MIoT system, multiple mobile camera nodes have to compete for the limited wireless transmission data rate. Due to the packet loss in the wireless network transmission, FEC needs to be used for source data protection. Therefore, we develop a joint source and channel rate allocation scheme for both video and FEC rate allocation with the objective to maximize the product of tracking precision of each uploaded GOP. Let M be the number of mobile camera nodes in system, and on layer :(17)f of the current raw GOP and sends these features to the server. The server jointly optimizes the source coding rate and FEC rate for all camera nodes based on these features and packet loss rates of all camera nodes. Then the optimal source coding rates and FEC rates are fed back to each camera node for current GOP coding. The encoded GOPs of six camera node are uploaded to the server. Next GOPs are also processed like this. Finally, we evaluate the tracking performance on the uploaded GOPs. In our simulation, we selected six video sequences from OOTB [We simulate a scenario where six mobile camera nodes capture different scenes and upload videos to a server via lossy wireless networks. The six mobile camera nodes have to compete for the limited transmission resource which in our simulation is represented by the total transmission bitrates. The channel condition of each camera node is represented by its packet loss rate. Each camera node extracts the two content features (rom OOTB  and visurom OOTB  data setm. We compare our proposed scheme with three schemes: the mean square error (MSE) driven rate allocation scheme, QoE driven rate allocation scheme and constant bitrate control (CBR) scheme. In CBR scheme, the total transmission rate is equally allocated to each camera node, and the packet correct rate of each node is set to be 0.995. For the MSE-driven and QoE-driven schemes, we adopt a classical rate-distortion model in  and a QoWe set a fixed total data rate of 800 kb/s for the proposed tracking-precision-driven scheme (TP-driven scheme), MSE-driven scheme, QoE-driven scheme and CBR scheme. The packet loss rates of all camera nodes are set to be 1%. Next, we conduct tracking on these video sequences by KCF scheme. The number of correct tracking frames in each GOP is illustrated in Here, we assume that the ground truth of the first frame in each video sequence is assigned by the server. From The performance of the proposed TP-driven scheme, MSE-driven scheme, QoE-driven scheme and CBR scheme is tested under different total rate constraints. We test the FEC rate allocation by setting different packet loss rates for each video. Since in the CBR scheme, the source data rates of all camera nodes are equally allocated to all videos, only the TP-driven, MSE-driven and QoE-driven schemes are tested. In the first case, the packet loss of each video is 1%. In our simulation, the GOP structure is IPPP. In fact, several B frames are allowed in the GOP, GOP structure like IBBPBBP can provide similar performance like IPPP. However, the GOP structures which have a lot of consecutive B frames (IBB…BP structure for example) are not suitable for our scheme. Due to the blurring and fast movement in the video, a lot of consecutive B frames can cause obvious video quality degradation. Consequently, tracking precision decreases. Therefore, I or P frames are needed to be inserted every several B frames in the GOP for high quality service.In this paper, we have proposed a joint source and channel coding algorithm for vehicle tracking in MIoT systems. Our algorithm aims to provide better tracking performance in MIoT servers, which is different with previous QoS/QoE-driven schemes which aim to improve the human visual experience. We first study the effect of video content and compression on the KCF tracking scheme and establish a content-aware tracking precision prediction model, which can estimate the tracking precision of videos with different content features and coding rates. In this model, video luminance level and adjacent block difference are extracted to represent the effect of lighting conditions and video content complexity on tracking precision, respectively. Then, based on the precision prediction model, the optimization problem is formulated to maximize the product of tracking precision of each surveillance video under the constraint of the total transmission rate in the wireless network. Considered the transmission error in wireless network, FEC is adopted to improve the reliability of data transmission. Compared with the traditional QoS/QoE-driven algorithm and CBR algorithm, our joint source and channel rate allocation algorithm more efficiently allocates the bit rates under the constraint total transmission rate and achieve better tracking performance.


Branching density  is an important trait which contributes to defining the number of roots in individual plants. The environmental and local variations in inter-branch distance have often been stressed, and simulations models have been put forward to take them into account within the dynamics of root system architecture (RSA). However, little is known about the interspecific and intra-plant variations of inter-branch distance. In this paper, we present an analysis which draws on 40 samples of plants belonging to 36 species collected in homogeneous soils, to address how the variations in inter-branch distance are structured within individual plants, and how this structure varies from one species to another. Using measurements of inter-branch distance on various roots of the same species and our knowledge of the branching process, we defined a simple and generic model dedicated to the simulation of the observed variations. This model distinguishes between two sub-processes: i) the longitudinal location of potential branching sites and ii) the effective emergence of lateral roots at these sites. Thus, it represents the variations in distance between the potential sites (with two parameters), and the probability of emergence of a lateral root at each site (one parameter). We show the ability of this model to account for the main variations in inter-branch distances with a limited number of parameters, and we estimated them for the different species. These parameters can be considered as promising traits to characterize—in a comprehensive and simple way—the genetic and environmental variations in the whole branching process at plant level. Based on the results, we make recommendations for carrying out comparable measurements of the branching density in developed plants. Moreover, we suggest the integration of this new model as a module in future RSA simulators, to improve their capacity to account for this important and highly variable characteristic of plant species. The linear branching density for a given root can be defined as the number of lateral branches per length unit along this root. It can be characterized by the reciprocal variable, the inter-branch distance (IBD), and is often measured on the branching parts of roots, i.e., excluding the distal end where laterals have not emerged yet .It is a very important trait at the root system level, because it largely contributes to defining the total number of roots in the whole plant. Since the root system is usually ramified up to several branching orders, typically between 3 and 5, the effect of inter-branch distance on the total number of roots is potentially raised to the power of the number of branching orders. However, note that some species, among the Amaryllidaceae, for example, have roots which are not branched, but they seem to constitute exceptions.Several authors have quantified IBD (or branching density) in certain species  observed in homogeneous soils to investigate how inter-branch distance varies within individual plants and from one species to another. Using this analysis and knowledge of mechanisms, we put forward a new generic and quantitative model to depict and simulate the observed variations.In Natura or in pot cultures  were extracted around the sampled plants, inserted into a mesh bag, immersed in a large bucket of water, and gently cleaned. Pot-grown plants were cultivated in greenhouses using long PVC tubes  filled with either sieved soil or a mixture of sandy soil and sieved compost. Each species was sampled from two to five well-developed plants, usually before flowering. Sampled trees were young, between 2 and 4 years old.The data used in this paper come from a large data set of plants sampled since spring 2013 either cultures Table 1.cultures . Plants From our initial data set, containing more than 200 samples , we selected only those which had at least 150 branch roots that were measured on parent roots with a sufficient range of diameters to study the effect of this factor. Thus, we kept 40 samples belonging to 36 different species  at a resolution of 2,400 to 3,200 dots per inch, using the transparent mode. The resolution was adjusted for each species so as to get at least 10 pixels transversally for the finest roots to measure them with sufficient accuracy. Images were stored in jpeg format.http://rsbweb.nih.gov/ij/). We measured the diameter of the parent root and the distance along the parent root from each lateral to its proximal closest neighbor (from axis to axis). We quantified branching density based on the inter-branch distance , because this variable could be measured for each lateral root. Measurements were done in the young branching zone, where laterals had reached at least 2 mm, to discard the very distal zone where emergence was taking place and where lateral roots might be not visible. Let us note that some lateral roots were broken during washing and imaging, but their trace was clearly visible because of the relative transparency of the parent root cortex.Measurements were made on the computer screen by mouse clicking on the displayed images using the measuring tools  provided by the ImageJ software , as well as analyses of distributions using several functions of the “moments” package of R . We represented the distributions of inter-branch distances graphically using boxplots , and its coefficient of variation is DCV. This is the first step that gives a distribution of potential branching sites along the virtual parent root. Second, it is assumed that each potential branching site may actually give rise to an emerged lateral root  with a given probability Pem, or it may remain unbranched (branching failure), because one of the following steps could not be accomplished, with a probability of (em1-P). This is the second step that simulates the positions of the emerged lateral roots along the parent root and allows calculating the distribution of inter-branch distances that can be compared with the observed ones. This simple model does not distinguish between reasons of failure. It could occur during the inception or later development of the primordium or during the emergence as a lateral root. We did not specify the radial angle that defines the orientation of the laterals around the parent root. The proposed model has three parameters: average distance between the potential sites (ISD), coefficient of variation of this distance (CVD), and probability (Pem) of success of the whole branching process. These parameters, which quantify the two sub-processes, had to be estimated on the whole, using the observed IBD distributions. The calibrations and tests were done on the whole root population for each species first, and then the model was used to analyze the effects of the diameter of the parent root.Then, we wrote a specific R function  and 5.63 mm (Acanthus mollis at Thouzon). When several samples exist for the same species , these samples are rather close to each other.The IBD distributions were neither normal nor symmetrical. The Shapiro–Wilk normality test rejected the normality hypothesis for all species. The values of skewness and the Agostino test also showed a clear asymmetry for almost all species, with a systematic right-hand tail of the distributions, i.e., an excess of high values above the last quartile, as we can see in P < 0.001).When faced with such distributions, it is common to make a logarithmic transformation e.g.,  to obtaiWe also explored the effect of the diameter of the parent root on IBD using correlation tests. Among the 40 samples, the correlation between the diameter of the parent root and IBD was not significant in 12 cases, significant and positive in 5 cases, and significant and negative in 23 cases. Thus, for a majority of species, the laterals tend to be more spaced out on the fine parent roots, but the inverse phenomenon also occurs, although less frequently.D, Pem); iii) low correlations between these indicators. For the construction, we made 8,000 simulations of distributions with 60,000 laterals for each of them, combining parameter values distributed within the plausible ranges for each parameter , and we calculated a number of common characteristics for each simulated distribution of IBD: mean, standard deviation, mode, quantiles at probability levels of 0.10, 0.20, 0.30, …, 0.90, 1.0. Based on this preliminary work (not shown), we retained three indicators that fulfilled the three conditions relatively well. They are: the mode of the IBD distribution, which was tightly correlated to the parameter ISD; the ratio: quantile 0.10/mode, tightly correlated to CVD; the ratio quantile 0.80/mode, tightly correlated to Pem. Then, we estimated the three model parameters independently using these three indicators via the linear interpolation of the relationships between average parameter values and indicator values that we obtained from preliminary simulations.Since the proposed branching model is a numerical and stochastic simulation model, the fitting is not straightforward. To adjust it to the empirical (measured) distributions of IBD, we used a set of intermediate indicators that we constructed first based on simulated distributions and that met three conditions: i) simplicity to estimate these indicators based on the empirical distributions; ii) tight correlations between these indicators and input parameters . We calculated chi square values for each sample to evaluate the quality of the fitting. These values were significant and rejected the hypothesis of identical simulated and empirical distributions in only 3 of the 40 cases , confirming the importance of the species effect on this trait. Other parameters were less variable: DCV varied between 0.28 and 0.69 (2.5-fold factor), and Pem varied between 0.37 and 0.98 (2.6-fold factor). We also obtained positive and significant correlations between estimates of the three parameters, but the R2 values were below 0.3.The distributions of parameters are presented in For each sample, we divided the population of lateral roots into two sub-populations: those roots originating from parent roots that are finer than the central value of the diameter [0.5*] and those on parent roots with a diameter above this central value. We calibrated the model separately for each sub-population.ISD parameter values as estimated for the two populations. We observed a highly significant correlation between both sets, and we tested that the regression line was not significantly different from the bisecting line  on the contrary, we did not observe correlations between the two estimates. Again, the Student t test did not reveal differences between the two sub-populations. However, the values of emP tended to be slightly higher for thick roots in a number of species, confirming a tendency for these species to have a heavier right tail in their IBD distribution for the finer roots.ISD were correlated to the relative differences of Pem, as shown in Merculialis annua observed in Thouzon and Pseudotsuga menziesii observed in Nozeyrolles, whose IBD distributions are represented in Mercurialis annua, the lateral roots on fine roots were more spaced out, whereas Pseudotsuga menziesii showed higher densities on its fine roots.We also observed that the relative differences of estimated In this study, we confirm that IBD exhibits large variations in each of the 40 species samples, both within the species and between species, as reported for example by Despite the large intra-plant variations of IBD, the inter-species variability was so high (nine-fold factor in our study) that the species effect was significant. Because of the large intra-plant and intra-root variations, testing a species or genotype effect requires a sufficient number of measurements for each one of them as well as measurements made in very comparable situations for all. In this study, we had at least 150 measurements per sample combined with 40 different samples. The species effect was not specifically the subject of the present study, but it was demonstrated earlier by ISD), its variations (model parameter DCV), and the probability of accomplishing the whole branching process until lateral emergence takes place on each branching site (summarized by the emP parameter). These different sub-processes are worth separating because they occur successively, involve various molecular mechanisms, and respond to different environmental stimuli  to the parent root diameter and possibly to the root position in the surrounding soil. This method would lead to a reduced number of parameters with a more clear and biological meaning.This model can be integrated as a component (or module) in larger models of the root system architecture . BranchiISD is rather genetically controlled, whereas Pem may be more sensitive to local conditions and would reflect plasticity. These speculations based on the present results are worth verifying using suitable experimental designs that cross genotypes and environmental conditions. We have seen that ISD can easily be estimated based on the mode of the empirical distribution of IBD, which represents the most frequent inter-branch distance. This value seems relatively easy to obtain, and could be estimated on a reduced number of lateral roots, several tens for instance. The probability parameter emP has a very different impact on distribution. Its value modifies the right tail of the distribution, so that it can be conveniently estimated via the high-probability quantiles of the distribution. We used the 0.8 level in the present study. According to our model, a Gaussian distribution is obtained as the limit distribution when emP = 1, i.e., when all branching sites give rise to an emerged lateral root. The importance of the right tail increases with decreasing values of emP. Therefore, ISD would summarize a genetic potential for the plant regarding branching density, and emP would summarize the accomplishment of this branching density in the given environmental conditions. The impact of the third parameter (DCV) on the global distribution is lower, and its estimation is more difficult. Thus, its interest from a phenotyping perspective is much lower.The parameters of the present model may also be used as interesting traits for root phenotyping approaches. Among the three parameters, we can guess that ISD values were not impacted by the parent root diameter. Thus, for all these species, the fineness of roots did not modify this archetypal distance. On the other hand, the diameter of the parent root had a higher -although unpredictable- impact on the probability of accomplishing the branching process (emP parameter). For a majority of species, this probability decreases with decreasing diameters, but it is not a generally observed phenomenon. We also observed opposite cases, where it seems that failure is more probable for thick roots than fine roots.Thanks to our original data set which relies on many different species and well-developed and highly branched root systems, we could study the inter-branch distance on main roots as well as lateral roots, and chose to study the effects of the diameter of the parent root. We showed that in most species, the Since the differences observed for the values of ISD and Pem in the sub-populations of parent roots (thick versus fine) were not independent, we can suspect that it might reflect different plant strategies regarding the positioning of their laterals. Some species had higher densities on their main thick roots, whereas others favored higher densities on their fine roots. Both strategies can make sense and participate in the development of a diversity of branching patterns, in coordination with other developmental processes, as described by It would be very interesting to use this diversity to further explore the molecular mechanisms explaining that laterals have a higher chance to emerge on thick or fine roots within the same genotype.Therefore, studying simultaneously intra-plant and inter-species variations and using well-developed root systems is an interesting means of raising new questions regarding both the diversity in the foraging strategies of plants and regarding the physiological mechanisms underlying this diversity. The construction and use of such a simple and generic model is a prime way of allowing the quantitative exploration of this diversity.The datasets generated for this study are available on request to the corresponding author.LP made the conception, data acquisition and analysis, and wrote the paper.The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


The experience feedback committee (EFC) is a tool designed to involve medical teams in patient safety management, through root cause analysis (RCA) within the team.To investigate the functioning of EFCs in the departments of a large university-affiliated hospital in France and to consider its potential contribution to the management of patient safety.Cross-sectional, observational study, based on an analysis of the documents produced by the EFCs for 1 year. Data were collected independently by two investigators in meeting minutes, adverse event reports and event analysis reports.The study included all 20 EFCs operating in the hospital's medical departments. During the study year, committees held 164 meetings, reviewed 1707 adverse events, conducted 91 event analyses and decided on 206 corrective actions. The median number of corrective actions adopted by each EFC was five actions . A root cause analysis (RCA) was present in 76% of the analysis reports, and these analyses were complete in only 23% of the reports. There was also a lack of planning corrective actions: an implementation deadline was only defined in 26% of the actions.Healthcare professionals adhered to the system-based approach to patient safety, but we observed difficulties in holding regular meetings and deviations from the theoretical framework. These findings confirm the difficulties of practicing RCA in the healthcare setting. Nevertheless, EFCs can be vectors of safety culture and teamwork. The improvement of patient safety remains one of the greatest challenges of all healthcare systems. Despite In France, the vast majority of healthcare facilities have a similar program called the experience feedback committee (EFC).  The mainThrough the analysis of the 20 EFCs implemented in a large university-affiliated hospital, the objective of this study was to investigate the functioning of the EFC and to consider its contribution to the management of patient safety.We conducted a cross-sectional, observational study of EFCs established in the Grenoble University Hospital (France). The study was approved by the Institutional Research Ethics Committee of Southeastern France .The study was conducted in a 1347-bed acute-care university hospital including 42 clinical and medical-technical departments.The hospital’s central risk management system was described elsewhere. Adverse The functioning of the EFC is defined in local guidelines in accordance with the framework proposed by Air France Consulting. The depaThe Committee is composed of volunteer representatives of the department’s various professions. A few days before the committee meeting, the EFC leader receives a file including the department’s events reports from the central safety unit. The Committee can also set up a specific reporting system for the EFC.The committee meets regularly, usually once a month, according to a fixed schedule. Meetings last between 1 and 2 h. Committee meetings are conducted according to a standardised framework: 1) reading the list of reported events, 2) choosing a priority event to investigate, 3) choosing the professional responsible for the investigation, 4) reviewing the analysis carried out for the event chosen the previous month, 5) deciding on corrective actions and 6) monitoring on-going actions.The investigation is carried out during the month following the EFC meeting by a designated person using the Orion method developed by aviation safety experts. The mainAll EFCs established in the hospital departments more than 1 year before were eligible. The purpose of the study was presented to the EFC leaders and their consent was was requested orally to participate in the study and to allow investigators to analyze the documents produced by the EFC. These documents did not contain any patient-level identifying information.We used the data collection method previously described elsewhere. All writth and 75th percentiles) for continuous variables. To identify the most potentially productive EFCs, we generated another variable based on the median number of corrective actions decided per year. Thus, EFCs that decided to implement a number of actions greater than the median were considered potentially more productive than the others. Secondarily, we compared the EFC characteristics across subgroups of EFCs defined by this dichotomised variable using the chi-square test or Fischer’s exact test, when appropriate, for categorical variables, and the Kruskal-Wallis test for continuous variables. P-values less than 0.05 were considered statistically significant. Analyses were performed using Stata 14.0 .EFC baseline characteristics were reported as numbers and percentages for categorical variables, and median and interquartile range , six medical-technical departments , five emergency or intensive care departments and two surgical departments. Five EFCs had periods from 6 to 26 months of inactivity resulting from staffing issues, such as resignation of the EFC leader or understaffing in the related department.During the 1-year period studied, the EFCs held 164 meetings (2–12 per EFC). They examined 1707 reported events, conducted in-depth analysis of 91 events and decided on 206 corrective actions.Meeting minutes were found for 160 (98%) of the 164 committee meetings. These reports mentioned 351 participants including 99 physicians, 48 head nurses and 76 nurses or other paramedics . The revOf the 20 EFCs, 11 used event reporting from the central unit for risk management, six used only the reports collected in the EFC department and three used the two sources of reports . Most evAmong all the events reviewed, 98 were selected for in-depth analysis. An analysis report presented at a committee meeting was produced for 91 event investigations, and 72 of them (79%) were presented according to the format of the Orion method . The datThe annual number of corrective actions decided by each EFC ranged from 0 to 62, with a median of five . The mosp = 0.02) and achieved more investigations  (p<0.001). The most active committees designated a person in charge of the corrective action more often (p<0.001) and their actions had a deadline for implementation more often (p = 0.01).The most productive committees, defined as EFCs that decided at least five actions in the year, reviewed more events  . The anaThis study shows that nearly half of all medical departments voluntarily implemented an Experience Feedback Committee (EFC). Healthcare professionals adhere to the method that is implanted in a wide variety of medical departments. Reported adverse events are analysed and corrective actions are decided by the committees.However, this picture is mitigated by the problems maintaining this activity in the healthcare teams’ routine. The number of meetings varied over time and from one department to another; some even had long periods of inactivity. Healthcare professionals explained these variations in activity by the departure of a leader who was not replaced and, above all, by the lack of time and resources. Indeed, carrying out investigations to identify the causes of events, as well as writing analysis reports, takes a lot of time for professionals who are already very busy., 15The study also shows the sometimes significant deviations in practices compared to the theoretical functioning of the EFC. The EFC is based on a systems approach to patient safety and it provides a formal method for the root cause analysis (RCA) of adverse events. The Orion method, based on the Reason model, is close to the “Association of Litigation And Risk Management” (ALARM) method and includes the same steps. InitiateThis weakness found in the practice of RCA in the field of healthcare is reported by many authors., 17–19 EThis lack of rigor in RCA application might suggest that better training is needed to involve healthcare professionals more effectively in RCA within the team. More generally, these findings raise questions about the relevance of performing RCA by caregivers rather than by risk management experts.When interviewing healthcare professionals who participated in RCA training programs, paradoxical responses were obtained., 15, 19 RCA effectively improves safety in various industries such as civil aviation. Because of this potential, RCA has become an important part of all healthcare safety management programs around the world., 17 TherIn agreement with the literature, we found that the practice of RCA in the EFC is imperfect and that it would be illusory to expect short-term effects on the incidence of adverse events and patient safety. However, we hope that the EFCs will have long-term effects by increasing safety culture and learning by error. Indeed, the EFC is a particular mode of implementation of RCA in the field of healthcare that aims to directly involve members of the staff in the management of adverse events affecting their department. A person who reports an incident is invited to participate in the analysis and solution development and can then see the implementation of corrective actions and observe their effects. In this context, the professionals receive feedback on the reports, within a short feedback loop. This can help reduce a traditional barrier of incident reporting that is related to the opacity of reporting systems and the lack of feedback to the reporters about actions decided following such reports. We suppoWe also hypothesise that the system-based approach to managing adverse events within a team is likely to improve the perception of the collective dimension of healthcare and thus foster teamwork. The social and cultural functions of the EFCs can be compared to those of the mortality and morbidity conferences when they are used to improve the quality and safety of care., 28 LikeThe main limitation of this study is that it concerns only one hospital. It is probably not representative of the functioning of all EFCs in all French hospitals. However, this study is exploratory, it is the first one that analyses the functioning of several EFCs implanted in different medical specialties. It shows the difficulties of running a risk management system on a regular basis and opens up leads for carrying out broader studies on several hospitals.We can also discuss the relevance of the criterion based on the number of actions decided to estimate the effectiveness of EFCs. This is an intermediate criterion, indicative of EFC functioning. To evaluate EFCs’ ability to improve care safety, it would be necessary to verify the actual implementation of these actions and their effect on adverse event incidence.Another limitation is related to the retrospective collection of data, including missing data due to absent or incomplete reports. However, this lack of traceability is itself a result confirming how difficult it is for teams to follow a rigorous method.In spite of these limitations, this study provides leads that may improve the functioning of EFCs. Although the method seems simple, its implementation requires training and it will be necessary to strengthen the training of healthcare professionals and to offer long-term methodological support by risk management specialists.The EFC is a way to involve healthcare professionals in system-based analysis of adverse events associated with medical care. The study identifies the limitations of this type of activity, which requires time for professionals and skills that are not easy to acquire. But the main barriers to implementing RCA in healthcare teams are psychological, social and cultural. However, we observe the approval of professionals persisting over time, and we remain hopeful that RCA will contribute to improving the safety culture of healthcare professionals.S1 Dataset(XLS)Click here for additional data file.


Muscle synergies are now widely discussed as a method for evaluating the existence of redundant neural networks that can be activated to enhance stroke rehabilitation. However, this approach was initially conceived to study muscle coordination during learned motions in healthy individuals. After brain damage, there are several neural adaptations that contribute to the recovery of motor strength, with muscle coordination being one of them. In this study, a model is proposed that assesses motion based on surface electromyography (sEMG) according to two main factors closely related to the neural adaptations underlying motor recovery: (1) the correct coordination of the muscles involved in a particular motion and (2) the ability to tune the effective strength of each muscle through muscle fiber contractions. These two factors are hypothesized to be affected differently by brain damage. Therefore, their independent evaluation will play an important role in understanding the origin of stroke-related motor impairments.The model proposed was validated by analyzing sEMG data from 18 stroke patients with different paralysis levels and 30 healthy subjects. While the factors necessary to describe motion were stable across heathy subjects, there was an increasing disassociation for stroke patients with severe motor impairment.The clear dissociation between the coordination of muscles and the tuning of their strength demonstrates the importance of evaluating these factors in order to choose appropriate rehabilitation therapies. The model described in this research provides an efficient approach to promptly evaluate these factors through the use of two intuitive indexes. As a consequence of the population aging around the world, stroke has become a widespread concern . DependiIn recent years, the concept of muscle synergies has been widely discussed to clarify the biological basis for activating the redundant musculoskeletal system of stroke survivors –10. AccoLearned motions are perceived by healthy individuals as simple and easy, even though they require the complex coordination of muscles. The concept of muscle synergies was originally developed to explain the neural processes that provide the brain with tools to reduce the kinesiological complexity behind this coordination. However, after a stroke, the failure in motor control occurs not in the spinal cord but in the brain. Although studying the appearance of new muscle synergies after brain damage is necessary, current knowledge on this topic is not enough to choose an appropriate rehabilitation treatment. Neural recovery starts at the brain level, and depending on the severity of the injury will trigger changes at the spinal and muscular levels . In thisDuring motion in healthy individuals, the mechanisms of muscle coordination and the tuning of effective muscle strength compensate for each other, thereby creating stable muscle synergies. In this research, it is hypothesized that after brain damage, these two factors are affected differently , thus generating a wider range of motor impairments. As these factors are directly related to the neural adaptations underlying motor recovery, their measurement will facilitate the prompt identification of the neural origins of the motor impairment, and therefore the selection of an effective rehabilitation therapy.Subjects were asked to perform two different bimanual symmetric motions which have been proved not only efficient tasks to asses stroke , 25, butThe experimental trial consisted of 20 s of motion followed by 20 s of rest. The experimental session was composed of three 20-s elbow flexion trials and three 20-s dual steering trials. These trials were presented in a randomized order to reduce possible biases in subjects’ motor behavior owing to constant repetition of the same task.healthyelder × 3 trials = 60 trials per motion). Finally, each healthy young subject was asked to perform six trials per motion . Healthy subjects did not present any known diseases and had no medical record history associated with motor disfunction. All subjects were previously informed about the experimental procedure and provided written informed consent, in accordance with the Declaration of Helsinki. The experimental procedure was approved by the ethics committees of the National Center for Geriatrics and Gerontology and RIKEN.EMG data were recorded from 18 stroke patients (9 men and 9 women) between the ages of 66 and 96 years (74.88 ±9.63). Table Muscle activity was recorded using 18 wireless sEMG sensors  symmetrically located on the following nine upper limb muscles (on both the left and right sides of the body): brachioradialis, pronator teres, biceps, triceps, anterior deltoid, posterior deltoid, pectoralis, infraspinatus, and elector spinae. These muscles were defined according to the guidelines of the Surface Electromyography for the Non-Invasive Assessment of Muscle Project . AcquireRaw sEMG data were high-pass filtered at 20 Hz to remove possible motion artifacts, rectified, and low-pass filtered at 32 Hz to preserve the key frequencies associated with muscle contractions. In both filtering stages, a fifth-order Butterworth filter was applied .P is defined as the summation of the electrical power used by each muscle, Pi, for i=1,2,3,...N, with N being the total number of muscles contributing to the movement  in synchrony with the whole set of contributing muscles , 34. At Pi value is redefined according to its proportional contribution to the total electrical power P through the use of ai coefficients with i=1,2,3,...N , which is defined as the comparison between d Pl Eq. . Changesvely Eq. . This paai and bi were extracted to show the benefits of their evaluation in assessing the importance of each muscle depending on the motion. Sets of ai and bi were represented as 18 values (associated with the nine muscles measured on each side of the body). In addition, the values of ESB and MCS were computed over the three groups of subjects to assess how different severity levels are affected by the two groups of neural adaptations that participate in motor performance.After sEMG processing, the power per second of each muscle was quantized as the RMS of the rectified signal. In addition, the indexes defined in the motion model described in the previous section were computed. Coefficients ai and bi. Figure The muscle activity print associated with each movement and subject group is summarized in Fig. The healthy young and elder muscle prints show how the important muscles involved in voluntarily motor control differ between the movements performed. During elbow flexion Fig a, bicepsx-axis) and MCS values (y-axis). Each point corresponds to the ESB and MCS values computed from a single experimental trial. In this representation, points with values close to x=0 are related to movements in which the left and right sides of the body require equal electrical power to contract the muscles. Depending on the level of dominance of one side of the body over the other, the value of the x-axis varies from -1 to 1, with -1 and 1 representing total dominance of the left and right sides, respectively. On the other hand, the y-axis shows the MCS values: y=1 represents the situation in which muscles are coordinated on the left and right sides of the body following the same strategy, that is, a situation of perfect muscle mirroring; y=0 implies a loss in the correlation of muscle activity between sides; and y<0 indicates the appearance of an asymmetrical correlation, which implies that symmetric muscles have switched roles, consisting of high activity on one side and poor activity on the other. This is a rare situation that must be considered to evaluate motion in severe stroke patients who are barely able or completely unable to move their paretic arms. Figure The scatter plots in Fig. y-axes represent MCS (A and C) and ESB (B and D) values for each SIAS level shown along the x-axes. Results are also represented in the form of boxplots showing the first and third quartiles as well as the 1.5 inter quartile ratio average, and outliers of each distribution. The results for healthy elder and healthy young subjects are also included to demonstrate the differences between these groups of subjects. The absolute value of ESB was used to assess these results in order to highlight how the SIAS level influences the side balance independently of the paretic arm. To evaluate the significance among groups, an all-vs-all statistical comparison was applied to each graph, namely, a Wilcoxon sum-rank test with a confidence interval of 95% [In Fig. l of 95%  followedl of 95% . MultiplIt was initially argued that the mere evaluation of muscle coordination in patients recovering from strokes is insufficient to elucidate all neural processes involved in motor recovery. Even though the proper coordination of muscles is an important factor that contributes to the performance of an efficient motion, the ability to properly tune the effective strength of muscle contractions also plays an important role. By exploiting the symmetry of the human body, the model described in this work proposes two indexes directly related to the factors involved in motor recovery. The computation of these indexes for subjects ranging from healthy individuals to severe stroke patients provided interesting muscle performance results in terms of the factors involved in motor recovery.For the healthy young and elder subjects, there were clear similarities between body sides both in muscle coordination (MCS) and muscle effective strength (ESB) Fig. a-c. AlsoHowever, in the case of stroke survivors, the ability to coordinate muscles and tune their effective contraction strength are affected differently depending on the area and size of the lesion. This is clearly represented by MCS and ESB indexes in Fig. Accordingly, the application of models that explain motor control based on neural processes in charge of motor coordination is incomplete in studying motor recovery. In this scenario, other neural processes often related to recovery of motion strength also play an important role. The identification of the neurological origins of post-stroke motor impairment is a relevant factor for the assignment of effective rehabilitation therapies. In that regard, as supported by the data analyzed, it appears that the model described and the indexes inferred from it have the potential to distinguish among the different factors affecting motor recovery based on the analysis of sEMG signals. In addition, the reduced setup and simplicity of calculating these indexes make them easy tools for standardizing rehabilitation therapies.This research focused on the development of a model for extracting information from sEMG signals to quantify the neural mechanisms behind motor recovery. Given the physiological properties of sEMG signals and current knowledge about the neural processes in charge of recovering motor strength , 23, 23,Finally, it is important to note that the present method is based on a comparison between paretic and non-paretic body sides. Even though this is a commonly accepted methodology in stroke research, it assumes a healthy condition of the non-paretic side. To avoid the misinterpretation of results, researchers using these parameters should confirm the condition of the non-paretic area used as the empirical expectation. However, the current work is not intended to study the rehabilitation benefits of the specific motions evaluated. The recorded sEMG data were used only to validate the model proposed and show its potential as a tool to evaluate motor recovery. The model consistency should be also further analyzed by including a wider range of tasks to those evaluated on this work. It should be also mention that the methodology proposed do not intend to provide ultimate solutions to all current challenges in the field of motor rehabilitation. For instance, many daily tasks are unimanual or require independent actions on each limb. The system proposed, as happens also with many other rehabilitation techniques, cannot be used on a daily environment. However, it allows the use of the non-paretic areas of the body, within a controlled clinical environment, in order to potentiate the recovery of the paretic areas by exploiting the symmetrical structure of human body. In fact, the recovery of specific motions on the paretic limb using feedback from the non-paretic side is a common technique on rehabilitation . NeverthThis work is the beginning of a wider research program that is intended to increase the current understanding of motor recovery in order to improve rehabilitation therapies. Accordingly, the next research stages include the following objectives: (1) development of a system that provides real-time feedback about ESB and MCS to patients during rehabilitation; (2) integration of ESB and MCS with the study of muscle synergy development during rehabilitation to better understand the neural processes following recovery; and (3) development of a rehabilitation protocol adaptable to specific patient conditions and neural recovery strategies based on the results obtained in the this and future studies.


Moving ship targets appear blurred and defocused in synthetic aperture radar (SAR) images due to the translation motion during the coherent processing. Motion compensation is required for refocusing moving ship targets in SAR scenes. A novel refocusing method for moving ship is developed in this paper. The method is exploiting inverse synthetic aperture radar (ISAR) technique to refocus the ship target in SAR image. Generally, most cases of refocusing are for raw echo data, not for SAR image. Taking into account the advantages of processing in SAR image, the processing data are SAR image rather than raw echo data in this paper. The ISAR processing is based on fast minimum entropy phase compensation method, an iterative approach to obtain the phase error. The proposed method has been tested using Spaceborne TerraSAR-X, Gaofeng-3 images and airborne SAR images of maritime targets. Synthetic aperture radar (SAR) is widely employed in military surveillance, geography mapping and resource surveying. High-resolution SAR image is of great significance for homeland and military security ,2,3,4. AThe motion between the moving target and the radar contains translation and rotation motions ,9. TransMethods for phase compensation may be divided into three categories. The first category is scatter-based algorithms, such as dominant scatter processing (DSP) method ,13, phasThere are two issues involved in processing raw echo data. One is that the aforementioned methods are often applied to raw echo data, while the moving ship’s position is hard to ascertain in raw data . Hence tTo deal with these problems, a refocusing method for moving ships based on fast minimum entropy phase compensation is proposed in this paper. The processing data are sub-images containing moving ships rather than the raw echo data, and the ISAR technique is based on fast minimum entropy phase compensation. The refocusing method has three advantages, the first is that the computational burden is low, due to the smaller data size of the sub-image than the raw echo data. The second is the procedures of the inversion algorithm and image reconstruction are simple. The last is that the ISAR technique based on fast minimum entropy phase compensation has good image quality and computational efficiency.The remainder of this paper is organized as follows: in Moving ships are regularly blurred and defocused in SAR images due to the translation motion with respect to the scene center. ISAR technology is a common method to form well-focused images, but it has two basic problems. One is that SAR images contain a very large number of ships, each with its own motion. Hence, it needs sub-images which only contain a single ship target. The other is the input of the ISAR processor is raw data, not SAR images, so it needs to get input data for the ISAR system which can serve as raw data. The basic procedures of refocusing moving ship in SAR image are represented in (a)Input a single look complex (SLC) image;(b)Implement ship detection with software;(c)Select sub-images, where each sub-image includes only a single defocused ship and has the same spatial resolution as the original image;(d)Invert the sub-image to the equivalent raw data domain via an inversion method ,27,28,29(e)Exploit ISAR processing to generate a focused image of the ship.The detailed procedures are as follows:The sub-image needs to be inverted to equivalent raw data-like data containing only the target echo, background and residual clutter. The common inversion method is known as the range Doppler inversion .When the angle variation is not too large, and the rotation vector is sufficiently stable during radar imaging, the range Doppler (RD) algorithm is applied for reforming SAR or ISAR images with high accuracy after motion compensation. The polar grid in the spatial frequency domain can be taken as a nearly regularly sampled rectangular grid with no need for interpolations. The main advantage of this method is the low amount of computation and it is the reason why it has been employed in many references ,29. The We propose an ISAR processing method based on fast minimum entropy phase compensation to refocus sub-images of moving ship targets. The approach is a RD algorithm exploiting a Fourier transform to create the image. The fast minimum entropy phase compensation method is a non-parametric method and can be applied to arbitrary targets, even high-order polynomial models.The geometry of the ISAR system is depicted in  follows ,31:(1)SRThe radial motion compensation can be achieved by removing the phase term The distance The quadratic term R images ,32.The features of ship, image entropy and image contrast (IC) are regarded as the evaluation criteria of image focusing. If the image is well-focused, the entropy and IC attain their minimum and maximum values, respectively . BecauseAn ISAR image  follows :(5)E(I)=The entropy is relatively small when the image is well-focused, and image refocusing is assessed via Equation (5). The IC denotes the normalized effective power of the image intensity and gives a measure of the image focus. If the image is well-focused, the IC value of the image is large. IC definition is considered as the ratio of the standard deviation to the mean of the amplitude. The IC is written as follows :(7)IC(I)The motion compensation mainly compensates for the translation motion between the target and the radar. It contains two steps, one is range alignment, and the other is phase compensation ,11. The Phase compensation and ISAR imaging can be written as follows ,32,35:(8It means that the Equation (9) should be satisfied with (10):Since the total energy Therefore Equation (10) is equivalent to:The derivation function of the entropy with respect to Since Then substituting Equation (14) into Equation (13), we have:The derivative of Substituting Equation (16) into (15), one obtains:Finally, Equation (17) is equal to zero, and The flowchart of the proposed phase compensation method is shown in Step 1: Input the raw data-like data and utilize DCT method [T method  to obtaiStep 2: Compensate phase error by substituting Step 3: Calculate entropy of the ISAR image and set tolerance Step 4: Obtain Step 5: Update phase error estimation Experimental results based on real SAR images are presented in this section to quantitatively evaluate the validity of the proposed method. The experimental data include spaceborne and airborne SAR data. The parameters of the SAR images are presented in The SAR image perform ships detection with software  and sub- the DCT  and PGA  the DCT  methods.The spaceborne SAR data contains data from two kinds of satellite system. One is TerraSAR-X system images, the other is Gaofeng-3 system images. The defocused ship targets of TerraSAR-X images are depicted in The refocused results of ship01 with the three methods are shown in Image entropy and IC value are the two criteria to assess the image quality. The convergences of the two criteria versus the number of iterations are depicted in The geometrical features of the ships are extracted for further assessment. The features include the length, width and area. Since there are no ground truths for the ship features, it is impossible to compare extracted geometrical features with ground truths. The geometrical features of ships in focused images should be smaller than the defocused images.The extracted features of ship01 and ship02 are illustrated in The defocused ships detected in two Gaofeng-3 SAR images are illustrated in The refocusing results of ship03 are presented in The results of ship04 are shown in The entropies and IC values during iteration are represented in In addition to spaceborne SAR data, an airborne SAR dataset is exploited to further validate the procedure. The defocused ships are marked in the airborne SAR image in The defocusing of moving ship in airborne SAR image is more evident. The Doppler rate shift induced by target motion in the airborne SAR system is greater than in the spaceborne SAR system. The Doppler rate has great influence on the image quality and a little error will result in image defocusing and decreased resolution. The degree of defocusing is directly related to the Doppler rate shift .The experimental results of ship05 are shown in The ISAR processing is not able to produce a well-focused target from airborne SAR images and the refocusing are not as good as for spaceborne SAR sub-images. The reason is that the IRD inversion is applied in SAR images for which the resolution is of the order of meters, while the resolution of airborne SAR images is higher than one meter. The inversion of high-resolution SAR images (higher than 1 m) would be need a more accurate inversion algorithm.The entropies and IC values change are illustrated in The entropies and IC values of the original and refocused images are calculated for quantitative evaluation of the focusing. The performance of the proposed method is compared with the DCT, and PGA methods and the original images. The entropies of sub-images with different method processing are presented in By exploiting the ISAR technique based on fast minimum entropy phase compensation, a refocusing method for moving ships in SAR images is proposed in this paper. The basic procedures of refocusing in SAR images are built. The processing data are SAR images rather than raw echo data. The algorithms inverting images to the raw data domain are described. This makes the processing flow simple and reduces the computational burden. The ISAR processing based on the fast minimum entropy phase compensation method iteratively obtains the phase error estimation by constructing a cost function of entropy. The experimental results based on spaceborne and airborne SAR data verify the effectiveness of the proposed method.Though the experimental results are good, the airborne SAR sub-images (resolution higher than 1 m) are not well focused via ISAR processing. The high-resolution SAR image inversion needs to be investigated in future work.


This paper simultaneously calibrates odometry parameters and the relative pose between a monocular camera and a robot automatically. Most camera pose estimation methods use natural features or artificial landmark tools. However, there are mismatches and scale ambiguity for natural features; the large-scale precision landmark tool is also challenging to make. To solve these problems, we propose an automatic process to combine multiple composite targets, select keyframes, and estimate keyframe poses. The composite target consists of an aruco marker and a checkerboard pattern. First, an analytical method is applied to obtain initial values of all calibration parameters; prior knowledge of the calibration parameters is not required. Then, two optimization steps are used to refine the calibration parameters. Planar motion constraints of the camera are introduced in these optimizations. The proposed solution is automatic; manual selection of keyframes, initial values, and robot construction within a specific trajectory are not required. The competing accuracy and stability of the proposed method under different target placements and robot paths are tested experimentally. Positive effects on calibration accuracy and stability are obtained when (1) composite targets are adopted; (2) two optimization steps are used; (3) plane motion constraints are introduced; and (4) target numbers are increased. Odometry and monocular camera have been widely used in indoor mobile robots owing to low cost and rich information. Fusing these two sensors for robot navigation is a popular research topic ,2,3. TheOdometry errors can be categorized into two types: systematic and non-systematic errors . The sysDetermining the relative pose between the camera and mobile robot is referred to in this paper as robot extrinsic parameter calibration. The purpose is to identify the best pose transformation to connect the mobile robot trajectory and camera trajectory. Estimating the robot trajectory typically resorts to wheel odometry ,13, wherIn fact, if robot extrinsic parameters were known, then a camera could be used as an external sensor to measure the robot pose and facilitate odometry calibration. Similarly, if the odometry parameters were known, then the robot extrinsic parameters could be easily obtained. Odometry parameter calibration and robot extrinsic parameter calibration are chicken–egg problems, which some researchers have tried to solve simultaneously. The analytical method ,18 is onIn this paper, an automatic method is proposed to calibrate the odometry and robot extrinsic parameters simultaneously. This approach automatically selects keyframes, calculates initial parameter values, and has no constraints on the robot path. First, we use multiple specially designed composite targets. Then, a new approach is introduced to combine these composite targets, select keyframes, and estimate the keyframe poses. Next, we use an analytical method to obtain initial values of the calibration parameters. Finally, two optimization steps are applied to determine the refined parameters. In addition, planar motion constraints are introduced into the optimization functions.The remainder of this paper is organized as follows. In b is the wheel spacing. Two coordinate frames are established, namely, the robot coordinate frame The system of a differential-drive mobile robot equipped with a monocular camera is shown in k is represented by k and b.The pose of a mobile robot moving on a 2D plane can be expressed as ry model :(1)xk+1Many different camera models have been proposed in the literature ,24. A unIn addition, the camera intrinsic parameters are assumed to be carefully pre-calibrated.The robot extrinsic parameters to be calibrated represent the relative pose between the camera coordinate frame servable , which hservable . This meb, In short, eight parameters must be calibrated in total: The proposed automatic calibration method is discussed in this section. First, the main ideas and a calibration pipeline are introduced. Then, four key steps in the solution are explained.The proposed method is illustrated in Multiple composite targets are used to improve the accuracy of camera pose estimation, thereby enhancing calibration accuracy. It is difficult for natural feature-based methods to achieve high precision camera pose estimation due to mismatching, scale problems for the monocular camera, or a lack of features in the environment. Commonly used artificial landmarks provide few corner points and suffer from low precision problems, such as binary square fiducial markers. The composite target proposed in this paper can produce many accurate corner points via the checkerboard. Using an aruco ID also avoids mismatches. However, a small size target or landmark tool only covers part of the image, resulting in low camera pose estimation accuracy. Moreover, a high precision, large target or landmark tool is difficult to produce, has poor portability, and is costly. Given these drawbacks, multiple targets are employed here.After the targets are laid out, the robot is controlled to move arbitrarily in front of them. The encoders and image data with timestamps are recorded. The layout of the targets and the robot trajectory are not limited, but for the precision considerations, the robot is recommended to move slowly to avoid wheel slippage. Additionally, as many targets as possible should be within the field of the camera.Some key images  method .lth image comes in, its pose will be estimated. Multiple targets may be observed in the image. The features of each observed target from the image will be extracted firstly. The corresponding targets are searched in the map using aruco IDs. After this, matches of 3D map points to 2D features are obtained. Then, the PnP method is used to get the camera pose After initialization, the keyframe and the target exist in the map. The system then enters a normal process. When the lth image is checked to determine whether new targets exist, if so, they will be added to the map. First, the PnP method is used to calculate the homogeneous transformation Next, the The map points on the new targets can be unified to the world coordinate frame.Then, the selection of keyframes is introduced under two conditions. First, at least two targets must be observed by the image to ensure sufficient constraints between the keyframes and the targets to achieve higher accuracy in subsequent optimization steps. Second, the angle change and distance between the current image and the last keyframe should be greater than thresholds lth image is constructed as the jth keyframe and added to the map.The lve step . When boith target and the jth keyframe, the error term is defined as:Q is the number of corners in the target, and qth 2D feature in the target extracted from the jth keyframe, qth 3D map point in the target expressed in the target coordinate frame When a target is observed in a keyframe, a line in λ={Tb2w,Tjth keyframe:The second and the third parts are constraints generated by the planar motion of the camera. The second part considers the keyframe positions. The error term is defined as the square of the distance from the keyframe position The third part considers the keyframe rotations. For a planar motion camera, the axes of rotations A, B, C, and D are obtained via data fitting using all keyframe positions. The initial values of Good initial values are necessary for global optimization. However, the keyframe and the target insertion process can generate cumulative errors, resulting in low precision of initial poses of keyframes and targets. To address this problem, a local optimization process is introduced to reduce error accumulation. The local optimization is performed when a newly inserted target jth keyframe and the K is the number of encoder samples between two adjacent keyframes; and N keyframes. If Because the robot moves on a 2D plane and the quation a1sEquation . Then, Na1sEquatiWith the proper choice of keyframes as in , Rank(Ω)ank(Ω)=4 . Thus, ΩEquation , we decoEquation  is givenClearly, In this step, only the calibration parameters are optimized by minimizing the odometry observation errors:jth and The error term is calculated between two adjacent keyframes: the ribution :(25)sL/kth encoder is The mean is calculated by the two encoder readings, and the left/right variance is proportional to the absolute value of the increment. Assuming that the 3 × 3 covariance of the robot pose at the time point of the n method :(26)Σo,Equation :(27)Σe,K encoder samples between the jth keyframe and the If there are Equation  with iniz is assumed to equal to the minimum of variances of x and y:It is the odometry observation covariance between the two adjacent keyframes. It should be noted that the odometry observation covariance used in Equation  is a 6 ×Lastly, a total optimization is performed to minimize all errors. Ceres Solver  is used Several experiments were performed to verify the accuracy and the stability of the proposed method and to test the effect of four strategies: the adoption of multiple composite targets, the use of two optimization steps, the introduction of planar constraints, and the increases of the number of targets.b is 0.32 m, the reduction ratio of the left/right motor is 7.2:1, and the encoder produces 1024 ticks per motor round with the frequency of 100 Hz. A daheng MER-302-56U3M/C camera with a Kawa LM3NCM wide-angle lens  was installed. The camera resolution was set to 2048 × 1536 pixels, and the frame rate was set to 10 Hz. A Thinkpad laptop collected the encoder readings and images with a rosbag tool [bag tool  providedThe purpose of this experiment was to test the stability of our method on different data. To verify the impact of the target placement form, as shown in To verify the method, three other variants were compared as follows. In the NoPlanar method, the planar motion error terms in Equations  and 30)30) were Because the thresholds were random when selecting the keyframes, the result of each run of the algorithm could differ. For each data in The calibration results of different methods varied. In terms of the mean values of the results, relatively small fluctuations across different data appeared in the proposed method and the NoPlanar method because the two optimization steps improved the stability. Additionally, these two methods used the composite targets compared with the Aruco method. The composite targets contain many more precision map points; hence, these two methods achieved higher stability. The standard deviations represent the effect of keyframe selection. The standard deviations of the Analytical method were the largest given the absence of the two optimization steps. By contrast, the standard deviations of the proposed method and the NoPlanar method were relatively small compared with the Analytical method and the Aruco method, further confirming the positive effects of the two optimization steps and the composite targets. Moreover, the means and standard deviations of the proposed method and the NoPlanar method were similar, indicating that the introduction of planar constraints exerted little influence on the means and standard deviations of the calibration results; thus, they had little effect on calibration stability.Some differences also emerged between different target placement forms in the proposed method. Although we could obtain the mean and variance of the calibration parameters, we could not test the accuracy of the calibration results directly due to the absence of real calibration parameter values. To address this problem, two indirect experiments were designed to verify the calibration accuracy of the odometry and the external parameters of the robot.If the odometry parameters are calibrated accurately, then the pose estimated by odometry will be accurate. An experiment was designed based on this assumption. The robot was controlled to run for a long distance and controlled back to the initial pose. Ideally, the end pose should be the same as the initial pose with a value of 0. However, due to errors in the odometry calibration, the two poses may be different. We used the endpoint error to indicate odometry calibration accuracy. Data were recorded in this experiment as shown in In most cases, the proposed method obtained the highest odometry accuracy due to using multiple composite targets, introducing planar constraints, and adopting two optimization steps. The odometry errors with calibration results from target placement form C were smaller than those from A and B, suggesting that the circular target placement form achieved higher odometry calibration accuracy. Although the means and standard deviations of the calibration results of the proposed and NoPlanar methods were similar, the proposed method achieved higher odometry calibration accuracy, potentially because the introduction of plane constraints improved the estimation accuracy of keyframe poses. To test the accuracy of the robot extrinsic calibration, as shown in To test the impact of the number of targets on the performance of the proposed method, 1–10 targets were used and arranged as presented in The odometry parameters calibrated by the proposed method were also compared with the design odometry parameters. The left/right wheel factor The two sets of odometry parameters were compared using the method in Our approach is similar to the work of . The autWe have to point out that the proposed method uses multiple composite targets that are a bit complicated to fabricate and arrange. In contrast, the natural feature-based method ,13,14 doIn this paper, we propose an automatic pipeline to simultaneously estimate the odometry and robot extrinsic parameters of a differential-drive mobile robot equipped with a monocular camera. This approach does not limit the path of the robot and does not require an initial assumption of calibration parameters, producing relatively more accurate and stable results than other methods. To address the low-accuracy problem of traditional artificial landmark tools, we propose a composite target consisting of an aruco marker and a checkerboard pattern and introduce a method to automatically combine multiple composite targets, select keyframes, and estimate keyframe poses. Initial values of the calibration parameters are computed by an analytical method and then optimized via two optimization steps. Several experiments were conducted to test the stability and accuracy of the proposed approach as well as the effectiveness and roles of key strategies. Results confirm the comparable performance of this method.


The interest in fisheye cameras has recently risen in the autonomous vehicles field, as they are able to reduce the complexity of perception systems while improving the management of dangerous driving situations. However, the strong distortion inherent to these cameras makes the usage of conventional computer vision algorithms difficult and has prevented the development of these devices. This paper presents a methodology that provides real-time semantic segmentation on fisheye cameras leveraging only synthetic images. Furthermore, we propose some Convolutional Neural Networks(CNN) architectures based on Efficient Residual Factorized Network(ERFNet) that demonstrate notable skills handling distortion and a new training strategy that improves the segmentation on the image borders. Our proposals are compared to similar state-of-the-art works showing an outstanding performance and tested in an unknown real world scenario using a fisheye camera integrated in an open-source autonomous electric car, showing a high domain adaptation capability. The most critical task for autonomous vehicles is understanding their surroundings. A good real-time scene-comprehension is vital to a vehicle so it can drive in an unknown environment in a safe way. The semantic segmentation task proposes a solution for this challenge based on image pixel-level classification in multiple semantic categories such as vehicles, pedestrians, traffic signals, etc., satisfying most of the vehicle needs in a unified way .The remarkable success of semantic segmentation solutions during the last few years has been closely related to the breakthrough of deep learning methods, which have proven to widely outperform previous state-of-the-art machine learning techniques ,3. AmongThe comprehension of a vehicle’s surroundings becomes even more challenging in complex environments such as urban traffic scenes, where the behavior of dynamic traffic participants like pedestrians or vehicles is unpredictable, or specific situations such as intersections or roundabouts that require big volumes of information to be adequately handled. Accordingly, a full real-time perception of the scene is a compulsory need for autonomous vehicles. Different sensors can be used in order to cover this need as cameras, LiDAR, radar, ultrasound, etc. Cameras clearly stand out among other solutions as they are able to generate real-time high-level semantic information while remaining easy to manage, cheap and present low power consumption.However, the limited field of view of traditional cameras complicates the management of complex environments since cameras are expected to cover the 360Fisheye cameras have started to play an increasingly important role in autonomous vehicles because of their ultra-wide field of view. These devices allow for acquiring more scene information using only a sensor at the cost of radial distortion in the images. With fields of view higher than 180 degrees, only two of these cameras are theoretically needed to cover the all of the vehicle’s surroundings. In addition, current autonomous vehicles are betting on redundant and robust perception systems. Fisheye cameras can clearly help in the achievement of these objectives in order to reach safe and reliable driving.Despite the discussed advantages, distortion associated with these cameras prevents the use of standard computer vision algorithms on the acquired images, making the integration into autonomous vehicles difficult. Furthermore, the application of deep learning techniques to these kinds of images presents many problems such as the lack of large-scale annotated datasets or the management of the distortion, which has caused that only some few works of the state of the art have focused on adapting current semantic segmentation methods to fisheye cameras.This paper is an extension of our previous conference publication . This woThis paper is organized as follows: The main issue with fisheye cameras is how to correctly handle distortion. Distortion is heterogeneous over the different fisheye image areas , being aAn initial approach to deal with the problem is the undistortion of the captured images in order to apply traditional vision techniques . In 11]11], an eAuthors in  successfNevertheless, none of the previous works achieved a good quality corrected image in a reasonable processing time as the image undistortion process has several difficulties: the strong dependency on intrinsic camera calibration parameters, the high consumption of computational resources that penalizes real-time processes and, finally, a remarkable loss of image quality, leading to information loss all over the image, but especially in the boundaries as shown in These inconveniences have caused the sprouting up of other approaches that try to adapt existing image processing techniques to work with the distorted images directly instead of the opposite. The lack of available large-scale annotated datasets for non-conventional camera images, like fisheye, has forced the generation of synthetic datasets with additional fisheye distortion leveraging existing ones like CityScapes . In 14]14], the The most relevant features to be learned in the CNN learning process are the appearance of the detected objects, their shape and their contextual information . PreviouMultiple ideas have been proposed to incorporate more context information in order to improve the results of the classification task. Most works have focused on obtaining wide receptive fields to capture valuable information. This can be achieved by including down-sampling stages followed by a set of convolutional layers. However, the down-sampling operation implies the reduction of the feature maps’ scale and, hence, the loss of information. In , dilatedIn conclusion, multiple ideas are currently proposed to improve CNN performance on fisheye images and handle fisheye distortion correctly, but most of them are not able to achieve real-time semantic segmentation on real fisheye images without resorting to manually annotated images during training.The generation of large-scale datasets involves expensive and time-consuming tasks such as data acquisition and the corresponding data annotation. Semantic segmentation tasks require pixel-wise annotations, which makes labeling extremely difficult for this kind of images. As we advanced in our precious publication , our appf is the focal length of the camera and Conventional pinhole cameras have a limited field of view defined by their imaging projection, according to the following Equation :(1)ρpinhIn the case of fisheye camera modelling, there are several mathematical models that can be used to design fisheye lenses. Among them, the most widespread one is the equidistant fisheye, which is described by Equation :(2)ρequif parameter, which determines the level of added distortion as shown on Equation dc=ftLeveraging the previous equation and the CityScapes dataset, a new collection of synthetic distorted images was produced to carry out the training of CNNs with fisheye images representing urban scenes. CityScapes is an optimal dataset for autonomous driving applications, as it is focused on urban scene understanding. It provides 5000 dense pixel-wise annotated images separated into three different subsets  and, additionally, another 20,000 with coarse annotations from 27 European cities with 19 classes for evaluation. In our case, only the fine full dataset, including training and validation subsets images, for both RGB and annotated images, were transformed. Original images were resized to 640 × 576, using bi-linear interpolation for RGB images and nearest-neighbor for label images in order to adapt them to our CNN architecture. Some examples of the final synthetic dataset can be seen in The features learned by a CNN during training rely mostly on the specific images used during this process. Therefore, these features are limited to the domains in which these images were acquired and should have the property to generalize to different domains. However, achieving robustness in other domains is not an easy task, and deep networks are often prone to overfitting even with thousands of training images. This difficulty is even greater when synthetic images are employed during training, given that appearance in synthetic images is less rich and varied than in real images.In order to obtain general features, data diversity must be high, due to the huge, different patterns CNNs are forced to learn to be able to distinguish between multiple categories in changing detection conditions. Data augmentation aims to enlarge the training datasets, preserving the available labels by applying different transformations.f parameter from Equation , texture augmentations   or speciEquation  aiming tThe demanding needs of real-time applications have boosted the development of efficient network architectures, leaving behind large deep architectures that achieved outstanding performances at the expense of the consumption of computational resources, using different ideas such as Conditional Random Fields (CRFs) , residuaOur previous proposal ERFNet  achievedWe adopt ERFNet as our baseline CNN architecture. The basic architecture of ERFNet is presented in The decoder module includes upsampling and convolutional layers and a final classification log–softmax loss layer. The decoder stage is meant to preprocess encoded features up to the input’s resolution and provide the final probabilities for each of the trained classes. Thus, the final layer is a volume with a number of slices equal to the number of classes, where each slice contains the per-pixel probabilities of that class. In order to take the predicted (or most probable class), the argmax of this volume is calculated. Many networks use a large decoder, but we chose a relatively small one because the decoder is only meant to upsample features and convert to probabilities, without affecting much to the extraction of good features. Therefore, the encoder does most of the feature extraction job and our light decoder transforms these representations into meaningful outputs in the shape of probabilities.Considering the significance of context information in fisheye images, we also study the use of an alternative architecture, consisting of replacing the original ERFNet decoder by a handcrafted pyramidal pooling module . Four diThe training of both architectures (baseline and modified) is divided into two different stages: on the first one, the encoder module is trained individually using downsampled annotations as ground truth during 90 epochs with a batch size of 6. For the second one, the complete architecture  are trained together to produce end-to-end semantic segmentation for another 90 epochs.c = 10 during the entire training.The Adam optimization of Stochastic Gradient Descent is used, starting with a learning rate of duced in wclass=1lThe black corners in As it can be seen, these regions present a very heterogeneous segmentation that harms the context information on the borders of the useful parts of the image, which are essential because they contain an important area of the total FOV of the camera. Context information is the most determinant feature for the segmentation of the image borders, due to the strong distortion they present. As a consequence, the performance of the CNN in this area is clearly degraded.In order to preserve this context information and improve the segmentation on the borders, a new training strategy is proposed. We identify those areas a priori and add an additional 20th class to represent them during training.For the validation of the proposed architectures, data augmentation and training strategies, different experiments are presented. The first one evaluates our data augmentation strategy and the second one studies the performance of the different architectures compared to other proposals of the state of the art.To prove the benefits of our data augmentation proposal, a comparative experiment with other approaches of the state of the art was carried out. For the fixed zoom augmentation, three datasets were generated as in  with f0=e did in . AdditioAccording to In a second experiment, an alternative synthetic dataset with a lower distortion , but it is improved by the addition of factorized restricted deformable convolutions (56.1%) and restricted deformable convolutions (57.9%). As shown in This experiment aims to demonstrate the generalization abilities provided by the suggested architectures and training techniques based on synthetic images obtained by using distortion models over normal FOV images, applying them to the images captured by a real fisheye camera, over an urban driving scenario similar to the one used during training, but never seen before. With that purpose, a HD fisheye camera with a 180The previous training was not adequate for the real fisheye camera, due to the difference between resolutions and aspect-ratio of the synthetic images regarding the real ones. A new specific training adapted to the real fisheye camera was carried out, using nine new images with random distortions between Both ERFNet and ERFNetPSP architectures were trained using the new range of distortions and resolution. A quantitative validation was performed using the transformed CityScapes validation subset for these new parameters. Once again, the ERFNetPSP reaches the best performance, obtaining a mean IoU of 69.6% while the baseline ERFNet achieves an IoU of 68.3% for this validation subset.f. Therefore, following the graphics, ERFNetPSP with ImageNet pre-training and full data augmentation obtains the best semantic segmentation results.Fisheye camera was integrated in an open-source autonomous car prototype ,34 as a Due to the absence of annotated ground-truth, only qualitative results are exposed in this section. To provide a convincing validation, results are split focusing on the main groups of segmented classes defined in Cityscapes, and using some representative Campus images captured from the autonomous vehicle. To facilitate the understanding of the segmentation, we provide the Cityscapes color legend in The “object group” segmentation, composed of the following classes: pole, traffic light and traffic signs, is shown in Segmentation of the “nature group”, which includes vegetation and terrain classes, and the “sky group”, which only contains the sky class, are well represented in all of the previous images. Their influence is secondary in autonomous vehicles’ applications mainly due to the fact that they are faraway from the driving area. However, there are some cases where nature classes define the limits of the road a and shoResults demonstrate that the ERFNetPSP architecture provides real-time good quality semantic segmentation being able to detect even the classes with reduced number of training data and showing a robust behaviour to shadows and lighting changes.Despite the good results, segmentation has still some problems dealing with glares (which are common in fisheye cameras due to its wide FOV) and with big classes with changing appearances such as the sky, as we can see in An additional problem is that appearances of the classes present near the edges, corresponding to lateral objects located on the left/right FOV limits, are not included in the training dataset, which is captured from a conventional FOV camera. This fact degrades the obtained segmentation in the edge regions, which tend to associate small classes with more available classes such as building, road or sky.This paper proposed a methodology to achieve real-time semantic segmentation based on ERFNet over real fisheye images, leveraging only synthetic images and, therefore, solving the lack of large-scale fisheye datasets while avoiding the heavy task of data annotation. The two introduced architectures (ERFNet and ERFNetPSP) achieve better results than the best state-of-the-art works in various synthetic datasets. Furthermore, the ability of ERFNetPSP to handle distortion by the prioritization of context information is proven, showing a better performance than ERFNet. The model also demonstrates better leveraging our data augmentation strategy, reaching the ERFNet performance in the original CityScapes dataset. Additionally, alternative training with an extra class to segment the image borders is presented. Our proposals have been validated with images taken from a real fisheye camera in unseen scenarios, showing a high capacity of domain adaptation without using a fine-tuning process with manually annotated data.Future work involves the development of a full 360