Fusarium oxysporum and Rhizoctonia solani, performed using two constructions of a low-cost electronic nose. The first electronic nose used six non-specific Figaro Inc. metal oxide gas sensors. The second one used ten sensors from only two models (TGS 2602 and TGS 2603) operating at different heater voltages. Sets of features describing the shapes of the measurement curves of the sensors’ responses when exposed to the odours were extracted. Machine learning classification models using the logistic regression method were created. We demonstrated the possibility of applying the low-cost electronic nose data to differentiate between the two studied species of fungi with acceptable accuracy. Improved classification performance could be obtained, mainly for measurements using TGS 2603 sensors operating at different voltage conditions.Electronic noses can be applied as a rapid, cost-effective option for several applications. This paper presents the results of measurements of samples of two pathogenic fungi,  Analysis and detection of odours can be performed using many techniques of chemical analysis of gases. On the one hand, most of the information can be obtained by classical chemical analytical techniques, such as gas chromatography combined with mass spectrophotomety . It alloDifferent types of gas sensors can be used in the development of electronic noses, such as optical , gravimeAn essential task in the process of electronic nose construction is a choice of gas sensor types. Multiple reports describe approaches to optimisation of the gas sensor arrays ,23,24,25When building a low-cost electronic nose, the constructors are limited in their choices of gas sensors to the models on the market. Therefore, the standard approach is to use electrical circuit parameters according to recommendations of the sensor manufacturers—for example, the sensor’s heater voltage, which determines the sensor’s temperature. On the other hand, since the MOX gas sensors are used for other purposes in electronic noses than originally intended, different operation conditions could be more appropriate. Fonollosa and coworkers  reportedProposed constructions, applications, or limitations and challenges of e-noses designed for studies of fungal, bacterial, and viral infections can be found in several review papers ,41,42,43Fusarium oxysporum and Rhizoctonia solani. Fungi of the genus Fusarium, belonging to Ascomycota, and Rhizoctonia, belonging to Basidiomycota, have been of interest to microbiologists worldwide, for many years. Currently, these fungi are counted among the most pathogenic and phytotoxic microorganisms in the world. They are plant pathogenic fungi with broad host ranges and worldwide distribution. The species R. solani was discovered over 100 years ago and is considered a soilborne pathogen, and F. oxysporum is currently considered a species complex comprising a variety of species and strains that are ubiquitous in soils [Herein, we report experimental results of a simple, low-cost electronic nose device. When in a sensor array, we used two sensors and five individual sensors working in different temperatures. The device was successfully used to differentiate between odours of two common forest fungi pathogens, in soils . Most ofin soils .F. oxysporum has expanded considerably, and many new particular forms and races have been described. In an extensive literature review, Edel-Hermann and Lecomte [F. oxysporum. The difficulty of traditional detection of these fungi in practice is that the anamorph  does not produce spores, while the teleomorph  does not produce fruiting bodies, but only sexual spores, called basidiospores. A device that detects pathogens in the soil of the rhizosphere or at the base of the stems would be precious for early detection of pathogens and planning preventive measures.The most recent review paper, still referred to by many researchers, dates back more than 35 years . The aut Lecomte  enumeratAspergillus terreus, A. holandicus, and Eurotium chevalieri. In vitro tests showed that the conductive polymer sensor matrix exhibited different responses to each species at the two moisture levels. Discriminant functional analysis of the data showed discrimination between the control and the fungi. Cluster analysis showed significant separation of the control from each of the paper-damaging fungi. In situ tests on three paper grades showed that the volatile pattern produced by each fungus differed from the patterns of the others and the control when natural substrates were used. The results obtained were better at higher moisture contents. The three paper grades were successfully differentiated into clusters. The differentiation between the control and the perishable fungi was better for a single paper grade with higher moisture content. This study showed that this technology can detect fungal decay in library and archival materials early to protect cultural heritage better. We also want to acknowledge here a paper by Falasconi et al. [Fusarium verticillioides in corn is reported.Some authors  performei et al. , in whicIn the present paper, we report the results of measurements performed by two electronic nose devices. PW4 was already described in more detail in the previous paper , and PW6Pythium intermedium and Phytophthora plurivora. We also noticed that the low-cost electronic nose device could be simplified, as the best performing classification was obtained when data extracted from just one sensor signal were used. It was the TGS 2603 sensor, but also, in some cases TGS 2602 sensor data allowed us to achieve similar classification performance. In the current study, we have extended our research to detect odours of pathogenic fungi that are not closely related to oomycetes. Therefore, it can be expected that the emitted metabolites contain similar or different chemical components and yet can be detected by the same type of gas sensors.The operation of an electronic nose relies on measuring the resistances of non-specific gas sensors exposed to the studied odour sample. In the previous papers ,54,, we Sensor models TGS 2602 and TGS 2603 —the latter especially—are designed to detect low concentrations of odorous gases generated from waste or spoiled materials. Hence, it is not surprising that they react to the gases emitted by the growth of oomycetes or fungi, as this is a similar biological process of organic material consumption as during food spoilage.When building a low-cost electronic nose based on MOX sensors, we were limited to commercially available sensors, as manufacturing our own sensors would have been too costly. One idea for improving electronic nose performance is to use more sensors with overlapping target gases but with different response characteristics and to then rely on the machine learning algorithms to differentiate between response patterns. For that task, it would be possible to use sensors from various manufacturers. Another idea exploited in this paper is to change the conditions of sensor operation, which can be achieved by changing the sensor’s heater voltage, and thus the operation temperature. This approach is similar to proposals of using an array of custom-made gas sensors with a temperature gradient ,38.It may be interesting to recall the construction of an MOX-type gas sensor and its operation principle. The main elements of the sensor are a sensing material, typically tin dioxide, and an electric heater, usually made of platinum. When the sensing material is heated, usually to the temperature of a few hundred degrees Celsius, in clear air, oxygen is absorbed on the surface of the sensing element and attracts donor electrons, preventing electric current flow. In reducing gases, oxygen reacts with the gas particles and the surface density of adsorbed oxygen decreases; donor electrons are released into the tin dioxide, leading to electric current flow. This means that the temperature is an important parameter for controlling the physical processes of this gas sensor’s operating principle.To some extent, an MOX gas sensor behaves differently at different temperatures, and may exhibit different sensitivities and selectivities to various chemical components. Indeed, in Compared to PW4, the PW6 device has upgraded hardware: a 16-bit ADS1115 AD converter instead of a 12-bit one. To control more sensors, a better model of the Atmega microchip controller (Atmel 0814G Atmega 8-16PU) was used. Additional delays were added to ensure that the time of one measurement sequence was the same for both devices. As the sensors’ heaters voltage stability was critical in this experiment, an external power supply was used to power them. Sensors were power supplied, just as in PW4, via a computer’s USB port. This cable was also used to control the devices and send the data to the computer, where they were stored.The PW6 electronic nose device was built using ten metal oxide gas sensors  and humidity and temperature sensors. We decided not to use any air pumps or fans to force airflow to reduce costs and complications. The sensors were placed in an aluminium cover that just fit into Petri dishes 9 cm in diameter. The wires that connect sensors and multiplexers were connected alternately to signal and ground wires, to better shield electrical noise.The main difference in the PW6 e-nose, compared to the PW4, was the use of only two types of sensors, but each of them had an extra resistor R in the sensor’s heater circuit, presented in Due to the different original resistances of the heaters . Pathogens in this form can remain latent in the soil for a very long time and develop when optimal development conditions  occur. It is one of many reasons why their early detection is so essential in nursery production.Damping-off is a disease that causes decay of germinating seeds and young seedlings, especially those that grow in forest nurseries . Among aF. oxsyporum is among the top 10 most destructive fungal plant pathogens worldwide. The pathogenic strains of F. oxysporum are responsible for two types of symptoms. The most common is vascular wilt, and in some cases, rotting. In vascular wilt, Fusarium invades host roots in the xylem vessels, which it colonises upward, causing progressive yellowing and wilting of the plant. While residing in the soil, it can decompose dead organic matter (saprotrophic), or infest and damage healthy plants (parasite), or live a latent life in their tissues (endophyte). Over time, the endophytic presence of the pathogen may be manifested by reduced plant growth, followed by wilting, chlorosis, and premature death of the plant. This shows how important it is to be able to detect it at an early stage. The fungus F. oxysporum is the only representative of the genus Fusarium that develops in the vascular system of the host plant and spreads upwards within the plant tissue, mainly affecting the bark layer. Other fungal species spread outside the plant tissue. In the presence of suitable host plants, Fusarium changes its mode of life to parasitic . An infection by a pathogen of the genus Fusarium occurs more readily when the root system is mechanically damaged; nematode damage is particularly favoured.The pathogenic strains mentioned above develop readily in soil, which is their natural habitat. Their adaptability has led them to being considered one of the most damaging microorganisms in the world today. They cause root rot ,63, tubeF. oxysporum is that there are harmful, toxic, and pathogenic strains and beneficial ones. Some strains of F. oxysporum, such as Fo47 and CS-20, are indeed beneficial to the host and may even protect against root pathogens. At this stage, our research should only distinguish between F. oxysporum and R. solani in order to use effective selective fungicides. An additional consideration is that the F. oxysporum species complex mutates frequently, and virulence-related genes may lead to the continual emergence of new races within the species [The difficulty of controlling a species as complex as  species ,66,67,68R. solani, in turn, lives in the soil as vegetative hyphae or spores called sclerotia. The thick cell walls of the spores allow them to survive in the soil for many years. They begin to develop when exposed to chemicals secreted by a growing plant, or when exposed to chemicals produced during the decomposition of organic plant debris. Capturing these volatiles with an e-nose is a solution that provides some promise. The fungus can enter the plant through natural openings and through the epidermis. Upon contact with the plant, the cuticle of the pathogen produces appressoria and infectious hyphe that penetrate the plant cells. Enzymes produced by the pathogen are involved in the dissolution of the cuticle.Quercus robur) and were kept in the stock of the Department of Forest Protection in the Forest Research Institute in Sękocin Stary (Poland). The fungal samples were cultured on classical PDA-Agar media  in 9 cm Petri dishes. They were kept at room temperature until the mycelium grew over the surfaces of their dishes completely.The fungal strains were isolated in a forest nursery (Chojnów Forest District) from the rhizosphere soil of diseased European oaks  were the baseline. Next, the sensors were placed manually on the Petri plate with the sample. That made a small gas chamber under the sensors. After 100 readings, each sensor’s probe was removed and placed away from the samples. The last 500 resistance readings (6 min 10 s) recorded relaxation and sensor cleanings. When not being measured, all samples on Petri dishes were covered to avoid any infections. Electronic noses and samples were stored in a laminar flow cabinet to ensure stable environmental conditions.All data used in this paper were measured between the 23rd of June and the 7th of July 2021. That makes 10 measuring days, so any random noise in the environment could be reduced. The sensors were power supplied and heated during all those days. We kept them clean. For each of the specimens, 35 measurements were collected. That makes a total of 105 records to analyse.In our experiment, we maintained controlled conditions of constant temperature and humidity throughout the experiment. The sensitivity of the MOX gas sensors is highly dependent on these parameters, and we were interested in finding differences in the responses of the sensors to the odours emitted by the test samples. Since we held the humidity constant, we could consider a sensor’s response to it as a background signal, which was the same in all cases.The data analysis that we performed consisted of several steps. First of all, the sensors’ response resistance data, which are presented in The number of reads collected during one measurement of a sample was 3600 for the PW4 e-nose and 6000 for the PW6 e-nose device—each represents the number of sensors multiplied by the number of reads of sensor resistance magnitude in both the gas adsorption and desorption phases, as one can observe in Principal component analysis is an unsupervised method of data space transformation to a new space represented by factors. The magnitude of a factor is related to the variability of the dataset captured by the given factor. The factors construct a coordinate system in which the first coordinate vector sets the axis for which the variance of the dataset points is maximal, then the second coordinate vector sets the axis perpendicular to the first one and contains the maximum of the remaining variance of the dataset, and so on. The PCA transformation can be intuitively interpreted as a rotation of the coordinate system. One of the applications of PCA analysis is the reduction of the dimensionality of the dataset, as it allows the maximum of the content of the data variability in each of the components and decides to discard the less critical ones. In our analysis, the PCA method was used for visualisation purposes only. This allowed us to examine the data distribution in less dimensional space and understand the patterns. Since the input variables are represented in various measurement units in the studies we used and have various ranges, the input data were standardised, which allowed us to treat them on an equal footing.To build a machine learning classification model, we used the logistic regression method. The model’s performance was evaluated using an accuracy measure, defined as the proportion of correctly classified samples to all samples in the scored dataset. The accuracy was calculated using the cross-validation (CV) technique.It may be interesting to elaborate in more detail on the CV procedure used in our analysis. As we already mentioned, the measurements were performed over 10 days, and every day of measurement, each sample on a Petri dish was measured twice. In our opinion, that should be considered in the proper splitting of the experimental data to the training, validation, and testing datasets. Data collected from measurements of the same sample should not be used in both operations: model building and testing. In addition, as we noticed last time , the meaIt is essential to notice that, in the dataset we prepared, the number of modelling features is quite large, and to avoid overfitting, we performed a variable selection procedure using the recursive variable selection method . This taIn Rhizoctonia samples is larger than the separation between medium only and Fusarium. This may indicate that the classification models detecting Rhizoctonia perform better than the classification models differentiating between medium and Fusarium samples. In addition, we can observe that differentiation between the two studied species of fungi F. oxysporum and R. solani may be more challenging, as the data distributions for these samples are not totally separated and have a small overlapping region. As we present in the next section, such intuitive observations about the classification performances are confirmed by the accuracy measures based on the PCA visualisation.As one can observe in the following figure, there is a clear distinction between samples infected by fungi and non-infected samples containing only the growth medium. Additionally, we can notice that the separation between medium only and The first result of machine learning classification modelling that we would like to present compares the model’s performance using data collected by the PW4 and PW6 electronic noses. In For further analysis, we built more models to verify the performances of various cases, when data collected by just one of the sensors in the PW6 electronic nose were used for the model training, or when we used data collected from the sensor working at its heater voltage suggested by the manufacturer together with other sensors working at lower voltages . We notiFusarium samples, the difference was in the order of six percent, which considering the number of data points, is equivalent to two more samples being correctly classified. The highest increase of the classification accuracy was observed for the multinomial classification when the classification of all categories was attempted, and the increase was in the order of nine percent, which is equivalent to about five samples.As one can notice, in most cases, we achieved better accuracy performance when the heater voltage was lower than the nominal one. In the case of differentiation between medium and In One final note is worth mentioning. As we discussed in the previous subsection, the distribution of data presented in Phytophthora and Pythium. In the current paper, we reported further development of the constructed low-cost electronic nose device with an application to pathogenic fungal differentiation (Fusarium oxysporum and Rhizoctonia solani). Fungi of the genus Fusarium and Rhizoctonia are considered some of the most pathogenic and phytotoxic microorganisms in the world. They are plant pathogens with vast host ranges—worldwide, in fact. They are the most abundant pathogens in forest nurseries in Poland, where they cause damping-off, the disease that leads to the decay of germinating seeds and young seedlings.Electronic noses have been proposed as a rapid and non-invasive diagnostic tool in many industries and agriculture applications. In our previous papers ,54 we deThe first low-cost electronic nose created by our team (PW4) ,54 consiIn the data analysis part of the research, we applied a well-established methodology. In the first step, the responses of the sensors to the presence of odours were transformed, and a set of features describing the shapes of the response curves were applied. Then the classification models using the logistic regression method were trained, and in this process, the variable selection procedure was applied. The accuracies of classification models were estimated using the cross-validation method.The results demonstrated the possibility of differentiating between the studied samples using the data collected by the low-cost electronic nose devices. We also demonstrated an improvement of classification performance when data of the second proposed device were used.


Background: There is a current lack of any composite measure for the effective tracking and monitoring of clinical change in individuals exposed to repetitive head impacts (RHI). The aim of this study is to create a composite instrument for the purposes of detecting change over time in cognitive and behavioral function in individuals exposed to RHI.Methods: The data to derive the composite instrument came from the Professional Fighters Brain Health Study (PFBHS), a longitudinal study of active and retired professional fighters [boxers and mixed martial arts (MMA) fighters] and healthy controls. Participants in the PFBHS underwent assessment on an annual basis that included computerized cognitive testing and behavioral questionnaires. Multivariate logistic regression models were employed to compare active fighters (n = 117) with controls (n = 22), and retired fighters (n = 26) with controls to identify the predictors that could be used to differentiate the groups over time. In a second step, linear discriminant analysis was performed to derive the linear discriminant coefficients for the three groups by using the predictors from the two separate logistic regression models.Results: The composite scale is a weighted linear value of 12 standardized scores consisting of both current and yearly change scores in domains including: processing speed, choice reaction time, semantic fluency, letter fluency, and Barrett Impulsiveness Scale. Because the weighting of values differed between active and retired fighters, two versions emerged. The mean and standard deviation ratio (MSDR) showed that the new index had better sensitivity compared to the individual measures, with the ratio of MSDR of the new index to that of the existing measures of at least 1.84.Conclusion: With the increasing need for tools to follow individuals exposed to RHI and the potential of clinical trials on the horizon for CTE, the RHICI is poised to serve as an initial approach to a composite clinical measure. There is increasing interest in understanding the long term neurological effects of exposure to repetitive head impacts (RHI) including Chronic Traumatic Encephalopathy (CTE), with the anticipation of developing therapeutic interventions –3. The tTo develop and validate new test batteries takes significant time and effort. Composite scales can be produced by either consensus  or data driven  strategies –6. The dPrevious work on neuropsychological and behavioral changes in those exposed to RHI have tended to be cross sectional and have reported deficits in domains such as memory, information-processing speed, finger-tapping speed, complex attentional tasks, and frontal-executive functions –9. HowevThe aim of this study is to create a composite instrument that could detect change over time in clinical measures in individuals exposed to RHI. This was accomplished by utilizing data obtained from the Professional Fighters Brain Health Study . The comThe data used for the RHICI came from the PFBHS, a convenience sample consisting of active and retired professional fighters (boxers and MMA fighters) and healthy controls. Active fighters were required to have at least 1 professional fight within 2 years of enrollment and be training with the intent to compete. Retired fighters were included if they had been boxers, had a minimum of 10 professional fights, had no sanctioned fights for at least 2 years, and did not intend to return to competition (there were too few retired MMA fighters to include as a separate group). Control subjects were recruited from outreach efforts in the community and could not have any self-reported prior history of neurological disorders, head trauma, military service, or participation at a high school level or higher in a combat sport or a sport in which head impacts can be anticipated to occur, such as American football, wrestling, hockey, rugby, soccer, or rodeo. All participants were required to be able to read at a minimum at a 4th grade level but were not otherwise screened for cognitive status or subjective complaints prior to enrollment.Enrollment in the PFBHS began in 2011 and has been continuous since. Each participant is seen on an annual basis, and for active fighters, not sooner than 45 days after a sanctioned fight. The PFBHS was approved by the Cleveland Clinic Institutional Review Board and written informed consent was obtained from all participants. More detailed methods of recruitment and study procedures have been described previously .At baseline and each annual visit, a battery of tests and information were acquired including magnetic resonance imaging (MRI) of the brain, computerized cognitive testing, behavioral questionnaires, and exposure history. Participants answered questionnaires with the assistance of the study coordinator that collected information on demographics; educational attainment; medical history including concurrent illnesses and prescribed medications; previous head trauma, both related and unrelated to athletic activities; and prior involvement in other contacts sports.Cognitive function was assessed by two computerized cognitive test batteries and verbally administered measures of verbal fluency. One computerized battery consists of four subtests of the CNS Vital Signs™  including verbal memory, symbol digit coding, Stroop and a finger tapping test. CNS Vital Signs™ offers robust and reliable measurements of cognition, which are computerized; test performance is supervised by a technician . ResultsBehavioral assessment was obtained through administration of the Patient Health Questionnaire (PHQ 9) that evaluates the presence of depressive symptoms and the Barrett Impulsiveness Scale (BIS II) , 20. Then = 22) were used as the reference group with the assumption that they were the least likely to show decline in cognitive measures over a several year period. The conceptual framework informing the index was to identify and create a composite from those cognitive and behavioral measures that were able to best differentiate over time the active fighters (n = 117) from the controls and the retired fighters (n = 26) from the controls.Data from the PFBHS study was used to develop a new composite index for individuals exposed to RHI. We included participants having at least three study visits over a minimum of 2 years. The normal controls , the study team adjudicated which one was clinically more important.The initial step was to examine the correlation of the cognitive and behavioral variables with each other in a correlation matrix. This was performed to avoid the problem that could be caused by multiple co-linearity—when two highly correlated variables are both included in a statistical model may result in the sign change of parameter estimates. When two or more measures were highly correlated  and 4 were behavioral measures (B). Although we had longitudinal data from multiple time points, the change of the last measure from baseline was used to avoid the violation of the linear assumption of the longitudinal measures. For each measure, we utilized the score and the average yearly change in score based on the last measure and the baseline, comprising a total of 32 predictors.Multivariate logistic regression models were employed to compare active fighters with controls, and retired fighters with controls to identify the predictors that could be used to differentiate the groups. In a second step, linear discriminant analysis was performed to derive the linear discriminant coefficients for the three groups by using the predictors from the two separate logistic regression models.Assuming that CA and BA represent the average yearly changes of cognitive measures and behavioral measures, respectively, we standardized all the measures by using the range of each measure from all samples. The multinomial logistic regression between the active fighter group and the control group was presented as:p-value of 0.3 as the threshold to select the measures that can be used in the next step in developing a new composite index. The selected predictors that did not meet the direction were removed from the final model. The final model had a total of 12 measures among the 32 predictors from cognitive and behavioral assessments.where g(p) = log[p/(1–p)] is the logit link function, X are the observed data, and Y is the group with 0 for the control group and 1 for the active fighter group. A similar logistic regression model was used for the retired fighter group using the control group as the reference group. We utilized the backward model selection method with the In the final step of composite construction, we performed discriminant analysis in which the group variable  was used as the outcome, and the 12 measures were included in the model as independent variables. Discriminant analysis was performed to derive the linear discriminant coefficients for the 12 measures in each group. We then rescaled the new index to a range of 0–20 with lower scores considered to be normal.The characteristics of the participants that make up the data set are shown in The range of the new index was from 0 to 20. We evaluated the performance of the new index in the active fighters as compared to the individual measures with regards to sensitivity. The sensitivity was calculated as the mean and standard deviation ratio (MSDR) over a fixed follow-up time . A larger MSDR value represented a larger effect size, which leads to greater sensitivity.We computed the MSDR using the 2-year data of each fighter . The MSDR  showed tWe showed substantial improvement in sensitivity as compared to the individual measures. The improved sensitivity of the RHICI would be expected to reduce the sample size for a clinical trial if used as an outcome measure. As an example, consider a hypothetical 2-year study to detect a 5% decline from baseline for active fighters using the existing measures, to attain 90% power at the significance level of 0.05. The required sample size based on processing speed score was 126; it was reduced to 25 when the new index was used. Similar results were observed when other measures were used in sample size calculations.Assessment tools that are composed of elements of existing scales thought most likely to detect change in a particular disease are commonly used as outcome measures in clinical trials and observational studies . There aThough there has been increasing attention to the long-term sequelae of RHI including CTE, no validated composite measure is available. The RHICI is a first attempt using data-driven methods to construct a composite scale to be used in tracking longitudinal change and as a potential outcome measure in studies related to neurological effects of RHI. The RHICI has the advantage of easy administration on a desktop or tablet computer, with most subtests performed on the device itself or the score entered by the person supervising the testing .The components that constitute the RHICI currently, including measures of processing speed, attention, executive function and impulsivity, are consistent with domains that have been reported in the literature as affected in boxers and other groups exposed to RHI , 24, 25.The structure and weighting of items in the RHICI differed between active and retired fighters. The reason for this difference is speculative. One possibility is that the underlying pathophysiologic process may differ between these two groups. In the active fighter group, RHICI may be measuring the effects of accumulating axonal injury whereas the retired fighters may include some who are harboring a neurodegenerative process due to CTE. Longitudinal MRI regional volumetric data suggest such a dichotomy . The praA unique feature of the RHICI is the inclusion of both current values and rate of change from baseline in several elements. The RHICI is not intended to be used as a diagnostic test though a baseline score can be calculated to provide an anchor point from which clinical trajectory can be followed. Commonly used composite scales include measures that are thought to be characteristic of a particular condition and can differentiate patients from those without the disease. However, it is possible that the rate of change of some of these components may also be informative. By specifically assessing how certain tests change over time and integrating these elements into the RHICI, we expect this instrument to be of particular value in monitoring longitudinal change or as an outcome measure in clinical trials.The use of composite measures as endpoints in clinical trials have been encouraged by the Food and Drug Administration (FDA) and European Medical Agency (EMA) provided that the measures are carefully designed, are relevant to existing tools for which historical evidence exists and are validated in independent prospective cohorts , 27. GivMoving forward, it will be essential to assess the RHICI in other cohorts exposed to RHI if this tool is to have broad application. While the version we describe is derived from professional combatants, it is recognized that the clinical presentation of those exposed to RHI is heterogenous and potentially related to the type of exposure to RHI . For exaIn developing the RHICI, we utilized all the cognitive and behavioral data collected from the PFBHS, a longitudinal observational study of active and professional fighters. However, it is important to acknowledge the limitations of this study. The extent of testing included several computerized cognitive batteries and self-administered behavioral inventories but was not exhaustive, particularly in surveying memory and behavioral dysregulation. The number of participants (particularly retired fighters and controls) who had complete data for at least three time points was limited as was the number of controls in the cohort. In regard to the latter, we would anticipate as more data become available from our cohort and others that the elements that comprise the RHICI may be modified or weighted differently. The computerized assessments used are proprietary, placing limitations on their widespread use.p-values above 0.05, while letter fluency total correct score fails to meet the normality with the computed p-value below 0.05. A mathematical function  may be used to transform data to meet the normality assumption.For the statistical models, several assumptions are made. With multiple regression analysis and linear discriminant analysis, it is assumed that there is a linear relationship between the outcome and each independent variable. However, this assumption may not be satisfied in clinically collected data and could lead to errors in interpretation Furthermore, such models could be sensitive to outliers . Another assumption in these two statistical models is the normality of independent variables. For the three individual measures, we tested their normality by using the Kolmogorov-Smirnov test. The normality of processing speed and that of choice reaction time were satisfied with the Repetitive Head Impact Composite Index, a sensitive measure to change in executive function, provides a standardized approach evaluating long-term effects of RHI.There is an increasing need for tools to effectively track individuals exposed to RHI. With the potential of treating the consequences of RHI via clinical trials on the horizon, the proposed The raw data supporting the conclusions of this article will be made available by the corresponding author upon reasonable request.The studies involving human participants were reviewed and approved by Cleveland Clinic. The patients/participants provided their written informed consent to participate in this study.CB, GS, LB, and JC were involved in design of study, analysis and interpretation of the results, and writing of manuscript. GS performed statistical analysis. JA was involved in instrument development and interpretation of results. All authors read and approved the final manuscript.CB has received research funding from the Ultimate Fighters Championship, Top Rank promotions, Haymon Boxing, and Bellator/Spike TV. JA has authored intellectual property related to the C3 Logix software. JC has provided consultation to Acadia, Actinogen, AgeneBio, Alkahest, Alzheon, Annovis, Avanir, Axsome, Biogen, Cassava, Cerecin, Cerevel, Cognoptix, Cortexyme, EIP Pharma, Eisai, Foresight, Gemvax, Green Valley, Grifols, Hisun, Karuna, MapLight, Novo Nordisk, Nutricia, Orion, Otsuka, ReMYND, Resverlogix, Roche, Samumed, Samus Therapeutics, Third Rock, Signant Health, Sunovion, Suven, and United Neuroscience pharmaceutical and assessment companies. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Is it possible to define a set of performance indicators (PIs) for clinical work in ART, which can create competency profiles for clinicians and for specific clinical process steps?The current paper recommends six PIs to be used for monitoring clinical work in ovarian stimulation for ART, embryo transfer, and pregnancy achievement: cycle cancellation rate (before oocyte pick-up (OPU)) (%CCR), rate of cycles with moderate/severe ovarian hyperstimulation syndrome (OHSS) (%mosOHSS), the proportion of mature (MII) oocytes at ICSI (%MII), complication rate after OPU (%CoOPU), clinical pregnancy rate (%CPR), and multiple pregnancy rate (%MPR).PIs are objective measures for evaluating critical healthcare domains. In 2017, ART laboratory key PIs (KPIs) were defined.A list of possible indicators was defined by a working group. The value and limitations of each indicator were confirmed through assessing published data and acceptability was evaluated through an online survey among members of ESHRE, mostly clinicians, of the special interest group Reproductive Endocrinology.The online survey was open for 5 weeks and 222 replies were received. Statements  were considered accepted when ≥70% of the responders agreed (agreed or strongly agreed). There was only one round to seek levels of agreement between the stakeholders.Indicators that were accepted by the survey responders were included in the final list of indicators. Statements reaching less than 70% were not included in the final list but were discussed in the paper.Cycle cancellation rate (before OPU) and the rate of cycles with moderate/severe OHSS, calculated on the number of started cycles, were defined as relevant PIs for monitoring ovarian stimulation. For monitoring ovarian response, trigger and OPU, the proportion of MII oocytes at ICSI and complication rate after OPU were listed as PIs: the latter PI was defined as the number of complications (any) that require an  medical intervention or hospital admission (apart from OHSS) over the number of OPUs performed. Finally, clinical pregnancy rate and multiple pregnancy rate were considered relevant PIs for embryo transfer and pregnancy. The defined PIs should be calculated every 6 months or per 100 cycles, whichever comes first. Clinical pregnancy rate and multiple pregnancy rate should be monitored more frequently (every 3 months or per 50 cycles). Live birth rate (LBR) is a generally accepted and an important parameter for measuring ART success. However, LBR is affected by many factors, even apart from ART, and it cannot be adequately used to monitor clinical practice. In addition to monitoring performance in general, PIs are essential for managing the performance of staff over time, and more specifically the gap between expected performance and actual performance measured. Individual clinics should determine which indicators are key to the success in their organisation based on their patient population, protocols, and procedures, and as such, which are their KPIs.The consensus values are based on data found in the literature and suggestions of experts. When calculated and compared to the competence/benchmark limits, prudent interpretation is necessary taking into account the specific clinical practice of each individual centre.The defined PIs complement the earlier defined indicators for the ART laboratory. Together, both sets of indicators aim to enhance the overall quality of the ART practice and are an essential part of the total quality management. PIs are important for education and can be applied during clinical subspecialty.This paper was developed and funded by ESHRE, covering expenses associated with meetings, literature searches, and dissemination. The writing group members did not receive payment.Dr G.G. reports personal fees from Merck, MSD, Ferring, Theramex, Finox, Gedeon-Richter, Abbott, Biosilu, ReprodWissen, Obseva, PregLem, and Guerbet, outside the submitted work. Dr A.D. reports personal fees from Cook, outside the submitted work; Dr S.A. reports starting a new employment in May 2020 at Vitrolife. Previously, she has been part of the Nordic Embryology Academic Team, with meetings were sponsored by Gedeon Richter. The other authors have no conflicts of interest to declare.This document represents the views of ESHRE, which are the result of consensus between the relevant ESHRE stakeholders and where relevant based on the scientific evidence available at the time of preparation.The recommendations should be used for informational and educational purposes. They should not be interpreted as setting a standard of care, or be deemed inclusive of all proper methods of care nor exclusive of other methods of care reasonably directed to obtaining the same results. They do not replace the need for application of clinical judgment to each individual presentation, nor variations based on locality and facility type.Furthermore, ESHREs recommendations do not constitute or imply the endorsement, recommendation, or favouring of any of the included technologies by ESHRE. WHAT DOES THIS MEAN FOR PATIENTS?Performance indicators (PIs) are seen as a valid way to check that the healthcare provided is high in quality and operates within acceptable limits. Although there already is a list of PIs for the laboratories working on infertility and ART , a list of PIs for the ART clinical work is lacking. This paper looks at PIs for ART clinical work for different steps of the ART process, starting from fertility workup and diagnosis, extending to ovarian stimulation and oocyte pick-up (OPU), and ending with embryo transfer and the birth of a healthy baby. After consensus was reached in the working group, the PIs were sent through an online survey to find agreement within a group of medical experts. Finally, six PIs were defined: cycle cancellation rate (before OPU), rate of cycles with moderate/severe ovarian hyperstimulation syndrome, the proportion of mature oocytes at ICSI, complication rate after OPU, clinical pregnancy rate, and multiple pregnancy rate.The working group concluded that it was important for each clinic and/or individual doctor to monitor and check their own performance regularly using these PIs. The PIs are an important step towards improving the outcomes of ART and achieving high-quality ART services.Treating infertility with ART is a complex process combining clinical work and laboratory procedures. The process involves different steps, including hormonal stimulation and monitoring of the associated ovarian response, oocyte pick-up (OPU), fertilization, embryo development, and/or cryopreservation in the laboratory, and intrauterine embryo transfer (ET), leading to implantation, pregnancy and eventually, the birth of a healthy child  . SystemaFocussing only on the predefined objective of assisted reproduction, i.e. the live birth of a healthy child, as a proxy for quality may be problematic as reaching this objective is known to be affected by factors other than the quality of care  and together they aim to improve the overall quality and the outcome of ART practice.The current paper was developed based on the methodology for the development of ESHRE good practice recommendations (The working group composed a list of possible indicators and drafted their definitions . A meetiThe (narrative) summary of evidence and final list of PIs were discussed and agreed upon during a 2-day meeting in Maribor (November 2019). The results of the meeting  were used as the basis for a survey with the aim of assessing acceptability on a 5-point Likert Scale. The online survey (developed in SurveyMonkey) was distributed on 07/05/2020 to the members of the SIG Reproductive Endocrinology  and the members of the ESHRE committee of national representatives (CNR) (n = 68). The survey was open for 5 weeks. Statements  were considered accepted when 70% of the responders agreed (agreed or strongly agreed). Only indicators that were accepted by the survey responders were included in this consensus paper  and II. Additionally, the CNR members were asked to provide minimum expected, or competence, values , and aspirational, or benchmark, values  for the different PIs based on national data (if available), or data from their own centre. Mean and 95% CI were calculated from the collected competence values. If a range, rather than a fixed number was suggested, the upper or lower value was used, depending on the indicator. The upper value was used for all indicators aiming at maximum values, i.e. proportion of MII oocytes at ICSI (%MII) and clinical pregnancy rate (%CPR). The calculated values from the literature search and those reported by the CNR members were used as a basis for the working group to define appropriate competence and benchmark values for each indicator. The final draft of the paper was discussed among the working group members before publication.This paper describes PIs in four steps of a standard ART process: diagnosis of infertility and indications for ART treatment; ovarian stimulation; monitoring of ovarian stimulation, trigger, and OPU; and ET and pregnancy. In addition, indicators for training and competence are described .The survey, which consisted of a total of 31 statements and formulas on PIs described for each step of the ART process, received 222 replies. Overall, an agreement was reached for 83.87% of the statements, meaning at least 70% of the survey responders (strongly) agreed with the statement . AgreemeSuggestions for competence (minimum expected) and benchmark  values for the different PIs were received from 11 countries . The competence and benchmark values based on the data provided by the national representatives combined with the competence values calculated from published data are available in In the Vienna consensus on laboratory PIs, a reference population was defined as female patients <40 years old, using own fresh oocytes, ejaculated spermatozoa (fresh or frozen), any insemination method (i.e. routine IVF and ICSI), and no preimplantation genetic testing (PGT). Unless specified differently, this paper uses the same reference population. For PIs related to ovarian stimulation and ovarian response monitoring (cycle cancellation rate and rate of cycles with moderate/severe OHSS), a calculation in populations stratified according to ovarian response  is considered relevant .As a first step, a thorough medical history is collected and a physical examination is performed with the aim of finding definite or potential causal factors for infertility and in turn tailoring management accordingly .In the female patient, the physical examination should include a 2-dimensional transvaginal ultrasound (US). Further assessment can be performed depending on indication and includes 3-dimensional US, hystero-contrast-sonography (HyCoSy) or hysterosalpingography , and/or hysteroscopy or laparoscopy in distinct cases where co-morbidities might be present. Ovulation and its disorders should be examined by assessing the menstrual calendar and by laboratory testing. Ovarian reserve assessment may be considered useful in addition to female age as a predictor of ovarian response and treatment outcome ; severe endometriosis , peripartum haemorrhage, and caesarean section  is correlated with increased procedural risks (such as adverse events during OPU), obstetric complications, and perinatal risks  , gynaecological conditions (e.g. endometriosis), auto-immune disorders , thrombophilia, previous abdominal surgery, single organ or systemic diseases. In women with concomitant disorders, the risks of the ART treatment, the fitness for pregnancy, and required obstetric monitoring should be assessed, where needed in a multidisciplinary context.Transmission risk for infectious diseases.In addition to medical indications, ART treatments are being performed in many countries and centres , even thThe decision on whether to initiate or proceed with ART treatment must first and foremost be a medical one, based on the results of the diagnostic workup and assessment of risks, and in line with patient preferences, national legislation, reimbursement schemes, availability of services and/or socio-cultural aspects. In the absence of PIs for this section, the following recommendations were formulated and achieved agreement :A diagnostic fertility workup should be performed to assess the chance of natural conception, to identify causal factors for infertility, to predict the chance of success of ART treatments, and to detect risk factors for complications during ART treatment or pregnancy.In decision-making with regards to ART, four treatment dimensions should be considered: burden, effectiveness, safety, and costs . Although the statement represents a general principle in medicine, patient-centeredness and preference are highly relevant in ART decision-making.Cancellation of an ART cycle is an unexpected outcome that can occur prior to or after OPU. Cycle cancellation after OPU may result from zero oocytes being identified after oocyte retrieval , failed fertilization, or poor embryo development. Based on published data from 14 studies, the cancellation rate between OPU and ET is estimated to be 14% (weighted average). Cycle cancellation between OPU and ET is not considered a good indicator for clinical work since it can be related to biological factors (such as failure of sperm to fertilize), factors related to the capacity of the oocyte to be activated and fertilized, or factors associated with the clinic’s ET strategy (such as blastocyst transfer regardless of the number of oocytes or embryos available). Cancellation between OPU and ET related to laboratory performance can be monitored by other PIs . Therefore, the clinical PI cycle cancellation should focus on cancellation before OPU only.Cycle cancellation before OPU can be attributed to poor response to ovarian stimulation, premature ovulation, or errors in taking medications, but it may also be influenced by local reimbursement policy or patient preferences. The cycle cancellation rate for 2016 was calculated from the reported number of cycles and the number of aspirations in the European IVF-monitoring Consortium (EIM) data  in which GnRH agonist was used for triggering final oocyte maturation and no luteal phase support was administered , the estimated incidence of severe OHSS was 0.00 (95% CI 0.00–0.00) , although sporadic cases of OHSS under this approach have also been reported .In summary, if GnRH agonists are used for ovarian stimulation, the expected incidence of moderate/severe OHSS is 6.43% and 10.61% in the regular population and in patients with PCOS, respectively. When GnRH antagonists are used, the expected incidences of moderate/severe OHSS are 2.94% and 2.14% in the regular population and in patients with PCOS, respectively. A further reduction of OHSS incidence, by replacing hCG with GnRH agonist and not performing a fresh transfer, has been reported .In the survey, the indicator rate of cycles with moderate/severe OHSS was considered a relevant parameter to assess performance in ovarian stimulation, and its suggested definition of the number of cycles with moderate to severe OHSS over the number of started cycles was acceptable. Competence and benchmark values were defined for all patients, normal and high responders, and for agonist and antagonist protocols. In the survey, respondents commented on the lack of relevance of calculating the parameter for poor responders, which was corrected.Empty follicle syndrome (EFS) is defined as the complete failure to retrieve oocytes during OPU, despite apparently normal development of ovarian follicles and appropriate oestradiol production by granulosa cells  or the proportion of oocytes recovered (in stimulated cycles) is defined as the number of oocytes retrieved during OPU over the number of follicles on the day of trigger . ORR is a measure of the quality of patient monitoring and may be influenced by the timing of the trigger while it is a useful reference indicator for the laboratory  . In the Vienna consensus, no values on follicle size were included in the denominator, and suggestions have been made with a high variation ranging from 10 mm to 15 mm. It is clear that changing the denominator, and more specifically the cut-off follicle size, will largely affect the values for the indicator. A study from Although from the survey ORR is considered a relevant PI to be calculated, the suggested definition, i.e. the number of oocytes retrieved over the number of follicles (>10 mm) on the day of trigger, was judged as debatable. Comments were made on limitations related to the technical feasibility and subjectivity of counting follicles, especially as there is no clear agreement or exact measurement of the follicle size. ORR was not included as a consensus PI.The proportion of MII oocytes at ICSI or rate of mature oocytes was categorized as a reference indicator in the Vienna consensus and defined as the number of mature (MII) oocytes at ICSI over the number of cumulus–oocyte complexes retrieved . The indicator reflects the rate of mature oocytes from the oocytes collected and is a proxy indication of the effectiveness of ovarian stimulation . Another definition was suggested to be the number of mature oocytes over the number of follicles ≥16 mm in stimulated cycles. This would make the indicator a relevant parameter to assess the competence of the clinician for deciding the time of trigger, with the cut-off of ≥16 mm based on a generally used criterion for trigger. It is clear that this indicator can only be calculated if the collected oocytes are fertilized by means of ICSI, which includes the removal of cumulus cells required to assess oocyte maturation.From the survey, there seemed to be a preference for the first definition of proportion of MII oocytes at ICSI, although both definitions were acceptable with over 70% agreement.Complications of OPU include bleeding , infection , severe pain, or injury of pelvic structures. Complications related to sedation or anaesthesia have also been reported but are not considered a relevant PI for clinical practice in ART, as these complications occur with very low prevalence and are considered the responsibility of the anaesthetist. OHSS is excluded from the OPU complication rates, as it correlated to the ovarian stimulation and response, and it is suggested to be monitored as a separate PI.It has been reported that the incidence and severity of complications during OPU are largely dependent on the training of the doctors  cycles, including 983 bleedings, 117 infections, and 361 other complications  complications that require an  medical intervention or hospital admission (apart from OHSS) over the number of OPUs performed. Suggestions in the survey for also including mild bleeding and pain in the complication rate were balanced against comments on the feasibility of collecting the data on minor complications.An ART treatment culminates in the ET procedure. The success of this procedure, and, thus, the success of the ART cycle as a whole, is affected by several factors, including ART regulations and ET policies, patient characteristics, ovarian stimulation, embryo quality , and by ET operator skills and technique. In fact, it has been shown that notwithstanding patient characteristics and embryo quality, the main determinant of a successful ET is the operator  is considered the preferable route towards the key objective of ART, which is the birth of a healthy singleton child. For this reason, ESHRE is drafting an evidence-based guideline specifically on eSET. In the present document, the working group decided, based on the currently available evidence, to include multiple pregnancy rate as an indicator reflecting the transfer policy.Clinical pregnancy is a commonly used criterion for measuring the effectiveness of ART, despite not being the final objective of the procedure. However, CPR is associated with clinician skills and, therefore, relevant to be used as the main PI for ET. There are many different outcomes and definitions used in clinical trials  and this complicates the synthesis of evidence . For CPRCPR is a measure of efficacy, and the parameter does not consider aspects of safety, either ectopic or multiple pregnancies. Consequently, when used independently and without additional data, the indicator may incorrectly favour multiple ET strategies.The survey showed large agreement on considering CPR a relevant parameter to evaluate ET and on the definition, i.e. the number of pregnancies  over the number of ET cycles. Consistently, 69.19% of the respondents preferred per ET as the denominator for CPR in a specific question on the topic, while an additional 4.32% suggested calculating per ET and per OPU. Other suggested denominators included per OPU, per started cycle, per embryo transferred, and per patient. Per ET was considered the most appropriate denominator when using the indicator to assess operator skill. Clinical pregnancy is dependent on other factors, in addition to the operator skill, as discussed above, and using a reference population can be useful to perform meaningful comparisons.Using published data to deduce benchmark values for CPR or other indicators is problematic towing to lack of standardization in the definitions of the different parameters described. The lack of standardization is linked to the numerator  and the denominator . Furthermore, the recent shift in clinical management towards freeze-all cycles (up to 6.6% in the latest EIM report), either to prevent OHSS or as a strategy to improve clinical outcomes, resulted in more frozen and less fresh ETs  should be defined at a local level , to minimize the effects of institutional or legislative differences on clinical outcomes.In addition to differences between countries, variation in reported CPR values can be attributed to the origin of the oocytes and the type of cycle (fresh or frozen). Therefore, it is recommended to assess CPR in the reference population (fresh ET with own oocytes), but also in frozen ET (FET) cycles with own oocytes, fresh ET after oocyte donation, and FET after oocyte donation. For those clinics with enough volume of treatments performed, it is recommended to split all four mentioned categories for cleavage and blastocyst ETs, as results can be significantly different between these groups.An ectopic pregnancy  is a prThe survey showed that less than half of the respondents considered ectopic pregnancy rate as a valuable PI for ET and one-third of them considered ectopic pregnancies not associated with ET technique or operator skills. The ectopic pregnancy rate was, therefore, not considered a relevant PI. The definition of ectopic pregnancy rate was accepted, and it should be calculated as the number of pregnancies outside the uterine cavity over the number of pregnancies.A singleton live full-term healthy baby is the most relevant standard of success in ART  per aspiration. The cumDR was 29.6% for the countries that were able to provide data  subspecialty curriculum does not contain any specific minimum number of procedures to be performed , nor doeFor safety reasons, and wherever feasible, a simulator could be the initial part of structured training for novices who want to perform these procedures, enabling them to acquire basic skills and to reach a predefined level of performance in a safe and controlled environment, before applying the procedure to patients  certification scheme. The number of procedures to be completed for training is listed in After the training phase, maintaining competence or skills is essential. Criteria for assessing proficiency/competency on the technical aspects of OPU were recently included in a paper on good practice recommendations for OPU  and actual performance (the result or measure). Once the gap between a benchmark and the actual performance is identified, it is vital to put a specific plan in place to reduce the gap. It makes no sense to establish a benchmark and measure performance against it if no action is taken to act upon the information gathered .The survey results showed overall an acceptance of using PIs to internally evaluate the maintenance of skills. The proposed values, as displayed in The current paper recommends six PIs to be used for monitoring clinical work in ART in ovarian stimulation, monitoring of ovarian stimulation, trigger and OPU, and ET and pregnancy. Furthermore, training and competence in relation to PIs are discussed. No PIs were defined for the first step in a standard ART process; diagnosis of infertility and indications for ART treatment. PIs for the laboratory procedures have been previously defined .In addition to the working group members and published data, the opinion of the stakeholders involved in the survey was vital in defining relevant and acceptable PIs for clinical work. Overall, the indicators and their definition as drafted by the working group were considered acceptable by stakeholders that participated in the online survey. The PIs defined in the current paper are either associated only with the performance of the clinician and clinic, or they are associated with outcomes that depend both on the clinic and the laboratory, and both share responsibility for results.The PIs complement the earlier defined indicators for the ART laboratory . Together, both sets of indicators aim to enhance the overall quality of the ART practice, and should be part of the total quality management of ART centres. They are important and can be used for education during subspecialty training. Future developments and shifts in ART clinical practice, such as eSET and the freeze-all technique, may warrant an update of the defined PIs and the suggested competence and benchmark values. The latter values are now based on the literature and consensus. However, ideally they should be defined on a local level; therefore, national and by extension international registries should be encouraged to standardise data collection, in line with defined ART clinic PIs and laboratory KPIs to enable the collected data to be used for the derivation of PI standard values. These initiatives will eventually lead to more functional competence and benchmark levels.The data underlying this article are available in the article and in its online supplementary material.Supplementary_figure_S1Click here for additional data file.Supplementary_Table_SIClick here for additional data file.


A strong health information system (HIS) is one of the essential building blocks for a resilient health system. The Ministry of Health (MOH) of Ethiopia is working on different initiatives to strengthen the national HIS. Among these is the Capacity-Building and Mentorship Partnership (CBMP) Programme in collaboration with public universities in Ethiopia since November 2017. This study aims to evaluate the outcomes and share experiences of the country in working with universities to strengthen the national HIS.The study employed a mixed-methods approach that included 247 health organizations  of CBMP-implementing woredas (districts) and 23 key informant interviews. The programme focused on capacity-building and mentoring facilities and woreda health offices. The status of HIS was measured using a connected woreda checklist before and after the intervention. The checklist consists of items related to HIS infrastructure, data quality and administrative use. The organizations were classified as emerging, candidate or model based on the score. The findings were triangulated with qualitative data collected through key informant interviews.The results showed that the overall score of the HIS implementation was 46.3 before and 74.2 after implementation of the programme. The proportion of model organizations increased from 1.2% before to 31.8% after the programme implementation. The health system–university partnership has provided an opportunity for higher education institutions to understand the health system and tune their curricula to address real-world challenges. The partnership brought opportunities to conduct and produce local- and national-level evidence to improve the HIS. Weak ownership, poor responsiveness and poor perceptions of the programme were mentioned as major challenges in programme implementation.The overall HIS has shown substantial progress in CBMP implementation woredas. A number of facilities became models in a short period of time after the implementation of the programme. The health system–university partnership was found to be a promising approach to improve the national HIS and to share the on-the-ground experiences with the university academicians. However, weak ownership and poor responsiveness to feedback were the major challenges identified as needing more attention in future programme implementation.The online version contains supplementary material available at 10.1186/s12961-021-00787-x. Health systems consist of all the people and actions whose primary purpose is to improve health . The heaA weak HIS is characterized by poor data quality and low information use for evidence-based planning and decision-making. Incorrectly and poorly generated healthcare data create multifaceted problems that can seriously increase the burden of diseases on individuals and the community in general . Making Ethiopia has been implementing different initiatives to strengthen the national HIS, including the revision of the national health management information system (HMIS) in 2017, with the development of the information revolution roadmap that helped to guide the implementation of the information revolution. This initiative aimed to maximize the availability, accessibility, quality and use of health information for decision-making processes through the appropriate use of information communication technologies to positively impact the access, quality and equity of healthcare delivery at all levels. The information revolution roadmap also conceptualized the development of a connected woredas strategy, with data-use innovations to achieve the highest standards in data quality and use. This pathway begins with a grading process whereby health facilities are evaluated and scored against a common set of criteria related to HIS infrastructure and capacity, data quality, and administrative and clinical data use. Facilities and woredas that meet the highest standards and are able to access and share data with higher levels through offline mechanisms are recognized as “model facilities” and “model woredas”. Model facilities and woredas that take this one step further by enabling online data access and transmission are recognized as “connected facilities” and “connected woredas”.Following the development of the information revolution roadmap, efforts have been made to improve data quality and use of information at lower levels of the health system. These efforts have included capacity-building training on data recording and reporting, data analysis, quality assurance, routine data use for decision-making, and troubleshooting. Moreover, health information technology initiatives such as the District Health Information System 2 (DHIS2), electronic medical records, Electronic Community Health Information System (eCHIS) and human resource information systems have been implemented to facilitate information systems and improve the delivery of health services .Though Ethiopia has made remarkable progress in implementing HIS activities in the past few years, a thread of evidence shows that the level of data quality and use of information for decision-making is still not satisfactory , 12. PooEvidence has shown that engagement of higher education institutions could have a substantial impact in terms of capacity-building, technology transfer, training, collaborative research and internship –20. The Despite encouraging efforts, the engagement of higher education institutions in strengthening the HIS was limited. Hence, the MOH, in collaboration with higher education institutions, introduced a new model known as the Capacity-Building and Mentorship Partnership (CBMP) Programme. This programme envisioned establishing ownership and capacity among higher education institutions to collaborate with local health administrations on investigating gaps and designing and implementing innovative interventions to strengthen the HIS. Therefore, this paper aimed to share the vision, approach and experiences of the CBMP initiative with the global health community and assess its outcomes and the underlying implementation challenges in order to strengthen the programme in the next implementation phase.This study employed a mixed-method approach. For qualitative analysis, a phenomenological approach was used to explore the programme specifically from its inception to its current implementation, as the experts themselves were engaged in the design and implementation of the programme and documented the experiences, implementation challenges, success stories and future prospects, which are used as a base for this study. Additionally, key informant interviews were conducted with key decision-makers and implementers who were involved throughout the programme. This study considered a review of routine reports from implementing universities and national documents in line with the implementation of the CBMP programme. Quantitatively, the status of HIS was evaluated before and after the implementation of the CBMP programme.The CBMP programme was introduced by the MOH in collaboration with a group of universities. The CBMP is an initiative envisioned to establish a partnership between health systems and universities to strengthen the national HIS through different interventions. Six universities, namely Addis Ababa, Haromaya, Hawasa, Jimma, Mekele and the University of Gondar, were selected to implement this partnership programme. The programme included all Ethiopian regional states and city administrations. The partnership was aligned as Addis Ababa University with Addis Ababa City Administration; Haromaya University with the Somali and Harari regions and Dire Dawa city administration; Hawasa University with Sidama and the Southern Nations, Nationalities and Peoples’ (SNNP) region; Jimma University with the Oromia and Gambela regions; Mekele University with the Tigray and Afar regions; and the University of Gondar with the Amhara and Benishangul-Gumuz regions.The selected universities were to work closely with the consortium universities and the nearby health science colleges in capacity-building and mentorship activities. The intervention targeted a group of officials and data users from the woreda health office, health centres and healthcare providers. The intervention began in November 2017, immediately after baseline assessment. The end-line assessment was conducted after 3 years of implementation, and the programme is still active. With simultaneous effort on data quality, data analysis and data use, the staff at each level of the health system were poised to take advantage of increased available data. Through targeted training paired with mentoring and supportive supervision and training for supervisors, capacity-building efforts aimed to contribute to sustainable improvements in health system capacity..Components of the activity were engagement with stakeholders, training, mentoring and planning for sustainability Table .Table 1IThe primary outcome of the study was the HIS implementation status, which was measured using an information revolution-connected woreda checklist and scoring system developed by the MOH. This approach analysed health facilities and woreda health offices along three dimensions: HIS capacity and infrastructure, data quality and information use. The checklist was adapted from the PRISM [Performance of Routine Information System Management] assessment tool. To evaluate the HIS performance of the facilities and woreda health offices, a connected woreda checklist and scoring system was used  participated in the end-line assessment The overall score of the HIS implementation was 46.3 before implementation, 58 at the midline and 74.2 after implementation of the programme. The HIS infrastructure, data quality and administrative data use practice increased along the baseline, midline and end-line assessments. Data quality in particular showed a significant improvement over the course of the implementation For the data quality assessment, nationwide, a total of 242 woreda health offices, health centres and hospitals were included in the baseline assessment. In the baseline assessment, the level of data quality ranged from 4.9 in the Somali region to 17.6 in the Addis Ababa city administration. All regions except Sidama, Tigray and Addis Ababa scored below half. The overall data quality status score was 12.9 in the baseline assessment. All regions showed improvement, and the end-line assessment indicated that a massive improvement was observed in the majority of the regions. Four regions, namely Addis Ababa, Amhara, Sidama and Tigray, achieved the national target for data quality status, while only two regions, Harari and Somali, scored below 80% in the end-line assessment. The overall data quality status increased from 12.9 to 26.3  of the health organizations in the SNNP and two 9.5%) in the Sidama regions already had model organizations before CBMP implementation. Some regions, including Afar, Dire Dawa, Harari and Somali, had not yet achieved model organizations after the CBMP intervention. After implementation of the CBMP, the Tigray region achieved the highest number, with 19 (67.9%) model organizations, followed by 18 (50%) in Addis Ababa, 10 (45.5%) in Sidama and 15 (44.1%) in the Amhara region "The programme gave us a chance to understand our health system to adjust our curricula to the real context of our health system". Respondents indicated that the introduction of the CBMP programme has provided an opportunity to understand the real context of the health system. In the key informant interviews, participants reported that the programme offered mutual benefits for higher education institutions and the health industry. The higher education institutions have been given the opportunity to understand the real context of the health system in order to tune their curricula to incorporate contextualized content.The health system–university linkage and partnership has offered an opportunity to develop a national human resources roadmap for HIS in collaboration with the MOH, higher education institutions, regional health bureaus and the Data Use Partnership (DUP). The roadmap was developed for a 10-year period, from 2021 to 2030.The partnership has allowed different stakeholders to be engaged in assessing human resource demand and existing gaps in the competency of HIS manpower. The curriculum revision and harmonization undertaken involved 10 universities, one health science college, the MOH, the Ministry of Education and regional health bureaus in collaboration with the DUP. The revised curriculum is nationally harmonized and distributed to the universities by the Ministry of Education. Following the initiative, the number of universities that launched health informatics programmes expanded from 3 to 10 during the programme period.In collaboration with the MOH, local universities have developed a nationally harmonized bachelor’s-level health informatics curriculum for a generic and advanced standing programme. The curriculum was approved by the Ministry of Education and was distributed to all universities throughout Ethiopia. To date, about eight universities have begun offering pre-service health informatics programmes using the newly harmonized curriculum. Other universities are expected to launch programmes in the next academic year. The advanced standing curriculum, which is also approved, allows health information technicians to upgrade their qualification to a bachelor’s level.A joint team composed of the MOH, University of Gondar and other universities developed eight standardized pre-service training modules for major health informatics courses. This effort ensures that graduates are equipped with the knowledge and skills corresponding to the demand in the health sector. It also responds to the gap in trained HIS professionals in Ethiopia. Work will continue to further improve the pre-service teaching quality within partner universities.One of the aims of the CBMP is capacity-building of HIS cadres. The CBMP programme has given funding support focusing on research and learning opportunities for doctoral and master’s-level students.Before the implementation of the CBMP, four universities, namely Addis Ababa University, Mekelle University, Jimma University and University of Gondar, were providing HIS-related master's programmes in Ethiopia, each focusing its coursework on different aspects of HIS. Also, the University of Gondar, Debre Markos University, Metu University and Mekelle University were providing HIS-related bachelor’s degree courses. Before 2017, the four universities which were providing HIS-related courses in a bachelor’s degree had different curriculums. After the implementation of the CBMP, 10 universities, in collaboration with the MOH and Ministry of Education, undertook curriculum harmonization for both bachelor’s degree and advanced health information technician training programmes.Currently, a total of 10 public health universities and three public health science colleges have a bachelor’s degree programme in health informatics, and a total of 814 students are enrolled in the bachelor’s degree programme. More than 150 bachelor’s degree health informatics professionals have graduated.Also, before the implementation of the CBMP, scholarships for master's and doctoral degrees typically covered the cost of coursework, but they were not able to support additional academic research activities. The programme provided support for 27 master’s and nine PhD students to conduct operational research on developing and testing interventions to expand data use for evidence-based decision-making at health facilities and woreda health offices for better service delivery and management.The capacity-building activities are designed to link the training programme to information revolution outputs, targets and sustainability mechanisms. It is believed that building capacity in the areas of monitoring and evaluation, research, HIS and technology, and other related disciplines will contribute to achieving the goals of the information revolution and will ensure complete local ownership and sustainability of its programmes.The strategy that the universities follow is a “learning by doing” approach, which will ensure the availability of the staff services while they are in education. For example, the MOH and regional health bureau staff admitted to PhD and master of science (MSc) studies through this programme are expected to implement activities prioritized in the connected woreda and CBMP programmes in one of the regions. When implementing the programmes, they are expected to attend seminars in the universities and publish their implementation experience in reputable international journals. According to the human resource regulation, the MOH and regional health bureau staff who will be given this opportunity will have to sign a contract with the MOH regarding their service commitment after graduation.In general, the implementation of the CBMP programme has increased the number of universities and health science colleges with new HIS training programmes, which ultimately increases the number of students enrolled, leading to more competent graduates to fill the human resource gap.The study identified major challenges encountered during the implementation of the CBMP programme. Weak ownership at the lower level was reported as the main challenge by the majority of the respondents. Poor facility responsiveness towards the given feedback, perceiving CBMP as a research agenda rather than a partnership, and COVID-19-related difficulties were mentioned as challenges Fig. .Fig. 2Li"We have faced many challenges in implementing HIS activities. Low attention to health data, health information technicians’ negligence due to the job evaluation and grading, weak ownership for the CBMP activities, considering the CBMP activities as a research agenda and tasks of the universities, and weak response to the feedback provided to facilities were some of the challenges for implementing the CBMP". The qualitative findings also identified human resource-related challenges in HIS that included paying less attention to health data, negligence, poor ownership, and weak feedback and response as major challenges. One of the participants noted:This study has shown that the overall status of HIS implementation improved significantly after implementation of the CBMP programme when compared with the status before implementation, as envisioned initially. From the global experience, university–industry linkage brought about substantial results in technology transfer, training, collaborative research and capacity-building –19. SimiThe study showed improvements in HIS-related infrastructure, although it is still inadequate as compared to the expectation in the national information revolution guideline . AdequatThe results showed that the overall data quality status was improved at all levels of the health system. It indicated that the improvement in data quality could be due to the interventions that were provided, as most of the gaps identified at baseline were addressed at the health centre, hospital and woreda health office levels. Despite the improvement, data quality was still a major problem. Some of the major gaps, like poor documentation, lack of tally sheets, lack of registers, lack of HMIS guidelines and lack of skill, were identified during the baseline assessment. This is supported by the evidence from the Ethiopian Public Health Institute data quality review report and WHO , 27–29. The study found improvement in administrative data use from baseline to midline and finally to end-line assessment, which was attributable to the CBMP programme, though the overall data use level did not meet the expectation envisioned in the information revolution . The proThe study also revealed that there has been inconsistent performance across regions and health facilities over time. The reasons for the inconsistency may be related to the differences in the frequency and dose in implementing the intervention package. National documents have also reported that achieving cultural transformation in data use is the most challenging part of the information revolution agenda, as it requires strong coordination among different stakeholders at all levels , 34. UtiIn general, the connected woreda status showed a steady improvement from baseline to midline and end-line assessment attributable to the CBMP intervention. However, no single woreda achieved connected status. This implies that online data access and transmission remain challenging, and further efforts are needed to connect all woreda facilities digitally. The connected woreda plan considers the establishment of a widespread culture of using health information for decisions at all levels. However, the process requires reliable electricity and connectivity, digital literacy, standard processes and training for health workers to fully implement digital tools such as electronic patient registry, DHIS2 and eCHIS technologies .The CBMP partnership has also had a substantial impact on pre-service education, as the number of universities and colleges with health informatics training increased from 4 to 13 within 3 years of the partnership, which resulted in the addition of hundreds of professionals that can play a key role in a sustainable and strong national HIS in Ethiopia.As a limitation, this study focused on administrative data use. We did not address clinical data use practices, so the results may not show the full picture of data use at all levels of the health system.The overall HIS has shown substantial progress in capacity-building and mentorship programme implementation in woredas. A number of facilities became models in a short period of time after the implementation of the programme. The health system–university partnership was found to be a promising approach to improve the national HIS and to share the on-the-ground experiences with the university academicians. However, weak ownership and poor responsiveness to feedback were identified as major challenges that need more attention in future programme implementation.Additional file 1. Connected Woreda checklist and point allocations.