Humanitarian radio has been used in humanitarian aid efforts and after natural disasters over the last 15 years. However, the effects have barely been evaluated, and there are few scientific reports on the impact of radio as a disaster health response intervention. Therefore, this study aimed to provide an overview of the use and impact of humanitarian radio in natural disasters from a health perspective. A literature review of 13 scientific papers and grey literature resources was conducted. The results show that humanitarian radio could be used to promote both physical and psychosocial wellbeing by providing health-related information, advice and psychosocial support in natural disasters. Community resilience can be enhanced by the promotion of community engagement and can strengthen self-efficacy and community efficacy. Radio also has the potential to cost-effectively reach a large number of affected people in areas with severely damaged infrastructure. Radio could, therefore, contribute to health recovery and wellbeing from both individual and community perspectives. As such, health professionals; crises communication professionals, including radio journalists; and disaster-managing stakeholders should be prepared and trained to use humanitarian radio as an integrated part of the disaster health response in natural disasters.  In 2017, 335 natural disasters affected over 95.6 million people. These disasters mostly occurred in Asia in the form of storms or floods, but they also affected other parts of the world as earthquakes, storms and wildfires, which collectively caused deaths, injuries and human desolation . Damage Today, crisis communication is considered to be an integrated part of disaster health response . Access A literature review, which was inspired by the integrative review method  and baseAs a first step, a structured search of the Medline database and Web of Science was made, focusing on original research papers that aimed to describe the use or impact of radio from a health perspective during or in the aftermath (the first three months) of a natural disaster. Papers published in English between 1998 to 2018 were included in the present review. Papers describing the use or impact of Very High Frequency (VHF) radio communication and papers focused on disaster preparedness or prevention, including early warning systems, were excluded. Papers describing capacity-building initiatives in complex humanitarian emergencies or conflicts were also excluded. Keywords were combined in order to identify the available sources that met the inclusion criteria for this review See . Next, aThe search for studies in Medline and Web of Science yielded 172 papers. The grey literature search yielded 11 reports or papers. A reading of the titles reduced the number of possibly relevant papers to 30. The most common reasons for being excluded at this phase was reporting via VHF radios or amateur radio instead of radio broadcasting or referring to ‘radio’ as in radio nuclear studies. Also, papers describing the use of radio in humanitarian contexts other than natural disasters were common. In total, 17 papers and reports remained after the above exclusions; each paper was read in its entirety, and 13 papers or reports were selected for the analysis stage. The search strategy and results are presented in An original integrative review used quality scoring in order to conduct a quality appraisal . Due to After identifying the papers and reports that met the inclusion criteria, all documents were read several times in their entireties, and the data relevant to the purpose of this review were identified, extracted and reduced. Among the included research studies, only data from the results and conclusion sections were used in the present analysis. For the grey literature sources, all relevant data considered as a result or a conclusion was included in the analysis. Then, the extracted data were sorted into groups and clusters related to the aim of the study , and the reviewers compared and contrasted the information before summarizing the analysis into an integrated synthesis . The research studies and reports used in this study were drawn from 13 papers, covering the use or impact of humanitarian radio in several large-scale natural disasters see . The results will be discussed below in sections covering the practical use of radio, physical health promotion, psychosocial health promotion and the promotion of community resilience. Radio has been used in the immediate response phase of many kinds of natural disasters and with a range of stakeholders, dispatching messages, information or advice to affected communities ,20,21,22The providers of humanitarian radio were most often NGOs (based on request from the affected country) in all included studies except three, where ordinary public service radio was used ,27,28. Sn = 296) stated that they listened to disaster radio at some time during the first months after the disaster event. The reasons for listening to the radio were to obtain practical advice , for company , to obtain information on available services , to obtain information on the wellbeing of relatives and friends , to take a break  and to obtain general information  [To illustrate the use of radio as a natural disaster response intervention, disaster response operations by the NGO, First Response Radio (FRR) , are pre n = 65) . Radio has been used in many natural disasters to broadcast health-related information. By providing practical advice and stimulating self-care for both physical and mental health problems, wellbeing among survivors can be supported. The health-related information broadcasts after the Haiyan typhoon  showed tA multiple linear regression analysis on variables influencing perceived health among survivors 30 months after the Haiyan typhoon indicated that listening to the provided humanitarian radio after the typhoon positively influenced the general health of the survivors [Radio communications have been used in many natural disasters to transmit information, clarify situations and inform people about available support ,20,22,24In countries where radio is a trusted source of information or entertainment, humanitarian radio has been shown to be of importance as a source of both information and support after a natural disaster, as well as in situations where all other means of communication were lost ,21. RadiPromoting the health and wellbeing of a large number of people in distress in environments without functional infrastructure, such as electricity, roads and ordinary means of communication, is a great challenge. Such circumstances are the reality in most natural disasters. This review has demonstrated how disaster radio may be used to promote health during and in the aftermath of such a situation. By actively advising people on where to turn for medical services and promoting self-care advice, available medical resources may be more effectively used. Furthermore, proactively raising public awareness on how to reduce health problems might ease the burden on medical services. Radio campaigns have been proven to create substantial changes in health-seeking behaviors and reduce mortality in non-emergency contexts in Burkina Faso . There iFrom a historical perspective, awareness of the mental health impacts of disasters and the demand for psychosocial support after such events have increased . InformaWhen promoting hope, large-scale, community-based interventions may be more effective than individual interventions . This reThis review shows that radio has been used and has played an important role in natural disasters, not only in low- or middle-income countries and regions, such as the Philippines and Western Africa, but also in high-income countries, such as the United Kingdom and the United States. Today, the use of mobile applications and social media for information purposes and social contacts has increased and may also be tools that can be used in disasters. However, it should be remembered that these tools require a functioning internet service, and natural disasters often cause severe interruptions in mobile and internet coverage, sometimes for a long time. The studies included in this review all emphasized that radio was available and operational in situations with severely injured infrastructure, when the internet and mobile phones were not functioning. Also, studies suggest that people in emergencies and disasters tend to lean on information sources to which they are accustomed , and a lIt should be noted that there are many disasters where radio has been used as a response intervention, but there are only a few structured evaluations or research studies available. Quick disaster response requires preparation. For humanitarian radio, preparations should not only include training teams and logistical preparations but also legal procedures, such as those needed to gain the necessary permissions to broadcast. Radio broadcasters also need to have an understanding of the difference between ordinary media reporting and humanitarian radio as a disaster response intervention . This inThe limited number of scientific papers describing the use or impact of humanitarian radio from a health perspective made it impossible conduct a traditional systematic literature review based only on scientific papers. Mixing scientific published papers and grey literature added much value to this review. Grey literature can provide data not found elsewhere and can make important contributions to the synthesis of knowledge , especiaHumanitarian radio has been used to provide health-related information, advice and psychosocial support in natural disasters, thereby contributing to the health and wellbeing of the affected populations from physical, psychosocial and community perspectives. Radio has the potential to reach large populations in severely damaged areas and could, therefore, be a useful and powerful health response intervention after natural disasters. As such, health professionals; crises communication experts, e.g., radio journalists; and disaster managers should be prepared to use humanitarian radio as part of their response operations in natural disasters.


By using a comprehensive dataset of US and European universities, we demonstrate super-linear scaling between university revenues and their volume of publications and  citations. We show that this relationship holds both in the US and in Europe. In terms of resources, our data show that three characteristics differentiate the US system: (1) a significantly higher level of resources for the entire system, (2) a clearer distinction between education-oriented institutions and doctoral universities and (3) a higher concentration of resources among doctoral universities. Accordingly, a group of US universities receive a much larger amount of resources and have a far higher number of publications and citations when compared to their European counterparts. These results demonstrate empirically that international rankings are by and large richness measures and, therefore, can be interpreted only by introducing a measure of resources. Implications for public policies and institutional evaluation are finally discussed. During the last thirty or so years, public funding of research institutions and particularly of universities  has signSuch new conception of science is grounded on the belief that there are ‘universal’ measures of scientific ‘excellence’ that academic managers, policymakers and stakeholders can reliably use to assess the output of individual researchers and universities. A continuously evolving family of bibliometric indicators have been generated and used/misused both at the micro management level, to take decisions on salaries and career, and at the macro level, directly or indirectly, in performance-based university funding systems . In paraAcross countries, one model of research intensive institution, inspired by the US research university , has becHowever, such an approach under evaluates the institutional and historical diversity of local higher education institutions (HEIs) with their heterogeneity in missions and responses to specific local needs  and geneAt a more technical level, the bibliometric literature has demonstrated the potential flaws of such indicators and, specifically, of using them without benchmarking against the level of available resources . PrefereTherefore, the new governance of science suffers of two main shortcomings. First, while quantitative research evaluation methods are robust in identifying low performance (in the selected output measurement), they are less reliable when used to discriminate top performance. Moreover, research evaluation has to be carried out in comparative way, comparing apples with apples, and thus requires that inputs are taken into account and alternative outputs are compared. Specifically, and at the core of the contribution of this paper, funding needs to be considered if we want to properly understand productivity of individuals and organizations. Yet, beyond the obvious assumption that more resources translate into more output, we know little about the relationship at the institutional level between the amount of available resources on the one hand and scientific output and visibility on the other hand  23]. Al. Al23]. The focus of this paper is to examine the relationship between resources and standard bibliometric indicators that are widely used to compare universities for their ‘excellence’ . We aim to understand whether such indicators depict wealth rather than anything else.Our contribution is mainly empirical. We first test the association between university revenues and international visibility. Using a dataset providing input and output data for nearly all doctoral universities in the US and in Europe, we demonstrate that the number of publications and citations at the university level scales super-linearly in respect to revenues, and that these relationships are similar in the US and in Europe. This implies that the richest universities will systematically show up at the top of bibliometric indicators and of international rankings. Second, we show that the main path associating university revenues with bibliometric output is through additional revenues per staff, suggesting that the wealthiest universities compete for talented researchers by offering them more attractive funding packages .Third, we investigate to which extent such differences between US and Europe in international visibility are associated with different levels and distribution of resources by HEI. We show that the US system comprises a small number of universities with far larger revenues than their European counterparts. This suggests that the ‘transatlantic gap’ in research ‘excellence’  is by anFinally, from these results we derive implications for evaluation practices, academic management and public policies. Our results support claims from the evaluation community, such as expressed in the DORA-San Francisco Declaration  and in Our empirical strategy is described in First, we have created a dataset including the full population of HEIs delivering at least a bachelor degree in the two systems (excluding associate colleges in the US), i.e. 3,287 HEIs in the US and 2,243 HEIs in Europe. Data have been derived from the Integrated Postsecondary Education Data System for the US  and thedoctoral universities, defined as the HEIs awarding more than 20 PhD degrees in the reference year 2013 and excluding universities focused on a single topic such as medical schools . This sThird, we use the full dataset to compare the volume and distribution of revenues within the two systems and to examine to which extent this accounts for differences in resourcing of doctoral universities between US and Europe.The first group of variables is used to classify HEIs based on the Carnegie classification criteria and to identify doctoral universities in our population  out of the top-100 universities in the 2017 ARWU-Shanghai ranking.Coverage of revenues data is lower due to missing data in Europe; however, data availability is much higher for doctoral universities, which are the main focus of the analysis. For most European universities, the resources for the general functioning of the university cannot be broken down between basic state instalment and private donations; since available data show that the share of private funds is low in Europe, we use simple imputation based on the average of available cases (in Europe) to complete the dataset.Total Current Revenues are the amount of money received by the HEI during the reference period for its operations. It excludes revenues intended for long-term investment, such as state subsidies for buildings and large facilities. Excluding capital-related revenues is important for comparability purposes due to different treatments of capital costs and revenues depending on the university accounting system . Investment income (for example revenues generated from assets and endowments) is included. Revenues from ancillary enterprises are also excluded. This is important since, for US universities, sales and services from auxiliary enterprises and intercollegiate athletics might constitute a large share of total revenues. Finally, revenues of university hospitals are excluded, but educational and research costs of the medical faculties are included.total current revenues are divided into four streams:For a more fine-grained analysis, Basic state instalment, i.e. the funds provided by the state for the general functioning of the HEI.Private donations and pay-outs from the endowments, managed at the university level.Third-party funds mostly for research, e.g. research grants from public funding agencies and contracts from companies.Funding from student fees paid by students and families.To this aim, we devised a mapping scheme based on the revenues subcategories provided by IPEDS and ETER , this somewhat reduces funding level differences between the US and Europe.Academic Staff in Full Time Equivalents are based on working contracts; in ETER, all personnel involved in teaching and research is included, while for IPEDS, we use the number of instructional, research and public service staff as the nearest equivalent. In both cases, it excludes management, technical and support staff, as well as healthcare staff in the hospitals annexed to universities. Coverage of PhD students and postgraduate staff may not be fully complete. However, when using FTEs, this is less of a concern if part-time staff is not fully covered.Publications and Normalized Citations are derived from the Web of Science (WoS) copy maintained by the CWTS, University of Leiden, which is also the source of the Leiden ranking [ ranking . The lisThe Leiden ranking includes a substantial effort to delineate the perimeter of universities and to handle special cases, e.g. assigning publications correctly to members of confederate universities (e.g. University of London). University publications also include university hospitals, which are tightly integrated with the university, as revealed by publications with shared affiliations. This avoids comparability issues between situations where the hospital is part of the university (as in many US universities) or it is legally independent (as in most European countries).Bibliometric data were retrieved for 903 out of 930 doctoral universities. Most missing cases were special institutions such as distance graduate schools in the US.The indicators follow the definitions adopted for the Leiden Ranking :The count of publications (P) includes only the core publications in the WoS, i.e. those published in journals of international scope and highly referenced in the WoS. This is consistent with our focus on international research ‘excellence’. Fractional counting of publications was adopted. The reference period is 2014–2017.The total normalized citation score (TNCS) is the total number of citations of the publications normalized for field and publication year. The citation window is of variable length depending on the year of publication since citations are included only up to week 13.2019. Citations are also fractionalized.We carried out a robustness check of our main results using the other main international bibliometric database, i.e. Scopus. The list of doctoral universities was matched with the SCIMAGO Institutional Ranking (SIR). We were able to identify 867 out of 930 doctoral universities in the SIR. Most non-matched cases were either small institutions or multi-campus HEIs in the US, for which SIR only provided aggregated data.A well-known issue for bibliometric data are differences in publication behavior and database coverage across scientific fields; since universities have different subject compositions, this might weaken the observed relationship, as extensively analyzed in . We haveFirst, to test the association between revenues and bibliometric outputs, we regress the two bibliometric indicators, i.e. the number of publications and of field-normalized citations, against total university revenues (in euros PPPs). Since data refer to total revenues, including educational expenditures, we also control for the volume of education and for the share of students in social sciences and humanities, while we introduce a dummy for US vs. Europe in order to control for data comparability issues.The standard approach for fitting power-law relationships is to use an OLS regression on the log-transformed variables and to provide an analysis of residuals to check whether there are potential robustness issues . Two majAs a first step, we use OLS to detect and to exclude influential observations and outliers. Then we address heteroscedasticity through Feasible Generalized Least Squares (FGLS ). FGLS iFirst, the following OLS regression is estimated:Then, regressing the squared residuals of Model diagnostics and robustness tests). Complementarily, we perform quantile regressions [Weights are computed from the predictions of ressions , which aressions .Second, to test association paths between revenues, staff and bibliometric output, we run a mediation model that allows estimating the paths that associate revenues and publications  directly or through the number of academic staff . MediatiThird, to compare the level and the composition of revenues between US and Europe, we perform descriptive analyses using the full sample of 5,530 HEIs. We compare the aggregated level of revenues in the two systems, as well as the distribution between HEIs. Then, we analyse the distribution of total current revenues between subcategories. Finally, we compare the distribution of revenues, staff and bibliometric outputs among the sample of 930 doctoral universities to ascertain whether we observe differences in concentration of revenues and outputs between US and Europe.As reported in These findings have important implications for the use of bibliometric indicators for evaluation purposes. On the one hand, the coupling between revenues and bibliometric indicators is really tight, as shown by the coefficient of determination. On the other hand, super-linear scaling implies that bibliometric indicators increase more rapidly than revenues and so-called scale-free indicators, such as MNCS, become size-dependent . Such a We notice that the regression sample includes a large share of the population of internationally ‘excellent’ universities (76 out of the top-100 in the ARWU-Shanghai ranking) and almost all doctoral universities in the two regions. This emphasizes the significance of our results.Model diagnostics and robustness tests for full results), confirming previous results that, indicators from WoS and Scopus tend to be highly correlated at aggregation levels such as universities or countries [We also run the same regression using data from the other large international bibliometric database, i.e. SCIMAGO, and more specifically the number of documents for the period 2015–2017 and the field-normalized citation impact. Results are very similar to the WoS .We also perform a number of further robustness tests including separate regressions for the two regions, which produce similar results to the main regression, and an analysis of residuals and outliers; the latter shows that most deviant cases are in the left-tail of the small universities, while the fit is better for the top-ranked international universities , as expected since the main resource for universities is academic staff and, therefore, additional funds will be largely invested in hiring people. However, universities could also provide more resources per unit of staff, for example in the form of higher salaries or of starting packages for newly hired professors .As reported in As expected, student enrolments have a positive association with the number of staff, implying that, with increasing number of students, revenues are used to a larger extent to hire staff, generating some increase in output. However, the aggregate coefficient is negative (0.235*0.605–0.311) = -0.170, i.e. universities with more students have less publications and citations with the same resources.These results show that the stronger influence of resources on scientific production takes place through providing more resources per unit of staff, as this allows the richer universities to compete for the most talented researchers. However, increasing student enrolments push universities to expand their staff to manage educational activities and, in turn, this lowers scientific output. Therefore, for international ‘excellence’, not only the amount of resources matters, but also the extent to which revenues are decoupled from the number of students.While scaling relationships are similar, the two systems are characterized by large differences in the distribution and composition of HEI revenues.All top-25 HEIs by revenues in the dataset are in the US, with the list being topped by Harvard and Stanford, the first European universities are Cambridge (place 26) and Oxford (place 41), i.e. the highest ranked European HEIs in the 2017 ARWU-Shanghai ranking. Interestingly, HEIs in the same revenue class have similar levels of funding per student in the two systems, showing that the main difference lies in the distribution of revenues and, particularly, in the presence in the US of a group of about two dozens of universities with extremely high revenues.When combined with super-linear scaling of publications and citations over revenues, the distribution of revenues translates into a dominance of the US universities in the ranking by number of publications and citations , respective in the international rankings, which are closely correlated to bibliometric indicators.A deeper analysis reveals that a combination of institutional factors accounts for the observed difference in the distribution of revenues.First, the US higher education system is endowed with more resources. When comparing only HEIs that have financial data, the numbers of staff, students, publications and citations are similar in both systems, while the total amount of revenues is 282 billion euros PPS in the US and 133 billion euros in Europe , showingThe difference in resources in our data is compatible with international statistics where tertiary education spending was 2.7% of GDP in 2014 in the US and ranged between 1% and 2% in European countries . This difference is essentially due to revenues from the private sector and from students  that account for two-thirds of tertiary education spending in the US, but to less than 40% in most European countries (with the exception of the UK).Second, we observe a difference in the extent of institutional differentiation between the two systems, as revealed by applying the US Carnegie classification to the dataset  universities have a composite funding structure, in which state funds represent a sizeable (even if diminishing) share of the revenues , while pWe also observe differences in how the largest universities are funded. In the US, private donations and endowments are the main source for the largest institutions and are heavily concentrated at the top of the pile . The 16 Fourth, there are differences in the respective distribution of input and output within the group of doctoral universities . The levIn other words, European HEIs “scale up” with student enrolments, with the distribution of staff and revenues closely following students and with research outputs only moderately more concentrated. On the contrary, revenues are more concentrated than students (and staff) in the US, while publications are far more concentrated. This indicates that the funding mechanisms in the US allows top-ranked universities to receive more resources per unit of staff, without a parallel increase in the number of students. As suggested by our statistical models, this is a powerful driver for achieving international research ‘excellence’.Our results move beyond the current debate on the use of bibliometric indicators for evaluation showing the association of such indicators with the volume of resources . PreviouBeyond the obvious assumption that more resources produce more output, we have shown that this relationship is tight across a wide range of size and across the two main scientific systems worldwide; further, we have observed super linear scaling both for publications and field-normalized citations, i.e. bibliometric output increases more than proportionally with revenues.These findings add a further worrisome dimension to the evaluation debate. By and large and especially on the top of the pile, bibliometric indicators and rankings are a richness measure and is questionable whether by orienting their decisions to these indicators policy-makers and stakeholders would do more than enriching the richer, under the presumption of promoting international ‘excellence’. A key component of this process is the existence of a universal (context-free) and measurable definition of ‘excellence’ that might differ from (context-related) quality . Such a We believe that bibliometric indicators do provide valuable information for evaluation purposes at the policy and institutional level. However, we rejoin previous critiques against their de-contextualized usage without taking into account local situations and specificities of scientific fields, countries and institutions . In thisat the institutional level bibliometric indicators provide signals aligned with resources makes this effect pervasive and penalizes highlands of high quality within ‘average’ universities [Further, we have demonstrated that the strongest association between resources and bibliometric outputs is via additional resources per staff, rather than an increase in the number of faculty. This suggests that a key underlying mechanism explaining the observed patterns is academic mobility, where highly productive scientists move towards the ‘best’ places in terms of ‘excellence’, while in their hiring behavior universities attempt at maximizing ‘excellence’ by investing more resources in few highly productive people . The facersities .Our results demonstrate that fighting for the top-positions in international rankings must be associated with the concentration of large amounts of resources in a few places. The analysis of the funding system suggests that this is associated with the long-term construction of institutional structures that allow resources to follow international ‘excellence’ signals . In the Policy implications are therefore different for the US and for Europe. In the US, promoting international excellence should not be a major focus of public policies as private capital already ensures it; instead, public policies should continue to be focused on widening access and ensuring good quality of education and research throughout the country, following the longstanding tradition of support to colleges and state universities . The incIn turn, at the institutional level, our results confirm that the battle for international rankings should not be the main concern of most university managers for two reasons: first, this process is driven by largely endogenous mechanisms and, at the least in the short and medium term, there is important inertia that makes it difficult to substantially change the amount and distribution of resources. Second, even in a well-funded system like the US one, this concerns only a handful of universities that account for a tiny proportion of higher education activities, particularly for what concerns education and the contribution to society and economy.At the same time, in the current evaluative society, higher education institutions cannot avoid comparing themselves. In this respect, our results suggest that the level of resources should be a key criterion for higher education managers to identify peers to compare with, alongside other criteria such as institutional mission and subject areas covered. To that end, institutional rankings based only on size-dependent indicators, should be restructured with a measure of resources or, at least, offer the option of selecting groups of institutions with a similar resource level and provide proper benchmarking strategies oriented by the institutional mission. Our analysis shows that data availability and quality should be no excuse for that.S1 Table(DOCX)Click here for additional data file.S2 Table(DOCX)Click here for additional data file.S1 Text(DOCX)Click here for additional data file.S2 Text(DOCX)Click here for additional data file.


Neural plastic changes are experience and learning dependent, yet exploiting this knowledge to enhance clinical outcomes after stroke is in its infancy. Our aim was to search the available evidence for the core concepts of neuroplasticity, stroke recovery, and learning; identify links between these concepts; and identify and review the themes that best characterise the intersection of these three concepts. a priori, and separate searches were conducted for each concept. The methodology involved three main phases: data collection and filtering, development of a clinical vocabulary, and the development of an automatic clinical text processing engine to aid the process and identify the unique and common topics. The common themes from the intersection of the three concepts were identified. These were then reviewed, with particular reference to the top 30 articles identified as intersecting these concepts.  We developed a novel approach to identify the common research topics among the three areas: neuroplasticity, stroke recovery, and learning. A concept map was created n = 6,498), stroke recovery , and learning . A cluster concept map (network graph) was generated from the results; indicating the concept nodes, strength of link between nodes, and the intersection between all three concepts. We identified 23 common themes (topics) and the top 30 articles that best represent the intersecting themes. A time-linked pattern emerged.  The search of the three concepts separately yielded 405,636 publications. Publications were filtered to include only human studies, generating 263,751 publications related to the concepts of neuroplasticity ( Our novel approach developed for this review allowed the identification of the common themes/topics that intersect the concepts of neuroplasticity, stroke recovery, and learning. These may be synthesised to advance a neuroscience-informed approach to stroke rehabilitation. We also identified gaps in available literature using this approach. These may help guide future targeted research. Neuroplasticity can be defined as the ability of the nervous system to respond to intrinsic or extrinsic stimuli by reorganizing its structure, function, and connections , this included 25 subheadings and further 32 subheadings under these subheadings. As an inclusion criteria for the collected studies, we selected research where experiments were conducted on humans.https://github.com/alistairwalsh/informatician.The PubMed database was accessed using the Entrez Programming Utilities (E-utilities), a set of eight server side programs that provide a programmatic interface to the National Center for Biotechnology Information (NCBI) database system . A pythoThe three separate requests with the query terms “neuroplasticity[MeSH],” “stroke[MeSH],” and “learning[MeSH]” returned associated PubMed ID numbers, which were then used to retrieve all the information available for those articles. The resulting XML documents were then searched for an English abstract along with their article title, abstract, and index terms  to produce a collection of studies that were searched for terms of interest.Three sources of data were collected and analysed for each article retrieved: title, abstract, and index terms, as identified in the article by the authors. This data was not only selected for its availability but also based on the expectation that key topic words should be captured in these sources. Further, data collected across these data sources should be comparable as the type of information included in abstracts is relatively uniform, with clear expectations, and is usually word limited, thus minimising bias due to variance in article length.Following the filtering of the collected documents, text mining tasks were performed to gain insights on the associations between the three concepts. Text mining is the process of extracting useful information from unstructured data and customization according to the requirements. For this purpose, it was necessary to build a vocabulary/initial seed word list, which could be used as the guide for text mining to extract relevant information. Therefore, a clinical vocabulary comprising of prominent topics in all three research areas was required. The following steps were undertaken to develop the vocabulary.An initial vocabulary was formed using the domain knowledge from experts. These topic vocabulary terms are listed in Index terms (keywords provided by authors) and MESH terms used by the authors for each article were included to further enrich the vocabulary.Word embedding is a machine learning technique that intelligently captures the context of a word in a document, i.e., capturing semantic and syntactic similarity as well as identifying the relation with other words. This technique was used to extract synonyms for the original list of terms  techniques and text analysis were used for this purpose. NLP is known as the application of computational techniques to analyse natural language which is unstructured textual data . The devIntelligent search algorithms  were useOnce the terms were extracted, it was essential to identify the common terms between the three groups. We used NLP techniques to automatically group publications that have similar topics and thereby identify unique and common clusters of topics.The results were then used to generate a weighted concept link map illustrating the topics that connect the concepts together. The output concept map represented an overview of the topics that link the three concepts together. Each connection was given a score based on the number of publications, therefore allowing filtering out only the important connections.The high-level process of the text analysis engine is illustrated in We conducted a post hoc analysis to explore if any time-related patterns emerged in relation to the emergence of topics for each of the three concepts over time. First, the three core concepts were analysed with the date of the publication and for each topic; a percentage was calculated for each year indicating the use of that topic in a particular year . We then analysed how the three concepts have been linked together from 1975 to 2018 to explore the emergence of patterns in the linking of concepts over time.n = 6,498), stroke recovery , and learning .Searching the three core concepts separately yielded 405,636 publications. Publications were filtered to include only studies of humans, generating 263,751 publications from the three groups. This included studies related to the concepts of neuroplasticity  for each concept and the evolution of how the topics link together over time revealed two outcomes: (1) overall topic distribution over time—this indicated how frequently a given topic was addressed in research studies each year thereby demonstrating the patterns over time; (2) the emergence of topics—this indicated when certain topics first appeared and how they evolved over time. Based on the patterns identified by these outcomes, we further examined the time-based topical associations to observe how the link (intersection) between the three concepts  had emerged over time. For demonstration purposes, we created three sets of publications based on the patterns detected by the time-based topic distribution. Three time periods emerged: (1) Early era (1975-1990); (2) Emerging era (1997-2003); and (3) Recent era (2012-2018). These time periods emerged primarily from the topic flow graph of neuroplasticity. Using the publications in these three groups, we analysed the evolution of the link between the three concepts. This process was automated by the proposed text mining approach.The aim of this review was to identify the literature that links neuroplasticity, stroke recovery, and learning in order to advance our understanding of and provide direction for a neuroscience-informed approach to stroke rehabilitation. The concept map generated by the text processing engine provides an efficient and rigorous approach to identify associations between different research areas as well as insights on important research themes and topics within a large pool of research publications. Moreover, the weighted link map provided a quantitative measure of the significance of the relationship between the themes; thus, the important topics could be identified. Finally, the intersection between all three concepts was defined and common topics identified. Time-linked patterns emerged from our analysis of the evolution of the link between the three concepts.The methodology used to conduct this review is novel. Commonly, when commencing a literature review, a basic search term of interest will return a very large number of articles. Subsequently, more complex search terms are added until a manageable number of articles are returned. This often means there is little knowledge of the articles being excluded before the human reviewers' start to look at the final articles. The approach detailed here of conducting an extremely broad search of the literature databases and using natural language processing to understand what is present means the choice of articles to include and perhaps more importantly, knowing what is being discarded from review, has the advantage of being controlled and repeatable.The intent of our approach was to identify key topics related to the core concepts in a systematic and comprehensive manner, thus scoping the currently available literature in the field. To achieve this, our approach employed a broad range of terms that represent the current literature and captured words that might have similar or overlapping meaning between studies and over time. The use of machine learning approaches involving text mining, word embedding, and natural language processing enhanced this feature of our review. However, there are two important considerations when conducting a literature search across different domains and across large spans of time. First, do the different domains use the same term to mean the same concept or are the same terms used to mean different things in their own domain? Second, has the meaning of a term changed over time or were concepts referred to by a different term in the past? Word embedding, which maps words to vectors of real numbers, can help with this, as it understands the context. The meaning of words and word relationships is derived from their use in the text rather than any dictionary definition. In line with this, it can describe what is in the current literature. It does not however attempt to define or evaluate the terminology used.The approach used allowed the existing literature to inform the themes and topics that link the three main concepts. In this way, it not only confirmed but also expanded the topics identified by domain experts. The topics identified that linked only two concepts were often quite specialised and limited. In comparison, 23 common themes/topics emerged from the intersection between all three concepts. This is reinforcing and provides direction to inform an integrated neuroscience and learning-based approach to rehabilitation.Cognition was the major theme identified  [As expected, on (FES) . This thtask-based learning and activity-based learning. The common focus on learning in the context of tasks and/or activities  was identified using this approach. The topic of task-specific training, a term often used in clinical settings, was also aligned. These learning approaches are seen as potential enhancers of neural plasticity [task-based training as a core element of therapy, consistent with recommendations [Different types of learning were identified in the context of neuroplasticity and stroke recovery, representing a clear intersection of all three concepts (Link 3). These included asticity . Task-baasticity . In factndations , 23, 57.skill and skill learning, focusing on the outcomes of learning. Skill learning in the context of stroke recovery and neurorehabilitation links learning-dependent plasticity with restorative therapies. The goal of learning-dependent plasticity is often the learning of a skill, such as juggling and playing a musical instrument. In the context of stroke recovery, it may be learning a sensorimotor skill, such as learning to grasp a cup in a more normal manner following paresis. We have clear evidence from animal studies that training is a critical ingredient to this change [Aligned with this focus on task- and activity-based learning is s change , 62. In s change , 63. Formotor learning, movement, and motor control. This finding highlights the current focus on movement outcomes, potentially at the expense of other functions or more complex outcomes. A relatively small proportion of articles focused on language and speech (9.2%). In comparison, focus on sensation (vision or touch) appeared to be missing as did more complex outcomes such as daily activities and or transfer to novel and/or complex activities. This likely reflects where the field currently is, i.e., in its infancy, in relation to applying knowledge that integrates neural plasticity with learning and valued stroke recovery outcomes. Nevertheless, the value of learning paradigms, in particular motor learning paradigms, is growing and a push to “infuse” motor learning research into neurorehabilitation practice is argued for in this literature [A relatively large proportion of the publications (20.78%) were focused on terature . An inteterature . The autexperience-dependent learning was identified as a topic linking only neuroplasticity and learning (not the 3-way intersection) in our review (Link 1). Experience-dependent learning is closely aligned with experience-dependent plasticity [Interestingly, asticity . Experieasticity . The potasticity ; howeverconnectivity, neuroimaging, BDNF, functional connectivity, and brain activation. The neurobiological mechanisms underlying recovery in patients with varying severity of impairment and in the longer term, are incompletely understood. New technologies are emerging and have a role in providing new insights [A few topics highlighted outcomes and/or mechanisms of change at a neurobiological level. Those topics that spanned underlying mechanisms or biomarkers included insights  and in hinsights , 65. Forinsights . This wainsights .sequence learning to relearning and neurorehabilitation, were also identified as themes. Given the focus on learning and search terms used, it was interesting to note that the current literature often did not include topics that reflect a greater specificity in the nature of the learning, e.g., implicit and explicit learning. An exception was the identification of sequence learning as the approach to motor skill learning by Wadden et al. [The process of and application of learning, including n et al. . Again, n et al. . Neverthn et al. .generalization and transfer (included in the MESH term for learning) did not emerge in any of the common themes. This is of potential concern given that outcomes associated with training and therapy need to be able to transfer to novel tasks and complex settings. The issue of sustainable and generalizable gains in motor skills and associated behaviors is highlighted in the rehabilitation literature [Of further interest is the fact that learning terms such as terature , 57. It terature . HoweverNeuroplasticity, learning, and transfer to novel tasks may be promoted by task complexity , 66, 67.learning modifiers was also identified as a topic. Factors that modify learning, its effectiveness, and impact at different times in the recovery trajectory are of interest. These factors ranged from factors such as BDNF [Finally,  as BDNF  to adjun as BDNF  and robo as BDNF , 51. One as BDNF . Other fFurther analysis was carried out to explore the evolution and associations of topics over time. Our objective was to observe how the topics in neuroplasticity, stroke recovery, and learning had evolved over time (1975-2018) using the collected sample of research studies from 1975 to 2018. Only a few topics were identified in the early time period (1975-1990). The link between neuroplasticity and stroke was established via research focused on “Brain,” while the link between neuroplasticity and learning was established via studies on “Stimulation” and “Consolidation.” In contrast, the Emerging era (1997-2003) showed the appearance of many more topics in neuroplasticity and the links have more weight indicating the availability of more research studies. The analysis of research in the Recent era (2012-2018) disclosed the emergence of many new topics. The link between neuroplasticity and stroke recovery was further expanded by studies on “cortical activation” and “neurostimulation.” It was also observed that the link between stroke recovery and learning was established in this time period, thus linking all three concepts together.As this analysis was automated by the text mining approach described, further analysis and comparison using different time periods will allow disclosing other interesting patterns and insights regarding the associations among the three concepts. We present this time-based topic analysis as further contribution to the proposed approach as it enables researchers to mine useful time-based patterns from many publications without manual processing.Some recommendations for future research emerge from our review. The development of computational models of salient neural processes , includiThe development of future interventions should match neuroscience and learning principles to specific outcomes. In particular, the need to systematically target the intersect between neural plasticity and learning to achieve better generalisation of training effects and transfer to novel tasks in the context of stroke rehabilitation is critical. With further understanding, the potential to individualise therapy emerges. This may include the recognition of underlying capacities that support a particular type of learning, through genetic variations and strategies that influence modifiers of learning, such as BDNF. Finally, future research should be directed at discovering drivers of the different types of plasticity, as well as when they might best be applied at different times in the recovery trajectory.In summary, the novel approach taken in this review allowed us to identify and characterise not only the topics that are currently being investigated in the literature but also those that are not or are only infrequently mentioned. Identification of the common intersecting themes linked with the core concepts proposed now provides a foundation of literature that may be synthesised to advance a neuroscience-informed approach to stroke rehabilitation. Further, such an approach helps to identify gaps in the field that may be important, as researched and recommended in related fields. For example, the topics of transfer and generalisation have been extensively researched in the field of learning, but did not emerge as an intersection with neural plasticity and stroke recovery. The review of the concepts of neural plasticity, learning, and stroke recovery and the common themes and topics that link them has provided direction for future research, important in the development of new neuroscience and learning-based therapeutic approaches. Finally, the potential also exists to develop theoretical frameworks by which new interventions may be conceptualised, incorporating knowledge of the intersection between contributing fields of research.


The rapid advances in metabolomics pose a significant challenge in presentation and interpretation of results. Development of new, engaging visual aids is crucial to advancing our understanding of new findings.We have developed MetaboCraft, a Minecraft plugin which creates immersive visualizations of metabolic networks and pathways in a 3D environment and allows the results of user experiments to be viewed in this context, presenting a novel approach to exploring the metabolome.https://github.com/argymeg/MetaboCraft/; https://hub.docker.com/r/ronandaly/metabocraft/Bioinformatics online. Metabolomics as a field has expanded rapidly in recent years, owing both to increased interest as well as advances in instrument throughput and resolution, accordingly resulting in the accumulation of a vast metabolomic data repository , howeverPiMP (At a conceptual level, metabolic maps can be interpreted as mathematical graph objects, enabling the application of widely applied methods towards both their design and their interpretation . MetExplMinecraft is an open-world, sandbox building game written in Java, similar in its conception to a virtual Lego construction set, which allows the player to construct arbitrarily large and complex structures, using a variety of visually and even functionally different building blocks. It has been extensively used in educational and scientific contexts in the past, including the illustration of physical, chemical and biological phenomena . Such apThe MetaboCraft server stack is comprised of three components: a Spigot server, a Shiny server and a plumber server. The Spigot server runs JavaScript code, while the Shiny and plumber servers run R code.A series of shell scripts are provided for installing, running and maintenance. The installer automates the downloading and compilation of Spigot, the downloading and installation of the relevant plugins and finally the JavaScript component of MetaboCraft, on top of Spigot. It initializes the configuration of the Minecraft world with a variety of settings appropriate for the desired functionality. It also attempts to automate the installation of the necessary R packages, which due to variations in R setups is not always achievable. The script attempts to anticipate such cases and prompt the user to manually install the missing packages, if necessary. As well as being able to install the components by hand, a Docker image is available that provides a one-click installation option for quick and easy setup.By default, MetaboCraft displays data from MetExplore BioSource 4324, derived from the Recon 2 reconstruction of the human metabolome (The Shiny web application can be accessed on port 38909 on the local machine. After optionally uploading their files, on logging into the server using Minecraft, the user is presented with a welcome message informing them of the default viewing parameters and the data files uploaded under their name and available to them. They are presented with a panoramic view of the full default metabolic network map, along with a series of control blocks presenting the choice of cellular compartments. Choosing a compartment presents the user with a view of the network map of that specific compartment, while choosing a pathway from the map presents the 3D graph of that pathway, along with an option to go back to the map view. The various entities represented in a metabolic pathway graph are colour-coded for legibility .Several commands are also implemented for the user to input into the Minecraft console, allowing direct access to these features, as well as additional control over the behaviour of MetaboCraft, such as switching between uploaded files and changing the pathway network map layout.MetaboCraft joins an already crowded field of visualization frameworks for metabolic networks, such as MetExploreViz  and TULIThis work was supported by the Wellcome Trust [105614/Z/14/Z].Conflict of Interest: none declared.Supplementary DataClick here for additional data file.


To summarise the definitions and combinations of codes used to identify outcomes of anxiety, depression, fatigue, cognitive dysfunction (including mild cognitive dysfunction and dementia), sexual dysfunction, pain, sleep disorders, and fatal and non-fatal self-harm in studies using electronic health records from primary care databases in the UK.Systematic review.Medline, Embase and lists of publications of the main primary care databases in the UK.Included data from a UK primary care database and studied outcome(s) of interest.We abstracted information on the outcomes definition and codelists. When necessary, authors were contacted to request codelists.120 studies were eligible. Codelists were available for 17/42 studies of depression; 21/41 studies of fatal and non-fatal self-harm; 17/27 studies of dementia/cognitive dysfunction; 5/12 studies of anxiety; 4/8 studies of pain; 3/6 studies of fatigue and sexual dysfunction; 1/2 studies of sleep disorders. Depression was most often defined using codes for diagnoses (37/42 studies) and/or antidepressants prescriptions (21/42 studies); six studies reported including symptoms in their definition. Anxiety was defined with codes for diagnoses (12/12 studies); four studies also reported including symptoms. Fatal self-harm was ascertained in primary care data linked to the Office for National Statistics mortality database in nine studies. Most studies of cognitive dysfunction included Alzheimer’s disease, and vascular and frontotemporal dementia. Fatigue definitions varied little, including chronic fatigue syndrome, neurasthenia and postviral fatigue syndrome. All studies of sexual dysfunction focused on male conditions, principally erectile dysfunction. Sleep disorders included insomnia and hypersomnia. There was substantial variability in the codelists; validation was carried out i21/120 studies.There is a need for standardised definitions and validated list of codes to assess mental health and quality of life outcomes in primary care databases in the UK. Comprehensive systematic review of the literature aiming at describing the definitions and combination of codes used to identify outcomes of mental health and quality of life in electronic health records databases in the UK.Potential for error in the selection of the eligible studies minimised by duplication of the screening.The authors of the original studies were contacted to obtain the list of Read codes used when these were not publicly available.We only considered definitions of study outcomes, and did not consider studies where mental health or quality of life variables were covariates or exposure variables, limiting the generalisability of our results to these other contexts.Primary care databases of electronic health records (EHRs) in the UK such as The Clinical Practice Research Datalink (CPRD), QResearch or The Health Improvement Network (THIN) have been widely used to study mental health outcomes such as depression,Strategies to identify patients with a given condition in the EHRs typically include generating lists of relevant codes, then searching the patients’ record for these codes to identify symptoms, diagnoses, referrals, appointments for disease management and monitoring, and/or prescriptions of interest.Given the broad interest in mental health and QoL outcomes, and the strong potential for primary care data to contribute to studying these outcomes, our aim was to systematically review and summarise the strategies used to define such outcomes in previous studies, and the extent to which case definitions have been validated.This review followed the a priori defined methods specified in the systematic review protocol online .10.1136/bmjopen-2019-029227.supp1Supplementary dataThe outcomes of interest for this review were: anxiety, depression, fatigue, cognitive dysfunction, pain, sexual dysfunction, sleep disorder and fatal and non-fatal self-harm. We considered that a study provided data for cognitive dysfunction when dementia, mild cognitive impairment or single domains of cognitive function were studied . Composite outcomes of two or more of these outcomes  were also eligible.We searched MEDLINE and Embase via Ovid, from inception up to 28 June 2018, to identify studies that involved EHRs from primary care databases and studied one of the outcomes of interest (see above). The search expressions are provided in online We considered eligible the studies that used data from a primary care database that routinely gathers EHR data from primary care practices in the UK, and in which the outcome of interest was one of those of interest for this study (see list above). This included purely descriptive studies on the incidence/prevalence of the outcome and analytical studies where the condition of interest was one of the main outcomes of the study. Studies of primary care data linked to other sources of data, such as the Hospital Episode Statistics (HES) or the Office for National Statistics (ONS) mortality data, were also considered eligible.Abstracts from conferences were excluded, as it was unlikely that the methods section would provide sufficiently detailed information on the definition of the outcomes. Studies of pain caused by infectious agents  were excluded; similarly, studies of sleep apnoea and narcolepsy were excluded due to their unlikely psychological origin.The eligibility of the studies was determined by two authors (HC and HS) reviewing all records retrieved from the publications databases. First, the title of each study was read to determine the eligibility for the review; when the information provided in the title was insufficient for a clear exclusion of the study, the study was considered for further assessment. Second, the full text of each study not previously excluded was read, in order to determine the eligibility. Disagreements over study eligibility between the two reviewers were resolved by discussion, including with a third researcher (KB or RW) where needed.We abstracted data on study characteristics , the primary care database used, any database(s) linked to the primary care data, outcome(s) reported, definition of the outcome(s)  codes, etc) and any codelist available. When there were two or more definitions of the outcome , we abstracted all information but considered only the main outcome for data analysis in this review. We also abstracted data on whether the codelist had been validated, and any description related to the handling of past or prevalent (at baseline in cohort studies) episodes of these outcomes. We considered that the study had attempted to validate the list of codes when the results were compared with data from another source, or when outcomes were confirmed by enquiring the patients’ GP or by reviewing the patients’ medical record. The data extraction process was repeated by a second author (HS) for 10% of the papers included for each outcome, to check for reliability in the extraction process.When a study did not provide the codelist for the definition of the outcome in the original publication or in a publicly available repository, we contacted the corresponding author of the study by email seeking this information online . In the 10.1136/bmjopen-2019-029227.supp2Supplementary dataWe produced descriptive tables showing the number and proportion of studies eligible for each outcome, by primary care database and codelist availability. We described, for each outcome, the types of codes used in the definition of the outcomes . The lists of codes were also reviewed to assess the clinical characteristics of the disorders included ; this was done by manually reviewing the list of codes to identify codes related to different clinical characteristics of the specific outcome. To describe the Read codes most commonly used to identify these outcomes in the data, we produced a list of Read codes sorted by number of studies that used the code. The results of the validation studies described in the original papers were reported descriptively.No patients or public were involved in the design and conduct of this study.Of 5946 records initially identified in the bibliographical references search, 2979 were discarded as being duplicated, which left 2967 records to be assessed for eligibility . The titTwelve studies had anxiety as an outcome ; of thes10.1136/bmjopen-2019-029227.supp3Supplementary dataOf the five studies for which codelists were available, five included codes for generalised anxiety disorder (100%), four for phobia (80.0%), four for panic disorder/attacks (80.0%), three for mixed anxiety and depression (60.0%), and two for stress-related disorders (40.0%) . Codes fOnly one study reported including drugs prescriptions in the definition of the outcomeTwo studiesOnline 10.1136/bmjopen-2019-029227.supp4Supplementary dataForty-two studies identified outcomes of depression . The lisOf the 17 studies for which the codelists were available, 10 included codes for mixed anxiety and depression (58.8%), 4 for bipolar disorder (23.5%) and 3 for depression in dementia (17.6%) .Antidepressant prescriptions, in isolation or combination with diagnostic/symptoms Read codes, were considered in the identification of patients with depression in 21 studies (50.0%); in six studies, depression was solely defined by the prescription of antidepressants . The lisFive studies (11.9%) assessed the performance of the list of codes to identify patients with depression . The proThe list of Read codes used to identify patients with depression is provided in online Two studies provided data for composite outcomes of anxiety and depression . The codet alJohn Twenty-seven studies reported outcomes of dementia or cognitive function . The codOf the 17 studies for which the codelists were available, 13 reported codes for Alzheimer’s disease (81.3%), 13 for vascular dementia (81.3%), 12 for frontotemporal dementia (75.0%) and 11 for Lewy bodies disease (68.8%) . Three sFour studies (14.8%) used prescriptions in the identification of patients with dementia; all four included anticholinesterases and dopaminergic agents .Five studiesThe list of Read codes used in the studies is provided in online Six studies had fatigue as the outcome . All stuThe three studies for which the codelist was available included codes for chronic fatigue syndrome, neurasthenia and postviral fatigue syndrome . FibromyPain was the outcome in eight studies . The lisThree studies validated the list of patients selected with the codelist . The proOnline Six studies had sexual dysfunction as an outcome, all of which focused on male sexual dysfunction . Three sTwo studies were eligible for sleep disorders ; the twoForty-one studies had outcomes related to fatal and non-fatal self-harm . The lisOf the 21 studies for which the codelist was available, 17 studies (80.9%) included completed suicide, while 4 studies focused on attempted suicide only (19.1%). Of the 17 studies including completed suicide as an outcome, eight reported only completed suicides, six considered completed and attempted suicides and four included complete and attempted suicide, as well as self-harm . All stuet alNine studies involved some method of validation of the list of cases identified via code search . Four stOnline This review summarised the definitions and combinations of codes used to identify outcomes of anxiety, depression, dementia and cognitive impairment, fatigue, pain, male sexual dysfunction, sleep disorder and self-injurious behaviour in primary care databases of patients in the UK. The list of codes used in the original studies was obtained for approximately half of the papers; the lack of detailed information on the definition of the outcomes in most studies raises important questions as to whether studies can be replicated by others. In the studies where the codelist was available, for all outcomes, there was substantial heterogeneity in the type of codes included  and drugs selected to identify outcomes; for the remaining studies, the details provided in the original publications suggest a similar pattern. We also noted considerable variability in the clinical definition of some outcomes . Validation of codes used to identify these outcomes was rarely carried out; where done, positive predictive values of case definitions were variable but mostly above 80%. To overcome these issues in the current context of limited number of studies with validation efforts, it is imperative that researchers develop, validate and make publicly available code lists for these outcomes.This review is based on an extensive search of the studies involving EHRs in the UK. Errors in study selection and data extraction were minimised by the independent assessment of the studies by two investigators. We contacted the authors of all original studies where the list of codes had not been provided in the original publication to seek this information; this largely increased the number of studies for which lists of codes were available, and contributed to a more detailed characterisation of the combination of codes used to define mental health outcomes in primary care databases of EHR in the UK.However, this review has limitations. Some relevant studies may have been missed due to imperfect search terms, as there is no Medical Subject Headings (MeSH) term for the primary care databases, and studies could be potentially missed if the keywords did not appear in the title and abstract, or due to inaccurate indexing in the publications database. We attempted to minimise the risk of missing potential eligible studies by using broad search terms incorporating both indexing terms and keywords, two databases with different indexing systems, and an additional manual check of the eligible studies and list of bibliographical references from the main EHR databases. We only considered studies where mental health or QoL variables were the outcomes of interest, limiting generalisability to other contexts. For example, we excluded studies where these variables were covariates because we expected that detailed information about covariate definitions would rarely be available. We also excluded studies where the mental health or QoL variable was used to define the patient population , on the basis that decisions about how to define cases may have had quite different motivations, compared with studies where the condition was the outcome of the study, making case definitions difficult to meaningfully compare. We included studies that explicitly referred to using prescription data as a proxy for the definition of the condition , but we acknowledge that it was not always clear to decide whether treatment of the condition was being used to define the condition. This could have resulted in a few studies erroneously excluded, even though this should have been minimised by the duplication of the search and study selection process by two researchers working independently, with discussion of all discordant results. It is unclear if the list of codes that could not be obtained differ in any systematic way to the ones obtained. Some authors expressed concerns over intellectual property when sharing the list of codes, and this may have been a bigger concern among those who put a lot of time and thought into their codelists; on the other hand, authors who have concerns about the quality of their codelist may have been less willing to share them. Lastly, we summarised the types of codes used to define the outcome based on what was stated in original studies’ methods sections , but this may have been inaccurate, for example, some studies that reported in their methods only including diagnosis codes then provided code lists that appeared to also contain symptom codes.The list of codes was provided in the original publications for just over a quarter of the studies. Contacting the authors resulted in codelists being made available for approximately half of the studies. For the remaining studies, the authors either could not be contacted  or could not locate the relevant codelist (including for some studies where the paper had stated that the codelist would be available on request). Provision of codelists within the publication or in a web repository would eliminate the difficulties of authors having to be contacted and archived codelists retrieved. Most journals currently accept codelists in online Anxiety was often defined with diagnostic and symptoms codes, and in a few studies by the prescription of anxiolytics and hypnotics. Even though the sensitivity of symptoms codes for anxiety is expected to be high, the positive predictive value is unknown. Anxiolytics may also result in misclassification of the outcomes, as they are currently discouraged as first line of treatment for anxietyet al explored the indications of antidepressants; more than half of the new antidepressant prescriptions were for depression, with increasing but low incidence of prescriptions for pain and anxiety, but the authors could not identify the indication for antidepressants in 17% of the new prescriptions.The inclusion/exclusion of codes for symptoms may have a larger impact in the definition of depression, as it has been shown that GPs switched from diagnostic to symptoms codes after the introduction of performance indicators in the GP contract Quality and Outcomes Framework in 2006Regardless of the type of codes included, authors will need to often choose the inclusion/exclusion of codes relating to the clinical profile of the patients. This may have a particular impact for conditions that are highly comorbid. For example, the code for ‘mixed anxiety and depression’ was sometimes used in the definition of anxiety and in the definition of depression; anxiety and depression are highly comorbid and the inclusion/exclusion of these patients may have an impact on the results. In addition, for depression, the inclusion of codes related to depression in the context of bipolar disease, dementia and schizophrenia may raise issues as to whether it represents a primary depressive episode.Part of the heterogeneity in the list of codes used to identify these outcomes may be explained by the complexity of these conditions and by the purpose for which these data are collected. Electronic healthcare data are primarily collected to provide patients with treatment, and distinctions between diagnosis and symptoms may have less weight at the point of care than when researchers aim to define these conditions using data routinely collected.et al reviewed free text to exclude those who did not seem to have attempted suicideRoutinely collected primary care data were shown to have low sensitivity to detect cases of suicide.The aetiology and location of pain in the studies involved in this review varied due to our broad inclusion criteria. When pharmacological treatment was included in the definition of pain, this was most often done with prescriptions of antidepressants and antiepileptics. Antidepressants such as first-generation tricyclic antidepressants have been used for over 30 years to manage neuropathic pain .We considered cognitive dysfunction as a composite outcome including studies from mild to severe impairments such as those in dementia; between 10% and 20% of the patients with mild cognitive impairment are expected to convert to dementia.Outcomes identified in EHRs may lack of validity: a person meeting the operational definition for the outcome based on specific codes may not have the diagnosis or vice versa. Only a small number of studies assessed validity in their studies, and this was almost always about assessing positive predictive value of the case definition, with sensitivity and specificity rarely explored. Of these, some studies only stated that validation had been carried out, but did not report the results, which makes the performance of the case definition unclear. However, the studies that reported results tended to show a high proportion of cases confirmed by their primary care physician or by further investigations . This is in accordance with the results of two systematic reviews that assessed the validity of the diagnostic coding within the CPRD primary care database.Mental health and QoL-related outcomes are difficult to identify in EHR databases; and thus, extra care needs to be used when defining these outcomes. The use of a particular code can vary between GP practices; for example, a study on the interpractice use of Read codes for diabetes showed that the most generic code was used in 14%–98% of the patients with diabetes in the practices.Validation of the outcomes appears to be essential to understand the validity of case definitions. A balance between sensitivity and specificity may be considered depending on the aim of the studyPrimary care databases of EHRs have made important contributions to medicine worldwide, particularly in the fields of infectious, respiratory and cardiovascular diseases. The burden of mental disorders in high-income countries has increased substantially in the last decades,Despite the difficulties of assessing each separate outcome, we must take into account that mental health disorder symptoms often overlap, and is difficult to disentangle what is attributable to each condition. Lastly, these conditions have a long period of exposure to medication after symptoms have disappeared, besides a high probability of relapse and recurrence,Detailed information about codes used to identify outcomes of anxiety, depression, fatigue, cognitive dysfunction, sexual dysfunction, pain, sleep disorders, and fatal and non-fatal self-harm in studies using EHRs from primary care databases in the UK was unavailable for around half of studies of these outcomes. Where available, there was substantial heterogeneity in the list of codes used to ascertain cases. Most studies did not validate case definitions, though when this was done, positive predictive values were generally high. This review focused on common mental health disorders and QoL outcomes, but our conclusions are likely to be generalisable to other mental health outcomes. Caution is needed when interpreting and comparing results between studies, as heterogeneity in case definitions may be large. Future studies should fully report outcomes definitions, use sensitivity analysis to mitigate uncertainties about the impact of the case definition on studies’ reported outcomes, and seek to validate the list of codes used to identify these outcomes.


Outbreaks of several serious infectious diseases have occurred in recent years. In response, to mitigate public health risks, countries worldwide have dedicated efforts to establish an information system for effective disease monitoring, risk assessment, and early warning management for international disease outbreaks. A cloud computing framework can effectively provide the required hardware resources and information access and exchange to conveniently connect information related to infectious diseases and develop a cross-system surveillance and control system for infectious diseases.The objective of our study was to develop a Hospital Automated Laboratory Reporting (HALR) system based on such a framework and evaluate its effectiveness.We collected data for 6 months and analyzed the cases reported within this period by the HALR and the Web-based Notifiable Disease Reporting (WebNDR) systems. Furthermore, system evaluation indicators were gathered, including those evaluating sensitivity and specificity.Streptococcus pneumoniae, Mycobacterium tuberculosis complex, and hepatitis C virus were 99.8%, 96.6%, and 97.4%, respectively. However, the specificity for influenza virus and hepatitis B virus were only 79.9% and 47.1%, respectively. After the reported data were integrated with patients’ diagnostic results in their electronic medical records (EMRs), the specificity for influenza virus and hepatitis B virus increased to 89.2% and 99.1%, respectively.The HALR system reported 15 pathogens and 5174 cases, and the WebNDR system reported 34 cases. In a comparison of the two systems, sensitivity was 100% and specificity varied according to the reported pathogens. In particular, the specificity for The HALR system can provide early reporting of specified pathogens according to test results, allowing for early detection of outbreaks and providing trends in infectious disease data. The results of this study show that the sensitivity and specificity of early disease detection can be increased by integrating the reported data in the HALR system with the cases’ clinical information  in EMRs, thereby enhancing the control and prevention of infectious diseases. Electronic laboratory reporting (ELR) generally refers to the automated transmission of reportable laboratory findings from public health services, hospitals, and other laboratories to local or state public health agencies. Many communicable diseases currently under nationwide surveillance can be identified and confirmed by laboratory test results. Thus, ELR has become a critical part of the disease surveillance process. Previous studies have shown that ELR improves the timeliness, accuracy, and completeness of reported laboratory data, which, in turn, improves the effectiveness and efficiency of public health responses to outbreaks and cases of notifiable conditions -4. It haAn electronic medical record (EMR) is a systematized collection of computerized patient and population health information. An EMR supports secure, real-time, point-of-care, patient-centric information availability and is a resource for clinical care as well as research and education. Recently, EMR systems have become an increasingly pervasive technology in health care settings -14. SeveIn Taiwan, as required by law , clinicaIn order to reduce the workload of surveillance professionals, and address the lack of specificity of ELR, the TCDC launched a pilot project, called Automated Laboratory Reporting (ALR), in 2014 . The ALRIn this study, we implemented a counterpart of the TCDC’s ALR system in a teaching affiliate hospital with about 1000 beds starting in August 2014 in Taiwan. The hospital has an LIS to manage laboratory orders and results, a computerized physician order entry (CPOE) system to support clinicians in their daily clinical practice including entering test orders and reviewing test results, and other EMR systems. It can also access the WebNDR system to report notifiable diseases manually.Based on the framework of the TCDC’s ALR system, this study documents the implementation in a hospital setting, termed the Hospital Automated Laboratory Reporting (HALR) system . The HALWhen the LIS releases a pathogen test result, the ADRP module checks the pathogen against the NPDB. LIS pathogen test results include patient ID, name, gender, test item, specimen type, and pathogen report content. If the pathogen can be found in the NPDB and its test result is positive, the test case is labeled as reportable. The patient’s information is then included with the pathogen test result to form a reportable case. The laboratory test is encoded with a hospital code; however, the TCDC requires a laboratory test with a Logical Observation Identifiers Names and Codes (LOINC) code.Thus, the ADRP module can translate the hospital code of a pathogen test into its corresponding LOINC code by referencing a local LOINC cross-mapping database. The reportable test cases detected by the ADRP module are then stored into the Reportable Test Case Database (RTCDB).Finally, the NCR module retrieves a reportable test case from the RTCDB and links the test data with the patient’s clinical information, such as disease name, diagnosis code, and the date when the condition was diagnosed, which are stored in the CPOE system. Then, the linked data are compiled into a laboratory-reportable case in a digital format such as XML that is defined by the TCDC. Subsequently, the HALR Gateway sends the laboratory-reportable case to the TCDC ALR system; this belongs to the in-hospital system data and is indicated by a thin line.When the LIS confirms a pathogen test result that meets a laboratory reporting condition, the pathogen test result and other information, such as order number, local code, specimen type, and time of result, will be automatically written into the RTCDB.After retrieving new reporting data, the program converts the laboratory test items to LOINC codes and retrieves other patient information required by the TCDC from different hospital information systems and then submits the combined data to the hospital reporting module within the TCDC gateway in the required format.Since the WebNDR system is dedicated to reporting communicable diseases, the five disease categories reported by this system are used as a basis of evaluating the HALR system. The TCDC provides a table for mapping between pathogens and their related diseases .After the HALR system was implemented in a regional hospital in Taipei, we collected reported cases from the HALR and WebNDR systems for 6 months, from December 2014 to May 2015. Since the reported cases from the WebNDR system were confirmed by infectious control professionals, they served as a gold standard for evaluating the HALR system in this study. Thus, the sensitivity and specificity of the reported cases by the HALR system could be evaluated according to the following definitions:Sensitivity (true positive rate): If a patient has a notifiable disease reported by the WebNDR system, the sensitivity is the probability that the HALR system would report the case. The numerator is the number of cases identified as positive and reported by both the HALR and WebNDR systems, whereas denominator is the total number of cases reported by the WebNDR system (# of reported cases by both the HALR and WebNDR systems)/(# of cases reported by the WebNDR system).Specificity (true negative rate): If a patient does not have a notifiable disease and is not reported by the WebNDR system, the specificity is the probability that the HALR system would not report the case. The numerator is the number of cases in whom test results were identified as negative and were not reported by the WebNDR system, whereas the denominator is the total number of cases not reported by the WebNDR system (# of not reported cases by neither the HALR nor WebNDR system)/(# of cases not reported by the WebNDR).The cases reported by the WebNDR system were patients with notifiable diseases, whereas those reported by the HALR system were patients whose pathogen test results were positive. During his or her hospital stay, a patient with the same pathogen reported more than once  was excluded.Furthermore, since a pathogen may cause more than one disease, the HALR system includes clinical diagnosis information such as that from the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM). Thus, the test results were combined with ICD-9-CM data to determine whether the HALR system performance could be improved.Sensitivity (with ICD): The numerator is the number of cases identified as a notifiable disease with a positive test result, which were automatically reported by the HALR and WebNDR systems. The denominator is the total number of cases reported by the WebNDR system.Specificity (with ICD): The numerator is the number of cases identified as negative and, therefore, not reported by either the HALR or WebNDR system. The denominator is the total number of cases not reported by the WebNDR system .Based on the TCDC’s reportable pathogens and reporting criteria, 15 pathogens were reported to the TCDC ALR system . Among tStreptococcus pneumoniae infection (2), and acute viral hepatitis B (1).Simultaneously, 34 cases in five disease categories were reported by the WebNDR system . The numS. pneumoniae, Mycobacterium tuberculosis complex, and hepatitis C virus were 99.8%, 96.6%, and 97.4%, respectively. In addition, the specificity for influenza and hepatitis B virus were only 79.9% and 47.1%, respectively.Because the HALR system collects a reported case that includes not only laboratory test results but also clinical information (such as ICD-9-CM) from the TCDC ALR system, only cases with associated laboratory positive test results and an associated diagnosis code that meets the TCDC’s notifiable disease definition are reported to the TCDC ALR system . The senIn this study, we developed an HALR system for the automatic reporting of pathogen test results and clinical information. Once laboratory test results are released, the HALR system can automatically detect pathogens that meet notifiable conditions as defined by the TCDC and report the cases to the TCDC ALR system. Since the patients’ laboratory test results are usually released to their physicians far in advance of the physicians’ final diagnoses of any notifiable diseases, the HALR system can improve the timeliness for notifiable disease surveillance and control. Moreover, this system’s effectiveness is also improved as long as the doctor has included a working diagnosis.As indicated in previous studies -10, an EIn response to emerging infectious diseases, the HALR system detects reportable cases based on the NPDB. This design allows the TCDC to add, delete, or update the definitions of reportable pathogens as new emerging threats are identified. Hence, as this system can routinely download and update the NPDB, it can quickly respond to altered requirements.The improvement of the sensitivity and specificity of reportable cases depends not only on timely and accurate laboratory test results but also on the availability of clinical information required to identify cases. This study simultaneously examined reporting data on the basis of test results and clinical evidence in order to enable data sharing between information systems. Through double verification, with this approach, hospitals can immediately confirm patients’ conditions and submit reports when required. Thus, the sensitivity and specificity for early disease detection can be improved. The findings of this study can serve as a reference for disease prevention measures in clinical care.The following are some limitations of this system. First, the system will be limited to tests accepted by patients in the hospital. This system can only automatically link information when required clinical data are complete. Second, the notification condition of the test and the positive definition of the pathogen remain important factors affecting the accuracy of the content of the notification.In this study, we developed an HALR system in a hospital to automatically and actively report pathogen test results and relevant clinical information to the TCDC ALR system. The study results show that the HALR system can improve the timeliness, sensitivity, and specificity of reported cases. Furthermore, it can provide the flexibility to integrate frequent changes to the definitions of notifiable cases when the TCDC finds new, emerging threats or diseases.The improvement of the sensitivity and specificity of reportable cases depends not only on timely and accurate laboratory test results but also on the availability of clinical information that may be required for identified cases. This study recommends the following feasible solutions: integrate automatic pathogen reporting by the HALR system with related medical data in patients’ EMRs through the automatic system determination, with EMR data provided according to the TCDC clinical standards specifications for infectious diseases