The NIST/NASA/CARB Biological Macromolecule Crystallization Database (BMCD), NIST Standard Reference Database 21, contains crystal data and crystallization conditions for biological macromolecules. The database entries include data abstracted from published crystallographic reports. Each entry consists of information describing the biological macromolecule crystallized and crystal data and the crystallization conditions for each crystal form. The BMCD serves as the NASA Protein Crystal Growth Archive in that it contains protocols and results of crystallization experiments undertaken in microgravity (space). These database entries report the results, whether successful or not, from NASA-sponsored protein crystal growth experiments in microgravity and from microgravity crystallization studies sponsored by other international organizations. The BMCD was designed as a tool to assist x-ray crystallographers in the development of protocols to crystallize biological macromolecules, those that have previously been crystallized, and those that have not been crystallized. The three-dimensional structures of biological macromolecules have become essential elements for guiding studies and developing an understanding of biological processes. The coordinates provide essential information for protein engineering, rational drug design, and other biotechnology efforts. X-ray crystallographic studies of biological macromolecules provide a mechanism of attaining this information. The diffraction studies require the production of large single crystals of the biological macromolecule of interest. An example of a crystal of ribonuclease S suitable for crystallographic studies is shown in The solution properties of a biological macromolecule, determined by factors such as size, shape, surface complexity, and conformational stability, directly relate to its ability to crystallize. A number of recent studies of the biophysical properties of biological macromolecules to discover simple methods that would have predictive capabilities have been prompted by recognition of these facts . Traduniversal strategy for searching for the crystal growth parameters for a biological macromolecule has gained widespread acceptance. Many laboratories have recently begun using experimental procedures called “fast screens” [Although several systematic procedures and strategy suggestions have been put forth –11, no uscreens”  that usescreens” . These pThe development of a productive crystallization strategy may yield crystals, but the internal order of a crystal dictates the quality of x-ray diffraction limiting the information that can be obtained from the ensuing crystallographic studies. An improvement in the diffraction quality of the crystals will not only facilitate the structural investigation, but also directly influence the accuracy of the coordinates obtained from the structural investigation. Recently, a number of crystallization experiments undertaken in space indicate microgravity has a positive influence on the crystal growth process, i.e., an improvement in the diffraction quality of crystals of biological macromolecules . BecauseThe Biological Macromolecule Crystallization Database (BMCD), besides cataloging and summarizing the information concerning crystallization that is available in the literature, now includes the National Aeronautics and Space Administration (NASA) Protein Crystal Growth Archive. A description of the data and software that constitute the BMCD and the NASA Protein Crystal Growth Archive along with illustrations of how the BMCD can be used to address the needs of macromolecular crystallographers in the search for crystallization protocols is presented here.In 1989, with assistance from the National Institute of Standards and Technology Standard Reference Data Program, the NIST/CARB (Center for Advanced Research in Biotechnology) BMCD database was created. The menu driven software used to access the data is written and compiled with ClipperA second version of the NIST/CARB BMCD software and data was released in 1991 . This reRecently, Version 3.0 of the BMCD has been ported to a UNIX platform to take advantage of the development of network capabilities that employ client-server tools . This imThe BMCD includes crystallization data for peptides, proteins, protein-protein complexes, nucleic acids, nucleic acid-nucleic acid complexes, protein-nucleic acid complexes and viruses for which diffraction quality crystals have been obtained and reported in the literature. In addition, the BMCD, serving as the NASA Protein Crystal Growth Archive, contains the crystallization data generated from ground-based and microgravity crystallization studies supported by NASA. Crystallization data from microgravity experiments sponsored by other international space agencies are also included.The information contained in the BMCD is divided into three major categories, biological macromolecule, crystallization, and summary data. Each biological macromolecule included in the database has a unique biopolymer sequence. For example, a mutant protein with a single amino acid change from the wild type protein is considered a separate macromolecule entry. Analogously, a crystal reported in the literature must have undergone a preliminary x-ray crystallographic study and have unique unit cell constants. Crystal entries that are nearly isomorphous with previously reported crystals are included if the author(s) describes significant differences in the intensity distribution of the diffraction pattern from that previously reported. The BMCD data elements for each category mentioned above are described below. Many data items, which are not appropriate for the biological macromolecule or crystal form or which may be missing from the published report, are not included in the data entries.in vivo assembly associated with other parts by noncovalent interactions. For example, the two monomers of a dimeric enzyme and the two oligomeric nucleic acid strands of a double-stranded nucleic acid fragment are considered in both cases as two subunits. The name of a prosthetic group associated with a biological macromolecule is included if it is reported in the crystallographic studies. If the macromolecule is an enzyme, the EC number [The entries for each biological macromolecule are composed of data elements that provide a unique description of the biopolymer or biopolymer complex. The data elements include the preferred name, the systematic name, and other aliases of the macromolecule. Each entry contains biological source information that consists of the common name, genus-species, tissue, cell, and organelle from which the macromolecule was isolated. Attempts have also been made to include this information for recombinant proteins expressed in a foreign host. The subunit composition and molecular weight are present. This information consists of the total number of subunits and the total molecular weight, and the number and corresponding molecular weights of each type of distinct subunit associated with the macromolecule. A subunit of a biological macromolecule is defined as a part of the C number  and cataa, b, c, α, β, γ), the space group, the number of molecules in the unit cell (Z), and the crystal density. Data entries are also cross referenced to other structural biology databases such as the Brookhaven Protein Data Bank [Each crystal form of a particular biological macromolecule characterized by diffraction studies is described as a crystal entry. The data in each crystal entry includes the crystallization procedure, the crystal data, crystal morphology, and complete references. All of the details required to reproduce the crystallization procedure are data elements of the crystal entry. These elements consist of the crystallization method, the macromolecule concentration, the temperature, the pH, the chemical additives to the growth medium , and the length of time required to produce crystals of a size suitable for diffraction experiments. If the crystallization deviates from standard protocols , an outlata Bank . Each crThe summary information gives the user a convenient mechanism for browsing the BMCD data. The user has access to a complete list of macromolecule names, tabulations of the number of macromolecules and crystal forms for each source, prosthetic group, space group, chemical addition, crystallization method, and a listing of complete references. The panels in Facilities are provided for comprehensive searches of the reference information. References can be queried for matches with a particular author, a key word or phrase. The BMCD also provides a listing of general references concerning all aspects of crystal growth. These references have been divided into categories that include reviews and books, articles concerning procedures, and references concerning nomenclature. Most references include complete titles, and remarks are often added to emphasize important aspects of a reference that may not be evident from the title.As briefly mentioned above, a new function of the BMCD is to serve as the NASA PCG Archive. A complete description of the biological macromolecule crystallization experiments performed in microgravity in the NASA Space Shuttle and in ground control experiments is included in the BMCD. In addition, the BMCD includes data and results from microgravity crystallization experiments sponsored by other international space agencies. The data are organized in a similar way as other entries in the BMCD. The data comprising the PCG Archive mentioned above contains biological macromolecule and crystal entries as described above with additional data items that relate to the microgravity experiment(s). In fact, in the PCG Archive a single crystal form of a macromolecule may have multiple entries, one for each microgravity experiment. Each of these crystallization entries is also given a four character alphanumeric identifier beginning with the letter C. An example of a microgravity crystal entry for a ribonuclease A  homology to a set of macromolecules that have previously been crystallized. The BMCD can be used to tabulate the crystallization conditions for the members of the family, and then this information can be used to limit the parameters for the initial crystallization trials with the biological macromolecule. For example an examination of crystallization conditions for the single-chain acid proteinases listed in The use of the BMCD has been incorporated into general procedures required for the crystallization of a unique biological macromolecule that has never been crystallized , 10. A gVapor diffusion experiments are set up after selecting from 3 to 10 of the reagents that have been most successful at inducing crystallization  for soluOnce the precipitation points have been found, sets of vapor diffusion experiments covering the range of reagents and pH conditions are initiated at both room and coldroom temperatures. Small droplets of protein (5 μL) are mixed with an equal volume of a reservoir solution and placed in a sealed chamber that allows the droplet to equilibrate with 0.5 mL to 1.0 mL of the reservoir solution. At the selected pH, successive experiments are initiated that increase the precipitant concentration to a value greater than that observed in the titration experiment. For example, if a protein precipitates at 50 % saturated ammonium sulfate, experiments that equilibrate droplets against 44 %, 47 %, 50 %, 53 %, and 55 % saturated ammonium sulfate. An equivalent set of vapor diffusion experiments with protein solution containing an effector  can be carried out in parallel.Besides the precipitant  reagents mentioned above, an analysis of the data contained in the BMCD reveals that about 10 % of the soluble proteins crystallize at low ionic strength (< 0.2 M). Thus, the procedure outlined in After initiating the experiments above periodic assessment of the experiments is done. The time for crystals to appear ranges from a few hours to many months, but usually if there is no success within three weeks, further crystallization trials are suggested. For the vapor diffusion experiments, additional reagents should be selected from the BMCD summary and the titration carried out before setting up a new set of experiments. Alternatively, a finer increment in pH and/or other temperatures should be considered. For the microdialysis experiments additional experiments at a finer increment should be carried out. For both the vapor diffusion and microdialysis experiments other effectors or additives should also be considered.If crystals are discovered in either the vapor diffusion or the microdialysis experiments, the parameters that are influencing crystal growth should be optimized. This requires small perturbations in the pH, temperature, and the reagent and protein concentrations. At this point seeding experiments should also be considered to improve crystal size or reproducibility. Even if crystals are large enough for diffraction studies, optimization may be necessary to improve the diffraction quality. If the crystals diffract well, the three-dimensional structure determination can be initiated, but if the crystals do not diffract, or diffract poorly, the search should be continued as mentioned above.The primary goal of the BMCD will continue to be a complete, error-free, and up-to-date crystallization database and NASA PCG Archive. The availability of the BMCD on the Internet will simplify user reporting of errors and distribution of new data. The current version of the software is being expanded to include alternate ways for users to display the data. Because of the BMCD being one of several biological and crystallographic databases on the internet, direct links to related entries in other databases are being considered. Finally, software tools for users engaged in crystal growth studies are being developed that will assist the user in deciding what experiments are appropriate and in designing the experiments for a particular biological macromolecule. These tools incorporate the principles outlined in the discussion of crystallization strategies outlined above.


In order to adapt to societal changes, healthcare systems need to switch from a disease orientation to a patient-centered approach. Virtual patient networks are a promising tool to favor this switch and much can be learned from the open and user innovation literature where the involvement of online user communities in the innovation process is well-documented.The objectives of this study were 1) to describe the use of online communities as a tool to capture and harness innovative ideas of end users or consumers; and 2) to point to the potential value and challenges of these virtual platforms to function as a tool to inform and promote patient-centered care in the context of chronic health conditions.A scoping review was conducted. A total of seven databases were searched for scientific articles published in English between 1995 and 2014. The search strategy was refined through an iterative process.A total of 144 studies were included in the review. Studies were coded inductively according to their research focus to identify groupings of papers. The first set of studies focused on the interplay of factors related to user roles, motivations, and behaviors that shape the innovation process within online communities. Studies of the second set examined the role of firms in online user innovation initiatives, identifying different organizational strategies and challenges. The third set of studies focused on the idea selection process and measures of success with respect to online user innovation initiatives. Finally, the findings from the review are presented in the light of the particularities and challenges discussed in current healthcare research.The present paper highlights the potential of virtual patient communities to inform and promote patient-centered care, describes the key challenges involved in this process, and makes recommendations on how to address them. During the last decades, the world has experienced a demographic and epidemiological transition , charactThe involvement of patients in the healthcare innovation process is increasingly recognized as a main asset to favor this switch. Guided by Omachonu’s and Einspruch’s definition of healthcare innovation, we consider the healthcare innovation process to be concerned with the identification and introduction of new concepts and ideas related to services, processes or products that seek to improve treatment, diagnosis, education, outreach, prevention, and research with the ultimate goal of improving health outcomes, quality, safety, efficiency, and cost-effectiveness ,5. It isIndeed, an emerging body of literature documents the value of peer-led self-management support workshops and their potential to improve health literacy and foster patient empowerment ,8. Two oIn this paper we propose virtual patient networks as a promising tool to favor and promote the active involvement of patients in the innovation process. Health-related online communities have in fact become a main platform for patients to exchange and discuss their health related experiences . HoweverBy reviewing the open and user innovation literature where the active involvement of online user communities in the innovation process is well-documented, we seek to leverage existing evidence from other fields to inform healthcare research and practice. We believe that this evidence can be a main source of information and innovation to foster the development of virtual patient-innovation communities. The idea underlying open innovation is that firms should leverage internal as well as external sources of innovation and commercialization to capture value and maximize economic profit ,31. ReseTaking evidence from the study of these online user innovation communities as a starting point, we focuses on the role of virtual patient networks for people with chronic health conditions to foster the integration of the patient’s perspective into the healthcare innovation process. The objectives of this paper are to 1) describe the use of online communities as a tool to capture and harness innovative ideas of end users or consumers in different fields; and 2) point to the potential value and challenges of these virtual platforms to function as a tool to inform and promote patient-centered care in the context of chronic health conditions. The paper proceeds as follows: In the next section, we present our methodological approach, describing our search strategy and analytical procedure. In the results section, we address our first research objective by providing a narrative account of the included studies. The discussion section of the paper then addresses our second research objective by discussing the results of the scoping review in the light of the particularities present in the healthcare context. In the discussion section, we also draw attention to promising directions for future research and acknowledge the limitations of our study. We conclude by highlighting the contribution of our work and its implications for healthcare research and practice.In seeking to link the available evidence from the open and user innovation literature to the healthcare context, we expected a highly heterogeneous and methodologically diverse body of literature . As prioIn a first step, the research team identified a number of questions based on a preliminary review of the literature, which resulted in two central research questions: What do we know about the use of online communities as a tool to capture user insights in other fields? And to what extent are these findings transferable to the healthcare context? We then proceeded to the identification of relevant studies. The literature search was performed through the databases ISI Web of Science, Communication & Mass Media Complete, PsycINFO, Business Source Premier, MEDLINE, ABI Inform Global, and ABI Trade & Industry. These databases were chosen to cover a broad range of social sciences content. The search strategy was developed based on a preliminary literature review, one lead to the identification of terms relating to open and user innovation in online communities. The full search strategy is described in more detail in the review protocol see: . ArticleAfter de-duplication, the remaining records (n = 11868) were converted to RIS format and imported into EPPI reviewer 4, a web-based software tool for research synthesis. In a next step, the research team proceeded to select relevant studies. Inclusion and exclusion criteria were refined through an iterative process comprised of three levels of relevance screening. The first level of screening comprised a review of title and abstract to eliminate items that did not meet our inclusion criteria. Two coders were involved in the screening process. Inter-coder reliability was assessed for 20% of the records and yielded 90% correspondence. 8754 records were discarded at this first stage. Upon completion of a second round of title and abstract screening, 426 articles were retrieved for full text screening. To be eligible for the review, studies had to be published in English between 1995 and 2014 and describe the user innovation or co-creation process taking place within one or more firm-hosted, firm-related or independent online user innovation communities. For the purpose of this study, we defined online user innovation community as a virtual gathering of end users engaging in product and service development related tasks and activities, such as proposing ideas, evaluating and commenting others’ ideas for product or service improvement, or prototyping. Studies were excluded if they examined online communities that did not involve user innovation activities (e.g. dating communities) or that did not focus on end-user (consumer) involvement but rather on employee or supplier integration. Studies reporting on offline initiatives to integrate consumers in the innovation process, studies investigating innovation intermediaries or crowdfunding practices were also excluded.Once we had identified and selected relevant studies, data-charting form was developed by the research team to determine which data to extract. As described by Arksey and O’Malley , we usedA total of 144 original studies were included . There hMost of the included studies addressed user innovation initiatives in the private industry –48 with The first category included articles focusing on factors related to the individual users of the platform and the community as a whole. In this context, the studies examined the interplay of different aspects related to individual and community roles, behaviors, and interaction that contribute to the functioning of an open innovation community. In this section, we present findings related to user motivations and factors influencing users’ contribution behavior in virtual innovation communities as well as findings concerning user interaction and the structure of virtual communities.For the purpose of this review, we are guided by the distinction as proposed by Raasch and von Hippel, who refer to output-related and process-related motivations . Output-The papers included in this review addressed motivational factors in a variety of different settings ranging from open source software development –75 to coThe key inference drawn from this section was that user motivations in virtual communities are complex, heterogeneous, and highly context dependent. The reviewed literature did not provide clear directions with respect to the importance or significance of individual motivational factors but rather highlighted a number of aspects that shape user contribution behavior in virtual innovation communities.Studies also examined different user characteristics and roles, investigating ways to identify different user types (such as user innovators and community leaders) based on contribution behavior and roles taken on within the community ,106–109.Further, studies shed light on the knowledge creation process by identifying different kinds of contributions and contribution activities as well as their specific role in the value creation process ,112–115.Findings from the studies reviewed in this section drew attention to the fact that virtual innovation communities are usually run and dominated by a small, active core community of self-selected individuals. Contrary to clinical research, authors did not view self-selection as a bias in this context but rather an advantage, acknowledging that not all users will make valuable contributions. The findings presented in this section also highlighted the importance of interaction between users to enable knowledge creation, for example in form of product-related discussions. Finally, they also demonstrated how firms can adopt an active role in this process by moderating and triggering discussions.The second category comprised studies focusing on aspects related to firms’ strategy, such as the design and functionalities of the open innovation platform ,90,133, One of the key findings was that only a small percentage of initiatives aiming at eliciting user contributions managed to do so successfully , as manyThe studies reviewed investigated a variety of different practices for involving end users in the innovation process including toolkits for user innovation ,143–145,Studies reviewed in this section focused on managerial challenges encountered when collaborating with virtual user communities, acknowledging that the integration of end users in the innovation process is a resource intensive endeavor that requires a clear strategy. Findings presented in this section also pointed to different forms and opportunities to integrate virtual user communities and highlighted the importance of ensuring a win-win situation between the virtual community and firm.For the purpose of this study, we refer to output as the ideas generated by the user community, whereby we consider outcomes to be direct and indirect market outcomes related to the implementation of these ideas. Studies in this third category investigated processes related to idea selection, implementation, and the resulting market outcomes.Regarding idea selection, we found that there were various modes and tools for selecting ideas, such as involving users or expert juries in the rating and ranking process ,165,166,Most of the included studies highlighted the success of open innovation communities in terms of their ability to generate valuable ideas and solutions. Authors, for example, compared the ideas generated within an online community to those resulting from traditional forms of market research showing that the use of social media tools for ideation led to more and higher quality ideas at a lower cost per idea in a shorter period of time, as compared to traditional forms of market research ,141. OnlFindings from the studies reviewed in this last section showed that online communities can in fact be an appropriate tool to help firms generate, identify, and collect end user ideas, when used appropriately. Authors also illustrated different approaches for identifying and selecting promising ideas generated by online communities. The reviewed studies further showed that the involvement of virtual communities can in fact positively affect firm performance with respect to the product and service innovation process but also with respect to customer relations.By reviewing the open and user innovation literature, this review described the use of online communities as a tool to capture and harness innovative ideas developed and shared by end users in different fields. We now seek to link these findings back to the healthcare context, presenting them in the light of the particularities and challenges discussed in current healthcare research. We thereby aim to point to the potential value and challenges of these virtual platforms to function as a tool to inform and promote patient-centered care in the context of chronic health conditions. In order structure this section in a meaningful and pragmatic way, we were guided by the innovation management literature, describing models of product and service innovation and knowledge integration as stage processes ,184, in Setting the stage for online user innovation initiativesAttracting and maintaining an active user communitySelecting and absorbing ideasProcess and outcome evaluationOur findings suggest that the operation of open innovation communities is highly context dependent ,142,187.Before investing in an online platform, it is, first of all, important to consider the significant amount of resources required to initiate, maintain and harness an open innovation community ,138. SecThe first point is best illustrated by a concrete example. Taking a hospital for instance, we expect different actor groups such as nurses, surgeons, hospital managers and patients to have different priorities, expectations, and perceptions related to the healthcare process ,196,200.The second challenge, the dominance of the top-down, paternalistic approach to care, is very closely related to the first one. We assume that there is an imbalance in power due to the dominance of the top-down approach in healthcare –198, whiHaving addressed the challenges related to the different actors in the healthcare setting, we now turn to the third challenge: the lack of best practice approaches. Despite successful examples from other fields ,152, litDrawing parallels to the healthcare setting we observe similar patterns. Departing from the paternalistic ‘closed’ approach of healthcare, patients have started to develop their own solutions to health problems they are facing. Since pursuing legal action against these patients is not a viable option, healthcare providers will have two options: a) Shut patients out and exclude them from the healthcare process taking the risk that they will create their own solutions anyways, or b) Include patients in the healthcare process and make them partners. The rising trend of shared decision-making and patient empowerment  suggestsSo far we have discussed issues related to setting the stage for online user innovation. We now shed light on the challenges related to attracting and maintaining an active user community. From the literature reviewed in this study we identified five key elements that are crucial to the initiation and maintenance of an open innovation platform.Users willingness to share their innovative ideas ,70,136Attractiveness and usability of the platform ,90,133Motivations ,90,95,96Interaction and Involvement ,137,172Partnership between user community and firm ,164,165When looking at the literature on health-related online communities, we find that the evidence on attracting and maintaining active user communities is scarce. However, authors have shown that patients are not only willing to share their personal health information for research ,213, butRelated to the second element, we find that in terms of the platform technology, reliable technology seems to be preferable over state-of-the-art technology . This asWith respect to the third element, there is very limited understanding of patients’ motivations to share their experiences online. This is partially due to the fact that online communities are frequently designed as an intervention tool tested on randomly assigning participants that were recruited for the study with the purpose of assessing health outcomes . This, hWhen addressing the fourth aspect in the healthcare setting, we encounter a key challenge that did not emerge from the studies included in the review: the potential danger resulting from incorrect, incomplete, or simply misinterpreted information and ideas. To illustrate this with an example, imagine a company collaborates with a user community to develop a new MP3 player. Of course the company aims for quality solutions and ideas that will lead to a profitable product, but not all ideas will be good. Even though these bad ideas may prolong the co-creation and idea selection process, there is no harm in bad ideas (as long as they are not implemented). On the contrary, if a person with Type 1 diabetes does not recognize bad ideas as such and implements them, this may pose a serious threat to his or her health. Some patients, particularly those suffering from rare, chronic, or life-threating conditions with no available standard treatments may be desperate for solutions and may be willing to try just about anything –223. ThiWe now turn to the last aspect, the practice of treating users as partners in the innovation process. In the healthcare setting, we propose that despite keeping a certain level of control over community activities, it is important to treat community members as a partners. In other words, they should be recognized as knowledge resources rather than data sources, or passive receivers of care. As pointed out in the introduction of this paper, the lived-experience of people with a chronic condition constitutes an invaluable resource for research and practice. Despite the obvious advantages an online patient community presents for research, such as facilitated data collection and patient recruitment , their aIn the previous section we discussed issues related to the idea generation process. But once ideas are being generated, how can we identify the most viable ones? Findings of our review highlighted various modes and tools for selecting user-generated ideas; however there is no consensus with respect to their effectiveness and efficiency in identifying the most promising ideas ,165,168.When discussing the process of idea selection in the healthcare context, issues related to idea quality, that were discussed previously, lead us to the conclusion that in addition to the aspects highlighted by the open and user innovation literature, such as novelty and high customer benefit , componeMost of the included studies highlighted the success of open innovation communities in terms of their ability to generate valuable ideas and solutions. Whether an online user innovation initiative was considered as successful or not was usually measured by the quantity, diversity, and quality of ideas generated ,129,173.The present paper contributes to the growing body of literature on virtual patient communities, highlighting some of the key aspects relevant to the integration of these communities in the healthcare innovation process. However, some of the limitations should be acknowledged when interpreting our findings. Firstly, due to the heterogeneity of the field in terms of terminology our search strategy, particularly our selection of key words, may have failed to identify some relevant studies. We tried to minimize this risk by conducting a preliminary review of the literature to inform our key word selection and choice of databases. Secondly, the diversity of the included studies with respect to their theoretical framework, study design, and research focus proved to be challenging when seeking to structure the evidence in a meaningful way. We chose to present our findings by grouping studies according to their research focus; however, evidence could have, for example, also been structured according to the type of online community investigated (firm-hosted vs. firm-related vs. independent). Thirdly, we did not account for the quality of the included publications, nor did we account for reporting or publication bias.Despite evidence suggesting that there is great potential in harnessing the knowledge and lived-experience of patients and their carers ,198,214,In the light of the shift toward patient-driven care, efficient strategies for integrating the ideas and experiences of patients into healthcare research and practice will become crucial. Our findings suggest that much can be learned from online user innovation practices employed in other fields and industries. Future research should further explore these areas and identify strategies to harness the innovative potential of online communities in the healthcare context. It will further be important to identify indicators of success and best practice approaches which will ultimately support healthcare institutions in adopting online communities as a means to integrate patients in the healthcare innovation process.S1 PRISMA Checklist(PDF)Click here for additional data file.S1 Protocol(PDF)Click here for additional data file.S1 Table(PDF)Click here for additional data file.


The first and last occurrences of hexapod families in the fossil record are compiled from publications up to end-2009. The major features of these data are compared with those of previous datasets (1993 and 1994). About a third of families (>400) are new to the fossil record since 1994, over half of the earlier, existing families have experienced changes in their known stratigraphic range and only about ten percent have unchanged ranges. Despite these significant additions to knowledge, the broad pattern of described richness through time remains similar, with described richness increasing steadily through geological history and a shift in dominant taxa, from Palaeoptera and Polyneoptera to Paraneoptera and Holometabola, after the Palaeozoic. However, after detrending, described richness is not well correlated with the earlier datasets, indicating significant changes in shorter-term patterns. There is reduced Palaeozoic richness, peaking at a different time, and a less pronounced Permian decline. A pronounced Triassic peak and decline is shown, and the plateau from the mid Early Cretaceous to the end of the period remains, albeit at substantially higher richness compared to earlier datasets. Origination and extinction rates are broadly similar to before, with a broad decline in both through time but episodic peaks, including end-Permian turnover. Origination more consistently exceeds extinction compared to previous datasets and exceptions are mainly in the Palaeozoic. These changes suggest that some inferences about causal mechanisms in insect macroevolution are likely to differ as well. A key contribution of palaeontology to the study of the diversity of life has been the elucidation of macroevolutionary patterns and processes through deep time, with fossils providing the only direct temporal evidence of how life has responded to a variety of biotic and abiotic forces –4. If thIn addition to such strategic questions, palaeontological data can help solve many basic questions of perennial interest. Comprising over 50% of described species , hexapodLagerstätten taxa) are likely to have substantially incomplete fossil records where the potential for change is much greater.Palaeodiversity data are typically compiled in the form of taxonomic databases of fossils that provide either temporal ranges or discrete occurrence data. Commonly, criticisms of such databases focus around the integrity of the data and its resilience to the addition of further information . SubstanLagerstätten) or stages where insect-bearing fossil deposits are poorly known and so were cautious with any such interpretations. Jarzembowski and Ross )et al. )Despite major changes to the ranges of insect families over fifteen years of discovery, changes to the pattern of described richness through time derived from those data seem less extensive. Correlations between the time series of the new and previous datasets show that the broad pattern of rise in discovered taxa through time is very similar to that previously described. The generally steady rise in richness through time suggests support for the previous conclusion  that no Other important features preserved in the NEW richness series include evidence for a mass extinction at the end-Permian. The Permian drop in richness is however less abrupt than in FR2. This effect is probably due to the improved temporal resolution from epoch to stage, which pulls the ranges of taxa in FR2 forward to the end of the Permian. At stage level resolution, many of these families are instead seen to have last occurrences before the end Permian. In turn, the asynchronicity in extinction may be genuine, but probably is partly an artefact of an incomplete record (the Signor-Lipps effect ) which tDespite the evidence for an end-Permian extinction, the NEW richness data leave no evidence of an end-Cretaceous extinction, in common with previous data ,53. GiveAlthough all datasets show an increase in richness in the Triassic, a subsequent drop is suggested by the NEW RT series . Many noSurprisingly, the overall level of richness in the NEW data is not always higher than the older data. This is mostly the case in the Palaeozoic, where there was an historical tendency by early workers such as Handlirsch and Tillyard to oversplit taxa, while revisions have decreased the number of valid families. Additionally, and perhaps more importantly, of the 324 families in the new data with ranges in the Palaeozoic, 28% of them represent contractions with respect to LAB. This suggests a specific effect of taxonomy on apparent richness that may be important for other researchers.The correlations between the differenced time series for the new and old data, although positive, are much less strong than for the raw time series, suggesting moderate differences in the shorter term variation in richness from stage to stage. This is potentially important when assessing the drivers behind diversity change, as time series are generally detrended to remove spurious correlations, and it is the short term variation around the long term trends that are analysed e.g. ,61). The. The61])Labandeira  picks ouper capita rates , althouIn summary, a new compilation of the fossil ranges of insect families shows changes in the ranges of a high proportion of families, and significant changes in described short term richness and some origination and extinction patterns, but little change in broad temporal patterns. Representing an additional 15 years of data in a rapidly expanding field compared with previously available compendia ,14, it iS1 Appendix(PDF)Click here for additional data file.S1 Dataset(XLSX)Click here for additional data file.


The ‘Sponsoring National Processes for Evidence-Informed Policy Making in the Health Sector of Developing Countries’ program was launched by the Alliance for Health Policy and Systems Research, WHO, in July 2008. The program aimed to catalyse the use of evidence generated through health policy and systems research in policymaking processes through (1) promoting researchers and policy advocates to present their evidence in a manner that is easy for policymakers to understand and use, (2) creating mechanisms to spur the demand for and application of research evidence in policymaking, and (3) increased interaction between researchers, policy advocates, and policymakers. Grants ran for three years and five projects were supported in Argentina, Bangladesh, Cameroon, Nigeria and Zambia. This paper seeks to understand why projects in some settings were perceived by the key stakeholders involved to have made progress towards their goals, whereas others were perceived to have not done so well. Additionally, by comparing experiences across five countries, we seek to illustrate general learnings to inform future evidence-to-policy efforts in low- and middle-income countries.J Health Serv Res Policy 8(2):94–9, 2003) as a framing device to reflect on project experiences across the five cases. Using data from the projects’ external evaluation reports, which included information from semi-structured interviews and quantitative evaluation surveys of those involved in projects, and supplemented by information from the projects’ individual technical reports, we applied the theoretical framework with a partially grounded approach to analyse each of the cases and make comparisons.We adopted the theory of knowledge translation developed by Jacobson et al. (There was wide variation across projects in the type of activities carried out as well as their intensity. Based on our findings, we can conclude that projects perceived as having made progress towards their goals were characterized by the coming together of a number of domains identified by the theory. The domains of Jacobson’s theoretical framework, initially developed for high-income settings, are of relevance to the low- and middle-income country context, but may need modification to be fully applicable to these settings. Specifically, the relative fragility of institutions and the concomitantly more significant role of individual leaders point to the need to look at leadership as an additional domain influencing the evidence-to-policy process. A heightened awareness of the mismatch between our ever increasing wants and scarce resources has brought to the forefront questions of resource allocation and priority setting, and the need to justify decisions taken at every level of the health system. This is all the more important in low- and middle-income countries (LMICs) where there are major health challenges accompanied by limited resources to address these. It is against this background that the use of evidence-based and evidence-informed decision-making have gained ground, both in the field of medicine and, more recently, in health policymaking .However, clinical decision-making is a fundamentally different process from policymaking. Clinical medicine, steeped in the positivist paradigm, seeks evidence of generalizable cause and effect relationships, which are universally applicable as the basis of decision-making. However, evidence is only one of the many inputs that policymakers consider. Policymaking is complex and context dependent, influenced by ethical values, interest group and party politics, as well as social and economic factors. Recognizing this complexity, evidence-informed policymaking has been described as an approach that seeks to ‘ensure that decision-making is influenced by the best available research evidence’, even while it acknowledges the multiplicity of factors that influence policymaking .This paper reflects on the practical experience of a multi-country program  programme) aimed at enhancing evidence-informed policymaking in five LMICs supported by the Alliance for Health Policy and Systems Research (AHPSR), an international partnership based within WHO. Over the past decade, WHO has played an important role in promoting evidence-informed policymaking, from the 2004 Ministerial Summit on Health Research, held in Mexico , to the The approaches used in this programme included (1) encouraging researchers and policy advocates to present their evidence succinctly and vividly, (2) increasing interaction between researchers, policy advocates and policymakers, and (3) strengthening the capacity of Ministries of Health (MOHs) to demand and utilize evidence. While reflections on single projects as well as on specific components of this program have been previously published -9, this There are a number of models that seek to explain the incorporation of research evidence into decision-making. These include (1) ‘push’ models, which emphasize the role of producers of research to provide information to decision-makers, (2) ‘pull models’, which give primacy to efforts by research users such as decision-makers to ‘extract information’ from the world of research, (3) ‘exchange models’, which prioritize the creation of linkages or partnerships between producers and users of research, and (4) integrated approaches combining elements of the three aforementioned approaches that make use of knowledge translation platforms that bring together research consumers and producers and encourage both push and pull efforts towards clearly defined goals, often established through priority setting mechanisms . Given, There exists a significant literature largely centred around high-income countries that examines the use of research evidence in the health policymaking process in an attempt to identify factors influencing the incorporation of evidence into decision-making. These include systematic reviews by Innvaer et al.  and by OThere is also a growing literature in this area from LMICs. Evidence-informed decision-making faces additional hurdles in these settings. Reflecting funding availability and institutional research capacity, there is often a paucity of locally relevant research evidence available to inform decision-making. This is often accompanied by a shortage of skilled human resources in MOHs to demand, evaluate, synthesize and adapt available research evidence. Frequent transfers of officials within MOHs, something that is not unusual in these settings, further exacerbates this problem. Examples of studies based in LMICs include a literature review and case study on Thailand by Sauerborn et al. , a studyhttp://www.who.int/alliance-hpsr/projects/national_processes/en/). The second section discusses our preferred theoretical framework, whereas the third describes data sources and methods used to analyse the data. Section four presents the findings and examines these in terms of the theoretical framework. The final section discusses the implications of our study in terms of lessons learned for evidence-informed policymaking for health systems in LMICs.The remainder of this paper is divided into five parts. We begin with a description of projects carried out under the AHPSR’s SNP program  promoting researchers and policy advocates to present their evidence in a manner that is easy for policymakers to understand and use, (2) creating mechanisms to spur the demand for and application of research evidence in policymaking, and (3) increased interaction between researchers, policy advocates and policymakers. A number of strategies were suggested including the creation of platforms to produce and communicate research to policymakers in an accessible manner, training policymakers and establishing units within MOHs to strengthen MOH capacity to demand and use research evidence, developing policy briefs (defined as documents that bring together the best available evidence on a problem and viable solutions to address the problem) , and sup promotinThe SNP program is thus best viewed as a joint effort involving national stakeholders (including researchers and former policymakers) who designed and implemented individual projects, and an international collaboration, namely the AHPSR, which conceptualized and funded this program as well as mandated the evaluation carried out by a team of international researchers.We use the theory of knowledge translation developed by Jacobson et al.  as a fraJacobson et al.  give greA document review of reports produced by the AHPSR SNP program was conducted at the outset. The projects’ formal evaluation report was the main source of data for this paper . This waThe project evaluation report was produced by two AHPSR appointed external researchers who between them carried out the evaluation across the five projects. Country visits were carried out to monitor project progress, review documents and hold interviews with project leaders, team members, and selected decision-makers, as discussed below. Visits were made to all project sites except Argentina, where this was difficult due to logistic reasons. Additionally, five meetings were held with project teams taking advantage of other events including SURE  meetings as well as the McMaster Health Forum policy dialogues . The evaSemi-structured interview guides were developed by the Knowledge Translation Platform Evaluation Team at McMaster University . InitialThe evaluation report also included results from surveys on (1) policy briefs produced, (2) policy dialogues held, and (3) project outcomes. Potential respondents for these surveys were identified by project teams based on their likely involvement in policymaking around issues that were being examined by the project teams. Respondents included national and sub-national level policymakers, managers at healthcare institutions, staff members of donor agencies, non-governmental organizations, or professional health associations and researchers at universities and other research institutions ,24.For the policy brief survey, identified individuals were posted a package containing policy briefs prepared by the country projects and an invitation to take part in a policy dialogue. Along with the brief and invitation, respondents were provided with a questionnaire to assess the policy briefs that they were required to complete prior to attending the policy dialogue. The questionnaire included questions on respondent perceptions of the process by which the brief was designed and developed as well as an overall assessment of the policy brief itself examined through a 7-point Likert scale ,24. A toImmediately after the policy dialogue, participants were provided with a questionnaire to assess the policy dialogue process and were required to complete and return the questionnaire on the spot . The queFinally, for both the Nigeria and Zambia projects, the evaluation report included results from an outcomes evaluation survey conducted after more than 2 years of project implementation to examine views of these same groups of stakeholders on (1) research availability for issues with high priority in health policy, (2) relationships between researchers, government officials and other policymakers, and (3) the capacity of both researchers and policymakers to promote evidence-informed policymaking in the health sector. The outcomes evaluation survey contained a mix of questions, some of which used a 7-point Likert scale while others were open ended questions. The target sample size was up to 50 respondents in each setting, the final sample size was 66 respondents in Nigeria and 48 respondents in Zambia .Information from the evaluation report was supplemented by findings from individual final project technical reports submitted by project teams to the AHPSR -32. ThesUsing the sources above, we first summarized information on each project in terms of outputs and activities, respondents’ perceptions of policy briefs and policy dialogues, and their perceptions of outcomes, enabling us to categorize projects according to their performance in (1) producing policy relevant evidence, (2) fostering interactions between policymakers and researchers, and (3) building capacity of policymakers to demand and use research.We then applied the theoretical framework within a partially grounded approach to analyse each of the five cases. This involved coding the data from each of the five projects in terms of the chosen framework as well as allowing for new categories to emerge from the data in order to not force the data to fit the theory . We emplInformation on project areas and activities is provided in Table The research team in Argentina appears to have focused its attention solely on the production of policy briefs and holding policy dialogues with little evidence of sustained advocacy and dissemination efforts. There was no information on the impact of the policy briefs or dialogues on the policymaking process nor any evidence of the creation of structural-links across researchers and policymakers or of capacity-creation or strengthening efforts ,25,30.had a huge impact on specific decisions taken by policymakers at the national level” by the project leader during the semi-structured interview [In Bangladesh, policy briefs regarding the use of the H1N1 vaccine and non-communicable diseases were assessed to have “nterview ,31. The nterview ,31.In the Zambia case, while the impact of the policy briefs on policymaking itself remains unclear, the great majority of respondents felt that the policy briefs and dialogue both achieved their desired objectives Tables  and 4. TTwo of the policy briefs produced by the Cameroon project, on community health insurance and on scaling up malaria control in the country, were asserted by the project leader to have influenced national policies in these areas . RespondOver 85% of respondents felt that the policy briefs and dialogues undertaken as part of the Nigerian project had largely achieved their goals Tables  and 5. TFrom the above information and Table Next, we used the theoretical framework to frame team experiences and explain variations in project performance. We recognize that the relative importance of each of the five domains is context specific and will vary across settings, and that interactions among the domains are critical. For example, interested user-groups (domain: characteristic of the user-group), are more likely to be concerned about the research produced on the area of work that interests them and would thus tend to commission research that is relevant to them (domain: issue characteristic). They would also tend to make clear to the researcher the type of output that they would find useful in their work (domain: characteristic of dissemination strategy). Conversely, in the presence of an indifferent user group, researchers are more likely to pick issues that they think important and produce outputs that they deem relevant; a situation that is far less likely to lead to evidence incorporation into decision-making. Table The findings from Argentina clearly illustrate the importance of the coming together of domains. While we lack much information on user-group perceptions of research and researchers, the project technical report explicitly mentioned the difficulty in getting together experts and policymakers to hold policy dialogues as an obstacle to the projects’ functioning, suggesting a generally low level of policymaker enthusiasm on the topic. Additionally, the groups had never worked together before and there was a frequent turnover of MOH staff, which made the establishment of relationships between researchers and policymakers difficult. The need for the project to actively work to ‘spur political interest’ in the issue of provincial health insurance (the main focus of the researchers’ effort), as mentioned in the project technical report, suggests that this was not an issue that was on the policy agenda at the time . ResearcThe experience from Bangladesh, on the other hand, demonstrates how critical perceived relevance of research on a particular issue at a particular instance of time is to explaining the success of the evidence-to-policy process. While we lack quantitative information on user group perceptions, data from the merit review indicates that policymakers were keen to increase their interactions with researchers. In addition to forming the basis of a trusted relationship, the pre-existing links between an established research institution International Centre for Diarrhoeal Disease Research, Bangladesh and health policymakers in Bangladesh facilitated the latter’s early involvement in the project and potentially played an important role in enabling the project to overcome the challenge posed by frequent staff changes in the MOH . It alsoUser groups in Zambia tended to have a positive view of research and researchers, with 97% of respondents mentioning that the use of research evidence was extremely beneficial to their work. An overwhelming 86% of the respondents also felt that they were expected to use research evidence in their work Table . In commIn common with the information from other African countries, members of user groups in Cameroon appeared to have experience in carrying out research, with 67% of respondents who did not describe themselves as researchers indicating that they had research experience. The project’s activities were facilitated by pre-existing institutional structures, being based at a research institution that had been set up to promote the cause of evidence-informed policymaking in Cameroon’s health sector . This stThe Nigeria project followed a similar pattern to Cameroon. User groups wanted to and were expected to use research evidence, with 98% and 87% of respondents, respectively, strongly agreeing with these assertions Table . In commReflecting on our experiences from these AHPSR-supported projects in five countries across three continents, we can conclude that the successful incorporation of evidence into policymaking is indeed greatly aided by the coming together of a number of domains . The comThe results of this study are supported by findings from a systematic review by Innvaer et al.  that ideHowever, we caution against an understanding of the evidence-to-policy process as a simple coming together of domains. First, as explained above, the domains have the potential to reinforce each other and reflect an interactive spectrum where the whole can be greater or lesser than the sum of the parts. The Argentinian case of apparent user group indifference, apparently weaker pre-existing user group-researcher relationships, and lack of issue salience lies at one end of the spectrum. The Nigerian project, with political interest and involvement, systematic priority-setting exercises and capacity building initiatives, represents the other. Second, the relative contribution of each of these domains varied greatly across the different projects suggesting the importance of other contextual factors in shaping observed effects. For example, it would seem that, in Bangladesh, the issue, including the urgency surrounding it, its technical nature and potential to affect key interest groups, determined whether or not evidence was considered in policy around that issue. On the other hand, in the Zambian experience, the project’s ability to reach out to policymakers determined whether or not research evidence was considered in policymaking around the issue.Additionally, in line with Harries et al. , we findThese findings suggest that, though relevant to a large extent, Jacobson’s theoretical framework, initially developed for high-income settings, may need modification to be applicable to LMICs. The relative fragility of institutions and concomitantly more significant role of individual leaders points to the need to look at leadership as an additional domain influencing the evidence-to-policy process.The evidence from Nigeria, Zambia and Cameroon (where we have this information) also suggests that the separation between the ‘two communities’ of researchers and policymakers discussed above in the theoretical framework  is not aFinally, the findings from this study point to the need for comprehensive mechanisms to facilitate the incorporation of evidence in policymaking and overcome what has been termed as the false dichotomy of the ‘know-do’ gap in health policy and systems research . EmbeddiThere are some limitations to this study. The most significant being that the paper is based primarily on the perceptions of stakeholders including researchers, policymakers, and a few civil society and development partner groups who were in turn identified by the country project teams. It is these perceptions as opposed to well-defined policy outcomes that have been used to assess the role of evidence in informing health policymaking. Additionally, individuals identified by the project teams may feel a need to report positively about project processes and outcomes, which is a potential source of bias in the data. The inclusion of a wider range of stakeholders, such as the media as well as politicians, in the sample would have further strengthened and potentially enhanced the generalizability of the findings, in addition to removing this potential conflict of interest. Second, data availability varied by project. Having identical data sources across all the projects would have enhanced our ability to draw inferences by directly comparing results across sites. Third, the project evaluation was led by a team of two outside researchers from major universities in the Middle East and Latin America. While, on the one hand, this reduces the chance of biased analysis and reporting, on the other, it has the potential to lead to a loss of insider perspective that critical self-evaluation can provide and that an outsider may not be able to access through the interview process. Notwithstanding these limitations, we do believe that this study, by comparing the five countries as cases, has enabled us to at least answer some of the ‘why’ questions behind each project’s achievements, to contribute to further build on the chosen theoretical framework and significantly inform efforts towards evidence-informed health policy in LMICs.


David Hubel Fig.  was bornAfter graduation and apprenticeship, he took 2 years residency in neurology. Already in Canada, under the direction of Herbert Jasper, he developed his interests in neurology. In 1953, he received American, while simultaneously retaining Canadian citizenship. In 1954 he moved to America, where in Baltimore he began residency in neurology at the John Hopkins University School of Medicine.In 1955, he was drafted into the military service, where fortunately he could devote himself to scientific research. He started working at the Walter Reed Army Medical Center in Washington. Both, his specialization and the long-time interest in electronics drove him to conduct research on the nervous system. Initially he engaged in research on the cord, than the cerebral cortex. Together with a team of neurophysiologists he conducted studies on the visual cortex in sleeping and awakened cats. Hubel invented the modern tungsten microelectrode, supported by a hydraulic control unit that allows detection of signals from that area. Hubel noticed that various light signals stimulate individual neurons whereas others are left inactivated. In 1958, Hubel returned to John Hopkins where he began working with a team under the leadership of Stephen Kuffer 1913–1980). There he had begun a close cooperation with Torsten Wiesel, with whom he discovered selectivity and columnar organization in striate cortex. The research was conducted again placing a microelectrode in the striate cortex of cats. The animals were subjected to subsequent images, light, and dark. Individual neurons responded differently to diverse light intensity and graphic elements such as rectangles, with various angles of inclination. Some fired rapidly with lines at one angle while others were activated with light at a different one. Different images caused the neurons activity to change. These findings led Hubel and Wiesel to hierarchize striate cortex cells , 3.–1980. ThIn 1959, a publication authored by Hubel and Wiesel appeared on this subject . It showEffective cortical stimulation depends on the size and shape of light stimuli—in accordance with the distribution of excitatory and inhibitory regions within the receptive area of the retina. Hubel and Wiesel also noted that the intensity of cortical response also depended on the direction of the light source movement. They examined 45 individuals within the cortex—corresponding to the respective regions in the retina—36 were driven from only one eye, 15 from the ipsilateral eye, 21 from the contralateral and the remaining 9 from the two eyes independently. Thy eyes of some cats were equally sensitive; in others the dominance of one eye over the other was noted. The fields within the striate cortex, which were sensitive to the stimuli from both eyes, corresponded to the receptive fields with a very similar construction in both eyes. Also their location in the retina was coherent as to the shape, size and the direction of the movement of light [Neurons which received signals that was strictly determined as to the shape and direction of movement were called by Hubel and Wiesel, simple cells; those which were more versatile, complex cells.This way the scientists, step by step, made the discovery that the striate cortex is organized in several columns, specialized in processing light signals. As a result the visual cortex became the first mapped area of the brain.In 1959, Kuffer along with the entire team composed of nine families including that of Hubel, moved to Harvard University in Cambridge. The group closely collaborated at Harvard, and after 5 years created a separate Department of Neurobiology where they continued research and once again undertook the notion of vision. After suturing an eye of a kitten it turned out that the eye turned blind as a result of lack of striate cortex stimulation. On the other hand they demonstrated the flexibility of the cortex where the temporary closure of one eye permanently shifts the ocular dominance of neurons in the striate cortex to the eye that remains open. This led them to the conclusion on the importance of light impulses on the development of visual cortex. This also led to the discovery of the ocular dominance columns within the visual cortex. In addition, the dominance of one eye over the other proved normal. They also discovered that covering one eye causes lack of proper development of the areas responsible for binocular vision. These studies have altered the way ophthalmologist perceive the most beneficial time for surgical treatment of eye diseases such as congenial cataract and strabismus. Prior to Hubel and Wiesel’s discovery, the decision on the surgical treatment of these conditions was postponed; contemporarily surgical treatment is done in early childhood [Since 1965, Hubel worked as a Professor of Physiology and since 1968 as a Professor in Neurobiology at the Harvard University. In 1979 he was awarded the Dickson Prize. In 1981, together with T. Wiesel, he was awarded the NN “for their discoveries concerning information processing in the visual system”. At Harvard, Hubel held lectures until his retirement in January 2013. He was also the laureate of 12 honorary doctorates.David Hubel was married to Shirley Hubel and they were an extremely harmonious marriage. They had three sons. Hubel was a man of many interests, he knew several languages, he was interested in music, he played the piano and the flute. He enjoyed pottering about how he conducted astronomical observations, played tennis and squash and liked skiing.David Hubel was an extremely liked and sociable person who was characterized by great sense of humor.David Hubel died of kidney failure on September 22, 2013 in Lincoln, MA.


The issue of collaboration in medical education is becoming prominent. Some faculty development programs have suggested an approach for promoting collaboration on a global level. However, non-English-speaking developing countries in Asia, especially in Southeast Asia, do not take advantage of them due to their unique context, such as language and culture. To address these issues, Seoul National University College of Medicine initiated a 6-week international faculty development program called the “Seoul Intensive Course for Medical Educators” for 16 fellows from five Asian countries . The aim of this study is to report the evaluation results of the outcome of the program and discuss better ways of collaborating with developing countries.Three levels of collaboration—intraorganizational, intranational, and international—were central initiatives of the program. Prior to setting up the program details, we first established four design principles, following which the contents, materials, and facilitators were determined to maintain consistency with the design principles. The evaluation of the program was done with Kirkpatrick’s four-level model. Most of the evaluation data for level 1 were collected by two questionnaires, the post-module survey and the post-program survey. Portfolios and progress reports were mainly used to collect outcome data for levels 2 and 3, respectively.The reaction was generally positive throughout the program and there was a significant correlation between satisfaction and relevance to one’s job or needs. Despite the fellows’ propensity for overestimating themselves, both the evaluators and fellows reported that there was significant improvement in learning. Opinions on the impact or urgency of the topics were slightly different from country to country; however, the answers regarding feasibility were fairly similar. Moreover, we could observe from the post-program progress reports that the transfer of learning was actively in progress, mainly for topics that were highly feasible.These results show that the program was successful in terms of its effectiveness. Consistent and timely support is essential for the sustainable development of the medical education systems in these countries. Further understanding of the underlying factors on transfer (level 3) could improve the effectiveness of faculty development programs for developing countries. The issue of collaboration in medical education is becoming prominent . The speDespite growing trends for international collaboration, developing countries are still experiencing hardships, not only in recruiting human resources for medical schools but also in making opportunities for academic education . Their eWhen focusing on a Southeast Asian context, there are region-specific issues to consider. First, Southeast Asia is one of the regions with the smallest number of medical schools per million population . The resAfter recognizing these challenges and disadvantages that non-English-speaking Asian developing countries face, Seoul National University College of Medicine (SNUMC) initiated an international FDP called the Seoul Intensive Course for Medical Educators (SICME). The Department of Medical Education (DME) and two affiliated institutions of SNUMC, the JW Lee Center for Global Medicine (CGM) and the National Teacher Training Center for Health Personnel (NTTC), had pivotal roles in planning and implementing the program. The primary and immediate goal of the SICME was to educate and prepare fellows as medical educators who would have a broad and general competency in the field. In the long term, we expect them to form a critical mass of leaders in medical education, and to drive educational reform in their own institutions and countries.In total, 16 fellows from five Asian developing countries were invited to SNUMC for the program. Four countries—Cambodia, Laos, Myanmar, and Vietnam—belonged to Southeast Asia and one country, Mongolia, to East Asia. Mongolia was the only country located in a different geographic region, but WHO classified it in the Western Pacific Region, together with Cambodia, Laos, and Vietnam. Indeed, they share common problems in human resources for healthcare  as well This study has two aims. The first one is to describe the key features of the program and to examine the evaluation results based on Kirkpatrick’s four-level training evaluation model to understand the effects and limitations of the program. The second aim is to discuss better ways to collaborate with developing countries in faculty development and education to overcome the challenges facing them. Even though the participants of the SICME are mainly from Southeast Asian countries, we also discuss the relevant implications that would be applicable to other non-English-speaking developing countries, especially those countries who have been marginalized from existing international collaboration due to language barriers.Three levels of collaboration were central initiatives of the program. One level was intraorganizational collaboration inside the host institution—DME, CGM, and NTTC—as mentioned above. Another level included two types of intranational collaboration—one among fellows from the same countries and the other among facilitators from various medical schools in South Korea. The last one was at the international level, which included collaboration among fellows from different countries and continuing international collaboration among institutions participating in the SICME.To promote the fellows’ ownership of the program, every component of the SICME must be organized based on the careful consideration of the needs of the fellows, their institutions, and their countries.To enhance the application and transfer of learning after the program, the contents must be adaptable in resource-limited settings. Additionally, every piece of educational material should be provided in both hard copy and soft copy formats, unless there are any copyright issues.To facilitate learning, participatory and learner-centered pedagogical methods rather than teacher-centered ones must be prioritized and maximized throughout all courses.For continuous improvement, evaluation of the program must be conducted based on Kirkpatrick’s four-level model and feedback from all participants must be considered.Prior to determining the detailed curriculum of SICME, the Planning Committee, consisting of ten faculty members from eight medical schools, reached a consensus on four overarching design principles of the program.Another major task of the Planning Committee was to determine the topics for the 6 week program. Because the NTTC possesses its own source of education programs for health professionals which have been developed, revised, and used for over 35 years since its inception, module topics and their contents and relative lengths were determined primarily based on the previous education experience of the NTTC. After developing the broad outlines of the program, the Planning Committee thoroughly examined the educational backgrounds, professional experiences, stated needs and interests of each participant in his/her application form for fine adjustments to the program contents. All the members of the Planning Committee had iterative meetings to reach a consensus on the final structure and detailed contents of the program. Finally, one week was allotted for modules 1, 2, 3, 4, and 5. Modules 6, 7, and 8 were given one-third each of week 6  and 5 (Expert/Mastery) referring to two widely used competency level frameworks suggested by Dreyfus H  and MillBecause it is recommended to do the level 3 evaluation 3 to 6 months after training, we requested that the fellows report their actual progress 3–4 months after they completed the SICME. Moreover, we had designed part of the PPS with the purpose to predict the transferability of their learning by asking about the anticipated impact (PP 7), urgency (PP 8), and feasibility (PP 9) of the topics in the 8 modules.t-test and Spearman’s correlation analysis were used to analyze the quantitative data obtained during the evaluation process.To ensure content validity, all the instruments for data collection were designed and developed with reference to the book by Kirkpatrick DL , the desFor this study, exemption from ethical review was approved by the Institutional Review Board (IRB) of Seoul National University College of Medicine. Before conducting the surveys, we obtained oral informed consent from all the fellows, and the IRB waived the documentation of informed consent later in the ethical review process.Table Two items, one from the PMS and the other from the PPS, showed statistically significant correlations with the level of satisfaction for each module. One was ‘PM 9’ which asked how much a fellow thinks the module will help him/her do his/her job better Table , and the“It is not easy to translate my impression into a quantitative measure. However, what I’ve found out during the SICME is that their (fellows’) way of thinking changed substantially and was constantly maturing as the SICME progressed. At week 6, the level of discussion and their attitudes as a medical educator were totally different from what I saw in the first week.”In the PMS, we asked the fellows to self-assess their improvement in knowledge, skills, and attitudes. One noticeable finding was that they assessed their improvement in attitude higher than that in knowledge or skills. Although the difference was not statistically significant, the trend was consistent throughout the program, except for module eight Table . This reTo evaluate learning, four evaluators independently assessed the portfolios of the fellows, and the fellows also self-assessed their own overall competencies Fig. . StatistBecause one of our major goals was to train trainers who can then facilitate educational reform at their own institutions, the fellows were expected to disseminate what they learned during the SICME after going back to their countries. Therefore, not only improvement in knowledge, skills, and attitudes but also the development of actual capability as a workshop facilitator who can train trainers is important. In this regard, in PP 6, we asked fellows to choose all the module topics for which they could run a workshop on by themselves after going back to their home institutions. Meanwhile, evaluators also evaluated the suitability of each fellow as a workshop facilitator based on his/her portfolio. The results from the fellows and the evaluators were in agreement in that the fellows are the most capable for facilitating workshops for modules 1 and 3, and the least capable for modules 7 and 8. The correlation between the evaluation results of the evaluators and the fellows also showed statistical significance Table .Table 5EPrior to collecting actual data on transfer (level 3), we tried to examine how fellows perceived each module topic in terms of its impact, urgency and feasibility in the PPS Fig. . As the The interim progress report for each country shows that fellows were actively sharing and transferring their experience from the SICME Table . There wNot long after the submission of the progress report, three of the five countries—Laos, Mongolia and Myanmar—requested that SNUMC provide further support and consultation as an external expert body. Hence, SNUMC organized three separate teams according to the topics and types of projects for which they requested support. Faculty members of SNUMC who participated in the SICME as facilitators took charge of each team as core members. In Myanmar, DME and CGM provided a three-day workshop in July 2014 on ‘Curriculum Development in Line with Accreditation’ to heads, directors, administrators from key organizations in Myanmar who have a major influence on medical education in that country including major medical schools, Myanmar Medical Council, and Myanmar Medical Association. In Laos, CGM, in collaboration with the Western Pacific Regional Office of WHO and University of the Philippines Manila, provided three one-week workshops that cover general topics of medical education to faculty members of the University of Health Sciences, the only medical school in Laos. Lastly, to support the Mongolian fellows, DME acquired funding for a three-year project from the Korea International Cooperation Agency (KOICA) which aims to strengthen the educational capacity of Mongolian health professionals. For this project, DME organized a joint team with experts in organization development who joined the SICME as facilitators in Module 5 and with Raphael International, a medical nonprofit organization whose primary activity is to provide medical aid to impoverished people, which has more than 8 years of experience in medical service provision in Mongolia.In this study, we organized an international FDP for Asian developing countries, based on four design principles using multi-level collaboration. The results show that it is was successfully implemented with fellows who had limited training opportunities to address their own needs. The training was also effective in terms of the first (reaction), second (learning) and third (transfer) levels of Kirkpatrick’s model. Despite the effectiveness of the program, further studies are needed on each level and on how the collaboration actually worked.First, for level 1 (reaction), the fellows showed generally high levels of satisfaction for the various aspects of the program. Although high satisfaction itself does not guarantee learning, this is important because participation might decline unless the fellows perceive that the time and effort they invested is meaningful. Thus, as we expected from our first design principle, it is important to consider the personal and professional needs of participants to motivate their participation , 25. ActRegarding level 2 learning, one of the most interest findings was that a larger increase in attitude, which is particularly important in adult learning, was more significant than that of knowledge or skills. The third design principle of the SICME, which emphasized facilitating learning through learner-centered teaching methods and teacher-centered lectures, seems to have contributed to this result. Based on this principle, we reduced the proportion of lectures, to as low as 30% and tried to maximize learner-centered methodologies, such as small- and large-group discussions, role-playing, and participatory workshops. Through those learner-centered activities, what we tried to achieve was to provide fellows from five different countries with the opportunity to rethink their attitude as medical educators while encountering the ideals and reality of other countries, finding similarities and differences with each other, and arguing for and against something. Through these experiences, they exhaustively reflected on themselves and their situations and thought about their future direction. Previous research showing that active learning and student-centered approaches foster positive attitudes towards the subject  further Although the portfolio evaluation results showed significant improvement in the overall competency of the fellows, we found a tendency for the fellows to overestimate their own competency in the self-assessment results. This is not unusual because prior research has shown that doctors have a weakness in evaluating their own abilities . HoweverLearning in a FDP is less meaningful unless it can then be applied to the workplace. Use of projects, in this regard, as an essential element for effective faculty development, is highly recommended , and manLooking at Fig. In addition to the four design principles, multi-level collaboration was another core strategy to overcome challenges that we are likely to face when attempting an international FDP, such as the SICME. First, there was an intraorganizational collaboration inside the host institution. Institutions that took initiative in the SICME have a distinct experience in educational development, either as a donor or as a recipient. In the 1950s, after the Korean War, there was tremendous financial and educational development aid from the U.S. provided to SNUMC in the form of the so-called Seoul National University Cooperative Project . A few dSecond, two types of intranational collaboration were key to organizing an English workshop with facilitators and fellows only from non-English-speaking countries. One form of the intranational collaboration was that of the facilitators. For most of the facilitators, who had experienced merely short-term training in a foreign country at most, preparing materials and running workshops in English was no small burden. We did, however, successfully lessen the burden of each facilitator by distributing that burden to more than 20 facilitators from 15 institutions while the module coordinators safeguarded the whole program keeping it aligned with the four design principles. The other was the collaboration among the fellows. The fellows were quite heterogeneous in terms of their English fluency, and sometimes, this was a barrier to delivering important concepts. To solve this problem, we encouraged active peer-tutoring and extensive discussions in their own language whenever they needed. The importance of this interaction is that our intended goal of collaboration among the fellows was not only a means to foster the learning of the educational contents during the SICME, but also a foundation for forming a ‘critical mass’ of leaders in medical education at their own institutions, and in their own regions beyond.The last one is characterized as continuous international collaboration which encompasses the whole process of the SICME—before, during, and after the program. Before the organization of the SICME, SNUMC and the partner institutions at the five participant countries already have been maintaining cooperative relationships. These relationships generated two main advantages. First, this allowed us to discern and select the most suitable fellows possessing an appropriate background, commitment to learning, and actual leadership to reach the long-term goals of the SICME. Second, it enabled us to develop the SICME program based on the up-to-date information of the participants which enabled us to address their needs more accurately. The positive impact created by the international collaboration among the participant countries did not end with the program. After completion of the program the fellows, rather than SNUMC, actively sought ways for further collaboration. They not only tried to transfer and share their learning through their own efforts, but also voluntarily looked for ways to utilize newly established international relationships by requesting the participation of SNUMC in their projects as an external expert group. There are two reasons that make this especially encouraging. First, projects initiated and led by the recipient side are likely to be more effective than donor-led projects in that they can target the needs and interest of the recipients accurately. Second, from the donors’ side, the fact that a project is proposed by a recipient institution that calls for external support would more strongly justify the necessity of the project and make it easier to prepare budgets for aid.The evaluation results for levels 1, 2, and 3 showed that we successfully overcame the major inherent challenges of this program through elaborated collaboration. However, there are still some limitations that need consideration. First, our strategy for inviting fellows to South Korea was effective in achieving high satisfaction and active participation in the program, but it places financial restrictions on any major increase in the number of fellows in the following years. More importantly, considering the current positions of the fellows at their institutions, leaving their positions for more than a month or two at a time is also demanding on them. Consequentially, it limits the length, timing, and frequency of the FDP. Ultimately, these limitations might restrict the formation of sufficient’critical masses’ in recipient institutions. Continuing the education of subsequent batches of fellows is important not only because most of the fellows in this study asked for it but also because one-time, short-training courses are at risk of having little or no effect without any further supports . Second,In this study, we implemented a FDP with the aim to help overcome problems in Asian developing countries, especially non-English-speaking countries, and those who are marginalized and only have had scarce opportunities in the area of faculty development. According to the evaluation results, fellows were highly satisfied with the program, and their learning significantly improved after every module. Moreover, there was active transfer of learning, mainly with topics that were highly feasible, depending on their current situation. Even though the format, content, and evaluations have limitations, the overall direction of the observed changes was in line with the initial purpose of the SICME. The foremost task in overcoming the current limitations is supporting current fellows by providing timely feedback at the individual level and establishing cooperative relationships for continuous support at the organizational level in the long term. Future research needs to examine the underlying factors of transfer in various contextual aspects, such as organizational structure, culture, and leadership. These could contribute to further improving the effectiveness of FDPs for developing countries.