The health care sector is an area of social and economic interest in several countries; therefore, there have been lots of efforts in the use of electronic health records. Nevertheless, there is evidence suggesting that these systems have not been adopted as it was expected, and although there are some proposals to support their adoption, the proposed support is not by means of information and communication technology which can provide automatic tools of support. The aim of this study is to identify the critical adoption factors for electronic health records by physicians and to use them as a guide to support their adoption process automatically.This paper presents, based on the PRISMA statement, a systematic literature review in electronic databases with adoption studies of electronic health records published in English. Software applications that manage and process the data in the electronic health record have been considered, i.e.: computerized physician prescription, electronic medical records, and electronic capture of clinical data. Our review was conducted with the purpose of obtaining a taxonomy of the physicians main barriers for adopting electronic health records, that can be addressed by means of information and communication technology; in particular with the information technology roles of the knowledge management processes. Which take us to the question that we want to address in this work: "What are the critical adoption factors of electronic health records that can be supported by information and communication technology?". Reports from eight databases covering electronic health records adoption studies in the medical domain, in particular those focused on physicians, were analyzed.The review identifies two main issues: 1) a knowledge-based classification of critical factors for adopting electronic health records by physicians; and 2) the definition of a base for the design of a conceptual framework for supporting the design of knowledge-based systems, to assist the adoption process of electronic health records in an automatic fashion. From our review, six critical adoption factors have been identified: user attitude towards information systems, workflow impact, interoperability, technical support, communication among users, and expert support. The main limitation of the taxonomy is the different impact of the adoption factors of electronic health records reported by some studies depending on the type of practice, setting, or attention level; however, these features are a determinant aspect with regard to the adoption rate for the latter rather than the presence of a specific critical adoption factor.The critical adoption factors established here provide a sound theoretical basis for research to understand, support, and facilitate the adoption of electronic health records to physicians in benefit of patients. In this study, the concept of electronic health records is used to describe applications for manipulating and processing any information, of an individual, that resides in electronic systems for the purpose of providing health care and health related services . Based oThese systems have been widely diffused in the medical domain, not only for the reduction in costs of information and communication technology, but also for supporting physicians tasks in three directions: 1) reducing medical error rates ,3; 2) suAdoption studies can be grouped in a more general theory called diffusion of innovations, as proposed by Rogers , which h• Domain: our main interest is the medical domain; the taxonomy is focused on this and takes into account its culture in the use of information systems. Some studies facilitate the understanding of successful adoption and the use of information systems, but such studies are not focused on the adoption of electronic health records ,14.• Approach: our taxonomy is aimed towards a knowledge-based technology approach. Some studies propose critical factors for adopting electronic health records, but they do not recommend the elements for supporting their adoption by automatic means ,18.• Study subjects: our taxonomy does address physician personnel. However, some work is oriented toward the medical domain, and is not focused on physicians, i.e. it is aimed at patients , nurses The innovation-decision process, showed in Figure On the other hand, technology can be used for creating, storing, distributing, and applying knowledge. Each one of these knowledge processes can be implemented with information and communication technology roles  such as According to all the latter, the aim of this review is to provide a taxonomy to identify the critical adoption factors of electronic health records, that can be supported by information and communication technology, in a proactive form, to assist the users of these systems during their adoption process. This taxonomy is based on a list of critical adoption factors of electronic health records, classified from a knowledge point of view Figure ; where, In this way, every critical adoption factor could be associated with one or several of the information and communication technology roles of the knowledge management processes  to apply such knowledge for assisting the adoption of the electronic health records Figure .We have searched for relevant English language papers based on keywords within their title and abstract. Since knowledge helps to reduce uncertainty about an innovation, and technology can provide that knowledge timely and automatically, our review was conducted to answer the question "What are the critical adoption factors of electronic health records that can be supported by information and communication technology?" and was based on the PRISMA statement . Figure First, some criteria for inclusion and exclusion of studies were established. We included studies that met the following two criteria: they were published between January 1, 1985, and December 31, 2009; and they describe critical adoption factors for adopting electronic health records by physicians, e.g. if they focused specifically on critical factors for nurse and/or administrative personnel, the studies were excluded. Although it is important to analyze the adoption of electronic health records by other stakeholders in the medical domain, as we stated before, physicians have a significant position in the use of these systems. We then excluded studies if they described critical adoption factors which can not be supported by the information and communication technology roles of the knowledge management processes, e.g. the establishment of financial incentives for the organizations that invest on technology has been defined as a critical adoption factor of electronic medical record systems , howeveradoption electronic health record in document titles and abstracts. In addition we selected periodicals and conference proceedings publications. In the PubMed Central search engine, we also searched for the keyword adoption electronic health record in title and in citation+abstract. In the Springer Link search engine, first we searched for adoption electronic health record in all items and then we excluded books, series, and web content results. In the ScienceDirect search engine, we searched for the keyword adoption electronic health record in the documents' title and abstract. In the SAGE search engine, we searched for the keyword adoption electronic health record in the paper title and in full text/abstracts for the selected journals. Finally, in the Scirus search engine, we searched for the keyword adoption electronic health record in the complete documents for articles and scientist homepages. The search keyword is very wide, and provides a large number of studies that do not correspond to the objective of the literature review, however, it also provides the possibility to review work that meets the two criteria that concerns us: 1) addressing the issue of innovation adoption in medical field, and 2) including innovations concerning electronic health records.We searched for journals and proceedings studies about information and communication technology as support of medical tasks in the following electronic libraries: IEEE, ACM, PubMed Central, Springer Link, Wiley InterScience, ScienceDirect, SAGE, and Scirus. The selected electronic libraries take into account that chosen journals and proceedings included articles with proposals for using information and communication technology to support automatic processes in the medical domain. To avoid excluding relevant articles for our review, we considered results from the Scirus search engine, which has a broad scientific spectrum and a coarser granularity in search in the scientific area, but includes useful results for purposes of our review. As mentioned earlier, we searched in selected journals and proceedings for articles published between January 1, 1985, and December 31, 2009. In the IEEE and ACM search engines, we searched for the keyword Two reviewers independently assessed each article for eligibility criteria. A third reviewer reviewed by adjudication, in cases of disagreement. While reviewing each study, the same two reviewers used standardized forms to independently extract and record four aspects: study id, list of critical adoption factors, country where the study was accomplished, type of practice, study setting, a knowledge-based classification in which critical adoption factors might engage type of electronic health records use, and study type. To collect the critical adoption factors for adopting electronic health records, the reviewers examined the critical adoption factors, defined in each study, taking into account the information and communication technology roles of the knowledge management processes presented in table Searching the online databases resulted in 2,920 articles from IEEE, ACM, PubMed Central, Springer Link, Wiley InterScience, ScienceDirect, SAGE, and Scirus. From the 2,581 titles that were initially identified, 441 were from IEEE, 279 from ACM, 971 from PubMed Central, 542 from Springer Link, 42 from Wiley InserScience, 28 from ScienceDirect, 278 from SAGE and 339 from Scirus. After removing duplicates and going through a screening process, we identified 168 eligible articles for further full text review. Based on a full text review, 100 articles were excluded for one or more of the following three reasons: 1) they do not address the physician staff, the adoption unit is either a nurse or administrative personnel; 2) the adoption unit is an organization, not physicians; and 3) they just describe critical adoption factors not supported by information and communication technology, for example, financial incentives to use electronic health records.From all these studies we categorized the critical adoption factors into six main categories, which were arranged from the highest to the lowest order of relevance, as following user attitude towards information systems, workflow impact, interoperability, technical support, communication among users, and expert support. That order of importance is given by the frequency at which such factors are reported in the studies included in the review of literature. The 68 studies are listed in Table We have defined user attitude towards information systems to characterize a subjective critical adoption factor of the electronic health records. As Ajzen  states, Fifty four studies determined user attitudes towards systems as a critical factor for adopting electronic health records -70,87,89Workflow systems enable knowledge artifacts to be dispatched through organizations by means of a relative fixed process . Twenty In this study, interoperability is related to compatibility, which is a perceived attribute of an innovation. The later is defined by Rogers  as the dTechnical support relates to technical assistance given from technical staff to physicians through personal contact or via documents. Fourteen studies defined technical support as a critical adoption factor for electronic health records ,66,74,76Communication among users refers to the act of interchanging thoughts, opinions, or information by speech, or writing. Twelve studies defined communication among users as a factor affecting the adoption of electronic health records ,66,75,87Expert support refers to the assistance provided from a physician to another physician. This can be divided in two aspects: 1) a physician with experience in electronic health records usage assists, with information about how to use the system, to another physician; 2) a physician has the knowledge to help another physician accomplish a medical task. Such assistance can be given through personal contact or via documents. Eleven studies established expert support as a very determinant factor for adopting electronic health records ,45,66,76The main limitation of the taxonomy is the different impact of the adoption factors of electronic health records reported by some studies depending on the type of practice, setting, or attention level; however, these features are a determinant aspect with regard to the adoption rate for these systems rather than the presence of a specific critical adoption factor, as discussed in the next section.In this section we present the main implications about the results of our literature review on critical adoption factors for adopting electronic health records. The following five aspects must be emphasized:1) The definition of critical adoption factors described here does not make a difference between inpatient and outpatient settings for electronic health records. Nevertheless, as it has been explained in other studies, these differences represent variations in the level of the effect of a critical factor, rather than the presence of a specific critical adoption factor .2) We have here depicted critical adoption factors for the use of electronic health records by physicians. But, it has been argued  that ful3) Different kinds of practices present different kinds of adoption levels. For instance, small practices are less likely for people to adopt than large ones , and the4) Some studies indicate that national culture has a significant influence on a specific country adoption rates to some innovations . On the 5) Our search does not differentiate between mandatory and voluntary settings for adoption studies. Adoption in voluntary use is reflected in the usage of the electronic health records, and for mandatory system use, the adoption is reflected in the overall user acceptance . TherefoStudies on identification of critical factors for adopting electronic health records show that it is possible to define a knowledge-oriented taxonomy for these factors Figure . In thisThe fact that some critical factors for adopting electronic health records were found to be dominant critical factors, may guide the efforts for assisting the adoption process of these systems. For example, attitude towards information systems and technical support are critical factors that may be supported with information and communication technology tools for facilitating knowledge dissemination, reducing uncertainty about the electronic health records. The former can be encouraged with friendly context-aware user interfaces which facilitate needed knowledge in the required format and timely; the later can be assisted by knowledge directories and hyperlinks to technical reports.Taking into account a knowledge-based approach for defining critical adoption factors of electronic health records enables us, in addition, to define what critical adoption factor is determinant in each of the innovation-decision process stages. For example, as we have stated, communication among users is determinant in knowledge and confirmation stages, which can lead to use information and communication technology roles of knowledge management to moderate the communication behavior of an adoption unit, and help it become aware of an innovation to induce an adoption decision.The information and communication technology roles described are available features in factual knowledge oriented information systems; and its relationship with the defined critical adoption factors can be used as a guide for a conceptual framework for assisting the adoption of electronic health records.As we have stated, using a knowledge-based approach for classifying critical adoption factors of electronic health records leads to define a relationship between these factors and information and communication technology. These kinds of approaches imply transforming practical knowledge into structured one . This isEach specific type of interaction a physician has with an electronic health record could be associated with a critical adoption factor and the latter with a role of information and communication technology in the knowledge management processes; this association helps physicians to support an adoption decision in an automatic manner. Since it is possible to associate a physician- system interaction with the type of electronic health record adoption behavior, then it is possible to use the approach proposed in Figure User attitude towards electronic health records is a strong determinant for their adoption. Due to the subjective nature of this, there is not a specific type of interaction between the physician and the electronic health record that can make this attitude evident; however, it can be inferred through the system's use. For example, a transaction log of the system showing a slight interaction between physician and the electronic health record may indicate the perception of a negative attitude towards the system. If so, an electronic bulletin board where a physician can express his doubts, concerns, and general reasons for not using it can be used; from there the physician may be provided access to local frequent asked questions lists, or depending on the nature of the motive that led to his attitude, provide other information and communication technology tools to support an adoption decision.Concerns about negative impact on workflow may cause uncertainty about the use of electronic health records, which can negatively impact on its adoption. For example, if a physician observes that the data filling a medical prescription form in the system reduces the time for patient care, the physician may stop using the features that this module provides. If the physician's interaction with the electronic health records indicates that he has stopped using that module, then it could automatically proposed prefilled templates through simple typed entry in this module, or reuse captured notes on subsequent encounters with patients.The lack of interoperability of the electronic health records with other electronic health records or organizational systems causes problems on their adoption. For example, if a physician needs to capture data in the system to provide health services in first level of attention and then reused the data in another unit system to provide other services, the physician requires that both systems are interoperable, if not, it can negatively affect his adoption decision. Although there is not a special classification of interactions between a physician and electronic health records that can indicate a lack of interoperability in a system, the adoption can be inferred based on their use. If this is the case, an electronic bulletin board can automatically provide assistance to understand the causes of non-use, when these are known then it can provide links to documents about standards or make reengineering of the system so that it can interact with other organizational units' systems or electronic health records.The deficiency or lack of technical support makes the uncertainty about changes in electronic health records persisting, impacting negatively on its adoption. For example, if a physician performs the incomplete capture of data in a module of the system and does not receive technical support to investigate the reason and solve the problem, the physician may stop using module or the system. If the analysis of the interaction between the physician and the electronic health records reveals that the system did not capture the data in that module, the automatic provision of support directories or technical report hyperlinks to specific system information about a module can reduce uncertainty to capture data in this.There is a need of communication between the users of electronic health records, so that when an event that causes uncertainty about the use of the system occurs, coworkers can share their experience using the system, therefore promoting its adoption. For example, when modifying the electronic health record user interface, less experienced users can communicate with more experienced users so that the former can provide support to the latter in any doubt about the change. If the interaction between a physician and the system indicates that the characteristics associated with a change in the user interface was not used, then a tool that can relate users to help each other may be invoked automatically enabling the communication of the physician who did not used the user interface with experienced practitioners who could provide support to use it. Such strategies to support communication between physicians can be extended to other electronic health records users taking into account users' profiles: given that the tools for collaboration between users are based on their profiles, if these tools take into account a different profile, they can support different users.When there is a lack or deficiency of medical expert support to other users of electronic health records that are less experienced, innovations may cause uncertainty about the use of the system, giving rise to problems in its adoption. For example, if a doctor has problems about a medical prescription in a module of the electronic health record and receives no assistance from his preceptor, the former could stop using the system. If the physician's interaction with the electronic health record indicates that the doctor has not made the prescription, the automatic provision of knowledge directories or knowledge artifact hyperlinks can provide the necessary knowledge to carry out the prescription.We have found that critical factors for adopting electronic health records can be classified from a knowledge oriented perspective to support the development of approaches for assisting the adoption of these systems.As we stated before, the critical factors for adopting electronic health records to which we can provide automatic support by means of a knowledge perspective are six: user attitude towards information systems, workflow impact, interoperability, technical support, communication among users, and expert support. Based on the frequency with which such factors are reported in the studies included in the literature review, these are arranged from more prevailing to less prevailing factors.Currently, there is little evidence of the use of information and communication technology to assist the process for adopting electronic health records in an automatic fashion. Focusing on the critical adoption factors here identified, benefits technology practitioners' efforts for designing and implementing information systems for supporting, in a proactive and knowledge oriented manner, the adoption of electronic health records. This provides four directions for future work: 1) defining knowledge representation structures for supporting the adoption of electronic health records; 2) scheming conceptual frameworks -and corresponding methods- for facilitating their adoption; 3) process-modeling for supporting in automatic manner the adoption of electronic health records; and 4) designing software architectures for software applications focused on supporting the adoption of factual electronic health records. We therefore consider that critical adoption factors and its relationship with the information and communication technology roles of the knowledge management processes provide sound theoretical basis that scholars should use for empirical and theoretical research to understand, support, and facilitate the physician's adoption of information and communication technology for the benefit of patients.The authors declare that they have no competing interests.VHC and AIMG developed the idea for the study and its design. All authors reviewed the literature. VHC and JRGP drafted the manuscript. All authors revised the manuscript critically and approved its final version.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/10/60/prepub


Burnout is a psychological reaction triggered by interaction between personal characteristics and stress factors. Reorganizations and downsizing with increased workload imply stress for health-care professionals. This is a study of burnout in nurses during a period with two comprehensive reorganizations.In this quasi-experimental retrospective cohort study, burnout was assessed in nurses with long work experience in three surveys during a 30 months' period with two comprehensive reorganizations and downsizing of a hospital unit with mostly seriously ill patients with cancer. Burnout was measured with Bergen Burnout Indicator (BBI) at each survey, and "Sense of Coherence" (SOC) with Antonovsky's questionnaire at the last survey.One man and 45 women aged 30 to 65 years were invited to the surveys. There was a significant increase in burnout during the study period, the mean increase in BBI-score was 12.5 pr year (p < 0.001). The proportion of satisfied nurses at the first and last survey were 84% and 35% respectively, and the proportions with burnout were 0% and 29% respectively (p < 0.001). Except for auxiliary nurses with experience from the medical department, all subgroups experienced a significant increase in BBI. Burnout was associated with low SOC .There was a significant development of burnout in a group of nurses during a period with two reorganizations and downsizing. Burnout was associated with low SOC. Working with seriously ill patients with cancer has probably made the nurses exceptionally vulnerable to the stress and workload related to the reorganizations. Burnout syndrome is a well-known psychological reaction among health-care workers -3. TriggReorganizations and downsizing have become an increasingly common stressful event in the health-care system and have been associated with a high degree of depression, anxiety and emotional exhaustion ,11. A feThis retrospective cohort study aimed at studying the development of burnout in personnel with long work experience during two large reorganizations with downsizing, and factors like SOC, profession and work experience related to burnout. Concern for the nursing staff and interest for the human costs of repeated reorganizations motivated the study.In one hospital unit with mostly cancer patients, all registered nurses and auxiliary nurses with permanent employment and more than five years' working experience were included. The group was selected because of low expected burnout potential.The design is a quasi-experimental retrospective cohort study. The participants were followed for 30 months during two large reorganizations of the department. The purposes of both reorganizations were significant resource reduction without impaired health care. The workload increased. Three surveys were performed, one before the first reorganization, the second between the two reorganizations, and the third after the last one. The head of the department (author KN) initiated the surveys aiming at implementation of preventive initiatives if necessary.The first survey was performed after three years of regular service of an oncological hospital unit consisting of an inpatient clinic with 17 beds and an outpatient clinic. At the first survey, there was no knowledge of the forthcoming reorganizations. The participants filled in questionnaires for evaluation of burnout, and demographics were obtained from the hospital's administrative computer system.At the first reorganization, the specialized inpatient clinic for oncology was closed down and the outpatient clinic for oncology was reduced. The rest of the oncology unit was merged with a specialized surgical unit which also was significantly reduced, into a new common medical and surgical unit with a mixture of employees with medical  and surgical  experience. In total 12 employees with less than 5 years' employment and therefore not included in this study, were relocated to other work or denied prolongation of temporary position.The second survey, 21 months after the first one, was performed when normal service was established in the new department. There was no knowledge or suspicion of the second reorganization starting immediately afterwards.The second reorganization started with a decision to close down the new department. After three months' operational planning with discharge and/or relocation of the employees, it was decided to continue with a reduced unit. None out of nine relocated or discharged employees with a short length of service participated in the study. At the third survey 9 months after the second one, when regular service was re-established anew, a questionnaire for measuring sense of coherence was added. Figure Gender, age, profession, duration and type of work experience and size of current position were noted.Burnout was measured with Bergen Burnout Indicator (BBI). The questionnaire has been developed and validated in Norway and has good psychometric properties  and high correlation with more commonly used questionnaires like Maslach Burnout Inventory and Stress Burnout Scale for Health Professionals -17. The Antonovsky proposed the concept "Sense of Coherence" (SOC) . SOC refThe authorities required that data were made completely anonymous before being analyzed and published. We were not allowed to follow one person from one point of time to the next, and lack longitudinal data for each participant. Therefore, data were analyzed as if they were independent, and the analyses are conservative (biased) in the sense that the p-values are expected to be higher than they would have been from analyses based on methods for longitudinal data.Chi-square (with linear-by-linear associations when appropriate) was used for the table analyses, Pearson's correlation analyses for the association between BBI and SOC, and linear regression analyses for the relations between BBI (dependent variable) and time (independent variable). P-values < 0.05 were considered statistically significant and the main results are given with 95% confidence intervals (CI).The aims of the surveys were surveillance and protection of the employees against burnout, and not research. Participation was optional. After having been made anonymous, the head of the hospital and the Norwegian Data Inspectorate, represented by Privacy Ombudsman for Research at Oslo University Hospital, Norway, approved the data for research.Forty-eight employees were invited to the surveys. Table Table The main finding was a marked and clinically significant increase in burnout score during the study period. The increase was statistically significant in the whole group of participants and in the subgroups with different professions and work experience. The only exception was a small group of auxiliary nurses with experience from medical departments. They showed the same order of deterioration, but due to a wide range in the results, the changes were not statistically significant.For comparison, a Norwegian database with BBI scores from 3582 persons is available, nearly all were health-care professionals . At the The time of registration probably affects the results. The first survey with a high degree of satisfaction was performed in a period with regular service and a well-functioning group without prior knowledge of forthcoming changes. The second and third surveys were deliberately performed after the summer and Easter holidays to reduce an acute, stress-related, blow off steam and an exaggerated response.Burnout takes at least one to two years to develop, but normally longer ,8. The sBurnout is related to personal and external factors. Personal factors of importance are low self-esteem, unrealistic expectations, weak sense of coherence, excessive conscientiousness and high education level ,4,22-24.In this study, the participants were working with seriously and incurably ill and dying patients with cancer. This type of work has been shown to be meaningful and personally stimulating for health-care professionals, but also stressful ,3,5,27. Sense of coherence (SOC) was measured at the last survey. In accordance with other studies, there was a significant association between burnout and low SOC ,23,24. SThe significant burnout seen during the 30 months' period was probably due to the comprehensive and rapid reorganizations that had an extraordinary high negative impact on nurses working with seriously and terminally ill patients with cancer. The finding that downsizing and reorganizations increased the risk of burnout clearly shows that implementation of initiatives to prevent burnout should be mandatory during such periods. The most efficient ways to prevent burnout under such conditions are largely unknown, and studies aimed at finding the best preventive methods are required.The restricted number of participants might affect the external validity. There are no indications of a particularly vulnerable group of participants. On the contrary, it was a group with robust personality, long work experience and high degree of job satisfaction as measured with BBI before the reorganizations. However, working with seriously and terminally ill patients with cancer might have made them exceptionally vulnerable. The results are clear, convincing and statistically significant.The comprehensive and frequent reorganizations in the hospital made it an ideal quasi-experimental situation for studies of effects of reorganizations and downsizing. The retrospective observational design has inherent weaknesses, but prospectively controlled trials are seldom possible.A few participants quit their jobs during the study period and some new employees with work experience were engaged. This reduces the validity, but is inevitable in such observational studies and is likely to reduce rather than increase the degree of burnout. The high participation rate increases the validity. Separate analyses of subjects participating in all surveys would have been of interest but was impossible since data had to be made completely anonymous prior to the analyses and publication.Measurement of SOC at all surveys would have given valuable information about changes in SOC during the stress period. Any influence from the head of the hospital unit, who performed the surveys, such as exaggeration or concealing of the problems, is unknown but unlikely.During a 30 months' period with two comprehensive reorganizations and downsizing, nurses with long experience and working with seriously and incurably ill and dying patients with cancer showed a significant and unexpectedly fast development of burnout. The reorganizations are the only likely explanation, but working with seriously and terminally ill patients with cancer might have hastened the development. Low SOC seemed to be a risk factor for burnout.KN was head of the department throughout all the reorganizations.KN has collected the data and has been the main contributor to the concept and design of the study, has taken part in the analyses of the data, made the first draft manuscript, has thoroughly revised the manuscript and is the main responsible for the professional content. MH has taken part in the design of the study, participated actively in the analyses and interpretations of the data, has supervised the writing and revised the manuscript. PF has taken part in the design of the study, is responsible for the statistical analyses and interpretation of the results, and has drafted and written the final manuscript. All authors approved the published version.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6955/9/8/prepub


Case study and narrative are essential and complementary ways to understand history: case study dissects historical events into their logical elements, whereas narrative connects us to the sum of these elements, which may be startlingly different from the outcomes predicted by logic. The historical stories in this report describe selected US public health events from 1900 to 1932 and link these experiences to modern leadership challenges in public health policy.In 1991 Richmond and Kotelchuck proposed a model for public policy that can be represented by a 3-circle Venn diagram . The cirThe stories in this article illustrate how the sectors of this model came together, for better or worse, in the early 20th century in the United States. Hookworms in the South (worms), cholera infantum in New York City (germs), adulterated alcohol in the Midwest (drink), and pellagra in the Mississippi Delta (dementia) are examined in terms of how diseases and their victims were viewed, diagnosed, and treated at that time. Medical progress also affected these events, but the history of medicine is not my primary focus.For public health professionals, the science and infrastructure sectors of the Richmond-Kotelchuck model are often where we feel most comfortable and best trained. The descriptions of Southeastern hookworm health education programs, infant milk distribution systems, adulterated alcohol investigations, and Goldberger's pellagra research all resonate with our experience and expertise.Political will is another matter. Why did Southerners refuse to believe hookworm and pellagra existed in their states? Why did a 20-year partnership of physicians and social progressives collapse and scatter the resources for infant health programs? Why was there little outrage when tens of thousands of Americans were paralyzed by a poisoned patent medicine? Why did public health leaders play only small roles in these widespread health events?It can be argued that most of our failures are in the arena of political will. Health policy courses appear in public health education curricula, but even experienced public health leaders trip on the complexity of political interactions. Public health practitioners in government agencies may be further restricted because of our responsibilities as public servants to remain apolitical. Our efforts to affect public policy are often frustrated.Health is inextricably woven into the geography, warfare, politics, poverty, population, infrastructure, economy, and prejudices of a region. As Szreter has noted:The problem, of course, with emphasizing the importance of politics, the conflict of ideas, the role of the state, conditions of citizenship, local government structures and services, civil institutions and social capital in accounting for the relationship between public health and social and economic change is that this makes for a devilishly complicated story .Kitchen Table Wisdom to describe the importance of sharing stories for her cancer patients, many of whom were terminally ill , established in 1874, had a branch in almost every county in the country by 1883 . The orgThe Gilded Age crashed in the Panic of 1893 . A run oAt the beginning of the 20th century, the Southeast was still recovering from the Civil War. Despite the proclamations of a New South, agriculture remained the primary source of employment, but Southern agricultural yield was the worst in the country. The states' public education and health infrastructures were weak to nonexistent. Twelve percent of white adults and 50% of African American adults were illiterate  formed in 1909. This national organization emphasized that social reform and health education were essential to support healthy children . The firAnother plank in the AASPIM platform was to foster communication and coordination among the many organizations involved in infant health. These political efforts led to the establishment of the federal Children's Bureau in 1912, which was mandated to "investigate and report on all matters pertaining to the welfare of children" . In turnBills to support these programs were introduced into Congress in several sessions, but not until 1921 did a bill appear that seemed likely to become law. It faced strong opposition. Politicians argued that it was too expensive and violated the separation of state and federal responsibilities. Physicians were coming into their own as a profession and feared any intrusion into their field; the American Medical Association and American Gynecologic Society claimed that the bill would interfere with the practice of medicine. Opponents of women's suffrage said it would weaken the family and transfer parental authority to the government, and that its premise arose from Bolshevik philosophy .Good Housekeeping, McCall's, and Ladies' Home Journal published supporting editorials , a chemical that at the time was not recognized as poison. The National Institute of Health conducted field investigations and collected more samples. Additional pharmacologic and animal testing indicated that the suspected samples were contaminated with TOCP and that this compound caused the jake paralysis ,41.The infrastructure of laws and bureaucracy begins this story. Individual efforts to subvert the infrastructure led to catastrophe, which in turn led to scientific efforts to identify a poison. But political will generated only a weak response.Two partners of a Boston firm were charged with illegal alcohol distribution and falsely labeled products and received fines and probation. In 1932, one eventually began a 2-year prison term . The FooNor did public policy change. The victims of jake paralysis were marked by their unique gait; conservative neighbors and even the afflicted themselves believed their own irresponsibility had destroyed their lives. Few of those who were poisoned made a full recovery: the United Victims of Ginger Paralysis Association claimed 30,000 members and sought federal compensation for those injured, without success .The Mississippi River wanders for more than 2,000 miles through the middle of the country, draining 41% of the continental United States ,44. For Inevitably, politics controlled river development. One-crop cotton farming still dominated the lower river basin after the Civil War. The plantation owners manipulated taxation at local levels, exploited the cheap labor of ex-slaves, and obtained taxpayer assistance at state and federal levels to maintain river flow. In the 1880s the river was channeled to allow shipping. In 1884 the Mississippi legislature created the Yazoo-Mississippi River Levee District to build levees along state tributaries of the Mississippi River, and in 1917 the Army Corps of Engineers took over levee building ,44. It wD's" of the pellagra syndrome , completing the "4 syndrome .By 1920 Joseph Goldberger had made his name from his pellagra research in southern orphanages, "insane asylums," and mill towns . He idenHenry Grady was long dead, but other New South spokesmen were infuriated. Once more Yankee voices had proclaimed a widespread disease of poverty in the South. Yet the Mississippi state board of health recognized the pellagra epidemic and in 1920 held a state conference to discuss action. In 1921 an investigation of the Delta region found it had 60% of the state's cases but only 20% of the state's population .Then came what Herbert Hoover described as the "greatest disaster in peace times in our history": the 1927 Mississippi flood . The rivGoldberger toured Tennessee, Mississippi, Arkansas, and Louisiana and estimated that 45,000 to 50,000 cases existed across the region. The Red Cross created a massive, uncontrolled natural experiment, distributing 12,000 pounds of yeast. The distribution was followed by a rapid decline in the number of reported cases and improvement in disease symptoms. For the first time, the pellagra-prevention diet proved effective in a large population  might be an example of successful technical problem solving. Research identified a major risk factor for SIDS, a national public education campaign ensued, and parents found it easy to put babies on their backs instead of stomachs to sleep (But these 4 stories bring us back to political will. Ronald Heifetz, author of problems . The firto sleep .adaptive change. Adaptive change is about political will. The change requires a significant paradigm shift in science, systems, and social attitudes, and the shift must occur in the face of substantial resistance from the affected sectors. Leaders may take the easier course of treating adaptive challenges as technical ones — people accept that professionals will provide technical answers, while no one may know the answer to or consequence of an adaptive change.The second leadership problem requires The need for paradigm shift exists when society's expressed values and its behavior differ. The Progressive Era arose from the public realization that laissez-faire economics and industrial power did not provide the individual economic security they claimed. The formations of the NAACP, the Grange, the AFL, and the GFWC marked the beginnings of a shift from laissez-faire economics to new expectations of government. The shift carried upheaval — women and blacks demanding social change, labor power rising to balance the employers', agricultural systems giving way to industrial ones. New stories appear when old ones are not sustainable; they are resisted because they arrive with economic and social costs.An adaptive need may be recognized without receiving action. Because change is unpredictable, only a sense of urgency will drive people to consider it. If pandemic influenza is sweeping the country, high levels of public distress will create the urgency for change. At other times the public refuses to accept a problem as urgent. The New South boosters, the industrialists, and the anti-immigration movement were not ready to accept that their stories were inadequate. In the early days of the recognition of acquired immune deficiency disease, most of the population believed the infection was restricted to a small number of men having sex with other men. Popular attitudes were similar to those toward the victims of jake paralysis: the affected individuals' behaviors were socially unacceptable, therefore they deserved their affliction and "normal people" need not worry.As stress increases, the temptation rises to find easier alternatives. Women leaders of the temperance movement believed outlawing alcohol would eliminate multiple social ills. The fight for temperance was long and difficult, but it was still easier to pass prohibition legislation than to control violence, prostitution, or poverty. The current debate about universal health care is a modern example. Over the past 2 decades provider-driven care, health maintenance organizations, health savings accounts, consumer-controlled health plans, government-based insurance, tax credits, and a host of other technical options have been proposed. The adaptive question, however, is whether our social contract includes providing health care for every citizen. Until we recognize the question, technical solutions will not succeed.Heifetz notes that the final, and perhaps most difficult, leadership role is giving the challenge back to the citizens. Leaders are tempted to "claim" big issues, while society waits for them to solve the problems and people go about their usual lives without accepting their essential roles in solutions. Prohibition did not prevent alcohol consumption, much less its associated social evils, because much of the population did not support its premise. Authority can direct attention to the challenges; it is not able to resolve adaptive problems without involving the populace.One of the greatest risks for public health professionals is to accept ownership of a problem. We recognize the importance of building partnerships, crossing boundaries, and involving communities, but we are not yet sufficiently humble about our ability to create adaptive change. We are strong in technical solutions, but how skilled are we at affecting the third sector of the Richmond-Kotelchuck model? We know that scientific evidence and infrastructure related to immunizations, safe sex, regular exercise, tobacco avoidance, and even universal health care are not always sufficient to drive political will. Do we have the ability to change social context? And can we gain wisdom from stories of the past?Now, in the beginning of the 21st century, societal pressures are building once more. In the United States another major economic transition is progressing, changing from an industrial base to a service-oriented, information base. Accompanying that change are labor migrations associated with a global economy; this time work is traveling abroad to the workers. At the same time, immigrants continue to arrive, and anti-immigrant bias is now directed toward those with Mexican and South American origins. The quality of child health programs and public schools is at risk. The percentage of older adults in the population is rising: a greater Medicare burden is predicted, but the generation now maturing will be healthier and live longer. What will this mean for the national workforce and health care demands? Shifting warfare, infrastructure, and ecology concerns overlay these other stresses.How do these stories speak to the role of public health workers? Perhaps we cannot force change, but we can offer leadership in our own arena, we can inform paradigm shift, we can tip events toward a successful resolution. The Rockefeller Commission and Southern Progressives combined an understanding of hookworm infestation, agricultural productivity, and school health to stimulate political will for better lives among rural Southerners. Goldberger's understanding of social constructs informed his research and recommendations for eradicating pellagra, and his work was essential, ultimately, for the fortification of bread. Sometimes our success is partial. Child health activists and the Children's Bureau placed an early emphasis on the connection between health and social welfare; later medical interests separated those concepts. Sometimes we fail. Public health researchers identified the cause of jake paralysis, but there was no call to improve the lives of the afflicted or to find ways to prevent future catastrophe.At its heart, public policy is about the choices of citizens in a cooperative endeavor. What are the early warning signs that these choices are at odds with professed beliefs? Where might we hear the first rumblings of a social shift? Will we understand the archetypal stories people use in health and policy decisions? Perhaps we need to teach ourselves another way of observing, to learn more about incorporating intuitive as well as data-driven evidence into action. Too often, of course, we have acted without evidence, and no one advocates a return to that practice. But given our struggles with political will, it seems we need not only new data but also new tools for understanding the data.The events described here grew out of the earlier history of the Gilded Age and in turn created the context for what we face today. The stories illustrate that public policy results from multiple elements, some of which developed long before current crises. In helping us understand political will, the past offers not only informative case studies but also narrative components of contemporary challenges. For us to offer leadership to influence change, we need to immerse ourselves in story, past and present.


Environmental health impact assessments often have to deal with substantial uncertainties. Typically, the knowledge-base is limited with incomplete, or inconsistent evidence and missing or ambiguous data. Consulting experts can help to identify and address uncertainties.Formal expert elicitation is a structured approach to systematically consult experts on uncertain issues. It is most often used to quantify ranges for poorly known parameters, but may also be useful to further develop qualitative issues such as definitions, assumptions or conceptual  models. A thorough preparation and systematic design and execution of an expert elicitation process may increase the validity of its outcomes and transparency and trustworthiness of its conclusions. Various expert elicitation protocols and methods exist. However, these are often not universally applicable, and need customization to suite the needs of a specific study. In this paper, we set out to develop a widely applicable method for the use of expert elicitation in environmental health impact assessment.We present a practical yet flexible seven step procedure towards organising expert elicitation in the context of environmental health impact assessment, based on existing protocols. We describe how customization for specific applications is always necessary. In particular, three issues affect the choice of methods for a particular application: the types of uncertainties considered, the intended use of the elicited information, and the available resources. We outline how these three considerations guide choices regarding the design and execution of expert elicitation. We present signposts to sources where the issues are discussed in more depth to give the newcomer the insights needed to make the protocol work. The seven step procedure is illustrated using examples from earlier published elicitations in the field of environmental health research.We conclude that, despite some known criticism on its validity, formal expert elicitation can support environmental health research in various ways. Its main purpose is to provide a temporary summary of the limited available knowledge, which can serve as a provisional basis for policy until further research has been carried out. It is widely recognized that exposure to environmental factors can cause adverse health effects. Many environmental health professionals are confronted with questions about the overall impact of environmental stressors on (public) health, or about the beneficial effects of policy measures to reduce environmental exposures. Oftentimes these questions are extremely difficult to address, due to limitations and inconsistencies in the scientific knowledgebase. Thus, benefits of policy measures directed to reduce environmental exposures are difficult to gauge and policy measures often have to be taken without conclusive scientific evidence. Integrated environmental health impact assessment (IEHIA) aims to support policy making by comprehensively assessing environmental health effects, while taking account of underlying complexities. In this paper, we will use the term IEHIA, but other (similar) forms of impact assessment  may serve the same purpose (see e.g. ). These Formal expert elicitation is one of the means towards a structured and transparent way to address such uncertainties. It refers to a structured approach of consulting experts on a subject where there is insufficient knowledge and seeks to make explicit the published and unpublished knowledge and wisdom of experts. Expert elicitation can serve as a means to synthesize the (limited) available knowledge in order to inform policies which have to be made before conclusive scientific evidence becomes available. The quality of the knowledge derived from experts, or at least its transparency and reproducibility, improves when expert elicitation is applied according to a systematic protocol. Halfway through the last century, the use of such formal expert elicitation arose in disciplines such as systems theory and decision analysis . The Delhttp://www.intarese.org. This paper illustrates the wide potential applicability of expert elicitation in environmental health research and provides practical guidance for its implementation. We do not attempt to describe all available methods for expert elicitation . Ins to e.g. ,30,31. W to e.g. ,29,32,33We illustrate the seven step procedure for formal expert elicitation using examples from three existing expert elicitation studies in the field of environmental health research: one on ultrafine particle exposure , one on Ultrafine particles are a very fine (aerodynamic diameter of <100 nm) component of particulate matter air pollution. They may play a role in initiating or aggravating adverse health effects in humans. As yet, insufficient information is available to assess the magnitude of potential ultrafine particle-related health effects. An expert elicitation was organized to assess the evidence for a causal relationship between exposure to ultrafine particles and health effects and to gain insight into the likelihood of several (patho)physiological mechanisms to explain a potential causal relationship. The resources available for this elicitation were sufficient. The study has been planned and organized using the seven step procedure for expert elicitation as presented in , which aBy genetic modifying plants, it is possible to develop crops that are able to break down a particular herbicide, which makes the crops resistant to that herbicide. As such, weed control can be made simpler, as herbicides can be applied without damaging the crops themselves. However, such genetic modifications might have partly unknown and potentially adverse effects on agricultural and cultivation practices, and potentially also on public health. An expert elicitation was organized in order to obtain qualitative and quantitative information regarding the uncertainties present in agricultural risk assessments on GM oilseed crops (canola or rapeseed). No formal protocol was used during this elicitation, as none existed for the broad concept of uncertainty employed here. The available resources were limited.When broiler-chicken carcasses are processed, there is a risk that pathogens such as Campylobacter are transmitted. When the broiler-chicken meat is consumed, this may lead to Campylobacter exposure in the population, posing a serious health threat. A formal expert elicitation study was organized to quantify model parameters for a risk assessment model describing the transmission of Campylobacter during broiler-chicken processing in the Netherlands. The model aims to assess exposure of the Dutch population to Campylobacter as a consequence of the consumption of broiler-chicken meat, as well as to compare the effects of various intervention strategies to reduce this exposure. Experimental data on the model coefficients were not available, and experts did not have enough information to reasonably quantify these values and their uncertainties. Instead, they were asked to provide probability distributions for other quantities with which they were sufficiently familiar to render a judgment. From these estimates, the necessary model parameters could be derived using probabilistic inversion techniques . The orgFigure Expert elicitation is one of several methods to deal with uncertain information in IEHIA. Other methods include for example modelling missing data, scenario analysis or sensitivity analysis . The chaThe typology described here is useful in the scientific analysis of uncertain environmental health issues. When the results of such analyses are communicated to policy makers or other stakeholders, further uncertainty can derive from vagueness or ambiguity in the wording and presentation of the results. This type of uncertainty, referred to as linguistic uncertainty  or linguThe typology of uncertainty, which we have presented more extensively earlier , disting1) The location of uncertainty specifies where in an IEHIA uncertainty manifests itself. Different locations can be distinguished: the context of an assessment, the underlying model, the parameters and the input data.The context of an IEHIA relates to the definitions and boundaries of the assessment: what are we assessing? This question is addressed in the 'issue framing' phase of an assessment  models may be used to represent this structure graphically. They are primarily designed in the 'issue framing' phase of the assessment and further specified in the 'design' and 'execution' phase Figure  , are used in statistical analysis of a model in the 'execution' phase of IEHIA Figure . Uncertax emission data.Input data uncertainty describes uncertainty in datasets used in the 'execution' phase of IEHIA, such as exposure data or disease incidence data. Expert elicitation can be used to estimate the uncertainty of existing datasets. Walker et al. have used expert elicitation to obtain distributions of benzene concentrations . Similarx emission data, addressed both epistemic and ontic uncertainty. The latter, referred to as aleatory uncertainty in their report, was for example found in variability of numbers of cars.2) The nature of uncertainty characterizes whether the uncertainty is primarily caused by incomplete knowledge (epistemic uncertainty) or by intrinsic properties of the system itself (ontic uncertainty). The latter refers to natural variability of a system, such as weather conditions or activity patterns. Most of the examples mentioned so far are examples of epistemic uncertainty: there is a lack of knowledge and expert elicitation is used to fill the gap. However, ontic uncertainty can also be the subject of an expert elicitation, if its extent is poorly known. For example, Titus and Narayanan  have est3) The range of uncertainty gives an indication about whether an uncertainty can be expressed in statistical terms (i.e. as a subjective probability distribution), or in terms of scenarios . This range of uncertainty affects the format of information to be elicited, e.g. as subjective probability density functions or as relative likelihoods of scenarios. The examples about parameter uncertainty ,50,51 me4) Recognized ignorance deals with aspects of uncertainty for which we cannot establish any meaningful estimate. In such cases, expert elicitation can be used to give insight into what is not known and to what extent this is considered important. Krayer von Krauss  identifi5) Methodological uncertainty reflects weaknesses in methodological quality of (part of) an assessment. This may relate to its theoretical foundation, empirical basis, reproducibility or acceptance within the peer community. Expert elicitation can be used to identify or prioritize areas of potential methodological uncertainty. Jelovsek et al.  have use6) The final dimension of uncertainty distinguished here is value diversity among analysts, which relates to personal values and normative judgments held by scientists . Value diversity occurs when different, potentially valid choices can be made about assumptions in an assessment. The assessors making these choices may have different normative values and hence make different choices . The expResources (time and money) often limit the scope of an expert elicitation: how many experts can be approached; can experts be compensated for time and additional expenses; can international experts be invited, etc . In geneThere is no absolute guideline on which to base the number of experts to be invited. According to a panel of expert elicitation practitioners , at leasResource-wise, personal interviews generally consume more time of those organising the expert elicitation, but less money, as compared to group elicitation sessions. The potential benefits of group interaction include sharing of knowledge and better appreciation of different disciplinary viewpoints . On the Information can be elicited from experts in various ways: by conducting interviews, by having questionnaires filled out, or by using specific software ,59-64. IIn the ultrafine particle elicitation , expertsWhich experts take part in an elicitation can greatly affect its outcomes and their acceptability in the wider community, so the selection of experts requires careful consideration ,13,66. TIn general, three types of professional experts can be distinguished: generalists, subject-matter experts, and normative experts ,17. GeneIn the ultrafine particle elicitation , expertsWhen there is a high degree of value diversity (see 'Typology of uncertainty'), when there are high stakes involved, or when results need to be accepted by a wide peer-community, it is particularly important to have a well-balanced expert panel. Opposing views need to be justly represented in the panel and experts should preferably not have strong commitments to a particular outcome. In order to enhance such balance, it is recommendable to use a formal selection procedure ,15, as wThe elicitation of information from experts hinges on the availability of expertise in the scientific community. Experts cannot make up knowledge that does not exist yet in one form or another. However, when issues are highly uncertain, controversial, unquantifiable or associated with potentially irreversible damage; or when decision stakes are very high ,70, therThe elicitation protocol contains the questions to be asked during the elicitation and the desired format for the answers. Expert elicitation can be used both for quantitative and qualitative estimates, as well as for the construction or evaluation of conceptual  models.Quantitative estimates are often elicited for parameter and input data uncertainty (see 'Typology of uncertainty'). These values are necessary in the execution phase of the assessment, when the relevant models and analyses are run. Estimates are often expressed in probabilistic terms , such as used in the Campylobacter-elicitation . ExpertsCommonly, quantitative information is elicited in the form of a number, its unit (e.g. grams or euros) and its uncertainty. The NUSAP approach  has beenContext and model structure uncertainty (see 'Typology of uncertainty') can usually only be addressed in a qualitative way: to determine which data and variables are relevant for analysis; to judge which analytical methods are appropriate; or to assess which assumptions are valid, as elicited in the GM-elicitation . Such quConceptual models can be used to graphically represent the causal relationships between different variables. The elicitation of conceptual models (model structure uncertainty) can be aided by using group model building techniques . These hSome experts may be better at making valid judgements than others. However, it is often difficult to check such performance of the experts, as the 'true values' of their estimates are unknown. Therefore, so-called seed variables can be included. The actual (measured) values of these seed variables are unknown to the experts, but known to the analysts. The performance of the experts on assessing these variables can be used as a proxy for their performance on the query variables, which are the variables of actual interest. Seed variables were for instance included in the Campylobacter-elicitation   motivations for their judgments and identify issues that affected them. This reduces the chance of heuristics and biases to remain unrecognised, and increases the proper interpretation of final results and potential outliers (see below). All three illustrative elicitations have encouraged experts to provide such supporting information.Biases and heuristics [uristics ,26,78-81People use various heuristics when judging uncertain information . Some ofBias in response may also results from motivational bias. This occurs when the response of an expert is influenced by factors such as moral or professional responsibility, legal liability or peer credibility. For topics with a high level of value diversity, experts can be particularly prone to motivational bias, because they may want to influence the outcome of the elicitation . This miExperts can be provided prior to the elicitation with the program for the expert elicitation and the protocol with the questions to be posed (see previous paragraph). In addition, background information about the IEHIA and the subject of the elicitation can be provided to the experts in a so-called briefing document. The information provided in this briefing document should balance potential disparate and disciplinary views , and may contain any of the following elements ,12:- Outline of the nature of the problem and the uncertainties related to it, including the conditions on which the information is to be elicited;- Key literature; optionally inviting experts to add missing papers;- A  summary of the literature. This might however unintentionally stimulate the experts to use primarily the provided material in their judgment;- Information about the elicitation procedure;- Information about heuristics and biases (see above section).The organizers of the ultrafine particle expert workshop  providedAt the start of an expert elicitation session, typically an introduction will be held about the field of interest and the purpose of the meeting in order to familiarize the experts with the subject matter. During this introduction, the uncertainties at hand and the elicitation format can be discussed. Experts can be informed about what is expected from them and how results will be used and distributed. It is important for this purpose to be communicated clearly to the experts and adhered to in the follow-up of the elicitation. Experts may, for example, be willing to make judgements if these are to be used for purely academic purposes, but not if they are meant to form the basis for policy regulations or court decisions.If quantitative estimates are to be made, the use of training questions is advised, because most experts are unfamiliar with quantifying their degree of belief in terms of probabilities. These test questions can also be used to explain and discuss the format of the elicitation . ExpertsMeans to guide experts in correctly synthesising their judgements in an expert elicitation have been described in previous sections. Possible explanations for differences in judgments between experts include  (a) diffPost-elicitation feedback can be given instantaneously (as in computer-assisted elicitation) or delayed (e.g. on paper) and may serve multiple aims. First, it enables experts to check whether their results reflect their thoughts, and revise if necessary. Second, it stimulates discussion, as individual results can be shown in relation to the judgements of others, in order to identify interpretation differences and potential mistakes. However, this might (un)consciously stimulate experts with extreme ratings to move towards what most others reported, which could result in unwanted regression to the mean.The elicitation on ultrafine particles consisted of a round in which experts individually gave a first estimate, followed by a group discussion and then a final individual rating. Any changes in ratings between rounds were motivated by the experts. These changes were considered to be mostly an effect of contemplating new arguments or of a more harmonized interpretation of the question, rather than of 'peer pressure' or anchoring . HoweverExpert judgments can be summarized into one single estimate, or they can be presented individually. There is no consensus among scientists about the conditions under which aggregation is warranted and if so, in what way ,82,83. IIf quantitative estimates such as probability density functions are to be combined into one final estimate, assessors have the choice of various methods to weigh the individual estimates of the experts. One of the most obvious is the equal weighting scheme, in which the estimate of each expert is counted equally in the summary estimate. However, various more complex aggregation processes can be applied ,58,84, wAs an alternative to performance based on seed variables assessments, experts may be qualified according to some other consistency test (see section on format of questions); their number of scientific publications or peer-nominations ; or accoInstead of statistical aggregation, one might also attempt to derive a summary estimate by generating agreement among the experts   models. Expert elicitation may help to structure a problem and can be used to focus new research on the most salient uncertain issues. As such, expert elicitation can be applied widely in environmental health research, and provide useful contributions relevant to all four phases of IEHIA. The transparency and reproducibility, and most likely also the quality of the elicited information increases when the expert elicitation is carried out according to a systematic protocol.In this paper, we have outlined a seven step procedure that can be applied to organise a formal expert elicitation in the context of environmental health research, and IEHIA in particular. It is based upon a broad definition of uncertainty and hence widely applicable. The procedure consists of the following seven steps: (1) characterisation of uncertainties, in which the type of uncertain information to be elicited is identified and characterized according to a typology of uncertainties; (2) scope and format of the elicitation, in which the number of experts to be invited and the most appropriate form of elicitation is determined; (3) selection of experts, in which the necessary types of experts and the balance between different disciplines or viewpoints is considered, preferably using a formal selection procedure; (4) design of the elicitation protocol, in which the types of questions and the appropriate format and wording of the questions and answers are determined, taking into account potential effects of heuristics and biases; (5) preparation of the elicitation session, in which a protocol for the elicitation session is developed and background information is distributed; (6) elicitation of expert judgments, in which the different phases of the actual elicitation  take place; and (7) possible aggregation and reporting, in which the expert judgements may be aggregated and the results of the elicitation are reported. Two recent expert elicitations ,65 relatWe have discussed how the types of uncertainties considered, the intended use of the elicited information, and the resources available affect the design and execution of the seven step procedure. First, the type of uncertainty to be elicited can be characterized using an uncertainty typology in the first step. This characterisation subsequently affects choices about the types of experts to invite, and the format of the questions and elicitation session . All of these may greatly differ if the uncertain information to be obtained is qualitative or quantitative, subjective or objective, controversial or undisputed, mono- or multidisciplinary, etc. Second, the intended use of the elicited information influences the design of the expert elicitation. If the information is to be used in the issue framing phase of an assessment, a multidisciplinary panel might be needed (step 3). Expert elicitation may then be used to structure a problem, identify its boundaries, explore different viewpoints or set a strategy for assessment. Such an expert panel usually engages in a rather open discussion, in which consensus is not necessarily strived for. On the other hand, if the estimates are to be used in the execution phase of assessment as a parameter in a mathematical model, then quantification and some aggregation of individual judgements is likely to be required . Finally, the available resources determine the scope of the elicitation (step 2). The seven step procedure presented here is designed for use in IEHIA. It is primarily meant for environmental health related topics and involvement of scientific professionals. Even though the protocol also provides useful information for application outside this primary field, it should not be seen as a universal protocol. As mentioned earlier, for use of expert elicitation in business or court, different procedures and expert qualifications are likely to be necessary . Also, tNotwithstanding the potential advantages of using expert elicitation, the method has regularly been subject to criticism and debate. The following issues have been mentioned as sources for criticism against the original Delphi method ,9, but tWhen there is a high level of value diversity, for example for controversial issues such as genetic modification, some people might be especially sceptical towards the use of expert elicitation. Of course, this is particularly so when the results of the elicitation do not agree with their personal views. In general, people think that their own judgments are less prone to biases than those of people holding opposing views ,92. In aOf course there are 'real' limitations to the applicability of expert elicitation. Results provided by expert elicitation represent the synthesis of opinion of a particular group. They can thus not be used to forecast the response of a larger population or even of a different expert panel. Also, experts cannot 'make up' knowledge, when no expertise -in one form or another- is available. Thus, expert elicitation is only limitedly applicable to issues that are extremely uncertain and controversial. In addition, this paper outlines other limitations, such as those related to the effects of heuristics and biases, the criteria about who is considered to be an expert, and the aggregation of results.Formal expert elicitation is one of the only options to synthesize scientific knowledge when the development of policy cannot wait for conclusive scientific evidence to become available. The seven step procedure presented in this paper, which draws from several existing protocols for expert elicitation ,29,32,33AH: Arie Havelaar, personal communication, March 2009; DALY: Disability Adjusted Life Years; DDT: dichlorodiphenyltrichloroethane; EU: European Union; GM: genetic modification/genetically modified; IEHIA: Integrated environmental health impact assessment; Intarese: Integrated Assessment of Health Risks of Environmental Stressors in Europe; IPCC: Intergovernmental Panel on Climate Change; Iqarus: Identification, Quantitative Assessment and Reduction of Uncertainties in burden of disease estimates for environmental Stressors; MKvK: Martin Krayer von Krauss, personal communication, March/April 2009; nm: nanometer; NOx: nitrogen oxides; NUSAP: Numeral, Unit, Spread, Assessment and Pedigree; RIVM: Rijksinstituut voor Volksgezondheid en Milieu ; U.S.: United States.The authors declare that they have no competing interests.PS, ABK and JPS studied the expert elicitation literature and designed the seven step procedure. ABK studied existing environmental health related expert elicitation studies in order to illustrate the procedure, and drafted the main manuscript. EL contributed expertise on the potential use of expert elicitation in IEHIA. All authors read and approved the final manuscript.We are grateful to prof. dr. ir. Arie Havelaar and dr. Martin Krayer von Krauss for sharing the details of their expert elicitation studies. In addition, we would like to thank prof. dr. Bert Brunekreef for his useful comments to earlier versions of this manuscript, and the three reviewers for their valuable suggestions. The work was funded through the RIVM Strategic Research Project IQARUS and through the EU 6th Framework Project INTARESE.