Evaluating the impact of federally funded research with a broad, methodical, and objective approach is important to ensure that public funds advance the mission of federal agencies.We aimed to develop a methodical approach that would yield a broad assessment of National Toxicology Program’s (NTP’s) effectiveness across multiple sectors and demonstrate the utility of the approach through a case study.A conceptual model was developed with defined activities, outputs (products), and outcomes  and applied retrospectively to NTP’s research on hexavalent chromium (CrVI). Proximal outcomes were measured by counting views of and requests for NTP’s products by external stakeholders. Intermediate outcomes were measured by bibliometric analysis. Distal outcomes were assessed through Web and LexisNexis searches for documents related to legislation or regulation changes.The approach identified awareness of NTP’s work on CrVI by external stakeholders  and citations of NTP’s research in scientific publications, reports, congressional testimonies, and legal and policy documents (intermediate outcome). NTP’s research was key to the nation’s first-ever drinking water standard for CrVI adopted by California in 2014 . By applying this approach to a case study, the utility and limitations of the approach were identified, including challenges to evaluating the outcomes of a research program.This study identified a broad and objective approach for assessing NTP’s effectiveness, including methodological needs for more thorough and efficient impact assessments in the future.Xie Y, Holmgren S, Andrews DMK, Wolfe MS. 2017. Evaluating the impact of the U.S. National Toxicology Program: a case study on hexavalent chromium. Environ Health Perspect 125:181–188; http://dx.doi.org/10.1289/EHP21 These efforts communicate NTP’s work to the public and any interested groups (stakeholders). To evaluate the impact of NTP’s work, we sought methods to assess the effectiveness of NTP’s science at advancing toxicology and being translated to public health decision-making.For more than 35 years, the National Toxicology Program (NTP) has conducted research, testing, and analysis activities and has disseminated information about potential health hazards in our environment. As the largest government program in toxicology, NTP has studied more than 2,800 substances for a variety of health effects, developed numerous new methods and tools, and published over 600 reports and monographs . NTP staIn 1978, NTP was established within the U.S. Public Health Service in response to growing scientific, regulatory, and congressional concerns that many human diseases and disabilities are linked to chemical exposures . NTP wasEvaluating the impact of federally funded research using a broad and methodical approach is necessary to ensure that public funds are advancing the mission of federal agencies. The impact of federally funded research has been evaluated in a number of studies, supported by public universities, private organizations, and federal agencies , 2012. IFederal Register notices for references to its work and through other ad hoc methods. Based on the literature, we aimed to develop a more formal and methodical approach that would yield a broad assessment of NTP’s effectiveness across multiple sectors and demonstrate the utility of the approach using a case study of NTP’s research on hexavalent chromium (CrVI).Prior to this project, NTP assessed its impact by tracking regulatory actions through Determining the ideal time window to perform an evaluation requires a balance between the availability of records and sufficient lag time to see the full impact of the work. Choosing a project that was completed many decades ago may allow a more thorough search for distal outcomes, such as changes in public health; however, there may be a lack of correspondence and electronic records to provide evidence for proximal and intermediate outcomes. In research fields, there seems to be a rise in mean citation counts for roughly the first 5–10 years after publication . Thus, tUsing a logic model approach , we defiThe logic model accounts first for the main activities in NTP’s research, including the nomination of CrVI to NTP for study, the conduct of studies, and the peer review of NTP study results. These activities led to a number of products, including journal articles and reports. While there were tangential products from NTP studies, such as fact sheets, this assessment focused on NTP’s scientific publications.The NTP Electronic Library stores research data, documents, and other information for NTP studies. Information about the nomination of CrVI to NTP for study was obtained from the NTP Electronic Library and nomination summary pages , 2014c, Proximal outcomes were associated with stakeholder groups  gaining knowledge or awareness of NTP reports and publications. This direct impact was represented by user data from Web-page views of NTP reports and requests for information about NTP’s work. For the CrVI case study, the number of Web-page views for NTP reports was obtained from server logs for July 2011 through October 2014. Web pages with Webtrends code  can tracIntermediate outcomes, or citation metrics of NTP publications , were used as a metric of informing science in stakeholder groups. For NTP journal articles  citationWe used the Query, View, and Report System (QVR), a tool that enables Department of Health and Human Services staff to search, view, and retrieve information from NIH databases about grant applications and awards , to idenUsing Lexis Advance® , a platfThe Google and Lexis Advance® search approaches described above were also applied to identify distal outcomes, or references to NTP’s work that related to changes in regulation(s). All documents were screened manually for explicit or implicit references to NTP’s work on CrVI. Explicit references were found in footnotes or reference lists, while implicit references were found in the text. For example, in California’s “Initial Statement of Reasons” to adopt a maximum contaminant level for CrVI in drinking water, NTP’s work was not present in the reference list . InsteadIn 2000 and 2001, the California Congressional Delegation, California Environmental Protection Agency, and the California Department of Health Services nominated CrVI to NTP for study , 2014c. a) a series of three final reports on the reproductive effects of potassium dichromate administered in the diet in rodents (b) the NTP “Report on Carcinogens” listing of CrVI compounds as “known to be human carcinogens” since 1980  , one tec rodents , 1997 annce 1980 .Web statistics and external requests showed stakeholders had direct awareness of NTP’s products. From 2005 through 2014, there were 14 requests related to NTP’s work on CrVI (see Table S1). These requests came from different groups including industry, U.S. EPA, U.S. House of Representatives staff, a New Jersey state agency, and academia. NTP did not have the capability to track Web-page views until July 2011. While Web statistics are not available prior to July 2011, the NTP toxicity report TOX 72 and NTP technical report TR 546 received 16,471 and 22,634 views, respectively, from July 2011 through October 2014. About 87% of Web-page views for TR 546 and 77% of Web-page views for TOX 72 were external, meaning the views came from non-NIH and non-NIEHS IP addresses.With respect to journal articles that cited NTP’s publications, all but one used NTP’s work contextually and informatively , such asNIH grants were analyzed to determine if NTP products were used as support for future research. Of the roughly 100 NIH funded grants with chromium in the title or abstract (2000–2014), 4 referenced NTP’s work. Two references were for TR 546. The other 2 references were for the “Report on Carcinogens” listing of CrVI compounds.U.S. states, federal agencies, and nongovernment groups cited NTP’s work to identify CrVI as a hazard (see Tables S2–S4). The World Health Organization’s International Agency for Research on Cancer and International Programme on Chemical Safety also cited NTP’s work with CrVI (see Table S5). Academics, U.S. federal agency leaders, a U.S. senator, and members of nongovernment groups have included references to NTP’s work in congressional hearings . In addition, NTP has informed the science in three lawsuits (see Table S6).Federal Register notice that draft TR 546 would be peer reviewed and posted publicly, the office of CA Representative Adam Schiff asked for the conclusions of NTP’s toxicology and carcinogenicity studies and announced that NTP had released the draft. Following the release of TR 546 and other NTP publications, California released several reports related to CrVI  and Occupational Safety and Health Administration (OSHA). In 2011, DoD issued a final rule to amend the Defense Federal Acquisition Regulation Supplement to minimize the use of materials containing CrVI in items they acquire (see Table S7). In 2006, OSHA amended the existing standard that limits occupational exposure to CrVI (see Table S7).We sought to develop a formal and methodical approach that would yield a broad assessment of NTP’s effectiveness across multiple sectors and demonstrate the utility of the approach through a case study of one of NTP’s research projects. Our approach was to use a logic model and apply it retrospectively to a case study of NTP’s research on CrVI. Because this case study is for assessing NTP’s effectiveness, the logic model focuses on activities, outputs, and outcomes; inputs  were not evaluated.We conducted an evaluation of proximal, intermediate, and distal outcomes of NTP’s work in multiple sectors and found challenges in each step. For proximal impacts, we learned that NTP did not track Web-page views for its reports before July 2011. Thus, we are not able to determine the number of downloads for the main products of NTP’s CrVI studies immediately following publication, when interest would presumably be the highest. This also makes it difficult to gauge if the Web-page views for TR 546 and TOX 72 are high or low compared to other NTP projects. For reports published after July 2011, we would be better able to make those kinds of comparisons. For future assessments, it would be important to track Web statistics immediately following the posting of NTP publications online. In addition, it would be worth obtaining information on the number of times NTP articles are downloaded from journal publishers, when available.For intermediate impact, we learned that NTP’s technical and toxicity reports are not completely indexed in subscription-based, scientific abstract and citation databases, such as Scopus and Web of Science. As a result, in order to identify publications that cite the NTP reports TR 546 and TOX 72, we searched PubMed Central (PMC). PMC is a free archive of full-text, biomedical and life science journal articles that enabled us to search for articles with NTP publications in the references. The number of papers in PMC is limited. There are about 3.7 million articles in PMC , while SAssessing whether NTP efforts led to distal outcomes had three main challenges: lag time, external factors, and attribution. First, the lag time between when outputs occur and a specific outcome is implemented can be extensive. With respect to CrVI, the lag time from the release of the draft report on NTP’s 2-year studies of CrVI in drinking water (TR 546) to approval of the maximum contaminant level of CrVI in drinking water by California was 7 years. Any improved public health impact resulting from this regulation may not be known for many years to come.The second challenge is that the opportunity to effect a change is not always straightforward and may rely on external factors. Whether NTP’s work will lead to drinking water regulation changes in other parts of the United States as in California is beyond NTP’s purview. For example, in Washington State, TR 546 was cited as a supportive scientific development in “Draft Revisions Model Toxics Control Act Method A Groundwater Cleanup Levels” ; howeverFinding clear attribution is a third challenge that applies to both intermediate and distal outcomes. While many reports, lawsuits, and other works use NTP publications on CrVI, they often lack consistent or clear citations either in-text or in a reference list. Thus, key word searches for citations to NTP’s work on CrVI often find false positives that must be manually eliminated or may miss true positives. Unlike NIH grants, which have unique identifiers that help NIH track the impact of grants , there aFinally, for all outcomes, it is difficult to understand the exact nature of NTP’s contribution even when NTP’s work is clearly included in a reference list. Many factors are involved in producing reports, lawsuits, regulations, and other works. It is impossible to assign a level of importance to NTP’s work for any action by an external group; we can only note when NTP had a contribution. When linkages from proximal to distal outcomes can be made, there is a stronger case that NTP’s work made a significant contribution. As was the case for the impact of NTP’s CrVI studies on activities in California, our evaluation methods were able to identify proximal, intermediate, and distal outcomes. In California, there was direct interest and use of NTP’s work in scientific documents, and this led to the reference of NTP’s work in the “Initial Statement of Reasons”  for a maIn carrying out this evaluation, we identified a need for better tools to enhance assessments of a research project’s potential research impacts. While obtaining the number of citations for NTP reports and journal articles in scientific publications is straightforward, performing content analysis to determine how NTP’s work was cited is much more resource intensive. Developing text-mining tools to scan the full text of each citation would allow quicker content analysis to be performed. Better methods are also needed to search for references in grants to determine if NTP’s work was used to support the proposed new projects. Currently, the references of NIH grants are not searchable through the QVR tool. Indirect searches have to be performed, such as searching for grants that relate to chromium and then searching relevant grant files for a list of references. This indirect search method is likely to yield an incomplete list of citations. Finally, there is a need for Web-mining tools to increase the efficiency and completeness of searches for explicit and implicit citations of NTP’s work in documents from state agencies, federal agencies, nongovernment groups, and international groups. Searching through the Web for outcomes was a time-consuming manual process that highlighted the need for automation. With Web- and text-mining software, NTP may more thoroughly and efficiently identify potential impacts. If such tools can be developed, impact searches could be performed on a regular basis and stored in a database for public access to enhance the communication of NTP’s work.The purpose of evaluating NTP’s impact is 3-fold: (1) collect and analyze data to learn how NTP is achieving its goals; (2) implement improvements based on what is learned; (3) following implementation of these improvements, use the methods developed here to evaluate whether the changes have helped NTP achieve its goals. For example, this case study identified that TR 546 was used more often than NTP journal articles in documents by nongovernment organizations, federal agencies, state agencies, and international groups . There are limited data for citations of TR 546 in scientific literature due to inconsistent indexing in subscription-based, scientific abstract and citation databases. However, the available data suggest that TR 546 has more significant citations in scientific literature than NTP journal articles. NTP technical reports, like TR 546, are more comprehensive with regard to study data and results than to journal articles. NTP journal articles usually focus on a deeper discussion of a subset of the data and results than are available in corresponding technical reports. Thus, it is possible that stakeholders have greater interest in NTP technical reports because they are a more complete reporting of data and results. It would benefit the goal of NTP to increase the discoverability of its data and results and to harmonize the format of this information for improved usability in databases like the NTP Chemical Effects in Biological Systems (CEBS) database . While aWe developed an approach with methodical steps to evaluate a research project’s proximal, intermediate, and distal outcomes. The approach we developed can be applied to future projects for assessing NTP’s effectiveness. The logic model is adaptable to different NTP research programs and could cover outcomes not measured in this case study if relevant to the research topic. By applying this approach to a case study, data and methodological gaps were highlighted. As a result, NTP is seeking to improve indexing of its reports in online scientific citation and abstract databases. NTP is also working to submit full-text reports to PubMed Bookshelf , which p(940 KB) PDFClick here for additional data file.


School is an ideal setting to promote and increase physical activity (PA) in children. However, implementation of school-based PA programmes seems difficult, in particular due to schools’ focus on academic performance and a lack of involvement of school staff in program development. The potential cognitive and academic benefits of PA might increase chances of successful implementation. Therefore, the aim of this qualitative study was: (1) to explore the perceptions of teachers and principals with regard to implementation of additional PA aimed at improving cognitive and academic performance, and (2) to identify characteristics of PA programmes that according to them are feasible in daily school practice. Twenty-six face-to-face semi-structured interviews were conducted with primary school teachers (grades 5 and 6) and principals in The Netherlands, and analysed using inductive content analysis. Teachers and principals expressed their willingness to implement additional PA if it benefits learning. Time constraints appeared to be a major barrier, and strongly influenced participants’ perceptions of feasible PA programmes. Teachers and principals emphasised that additional PA needs to be short, executed in the classroom, and provided in “ready-to-use” materials, i.e., that require no or little preparation time . Future research is needed to strengthen the evidence on the effects of PA for academic purposes, and should examine the forms of PA that are both effective as well as feasible in the school setting. Schools are considered an ideal setting for the promotion of a healthy life style among children, such as regular physical activity (PA) . It is wStudies on the effectiveness of PA interventions are increasingly accompanied by process evaluation studies to gain insight into the feasibility of the intervention, and barriers and facilitators of the implementation process . A recenAlthough process evaluation studies provide valuable information to adapt interventions developed by health professionals, school professionals are rarely involved in the development of school-based PA programmes ,8,12. AsIn particular, we lack knowledge on teachers’ and principals’ perceptions of feasible PA programmes in schools with regard to how often, how long, and in which setting  PA can be implemented . Some prPrevious school-based PA interventions have mainly focused on promotion of children’s health; however, several studies reported that not all teachers and principals consider health promotion their responsibility ,16,19. AThe current study has a qualitative research design with an inductive approach; we identified themes from the data, which allowed us to gather information from teachers and principals’ perspectives . We condA purposive sampling method was used to recruit participants . SelectiTeachers and principals of 53 regular primary schools were approached by email and follow-up phone calls to participate. We emphasized in our recruitment email that teachers and principals who are enthusiastic as well as those who are sceptical on the subject “PA and academic performance” were welcome to participate in our study. Twenty schools (38%) did not respond. Nineteen schools (36%) were not able to participate due to busy school schedules (n = 8), an overwhelming number of requests to participate in research projects (n = 3), already participating in other research projects (n = 1), no interest in the research topic (n = 1), or without specific reason (n = 6). Fourteen schools (26%) agreed to participate in the study with either their principal(s), their teacher(s), or both.All participants received written information of the purpose and procedure of the study and signed informed consent prior to participation. The Medical Ethical Committee of the VU University Medical Center Amsterdam concluded that the study does not fall within the scope of the Medical Research Involving Human Subjects Act, and approved the study protocol (2014.363).Two interview guides  were developed following the recommendations of Creswell . The intWe developed a protocol and organized a training day to make sure that the interview guides were applied equally by our four interviewers. The interview guides were pilot tested in two preliminary interviews and evaluated with the interviewers. Based on the results of the pilot testing, the order of the questions was slightly adapted.The four trained interviewers conducted 26 semi-structured interviews in Dutch concurrently between April and July 2015. At each school, one or two principals and/or one to four teachers were interviewed individually. The interviewers visited participants at their school, where the interviews were held in a quiet, private setting. Before the start of the interview, information on the aim of the study, the duration of the interview, and anonymity and confidentiality was provided. The importance of participants’ own opinions, experiences and ideas was emphasized.All interviews were audio- or video-recorded and transcribed afterwards by two interviewers and Vera van den Berg. The duration of the interviews ranged from 9 to 35 min (mean 20 min) for teachers and 17 to 26 min (mean 19 min) for principals. The raw data consisted of 70 transcript pages from the principal interviews and 90 pages from the teacher interviews .The data analysis started after the data collection. Inductive qualitative content analysis was used to systematically identify and extract categories and themes from the data ,27. ThreIn step 1 transcripts were read repeatedly, relevant text fragments were selected, and preliminary codes were allocated. Codes were supported by short phrases of the content of the text fragments, and coded transcripts were checked and revised repeatedly in order to make sure that identified codes were corroborated across the interviews . A shortFor example, categories “small classroom size”, “many students in the classroom”, and “full schedules of the external sports hall”, were grouped together under the subtheme “space constraints”, and theme “barriers”.In total, 15 fifth and sixth-grade teachers and 11 principals of the 14 schools participated. Schools were located in urban (n = 8), suburban (n = 2), and rural (n = 4) areas across The Netherlands. Teachers had a mean age of 41 years (range 25–61); principals were on average 48 years old (range 34–59). Work experience ranged from 1–37 years (mean 13.4) and 1–30 years (mean 11) in teachers and principals, respectively. Of the participants, 80% were female.Although data saturation was reached after analysis of approximately three quarter of the interviews, we analysed all interviews. One interview with a principal was excluded, since the interviewer used the teacher version instead of the principal version of the interview guide.The results of the interviews were organized around four key themes: attitude towards PA, barriers towards additional PA, characteristics of feasible PA programmes, and requirements for implementation of PA programmes in school. The findings were supported by quotations from the interviews, translated from Dutch into English. All teachers and principals showed a positive attitude towards PA, and gave several reasons why they believed PA is important for their students.Academic benefits. The most common belief of all participants was that PA positively influences children’s ability to learn. Teachers indicated from experience that the majority of children need to move in order to release their energy and to recover attention. For example, teacher #10 noticed “Even if you just do a little bit of exercise, because a lot of children really need it, after that you can start up your attention span again”.Physical health benefits. Almost all teachers and principals mentioned the importance of PA for physical health, and some  believed that “if you’ve got a healthy body, you perform better cognitively” (teacher #3). However, some participants argued that their “core business is getting children to learn” .Social and emotional benefits. A group of five teachers and five principals also observed the benefits of PA for children’s social skills and self-confidence. “They learn to function in a group, to collaborate, learn to deal with losing… But also to be in the limelight if you happen to be good at basketball and maybe can’t learn very easily, you notice that a child gets a kick out of it” .Willingness. Overall, teachers and principals were positive towards implementation of additional PA in school. However, they mentioned that their willingness depended on the perceived needs of children to move and the efficiency for learning. Teacher #7 mentioned for example: “If it isn’t showing any results, I say stop doing it and leave them sitting”. In addition, the willingness was restricted to certain forms of PA , lack of financial support, and space constraints.Time and priority. All teachers and principals emphasized a lack of time as a major barrier hindering implementation of PA programmes, and this topic recurred repeatedly throughout all interviews. The perceived lack of time was related to high demands of the government and parents on academic performance, competing priorities, and full school schedules. Teachers expressed that they feel overwhelmed by several responsibilities on top of teaching, like administrative tasks. Overall, it appeared that, according to teachers and principals, implementation of additional PA currently has no priority in school.“Sometimes you’re trapped between the everyday rush, inspection reports, entrance exams, and the things that have to be done… The question is, whether that’s possible, and whether it’s desirable in our system, our school system in The Netherlands… because you’re judged on how good your arithmetic and spelling results are” (teacher #5).Space constraints. According to the principals, implementation of additional PA in form of additional physical education classes is hindered by the available space. Principal #5 mentioned for example: “Then you have to go to the gym or the sports hall. And you have to take the timetable into account, because other schools also use the same facilities”. Concerns with regard to limited space in the classroom for PA were reported by one third of the teachers: “Make allowances for the fact that the space available—you can see that here—is often limited” (teacher #15).Financial constraints. A lack of financial support from the government was mentioned by more than half of the principals and a small group of teachers. This barrier was related to hiring a physical education teacher or professional education of classroom teachers, and playground or sport facilities.“We won’t be getting an extra PE teacher, which is a real shame. It would be great, but you need more money from the government. It would be really good if you had money to redesign your playground more effectively” .Teachers and principals’ ideas of feasible forms of PA followed logically from the perceived barriers. The vast majority of participants emphasized that additional physical education classes are not feasible in school. In general, teachers provided more concrete ideas than principals.Setting. In terms of feasibility and relative to the time barrier, teachers reported that PA programmes need to be implemented by teachers themselves in their classroom. Teacher #2 mentioned: “If you have to go to another location all the time, it just won’t work. You’re stuck with the time factor... If I start doing things here in the corridor, the other classes won’t be very happy”. In addition, being active with the whole class was considered important for supervision and safety reasons. “If they start moving around in small groups for example, someone has to be around, otherwise it’s irresponsible” (teacher #7).Duration. All teachers and principals agreed that additional PA needs to be of short duration. Teacher #4 said for example: “You can find 5 or 10 min somewhere during the school day”. According to both groups, a feasible duration varies between 1 and 5, up to a maximum of 10 min.Type. Most teachers (n = 13) and principals (n = 7) considered PA breaks within or between academic lessons feasible. Several teachers suggested to use dance, movement clips, movement games or exercises around the table, such as jumping and running in place. Some teachers mentioned that there are short movement games available, with which they already had positive experiences.Another suggested type of PA was integration of PA with academic subjects . A recurrent example was exercising while performing math sums. Teacher #2 explained: “That’s a way of calculating that focuses on automation, it’s about movement, so every time you do something, you jump”. However, some participants thought it would be difficult to use such methods in the highest grades of primary education (10–12 year olds). A small group of two teachers and four principals also mentioned other forms of being active while learning. For example, games with alternately sitting and standing, or walking to corners of the classroom when answering questions.Furthermore, some teachers (n = 4) and the majority of principals (n = 7) saw opportunities to increase PA levels outside learning hours. They discussed the possibility of stimulating PA during recess, since not all children are physically active during recess time. They suggested to let children choose from activities, or by giving them cards with PA exercises, and providing more playground facilities. Also, PA interventions after school hours were mentioned by six principals. “I also know that there are out-of-school situations where sports instructors are hired to get children to exercise. It would be very interesting to see if we could make it accessible for all children… to create a kind of extended school day” .Materials. Almost all teachers and principals reported that it is important that PA interventions “don’t cost too much time to prepare” (teacher #9). They recommended materials that are ready to use, for example on the digital screen, in a booklet, or in a card box.Usage frequency. The majority of teachers (n = 11) reported that they would implement PA spontaneously/incidentally, when children lost their attention, while a small group  preferred structural implementation.Involvement of children. A quarter of all teachers and several principals pointed out the importance of involving the children when designing PA programmes. They emphasized that PA must be varied, fun, challenging, and appropriate for the age group, which could best be explained by the children themselves. Several teachers reported that “up until the second last year at school (grade 5), things are just fine, it’s easy to make it fun… but in the final year (grade 6), they think it’s more important that it’s cool” (teacher #15).Teachers and principals addressed several prerequisites for successful implementation and sustainability of PA programmes in school.Team support. Everyone emphasized that school-wide team support is essential when trying to implement additional PA in school. Teacher #5 said for example: “We’re fantastic at imagining all kinds of complications ... which is why I say you have to make a statement as a team that you want this. That’s the most important thing”. And after the decision has been made “it’s important to make clear agreements with each other” .Evidence. To establish team support, both groups reported the importance of a clear rationale, supported by evidence about the usefulness of PA for cognition/academic performance. Participants emphasized that providing this knowledge is especially important to gain support from colleagues who are less familiar or willing to use PA. In this respect, principal #1 said: “If a teacher is not convinced it’s going to help and that it’s practical, it doesn’t matter how many programmes you set up, nothing much will change. Even if you say it’s been agreed within the school and we’re going to do it”.Embed and evaluate. Teachers and principals discussed the importance of structurally incorporating PA programmes in the school schedule, such that teachers are aware that they need to implement it. For example, teacher #11 mentioned: “Regular set periods during the week would be great. That you think to yourself, oh yeah, that’s still to come”. In addition, several principals reported that PA programmes must be part of the school policy in order to successfully integrate it in school.Both teachers and principals discussed the importance of regular evaluation within the team to prevent relapse. The majority recommended an internal PA-commission or coordinator responsible for the implementation. Teacher #8 explained: “You can say that everyone takes care of it themselves, but they coordinate it, or propose ideas, or monitor it. Because we all have so much to do that if nobody specifically reminds you, it just doesn’t happen”. The one(s) responsible should, according to teachers and principals, place the topic on the agenda, monitor the implementation, and motivate, inspire and coach teachers.Sharing ideas. Teachers and principals both indicated that no extensive instruction or course for teachers is needed when implementing short forms of PA in the classroom. A single workshop can be helpful to get ideas, but thereafter, team members need to share and exchange ideas, tips and tricks. A part of teachers and principals who had a physical education teacher in their school, suggested that he/she could take on a supporting role herein.Communication with parents. Half of the principals stressed the importance of informing parents when deciding to implement additional PA in school. Principal 7 explained: “You need to communicate properly about why you make the choices you make and what effect they have”.In this study, we aimed to explore the perceptions of teachers and principals with regard to the implementation of PA interventions aimed at improving cognitive and academic performance, and identify characteristics of feasible school-based PA programmes. Our results add new information to the existing literature that has mainly focused on the perceptions of school staff towards the implementation of PA programmes with the purpose of health promotion. The main findings will be discussed in light of the current evidence on the effects of PA for cognitive and academic performance, and can be used in the development of future school-based PA interventions.In line with earlier process evaluation studies, teachers and principals in our study had positive attitudes towards PA, attributed to academic, health, and social-emotional benefits for their students ,17,19,29Despite the expressed willingness, our participants emphasized similar barriers as found in previous studies that evaluated the implementation of PA interventions focusing on health outcomes, with a lack of time as the most prominent barrier ,19,22,31Participants’ feeling of a lack of time due to pressures on academic performance seemed contrary to their expressed willingness to use PA to improve learning, as the latter suggests that PA can help improve academic performance. Our results suggest that teachers and principals perceive PA as a tool that can be used to restore attention in the short-term, rather than a structural intervention that supports them in improving academic performance in terms of students’ grades. The majority of the teachers mentioned that they would implement PA only at times when students needed a break from learning, for example when they lost their focus. This aligns with the literature on the acute effects of PA for cognitive performance, which suggest that PA breaks can provide immediate improvements in for example attention and on-task behaviour ,32.None of our participants related PA to students’ grades or referred to scientific evidence when talking about the academic benefits of PA. This could suggest that the current level of evidence does not convince school staff of the long-term effects of structural PA interventions on cognitive and academic performance ,33, or tTeachers and principals stressed that evidence on the benefits of PA for cognitive and academic performance is needed to gain team support, which they considered crucial when attempting to implement additional PA in school. Participants furthermore reported that the implementation process needs to be organized within the school, under the supervision of an internal PA coordinator or PA commission, which is in line with earlier studies ,31. DespThe perceptions of the teachers and principals in our study were often similar, although there were some differences that align with their different roles. In general, teachers focused mostly on practical issues that they face in their daily routines , while principals also expanded on organizational barriers such as a lack of financial support and restricted availability of space. Additionally, establishment of policy and communication with parents were important to principals. Earlier studies suggest that principals, besides initiating the implementation, have an essential role in supervising and supporting the implementation process ,29. HoweDue to the perceived time barrier, and financial and space constraints, teachers and principals indicated that additional PA moments in the school-setting are most feasible when implemented in the classroom, carried out by the classroom teacher. In line with earlier studies, teachers and principals suggested that PA programmes need to be short and easy to implement ,18. MoreAlthough teachers in the study of McMullen et al.  preferreFurthermore, teachers and principals discussed that PA programmes must be fun, varied, challenging and age-appropriate; factors that have been found previously as facilitators of PA engagement in children and adolescents ,17,39. ALastly, many principals looked for opportunities to implement additional PA outside academic learning hours, for example during recess and after school. This finding is not surprising, considering their role, and again reflect the competing priorities within the school curriculum.Looking at the current line of research, we established a gap between the forms of PA that teachers and principals consider most feasible in school , and the forms of PA that have been examined and proved effective in research studies . We also suggest schools to discuss strategies to lower the perceived time barrier, and look for ways to prioritize the use of PA in school. Herein, the potential benefits of PA for learning could be central. Decisions should be made by both principal and teachers, as team support seems essential for successful and sustainable implementation. We recommend enthusiastic teachers to take a leading role in the promotion of PA in school. Principals can support these teachers and encourage them to take a role as “PA coordinator”. We also recommend principals to provide/organize a workshop on the background and practical use of PA in the classroom. In particular, we suggest to discuss the cognitive and academic benefits of PA perceived by teachers, as well as the current scientific evidence. In this respect, it is important to note that although the scientific evidence needs to be strengthened, we can safely conclude that additional time spent on PA will not negatively influence cognitive and academic performance . This maA strength of our study was that data saturation was achieved, and that we consulted participants from several regions across The Netherlands. Other strengths were the inclusion of the audit trail and the interview guides, which provided transparency and ensured trustworthiness of our study, thereby enabling replication. Also, we provided new information that could be used by researchers in future studies on school-based PA interventions, as well as ideas and suggestions that could be used by schools that are looking for ways to implement additional PA in school. A limitation was that we faced a group of fairly motivated teachers and principals, despite the attempt to recruit and include teachers and principals that were also sceptical on the topic. This was supported by the response rate of 26%, and limited the generalisability of the results to less motivated teachers and principals. However, even in this group of participants, several barriers were apparent that would also be applicable to less motivated school staff. Moreover, our findings may not be generalisable to teachers and principals in other countries with different school systems and cultures. Another limitation was that due to time constraints and short availability of interviewers, we could not start data analysis during the data collection period. Therefore, we were not able to adjust questions during the course of the study, which could have provided us with more in-depth information into some aspects of the topic.In summary, the 5th and 6th grade teachers and primary school principals who participated in the current study had positive attitudes towards PA. They expressed their willingness to implement additional PA in school, in which the academic benefits was an important argument. However, teachers and principals highlighted similar barriers as found in studies on implementation of health promotion-based PA interventions, with time constraints as the most important barrier. Our participants emphasized that feasible PA programmes should be short , provided in “ready-to-use” materials, and be executed in the classroom by the classroom teacher. According to our participants, PA could be best used as break at moments that attention spans decline. Future research on the effects of such PA programmes on cognitive and academic performance is needed to provide policy makers and school staff with clear information on its benefits for school practice. Furthermore, we recommend schools to look for opportunities to implement PA in the classroom, and discuss strategies to lower time barriers and prioritize PA in school.


Lugol’s solution and other preparations containing iodide have for almost a century been used as an adjuvant treatment in patients with Graves’ disease planned for thyroidectomy. Iodide has been shown to decrease thyroid hormone levels and reduce blood flow within the thyroid gland. An escape phenomenon has been feared as the iodide effect has been claimed to only be temporary. Lugol’s solution has many additional effects and is used in other settings beside the thyroid. Still, there are questions of its mode of action, which doses should be deployed, if it should be used preoperative in all thyroidectomies or only in a few selected ones if at all, what is its use in other forms of thyrotoxicosis besides Graves’ disease, and what is the mechanism acting on the vasculature and if these effects are confined only to arterial vessels supporting the thyroid or not. This review aims to collate current available data about Lugol’s solution and other iodide preparations in the management of Graves’ disease and give some suggestions where more research is needed. Lugol’s solution (LS) was developed 1829 by the French physician Jean Guillaume August Lugol, initially as a cure for tuberculosis. It is a solution of elemental iodine (5%) and potassium iodide  together with distilled water. It has been used as a disinfectant, a reagent for starch detection in organic compounds, in histologic preparations, in dental procedures and in diagnosis of cervical cell alterations, the Schiller´s test  binds to e.g. potassium and in this state can be more easily consumed or applied topically. However, often the terms iodide and iodine are used interchangeably. With the development of pharmacologic agents such as antithyroid drugs and with radioiodine therapy, LS is not routine anymore in many countries –6, althoGD is a common autoimmune disease, with typical symptoms as weight loss, heat intolerance, tachycardia and mental disturbances as tiredness and restlessness. There is a female preponderance (4:1 in a first episode of hyperthyroidism) with a peak incidence at 40–69 years . These pThe aims of this review are to provide a summary of currently available knowledge of LS including other iodide preparations in the treatment of GD and give some suggestions in areas where more research is needed.−) is essential for the synthesis of thyroid hormones, T4 and T3. The primary source is food, often salt fortified with iodine. The recommended daily intake is 150 μg in adults, with a preferred increase in pregnant and lactating females [+/I transporter where it is rapidly oxidized by H2O2 to iodine, catalyzed by thyroid peroxidase and is then incorporated into thyroglobulin. In healthy humans who are not predisposed to thyrotoxicosis, increased thyroid hormone synthesis can be induced by high doses of iodine such as iodine containing contrast agents, and after a long-time use of antiseptics such as povidone–iodine swabs [Iodide , and operative times (138 vs. 150 min), were also significantly less compared to controls . The redLS tastes bitter and is also corrosive, and this should be disguised by taking it with a sweet drink such as apple juice. The applied doses come from experience rather than by prospective randomized controlled trials Table . HistoriTakata et al. demonstrated that normalisation of free T4 was more rapid with a combination of LS and antithyroid drugs, than methimazole alone, however, remission did not differ during a 4–5 year follow-up . In anotIn uncontrolled GD, or if adverse effects of antithyroid drugs develops such as agranulocytosis or liver failure, rescue treatment with LS is one option. Radioactive iodine could also be deployed but the fear of thyroid storm if thyroid function exacerbates by this therapy could make LS followed by thyroidectomy a reasonable choice. We recently demonstrated that LS in this setting was effective and decreased both thyroid hormone levels and heart rate with few side effects . In a SwThere are several iodide compounds available. Classically LS is prescribed in a dose of 5–10 drops thrice daily with each drop containing 5–8 mg/drop , 42, or www.medicinenet.com/potassium_iodide-oral/article.htm Accessed 10 August 2017). In the study by Sato et al. adverse effects with a combination of methimazole 15 mg and inorganic iodine (KI tablets) at dose of 38.2 mg per day were less common than in a group of patients treated with methimazole alone with 30 mg daily [www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM080542.pdf Accessed 10 August 2017).LS and other iodide preparations seem to have low frequency of adverse effects. In doses of 1000 times the normal nutritional need, side effects may include: acne, loss of appetite, or upset stomach. More severe side effects are fever, weakness, unusual tiredness, swelling in the neck or throat, mouth sores, skin rash, nausea, vomiting, stomach pains, irregular heartbeat, numbness or tingling of the hands or feet, or a metallic taste in the mouth. (mg daily . We founmg daily . Okamuramg daily . The onlmg daily . Other amg daily , and esoMost studies on LS therapy have been retrospective and often with few participants. Recently the issues on blood loss and vasculature effects were questioned due to the lack of proper randomized trials . HoweverThere is a need for future prospective investigations with a sufficient number of participants but also randomized control trials as well as in patients with mild vs. severe GD. This would enable us to draw firm conclusions if LS, alone or in combination with antithyroid drugs, do affect operative time, blood loss, and surgical complications such as hypocalcemia and damage of the recurrent laryngeal nerve. The efficacy of long-term treatment with LS needs also to be further explored. It is not proven if the iodine background, ethnicity or other factors make the relatively low number of patients escaping from iodide blocking in Japan a phenomenon which should not be feared to the same extent as it has in the past. To explore this investigations in areas with different iodine backgrounds have to be performed with focus on control of GD, degree of escape from therapy and adverse effects. Adverse events need to be better recorded, both in the short and long-term use of LS therapy. The optimal dose of LS also needs to be clarified. Moreover, the mechanisms of LS, especially on the vasculature are intriguing and merit further research. It is fascinating that almost 200 years since this compound was invented, and 100 years since it became a treatment in GD, we still do not fully understand how the effects on the vasculature are mediated. We also do not know if the vasculature effects in GD are present in other thyroidal diseases such as goiter or multinodular diseases, or in other vascular beds in the body.LS has been advocated for almost 100 years in the treatment of GD. It has effects in decreasing thyroid hormones, and possibly in reducing blood flow during thyroidectomy. LS is used both in combination with antithyroid drugs preoperatively in planned thyroidectomies in certain centers routinely, and alone as a rescue therapy if severe side effects to antithyroid drugs have occurred. These effects, especially on the vasculature have to explored further, together with investigations on how common there is an escape from LS-blocking, the mechanisms of action, which doses that should be applied, and registration of side effects.


In an editorial published in this journal, Baltussen et al argue that information on cost-effectiveness is not sufficient for priority setting for universal health coverage (UHC), a claim which is correct as far as it goes. However, their focus on the procedural legitimacy of ‘micro’ priority setting processes , and their related assumption that values for priority setting are determined only at this level, leads them to ignore the relevance of higher level, ‘macro’ priority setting processes, for example, consultations held by World Health Organization (WHO) Member States and other global stakeholders that have resulted in widespread consensus on the principles of UHC. Priority setting is not merely about discrete choices, nor should the focus be exclusively (or even mainly) on improving the procedural elements of micro priority setting processes. Systemic activities that shape the health system environment, such as strategic planning, as well as the substantive content of global policy instruments, are critical elements for priority setting for UHC. To structure our argument, we begin by formulating three specific propositions that we believe are explicit (or implicit) in the editorial:While we endorse the call in the editorial by Baltussen et al3 focus (mainly and unduly) on the provision of cost-effectiveness information (p. 1);International efforts to assist priority setting in healthCost-effectiveness information  does not adequately support countries to make choices about priorities (p. 1); andPriority setting is … a “value-laden political process, in which multiple criteria … are important and [in which] stakeholders often justifiably disagree about their relative importance” (p. 1).4 , while neglecting the broader scope of WHO’s policies, guidance, recommendations, and programmes related to priority setting in health. To the extent that Baltussen et al subscribe to the first proposition, in our view they would be making the error of conflating technical methods (such as WHO-CHOICE), which contribute to the evidence base for priority setting, with the values and principles underpinning a priority setting process.Leaving aside here a detailed discussion of DCP3 (which we nevertheless mention in passing later on), and focusing primarily on World Health Organization’s (WHO’s) role, in the first proposition Baltussen et al seem to be restricting their attention only to “WHO-CHOICE”only cost-effectiveness information is relevant for priority setting. This proposition, if adhered to by Baltussen et al, would constitute a serious misunderstanding of the aims of WHO-CHOICE.If on the other hand they do not subscribe to this view, they nevertheless seem to be criticizing a cost-effectiveness programme for producing cost-effectiveness information. The second proposition, therefore, amounts to the assertion that the provision of information on cost-effectiveness  is somehow equivalent to a recommendation that This brings us to the third proposition, namely that priority setting involves “multiple criteria,” that it is “value laden,” and that “stakeholders often justifiably disagree.” In a related vein, the authors further observe that “society … has a wide range of social values,” and they argue that “the whole of these values should be considered when setting priorities.”procedural features of social bargaining. While we do not suspect that Baltussen et al actually believe the extreme implications of this third proposition to be true, we nevertheless observe that – in their editorial at least – they have left that possibility far too open.The third proposition appears to us to be the most problematic, since, in this view, the content of the values relevant for priority setting in health appears to be an open question: “stakeholders” with “diverging interests” engage in “pluralistic bargaining” to influence priority setting according to their particular “interests.” Taken to its logical conclusion, this proposition would suggest that – at least in an anarchy of values – legitimacy, reasonableness, and accountability can come only from widespread acceptability of the 9 Recently, UHC has been adopted as UN Sustainable Development Goal Target 3.8.10 We observe, therefore, in contrast to the apparent agnosticism of Baltussen et al, that these international agreements contain numerous positive assertions concerning the substance and values of priority setting in health.Although we acknowledge that, at country level, priority setting processes involve a range of stakeholders with different views, with regard to global level policy formulated by WHO’s Member States, it must be recalled that the relevant values enjoy both broad consensus and an international mandate. So, while we agree with the need for an “evidence-informed deliberative process,” such as the authors recommend , we believe that Baltussen et al have been too agnostic in their editorial about the criteria and values already embodied in UHC . The substance of UHC has been progressively defined and clarified by the Member States of the WHO, as well as by those of the United Nations (UN), in a voluminous series of resolutions, reports, and technical documents.For example, three main indicators have been identified within the UHC framework:Coverage: How many people of those who would stand to benefit are receiving needed health services?Financial risk protection: Are people put at risk of impoverishment, or of not accessing health services, because of out-of-pocket payments?Equity: Is access to health services  equitably distributed across population groups?11While not exhaustive, these indicators nevertheless constitute a minimum set of criteria that represent the values that Member States of WHO (and of the UN) have endorsed through the organs of governance which have been instituted to determine policy on these matters. In light of the lengthy, participative, transparent, and mandated processes leading to the development of policies on UHC, in our view the above indicators can be held to constitute three ‘essential pillars’ for priority setting in the context of UHC: that is, other values can be added, but it would be a clear distortion of UHC to ignore any of these three. Thus, we conclude that priority setting for UHC does not start with a blank slate, nor is UHC an empty slogan or a catch-all phrase whose content is to be supplied solely through improved  rules for social bargaining; in fact, the essential content and values required by UHC are articulated in the relevant policies.To return to what we have identified as the first and second propositions, above, an exclusive focus on WHO-CHOICE  in evaluating international efforts on priority setting is, therefore, incorrect. Furthermore, we note that DCP3 in fact emphasizes methods  for operationalizing the triple values of health maximization, financial risk protection, and equity for priority setting in health – values corresponding precisely to the UHC indicators mentioned above.For these same reasons, we would like to reiterate that, at global-policy level and as formulated by the authorized representatives of WHO Member States, there has been an unprecedented amount of agreement on UHC policy, a policy which has itself been shaped through an “evidence-informed deliberative process.” Baltussen et al come closest to acknowledging the centrality of UHC in priority setting when they write,Making fair choices on the path to UHC, [a consultative group appointed to advise] the WHO recently proposed the use of ‘cost-effectiveness,’ ‘priority to the worse off,’ and ‘financial protection’ as the three most essential criteria for countries to consider when setting priorities12 We consider these as ‘core’ criteria, representing social values for which there is broad consensus on their importance and which are of generic relevance across countries, disease areas and health interventions. Their identification can be seen as the product of international learning, particularly in academic circles, on priority setting.”13“In its report de jure in the core of UHC policy, a policy which has been adopted by WHO Member States and integrated in the framework of the Sustainable Development Goals.However, we would add that the criteria referred to in the cited report are not merely the reflections of “academic learning,” nor are they values for which there is simply “broad consensus.” These criteria are in fact those that are embedded Furthermore, whereas Baltussen et al stress priority setting in the context of decisions to reimburse specific healthcare interventions, in genuinely organizing health-system priorities around the goals of UHC, some of the required activities thus identified will almost certainly be systemic in nature, such as ‘improving health-system governance,’ ‘ensuring equitable access to quality services,’ ‘separating prescribing from dispensing,’ or ‘setting up a pooled funding mechanism to purchase services.’ None of these system-wide activities are explicitly mentioned by Baltussen et al, although they are arguably instrumental for achieving UHC and might accordingly be given higher priority than decisions, for example, about the introduction of specific services to the benefit package. As a further example, in considering the systemic aspects of priority setting, one should equally take into account other activities not mentioned in the editorial, including health-sector situation analysis, national health strategic-plan development, and health-sector reviews, which comprise an inter-related set of activities that strongly influence the operating environment of discrete decision-making.This brings us to the claim made by Baltussen et al, to the effect that evidence-informed, deliberative processes in support of priority setting have been mainly “theoretical.” In fact, a substantial portion of the WHO Secretariat’s ongoing work on priority setting is organized around routine strategic-planning activities such as those just mentioned. In this context, country actors and stakeholders are already convening consultative groups for the sake of obtaining consensus on issues of concern. National health-strategy development, health-sector mid-term reviews, and joint annual health-sector reviews are being routinely conducted, and arguably constitute “evidence-informed deliberative processes” of just the kind called for in the editorial. Supporting Member States to manage effectively such routine, regular, and periodic ‘priority setting processes’  is an ongoing area of activity for the WHO Secretariat, although we acknowledge that such activities could be further strengthened and supported.14 HTA is becoming a fast growing area of work within the WHO Secretariat. In WHO’s conception of HTA, a conception informed by global learning drawn from a variety of approaches, HTA brings together the work of a multi-disciplinary group of experts, including health economists, legal advisors, pharmacists, epidemiologists, ethicists, medical engineers and health systems governance specialists. This responds, at least in part, to another important question left open by Baltussen et al: who should be invited to the deliberative dialogue?Finally, Baltussen et al highlight the role of health technology assessment (HTA) agencies in priority setting for discrete choices, such as for reimbursement via inclusion in a benefit package. In the light of recent policies established by WHO Member States,In the relevant WHO policies, HTA is not viewed as a technocratic solution to decision-making, nor as one focused on cost-effectiveness analysis, but rather as a transparent and fair evidence-informed decision-making process underpinned by a strong legal framework. With such a vision, provided the content of UHC policy is understood to be at the forefront, and the systemic elements of priority setting receive appropriate emphasis, we believe we can unreservedly find common cause with the proposals of Baltussen et al.Not applicable.Authors declare that they have no competing interests.JAL drafted the text. DR and MYB revised the text. All contributed to the conception, design, and interpretation; all approved the final version.1Economic Analysis and Evaluation, World Health Organization, Geneva, Switzerland. 2Health Systems Governance, Policy and Aid Effectiveness, World Health Organization, Geneva, Switzerland.


Many emerging technologies are associated with ‘risk’.While the concept of risk is protean, it is usually conceived of as thepotential of something damaging or harmful happening. Thus, risks are a primarytarget of many regulatory regimes. In this article, after articulating anunderstanding of risk, we assess the European medical devices regulatory regimefrom a risk perspective, focusing on its handling of ‘smart’implantable medical devices. In doing so, we discuss the empirical evidenceobtained from expert participants in the Implantable Smart Technologies Project,which evidence is framed around three risk typologies: materiality, geographyand modality. We conclude that none of these risks are sufficiently addressedwithin the existing regime, which falls down not just from a standardsperspective, but also from the perspective of transparency and balance. In an attempt to isolate andavoid these risks, and mitigate their deleterious consequences, we invent healthtechnologies aimed at preventing, diagnosing and treating a wide range ofconditions. Many of these health technologies are medical devices, a field whichemploys over 500,000 people in about 25,000 companies in Europe alone, and whichgenerates annual sales of around €95 billion from some 500,000 products.1for us, but also onus and within us. By ‘smart’, we mean they exhibitone or more of computational intelligence, autonomous operation and responsivenessto environmental changes .3The medical devices field is characterised by rapid and acceleratinginnovation, with drug development, tissue engineering, information and communicationtechnologies and nanoscience playing a role in its evolution. As advances are madein materials, miniaturisation and energy generation, medical devices are becomingsmaller, faster, more powerful and increasingly embedded or integrated into thebody.Given the above, there are very real concerns about the extent to whichmedical device regulations address the risks associated with these smart implantedmedical devices (IMDs). In this paper, after unpacking the concept of risk, wediscuss evidence generated in relation to risks, smart IMDs and the fitness of theEuropean Union (EU) regulatory framework, which informs United Kingdom (UK)approaches to device testing and market authorisation.10The paper will unfold as follows. first, we introduce the empiricalinterdisciplinary project that informs this paper, highlighting some of the existingand emerging IMDs that are captured by the idea of smart IMDs. Second, we verybriefly outline the EU/UK devices regime, which formed the backdrop against whichour empirical evidence was taken. Third, we consider the risks that our respondentsperceived as being particularly relevant to smart IMDs, although we emphasise thatsome of the concerns are common to all IMDs, not just smart IMDs. Fourth, we brieflydiscuss that evidence, considering what it says about the existing framework fromrisk and regulation perspectives. We conclude that smart IMDs have a number offeatures that translate into risks for which the existing framework is ill-equipped.Indeed, we argue that smart IMDs have features that challenge regulatory frameworksbeyond the devices regime, and we make some modest suggestions for reform.2Many emerging technologies, especially those that interact with the humanbody, are associated with ‘risk’. While there is significantsociological, medical, legal and economic literature on risk, there is no commonlyaccepted definition of risk, which has both descriptive and normative aspects.what counts as riskwhat should be acknowledged as proper variables inassessing/measuring riskwhat constitutes an acceptable level of riskwhat actions are most appropriate or effective for avoiding ormitigating risk.contribute to the formation of new risks,which themselves might be individually or socially experienced.13The problems associated with risk are magnified (and multiplied) when thetechnologies that we devise to avoid, mitigate, or manage risk alsoThe ‘Implantable Smart Technologies Project’ (ISTP),Armed with this conception of risk, we undertook several empiricalencounters to explore the term further (as well as the term ‘smart’).The first was a 2011 pilot workshop involving 13 experts from seven Europeanjurisdictions and the USA undertaken by Harmon.All interviews were, with consent, recorded using a digital audio recorderand then transcribed verbatim. The transcripts were read multiple times by allmembers of the team, and analysis proceeded following a grounded theoryapproach,20Importantly, the ISTP was not concerned with scaffolding technologies suchas hips, knees or other prosthetics. Rather, our investigations centred on moreactive or ‘smart’ implantable technologies: cochlear implants (CIs);implantable cardiac defibrillators (ICDs); in vivo biosensors (IVBSs) and deep brainstimulators (DBSs). These particular devices were chosen as the objects of theempirical element because: (1) they implicate very different physiologicalobjectives and technical solutions all within the unifying field of IMDs; (2) theyare either clinically available or close to clinical testing; (3) they have what wepreliminarily considered to be ‘smart’ characteristics  and (4) they were familiar to ourparticipant sample.any instrument, apparatus, appliance, software, material or otherarticle, whether used alone or in combination, together with anyaccessories, including the software intended by its manufacturer to be usedspecifically for diagnostic and/or therapeutic purposes and necessary forits proper application, intended by the manufacturer to be used for humanbeings for the purpose of:As can be gleaned from the more fulsome descriptions in diagnosis, prevention, monitoring, treatment or alleviation ofdisease,diagnosis, monitoring, treatment, alleviation of or compensation foran injury or handicap,investigation, replacement or modification of the anatomy or of aphysiological process,control of conception, and which does not achieve its principalintended action in or on the human body by pharmacological, immunological ormetabolic means, but which may be assisted in its function by suchmeans.An ‘active medical device’ is a device reliant on a source ofelectrical or other source of power other than that generated by the human body orgravity.24Having outlined the project, and demonstrated that smart IMDs are caught bythe regulatory framework, it is useful to offer a very brief overview of thatframework before turning to the evidence on risks that was generated by ourqualitative encounters.3At the outset, we acknowledge that the EU is considering new legislation inthis area,28The notified bodies maintain a register of manufacturers, assessnotifications of clinical investigations, enforce compliance of ‘CE’marked devices ,the purpose and mode of action of the device;how long the device is intended to be in continuous use;whether the device is implanted, invasive or active;whether the device contains a medicinal substance that actsancillary to the device.31Based on these factors, devices are assigned to one of the followingclassifications:32Class I (non-invasive and low risk such as tables andwheelchairs);Class IIa (active and medium risk such as endoscopes andultrasounds);Class IIb ;Class III .The classification of a device determines the standards applicable forissuance of a CE mark, and the level of regulator involvement .334In this section, we explore the risks that the ISTP respondents consideredto be of concern in relation to smart IMDs. They can be organised around threeconceptual phenomena: ‘materiality’; ‘geography’ and‘modality’.4.1[W]e know that the risks in [IMDs] are toxicity, potentialradiation, and then rejection by the body, and possibly inflammation.These are the three known problem areas … If you had to make [anIVBS] out of aluminium, which the body really doesn’t like, thenyou’d have to think of some way of protecting the body from thealuminium by covering it up, sealing it off.Materiality relates to the tangible physical material that is used toconstruct the IMD and its components. Given that all IMDs will invariably havesome material element that will be of a different substance or nature to that ofthe receiving body, they will naturally pose some risk of physiological reactionin the recipient .R2-Engineer, reported: R3-Clinician-1 confirmed that some materials have good track records;gold and titanium, for example, are known to be inert in the body.[Companies] try and get a commercial edge by making features ontheir device … which make them attractive to us and to patients.So they’re trying to devise leads which are slimmer and easier toimplant, and they devised this fidelis lead which was quite a smalldiameter lead, but it just wasn’t robust enough … I thinkit’s now accepted … that the leads are as small asthey’re safely going to get …However, and arguably, risks associated with inflammatory orimmunological responses to a device’s materiality might be accentuated inthe case of smart IMDs because, as they become more sophisticated and performmore (and more complex) functions, they will become both smaller  and more invasive . R7-Clinician-3 recounted the problems that arose in relation to ICDs,which rely on narrow leads operating in a ‘hostile’ environment. Efforts to streamline ICDsresulted in the implantation of devices with leads that were too thin.Respondent-7-Clinican 3 went on to say: In addition to reducing the performance of the IMD, which leaves thepatient vulnerable to their disease/condition, chronic responses to IMDmateriality can lead to full foreign-body reactions that‘wall-off’ the device. This process, known as‘bio-fouling’, can generate its own symptoms quite apart from theoriginal condition that necessitated the device.34The existing regulations stipulate that devices must be designed andmanufactured in such a way as to remove or minimize as far as possible the riskof physical injury in connection with their physical or dimensionalfeatures,36the choice of materials used, particularly as regards toxicityaspects;mutual compatibility between the materials used and biologicaltissues, cells and body fluids, account being taken of the anticipateduse of the device;compatibility of the devices with the substances they areintended to administer;the quality of the connections, particularly in respect ofsafety;the reliability of the source of energy;if appropriate, that they are leak-proof …not undertaken. In fact, the ICD leadsmentioned by R7-Clinician-3 were licensed and administered in the absence ofhuman clinical data.38In short, both the responsibility for setting the risk tolerances andfor managing the risks associated with materiality are placed on the devicemanufacturers. While this may be both practical and sensible, and while there issignificant knowledge associated with materiality in the industry, ourrespondents expressed concerns about how this risk was approached under theexisting regime. R9-Policymaker cautioned that bespoke clinical investigationsaround materiality  are often Concerns around materiality become more critical when IMDs have abiological component. Work is underway to coat IMDs with polymers, medicines andsometimes tissue. It was agreed by our respondents that both biologics andnanomaterials introduce new levels of complexity and uncertainty because theynaturally react and interact with the host’s body. However, the evidencebase for how these materials interact is poor, and there is little regulatoryassurance that good evidence will be generated.A holistic view of our respondents’ evidence suggests that, forthe subject smart IMDs and similarly complex emerging devices, the regulatoryregime is insufficiently specific in the type and quality of evidence that wouldbe required before a device is accepted for a specific classification andawarded a CE mark. This regulatory uncertainty, when combined with thescientific uncertainty often associated with new smart IMDs and some of theexperimental materials, is undesirable, and potentially very harmful, and thematter will become more pressing as novel materials become more prevalent.4.2I think I’d be, in principle, more concerned withsomething that was going inside my head than something going inside anyother bit of me. Once you get south of the border, if the neck’sthe border, I guess I would be pretty level about everything. There maybe occasional bits and pieces that I might think about, but ifit’s the body and not the thinking bit, I would say that’sone thing; if it’s the thinking bit, I would say that’s adifferent thing.The second risk-related phenomenon, ‘geography’, refers tothe location of the IMD within the body. This raises issues of invasiveness andidentity. On the former, most smart IMDs require a surgical intervention wherebythe device or a part thereof is implanted . Sometimes, elements of the device (such as ICD and DBS leads)are run from one location within the body to another, sometimes through theveins. Obviously, the implantation process bears risks common to all surgeries,and this is addressed by other medical regulation. However, our respondentsexpressed the most concern in relation to IMDs implanted in the brain. It isalready known, for example, that brain-implanted smart IMDs, in addition toprompting intense responses by patients and others, can impact on recipientidentity and well-being .There must be a sliding scale. If it’s just under yourskin and in a kind of non-fetishized part of your body, a very boringbit of your body, then it’s nothing, it’s almost external.But when the organ in question becomes more about who you are and more,‘woohoo’, it becomes spookier. I think that’sprobably a spectrum from the wearable to the really banal implanted tothe really special secret places where normally you would not go pokingaround.R5-Lawyer 2 concurred, stating: R8-Bioethicist concurred, acknowledging that our understanding of thebrain is so limited and so recent that IMDs associated with it raise special andwidely agreed concerns about personality change, damage to thinking centres,external monitoring , and free will (i.e.‘mind reading’ or control). In short, as explained by R1-Lawyer-1,perceptions of risk change dramatically once the IMD goes into the brain, forimplantations in that organ are viewed as more invasive and more threatening toour capacities and identity.40manufacturer sensibilities, and the lack of understandingof certain organs and biological processes will continue to complicate smart IMDdevelopment aimed at certain interventions, leaving regulators with littleindependent knowledge on which to base decisions; it is themanufacturer, in cooperation with the notified body, whichprovides the evidence on which regulatory decisions are made, and whichdetermines the classification. The nationalcompetent authority (the MHRA) only audits the process. However, differentnotified bodies have different standards, expectations for evidence and levelsof competence, and manufacturers can choose which one to work with.44The regulation’s materiality provisions noted above would applyhere, as well as more general provisions which state that the device’suse must not compromise the clinical condition or safety of the patient,And there have been some very public failures in the context ofnon-smart IMDs. For example, in addition to the breast implants andmetal-on-metal hip failures, it has been reported that a distal protectionsystem for coronary artery interventions received an EU CE mark after a singlestudy involving 22 subjects showed that it worked as intended, whereas approvalwas only granted in the US several years later after a randomised studyinvolving 800 subjects.474.3The third and final risk phenomenon that concerned our respondentsrelates to ‘modality’. This concept refers to the IMD’smethod of action and functionality, and it has several elements –physiological responses, psychosocial responses, and third party access –each of which are addressed below.4.3.1With respect to the physiological, it was felt that, as IMDs becomemore complex, they will interact in novel and unanticipated ways with thebody, and have increased potential to malfunction, which could have profoundconsequences for recipients. R2-Engineer wondered about the repercussions of‘misfiring’ treatments once IMDs are medicinally coated, oronce drugs are put on chips for timed release or release in real-timeresponse to physiological developments (both of which are innovationtargets). The implications of concentrated doses of chemotherapy beingreleased improperly into the body, for example, could be immense.This is an issue of regulatory operation insofar as the frameworkalready addresses the concerns voiced. Existing provisions, for example,state that devices must be designed and manufactured in such a way as toguarantee their safety characteristics and performances, with particularattention being paid to the proper functioning of the programming andcontrol systems, taking into account the principles of developmentlifecycle, risk management, validation and verification.49The big question that remains, however, is whether the existingsystem is vigilant enough. It is generally considered not to be, withnotified bodies – privately run bodies who are not public healthagencies – seeing themselves as ‘clients’ of themanufacturers. They do not publish their decision-making process, nor theevidence provided by the manufacturer on which they base their decision, andthey often keep information relating to safety confidential.504.3.2On the psycho-social element of modality, R5-Lawyer-2 emphasisedthat, while modality can have physical implications, it can also have strongpsychological/emotional implications, particularly if the IMD interacts with(or indeed controls the operation of) an organ considered central toindividual well-being. DBSs, for example, have elicited diverse and highlyemotive responses. In particular, they have been known to cause significantpersonality change, such as hyper-mania, hyper-sexualisation anddisinhibition.54post-market surveillance andcoordination  is critical, but our respondents unanimously thought that this wasinadequate in the existing framework, from both a standards and practicesperspective. In the UK, the following has been claimed: The number of different types of devices on the market isabout 80,000 in the UK and over 200,000 in Europe. Uncertaintysurrounds the numbers because there is no publicly available list ofdevices being used … The MHRA does not know precisely whichClass III devices (the most risky) have been cleared for use in theUK or Europe. … [T]he knowledge problem is compounded by thefact that NHS procedures are poorly coded … So while we canget detailed information about which drugs are being used in theNHS, the same does not apply to devices.55Despite smart IMD innovations almost invariably having psycho-socialimplications, the regulatory regime is silent on this issue. The difficulty,highlighted by R1-Lawyer-1, is that these symptoms and impacts will belargely unknown (possibly even un-theorised) until they present in users.The result is that this risk is characterised by a high degree ofuncertainty. As such, robust [A] researcher or regulator can readily identify all thedrugs the person was taking before [a] procedure, but a detaileddefinition of the device itself is not routinely recorded in mostclinical databases. Therefore, when the modified wiring of apacemaker is found to cause potential fatal short circuits …it is hard to perform a systematic assessment, notification, orrecall. Even car manufacturers have better databases to identify whouses their products …56Similar observations have been made elsewhere: Moreover, the surveillance system is complaint-triggered rather thansystematic, which means the data held by regulators is incomplete.614.3.3The final aspect of modality relates to the inclusiveness of the‘circle of control’,The worry about having programmability remotely is that, ifthat programming goes wrong, you may not have a way to rescue thesituation. So if you inadvertently programme something that isharmful to the patient, which occasionally happens because these areincredibly sophisticated devices, it’s not a good idea tohave the person alone at home when that happens.With ICDs, for example, the clinician is directly involved inprogramming the settings and sensitivities to physiological changes. Forsafety reasons, that programming is performed in hospital. R7-Clinician-3reported: [S]ome patients ask us to make alterations to theirpacemaker which are clearly inappropriate. My worry is this balancebetween giving patients autonomy versus patient safety. Our job isto advise and facilitate treatments for patient safely, and in someways you’d be passing a big part of the medicalpractitioner’s role over to the patient. I would questionwhether, with some patients at least … you [should] give yourpatient the power to make significant changes to the parameters oftheir pacemaker when that person isn’t medically trained andmaybe doesn’t fully understand the effects on physiology ofmaking certain adjustments …Several respondents expressed caution about expanding the circle ofcontrol beyond the treatment team, and they were reluctant to includepatients themselves. R7-Clinician-3 explained: Some IMDs have transmitters which will inform the hospital that theyare active . This expands the circle of individuals who have knowledgeabout what the device is doing and when. Obviously, this has implicationsfor patient privacy, which is eroded as more people become involved intreatment and device maintenance.[W]hat I would be interested in is the extent to which [thedevice] could be, or is, controlled remotely, for example. Or itsability to self-regulate, and in what ways, and if it isself-regulating, then what other controls could be exercised over itif it became inappropriate or unsafe.A more threatening aspect of privacy erosion that exercised ourrespondents was the risk that persons outwith the treatment team –and so bearing no duty of care to the patient – might hack into thedevice and gain knowledge, or, more alarmingly, alter its parameters.R7-Clinician-3 stated that, although ICDs can be interrogated online, theydo not typically upload instructions remotely, largely because of thesesecurity concerns. Despite different levels of concern over hacking,R1-Lawyer-1 summed up many respondents’ fears: Modality, and this element in particular, was the most concerning inpart because our respondents viewed it as the least satisfactorily addressedby the regulatory framework. In particular, data transmission andperformance modification grounded in the operation of software (i.e.security and decryption) is little addressed in the regulation.67Some of the points of access to smart IMD software include deviceidentification, data retrieval, device reconfiguration and softwareupgrading, multidevice coordination and communication, and manufactureaudits in the event of failures, each of which could serve as a point ofvulnerability.70Ultimately, while the list of device (and patient) vulnerabilitiesstemming from increased connectivity is growing, discourses about theappropriate balance between security, utility and traceability areunderdeveloped.5Our empirical evidence relates to understandings of both the risksassociated with IMDs generally, and smart IMDs more specifically, and the regulationthat applies to them. The overall view of our respondents was that each of the abovephenomena impact on ‘safety’ and ‘efficacy’, both ofwhich ought to be well-defined and systematically tested for regulatory purposes.However, they also shared the broad view that more needs to be done to improve howsmart IMDs are managed.5.1Some [IMDs], like a biosensor for drug delivery, wouldn’tyet be considered standard, so people might have more concerns aroundsomething they see as experimental … I think more establishedtechnologies would be fine, [but] others would need a little bit of workto get patient confidence and longer term safety data … [I]fyou’re at the beginning of a technology then maybe you need a bitof confidence that it is going to be safe, but you don’t reallyknow until 10, 15, 20 years have gone by.On the issue of understandings of risk, the empirical evidencedemonstrates that risks were associated with each characteristic . While many of the concerns were applicable to IMDsgenerally, some  were uniqueto new and emerging smart IMDs, and all were accentuated by smart IMDs. Riskconcerns were driven in part by the invasiveness and novelty of the smart IMD inquestion, the former of which raises many well-known physiological risks andsome less well understood psycho-social risks, and the latter of whichintroduces uncertainty about both risks and benefits. R3-Clinician-1 noted thefollowing: Some of [the risk] has to do with how experienced and effectiveclinicians become. So, whilst you would say [for a] pacemaker decadesago, ‘Oh, that’s very risky’, I wouldn’t seethat as risky now. I see anything to do with the brain as beingrisky.R6-Clinician-2, R9-Policymaker, and R10-Government Researcher agreed,with the latter adding: Ultimately, then, risk has a close relationship with novelty, and it isimportant to develop a reliable evidence-base so as to erode the uncertainty. Itwas agreed that better evidence of how new smart IMDs perform and how well theyserve patients is needed. Developing this evidence-base, as noted byR1-Lawyer-1, is challenging, requiring cooperation and information exchange.When the implantation and/or operation of the device is invasive, the utility ofinformation-sharing is increased. The same is true when the device containsfeatures that might seem inherently risky, such as transmitting features.5.2Our respondents considered the regulatory framework to be wanting inrelation to all three phenomena relevant to our risk components of‘safety’ and ‘efficacy’. It was acknowledged thatsafety is addressed by the existing framework, but not sufficiently, and thatefficacy is almost completely ignored by the regulation. In getting marketapproval, manufacturers need not demonstrate efficacy; the device must performas intended by the manufacturer,78Despite the prevailing ‘bottom-up’ and‘responsive-mode’ approach of the EU being commended fordelivering devices to the market quicker than a more top-downcommand-and-control system might,816Given the general acceptance that interventions on humans should be in theindividual’s best interests, and should be evaluated continually for theirsafety, effectiveness, efficiency, accessibility and quality,88With respect to gaps, the regulatory framework exhibits some criticallacunae, most particularly around generating evidence (pre and post-market) aboutboth safety and efficacy. Given the proliferation, complexity and power of smartIMDs, and so the magnitude of their potential impact , there is a pressing need to better address safety, and tomore explicitly address efficacy. Concerns over unsafe materials which have alreadybeen used in non-smart IMDs underline the urgency for action on this front; IMDs aretoo often approved for use without sufficient understanding of their implicationsfor health outcomes under the present regime, which permits devices to be marketedand implanted with dismayingly little understanding of key technical issues. And thelack of follow-on oversight offers unscrupulous manufacturers ample opportunities totake shortcuts .89The proposed amendments to the European framework are certainly a step inthe right direction, but they are not enough.91