F=23.11, p<0.005, which indicates that all proposed hypotheses are valid. The results suggest that the overall mental health level of young new entrepreneurs is not high, and their consciousness of innovation and entrepreneurship needs to be strengthened. Under the wave of mass innovation and entrepreneurship, the research results can strengthen entrepreneurs’ innovative entrepreneurship consciousness and may have great theoretical and practical significance for improving and optimizing government innovation and entrepreneurship policies.Under the background of mass entrepreneurship and innovation, innovative entrepreneurship research is urgently needed for entrepreneurs. The aim was to explore the innovative entrepreneurship consciousness of new entrepreneurs. First, the regional competitive advantage theory is discussed. The research method of questionnaire survey is combined with statistical analysis to obtain relevant research data. Quantitative standards are used to measure the impact of regional advantages and policy support on entrepreneurs’ innovation and entrepreneurship. The policy content analysis and questionnaire survey are applied to discuss the impact of regional competitive advantage, educational psychology, innovation and entrepreneurship policy, and entrepreneurs’ innovation consciousness. Meantime, the policies related to entrepreneurs’ innovation and entrepreneurship are further explored. The results show that the proportions of young entrepreneurs suffering anxiety and depression are 29.4 and 27.5%, respectively, which are significantly higher than the national average. Besides, in the test,  The current international economic situation is unstable, and China, which is deeply integrated into globalization, is also facing great downward pressure. The employment difficulty of college students caused by the year-by-year enrollment expansion of colleges and universities has become increasingly prominent, and the development of the times needs innovative and entrepreneurial talents. In 2014, Premier Li Keqiang puts forward mass entrepreneurship and innovation, hoping to promote employment by entrepreneurship and create a new engine for economic transformation and upgrading . GovernmThe study aims to give better play to the advantages of regional resources, strengthen entrepreneurs’ awareness of innovation and entrepreneurship, and establish an advantageous and leading position in the process of enterprise competition. The theory of regional competitive advantage is combined with the statistical analysis data from the Questionnaire Survey (QS) and the policy research on the innovation and entrepreneurship of scientific and technological entrepreneur talents. Consequently, solutions are put forward based on quantitative standards to measure the influence of regional competitive advantages and policy support on entrepreneurs’ innovative entrepreneurship, and the influence of regional competitive advantage, educational psychology, and innovation and entrepreneurship policy on entrepreneurs’ innovation consciousness is explored through policy content analysis, principal component analysis, and linear regression. Significantly, the regional competitive advantage theory is innovatively applied instead of the traditional absolute advantage theory or the comparative advantage theory, and QS is combined to explore the influencing factors of innovative entrepreneurship of new entrepreneurs and policy optimization.Domestic and international scholars have studied the entrepreneurial characteristics of new entrepreneurs. Specifically, The traditional absolute advantage theory holds that if a country costs far less than other countries in specific product production, then it has the absolute advantage of that product, thus exporting that product. Each country has its favorable production conditions for specific products, and each country conducts international labor division according to its superior products and exchanges products.The comparative advantage theory based on the theory of absolute advantage holds that it is very accidental that the two countries have the absolute advantage of different products. Each country does not have to produce products with absolute advantage but should concentrate on producing products with comparative advantages to export and import products with comparative disadvantages. This specialization can improve the productivity of labor.Given the deficiency of comparative advantage theory, scholars creatively propose the theory of national competitive advantage based on the research of important trading economies for several years. The research suggests that in national and industrial competition, production factors are becoming less influential, while productivity has become the decisive factor, and productivity is determined by the competitive advantage. Although the theory of national competitive advantage is set at the national level, its analysis framework applies to regional and urban levels . RegionaBy integrating the guidance of educational psychology, innovation and entrepreneurship policy can improve the cultivation of innovation consciousness of new entrepreneurs. Educational psychology is the science to study the basic psychological laws, the psychological interaction between teachers and students, and the psychological phenomena in teaching and learning . EducatiInnovative entrepreneurship is an entrepreneurial activity based on innovation, yet different from pure innovation or pure entrepreneurship. Innovation emphasizes pioneering and originality, while entrepreneurship emphasizes the behavior of obtaining benefits through practice. Therefore, in innovative entrepreneurship, innovation is the foundation of entrepreneurship, while entrepreneurship is the embodiment and extension of innovation . With thAccording to the above analysis, the regional competitive advantage and the self-perceived efficacy of new entrepreneurs are closely related to innovative entrepreneurship. Self-perceived efficacy refers to an individual’s judgment of whether he can complete an activity at a particular level. In the regional competition, the greater the regional competitive advantage is, the more its attraction is to scientific and technological talents and new entrepreneurs, which can mobilize entrepreneurs’ innovation consciousness to start-up businesses, further developing local resources, form an industrial chain with comparative advantages, and accelerate economic development. The technology route map is shown in Based on the regional competitive theory, a research model is proposed, as shown in The questionnaire on entrepreneurs’ innovation and entrepreneurship awareness is designed to collect the experimental data. The pre-test questionnaire relies on previously validated measurements and uses the Likert scale. After Professor Roger’s review and referring to Professor Roger’s relevant opinions , the folConcerning the validity and reliability of the QS, the QS is distributed among 160 entrepreneurs in Xi’an, including 102 men and 58 women, who all have started their own businesses. Of these, 55 of them have started their own business once, 70 of them have started their own business twice, and 35 have started their own business three times. Analysis of their understanding of regional competitive advantage theory and entrepreneurship policy is conducive to the optimization of policy research. For entrepreneurs with multiple start-up experiences, QSs are issued to collect their understanding of entrepreneurship policies in different entrepreneurship periods, and the correctness and validity of the results are tested through statistical analysis software. Cronbach coefficient is the most commonly in the reliability test. When the Cronbach coefficient is greater than 0.8, the scale has good reliability, and when the Cronbach coefficient is greater than 0.9, the scale has very ideal reliability. Thus, the Cronbach coefficient is used to study the reliability of the QS here. The results show that the value of Cronbach’s α coefficient is greater than 0.8, and the value of the Kaiser-Meyer-Olkin (KMO) is greater than 0.8, thus proving that the designed QS is reasonable and valid and can be distributed on a large scale. From September to December in 2020, a total of 300 QSs are distributed among new entrepreneurs in Shaanxi Province, 255 are recovered, and 241 are effective, with an effective rate of 80.3%. Significantly, Cronbach’s α coefficient can test the reliability of the questionnaire. The larger the Cronbach coefficient, the higher the internal consistency of the QS. If the Cronbach coefficient is less than 0.65, the reliability of the QS is low. If Cronbach’s α coefficient is greater than 0.65 and less than 0.7, the QS is acceptable. If the Cronbach’s α coefficient is greater than 0.8, the reliability of the QS is very high. Thus, the designed QS has high reliability .The results of the QS are shown in The reliability of each measurement item in the QS is tested. Cronbach’s α coefficient and the signal value of the reliability test are shown in KMO test statistics can verify whether the samples, and QS can be verified through factor analysis. The KMO values of each variable in the QS and the statistical results of the Bartlett sphere test are shown in The QS results suggest that the mental health status of young entrepreneurs is not optimistic. Due to the professional characteristics and competitive environment, young entrepreneurs’ pressure mainly comes from work and social responsibilities, and their mental health problems are serious . The QS Based on the above analysis, following points can be summarized. To make entrepreneurship and innovation popular and fruitful, rather than a flash in the pan, the reform should be further deepened and institutional barriers should be broken to liberate and lead the innovation and entrepreneurship wave. Meanwhile, innovation and entrepreneurship education should be strengthened, resources and policies should be integrated and implemented. Moreover, various forms of innovation and entrepreneurship curriculums should be brought into higher education. Colleges and universities should improve the entrepreneurship methods for college students, encourage students to learn through enterprise practice, and ensure the combination of entrepreneurship curriculum and practice . CollegeFurther, traditional industries should be transformed through process innovation, led by product innovation, and based on mechanism research to improve the international market competitiveness . FirstlyLastly, financial support and credit policies should be fully implemented in mass entrepreneurship to guide financial resources to public goods and services, promote the innovation of financial products and services, expand the scale of direct financing, and build diversified financing channels for public goods and services . First, Here, the QS data are comprehensively used and deeply mined. As a result, patterns contained in the data are discovered to verify the role of regional competitive advantages for the innovative entrepreneurship of new entrepreneurs. The results show that the regional competitive advantages have a great influence on the innovative entrepreneurship of new entrepreneurs, but the mental health level of the new entrepreneurs is generally low: 29.4% of the surveyed entrepreneurs have felt anxiety, and 27.5% of the respondents have felt depression. This demonstrates that the psychological quality of scientific and technological entrepreneur talents is generally weak, and their consciousness of innovation is insufficient, so they cannot make good use of the theory of regional competitive advantage to solve the problem of innovative entrepreneurship. Here are the deficiencies. The QS only statistically analyzed the data of just over 160 new entrepreneurs in Xi’an City, so the research scale is relatively small, and in the subsequent research, the sample scale will be further enlarged.The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.The studies involving human participants were reviewed and approved by Northeastern University Ethics Committee. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


Schools provide an opportunity for developing strategies to create healthy food environments for children. The present study aimed to analyze the Healthy School Canteen (HSC) policy and identify challenges of its implementation to improve the school food environment in Iran.This mixed method study included two qualitative and quantitative phases. In the qualitative phase, triangulation approach was applied by using semi-structured interviews with key informants, documents review and direct observation. Data content analysis was conducted through policy analysis triangle framework. In the quantitative phase, food items available in 64 canteens of primary schools of Tehran province were gathered. The food’s nutrient data were evaluated using their nutrition facts label. The number and proportion of foods that met the criteria based on Iran’s HSC guideline and the World Health Organization nutrient profile model for the Eastern Mediterranean Region (WHO-EMR) were determined.The main contextual factors that affected adoption of HSC policy included health , political , structural , economic , and socio-cultural  factors. Assessment of the school canteens showed that a large proportion of available foods did not comply with the national guidelines (54.7 ± 2.54%) and WHO-EMR model (85.6 ± 2.34%). The main reasons identified for incomplete implementation of the policy were inadequate physical and economic infrastructure to set up standard school canteens, lack of scientific criteria for food categorization, poor monitoring, high price of healthy foods, and conflict of interest among the actors.The majority of foods and beverages available in the school canteens did not comply with national and regional standards. Iran HSC policy needs to be improved by using an evidence-based approach and active interaction between all key actors.The online version contains supplementary material available at 10.1186/s12889-021-11587-x. Childhood is a critical period to form lifelong dietary behaviors which influence future risk of obesity and diet-related non-communicable diseases (NCDs) , 2. Desprange of foods that can be accessed in the context where people live and can enable or restrict healthy dietary choices”  that has both nutritional value and beautiful and attractive package. … We don’t have a variety of products that can be offered in the school canteen, and that’s a weakness.” (A representative from Community Nutrition Office in MoHME).Therefore, it is necessary to include more diverse food items in the permitted foods list according to local conditions and individuals’ food habits in different regions of the country.-Not providing a solution to establish a standard and healthy food canteen in schools."... A large number of schools in the country, even in Tehran city, have almost worn-out buildings. The buildings are very old. A lot of money must be provided to repair and renovate school buildings and their physical spaces." (A representative of Student Organization)One of the major weaknesses of the HSC bylaw is little attention being paid to the condition of most school canteens in Iran. According to the interviews, most school canteens do not have proper building and physical condition. Therefore, it seems necessary to define specific criteria for a hygienic school canteen and express strategies to improve present canteens. Most interviewees mentioned school budget shortages as a limitation to improve canteen’s physical condition.Therefore, funding strategies is another point that should have been considered in the content of the policy.-Ambiguity in the role(s) of program stakeholders.The content of the program does not specify who should participate in the program. For example, food industry, as a private sector, plays an important role in the development of healthy eating, but its role in the program is not clear." "Despite having a wide range of actors and stakeholders related to this policy, reviewing the documents shows that the assigned roles in bylaw have been limited to the MoHME and MoE, while the expectations of stakeholders from each other have not been clearly defined.Thus, more transparency with regard to specific roles of these two and many other stakeholders, including food industry, parents and students is needed.-Unclear strategic and operational planning.One of the main weaknesses of the bylaw’s content is the lack of strategic planning along with a clear, comprehensive and step-by-step national operational plan. However, the executive guideline is annually provided by the MoHME and MoE to the relevant departments throughout the country, but the annual actions and short-term goals are not clearly described in detail.-Defect to evaluate the program.There is unclear mechanism for comprehensive monitoring and evaluation of the program. A review of the policy documents shows that although internal and external audits have been defined to monitor the program, no comprehensive evaluation mechanisms that include an assessment of the program’s inputs, process, outputs, impacts and outcomes at various national, province, county, and school levels has been defined.Five main themes were identified as the main contextual factors lead to the formulation and implementation of the HSC program, as follows:-Health factors.Following socio-demographic changes and urbanization in Iran, the nutrition transition is taking place rapidly in recent decades . Healthy-Political factors.Upstream documents in Iran, e.g. Fourth National Economic, Social and Cultural Development Plan  was a st-Structural factors.Existence of various executive offices in the MoHME and MoE related to school health policies, and relatively little interaction between them can lead to inadequate use of available human and financial resources, parallel work and hinder reaching the goals of the HSC policy. In addition, lack of unified and stable stewardship has sometimes caused disagreement between the decision-making centers of the two ministries in some cases about health program in schools. Centralized regulations without attention to regional or local needs and limited participation of school principal in decision-making committees regarding student nutrition are important reasons behind incomplete implementation of the program in schools. Moreover, according to informants at the Student Organization, many schools have old buildings and do not have enough physical space to establish a healthy canteen.-Economic factors.In the context of international economic sanctions on Iran, food prices are rising sharply . Many ex-Socio-cultural factors.Many interviewed mothers who were employed stated that they could not provide a suitable snack for their children because they did not have enough time for preparing it. Moreover, their children are influenced by their peers and classmates to purchase foods from the school canteen. Also, the abundance of unhealthy food advertisements in the outdoor environment, television and social media encourages their children to consume them.Some informants at MoHME expressed that limited nutritional knowledge of some school principals and most canteen managers is one of the main obstacles to stop selling unhealthy food in schools and compliance in implementation of HSC policy.The two main actors in different stages of the policy process of the HSC policy are the MoHME and MoE. The offices of community nutrition and environmental health in the MoHME, and health office in MoE have key roles. The office of community nutrition in the MoHME have the main responsibility for designing school nutrition guidelines and identifying permitted and not permitted food list. Supervision of the establishment and quality management of the healthy school canteens has been assigned to the Student Organization in the MoE."... The student organization is also involved in issuing license for school canteens. According to their instructions, they should receive some money from school principals to issue a license of healthy canteen. Some principals do not declare that their school has a canteen because they do not want to pay that money and this is a damage [to policy implementation]." .Some of the interviewees were dissatisfied with the performance of the Student Organization and stated that they had a profit-based approach and considered the regulations as a mean for the financial profit of this organization.It is necessary to have effective communication between key actors to justify policy goals and remove executive barriers.In the present study, most canteen managers were janitors (76.56%), followed by other school staff (12.5%), students (6.25%), and in some cases personals employed by private sector (4.68%). School principal is responsible for the status of the school canteen. According to interviews, in some schools, implementation of this bylaw creates conflicts of interest for the principals. On one hand, the supply of healthy food items helps to improve students’ nutrition and on the other hand, if the canteen is not financially profitable, principals cannot afford to compensate expenses through the school’s budget and may fall under economic pressure. As a result, some principles are not motivated enough to enforce regulations and resist, while as the leader of the program in the school, can play a valuable role in conveying the health messages of the canteen. In addition, in some schools, canteen managers do not fully comply with the rules due to the financial benefits of selling unhealthy food products.In eight schools (12.5%), students’ mothers cooperated in preparing cooked foods. Based on school officials, collaboration of mothers has a positive effect in health status of canteens. After all, the food industry can help improve the health of the children’s food environment by producing varied healthy food products, but according to the interviews and review of the documents, they have not been properly involved in this policy.Although the initial attention on healthy food environment goes back to 1968 through article 13 of food, beverage, cosmetics and hygiene products act , the ideThe heuristic stages method was used for analyzing the policy-making process . This moMost interviewees believed that from decades ago, the hygienic status of school canteens was the main concern of both the MoHME and MoE; however, the recent alarming increase in childhood overweight and obesity, along with evidence on high consumption of unhealthy foods by children (CASPIAN study) and access to unhealthy food in schools made the healthiness of school food environment a major priority. Increasing inter-sectoral communications between MoHME and MoE for joint programs such as Health Promoting Schools Program or Executive Regulations for Providing, Maintaining and Promoting Physical, Mental and Social Health of Students provided a good background for cooperation of the two ministries. The result was a joint working group for the formulation of a bylaw on healthy canteens in 2012 and holding several meetings in the MoHME.The first bylaw was signed by the two ministers in 2014 in the form of a joint memorandum of understanding. However, the policy formulation approach was not evidence-based and did not follow a valid scientific process for settling different stages of the bylaw, including the criteria to define the list of permitted and unauthorized food items. Therefore, it was largely based on the previous relevant experiences of policymakers and experts from the Offices of Community Nutrition and Health Environment in the MoHME and Health office of the MoE . HoweverAfter notifying school principals regarding the regulations, they are required to obtain a license from the Student Organization to set up a canteen to supply food items according to the permitted foods list recommended by the HSC bylaw. Various supervisory groups from the Student Organization, the Office of Health in the Department of Education, the Nutrition and Environmental Health Unit of Community Health centers, monitor the implementation process of the policy. Several committees at the MoHME or MoH and provincial committees are set up to review and coordinate policy implementation. Although it has been more than four years since the launch of this program, its implementation has always faced challenges, the most important of which are listed in Table The availability of categories of food items in the school canteens are presented in Fig. The degree of strictness, as shown by the proportion of food items classified as permitted to be accessed in school canteens, varied considerably between the different WHO-EMR model and healthy nutrition canteen guideline of Iran Table -4. WHO-EReady-made, convenience foods and composite dishes had the highest proportions of eligible foods in school canteens based on the WHO-EMR model Table . But, ca"We are environmental health experts [Inspectors]. Our priority is to supervise restaurants, delicatessen, and food preparation and distribution venues. In fact, there are all kinds of places that need to be inspected and monitored. The school canteen is one of them that we have to visit once every three months. Since it is not possible to visit all schools every three months, if the experts [Inspectors] have time, do it … [But] This schedule is idealistic and experts [Inspectors] cannot implement it." Many interviewees believed that it is necessary to monitor and evaluate the HSC program. Insufficient trained and capable human resources was one of the main limitations for monitoring and evaluation of canteens health status in the schools across the country. Presently there are separate supervisory checklists being used by the offices at the MoHME and MoE which is a redundant and making them as one can save time and resources. Moreover, interviewees stated that the checklists are more focused on the hygienic aspects of the environment and food items, rather than the nutritional value of the foods. Besides, dealing with violators has not been strict enough. Environmental health offices in community health centers are the only sector for dealing with violations legally. However, for them monitoring school canteens was a lower priority, compared to other tasks they are responsible for.Therefore, it is necessary to consider sufficient human resources supply to monitor the school canteens regularly.Another challenge in monitoring and evaluation of the program was the lack of an official report on the results of the program audits. Many interviewees stated that although annual auditing forms are completed in the province level and sent to the national level of the MoHME and MoE, so far no official report on results of the audits has been published. This default has caused stakeholders to disagree about the policy outcome(s).“I think the situation of schools’ canteens is wretched. … Health and standard conditions are problematic based on the current criteria. We have few schools where the rules are actually followed” "In the meetings that we had with the working groups in Tehran … based on the feedbacks and reports about the number of inspections and health status [of canteens] … . every step that we move forward it has improved … " To avoid any misunderstanding, it is necessary to publish official reports on the current situation on a regular basis. In addition, based on the interviewees no evaluation of the program has been carried out since its implementation. This has prevented policy makers and other actors to access objective evidence for taking decision(s) on changes required to improve the content and implementation of the program.The findings of the current study revealed that four years after implementation of the policy of healthy school canteens in Iran, more than 54% of available foods in school canteens of Tehran did not comply with the list of permitted foods of the national guideline. This figure was more than 85% when compared with the WHO-EMR recommendations for children. These findings indicate a higher rigidity of the WHO-EMR model compared to the HSC guideline. The most important reason is inadequate consideration to scientific, accurate, and transparent criteria in the HSC guideline. The cakes, sweet biscuits, and pastries group had the highest share among the food groups available in the school canteens. Although all types of food items belong to this group are prohibited in the WHO-EMR model, but all of them were allowed in the current HSC guideline during the study period, except for types of donuts and non-meat strudel without packaging and no hygiene licensed from the MoHME.The proportion of not permitted food items was significantly higher than permitted items based on both models. Cakes, sweet biscuits and pastries had the highest share among not permitted foods. Also, these food items have a large share of processed food products that Iranian children consume on a daily basis . A crossThere are similar experiences in many countries, e.g., Pakistan, India, Bahrain, Saudi Arabia, Oman, Lebanon, Egypt, Turkey, where foods sold in school canteens had low nutritional value and contained items high in sugar , fat , or salt  –63. The Changing the food environment to offer healthier foods is not easy . SeveralAlso, in higher-income countries, while provision of nutritious foods or standards to restrict the availability of foods and beverages high in energy, sugar, salt, and fat have had a positive influence on dietary behaviors , in AustIn the present study, comparing the proportion of permitted and not permitted food items based on the WHO-EMR model did not show a significant difference in public/private, girls’ /boys’ schools and in different districts of Tehran province; however, the situation in rural schools was worse than urban ones. Other studies showed that the level of obedience to the policies change by socio-demographic characteristics of schools , type ofOne of the questions the present study explored was why school canteens still supply and sell the unauthorized foods despite the national guideline. Based on the results, school canteen managers, as front-line implementers are mostly janitors and often do not follow the law thoroughly. One of the reasons for their reluctance in the implementation of the law is the financial benefits of selling unhealthy food. For achieving high levels of health in food canteens, it is necessary that the school principals have a clear and strong authority, with a pure commitment to healthy food on school environments. However, the findings showed that this is not the case for some principals and they collaborate in making money through selling more of unhealthy products. In addition, low diversity and high price of healthy foods in the market and children’s preferences towards sweet, fatty and salty foods are factors that limit the full implementation of the law. After all, dealing with the offenders by both school officials and health inspectors had not been strict enough and the policy has not been taken seriously. Over all, barriers in school nutrition policy implementation can be a combination of complexity of policies, students’ food preference, fears of revenue loss, limited resources, and a lack of coordination, support, and communication , 75–77.One of the main weaknesses of the HSC policy in Iran is applying a top-down approach in the formulation and implementation of the program that may have unpleasantly affected accountability and ownership of actions . FailureToday, many countries use the nutrient content  to set forth standards, guidelines or regulations for foods and beverages available in schools. In this regards, countries in the WHO South-East Asia Region have the most comprehensive inclusion of nutrients in their criteria . GloballIn the formulation of HSC policy in Iran, individual, socio-cultural and physical-environmental influences have not been addressed properly, while these factors have been shown to affect unhealthy snacks consumption among adolescents in the country . Also, nOther barriers identified in the implementation of HSC program in Iran, included limited financial and human resources , high workload of key actors , and poor monitoring systems. Similarly, Reeve et al. reported that despite the existence a strong policy mechanism to provide healthy food in the Philippines schools, limited human and financial resources for its planning and implementation, negatively affected the policy impact on the health of school food environment .To the best of our knowledge, the present study is the first analysis on school food policy in Iran. The strengths of this study lie in its stringent methodology. A depth policy analysis was performed by a combination of quantitative and qualitative data through a mixed method, and triangulation theory.There are some limitations that should be considered in interpretation of the current results. The interviewees might have responded with regard to their societal, political, and institutional commitments and tried to manner socially desirable, which could affect the information they provided. Another limitation is the recall bias that we tried to minimize through adding document content analysis. Moreover, in some schools, it seemed that some of the food items were hidden before evaluating food items available in the canteen. Therefore, data collection may have been associated with some bias.Based on the findings, the following actions are suggested. It is necessary for decision-makers to use an evidence-based approach to policy making and choosing food items for provision in school canteens based on precise and scientific criteria and recommendations. In this regard, using a regional tool, e.g., the Nutrient Profile model of the WHO-EMR , may be The current findings indicate that the HSC policy has not been fully implemented in primary schools in Iran. The adoption of bylaws and regulations to protect children from the adverse effects of unhealthy foods and beverages in the school setting was restricted by the capacities and resource shortages for implementation, poor intra/inter-sectoral partnership, and potential conflicts of interest with food companies and canteen managers. The findings provide valuable information for policy makers, school officials, and public health practitioners interested in supporting the health and nutrition of children.Additional file 1.Additional file 2.Additional file 3.


Medical devices are potentially good candidates for coverage with evidence development (CED) schemes, as clinical data at market entry are often sparse and (cost-)effectiveness depends on real-world use. The objective of this research was to explore the diffusion of CED schemes for devices in Europe, and the factors that favour or hamper their utilization.We conducted structured interviews with 25 decision-makers from 22 European countries to explore the characteristics of existing CED programmes for devices, and how decision makers perceived 13 pre-identified challenges associated with initiating and operating CED schemes for devices. We also collected data on individual schemes that were either initiated or still ongoing in the last 5 years.We identified seven countries with CED programmes for devices and 78 ongoing schemes. The characteristics of CED programmes varied across countries, including eligibility criteria, roles and responsibilities of stakeholders, funding arrangements, and type of decisions being contemplated at the outset of each scheme. We observed a high variability in how decision makers perceived CED-related challenges possibly reflecting country-specific arrangements and different experiences with CED. One general finding across all countries was that relatively little attention was paid to the evaluation of schemes, both during and at their completion.CED programmes for devices with different characteristics exist in Europe. Decision-makers’ perceptions differ on the challenges associated with these schemes. More exchange of knowledge and experience will help decision makers anticipate the likely challenges in CED schemes for devices, and to learn from good practices existing elsewhere.The online version contains supplementary material available at 10.1007/s10198-021-01334-9. At the time of the publication of the ISPOR ‘Good Practices for Performance-Based Risk-Sharing Arrangements (PBRSAs) Task Force’ report , it was Since the publication of the Task Force’s report, two trends can be observed. Firstly, there has been a growth in the popularity of finance-based agreements, or simple price reductions, as compared with performance-based schemes. In a recent review of managed entry agreements in Europe, Dabbous et al.  note thaIn principle, medical devices are good candidates for PBRSAs, particularly for CED schemes, since there are often considerable uncertainties concerning their (cost-)effectiveness. This is mainly because the data requirements to obtain market access are often less stringent than those for drugs, and therefore devices are generally adopted in clinical practice with relatively little clinical or economic evidence , 11.. InThe pace of innovation in medical devices is considerable, with many new products entering the market every year. For example, in 2017, the number of patents in the field of medical technologies filed with the European Patent Office (EPO) was more than double compared to the number concerning pharmaceuticals , and the total expenditure on medical technologies in Europe was roughly estimated as €115 billion . Given tThis study is part of the EU Horizon 2020 COMED project that has been reviewed and approved by the Bocconi University Ethics Committee .The research was conducted in three consecutive steps: (1) development of a structured interview guide (2) interviews with decision-makers from a sample of European countries, (3) synthesis and qualitative content analysis of the interview data, the data made available by the decision-makers during or following the interview, and data on scheme characteristics previously obtained . The steWe developed a structured interview guide (Online Resource 1) that consisted of three sections. Section A included general questions on whether CED programmes underpinning the individual schemes existed in the decision-maker’s country and for which type of technology they were used. Section B included questions on 13 challenges for CED schemes for devices . Where CED schemes for devices existed, we also asked respondents how the challenges were met in their country, and the interview proceeded to Section C. Otherwise, the interview ended here. Section C included questions on the detailed characteristics of individual CED schemes for devices that had been either initiated or still ongoing in the past five years. These questions concerned a description of the device under evaluation, its clinical application, the objective of the scheme, key sources of uncertainty, funding of the scheme, its design, the decision rule, and outcome (if re-assessment was done), and any public source of information on the scheme.A first draft of the interview guide was circulated for comments among the COMED project partners. Subsequently the final draft of the interview guide was pilot tested during interviews with one Italian policy maker and two academic experts with extensive experience of CED in Canada and the USA, two countries with a substantial number of schemes.The interviews were conducted face-to-face or by telephone between June and December 2019. Decision-makers from decision bodies at the central  level were identified from the professional networks of the members of the COMED project team or the websites of relevant decision bodies in the following European countries: Austria, Belgium, Bulgaria, Croatia, Czech Republic, Denmark, England, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, the Netherlands, Norway, Poland, Portugal, Romania, Scotland, Slovakia, Slovenia, Spain, Sweden, and Switzerland. Other countries from the EU/EEA were excluded because it was not possible to identify a relevant decision-making body for the technology assessment of medical devices. We invited decision-makers to participate in the study by sending them an email with information on the COMED project and the objective of our study. When we were unable to identify a decision-maker from the networks or websites, we sent the information and invitation to the relevant decision bodies. In three cases where no relevant decision-maker could be identified , we invited academic researchers with relevant expertise to participate. None of these countries had however any CED programme for devices in place. We interviewed more than one decision maker from a given country in cases where schemes were operated in more than one jurisdiction , where more than one decision body was involved in operating schemes , or where more than one decision-maker, from different parts of the relevant organization, agreed to participate . We excluded Croatia, Iceland, Romania, and Slovenia from our sample after repeated attempts to schedule an interview by December 2019 were unsuccessful. Information on the individual CED schemes provided by decision-makers during or following the interview was supplemented with information on individual schemes previously obtained , compileThe transcripts were subjected to qualitative content analysis using deductive coding to meet the objective of this research. The results of each interview were reported in a table by one author (CF) and assessed by two authors (CF and VRD) who independently extracted the relevant information. Agreement on the data to be reported was then reached through discussion and further analysis of the original transcripts. The data obtained from Sections A and C of the interview guide, together with the data obtained prior to and following the interviews were used to identify and classify the characteristics of the existing CED programmes for devices according to the four phases of CED schemes: 1) assessing the desirability of the scheme; 2) designing the scheme; 3) implementing the scheme, and 4) evaluating it (1). These phases are described in more detail in Table The data obtained from Section B of the interview guide were used to obtain insight into the participants’ perceptions of the 13 challenges and into the factors that influenced their score for a particular challenge. The quantitative data obtained from Section B were used to calculate the mean (SD) and median (IQR) Likert scores for the 13 challenges . Then, we calculated these statistics separately for participants from countries with and from countries without a CED programme for medical devices. Because of the small sample sizes, we did not examine the differences in scores by performing statistical tests, but all factors which were perceived as having a positive or negative influence on each challenge were synthetized in tabular form.n = 15); national health insurance bodies (n = 2); hospitals (n = 3); and universities (n = 3) (see Online Resource 3 for details). Eighteen participants had high-level managerial roles related to the HTA of medical devices or services, or were responsible for the CED programme in their jurisdiction; four participants were technical advisers directly involved in the assessment of medical devices, and three were academics with an expertise in conditional reimbursement schemes. In seven out of the 23 jurisdictions (30.4%), CED programmes existed that included (or were specific to) schemes for medical devices . In France, two different programmes were identified: Post Registration Studies (PRS) for devices submitting for registration into the positive list of reimbursable products and services (LPPR list); and Forfait innovation (FI) for highly innovative technologies early in their development phase. Of the remaining jurisdictions, 5 (21.7%) operated CED programmes for drugs only , and 11 (47.8%) did not operate any CED programmes , although some of these may have other types of PBRSAs such as performance linked reimbursement schemes . In addition, single ‘one-off’ experiences with schemes for specific devices were reported by participants from Emilia Romagna Region in Italy and Ireland, in the absence of formal programmes for CED schemes for devices.We interviewed 25 participants from 23 jurisdictions. Respondents were from national or regional health authorities , whereas England, the Netherlands and Belgium mainly operated schemes in which a device is reimbursed only for patients who enrol in a clinical study . In France schemes were either OIR in the FI programme or OWR in the PRS, whereas in Germany the type of schemes depended on whether the technology was intended for inpatient use (OWR) or outpatient use (OIR).We identified differences in the design of schemes between countries. For example, Spain and Switzerland mainly operated schemes in which a device is reimbursed for all indicated patients while data are collected in a subset of patients  the device under evaluation or to key sources of uncertainty. Moreover, the study designs were similar between schemes in each country and mainly concerned either observational designs utilizing real-world data, or experimental designs to ensure a high level of evidence .We observed differences in the governance of CED schemes as well as in the roles and responsibilities of the stakeholders involved, regarding the development of the research protocol and the subsequent monitoring of the scheme. Overall, we identified two main approaches. In the first approach , the responsibility for the development of the study protocol, the monitoring of the scheme and the quality of the generated data relies entirely on the scheme applicants . However, in defining the study protocol the applicants typically must follow the recommendations of the relevant decision body. Usually, the protocol requires formal approval before study initiation, to make sure that it is suitable for addressing the identified uncertainties regarding the device. In the second approach, the responsibility for the development of the study protocol and the quality of the generated data are coordinated centrally , and managed either directly or through third-party research centres .Patient representatives may be involved in the initial assessment phase on the desirability of the scheme  or later during the recruitment phase of the study , but their involvement in the design phase and the development of the protocol was generally limited.During the scheme, the costs of care provision (including utilization of the device) are usually funded through the public health care system. Specific funding arrangements may be defined at the onset to cover the additional costs of the device or procedure, by either establishing a forfeit or negotiating an add-on to an already existing DRG tariff. However, different arrangements exist for covering the additional costs associated with the research, including the costs of developing the study protocol, scientific monitoring, data collection and analysis. These costs may be either entirely financed with public funds  or they may be partially or entirely covered by the scheme applicant . Notably, in some cases, funding arrangements also include resources for data collection. In other cases, health care providers are required to perform this task without any additional funding, for example, as a condition of participating in the scheme and gaining market access for the device .Decisions at the end of schemes mainly concerned the confirmation of the reimbursement status of the device, the refinement of clinical indications or conditions of use. For most of the identified schemes, no ex-ante decision rules that explicitly linked the scheme results to future decisions were defined. In most countries the schemes solely concerned the collection of additional evidence to reduce the identified uncertainties, while the final decision on the reimbursement, coverage or use of the device was integrated in the routine decision-making framework. A notable exception was the Netherlands, where the level of effectiveness that must be demonstrated during the scheme to obtain unconditional reimbursement was predefined at the onset of the scheme, in a covenant agreement signed by all stakeholders. Moreover, the covenant also addressed how to manage the withdrawal of a device in case it proved to be insufficiently effective or the data did not allow an informed decision .Notably, all participants reported having no, or only very little experience, with schemes that led to a negative coverage decision. Indeed, of the 24 CED schemes for which information on final decisions were available, coverage was confirmed  in 22 cases.Of the 25 participants, 18 scored the 13 challenges on the six-point Likert scales. Of these, nine were from jurisdictions with CED programmes involving devices, and nine were from jurisdictions with CED programmes involving drugs only. The seven participants who did not score the challenges were from countries without CED programmes.For most of the assessed challenges, scores were observed across the full range of the Likert scales, indicating no clear patterns in the decision-makers’ perceptions. Table Table Devices were generally considered to be more difficult to identify and monitor than pharmaceuticals, given that their routes to market are often less clear and may not be observed by those who are responsible for selecting potential candidates for CED schemes. The intrinsic characteristics of devices were also reported to pose additional challenges in the design and implementation of schemes. For example, device-user interactions and the context-specific factors which may affect device performance in the real-world were considered as challenges for the identification of all relevant uncertainty at the time of scheme initiation, and for the definition of the study protocol. In addition, devices may be associated with uncertainties that cannot be easily resolved within a feasible time frame for a scheme, such as uncertainties over the devices’ durability or their long-term performance in patients with different clinical conditions and physiologies. This in turn may increase the tension between the need to pragmatically rely on surrogate endpoints, which are rarely validated for MD procedures, and the relevance of the data collected to inform decision-making at the end of the scheme. In addition, routinely collected data, such as administrative datasets or electronic health records were expected to be less often available, or relevant, for devices, as compared with pharmaceuticals.Relating to the possibility of product modifications during the timeframe of the scheme, one of the main concerns related to the fact that such modifications could bias the results of the study or compromise the relevance of the new evidence collected. In this respect, being able to anticipate product modifications by means of dialogues with manufacturers and sharing of information was considered a potentially mitigating factor. However, the possibility of product modifications was not perceived by most of the respondents as a major challenge, or something which is likely to occur during the duration of a scheme.Similarly, about half of the respondents did not consider the possibility that similar products would enter the market during the period of the scheme to be an important challenge. Possible reasons related to the fact that most of the schemes evaluate a class of devices or a procedure rather than a single branded device, or that, even if focussed on a single product, they collected mainly non-comparative data. However, other respondents emphasised the difficulty of anticipating which products would enter the market during the schemes and the possibility that relative effectiveness estimates may not be meaningful anymore by the end of the scheme, as clinical practice changes more rapidly in the context of devices compared to pharmaceuticals.Finally, with respect to the existence of a learning curve, interviewees acknowledged it as a challenge which affects both the collection and analysis of data, as well as the design of the study, such as deciding on the number of clinical centres authorized to use the device as part of the scheme. However, direct experience with this aspect was generally limited across all respondents.CED schemes and their application to medical devices are important items on the policy and research agendas. The objectives of this research were to explore the characteristics and use of CED schemes for devices in Europe, as well as the challenges that decision-makers face when designing and operating these schemes. Our study importantly adds to the existing knowledge base by providing a comprehensive and multi-country overview, which was directly informed by surveys with European decision/makers.We found that 78 device-related CED schemes have been operated over the last 5 years in European countries. However, only seven countries had CED programmes in place for medical devices. To a large extent, this result may reflect the uneven application of HTA within Europe, since it may be difficult to develop a policy for CED schemes without having an established HTA capacity. For example, deciding that more data are required post-launch implies that some form of assessment of clinical or cost-effectiveness has been made. Nevertheless, HTA capacity cannot fully explain these differences, since CED schemes seem to be less frequently used for devices than for drugs .The characteristics of the identified CED programmes underpinning the individual schemes for devices varied between countries, which may reflect local differences in how HTA is organised and practised. For example, schemes were either initiated by the authorities , often as a consequence of the findings of an HTA for the technology, or as a response to a request from a manufacturer. We found similar patterns in the relative responsibilities for the funding of schemes and the design of study protocols although the authorities always played some role in study design, either by outlining a general specification or recommending that an independent research centre be involved. These differences in roles were also found in the aspects of the implementation of schemes, including the collection and analysis of data, which was sometimes the responsibility of the manufacturer and sometimes an independent party.One aspect that deserves attention is how devices are selected for a scheme. Indeed, CED is not a costless activity and its (opportunity) costs and benefits should be considered alongside other policy options, such as adopting or refusing adoption of the technology, based on currently available data, or negotiating a lower price. Aspects to be considered should include: 1) the expected value of research option(s) in terms of reduced uncertainty; 2) the direct costs of collecting evidence; 3) the opportunity costs of any delay in providing access to the technology because of the scheme; and 4) the existence of any irreversibility in the process  . Howeverin itinere during data collection and at the time of the reassessment of the technology once the scheme reported its results. This mirrors the findings of other studies of CED and market access schemes more generally [In addition, one general finding across all countries was that relatively little attention seemed to be paid to the evaluation of schemes, both enerally , 20, 21 The policy responses at the end of a CED scheme for devices may be more complicated than, for example, deciding on whether to include a drug on a formulary or to determine prescribing guidelines, since the reimbursement of devices, and the policies to determine their use, are often linked to the use of broader surgical, or other treatment, interventions. Therefore, policies probably involve adjustments to DRG tariffs, or changes to clinical guidelines, and/or hospital practice more generally. Hence, decision rules and policies for discontinuing the use of devices require attention in this context.ex-ante criteria for evaluating the schemes were defined, as was the case for almost all schemes for devices in Europe.Notably, all participants reported to have no or little experience with refusing to confirm reimbursement at the end of the schemes. While this may reflect the degree and type of uncertainties existing at the beginning of the schemes, it may also signal a certain difficulty in reversing the preliminary reimbursement decision once a technology has entered a scheme , 22, 23.Based on our observations of variation in the characteristics of schemes, it is difficult to prescribe a single preferred approach to CED of devices in Europe. Each country has specific local differences in HTA practices, although knowledge on how CED schemes have been used elsewhere can be used to develop local guidance. However, ideally a primary driver of the initiation of CED schemes would be the outcome of HTAs for the technologies concerned, since this can help identify the uncertainties in (cost-) effectiveness that (in principle) could be resolved through CED.The participants’ perceptions of the various challenges in initiating, designing, implementing, and evaluating CED schemes were varied and did not indicate that, in general, some challenges were substantially more important than others. The reasons for this are unclear, although in some cases the participant’s perception of a given challenge reflected local circumstances. For example, funding was not perceived as a major challenge in settings where public funding was made available, but a major challenge in settings where it was not. In addition, the scores obtained for those challenges that were ‘device specific’ did not differ substantially from those for the other, more generic challenges. While this aspect requires further investigation, our general impression was that some of the low scores given for ‘device specific’ challenges are attributable to a lack of direct experience with addressing these issues, given that the use of CED schemes for medical devices in some European countries is generally quite recent. For example, it has been argued that manufacturers may be reluctant to engage in a scheme and generate new evidence if other competitors entering the market with fast-follower products could also benefit from it . So, oneWe observed that the scores for the challenges were lower for respondents in countries where there was direct experience in CED for devices, as compared with those having experience with CED for drugs only. However, although the numerical differences in the scores were substantial, the small sample size means that no firm conclusions can be drawn. This could be explored in further research by comparing decision-makers’ perceptions before and after operating CED schemes and relating these perceptions to the general (HTA) infrastructure in a country.We used a combination of methods to obtain insights in the use of and challenges related to CED schemes in the relatively understudied context of devices, including a large set of European countries. The insights obtained allow learning from experiences across countries and increase the chances of having successful CED schemes in the future, by highlighting how decision makers perceive and deal with specific challenges. Nonetheless, some limitations also need highlighting. First, although we studied experiences in many European countries, we cannot be sure that our overview is complete as some countries were not included in the study. Moreover, although in each country we interviewed the person we considered to be most knowledgeable about CED schemes, we cannot be sure that the views of the participants are representative of the views of decision-makers more generally. Additionally, we focussed on the detailed perceptions of decision-makers, with a focus on HTA agencies at the national or regional level and (some) national payers because recent research suggests that decision-makers may be hesitant to engage in CED schemes . This maCED schemes for medical devices offer a promising tool to increase value for money in health care. While they are currently used in Europe, this study has shown experience with these schemes to be limited to a relatively small number of countries. Moreover, considerable variation exists between countries in how schemes are initiated, designed, implemented, and evaluated., using our list of challenges as a starting point, and considering for each of them the factors that decision-makers in this study outlined as having either a positive or negative influence. If a given challenge is considered to be important locally, the highlighted experiences of other countries in this study can help in addressing or overcoming them. That way, this study directly contributes to making CED schemes for devices a more effective policy option in the future.While the identified challenges in using CED schemes were perceived differently, none of them was unanimously considered insignificant. Hence, all challenges should be considered when initiating CED schemes in a given country. Our recommendation is that each jurisdiction embarking on CED schemes for devices should undertake its own ‘risk assessment’Supplementary file1 (PDF 197 KB)Supplementary file2 (PDF 84 KB)Supplementary file3 (PDF 34 KB)Supplementary file4 (XLSB 84 KB)Below is the link to the electronic supplementary material.


To examine the various roles of radiologists in different steps of developing artificial intelligence (AI) applications.Through the case study of eight companies active in developing AI applications for radiology, in different regions , we conducted 17 semi-structured interviews and collected data from documents. Based on systematic thematic analysis, we identified various roles of radiologists. We describe how each role happens across the companies and what factors impact how and when these roles emerge.We identified 9 roles that radiologists play in different steps of developing AI applications: (1) problem finder (in 4 companies); (2) problem shaper (in 3 companies); (3) problem dominator (in 1 company); (4) data researcher (in 2 companies); (5) data labeler (in 3 companies); (6) data quality controller (in 2 companies); (7) algorithm shaper (in 3 companies); (8) algorithm tester (in 6 companies); and (9) AI researcher (in 1 company).Radiologists can play a wide range of roles in the development of AI applications. How actively they are engaged and the way they are interacting with the development teams significantly vary across the cases. Radiologists need to become proactive in engaging in the development process and embrace new roles.• Radiologists can play a wide range of roles during the development of AI applications.• Both radiologists and developers need to be open to new roles and ways of interacting during the development process.• The availability of resources, time, expertise, and trust are key factors that impact how actively radiologists play roles in the development process.The online version contains supplementary material available at 10.1007/s00330-021-07879-w. Today, applications of artificial intelligence (AI) in radiology have become too important to ignore. The AI applications developed in the radiology domain are growing rapidly . BesidesThe literature on AI development in general , and in actually happen in the real world and whether radiologists play other roles in the development of AI applications. We also need to investigate how the roles of radiologists may change depending on the organizational conditions .Despite these suggestions, we still have a limited understanding of how these roles In this article, we examine the various roles of radiologists during the development of AI solutions, across eight companies, specializing in the development of AI applications for the field of radiology. We show that radiologists play a wider range of roles than what the current literature proposes. These roles, which occur in different steps of the AI development, are contingent to organizational conditions, such as the size of the organization, the background of the company, and the forms of interactions with radiologists.In the next section, we describe how we conducted our study. Then, we report the roles of radiologists in various steps of AI development. Finally, we discuss the implications of our findings for the radiology community and their participation in the development of AI.diversity of the AI developers and the ways they interact with radiologists in the development process. We followed the principle of “maximum variety” [We followed a multiple case-study approach  to captuvariety”  . Third, radiologists sometimes become the dominant actor in making the final decisions on requirements and features (problem dominator). We observed that these three roles are often based on occasional consultations, instead of continuous, ongoing interactions. Although radiologists were involved in the definition of the use case, they were often absent in specifying the conceptual design of the solution, such as defining the operations, designing the interface, and determining how to integrate the application to the radiology workflow.In the first step, developers define the medical use case: a problem to be solved and certain improvements to be achieved in radiology work. The use case should be medically relevant and offer a viable solution that can be implemented and worked with. We observed three roles of radiologists in this step. First, as data champion).Relevant, rich data, in the forms of images and scans, should be collected and prepared in order to train the algorithms and test them. It is crucial for the developers to recognize the relevant types of data and have a deep understanding of their attributes (what information is there), as well as the limitations and challenges of sourcing and working with them . Radiologists, who have the knowledge of medical data , can play an important role in finding and selecting relevant, rich data sources. In addition, the sourced data need to be checked, wrangled, and organized in such a way that can be fed into the algorithms (data curation). For that, data engineers consult radiologists to ensure that their curation of data does not violate the informational richness and accuracy of the data . Nevertheless, large companies had resources to hire radiologists to play this role. Radiologists were able to use their professional relations with PACS managers, and data warehouse managers in radiology departments, to get the data that are needed for the development of the AI solution. Sometimes, they had to decide how to collect and curate data. They also facilitated the interaction with privacy officers and ethical committees for ensuring secure and eligible handling of data.data labeler to be played by radiologists. Experienced data labelers came forward from long-lasting relations with medical institutions. Sometimes, vendors used automated techniques or used trained coders (who are not radiologists) for labeling data. These trained labelers were part of the technological team , who were already familiar with the field of radiology and with the conceptual design of the AI application. This was mostly the case for small vendors, who had limited resources to hire radiologists for labeling. Then, they relied on radiologists as data quality controllers who checked the labeled data for accuracy and consistency and resolved the complex cases.Before training the algorithm, the acquired data should be labelled in order to set a performance standard for the algorithm for (semi-)supervised algorithms. Next to labeling, developers need to establish the ground truth, against which they can assess the accuracy and performance of their algorithm. AI vendors considered the roles of algorithm shaper, by offering their insights and feedback on whether the technical decisions are aligned with the medical practice  and whether the results of the training are accurate, correct, and understandable from the perspective of domain experts .Once the data is curated and labelled, they are used for training the algorithm. The algorithm learns how to recognize certain patterns through detection, classification, and segmentation of the medical cases . Training should be based on the datasets that consist of a wide range of representative cases . Training is often performed by data scientists and programmers who select the type of algorithm, configure it, set the parameters, define the thresholds, execute the various iterations, and decide on stable, accurate configurations. These decisions are based on certain assumptions about how medical practice should take place and what is a relevant, valid result. There, radiologists played the role of algorithm tester by suggesting how to validate the algorithm on real-life datasets , checking the biases and sensitivity of the algorithm, examining the false-positive and false-negative cases and identifying common issues, testing the algorithm on complex cases, and specifying the boundary conditions and limitations of it. Since there is no universal standard database and procedure for validating algorithms, radiologists play a crucial role in defining the composition of validation datasets and the procedures that should be followed to ensure an accurate and mindful validation. All these enable the technical team to revise the application and algorithm in order to reach the expected performance. In addition, vendors relied on radiologists as AI researchers to design and execute scientific tests and produce scientifically sound evidence to endorse the performance of the AI application. In this role, they are also active in writing scientific publications  and crafting scientific sections of the legal documents .After training, the developers need to ensure that the algorithm is accurate and stable when it is used on broader datasets that represent the diversity and complexity of the real-world settings  and can be effectively worked with under the clinical conditions (testing). During this step, developers need to critically check the algorithm and its application based on feeding multiple, heterogeneous datasets. For that, radiologists perform the role of Looking across the companies and different development steps, we see that radiologists play a diverse range of roles.We see that radiologists play several other roles than merely labeling data. In fact, companies are increasingly relying on other ways to annotate their data than only recruiting radiologists . Hence, companies can benefit from the scarce, valuable knowledge of radiologists as domain experts in other steps of the development. Many of these roles reappear along the way of using AI. For instance, evaluation and validation of AI tools is not a one-time event, but an ongoing process, since regular updates of the software and using new types of equipment require revalidation of the algorithm. Radiologists should define how the tools can and should be evaluated in the clinical environment.Next to offering their expertise, radiologists can play important roles in establishing connections between the companies and medical institutions  and creating trust among their fellow radiologists. Next to their medical knowledge, their technical engagement is also essential to guide the development process. In addition, they can provide guidance with the seamless integration of an application in the radiology workflow, as well as ongoing evaluation of the AI at work. This way, radiologists can contribute to the establishment of relations with the prospective clients, the introduction of the AI solutions to the clinical practice, and the definitions of viable business cases.co-develop their products with certain clients, some radiologists from the clients are engaged in the process. Even though these radiologists may not be formally affiliated to the developing company, they can play all of these roles.We see that companies deploy various modes of collaboration with radiologists. When companies prefer active collaborations and they have enough resources and significant development projects, they tend to hire radiologists as part of their development team, with dedicated roles. Sometimes, companies rely on occasional interactions with external radiologists, especially when they have limited resources to hire radiologists. In some cases, radiologists are actively engaged by continuously interacting with the development teams, although they do not have any formal responsibility. When companies Looking across the cases, we see that several factors influence the active engagement of radiologists in the development process. Companies with strong medical background and connection with radiology departments engage radiologists from the beginning of the development process. Nevertheless, when companies are dominated by the technological teams and have a tech-heavy background, they tend to engage radiologists limitedly and in later steps of the development. This sometimes causes major challenges down the roadOur study showed that radiologists play a diverse range of roles in the development process. Next to these roles, there are still further engagement opportunities for radiologists especially in shaping the production of AI applications, designing interfaces, and configuring and aligning them into radiology workflow. As more diverse databases become available and stricter quality levels are demanded for the algorithms, radiologists can play a stronger role in sourcing data and advising on the development and validation of the algorithms. We expect that radiologists play new roles as organizations mature in the development, data sources become more diverse and available, and more diverse use cases are identified.Over time, organizations should not limit their interactions with radiologists to only occasional consultations; rather, they should actively and systematically bring the radiologists on board, with responsibilities and authorities to become part of the development team. This close collaboration is crucial for iterative co-creation process between the developers and the clients. At the same time, AI developers need to recognize various roles that radiologists can play and find ways to more effectively and actively engage radiologists all through the development steps and facilitate their engagement through various forms of collaboration .For radiologists, it is important to recognize the large, diverse, and yet open field of contribution. Radiologists should proactively engage in the development process and define their roles and contributions. Not all radiologists need to become developers. Yet, they need to learn enough about the development of AI applications to be able to bring their valuable medical experience into the development process. Radiologists should know the basic principles, limitations, risks, and pitfalls, and the techniques and technologies used for developing these applications and learn how to evaluate and monitor the algorithms 3.Although we examined a diverse sample of companies, future studies can expand the empirical scope and diversity of the cases by comparing how radiologists play roles across geographical contexts and different modes of production . Future studies can dig into the skills and capabilities that radiologists (need to) learn (before and) during their engagement in the development process. We can still learn from following the various career trajectories of radiologists who pursued different ways of engaging in the development of AI . Radiologists can play important roles in the “implementation” and “use” phases of the AI systems , which deserves further investigations. Finally, it is important to study the radiology community to see how they themselves perceive and play various roles in the development of AI applications.ESM 1(DOCX 31 kb)


Refugees are often among the most vulnerable of society’s members. The Office of the United Nations High Commissioner for Refugees  reports Refugees and displaced populations have limited access to health care services, including health promotion, disease prevention, treatment and care, as well as financial protection which is concerning . This vuWith the pre-existing fragile situation of refugees, the COVID-19 pandemic outbreak exacerbated the precarious living conditions and public health challenges among this population . RefugeeAccording to the 2020 vulnerability assessment of Syrian refugees report in Lebanon (VASyR) following the impact of COVID-19, an increase in the prevalence of households living in extreme poverty was evident increasing from 55% in 2019 to 89% in 2020 . In the Similarly, in Jordan, the rapid needs assessment report (2020) following the COVID-19 outbreak reported that 90% of the refugees do not have enough money to cover their basic needs during the lockdown measures . In addiThe current pandemic highlights how a population without access to essential services creates a risk for the concerned individuals along with the entire population in general . AccordiSeveral international organizations play an important role in integrating refugees into the national health system. The UNHCR works with partner organizations and ministries of health to ensure refugees receive the health support and medical treatment needed in emergencies and in stabilized and protracted situations . The UniAchieving Universal Health Coverage (UHC) in Lebanon remains a challenge on various levels. The health care system in Lebanon is based on a public-private partnership between multiple sources of funding that is highly fragmented . Around With a total population of six million, Lebanon has the highest number of refugees per capita in the world, whereby Syrian refugees make up 1.5 million of the total population . Most ofCurrently, there are no formal refugee camps established in Lebanon for Syrian refugees as per government policy . Thus, SLebanon currently also approximately half a million Palestine refugees who are registered with the UNRWA. Close to half of these reside in camps with complicated health problems emerging from the deteriorating living conditions, low wages, high poverty and deprivation ,32. PaleWith the aforementioned precarious health-related conditions that refugees are facing, efforts to integrate refugees in the national health systems are important and have achieved some progress, however, there are many ongoing challenges and significant support will be required to continue this effort. The ability of the existing health system to provide UHC, even for its citizens, is challenging. One challenge is the unclear policy framework for health care provision from the government, another is the overwhelming predominance of unregulated private sectors in the financing and provision of health care. In fact, secondary and tertiary care provision is almost exclusively given by the private sectors. Additionally, the Lebanese health care system is fragmented by target groups where each social group has access to different services and a different level of social protection . This tyThe health sector in Jordan consists of a multi-segmented public and private program . The twoAs of 2021, Jordan hosts one of the largest number of refugees in the EMR, with approximately 2 million registered Palestinian refugees (with UNRWA), and 665 000 Syrian refugees registered with the UNHCR . Unlike As for the Syrian refugees in Jordan, five camps were formed to accommodate the displaced individuals with Zaatari camp being the largest, hosting approximately 12%  of such refugees. The camp consists of 12 distinct districts, 58 community centers, 32 schools and 8 medical centers which are under the joint administration of the UNHCR and the Syrian Refugee Affairs Directorate (SRAD) where refugees have free access to health care services and resources ,45. HoweDespite the enormous efforts by the humanitarian sectors, such as UNHCR and UNRWA to provide equal health care services to refugees, a UHC plan needs to be implemented. The humanitarian agencies, such as the UNHCR, are experiencing a shortage of funding which may have a great impact on the refugee population . AdditioThe ability of existing health systems to provide health care is impacted by many factors including the loss of physical, human and institutional infrastructure; insecurity and instability; protracted emergencies and forcible displacement; ongoing conflicts with no foreseeable resolution in place; and uncertainty in the areas of economic burdens and political will. To meet the shared sustainable development goals (SDG)’s on health, each and every person’s right to health needs to be upheld, respected, protected and fulfilled. UHC and the human right to health go hand in hand as mutually inclusive realities that governments must be held accountable for. Moreover, governments are not only responsible to their populations, but also to individuals present on their territory in compliance with their international humanitarian and legal obligations. In efforts to ensure the provision of UHC in compliance with the right to health, governments must reassess their policies, legal frameworks, health systems and practices in order to evaluate whether or not they uphold the rights of refugees.The universality of the SDGs aims to ensure that everyone – including refugees– have access to a minimum standard of health care services. The approach undertaken by the health care systems in Lebanon and Jordan at present are unable to include refugees in the access to essential services without the involvement of international humanitarian organizations, mostly due to the fact that both countries, similar to others in the region, are themselves fragile when it comes to context and capacity. In Lebanon, achieving UHC remains a main goal of the National Health Strategy. However, its feasibility is questionable due to economic and political obstacles. To begin with, financing sources in Lebanon are heavily skewed towards domestic private expenditure and OOP spending. In 2017, OOP spending accounted for 33% of funding for health service in the Lebanese health system. Additionally, Lebanon’s high health GDP of approximately 8%-10% (which is comparable to many European countries) does not reflect better care, services, or health for the Lebanese or refugee population as the majority of the expenditure is concentrated in high-tech-high-cost interventions used by a small number of patients, with little to no focus on basic primary and preventive health care services. Furthermore, with Lebanon facing one of its greatest economic crises, funding for achieving UHC, more so integrating refugees into the national health system, is difficult and may only be possible through the help of humanitarian organizations.Integrating refugees in the national health system now, at a time where Lebanon is facing a great political instability crisis, could lead to greater challenges. This is because, integrating refugees in the national health system would infer that the government is accepting their permanent presence in Lebanon. This is particularly true for Palestinian refugees, whose presence in Lebanon has lasted for decades amid historical and political sensitivities. However, having a parallel functioning system (with UNRWA and UNHCR) is not efficient or sustainable in the long run; and reforms are necessary. With that being said, integrating refugees into the national health system may be difficult to achieve in the short run; however, it is important to move closer towards UHC and gradually integrate refugees into the health system while factoring in the political and economic complexities of the country. Ultimately, the goal and hope is for refugees to settle back in their respective countries.On the other hand, Jordan’s health care system has been shown to be one of the strongest health systems after the COVID-19 pandemic. However, with its growing national population and influx of refugees, Jordan struggles to ensure that everyone, including refugees, can access quality health care. Jordan faces a dual challenge of extending the overburdened health services, to meet the increased demand, while continuing to strengthen the quality of services. Additionally, the limited financing in the health care system is also a great factor that delays the progress towards UHC in Jordan. With that being said, it is pivotal for Jordan to expand their health care infrastructure and workforce in order to accommodate more people. Jordan must aim at improving the accessibility of public health services, reaching the poorest, and most vulnerable. With their continuous efforts in achieving UHC, Jordan faces challenges pertaining to the financial and material aspect in including everyone in the national health care system.The main obstacle that both Jordan and Lebanon currently face is the lack of sustainable resources, and subsequently/in parallel the political will, to officially integrate refugees in the national health system. It is pivotal that the fate of refugees be detached from protracted conflicts and the political sensitivities surrounding them. Though Jordan has initiated steps in incorporating refugees into the society at all levels by allocating a department in the government, Jordan Response Plan (JRP), aimed at outlining the rights of refugees, human rights considerations remain inadequate in addressing systemic and intersectional issues in the areas of access to adequate standards of aid and protection. Both countries signed the Universal Health Coverage 2030 Global Compact, demonstrating the high level of commitment to take action and progress towards UHC, not integrating refugees in the health system will continue to be an obstacle to achieving that.