The ESR International Forum at the ECR 2021 discussed effects of artificial intelligence on the future of radiology and the need for increased visibility of radiologists. The participating societies were invited to submit written reports detailing the current situation in their country or region. The European Society of Radiology (ESR) established the ESR International Forum in order to discuss hot topics in the profession of radiology with non-European radiological partner societies. At the ESR International Forum 2021, different strategies, initiatives and ideas were presented with regard to radiology community’s response to the changes caused by the emerging AI technology. AI and radiology should be understood as having a symbiotic rather than a competitive relationship.The use of AI can increase the visibility of radiologists and allow them to focus on other issues and expand their current role.Use of AI can significantly increase the level of care offered to patients.Radiologists need to be viewed as thought leaders and medical experts in AI.The advent of AI can propel radiology into a high visibility technology-enabled speciality.The ESR International Forum was established by the European Society of Radiology with the aim to discuss pivotal themes in the field of radiology with international societies from outside Europe. The ESR International Forum is held every year at the European Congress of Radiology (ECR), and participation is by invitation only. Due to the global pandemic of COVID-19, ECR 2021 was again held as a virtual congress and the participants of the ESR International Forum submitted written reports describing their society’s approach to the important issue of the use of AI in radiology.Previous topics discussed in the ESR International Forum include measures undertaken by radiologist in the fight against COVID-19, the relation between radiology and nuclear medicine, the position of ultrasound in radiology, the relation of general radiology and subspecialty radiology, the implementation of clinical decision support and imaging referral guidelines in the clinical routine, the position of interventional radiology within radiology, value-based radiology, and the strategies to engage the younger generation.The following societies submitted a report and presented the point of view of their respective country or region: The American College of Radiology (ACR), the Asian Oceanian Society of Radiology (AOSR), the Brazilian College of Radiology (CBR), the Chinese Society of Radiology (CSR), the Colombian Association of Radiology (ACR), the Egyptian Society of Radiology and Nuclear Medicine (ESRNM), the Indian Radiological and Imaging Association (IRIA), the Inter-American College of Radiology (CIR), the International Society of Radiology (ISR), the Japan Radiological Society (JRS), the Korean Society of Radiology (KSR), the Mexican Federation of Radiology and Imaging (FMRI), the Mexican Society of Radiology and Imaging (SMRI), the Radiological Society of the Emirates (RSE), the Radiological Society of North America (RSNA), the Radiological Society of Saudi Arabia (RSSA), the Radiological Society of Thailand (RST), the Royal Australian and New Zealand College of Radiologists (RANZCR), the Royal College of Radiologists of Thailand (RCRT), and the Radiological and Diagnostic Imaging Society of São Paulo (SPR). The European Society of Radiology (ESR) presented the situation in Europe.Radiology: Artificial Intelligence, launched in 2019, highlights emerging applications of machine learning and artificial intelligence in the field of imaging across multiple disciplines and keeps practicing physicians and imaging researchers up to date on the best emerging science in this subspecialty. Additionally, RSNA’s flagship journal Radiology also publishes cutting-edge AI research. In mid-2020, the RSNA COVID-19 AI Task Force launched the RSNA International COVID-19 Open Radiology Database (RICORD), envisioned as the largest open database of anonymised COVID-19 medical images in the world and in December 2020, the first annotated dataset from RICORD was published by The Cancer Imaging Archive (TCIA). M. C. Mahoney also explained that each year since 2017, RSNA has organized “AI Challenges” to spur the creation of AI tools for radiology. Finally, M. C. Mahoney concluded that radiologists’ participation in and contributions to patients’ healthcare teams are vital and that AI has enormous potential to make radiologists even more visible and valued members of those healthcare teams.M. C. Mahoney submitted the report on behalf of the Radiological Society of North America (RSNA). Pointing out that the RSNA recognized early on the importance of harnessing the power of AI to advance radiology, improve workflow and enhance patient care, M. C. Mahoney explained that the RSNA leads the AI charge, convening radiologists, data scientists and industry toward the shared goal of gathering, organizing and labelling volumes of data to help solve some of today’s most pressing health problems. Core RSNA AI-related activities are done through the RSNA annual meeting, peer-reviewed journals and data-gathering initiatives and collaborations. In 2020, RSNA’s first all-virtual meeting offered more than 100 AI education and research sessions. M. C. Mahoney further emphasized that the RSNA’s online journal G. McGinty, on behalf of the American College of Radiology (ACR), emphasized that radiologists, as both experts in imaging and leaders in the application of data science to health care, are the best equipped physicians to lead healthcare teams augmented by machine learning. Pointing out that in the USA there continues to be continued high demand for radiologists , G. McGiG. M. Figueroa Sanchez, on behalf of the Mexican Federation of Radiology and Imaging (FMRI), reported on the current situation in Mexico. He pointed out that radiologists have a unique opportunity to become more visible thanks to artificial intelligence. By already managing various different systems, such as Radiology Information System (RIS) and Picture Archive and Communication System (PACS), and considering teleradiology, interdepartmental sessions, tumour boards, teleconsultations, etc., radiologists have the possibility to drive the technological revolution. Even though many processes are probably going to be handled by AI in the future, G. M. Figueroa Sanchez emphasized that radiologists would have a crucial role in the complex process of integrating all information, putting it in the context of the patient and offering meaningful interpretation for the appropriate diagnosis and treatment. G. M. Figueroa Sanchez particularly emphasized that a visible radiologist adds value in the interpersonal relationship with patients, their families and also with clinical colleagues. He further stated that radiologists are to become data scientists, playing a key role in Precision Medicine, implementing the applications of AI and Big Data, thus being a fundamental part of the health system. Finally, G. M. Figueroa Sanchez stated that an AI-empowered radiologists will better communicate diagnosis to patients and referring physicians, will be more creative, empathic, with intuition and imagination and that being visible in the near future will certainly entail becoming experts in teleradiology and developing web-based portals with and for patients.N. Rodríguez Pedraza on behalf of the Mexican Society of Radiology and Imaging (SMRI) stated that radiology as a specialty has experienced an unprecedented progress; however, that recent advances in technology have made the radiologist more vulnerable, which is best reflected in the lack of communication between radiologists and patients and referring physicians. N. Rodríguez Pedraza emphasized that while the processing of data and medical images through the AI algorithms has helped in reducing the burden of repetitive tasks and has increased overall diagnostic accuracy, the absence of contact between radiologists and patients still remains an issue. She further pointed out that the SMRI implemented AI modules in its congress in order to ensure that Mexican radiologists are more acquainted with the automation of data and can in turn improve diagnoses without losing the human perspective. Furthermore, the SMRI is developing multidisciplinary webinars and it is intended to apply this multidisciplinary approach to future congresses as well, which will in turn make radiologists more visible by putting them in contact with various other specialties involved in the diagnosis of a condition.Finally, N. Rodríguez Pedraza stated that including the name and a photograph of the physician interpreting the image on the X-ray film would make them more visible.M. Arrieta Usta, reporting on behalf of the Colombian Association of Radiology (ACR), stated that currently many patients are not aware of the usefulness of diagnostic imaging studies and imaging procedures, with some of them also not knowing the role of the radiologist and other professionals in the treatment and diagnosis of their medical conditions. Additionally, many other factors, such as isolation, anonymity, high workflows in short time, diagnosis by other physician and improper administrative protocols, affect radiologists. And, with the application of artificial intelligence, radiologists are at risk of becoming even more distant from the patient. That is why M. Arrieta Usta pointed out the importance of person-centred care. In that sense, the ACR has created “Radiology for life”, a community of patients that allows ACR to inform patients about various topics related to diagnosis and image intervention. M. Arrieta Usta explained that through this community, the ACR is developing a series of actions that allow radiologists to approach patients, highlight the importance of radiology, diagnostic images and therapy by images , in addition to highlighting the role of professionals linked to the specialty. The ACR has also developed various scientific groups, one of which is Artificial Intelligence scientific group, coordinated by a committee consisting of an interdisciplinary team that works toward the proper use of artificial intelligence tools in the health sector in Colombia, in addition to promoting and supporting the developments in this area. Finally, M. Arrieta Usta explained that since 2019 the ACR has been venturing into the field of artificial intelligence through the promotion of different academic fora, in which issues related to the impact of these technologies on the exercise of radiology and diagnostic images could be discussed. In addition to education, the ACR Artificial Intelligence committee has focused its activities on three specific thematic pillars: regulation, ethics and technical aspects.V. Muglia, on behalf of the Brazilian College of Radiology (CBR), reported on the situation in Brazil. He explained that as diagnostic imaging is intrinsically linked to the development of new technologies its transformation through technological advances is inevitable. This is in turn also reflected in communication with other specialties and patients. V. Muglia stated that modern radiologists should increase their visibility by being present in multidisciplinary rounds, by being always available for consulting complex cases and by being accessible for patients. Pointing out that patients’ expectations are essential for changing how radiologists are seen, V. Muglia emphasized that radiologists should be readily available and should provide a good experience for patients in radiology centres, while also providing rapid turnover of reports. He further explained that the idea of AI as more of an ally than a threat to radiology is slowly gaining track. After an initial period in which AI was viewed as the panacea for solving all the problems of medicine, currently the irreplaceable role of the radiologist as the professional who is able to provide the best interpretation for imaging findings is being less questioned. V. Muglia also stated that currently several challenges prevent AI’s use on larger scale, namely the issue of generalisability and the model bias. However, it is possible to identify AI tools that can perform the simplest and most repetitive tasks of searching imaging findings, allowing radiologists to focus on the most complex role of finding the best interpretation. Commenting on the role of radiological societies, V. Muglia explained that they need to lead the narrative of the fundamental role of Diagnostic Imaging in patient’s care and emphasize the importance of Value-Based Health Care. Finally, V. Muglia stated that the notion that radiologists have a central role in clinical care and would not be replaced by AI should be strongly and repetitively delivered.M. Brandão da Costa, reporting on behalf of the Radiological and Diagnostic Imaging Society of Sao Paulo (SPR), pointed out that the use of AI allows for greater efficiency in regard to diagnostic imaging and helps radiologists find a more precise diagnose. In that sense AI will never replace radiologists but will allow them to become better professional who help patients more efficiently.B. E. González Ulloa, representing the Inter-American College of Radiology (CIR), explained that radiologists, who were on the forefront of the digital era in medicine, can guide the introduction of AI into health care. They will, however, not be replaced because radiology includes communication of diagnosis, consideration of patient’s values and preferences, medical judgment, quality assurance, education, policy making and interventional procedures. The higher efficiency provided by AI will allow radiologists to perform more value-added tasks, becoming more visible to patients and playing a vital role in multidisciplinary clinical teams. Commenting on the creation of algorithms used in radiology, B. E. González Ulloa stated that scientists creating them might need to learn more about medicine which is something where radiology professionals could serve as guides. Proper understanding of the limitations of algorithms by clinicians and proper understanding of clinical data by programmers is key to creating algorithms usable in the clinical work. In conclusion, B. E. González Ulloa stated that in Latin America, like other regions of the world, clinical radiologists play an important role in diagnosis integration. AI and its applications on imaging are powerful tools designed to increase diagnosis accuracy but will not replace radiologists.T. El-Diasty, on behalf of the Egyptian Society of Radiology and Nuclear Medicine (ESRNM), stated that it is important to explore the role of the radiologist in building AI models in collaboration with the engineering teams. In that sense the current role of the radiologists could be additionally expanded as radiologists can provide additional valuable information to engineers. Furthermore, T. El-Diasty mentioned an example of the Urology and Nephrology Centre at the Mansoura University, Egypt, that has an outstanding collaboration with an engineering team from the University of Louisville, KY, USA. This collaboration was established in 2004 and is still ongoing in many different projects related to the early detection and diagnosis of different diseases. T. El-Diasty concluded his report by stating that AI is there to assist radiologists and the radiologists and AI will continue to complement each other in the future.L. Jamjoom, on behalf of the Radiological Society of Saudi Arabia (RSSA), explained that it is important for a visible consultant to be accessible, affable and accomplished that a visible radiologist that uses AI will never be replaced by it. L. Jamjoom emphasized that AI has a complementary role and helps with tedious points in workflow, resulting in improved patient outcome, and enables radiologists to spend more time on what is important. Furthermore, she mentioned that a survey on AI, conducted in Saudi Arabia, examined radiologists’ attitude towards AI and that many radiologists found that AI will increase overall efficiency by reducing analysis and reporting time, enhance the diagnostic accuracy and provide medical facilities to remote areas, help with triaging and pinpointing alarming findings, improve health care, decrease diagnostic errors and will increase standardization. At the same time radiologists were worried that AI might cause a reduction in the number of doctors and specialists, increase the number of false results and lead to stereotyping and stiffness, wrong pattern recognition, decrease in the need for primary care physicians, lack of accountability and decrease when it comes to the communication with patients and case discussion. Finally, L. Jamjoom stated that Saudi Arabian expenditure on AI projects accounts for 63% of global investments and that AI implementation will involve a total of 14 hospitals and 56 small healthcare centres.Dr. AlBastaki, on behalf of the Radiology Society of the Emirates (RSE), explained that it is important to note that with the increase in the use of AI radiologists should have a more effective role in supervising the results procured by the AI. In that sense, radiologists should not feel threatened by AI but should instead consider it as a chance to evolve and focus on other issues. Dr. AlBastaki pointed out that radiologists should be receptive to the new developments and should offer continuous feedback for the developers to improve the systems. Finally, Dr. AlBastaki mentioned that currently in the United Arab Emirates there are more than 5000 chest X-Rays performed daily, which leads to great pressure on radiologists in the country and is something where future AI developments will certainly help.intelligent assistant to radiologists, could help. C. Amarnath explained that in this role AI could bring efficient synergies to radiology: reduce radiologists’ visual fatigue, address the exponential workload burst, triage normal, delegate repetitive tasks to machines and accomplish tasks that do not need human intervention. Reporting on the situation in India, C. Amarnath reported that the AI-division of IRIA is aiming to identify the needs and demands in India, generate new ideas and use clinical data to accomplish algorithms and build products. He also stressed the importance of introducing some sort of governance for decision-making based on autonomous AI-based algorithms. New policies regarding regulation of data protection, privacy of sensitive information, cybersecurity, accountability and responsibility issues need to be addressed as well. C. Amarnath concluded his report by stating that it is high time for radiologists to reclaim lost ground, utilize AI to optimize workflow, re-establish their role in clinical interdisciplinary teams for value added work, decision making and intervention.C. Amarnath, on behalf of the Indian Radiological and Imaging Association (IRIA), emphasized that radiologists have always adopted and adapted to major technological changes, digitalisation and new modalities. He pointed out that one of the main roles of the radiologist is to serve as a bridge between clinicians and patient diagnosis. However, with clinicians’ dependence on imaging increasing exponentially, studies have become more complex, and at the same time, there is a huge surge in volume, as witnessed during the current COVID-19 pandemic. Expectations towards radiologists have escalated with need of high-resolution data, 3D reconstructions, specific reporting standards, quantitative parameters, frequent follow-ups, quality control and research. Pointing out that imaging has become a commodity in medical practice, C. Amarnath stated that radiologists need to emerge as the conductors of multi-disciplinary teams at the forefront of clinical care and that this is something where AI, as an W. Tanomkiat, reporting on behalf of the Royal College of Radiologists of Thailand (RCRT), stated that RCRT established a dedicated committee, consisting of radiologists, data scientists and data analysts, in order to study the impact of AI on diagnostic radiology, advise and recommend best courses of action to the RCRT Broad of Directors. The Committee quickly recognized that AI will inevitably affect radiology in Thailand and the RCRT announced the first version of the suggestions for AI users in diagnostic radiology and signed a Memorandum of Understanding (MOU) on exchanging the resources with the Thailand Centre of Excellence in Life Sciences (TCELS) . W. TanoJ. Chaiyakum, on behalf of the Radiological Society of Thailand (RST), reported that the RST sent a questionnaire to Thai radiologists, to survey their opinions regarding the effect of AI on their practice and income. Almost 80% of the respondents stated that AI currently has no role in their workplace, and out of the remaining 20% most respondents stated that AI is used mostly for screening and as a help with detecting lesions. It was reported that more than 2/3 of respondents believe that AI will play a significant role in the future, in particular when it comes to the aforementioned issues of screening and detecting lesions. Radiologists in Thailand are overall not fully concerned that they will be replaced by AI and more than 50% believe that their income will not be decreased or will be only slightly decrease due to AI. Finally, J. Chaiyakum reported that a majority of radiologists would encourage young doctors to pursue a career in radiology.replacement of radiologists by the AI, S. Liu emphasized that radiologists are becoming more dependent on AI in some specific scenarios after few years of applications, but also have to dedicate more time to reassuring the patients and collaborating with clinicians. Thus, AI is not replacing radiologists’ “traditional work” but is extending their work boundaries. Finally, S. Liu concluded that AI has brought a profound change to the industry of medical imaging and that those who accept change, embrace change, and accommodate to change, are the ones who will own the future.S. Liu, on behalf of the Chinese Society of Radiology (CSR), reported on the situation in China. He began his report by stating that the basic consensus that radiologists and AI are in a symbiotic rather than a competitive relationship has long been existing in China. Therefore, a focus was placed on how AI can assist doctors in China and as a result number of radiologists using AI as an assisting tool is increasing year by year. Quoting Dr. Curtis Langlotz, a radiologist from Stanford who stated that “AI won’t replace radiologists, but radiologists who use AI will replace radiologists who don’t”, S. Liu explained that this is becoming reality in China. Commenting on the notion of potential S. H. Park submitted a report on behalf of the Korean Society of Radiology (KSR). He explained that main activities of the KSR regarding AI include educational programs, not only for radiologists but also for industry and government agencies, as well as making policy suggestions by collaborating with government agencies regarding the regulatory approval, insurance coverage and clinical implementation of AI devices. One of the central positions that KSR has in those AI-related activities is "human in the loop". As the strengths of humans and AI are different, it is critical both technically and ethically to include them both in order to ultimately improve health care. S. H. Park stated that Korean radiologists are overall no longer swayed by the initial hype regarding their replacement by AI and are instead examining how they can engage with and embrace AI to improve radiology practice for the good of the patients. Consequently, they understand that any substitution of radiologists by AI only occurs if radiologists cannot be found when needed which is why visibility is of utmost importance. S. H. Park also reported that in 2020 the KSR accomplished several AI-related projects, including an e-learning program on AI as well as a mini-textbook on AI in health care. S. H. Park concluded his report by stating that the KSR plans to expand its scope of AI education by designing a more in-depth, hands-on course that aims to train radiologists who could serve as leaders for any AI research project or any similar multi-disciplinary teamwork and perform as educators regarding the use of AI in medicine.Glow to generate images that emphasize deviations from normal distributions. Since this method does not depend on supervised learnings by annotating specific image findings by human radiologists, it is expected that is will reduce the annotation cost and will be able to emphasize any abnormal findings.S. Aoki, on behalf of the Japan Radiological Society (JRS), reported that the JRS is engaged in several actions in order to cope with, co-exist and master AI. He explained that some universities in Japan, such as the Kyoto Prefectural University of Medicine, have started daily case conferences thus facilitating communications between radiologists who work in different cities. Radiologists are currently at the centre of these conferences with other departments and at the centre of management and utilization of image information. S. Aoki also reported that the JRS is creating a nation-wide diagnostic imaging database called the Japan Medical Imaging Database (J-MID), which collects radiation exposure records, DICOM images and diagnostic imaging reports. In the J-MID project, the JRS is collaborating with the Japanese National Institute of Informatics to develop an artificial intelligence system for subarachnoid haemorrhage detection using datasets obtained from multiple institutions. S. Aoki also stated that the University of Tokyo is currently working on boosting the broad capabilities of radiologists by applying generative models such as D. Varma, on behalf of the Asian Oceanian Society of Radiology (AOSR), stated that the work of radiologist is not only interpreting imaging findings but more importantly being integral to a clinical team deciding if an imaging is indicated and justified and if so, what is the most appropriate imaging that will progress patient care as well as timely communication of significant findings. He pointed out that radiologists are more and more visible and clinically orientated and that AI, unless it is moderated by a human being, in this context, the radiologist, will not be able to be harnessed in the manner that will be safe for the patient. Therefore, a radiologist that is not visible and that does not use AI will be the one that is replaced by AI. D. Varma also pointed out that mutual communication between clinicians and radiologists can never be replaced by AI and that skill-based examinations, such as ultrasound or interventional procedures, cannot be replaced by AI no matter how delicate the robot is. On the other hand, AI can enhance the safety and efficiency of a radiologist’s work, helping to pick up accidentally missed lesions, prioritise the examinations waiting for reporting and improve the quality of radiology reports. D. Varma concluded that AI would add value to radiologists in reducing errors and triage of workflow; however, if the “invisible” radiologists choose to be completely dependent on AI, they will be the most at risk of being replaced by it.Ethical Principles for Artificial Intelligence in Medicine [Standards of Practice for Clinical Radiology [L. Lawler, reporting on behalf of the Royal Australian and New Zealand College of Radiologists (RANZCR), stated that AI and machine learning technologies offer wide-ranging benefits to radiology and health care in general, but they will not replace radiologists in the foreseeable future as radiologists have a central and critical role in clinical decision-making and are becoming increasingly accessible to patients and referrers through multi-disciplinary care teams and advances in interventional radiology. L. Lawler explained that the adoption of AI presents many opportunities to support and enhance current workflows, allowing for a more efficient and accessible healthcare system that delivers improved outcomes for patients and that the RANZCR have been leaders in the adoption of AI in health care for the past 3 years. In August 2019, RANZCR became the first healthcare body in the world to develop and release adiology  in SepteValue-Based Radiology: A New Era Begins [A. Rockall reported on behalf of the European Society of Radiology (ESR). She explained that media attention in 2015 brought radiology into the spotlight and opened a wide conversation concerning the role of the radiologist in value-based patient care , an impoa Begins . This ina Begins  that wasa Begins , 15. AdaWHO Rapid Advice Guide on the Use of chest imaging in COVID-19 [R. A. Mendonça, on behalf of the International Society of Radiology (ISR), reported that the ISR collaborated in the COVID-19 , which iThe future of radiology might be here sooner than anticipated. Despite initial apprehension many radiologists today believe that AI will optimize radiologists' workflows and develop new avenues for practicing radiology. Crucial element in that regard is going to be the increased visibility of radiologists which can only be achieved through the use of AI. In that sense only radiologists who embrace the possibilities of AI and are open to change will be able increase their visibility and establish themselves as an irreplaceable part of the medical teams of the future.Radiologists around the world are experiencing fundamental shifts in their profession brought with the advent of AI. However, AI is creating new opportunities and is not only increasing the level of care that can be provided to the patients but is at the same allowing radiologists to redefine their role. Instead of making the radiologists obsolete AI will allow them to reposition themselves and increase their visibility by positioning them technology leaders in the inevitable change that will affect the whole of medical profession.


We aim to evaluate whether the first National Emergency Medical Service (NEMS) improved access to hospital care for the people of Sierra Leone. We performed an interrupted time-series analysis to assess the effects of NEMS implementation on hospital admissions in 25 facilities. The analysis was also replicated separately for the area of Freetown and the rest of the country. The study population was stratified by the main Free Health Care Initiative (FHCI) categories of pregnant women, children under 5 years of age, and populations excluded from the FHCI. Finally, we calculated direct costs of the service. We report a 43% overall increase in hospital admissions immediately after NEMS inception . Analyses stratified by FHCI categories showed a significant increase among pregnant women  and among individuals excluded from the FHCI . The observed effect was mainly due to the impact of NEMS on the rural districts. The estimated recurrent cost per ambulance ride and NEMS yearly cost per inhabitant were 124 and 0.45 USD, respectively. To our knowledge, this is the first nationwide study documenting the increase in access to healthcare services following the implementation of an ambulance-based medical service in a low-income country. Based on our results, NEMS was able to overcome the existing barriers of geographical accessibility and transport availability, especially in the rural areas of Sierra Leone.  Multiple barriers prevent an equitable access to healthcare services in low-income countries (LICs), leading to higher morbidity and mortality rates for both acute and chronic diseases, especially in rural communities ,2,3.Sierra Leone is one of the least developed countries worldwide, where access to healthcare is mostly constrained by geographical barriers, extremely high out-of-pocket expenditures, lack of skilled medical staff, and poor service quality ,5. FurthTo tackle the high rates of maternal and neonatal mortality reported in its territory, in 2010, the Sierra Leone government launched the Free Health Care Initiative (FHCI), which waived all medical-related fees for pregnant and breastfeeding women, children under the age of 5 years, and Ebola survivors . AlthougSimilar to most of the other African countries, Sierra Leone has been long devoid of any formalized prehospital care system . Since tIn line with recommendations issued by the World Health Organization (WHO), in 2013, the African Federation of Emergency Medicine advocated for the development of prehospital care systems to reduce the high morbidity and mortality rates reported in African countries ,12. ThusThe aim of the present study was to evaluate the impact of this initiative on access to hospital care for the general population, with a special emphasis on underserved rural communities. We also provided an overview of the direct costs of the service. This was a retrospective study using monthly aggregated data on hospital admissions across Sierra Leone, recorded between October 2017 and November 2019. The study encompassed 25 hospital facilities distributed across the 14 districts of Sierra Leone, including government district hospitals, faith-based clinics, and health centers managed by non-governmental organizations. As the NEMS was introduced in the various districts at different times , in charge of the primary assessment and care of patients, a fleet of 81 ambulance units, and healthcare facilities at different levels . It recoAmbulances were staffed by a paramedic and an ambulance care driver, trained in the basic principles of prehospital emergency care, including management of medical emergencies, trauma, obstetrics and gynecologic and pediatric emergencies . TreatmeData were retrieved from the national referral coordinators’ database, storing information on all incoming referrals collected at each hospital facility . After iReferral and in-hospital data have been recorded nationwide by the network of local referral coordinators since September 2017, before the implementation of NEMS, when the vast majority of patients used either private or public means of transport to reach the hospital from the peripheral health units. This structured data collection system has been incorporated in the NEMS system upon its inception, and scans of patient referral forms filled by ambulance teams have been used to crosscheck data collected by referral coordinators at the hospital level. Data extracted for this research included age, gender, FHCI category, priority (“urgent” or “non-urgent”), date and time of arrival at the hospital, and transport mode (“NEMS ambulance” or “other”). Data on the population of Sierra Leone and its districts were extracted from the 2015 Sierra Leone Census, as reported on the Sierra Leone Statistics website . AdditioWe adopted standard methods for interrupted time-series to assess the effects of the introduction of the NEMS ,18. We cCalculation of the cost included recurrent costs of the intervention, while capital costs have been excluded from the analysis as the implementation phase of the project was entirely funded by the World Bank and relied on vehicles donated from around the world during the 2014–2016 Ebola crisis.A total of 28,574 hospital admissions in the 14 districts of Sierra Leone were included in the analysis. Both in the period before and after the implementation of the NEMS, women’s access to hospital facilities was much higher compared to men. The majority of hospital admissions were represented by patients included in the FHCI, namely pregnant women and children under 5 years of age .An overview of the obstetric diagnosis associated with “red” codes assigned in the category of pregnant women is presented in the In the different subgroups, a statistically significant increase was reported among pregnant women  and those exempted from the FHCI , but not for children under 5 years of age . We did not find evidence of an additional gradual effect of the intervention over time, as there was no statistically significant change in the underlying trend in admissions after the implementation of the NEMS. Analyses stratified by area showed a very different pattern in Freetown compared to the rest of the country . OverallConversely, in the rest of the country an immediate increase in the number of hospital admissions was evident both overall and for all considered subgroups, except for children under 5 years of age .The results of our study show that the implementation of NEMS in Sierra Leone was associated with an overall increase in access to hospitals. The rise in hospital admissions was observed not only among individuals with free access to healthcare services, but also among those not exempted from payment, suggesting that prior to NEMS inception, geographical and transportation barriers played an important role in limiting the access to the country’s healthcare services, irrespective of the possible financial constraints of subjects. This aspect corroborates findings from studies performed in other LICs, where distance or travel time to health facilities was inversely related to the healthcare seeking behavior of the population ,20.Moreover, the observation that underserved rural communities benefited the most from the activation of the free ambulance service in terms of ability to reach hospital care suggests that NEMS was effective in reducing health inequalities between urban and rural areas in Sierra Leone. Fittingly, we did not observe any major increment in hospital admission rates in the Freetown area, the most heavily populated district in the country, characterized by a higher concentration of healthcare facilities and better road conditions. Indeed, it is well established that urban residents of LICs have a much better prospect of accessing healthcare services compared to that of people living in rural communities, inevitably affected by structural constraints and health inequities ,22,23.The results stratified by FHCI categories show an increase in the number of hospital admissions of pregnant women upon NEMS inception. Although healthcare service utilization rates among pregnant women increased after the implementation of FHCI, this improvement was deemed not adequate enough to meet the sustainable development goals of the country ,24,25. IAnother interesting finding of our study is the decreased number of hospital admissions among “children under 5” in the Freetown area, which was evident immediately after the implementation of the NEMS. This observation may be explained by the presence of a big referral facility in Freetown, the Ola During Children’s Hospital, which used to treat a substantial number of pediatric cases from other districts before NEMS inception. Our results suggest that the NEMS allowed a better distribution of pediatric patients among the districts surrounding the capital, with a possible improvement in the use of available healthcare services.To our knowledge, this is the first nationwide study documenting the impact of a newly developed EMS on health service utilization rates in a LIC. Although previous studies performed in low- and lower middle-income African countries attempted to evaluate the few existing referral systems, analyses were restricted to the local, district, or regional level ,33,34,35A relevant aspect that arose from the implementation of NEMS in Sierra Leone was the importance of developing a robust and reliable patient referral network, featuring an efficient communication network and well-defined standard operating procedures . ReferraOf note, the vast majority of hospital admissions recorded after the implementation of NEMS involved the use of NEMS ambulances. This high utilization rate is associated with an average cost of the service that is slightly higher compared to other African countries, where, however, referral systems are only regionally developed and less structured than NEMS ,39. It iThe findings of this study should be interpreted considering some limitations. First, although the systematic collection of data was deemed accurate, information regarding admission and discharge diagnoses of patients was recorded as free text and were often either unreliable or missing. Additionally, information regarding in-hospital treatment was missing. For this reason, it was not possible to evaluate whether the implementation of the NEMS effectively translated in better clinical outcomes for patients nor if hospital admission had any detrimental effect on patients excluded from the FHCI who had to bear out of pocket expenditures to receive treatments. To overcome these technical limitations, we plan to perform additional studies on selected healthcare facilities, where more accurate clinical information might be available. Moreover, as the data analyzed did not include patients’ follow-up after hospital discharge, we were unable to provide information on how patients returned home from hospital facilities, and patients’ perspectives were not investigated in a qualitative manner. As in any observational study, the role of possible confounders cannot be definitely ruled out. However, confounding in time series studies is usually limited to factors that are related to the outcome of interest and change at the time of the intervention . In contIn conclusion, the present study demonstrates that the implementation of NEMS in Sierra Leone enhanced access to hospital care among vulnerable populations by overcoming existing barriers, such as geographical accessibility and transport availability, especially in the rural areas of the country. Our findings suggest that NEMS was able to boost the effects of the FHCI, therefore building on the existing governmental strategy to improve maternal and child health indicators and to achieve the sustainable development goals of Sierra Leone. Altogether, our findings may serve as the basis for the development and evaluation of prehospital emergency and referral services in other LICs.


The coating of materials with specific films is widely used to improve material properties and many technologies exist to perform it. In the last few years, the replacement of wet electrodeposition processes has been continuously encouraged in the EU due to the problematic waste management linked to those processes. In this paper, magnetron sputtering is studied as an alternative to conventional electrodeposition by comparing the technologies’ environmental impacts and costs. From the study, it appears that while magnetron sputtering greatly reduces hexavalent chromium emissions over the production, it has an increased electricity consumption mostly due to its lower production capacity, thus leading to more greenhouse gas emissions. Furthermore, a short discussion on the quantification of the impact of hexavalent chromium emissions is conducted. Regarding costs, the electrodeposition process has a lower cost of investment and of consumables, but requires more work time for the different steps of the process, making the total price per functional unit roughly equal. However, the cost per functional unit strongly depends on assumptions on the required work time, for which a sensitivity study is performed. Finally, the impacts of these two competing coating processes are discussed to complete the technological comparison for the case of hard chromium deposition. With the current focus of the scientific world on sustainability, finding more environment-friendly, cost-effective alternatives to current processes and technologies has become one the most important goals of researchers all around the world. In particular, thin films and coatings are increasingly prominent in everyday life, thanks to the ever-growing interest in micro-electronics, screens, solar panels; for instance, the global number of smartphone users increased by 44% in just 4 years, from 2.5 billion in 2016 to 3.6 billion in 2020 .In this paper, hard chrome coatings are studied. These coatings are, among other applications, widely used in the aerospace industry to improve the mechanical properties of many aircraft components: landing gears, engine bearings; these coatings increase the lifetime of the components by reducing friction and improving hardness, as well as corrosion and wear resistance. They also play a large role in the automotive industry for the same reasons. In addition, the use of chromium for decorative applications account for about 70% of the applications in the automotive coatings market .3) used in the process. As such, viable alternatives to the electrodeposition process are highly sought after. In this context, it is relevant to study processes with little to no waste generation as alternatives to the electrodeposition process such as magnetron sputtering. However, magnetron sputtering is not the only alternative, as, for the deposition of chromium, one may think to the use of trivalent chromium, thermal evaporation or electron beam evaporation. Many more processes using a different coating material such as high-velocity oxyfuel (HVOF) coatings for instance could also be considered substitutes. While all these technologies deserve full attention, they are beyond the scope of this paper, which focuses on magnetron sputtering.The main deposition process used is electrodeposition with hexavalent chromium. There is, however, a push from governmental institutions to forbid the use of the carcinogenic chromium trioxide , but that they consider several of them in order to draw a picture as complete as possible. The present paper can thus be seen as a case study for technological comparison, combining life cycle assessment and economic considerations.In this paper, a brief description of magnetron sputtering and electrodeposition is first made to highlight the context of the present work. Then, the assumptions used for describing the coating processes, along with description of the life cycle assessment (LCA) and techno-economic assessment (TEA) methodologies are detailed in the section dedicated to methods. The results of the LCA and TEA are then presented and discussed, followed by a discussion on the toxicity and potential human impacts. Finally, the paper concludes by highlighting the main challenges of magnetron sputtering in view of these results, as well as proposes some perspectives to improve this comparison of competing technologies.Electrodeposition is a deposition technique that has been mentioned already in the 18th century , and forIn Europe, the electrodeposition process has been declining in use. Chromium compounds are not manufactured in Europe anymore and are imported less than they were previously. For example, imports of chromite decreased by 18% between 2012 and 2016 . In addiIII, however, shows promise in the form of better current efficiency and deposition rate [Electrodeposition is a mature process, and recent research around the chromium plating technology has mostly targeted pulsed plating (PP), trivalent chromium deposition and waste water management. Each of those areas of research aims to overcome part of chromium plating’s drawbacks. In the case of PP, research seems to indicate that using pulsed current can improve the coating’s properties, such as adhesion, wear and corrosion resistance, current efficiency, microstructure and more . That teion rate . In receion rate ,19. AmonSputtering technologies have been studied since the 1800’s ,21. The 2 compared to around 10 W/cm2 for typical DC magnetron sputtering). This increases the ionization rate and allows for a better control of ion bombardment and tailoring of the phases of the growing film [Magnetron sputtering has been used for decades to produce chromium coatings , with haing film .In the literature, LCA has been used to assess the environmental impact of electrodeposition and to compare it with potential coating alternatives such as HVOF , or combBenveniste et al. did an LCA study to compare different alternatives for surface functionalization , includiIn the present chapter, the methodologies used to assess the environmental impacts and costs will be presented. They require the identification of a complete and justified inventory of inputs and emissions, as well as the definition of a functional unit for the comparison of technologies.The LCA being a standardized methodology, the inventory of mass and energy flows as well as the definition of a functional unit are precisely described in international standards (see below). In the present work, the functional unit and process inventories will be kept unchanged for the TEA study. A detailed description of the processes is conducted in the Life cycle assessment (LCA) is a methodology that assesses the environmental impacts of a process or product over its whole life cycle. It quantifies the environmental impacts linked to emissions or resource consumption at each step of production. Those impacts can then be sorted in impact categories and assessed. Following the ISO 14040 and 14044 norms for LCA ,40, the 2. This functional unit has been chosen because such cylinders are commonly coated with chromium film to improve their tribological and mechanical properties for use in several sectors, such as the textile industry for example [A functional unit is the amount of product generated for which the environmental and economic impacts are evaluated. In this study, it will be the coating of a steel cylinder’s side with a 20 µm thick chromium film. The cylinder has a height of 80 cm and diameter of 40 cm, corresponding to an area to be coated of about 1 m example . In totaIn the present case, the goal of the study is to assess and compare the environmental and techno-economic impacts for the coating of a functional unit with two deposition techniques: electrodeposition and magnetron sputtering. Regarding the environmental impacts, the main categories studied are the global warming potential and the hexavalent chromium emissions. Justifications for those choices are given in the impact assessment section . The stuRegarding the scope, this study will only focus on the deposition step itself and will not consider: (i) cylinder manufacture, (ii) cylinder transport, (iii) deposition equipment manufacture, (iv) use, maintenance and end phases of the coated cylinder’s life cycle, (v) maintenance steps for the equipment. This study will however take into account: (i) the energy used in the coating process, (ii) the materials used, from extraction to disposal when data is available, (iii) process emissions.Because the properties of coatings deposited with electrodeposition or with magnetron sputtering are considered equivalent, the use and the end-of-life phases of the cylinder are not taken into account.Material and energy flows inventories for the coating of a functional unit are described in the Through the emissions listed in the inventory, impact scores in 18 different midpoint categories can be calculated using the formula:x is the impact score of the impact category x; Xi is the emission i and CFxi is the characterization factor of the emission i in the impact category x.IMThese impact scores are usually hard to interpret outside a purely comparative setting. To better show the outliers in the impact categories, the normalized results will be shown. The normalization is done by correcting the impact score with a normalization factor:x is the normalized impact score in the x category and NFx is the normalization factor of the x category.NIThe normalization factor, in the case of ReCiPe 2016, corresponds the global average emissions of one person in 2010. Characterization factors and normalization factors can be found on the RIVM website .TOT is given in Equation (3).As mentioned earlier, the goal and scope of the cost study are similar to the ones identified for the LCA. The economic assessment is based on the calculation of the processes OPEX and CAPEX. To determine the operational costs for the coating of a functional unit , an inventory of process streams is available in OP is the operating cost; CL is the labor cost; CEQ is the equipment cost; CELEC is the cost of electricity; CMAT is the cost of the materials used; CM is the maintenance cost and CA is the cost of annuity.CAnnuity and maintenance costs can be derived based on some hypotheses. First, a yearly maintenance cost of 2% of the equipment investment is assumed . Then, eIn this section, environmental results are presented. First, using the previously obtained inventories, environmental impacts are worked out using the ReCiPe method to have a global view on the impacts of each process. Results are normalized corresponding to a standardized way of showing LCA outcomes.In a second step, the different impacts of chromium production for each process are studied. As the synthesis path for the chromium source is different for each process, it is relevant to focus on the way the production impacts differ. This implies studying both metallic chromium production for magnetron sputtering as well as chromium oxide production for electrodeposition.Finally, a discussion on the two most relevant environmental impacts is considered. From the study, it appears indeed that global warming potential and human carcinogenic toxicity are the most representative indicators to highlight differences between the two processes. Global warming potential is relevant to assess the very different energy consumptions for each process, while human carcinogenic toxicity is more representative of the harmful emissions of the electrodeposition process. However, due to the way toxicities are handled in LCA, with a large uncertainty on the link between emissions and impacts , the focThe normalized environmental impacts of each process are shown in VI emissions to water during the production of the chromium source. CrVI emissions are also responsible of the majority of ecotoxicities’ impacts. As a consequence, and in agreement with the poor consideration of ecotoxicities in current LCA methodologies already mentioned earlier, the focus will be set on CrVI emissions rather than human carcinogenic toxicity in the followings.The most prominent impact category in this analysis is human carcinogenic toxicity, with an impact higher than most other impact categories by several orders of magnitude. The two other most impactful categories are freshwater and marine ecotoxicities. In the case of human carcinogenic toxicity, more than 99.9% of the impact is attributed to CrFor most other impact categories, electricity production is usually the most impactful process . Due to the high differences in power consumption between electrodeposition and magnetron sputtering, an analysis of the global warming potential of those two processes is presented in 3 salts. Different production paths then lead to different environmental impacts which have been retrieved from the Ecoinvent database and are given below. It should be noted that metallic chromium is considered to come from aluminothermy at 75% and from electrolysis at 25%. The same amount of chromium atoms is considered, leading to different weight depending on their form: 520 g of metallic Cr and 1 kg of CrO3.As mentioned in the VI, and consequently the human carcinogenic toxicity, are lower. However, As it can be observed in In conclusion, the production of metallic chromium used in magnetron sputtering, while more energy intensive, produces less hexavalent chromium emissions than chromium oxide production, used in electrodeposition. Using this insight, a more detailed analysis of the most relevant environmental impacts can be made for the two processes in the next sections.First, the normalized global warming potential (GWP) for both processes is given below , top. AsIII and CrVI. While the former has low mobility and is harmless to most organisms, the latter is highly toxic over long exposure [VI emissions are the main factor in toxicity assessment. Therefore, this section will focus exclusively on the hexavalent chromium emissions. The different emission pathways for chromium have different effects on the human population and the environment. CrVI emissions can be put in two compartments: airborne and waterborne emissions. Both hexavalent chromium pathways emissions will be listed.There are two stable forms of chromium in nature: Crexposure . Its redexposure . As mentVI in rivers and freshwater can potentially harm the wildlife, depending on CrVI concentration [VI in the oral tract [VI is harmful to many organisms, it tends to be reduced relatively quickly in nature to CrIII, whose toxicity is dramatically lower, or at least not yet proven isolated from hexavalent chromium [On one side, airborne chromium  affects the workers mainly during occupational exposure. Chronic exposition leads to an increase of respiratory cancers. On the other side, waterborne chromium  can affect organisms differently depending on the environment where it is emitted. For example, Crntration . Hexavalntration . It may ntration ,53, althal tract . While Cchromium ,56.VI production and serve to the present discussion. Indeed, a more complete inventory of chromium trioxide production has been made by Harscoet and Froelich [More detailed data than the Ecoinvent database are available for CrFroelich , specifiVI discharge in water in the order of 13 kg per ton of chromium trioxide produced. In the present work, that would amount to 2.96 g of emissions per functional unit produced. The rest of CrVI emissions comes from wastewater treatment. Most of that wastewater originates from the plating itself, for which the hexavalent chromium emission is assumed to be 54.5 mg  for magnetron sputtering, while water takes the second rank in the electrodeposition process.With a consumable cost of EUR 7.5 per functional unit for magnetron sputtering compared to electrodeposition’s EUR 4, the most economic option tends to be the latter, due to the electricity consumption. However, labor costs should also be considered as they are usually important costs in Western Europe, accounting for about 75% of operating costs for an electrodeposition process in Belgium . For botFor both processes the amount of work needed for those two operations will vary. In the case of electrodeposition, due to the higher number of preparation steps and to the rectification required, the coating process takes more work time. This duration is here assumed to be 30 min for the electrodeposition process, while it is estimated at 15 min for magnetron sputtering. Regarding the time spent on process monitoring, the worker needs to be present only during a fraction of the coating step. Due to the higher deposition rate of electrodeposition, this duration is thus lower for a given monitoring rate (fraction of coating operation time for which operator attention is required). However, the impact of this lower duration on the total labor cost varies with the assumption that is made on the monitoring rate. A sensitivity analysis of the labor cost in function of the monitoring rate is then included in When comparing Besides operating costs, equipment costs play an important role in the economic feasibility of a process. In the present study and for the considered functional unit, it has been assumed that working capital and plant infrastructure are not taken into account. Indeed, working capital does not affect the economic viability of a project as it does not depreciate, while plant infrastructure is neglected because its impact is supposed to be similar for both processes. The cost breakdown for electrodeposition is given in It should be noted that there is a fairly important variation of prices between equipment from different sellers . However, the fact that the current rectifier  is one of the biggest contributors, as well as its indicated price range, is confirmed by a first-hand source .A cost breakdown for magnetron sputtering is given in It should be noted that, while this study estimates the total cost at around EUR 85,000, installations of similar sizes and functions have been found in catalogues with higher prices , hydroxyl radicals, sulfites or even microbes or humic acid are all factors that increase the rate of reduction of hexavalent chromium ,65. ThatVI concentration. The main way to assess this increased risk is to find out the number of workers potentially affected with a form of respiratory cancer due to their occupation. To that end, 3 figures must be worked out: the number of workers in the chromium industry, the number of workers that statistically should be affected by cancer independently of their occupation and the increase of cases directly caused by their occupation . For the cohort of workers studied in Gibb’s work [Even if an effect is localized, it can, nevertheless, be extremely harmful. This leads into the second point of discussion which is the effect of the contaminant on humans or wildlife. As mentioned earlier in this paper, it mainly affects humans through airborne droplets, although contamination through drinking water or consumption of contaminated fish is possible. Several studies ,49,67) h,67 h49,6b’s work , the num3 would be directly responsible of around 1638 lung cancers (2% of workers) in Europe. One should note that the assumptions made above are over-generalizing and should not considered as highly accurate due to the variability between data origin, regulations, etc. However, this figure gives a rough estimate of the potential risks for workers and related impact on public health organizations. Additionally, as mentioned in an earlier section, a large amount of CrVI emissions come from the production of chromium trioxide rather than from the plating process itself, so that the impact would be even greater for workers in the countries in which that chemical is produced. Following Transposing this calculation to a higher level, an estimated 32,000 workers are directly involved with working with chromium trioxide in Europe . In ordeConcerning cost estimations, while it is undeniable that magnetron sputtering equipment has a higher equipment cost as well as a higher energy cost, the lower time of pre- and post-treatments compared to electrodeposition is an indication for potential lower labor costs for magnetron sputtering. Further analysis in plant operating conditions is required to draw that conclusion. In addition, magnetron sputtering being a less industrially developed technology as well as better suited to automation , it coulFinally, several technical advantages of each technology have been omitted to streamline the assessment process. For electrodeposition, for example, the ability to use the technology to remove former coatings and make repairs on work pieces has not been considered. That application actually makes up more than half of the work flow of plating industries. Then, regarding magnetron sputtering, the technology’s polyvalence is not taken into account. Indeed, besides pure metallic chromium coatings deposited by magnetron sputtering, other high-performance coatings can only be deposited by magnetron sputtering, possibly making this technology even more competitive when compared to electrodeposition. For example, carbides or nitrides deposited by magnetron sputtering have a hardness greater than chromium coatings , meaningThe present work has performed an environmental and cost assessment for the case of hard chromium coatings, leading to the following conclusions: while magnetron sputtering can greatly decrease the hexavalent chromium emissions over the whole process, its larger electricity consumption increases the emissions of greenhouse gases during the process due to the selected electricity mix (Belgian mix) that entails a large use of fossil fuels. Additionally, although the magnetron sputtering process itself is costlier in terms of consumables and equipment, it can be economically competitive despite its lower deposition rate thanks to lower labor costs. Moreover, as magnetron sputtering further evolves, its drawbacks could be mitigated: recent developments have shown that better deposition rates are possible with segmented targets or heated targets/substrates for example. Furthermore, if the energy mix tends to be decarbonized, most of the environmental drawbacks of magnetron sputtering will be greatly mitigated, making it even more appealing compared to electrodeposition. As a consequence, it clearly appears that magnetron sputtering suffers from operational drawbacks (mostly due to lower deposition rate) compared to the benchmark electrodeposition, but nevertheless appears as a promising technology from combined environmental and economic perspectives once the right developments have been made.This paper opens a certain number of perspectives. For example, the study of high-performance coatings and further magnetron sputtering improvements such as High-Power Impulse Magnetron Sputtering (HiPIMS) , could bFurthermore, the present study has confirmed that insightful information can be gained both from a study of environmental impacts as well as from a study of process economics and that a fair comparison of technologies should consider both aspects. Life cycle and techno-economical assessments are clearly not redundant as they highlight different but equally important aspects of competing technologies. Performing them in parallel makes the evaluation more relevant but also more efficient, as the same inventory can be used for both assessments and trade-offs between cost and environmental impacts may be identified in an easier way.


The aim of this study was to compare virtual reality simulation with other methods of teaching interventional radiology. We searched multiple databases-Cochrane Library; Medline (PubMed); Embase; Trip Medical; Education Resources Information Center; Cumulative Index to Nursing and Allied Health Literature; Scientific Electronic Library Online; and Latin-American and Caribbean Health Sciences Literature-for studies comparing virtual reality simulation and other methods of teaching interventional radiology. This systematic review was performed in accordance with the criteria established by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses and the Best Evidence Medical Education (BEME) Collaboration. Eligible studies were evaluated by using the quality indicators provided in the BEME Guide No. 11 and the Kirkpatrick model of training evaluation. After the eligibility and quality criteria had been applied, five randomized clinical trials were included in the review. The Kirkpatrick level of impact varied among the studies evaluated, three studies being classified as level 2B and two being classified as level 4B. Among the studies evaluated, there was a consensus that virtual reality aggregates concepts and is beneficial for the teaching of interventional radiology. Although the use of virtual reality has been shown to be effective for skill acquisition and learning in interventional radiology, there is still a lack of studies evaluating and standardizing the employment of this technology in relation to the numerous procedures that exist within the field of expertise. Being able to acquire learning means being aware of that process as a whole, which encompasses knowledge, skill, and attitude.Learning is defined as the smallest independent structural experience that contains an objective, an activity to carry out, and an assessment. To that end, new resources are being tested in order to improve surgical skills in the current setting, in which procedures are becoming increasing invasive and complex-6. Because instruction in catheter-based endovascular interventions has become continuous in hospitals, a structure involving mentors is needed for resident training.In recent decades, minimally invasive procedures have replaced many open surgical procedures, one of the key aims being to reduce surgical morbidity and mortality,8. Simulators based on this technology mimic realistic situations and relevant scenarios, which can then be explored by various professionals. In interventional radiology, VR has been increasingly used for improving procedural skills, being widely employed for the teaching and improvement of surgical techniques, such as angiography, angioplasty, vascular catheterization, catheter placement under fluoroscopic guidance, and stent placement, as well as for the teaching of basic procedures such as the Seldinger technique,7,10-12. Currently, VR involves numerous devices and technologies, which can be adapted to work with a variety of equipment used around the world. However, in a study published in 2019, Nesbitt et al. stated that a VR simulator (VRS) cannot be purchased for less than £100,000.Virtual reality (VR) is a technology that aims to immerse the user in a particular location, through the perceptual deprivation of the actual environment, using computerized equipment or previously captured video to create a setting resembling aspects of the real world. Therefore, seeking to enrich the medical teaching methodology, administrators have combined VR with traditional methods, thus broadly aggregating concepts to increase effectiveness as well as to improve surgical skills. Preliminary studies comparing the combined use of VR and traditional teaching have obtained promising results regarding the employment of VR as a teaching method,6,11.The handling of basic materials, such as guidewires, catheters, drains, thermal ablation equipment, and needles, typically presents a challenge at the beginning of the learning process in interventional radiologyThe aim of this study was to identify, systematically evaluate, and summarize the best available scientific evidence comparing VR with various other methods of teaching interventional radiology.https://www.bemecollaboration.org/). It was registered in advance via the Open Science Framework (https://osf.io/wn762). The study was deemed exempt from formal institutional review by our institutional review board because no human or animal subjects were involved.This systematic review was conducted in accordance with the guidelines established by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses and by the Best Evidence Medical Education (BEME) Collaboration ; Embase; Trip Medical; Education Resources Information Center; Cumulative Index to Nursing and Allied Health Literature; Scientific Electronic Library Online; and Latin-American and Caribbean Health Sciences Literature. As search terms, we used US National Library of Medicine Medical Subject Headings, as follows: interventional radiology; virtual reality; augmented reality; video games; computer simulation; education, medical; teaching; and simulation training. We imposed no restrictions regarding language, origin, date of publication, publication status, or population evaluated. Reference lists of the studies selected and the main reviews on the subject were also evaluated. Manual searches were also carried out in the reference lists. All searches were performed on July 29, 2020.We included studies that compared VR with other methods for teaching interventional radiology. The comparator methods included the use of a pulsatile human cadaver model (PHCM), the traditional cadaver model, didactic classes, and the porcine model..Eligibility was determined on the basis of the relevance of the articles or their abstracts and the relevance of the respective journals. The identification of eligible studies was carried out in two stages by two reviewers, working independently. Disagreements were resolved by consensus. In the first stage, after excluding duplicates, the reviewers evaluated titles and abstracts, thus pre-selecting potentially eligible studies. In the second stage, the full texts of those same studies were assessed in order to confirm eligibility. The selection process was performed with the Rayyan QCRI software. Eligible studies were evaluated by using the quality indicators provided in BEME Guide No. 11 and the Kirkpatrick model of training evaluation described in BEME Guide No. 8 by Steinert et al.. The tools are based on instruments that cover a wide range of methodological issues in studies evaluating teaching methodology.The Cochrane Collaboration tool was applied in order to assess the risk of bias for individual studies and across studiesThe systematic review yielded 5,189 articles, of which 50 were found to be duplicates. After the two independent evaluators had read the titles and abstracts of the remaining 5,139 articles, using the Rayyan online platform, 51 articles were chosen for full-text reading., those analyzing factors other than medical teaching, and randomized clinical trials unrelated to the field of interventional radiology. Thus, 46 studies were excluded, resulting in a final sample of five studies  was used in combination with a PHCM. The students were randomly and equally divided into two groups: those using the PHCM alone (control group); and those using the VRS and the PHCM. A pre-task questionnaire was applied, after which the students were familiarized with the operating instructions for the devices. The procedure chosen for evaluation was left renal artery catheterization, confirmed by fluoroscopy (angiogram). Videos of the procedures were then evaluated by two experienced vascular surgeons who were blinded to the methods used. Given that the mean number of attempts required to achieve technical success in a particular procedure ranges from two to seven, the authors established a limit of eight attempts per student, to avoid a learning bias. After that step had been completed, a crossover step with a single attempt was carried out in order to evaluate the applicability of using the different teaching methods. The authors concluded that the learning curve of the students who used the VRS reached a plateau after the second attempt, whereas the students in the control group had greater difficulty and consequently a longer learning curve. In the VRS group, the fluoroscopy times were shorter and smaller quantities of contrast medium were administered. The authors emphasized the importance of comparing different methods of medical teaching in endovascular procedures and recognized the need for additional studies to verify that such skills are transferable to real life.In a comparative study involving 24 medical students with no previous endovascular training, Nesbitt et al. evaluated the effectiveness of using a simulator as a teaching method among general surgery residents with no prior endovascular experience. All of the participants were provided with reading material on basic catheter-based intervention for lower-extremity occlusive disease and attended a lecture on the topic. They were then randomly divided into two groups of 10 participants each. Those in one group received additional training in the technique on a simulator, whereas those in the other group did not. The result was evaluated by experienced surgeons who were blinded to the training status of the residents at the time of the surgery and used an 18-point checklist of the steps required to complete the procedure, including diagnostic angiography, angioplasty, and stenting. After two consecutive mentored interventions, the authors found that the group trained with the simulator received better scores for almost all aspects of the procedure, the difference in relation to the control group being even more pronounced for the second procedure. Overall, there were no significant differences between the two groups for the following aspects: advancing the femoral wire; mounting and advancing the catheter over the wire; and acquiring the image after percutaneous transluminal angioplasty. However, the participants trained with the simulator scored significantly higher for aspects such as knowledge of the anatomy, correct catheter positioning, balloon inflation, balloon pressure, and definition of the area of stenosis.Chaer et al. compared it with training on a VRS . Given the space required, the ethical context, and the fact that healthy animals are used, the porcine training method does not have the high accuracy needed for the transfer of the skills learned to a sick patient. Twelve novice endovascular surgeons received two days of training for internal iliac artery revascularization with one or both of the methods, in different orders, prior to performing the procedure. Two experienced interventional radiologists, working independently, evaluated the videos of the procedures. The students who received VRS training had scores that were consistently and significantly higher than were those of the students who received porcine training. After the procedures, the participants completed a questionnaire designed to evaluate their opinions. Most agreed that both methods should be included in the academic context and that mentors are undoubtedly irreplaceable in surgical practice. It is also notable that when the students received the VRS training prior to the porcine training, the total score for both methods improved for the second procedure performed. However, when the porcine training was received first, the total score for the VRS method did not improve. The authors stated that the skills learned via VRS training may be transferable to the porcine model. The students who received VRS training had an advantage and acquired skills that were transferable, although few were able to finalize the iliac artery revascularization without a mentor, demonstrating that VRS training should be a complement to, rather than a substitute for, traditional teaching methods.Employing porcine training, a teaching method that is lacking in robustness but is widely used in medicine, Berry et al. evaluated methods of teaching the Seldinger technique in a randomized clinical trial conducted in three stages. With the aim of evaluating the learning of this technique by a VR-based method, 35 interventional radiology mentors carried out, on a VRS, the necessary steps to perform the procedure. In the second stage, the authors evaluated the performance of the experienced interventional radiologists in relation to the novices and concluded that those who already had technical skill scored higher in almost all aspects, especially regarding puncture time. In the final stage, designated “validation of the transfer of training”, the authors defined the applicability of this teaching method in real life. To that end, 14 interventional radiology residents were randomly divided into two groups: those receiving technical instructions with training on the VRS; and those receiving only verbal instructions (control group). The students who had previously trained on the VRS performed significantly better for all of the criteria than did those in the control group, indicating that the VR-based educational intervention had a direct impact.One of the puncture techniques most widely used in interventional radiology, known as the Seldinger technique, consists in making a single incision and inserting a coaxial system, which makes it possible to introduce and change the various materials, such as guidewires and catheters, that are used during interventional procedures. Johnson et al. evaluated the applicability of VR to the teaching of percutaneous nephrostomy. The participants first underwent baseline testing on a VRS , after which they were randomized to receive either no additional training (control arm) or two 30-min sessions of training on the VRS (intervention arm). The participants in the intervention arm showed significant improvement in 11 of the 14 parameters measured, whereas those in the control arm showed no significant improvement in any of the parameters measured. The values for variables such as fluoroscopy time, puncture attempts, and vascular injuries were lower in the intervention arm, demonstrating a real shortening of the learning curve, as well as suggesting that VR-based teaching could reduce morbidity and mortality among patients undergoing percutaneous nephrostomy.In a study involving 63 participants , Knudsen et al.. Since then, it has been shown to have numerous benefits and is constantly being transformed. However, it has some inherent limitations that should be mentioned. Headaches and dizziness, as well as the cost and ergonomic limitations, continue to be the main challenges to be met in the use of the technology.Created in 1965 by Ivan Sutherland and initially tested in the field of computer science, VR has grown to cover a variety of scenarios. It has been available for use in interventional radiology since 1996, when the first interactive simulator was developedAmong the studies evaluated in this systematic review, there was heterogeneity of the populations evaluated and of the procedures included for analysis. The aim, therefore, our objective was to provide up-to-date information regarding the use and effectiveness of VR as an educational method in interventional radiology.In all five of the studies analyzed, the authors concluded that VR-based teaching promotes the transfer of skills, the Kirkpatrick level of impact ranging from 2B to 4B. In three of the studies, the analysis referred mostly to the learning/performance of the students or residents as a direct result of the educational intervention (Kirkpatrick level 4B), whereas it referred to changes in the attitudes or perceptions of the participants in relation to the teaching and learning (evidence level 2B) in two. Among the analyses that were classified as Kirkpatrick level 2B, the control group was analyzed using other methods of teaching, such as porcine and cadaver training, which probably differ less significantly than do classes and seminars, because they involve dynamic learning.Three of the studies evaluated involved basic interventional radiology techniques-catheterization, puncture, and the Seldinger technique, respectively. Another study evaluated percutaneous nephrostomy, and yet another assessed a slightly more complex procedure known as iliac artery stenosis revascularization. It should be stressed that the questions evaluated varied among the samples, although the main focus in all of the studies was the effectiveness of the teaching method. In relation to the population evaluated, residents and novice physicians were more often involved than were experienced surgeons, underscoring the fact that the former take greater advantage of VR-based training, thus shortening the learning curve.. Interventional radiology, which is one aspect of surgery, encompasses minimally invasive procedures, translating into lower cost, less morbidity, and shorter hospital stays. The greatest difficulties found during the process of learning interventional radiology procedures involve spatial and cognitive proprioception, as well as motor difficulties in using the new equipment, which can have catastrophic consequences at the beginning of a career,22.Surgery is a medical specialty that changes constantly over the years. The traditional “see one, do one, teach one” method is quickly being supplanted by the concept of “learning before doing”, saving inexperienced doctors from committing potentially avoidable errors,23. According to the British National Health Service, approximately 10% of hospitalized patients suffer from potentially avoidable injuries, most of which are attributable to inadequate medical training. However, in a study involving experienced doctors, Willaert et al. evaluated the concept known as “warm-up” and found that when carried out with a simulator, the “warm-up” improves the surgical performance of the team, reduces medical errors, reduces surgical time/costs, and increases the self-confidence of the team members. This concept, although new in medical practice, is long-established in music and sports, with the aim of complementing the development of the professional.With the aim of improving the education, as well as improving and enriching medical knowledge, new VR-based methods are currently being introduced, as are methods employing augmented reality or even hybrid forms of the two. Studies suggest that approximately 10,000 course hours are needed in order to acquire these procedural skills at an experienced level. With the aim of achieving this level of training in resident physicians, the use of simulators has come to the forefront in the teaching of interventional radiology procedures. When VR-based methods are added to the teaching protocol, the operator is able to learn a technique safely and effectively, which promotes patient well-being. Reductions in procedure times and in the number of technical errors are the most desired effects of this new resource.The term “practice makes perfect” reflects a scenario in which interventional radiologists need to perform a certain number of procedures in order to then execute them accurately and thoroughly. Medical errors that are considered foreseeable are responsible for 21,000 deaths annually in the country. In a study conducted in 2006 at Northwestern University Feinberg School of Medicine, residents completed a simulation-based course in CVCs before starting an intensive care unit internship. The authors found that, after one year, the CVC-related bloodstream infection rate fell from 4.2 to 0.42 cases for 100 patients. In addition to reductions in morbidity and mortality, that resulted in significant cost savings. According to those same authors, the focus on minimizing foreseeable complications and shortening hospital stays has been the motivation for studies aimed at identifying effective methods of medical training.In the United States, more than five million central venous catheters (CVCs) are used every year, and complication rates are 5-8% higher when such catheters are manipulated by professionals with little or no experience concluded that although the cost of a VRS can be prohibitive, simulation-based courses could result in a five-year savings of more than US$ 390,000 in comparison with the use of traditional porcine training. The authors estimated that, in comparison with traditional teaching methods, such as classes, books, and the use of animal models, the annual savings achieved with simulation-based training would be US$ 62,410. When taking into account spending on treatment for CVC-related infections and on hospitalization, Cohen et al. estimated an annual savings of approximately US$ 820,000, together with annual reductions of 137 fewer hospital admissions and approximately 120 fewer days in intensive care units. In addition, fluoroscopy time and the quantity of contrast medium used are two factors commonly found to have decreased in most of the randomized clinical trials evaluated in this review. The associated cost savings likely increase the overall savings attributed to the use of simulation-based training.In an economic analysis of endovascular skills training, Berry et al.. However, it remains unknown which simulators are most effective for teaching, in which populations they should actually be implemented, and for what purposes they are indicated. Additional time will be needed in order to synthesize data that are more homogenous and objective, in order to perform a robust analysis. However, VR, as evaluated in the studies reviewed here, has a promising future, with numerous advantages in terms of skill acquisition and cost benefits, as well as reductions in patient morbidity and mortality. Incorporating this technology into medical school curricula will undoubtedly improve learning, as well as honing the technical skills of surgeons.Institutions such as the European Board of Vascular Surgery and the German Society for Vascular Surgery and Vascular Medicine have already implemented or expressed an interest in including simulation-based training as a compulsory part of medical educationThere is still a lack of studies involving homogenous target populations and specifically evaluating the application of the currently available simulators for teaching the various interventional radiology procedures. There have also been few studies evaluating VR-based training of experienced professionals.Our efficacy analysis of the studies selected in this systematic review revealed that the use of VR for training in interventional radiology resulted in the acquisition of transferable skills. Among the studies evaluated, there was a consensus that VR-based methods of teaching have many benefits. Although such methods can be used in conjunction with other methods of teaching interventional radiology, VR-based training should not be viewed as a substitute for traditional modes of training or for mentoring.