This study describes the development, implementation, and evaluation of an initial stakeholder engagement experience designed to facilitate knowledge co-production. The engagement experience is part of a collaborative research framework (CRF), which facilitates iterative interactions among diverse researchers and stakeholders around the topic of enhanced climate resilience. Here, we describe the: (1) need for and development of a CRF as it relates to stakeholder engagement and knowledge co-production; (2) implementation of the initial engagement experience, focused around individual semi-structured interviews, in the context of a snow-dependent, arid river basin where historical water over allocation, climate change, and diversified water uses challenge the basin’s resilience; and (3) formative evaluation of the engagement experience using an online survey to inform the development of more effective engagement practices. Results of the evaluation indicate that, after participating, most stakeholders understand and recognize the importance of research goals, demonstrate positive attitudes toward collaborative research and researchers, view their contribution of knowledge and expertise as critical to research, and perceive researchers as eager to use their expertise. Moreover, stakeholders emphasized various context-specific goals for knowledge co-production, such as finding innovative ways to adapt to increased competition for diminishing water supplies. To achieve these goals, stakeholders suggested researchers learn about their basin, including its water allocation history and agricultural practices. These results highlight the importance of centering stakeholder engagement experiences within a broader CRF and formatively evaluating such experiences to adapt them to achieve research goals. Despite increases in funding for scientific research on climate change impacts, mitigation, and adaptation, challenges remain for closing the gap between science research products and the public’s use of those products to achieve outcomes  grounded in Reed et al.’s  and helpThe SNOWPACS project aims to co-produce new knowledge to enhance climate resilience by supporting the adaptation of irrigated agricultural communities in the arid western USA to shifts in the timing and quantities of snowmelt-derived water supplies. Many western river basins are over-allocated, meaning more water has been granted to users through a water rights-based system than is available in most years, even when snowpack is 100% or more of its expected accumulation Libecap . ClimateIn this project, a subset of the SNOWPACS team (referred to as the CRF sub-team going forward) was tasked with designing and implementing a framework for collaboration among: (1) the multi-disciplinary members of the research team; and (2) stakeholders in selected “case study” river basins who represent diverse, competing water uses. The overarching goal of our CRF is to better understand the complex interrelationships  introduced the SNOWPACS project based on the one-page description of the research that had been sent to stakeholders and (2) asked interview questions about stakeholders’ roles in the basin, types of water management challenges and decisions they make, what kinds of information water managers in the basin need, what coping actions they were taking in response to changing supply and demand, and similar topics. We also invited stakeholders to ask any questions they might have about SNOWPACS research goals and asked if they would be willing to continue engaging with researchers to inform the development of climate–hydrologic–economic models and verify model results. This question was intended to help stakeholders understand that the interviews were only a first step in a longer-term engagement with researchers and to communicate the importance of their knowledge to the SNOWPACS research.This engagement experience served multiple purposes under the umbrella of the CRF. First, developing the interview questionnaire facilitated collaboration among the broader research team and helped connect scientists, beyond those working directly with stakeholders, to the engagement portion of the project (top-down portion). Second, the interview experience (bottom-up portion) provided foundational engagement between project researchers and basin stakeholders, informed stakeholders about the project goals and objectives, set the stage for on-going stakeholder participation, and allowed for baseline evaluation of stakeholder engagement experiences  completed the survey.The survey questions were developed to assess the stakeholder engagement experience based on recommended practices grounded in participation theory  identify the largest water related concern in their area; (2) describe how they would define success for this project; and (3) explain what they believed researchers should know about their local area to inform this research. The qualitative responses were analyzed using thematic content analysis , although the standard deviation is large, suggesting highly varied responses for this question across our sample. Stakeholders perceived SNOWPACS researchers as eager to incorporate their knowledge into project research , with 91% of respondents responding positively, while the remaining 9% of participants neither agreed nor disagreed. From these results, we glean that we selected a set of stakeholders with fairly high levels of engagement experience through our stakeholder analysis, and the respondents believe that researchers are interested in what they have to say. Additionally, most stakeholders think their participation will be critical to the success of SNOWPACS , and the average respondent also believed they have a lot of expertise to offer this research , with 39% agreeing with this statement and 43% somewhat agreeing. These results suggest that stakeholders see value in engaging in collaborative research, making this project more likely to lead to knowledge co-production.Respondents were most mixed on the question item, “I am not personally expecting to get anything out of this project,” , as this relatively large standard deviation reveals, with 26% of participants agreeing with this statement and 17% of participants disagreeing. That said, respondents generally agreed that their knowledge would be used by researchers . Respondents also believed it was important that they know the goals of SNOWPACS . These results suggest that while stakeholders may see their knowledge and input as valuable to scientific research projects that they understand, they may not necessarily benefit personally from this research, though perhaps their broader stakeholder community might.Finally, 62% of the participants reported only moderate satisfaction with the basin’s existing water allocation models and water supply projections , with 8% of participants reporting moderate dissatisfaction, 24% reporting being slightly satisfied, and 6% reporting being neither satisfied nor dissatisfied on this question. This tepid satisfaction with information available to guide water management decisions under changing snowpack and snowmelt timing suggests that stakeholders may be more open to investigating alternative management regimes and coping actions, which is a key goal of the SNOWPACS project.Laws governing water resources are outdated and are not based on science or the hydrology of the basin.” Stakeholders also reported that climate change and inconsistent weather patterns presented new challenges to their existing water management responsibilities. One stakeholder summarized these challenges as follows: “Changes in the amount and timing of precipitation are combining with increases in temperature and with continued population growth and residential development to exacerbate existing imbalances and tensions between the available supply of fresh water and the demands of urban, suburban, exurban and rural communities and the needs of the environment. Within the context of this increasingly sharp competition over limited available freshwater resources, poorer communities and the environment are in danger of losing out in ways that jeopardize their continued viability.” Similar responses to this question suggest broadly that stakeholders acknowledge the potential for growing competition over increasingly limited water resources to increase conflict, further validating the need for knowledge that can aid stakeholders in finding both common ground and new solutions.The first open-ended question asked stakeholders what the largest water-related challenges in their area were. Sixteen participants responded to this question. Most highlighted the increasing competition for available water supplies, which create imbalances and tensions within their local area. This includes the perception that existing water allocation institutions may be out of step with current water management challenges. As one stakeholder commented, “Allowing for additional forecasting that may help with water acquisition strategies for instream use ” and “Water users, administrators, and law makers are able to rely on the final product to improve how water is managed.” Stakeholders also described project success in terms of help in dealing with repercussions from climate change already being felt. Such comments included, for example, “Realistic and implementable strategies by water users to adapt to changing water supply” and “The development of a tool or model that operators can use to make seasonable [sic] decisions for their operations.” Some participants encouraged researchers to aim for project outcomes that address larger social welfare concerns including, for example, “Models need to include the effect that government agency regulations have on a watershed and not just climate, streamflow, and water management,” and “A resource [is needed] that builds consensus and understanding.” As one stakeholder commented, “Success would be represented by water modeling of reliable accuracy that factors in the needs of water-dependent natural resources and land-based rural communities, and that explores the ramifications of various alternative water resource management scenarios for meeting those needs.” Together, these responses suggest that stakeholders need both procedural and substantive solutions to climate resilience challenges in their river basin.Second, we asked stakeholders how they would define success for a collaborative research and water modeling project such as SNOWPACS. Seventeen participants responded, with most defining project success as improved management practices or new information that could inform such efforts. Comments included, for example, “First they need to understand and study Nevada water law, then finding a balance for all users of water.” Others suggested researchers learn about water cycles within their basin as well as other relevant basins. Comments included for example, [researchers need to learn] “History of our waters and how we got to a point in time and stopped improving,” and “I think understanding water rotations in a given ditch system is very important to how water losses are shared.” Stakeholders also encouraged researchers to learn about ways that local farmers were practicing sustainable agriculture, as well as the effect of agricultural diversions on other parts of the system: “I think researchers need to know about and consider how to meet the need for additional flows of waters to restore and maintain a reasonable state of ecological health and function in Walker Lake and the riparian and estuarine habitat areas of the lower reach of the Walker River.” Similarly, one stakeholder noted that [researchers need to] “understand the efficiency and extent of aquifer recharge from mountain snowpack infiltration and runoff on a basin-by-basin scale—essentially the perennial groundwater yield of a basin and how the perennial yield relates to the system yield as in a stream-dominated system like the Walker.” Ensuring researchers understand the constraints and opportunities for sustainable water management, whether physical or social, appears to be a common concern, regardless of water use sector.Finally, we asked participants what researchers should know about their area to better inform project research. Responses to this item were quite diverse. Some participants suggested researchers might need to learn about local and state water laws and regulations that govern water use in the basin. As one stakeholder summarized, “Research to date on climate science knowledge co-production suggests that success hinges in large part on the quality of collaborative interactions between researchers and stakeholders (Cundill et al. Our study approaches this empirical gap through a formative evaluation of an engagement experience aimed to support the adaptation of agricultural communities in the western USA to shifts in the timing and quantities of snowmelt-derived water supplies. To incorporate concerns of context, process design, power dynamics management, participation culture, and scalar fit (Reed et al. Our survey results indicate that our CRF has been successful in helping to identify and engage stakeholders who possessed prior engagement experience, which has been shown to improve the likelihood of learning and knowledge co-production in other contexts Koebele . The resStakeholders reported only moderate satisfaction with existing water allocation models and water supply projections for the basin, with two participants reporting moderate dissatisfaction, which emphasizes stakeholders’ perceived need for new knowledge to inform and improve water management decisions in the basin. At the same time, stakeholders expressed an urgent need for the SNOWPACS project to co-produce new knowledge relevant to issues specific to the Walker River Basin, which speaks to concerns about the scalar fit of the research. For instance, stakeholders described increasing competition from the environmental water use sector to acquire agricultural water rights, compounded by inconsistent weather patterns, as substantive issues they face, which also reaffirms the need to manage power dynamics in co-production processes. Further, stakeholders recommended that SNOWPACS researchers should learn about their basin in terms of historical water cycles and to identify and provide information to help manage water at the farm level. This outcome of engagement reiterates and verifies the importance of both context and spatial consideration in the design of engagement processes (Reed et al. The formative evaluation results from the stakeholder engagement experience reported here are only one part of the bottom-up portion of the CRF. At the time of writing this manuscript, stakeholders who offered to participate in SNOWPACS on an on-going basis were recruited as technical advisors who will collaborate with researchers to inform model specification, refine model performance, and verify model outputs. These activities will further inform best practices for stakeholder engagement toward knowledge co-production by providing additional insight into how to best structure interactions between researchers and stakeholders. They also help to integrate the top-down and bottom-up portions of the CRF, as researchers beyond the CRF sub-team will interact directly with stakeholders. The CRF sub-team is also replicating this initial engagement experience in a second river basin, which will allow for a comparative assessment of engagement experiences across settings with different water challenges, geographic scales, demographics, economics, and institutions.The top-down portion of the CRF additionally includes assessing SNOWPACS researchers’ attitudes toward collaborative research generally, and specifically toward the CRF designed and implemented for this project. Understanding researchers’ perceived challenges for collaborating across disciplines and with stakeholders will help to inform future research design and engagement practices.Project implementation coincided with the beginning of the SARS-COVID-19 pandemic, which introduced unexpected and unprecedented challenges for stakeholder engagement modalities planned for the project that originally had included multiple iterative in-person interactions. The authors’ University Office of Research Integrity imposed public health safety precautions that included several months of prohibited in-person field research. These unexpected restrictions required that we adapt our engagement, replacing planned in-person interviews with virtual video conference technology and telephone interactions.While the pandemic continues to influence engagement modality, slowing the overall pace and texture of this research project, we have discovered that Walker River Basin stakeholders show participatory resilience. That is, despite the impact of the pandemic on their respective professional and personal lives, most of the stakeholders we have engaged consistently show a willingness to take part in this project’s knowledge co-production processes. The formative evaluation results presented here suggest that this willingness may in part represent a real need for new co-produced knowledge for adapting to climate change impacts, as well as an effective process design with products that could inform future local water management decisions.Finally, the potential for response bias exists in the evaluation data reported here. Nine of the 33 stakeholders who did not complete the online survey may have been dissatisfied with their initial engagement experience. Subsequently, it is possible that the responses reported here may be positively skewed.and scope of work. Recognizing, seeking out, and embracing this level of engagement diversity can serve to advance collaborative research processes and impacts (Mach et al. Recent years of federally funded co-produced knowledge initiatives in the USA specific to climate change have prompted scientists and stakeholders alike to advocate for its widespread use and funding (Cundill et al. Defining what constitutes effective knowledge co-production and best stakeholder engagement practices continues to evolve, in part thanks to a growing international body of scholars pursuing these linkages (see Eaton et al. Federally sponsored, co-produced climate knowledge projects across the USA are reporting enhanced utility of that knowledge (e.g., Prokopy et al. Finally, transdisciplinary collaborative research and knowledge co-production may not persevere without protecting researchers and stakeholders from their respective organizational culture, which may reward more conventional and narrowly focused endeavors. Protection may include combinations of moderating performance criteria that explicitly nurture or at least support a culture of knowledge co-production and participation (Boon et al. Knowledge co-production to enhance and support climate adaptation of agricultural communities across the western USA can inform similar undertakings in arid, snow-dependent riverine environments around the globe—and vice versa. Many agricultural water users worldwide share similar circumstances in that they face more variable water supplies (Qin et al. Further, as collaborative research continues to bring together diverse perspectives and knowledges to address wicked problems of the twenty-first century (Chan and Xiang


Concerns arise when patients with pneumocephalus engage in air travel. How hypobaric cabin pressure affects intracranial air is largely unclear. A widespread concern is that the intracranial volume could relevantly expand during flight and lead to elevated intracranial pressure. The aim of this systematic review was to identify and summarise models and case reports with confirmed pre-flight pneumocephalus.The terms  AND (flying OR fly OR travel OR air transport OR aircraft) were used to search the database PubMed on 30 November 2021. This search returned 144 results. To be included, a paper needed to fulfil each of the following criteria: (i) peer-reviewed publication of case reports, surveys, simulations or laboratory experiments that focussed on air travel with pre-existing pneumocephalus; (ii) available in full text.Thirteen studies met the inclusion criteria after title or abstract screening. We additionally identified five more articles when reviewing the references. A notion that repeatedly surfaced is that any air contained within the neurocranium increases in volume at higher altitude, much like any extracranial gas, potentially resulting in tension pneumocephalus or increased intracranial pressure.Relatively conservative thresholds for patients flying with pneumocephalus are suggested based on models where the intracranial air equilibrates with cabin pressure, although intracranial air in a confined space would be surrounded by the intracranial pressure. There is a discrepancy between the models and case presentations in that we found no reports of permanent or transient decompensation secondary to a pre-existing pneumocephalus during air travel. Nevertheless, the quality of examination varies and clinicians might tend to refrain from reporting adverse events. We identified a persistent extracranial to intracranial fistulous process in multiple cases with newly diagnosed pneumocephalus after flight. Finally, we summarised management principles to avoid complications from pneumocephalus during air travel and argue that a patient-specific understanding of the pathophysiology and time course of the pneumocephalus are potentially more important than its volume. Pneumocephalus refers to the presence of gas within the neurocranium. Aetiologies of pneumocephalus are numerous and include neurosurgical procedures, trauma, congenital or erosive skull defects, and infections with gas-producing organisms.Great concerns arise when patients with pneumocephalus must or wish to engage in air travel. These concerns primarily arise from the potential effect changing cabin pressure could have on the intracranial air volume and ultimately on intracranial pressure. Although commercial airplanes usually cruise at altitudes of 7000–13,000 m above sea level, the passenger cabin is pressurised to an altitude of 1500–2500 m above sea level.Boyle’s law states that, given a constant mass and temperature of an ideal gas, pressure is inversely proportional to the volume  peer-reviewed publication of case reports, surveys, simulations or laboratory experiments that focussed on air travel with pre-existing pneumocephalus; (ii) available in full text. A search of the reference lists of the included articles was conducted to assess for further studies that may fulfil the inclusion criteria.The initial search returned a total of 144 results. After title or abstract review, thirteen articles fulfilling the inclusion criteria were viewed in full text. The reference lists of these articles were searched for further studies that may fulfil the inclusion criteria, yielding five additional studies to be included. Thus, 18 publications met all inclusion criteria and were finally included.Donovan et al.  looked rA patient with an osteoma invading the frontal sinus and secondary periorbital emphysema, pneumocephalus and cerebrospinal fluid leak was safely air-transported by a commercial airline to a speciality care facility . InteresA patient with severe obtundation from tension pneumocephalus due to a fracture of the cribriform plate after endoscopic sinus surgery for chronic sinusitis on the preceding day was safely flown by fixed wing aircraft for urgent neurosurgical decompression and skull base repair at an urban centre . During Of note, a patient with a small anterior table defect as well as a minimally displaced posterior table fracture of the frontal sinus underwent shrapnel removal, primary closure and nasal packing at a combat support hospital before evacuation by air . DespiteThe hydrodynamics of a pneumocephalus patient during flight was investigated in an experimental setup using an acrylic box (skull), air-filled balloon , water-filled balloon  and agarose gel (brain) all placed in a custom-made pressure chamber . An intrTwo simulation studies used the same model for the nonlinear transformation of air volume expansion into intracranial pressure , 5. The The presence of pneumocephalus causes great uncertainty and safety concerns when patients with intracranial air inclusions must or wish to engage in air travel. These concerns mostly stem from the potential effect changing cabin pressure could have on intracranial air and consecutively on intracranial pressure. A systematic review was conducted to gain and summarise further information from case reports and series, surveys, laboratory experiments and simulations concerning patients with pneumocephalus engaging in air travel.A notion that repeatedly surfaced in this body of literature is that any air or gas contained within the neurocranium expands at higher altitude, comparable to extracranial gas, potentially resulting in increased intracranial pressure or even tension pneumocephalus.indirectly affect intracranial pressure, and intracranial air is only then affected by a changing intracranial pressure. Moreover, there are compensatory mechanisms  and the expansion of the intracranial air would be limited by the tension of surrounding tissue and fluids as well as a possibly increasing intracranial pressure (given the expansion of the air inclusion within a closed-compartment).The model in both simulation studies , 5 assumIn the only laboratory experiment conducted so far to investigate pneumocephalus during flight, four grooves were cut into the lid of the acrylic box (skull model) in order to allow the intracranial air to react with the changes in air pressure inside the pressure chamber .Based on their underlying assumptions, the models so far reported in the literature suggest overly conservative thresholds for patients flying with pneumocephalus.The concern that intracranial air relevantly expands with increasing altitude is widespread . NeverthThe case series by Donovan et al.  is reassWhile all case presentations fulfilling the inclusion criteria of this review showed no decompensation of the pneumocephalus, we would like to advise caution to clinicians regarding patients with a potential extracranial to intracranial fistula. Complications with intracranial air inclusions after flight have been reported in patients without a pre-flight diagnosis of pneumocephalus. A patient with a likely persistent intracranial to extracranial fistulous tract after parietotemporal craniectomy, subsequent debridement due to infection and still persistent complex dehisced scalp wound was found unarousable and with dramatic pneumocephalus after air travel . One patAnother impending dramatic in-flight evolution of pneumocephalus with an extracranial to intracranial fistulous tract was presumably averted by maintaining the atmospheric pressure of the departing airport and by the patient receiving 10 l/min oxygen via a non-pressurised, non-rebreather mask in a flat supine position .One patient suffered multiple facial fractures and a dural tear on his way to the airport and without neurologic deterioration in-flight eventually presented to the emergency department with an extensive pneumocephalus . The CT A patient with a minimally displaced posterior table fracture of the frontal sinus showed no intracranial air inclusion after evacuation by air . SubsequComplications with intracranial air inclusions after travel by air have been solely reported in patients with persistent extracranial to intracranial fistulous processes, regardless of a pre-flight diagnosis of pneumocephalus. Potential mechanisms for the generation of pneumocephalus with extracranial to intracranial fistulous processes include valve mechanisms, Valsalva manoeuvres or the presence of a cerebrospinal fluid leak , which idirect effect on the intracranial air.Specific precautions to be observed to avoid complications from pneumocephalus during flight can be suggested but ultimately lack conclusive data . A low-lBesides hypobaric conditions, other constellations should be considered when flying with pneumocephalus. For instance, stresses caused by accelerations (including changing g-forces), noises or hypoxaemia because of decreased partial pressure of oxygen at higher altitudes are relevant factors . HypoxaeThis review is limited by the exclusion of non-English articles. No formal assessment of publication bias was conducted. The overall sparse literature on this topic, the paucity of case reports and the heterogeneous ways of reporting limit generalising the results of this review, drawing final conclusions and making concrete recommendations. Moreover, clinicians might tend to refrain from reporting adverse events giving rise to a possible bias and the reports do not present enough basic data enabling a good scientific evaluation.Given the current lack of clear clinical guidelines, an informed decision as to which patients with pneumocephalus can travel safely is still not possible. Rigorous reporting of future case reports and in vivo recordings of intracranial pressure during flight will help addressing this important clinical question and spark discussions to prevent repetition of errors leading to complications. In addition, more accurate simulation studies and laboratory experiments are required to better understand what occurs to intracranial air accumulations in closed intracranial compartments. Correctly identifying patients with pneumocephalus that are at increased risk during flight could help prevent the occurrence of increased mass effect by postponing the flight or establishing management principles in case air travel cannot be delayed.Concerns arise when patients with intracranial air inclusions engage in air travel, mainly due to an uncertainty of how hypobaric cabin pressure affects intracranial air. The notion that intracranial air expands much like extracranial gas seems overrated and fuelled by results from models and experiments with assumptions and conditions that can be discussed. Along this line, we found no case reports of decompensation secondary to a pre-existing pneumocephalus during air travel. Nevertheless, the quality of examination varies and clinicians might tend to refrain from reporting such adverse events. A red flag in the form of a persistent extracranial to intracranial fistulous process was identified to be the cause for complications with intracranial air inclusions in multiple cases without a pre-flight diagnosis of pneumocephalus. A patient-specific understanding of the pathophysiology behind the pneumocephalus and its time course generally seems to be more important than the mere volume when assessing fitness to fly.


Finally, open questions and further research steps are outlined.Recent clinical studies have shown that agonists at muscarinic acetylcholine receptors effectively reduce schizophrenia symptoms. It is thus conceivable that, for the first time, a second substance class of procholinergic antipsychotics could become established alongside the usual antidopaminergic antipsychotics. In addition, various basic science studies suggest that there may be a subgroup of schizophrenia in which hypofunction of muscarinic acetylcholine receptors is of etiological importance. This could represent a major opportunity for individualized treatment of schizophrenia if markers can be identified that predict response to procholinergic vs. antidopaminergic interventions. In this perspective, non-response to antidopaminergic antipsychotics, specific symptom patterns like visual hallucinations and strong disorganization, the presence of antimuscarinic antibodies, ERP markers such as mismatch negativity, and radiotracers are presented as possible  For many decades, the dopamine theory has been the most influential neurobiological concept in understanding psychosis  and it iFirstly, a glutamate theory of schizophrenia has been established mainly based on the observation that antagonists at the glutamatergic NMDA receptor, such as ketamine, phencyclidine, or endogenously formed anti-NMDA antibodies, directly and reliably cause psychotic symptoms and genetic studies showing an increased risk of schizophrenia in individuals with gene variants affecting the glutamate system , 7. HoweSecondly, reduced neurotransmission at acetylcholine receptors has been proposed as a non-dopaminergic mechanism underlying schizophrenia symptoms. Acetylcholine receptors are generally divided into nicotinic and muscarinic receptors according to binding ligand, and both have been implicated in the etiology of schizophrenia. Based on the extremely high rate of nicotine use in patients with schizophrenia, which has been interpreted as self-medication of existing deficits, nicotinic acetylcholine receptors, particularly the α7-subtype, have initially become the focus of research interest , 9. An iA deficit at muscarinic acetylcholine receptors in at least a proportion of patients with schizophrenia is suggested, for example, by post-mortem studies showing reduced density of muscarinic receptors in patients, by imaging studies with radioligands showing reduced availability of muscarinic receptors, and by the ability of muscarinic antagonists such as scopolamine to elicit schizophrenia-like symptoms in healthy individuals , 15. Basin vivo markers of muscarinic deficit in schizophrenia that ideally in the future could be used for personalized therapy planning and help in deciding whether to initiate treatment primarily with a dopamine antagonist or a muscarinic acetylcholine agonist.These developments make it seem possible that in the near future, a second substance class of muscarinic antipsychotics may be established alongside current antidopaminergic antipsychotics. This would lend new urgent importance to the question of whether there is an “antimuscarinic subtype” in schizophrenia and, particularly, how to identify it in clinical settings since this subtype might show non-response to antidopaminergic medication and response to procholinergic treatment. This perspective is intended to present several potential The following sections describe markers that could indicate a muscarinic deficit in patients with schizophrenia. A summary is provided in About 20% of schizophrenia patients show no improvement at all when treated treated with common antipsychotics . CurrentA hypothetical subtype of schizophrenia with a predominant underlying muscarinic deficit might show specific symptom patterns compared with other subtypes. Information about these clinical correlates of a muscarinic deficit may be provided by the symptoms elicited by anticholinergic agents. First and foremost here are cognitive impairments, which are caused by anticholinergics  and are Mismatch negativity (MMN) refers to the electrophysiological response to a rule violation  in a sequence of (in most studies auditory) stimuli. A reduction of MMN in patients with schizophrenia compared to controls has now been shown in many studies and is considered one of the most promising biomarkers for predicting the risk of conversion to clinical psychosis in people at high risk and the response to specific therapies –35. In aThe exact pathomechanism underlying a potential cholinergic deficit in schizophrenia is unclear, with post-mortem studies suggesting reduced receptor density . Interesin vivo using radioactive tracers, such as 123I-IQNB for SPECT  antipsychotics. Possible candidates for such predictors include treatment resistance to antidopaminergic treatment, the presence of severe cognitive impairment and/or visual hallucinations, reduced mismatch negativity, the presence of antimuscarinic antibodies, and reduced availability of muscarinic receptors on radionucleotide imaging.Muscarinic acetylcholine receptors can be divided into 5 subtypes (M1–M5), with mainly agonists at M1 and M4 subtypes being considered candidates in schizophrenia treatment, whereas stimulation of M2 and M3 subtypes is related to gastrointestinal and other peripheral side effects. Animal research furthermore suggests that enforcing stimulation at M4 receptors through positive allosteric modulators (PAM) particularly improves proxy markers of positive schizophrenia symptoms such as hyperlocomotion or impaired prepulse inhibition induced by dopamine agonists or NMDA antagonists , 57. In A crucial question in attempts to find biomarkers for response to muscarinic agonists concerns the precise cause-effect relationship between muscarinic deficit and psychotic symptoms, specifically the question, if muscarinic deficiency causes psychosis through dopaminergic or dopamine-independent pathways. Obviously, the hope to identify distinct predictors for the response to antidopaminergic vs. procholinergic treatment rests on the assumption that there are dissociable etiologies behind psychotic syndromes . A variety of complex interactions between muscarinergic and dopaminergic neurotransmission have been established , leadingThe most rigorous way of testing the capacity of the suggested biomarkers to identify those schizophrenia patients who will likely benefit more from a procholinergic as compared to an antidopaminergic treatment would be to test their specific ability to predict individual outcome of a clinical trial of, for example xanomeline-trospium compared with amisulpride treatment. A specific predictor, as compared to a general prognostic factor, is a pretreatment variable , whose influence on the treatment outcome  differs between treatments . For insThe original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.The author confirms being the sole contributor of this work and has approved it for publication.


INTRPRT guideline, a design directive for transparent ML systems in medical image analysis. The INTRPRT guideline suggests human-centered design principles, recommending formative user research as the first step to understand user needs and domain requirements. Following these guidelines increases the likelihood that the algorithms afford transparency and enable stakeholders to capitalize on the benefits of transparent ML.Transparency in Machine Learning (ML), often also referred to as interpretability or explainability, attempts to reveal the working mechanisms of complex models. From a human-centered design perspective, transparency is not a property of the ML model but an affordance, i.e., a relationship between algorithm and users. Thus, prototyping and user evaluations are critical to attaining solutions that afford transparency. Following human-centered design principles in highly specialized and high stakes domains, such as medical image analysis, is challenging due to the limited access to end users and the knowledge imbalance between those users and ML designers. To investigate the state of transparent ML in medical image analysis, we conducted a systematic review of the literature from 2012 to 2021 in PubMed, EMBASE, and Compendex databases. We identified 2508 records and 68 articles met the inclusion criteria. Current techniques in transparent ML are dominated by computational feasibility and barely consider end users, e.g. clinical stakeholders. Despite the different roles and knowledge of ML developers and end users, no study reported formative user research to inform the design and development of transparent ML models. Only a few studies validated transparency claims through empirical user evaluations. These shortcomings put contemporary research on transparent ML at risk of being incomprehensible to users, and thus, clinically irrelevant. To alleviate these shortcomings in forthcoming research, we introduce the  However, translating these ML models from the bench to the bedside to support clinical stakeholders during routine care brings substantial challenges, among other reasons, because of the high stakes involved in most decisions that impact human lives. When stakeholders interact with ML tools to reach decisions, they may be persuaded to follow ML’s recommendations that may be incorrect or promote unintended biases against vulnerable populations, all of which can have dreadful consequences2. These circumstances motivate the need for trustworthy ML systems in healthcare and have sparked efforts to specify the different requirements that ML algorithms should fulfill. Most of these recent efforts focus on achieving a certain on-task performance requirement but neglect that for assisted decision making not ML system performance alone, but human-ML team performance is the most pertinent to patient outcome. How to achieve adequate human-machine teaming performance, however, is debated. While some argue that rigorous algorithmic validation, e.g., similar to the evaluation of drugs, tests, or devices, demonstrates safe and reliable operation and may thus be sufficient for successful human-machine teaming4, others reason that transparency in an ML model, e.g., by revealing its working mechanisms and presenting a proper interface, is necessary to invoke user trust and achieve the desired human-machine teaming performance7. The growing interest and convergence of recent works on the importance and need of transparency have stressed that not addressing the opacity of ML techniques might hinder their adoption of in healthcare, limiting the potential positive impacts12. The inability to make the decision making process transparent might affect the misuse and disuse of ML models in the clinical domain, as the utility of the model might be limited if it does not reveal the reasoning process, limitations, and biases9. We believe that this dichotomy is artificial in that, first, rigorous validation and transparency are not mutually exclusive, and second, both approaches augment an ML model with additional information in hopes to justify  the recommendation’s validity which is hypothesized to achieve certain human-factors engineering goals such as understandability, reliability, trust and etc. However, as we will highlight in detail through a systematic review, current approaches that aim at advancing human factors goals of ML systems rely on developers’ intuition rather than considering whether these mechanisms affect users’ experience with the system and their ability to act on ML model’s outputs.There have been considerable research thrusts to develop Machine Learning (ML) models in the healthcare domain that assist clinical stakeholdersaffordance, a concept that is commonly employed when designing effective Human-Computer Interactions (HCIs)13, and we argue that transparency in ML algorithms should be viewed as such. There are several consequences from this definition:Developing transparent ML algorithms is not purely computational.Specific design choices on the mechanisms to achieve explanations or interpretations may be suitable for one user group, but not for another.Creating transparent ML systems without prior groundwork to establish that it indeed affords transparency may result in misspent effort.Designing ML algorithms that are transparent is fundamentally different from merely designing ML algorithms. The desire for transparency adds a layer of complexity that is not necessarily computational. Rather, it involves human factors, namely the users to whom the ML algorithm should be transparent. As a consequence, transparency of an algorithm is not a property of the algorithm but a relationship between the transparent ML algorithm and the user processing the information. Such relationship can be understood as an Given the user- and context-dependent nature of transparency, it is essential to understand the target audience and to validate design choices through iterative empirical user studies to ensure that design choices of transparent models are grounded in a deep understanding of the target users and their context. In addition, to maintain a user-centered approach to design from the early stages, rapid prototyping with users provides feedback on the current, low- to high-fidelity embodiment of the system that is going to be built eventually. Involving users early by exposing them to low-fidelity prototypes that mimic final system behavior allows designers to explore multiple alternatives before committing to one pre-determined approach that may not be understandable nor of interest to end users.However, following a human-centered design approach to build transparent ML systems for highly specialized and high stakes domains, such as healthcare, is challenging. The barriers are diverse and include: (1) the high knowledge mismatch between ML developers and the varied stakeholders in medicine, including providers, administrators, or patients; (2) availability restrictions or ethical concerns that limit accessibility of potential target users for iterated empirical tests in simulated setups for formative research or validation; (3) challenges inherent to clinical problems, including the complex nature of medical data  and decision making tasks from multiple data sources; and last but not least, (4) the lack of ML designers’ training in design thinking and human factors engineering.Starting from the considerations around designing and validating transparent ML for healthcare presented above, we investigate the current state of transparent ML in medical image analysis, a trailblazing application area for ML in healthcare due to the abundance and structure of data. Through a systematic review based on these aspects, we first identify major shortcomings in the design and validation processes of developing transparent ML models. These deficiencies include the absence of formative user research, the lack of empirical user studies, and in general, the omission of considering ML transparency as contingent on the targeted users and contexts. Together, these shortcomings of contemporary practices in transparent ML development put the resulting solutions at substantial risk of being unintelligible to the target users, and consequently, irrelevant.INTRPRT guideline to help model designers for developing transparent ML for medical image analysis step by step. Figure This paper aims to encourage model designers to actively consider and work closely with the end users during the design, construction, and validation of ML models for medical imaging problems. Acknowledging the barriers to widespread adoption of human-centered design techniques to develop transparent ML in healthcare and grounded in our systematic review of the literature, we further propose the INTRPRT; the themes are incorporation (IN), interpretability (IN), target (T), reporting (R), prior (PR), and task (T). Incorporation refers to the communication and cooperation between designers and end users before and during the construction of the transparent model. Formative user research is one possible strategy that can help designers to understand end users’ needs and background knowledge15, but other approaches exist16. Interpretability considers the technicalities of algorithmic realization of a transparent ML system. Figure Target determines the end users of the transparent ML algorithms. Reporting summarizes all aspects pertaining to the validation of transparent algorithms. This includes task performance evaluation as well as the assessment of technical correctness and human factors of the proposed transparency technique . Prior refers to previously published, otherwise public, or empirically established sources of information about target users and their context. This prior evidence can be used to conceptualize and justify design choices around achieving transparency. Finally, task specifies the considered medical image analysis task, such as prediction, segmentation, or super resolution, and thus determines the clinical requirements on performance. We emphasize that these themes should not be considered in isolation because they interact with and are relevant to each other. For example, technical feasibility of innovative transparency mechanisms based on the desired task may influence both, the priors that will be considered during development as well as the incorporation of target users to identify and validate alternatives.Compared to developing generic ML algorithms, designing and validating transparent ML algorithms in medical imaging tasks requires consideration of human factors and clinical context. We group these additional considerations into six themes according to the initial review, iteratively defined prior to data extraction and abbreviated to incorporation theme, cross-disciplinary study teams may constitute a first step towards incorporating target users during ML design, however, only 33 of the included articles were authored by multidisciplinary clinician-engineering teams. More importantly, no paper introduced formative user research to understand user needs and contextual considerations before model construction, which is reflected in the lack of justifying the prior theme. Around half of the selected articles (n = 28) chose clinical priors and guidelines as an inspiration for transparent systems. In the target theme, we found that only 30 of the included articles specified end users, and all of these papers were aimed at clinical care providers, a stark imbalance considering the variety of stakeholders. In the task theme, prediction tasks were by far the most common application for transparent ML algorithm design (57/68). In the Interpretability theme, methods relying on clinical guidelines resulted in algorithms that adopted multiple sub-steps of a clinical guideline to build the model and generate outcomes, while methods that were based on computer vision techniques for transparency most commonly relied on post-hoc explanations. In the Reporting theme, the methods used for assessing transparency varied with the problem formulation and transparency design, and included human perception, qualitative visualizations, quantitative metrics, and empirical user studies; we note that an evaluation with end users was highly uncommon (only 3 of the 68 included studies). However, no paper considered the six themes comprehensively. More importantly, there is no evidence that any papers considered the dependency and interaction between different themes. The reviewed literature further supports that one guideline considering all themes and the interaction between them is highly desired in the medical image analysis community to construct transparent ML models following human-centered design practices.Having identified and refined the themes iteratively after an initial review, we structured the systematic review according to the six themes. We identify and summarize dominant trends among the 68 included studies aiming to design transparent ML for medical image analysis. In the INTRPRT guideline. The INTRPRT guideline provides suggestions for designing and validating transparent ML systems in healthcare in hopes to increase the likelihood that the resulting algorithms indeed afford transparency for the designated end users. The guidelines also address the challenges of following a human-centered design approach in the healthcare domain, propose potential solutions, and apply to different kinds of transparency ML algorithms. To further illustrate the INTRPRT guideline, we introduce a case study .The first step to designing any ML algorithm for healthcare is to well define the clinical scenario, the constraints the solution will have to abide by, and all hard or soft requirements the algorithm needs to meet for the clinical cf. Fig. . For ML There exists a wide gap in domain expertise and contextualization between target users and ML model designers in most use cases in healthcare. Furthermore, there are multiple “transparency” techniques and choices, such as the transparent working mechanism or user-centered interactions in practice. Simply selecting a “transparency” technique, without incorporating and consulting target users puts the resulting ML models at risk of not achieving the desired transparency. The human-centered design approach addresses this challenge through iterative empirical studies that over time guide the development and refinement of the technical approach such that, upon completion, the design choices are well justified by empirical target user feedback. This approach may not always be feasible in healthcare due to accessibility and availability barriers of target users. To address this limitation while still enabling technological progress in transparent ML, we introduce four distinct levels of evidence. These levels allow designers to classify the level of confidence one may have that the specific design choices will indeed result in a model that affords transparency.Level 0: No evidence. No dedicated investigations about the end users are performed to develop transparent ML systems.Level 1: One-way evidence. Formative user research techniques, such as surveys and diary studies, are only performed once without further feedback from end users about the findings extracted from the research phase, resulting in one-way evidence. Such user research suffers risks of potential bias in concluding about justification of transparency because there is no opportunity for dialog, i.e., designers may ask irrelevant questions or target users may provide non-insightful, potentially biased responses.Level 2: Public evidence. Public evidence refers to information about target user knowledge, preference, or behavior that is public domain and vetted in a sensible way. Public evidence includes clinical best practice guidelines, Delphi consensus reports, peer-reviewed empirical studies of closely related approaches in large cohorts, or well documented socio-behavioral phenomena.Level 3: Iteratively developed evidence. Iteratively developed evidence is transparency evidence that is iteratively refined through user feedback where designers and end users communicate with each other throughout method development. The purpose of iteratively validating and refining the current transparency mechanism is to identify any potential bias in the assumptions that motivate the transparency technique while ensuring that it is understandable to end users.The levels of evidence are based on increasingly thorough approaches to understand the chosen end users in context of the envisioned task:Being actively cognizant of the level of evidence that supports the development enables trading off development efforts between ML method development vs. gathering richer evidence in support of the intended developments.This guideline is designed to ensure that the transparency technique used in the ML model is indeed consistent with the assumptions made during its justification. While complying with this guideline is trivial if the model is developed in a human-centered design approach (Level 3: Iteratively developed evidence), in all other cases designers should be explicit about the intellectual proximity of the developed technical approach to the motivating evidence. To this end, after specifying which components of the ML model require transparency for users to capitalize on the intended benefits, it is desirable for the method to be as simple as possible so that it can be easily derived from and linked to the justification of transparency. Once confirmed that the envisioned model is indeed consistent with the justification, computational development of the model, including training, refinement, and validation, begins.18, channel 19, and interactivity (e.g. whether users can provide feedback or refine model outputs) can drastically affect users’ experience and performance22, and therefore, must be aligned with the goal of transparent system development. Clearly, the selection of presentation mode should be incorporated early and supported by some degree of evidence, that emerges naturally when following human-centered design principles but requires justification if not (as posited for transparency in Guideline 2). Ultimately, users’ experience with the systems plays an important role in their willingness to adopt it in a real setup23.In addition to content , seemingly peripheral factors on the presentation of information may play a disproportionate role in the perception of transparency. It is well known that factors like format 24.Similar to ordinary ML models, the transparent system must be evaluated quantitatively using appropriate metrics that well reflect the desired performance. In addition, the data used to evaluate the algorithm and its relevance regarding the clinical target task must be specified. Metrics and evaluation protocols should be selected to well determine the model’s abilities in regard to the clinical requirements specified per Guideline 1. Reporting task performance of the algorithm in standalone deployment is important as a baseline for empirical studies in which users may interact and collaborate with the system to complete a task, and team performance (human + ML system) metrics can be measured. Such comparisons are relevant to the goal of improving team performance when integrating intelligent systems to assist humans in complex tasks18, reliance, satisfaction22, mental model formation25, and system acceptance. Reporting of these user studies should include details of the experimental design, participant sample, and techniques to analyze the results.In addition to task performance, transparent ML systems must be evaluated with respect to their transparency claims. This validation is generally necessary even for Level 3 (iteratively developed evidence) justified transparency mechanisms, because user studies in the design phase commonly rely on mock-up prototypes of the envisioned system, and therefore, may have different modes of failure than the final ML system. Within this guideline, we distinguish two types of evaluation: (1) Validating the correctness of the transparency technique, which objectively assesses whether the information supplied to achieve transparency is in agreement with the justification. Achieving correctness is particularly important for systems that rely on post-hoc explanations, since explanations may rely on a second model that is distinct from the ML algorithm generating recommendations. (2) Validating the effectiveness of transparency in a human-centered approach, to demonstrate that the transparent ML system applied to relevant data samples and in cooperation with target users achieves the desired goals. This empirical evaluation determines the efficacy of human factors engineering. The dimensions that are often considered include users’ trustTransparency of ML algorithms for medical image analysis is commonly motivated by the desire of automating complex tasks while retaining a clear interface for human interaction, e.g., to improve trust, avoid over-reliance, or increase acceptance. However, achieving these design goals through transparency requires the development of transparent ML algorithms that are intelligible by the envisioned end users. In design thinking, aligning technological developments with user needs is accomplished through user involvement in the design process and iterative user testing, which is largely infeasible in healthcare due to varied barriers to end user involvement. We propose a design and evaluation framework where ML designers actively consider end users’ needs, knowledge, and requirements, allowing designers to classify the reliability of their understanding of end user needs using four levels of evidence. Explicitly thinking about the confidence one may have in the assumptions about end users that drive transparent ML system development may mitigate the risks of developing solutions that are unintelligible to the target users, and therefore, neither achieve the desired human factors engineering goals nor benefit clinical practice. Similarly, quantifying the level of evidence currently available to motivate transparency claims then allows developers to trade-off resources between technical ML developments and additional formative research of their target users to ensure that the resulting systems are fit to meet the requirements of the clinical task but foremost the target users.INTRPRT guideline in the context of the key observations from different themes of our systematic review to identify opportunities to improve the design of transparent ML systems for medical image analysis. Each discussion point focuses on one or more themes that we introduced before. Furthermore, the second last subsection presents successful examples of ML systems designed with clinical end users and the last subsection also includes the comparison of our guideline and systematic review with existing literature.We discuss the incorporation and prior) and empirical user testing (theme reporting) are critical to ensure that solutions meet user needs (theme target). On the one hand, formative user research helps designers navigate and understand end users’ domain practice and needs. On the other hand, empirical user testing assesses whether the designed algorithm indeed achieves the human factors engineering goals, such as affording transparency, promoting trust, or avoiding over-reliance. Additionally, early user involvement in the design process using prototypes of increasing fidelity provides opportunities to review and iterate over design choices. From our systematic review of the literature presented in the “Detailed Analysis of Findings during Systematic Review” section, we find that although most contemporary studies on transparent ML formulate human factor engineering goals, no study reported formative user research or empirical testing to inform and validate design choices. We must conclude that contemporary research efforts in medical image analysis have disproportionately prioritized the technological development of algorithmic solutions that alter or augment the predictions of complex ML systems with the implicit—though unfortunately often explicit—assumption that those changes would achieve transparency. However, because of the substantial knowledge imbalance between ML engineers and target users among other reasons detailed above, it is unlikely that, without formative user research or empirical tests, those systems truly afford transparency or achieve the promised human factors engineering goals. While demonstrating the computational feasibility of advanced transparency techniques is certainly of interest, grounding the need for these techniques in solid understanding of the target users should be the first step for most, if not all, such developments.Both formative user research (theme incorporation and prior), upon completion of ML development, system transparency needs to be empirically validated (theme reporting). During the literature review, we observed that hardly any study reported quantitative empirical user evaluations as part of final method validation, and many of the included articles limited analysis of transparency goals to qualitative analysis by presenting a limited number of illustrative examples, e.g., pixel-attribution visualizations. While such analysis may suggest fidelity of the transparency design to the cause of the prediction in those few select samples, its utility beyond is unclear. In cases where no empirical user evaluation is conducted, neither during conceptualization nor during development, claims around system transparency or human factors are at high risk of being optimistic and should be avoided.In addition to user involvement or formative research during the design phase  by those target users. One motivation for investigating transparency in such tasks is the existence of clear and systematic clinical workflows and guidelines, e.g., the Breast Imaging Reporting and Data System (BI-RADS) system for mammography, the AO/OTA Tile grading of pelvic fractures, or other easily intelligible covariates associated with outcomes. The availability of such human-defined baselines that are already used for clinical decision making provides immediate Level 2 evidence of transparency for ML systems attempting their replication. In addition, it facilitates data collection and annotation, because intermediate outputs that may be required to build such system are known a priori. Conversely, justifying specific attempts at achieving transparency is much more complicated for tasks that do not readily have human-based baselines or clinical best practice guidelines. Some such tasks may already be performed in clinical practice, such as segmentation or super-resolution, the interpretation of which may be ambiguous and result in high variability among observers27. Other tasks may be beyond the current human understanding of the underlying mechanisms that enable ML-based prediction, e.g., ethnicity prediction from chest X-ray28 or various tasks in digital pathology30. In these scenarios, while it may be possible to derive some justification from the literature, e.g., how target users generally approach tasks of the kind, achieving even Level 2 justification is difficult if not impossible. Empirically validating the envisioned mechanisms for transparency with respect to their ability to afford transparency and achieve the human factors engineering goals is thus paramount when attempting to benefit such tasks.Clearly specifying and formulating the medical task that the ML system solves is fundamental to determine the assistance that it can provide to clinical practice. Along with the disproportionate consideration of clinicians as end users goes a disproportionate focus on clinical tasks that are routinely performed in current clinical practice  enabled analysis tool for Chest X-ray (CXR) images, a user-centered iterative design assessed the utility of potential explanatory information in the AI system15. Users’ needs during their introduction to an AI-based assistance system for digital pathology were identified through open-ended interviews and a qualitative laboratory study31. Iterative co-design processes were followed to identify clinicians’ perceptions of ML tools for real clinical workflows, e.g., antidepressant treatment decisions16 and phenotype diagnosis in the intensive care unit32. Determining the efficacy of envisioned ML systems or ML-enabled interaction paradigms in empirical user studies before committing resources to their fully-fledged implementation has become common practice in human-centered AI, e.g.,34, with many studies considering tasks that are related to medical image analysis35. Increasing the acceptance of empirical formative user research as an integral component of human-centered ML design for healthcare tasks, including medical image analysis, will be critical in ensuring that the assumptions on which human-centered systems are built hold in the real world.Early identification and direct communication with end users, as it is emphasized in the target). Generally applicable design guidelines were compiled and iteratively refined by HCI experts to design and evaluate human-AI interactions36. Although these guidelines are relevant and suitable for a wide range of common AI-enabled systems, more nuanced guidelines are desirable for domains where study participants cannot be recruited nor interviewed in abundance. Similarly, previous attempts to guide the design of effective transparency mechanisms acknowledge that real stakeholders involved should be considered and understood37. Starting from the identification of diverse design goals according to users’ needs and their level of expertise on AI technology, and a categorization of evaluation measures for Explainable Artificial Intelligence (XAI) systems38, addressed the multidisciplinary efforts needed to build such systems. A set of guidelines, summarized in a unified framework, suggests iterative design and evaluation loops to account for both algorithmic and human factors of XAI systems. However, similar to ref. 36, these guidelines are intended for generic applications, e.g., loan and insurance rate prediction39 and personalized advertisements40, and do not consider additional challenges, barriers, and limitations when developing algorithms for domains that exhibit users with very specific needs and in highly specific contexts, such as healthcare. Other considerations to build interpretable AI systems have been identified from a multidisciplinary perspective12. For instance, the approach presented in ref. 41 summarized four guidelines that included the application domain, technical implementation, and human-centered requirements in terms of the capabilities of human understanding. A requirements list formulated as a “fact sheet” was introduced in ref. 42 to characterize and assess explainable systems along five key dimensions: functional, operational, usability, safety and validation. While the five dimensions allow to systematically compare and contrast explainability approaches theoretically and practically, the properties that were included failed to consider where and how to formulate the justification of transparency. Formative user research and validation of the justification of transparency are especially essential in healthcare, where a huge knowledge imbalance exists between ML designers and end users of AI systems.Motivated by advances in AI technologies and the wide range of applications in which it can be used to assist humans, there are ongoing efforts to guide the design and evaluation of AI-infused systems that people can interact with , and focus on more recent ML techniques such as Convolution Neural Networks (CNNs). Likewise, transparent ML in cancer detection is also reviewed in ref. 50 and structured following the same aspects of generic transparent ML techniques, such as Local vs. Global and Ad-Hoc vs. Post-Hoc. distinctionsWith the trend that ML is more popular in clinical decision making tasks due to its performance, recent surveys and systematic reviews have aimed to summarize existing literature to create transparent ML in healthcare. However, these surveys failed to consider all the themes proposed in this paper and each aspect of transparent ML is reviewed in isolation. More importantly, current reviews mainly focus on the existing transparency techniques and evaluation, ignoring how and where justification of transparency emerges. For example, a survey categorized research works related to the interpretability of ML in general, and then applied the same categories to interpretability in the medical fieldThe guidelines and systematic review of the state of the field presented here aim at emphasizing the need for formative user research and empirical user studies to firmly establish the validity of assumptions on which human factors engineering goals (including transparency) are based; a natural first step in human-centered AI or HCI, but not yet in medical image analysis. As methods for the human-centered development of transparent ML for medical image analysis mature, the guidelines presented here may require refinements to better reflect the challenges faced then. At the time of writing, supported by the findings of the systematic review, we believe that the lack of explicit formative research is the largest barrier to capitalizing on the benefits of transparent ML in medical image analysis.INTRPRT guideline that emphasize the importance of user and context understanding for transparent ML design, but provide alternatives to empirical studies for formative user research. By following these guidelines, ML designers must actively consider their end users throughout the entire design process. We hope that these design directives will catalyze forthcoming efforts to build transparent ML systems for healthcare that demonstrably achieve the desired human factors engineering goals.To conclude, transparency is an affordance of transparent ML systems, i.e., a relationship between models and end users. Therefore, especially in contexts where there exists a high knowledge gap between ML developers and the envisioned end users, developing transparent ML algorithms without explicitly considering and involving end users may result in products that are unintelligible in the envisioned context and irrelevant in practice. Efforts to build ML systems that afford transparency in the healthcare context should go beyond computational advances, which—based on the findings of our systematic review—is not common practice in the context of transparent ML for medical image analysis. While many of the approaches claimed transparency or derivative accomplishments in human factors engineering, they did so even without defining target users, engaging in formative user research, or reporting rigorous validation. Consequently, for most of the recently proposed algorithms, it remains unclear whether they truly afford transparency or advance human factors engineering goals. We acknowledge that building systems that afford transparency by involving end users in the design process is challenging for medical image analysis and related healthcare tasks. In this context, we propose the 51.The aim of the systematic review is to survey the current state of transparent ML methods for medical image analysis. Because ML transparency as major research thrust has emerged following the omnipresence of highly complex ML models, such as deep CNNs, we limited our analysis to records that appeared after January 2012, which pre-dates the onset of the ongoing surge of interest in learning-based image processing52. We searched PubMed, EMBASE, and Compendex databases to find articles pertinent to transparent ML (including but not limited to explainable and interpretable ML) for medical imaging by screening titles, abstracts, and keywords of all available records from January 2012 through July 2021. Details of the search terms and strategy can be found in Supplementary information We conducted a systematic literature review in accordance with the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) methodFollowing the removal of duplicates (1731 remained), studies were first pre-screened using the title and abstract. Studies that did not describe transparent methods nor medical imaging problems were immediately excluded (217 remained). We then proceeded to full-text review, where each study was examined to determine whether the study presented and evaluated a transparent ML technique for medical image analysis. Failure to comply with the described inclusion/exclusion criteria resulted in the study’s removal from further consideration. Detailed statistics and a complete description of the pre-screening and full-text review can be found in Supplementary information 53. This fact makes intractable to identify all articles about transparent ML methods. Finally, the review is limited to published manuscripts, long articles and novel approaches. Publication bias may have resulted in the exclusion of works relevant to this review.For the 68 selected articles that met the inclusion criteria, two authors (H.C. and C.G.) performed detailed data extraction to summarize important information related to the six themes described in INTRPRT Guideline Section. A data extraction template was developed by all authors and is summarized in Supplementary information INTRPRT guideline, the adequacy of which was confirmed during data extraction. Therefore, we summarize our findings for each theme and provide details of the extraction results for each article in Tables 2&3 in Supplementary information The data extraction template for studies included in the systematic review was structured using the six themes of the n = 33) was that the presented methods were developed by multidisciplinary clinician-engineering teams, as was evidenced by the incorporation of clinical specialists, such as physicians, radiologists, or pathologists, in the study team and on the author lists. In light of the current bias towards clinicians as end users of transparent ML algorithms, this observation suggests that designers may have communicated with a limited subset of the intended end users. However, no formative user research is explicitly described or introduced in these articles to systematically understand the end users before implementing the model. Further, we found that incorporating clinical experts did not have a considerable impact on whether clinical priors or standard or care guidelines  were used to build the ML system . Regarding the technical approach to provide transparency, the incorporation of medical experts motivated designers to incorporate prior knowledge directly into the model structure and/or inference for medical imaging (73%/64% articles with/without the incorporation of end users do not need a second model to generate transparency).A common trend among included studies (n = 15), use of human-understandable features (n = 11), a combination of deep neural networks and transparent traditional ML methods (n = 7), visualization approaches (n = 5), clustering methods (n = 4), uncertainty estimation/confidence calibration (n = 3), relation analysis between outputs and hand-crafted features (n = 3), and other custom techniques (n = 20).Transparency of ML systems was achieved through various techniques, including attention mechanisms 58 and multiple sclerosis segmentation in MRIs61. Clinical prior knowledge was also inserted into the attention mechanism to make the whole system more transparent. For instance,65 split brain MRIs into 96 clinically important regions and used a genetic algorithm to calculate the importance of each region to evaluate Alzheimer’s Disease (AD).The use of an attention mechanism was the most common technique for adding transparency. Attention mechanisms enabled the generation of pixel-attribution methodse.g., morphological and radiomic features, from predicted segmentation masks generated by a non-transparent model79 followed by analysis of those hand-crafted features using a separate classification module; (2) Directly predicting human-understandable features together with the main classification and detection task84. In these approaches, all tasks usually shared the same network architecture and parameter weights.Human-understandable features, e.g., hand-crafted low-dimensional features or clinical variables  were frequently used to establish transparent systems. There existed two main ways to use human-understandable features in medical imaging: (1) Extracting hand-crafted features, 90. Rule-based algorithms91 and regression methods92 were used to promote transparency of the prediction93 created a Graphical Convolution Network (GCN) based on clinical knowledge to model the correlations among colposcopic images captured around five key time slots during a visual examination.Instead of explicitly extracting or predicting human-understandable features, other articles further analyzed deep encoded features with human-understandable techniques by following clinical knowledge. Techniques such as decision trees were constructed based on clinical taxonomy for hierarchical learningWe also identified various other methods to create transparent systems. These methods can be categorized as visualization-based, feature-based, region importance-based, and architecture modification-based methods. Each approach is discussed in detail below.95, pixel-level predicted probability96, or a combination of different levels of feature maps98. (2) Latent feature evolution: Encoded features were evolved according to the gradient ascent direction so that the decoded image  gradually change from one class to another101.Visualization-based methods provide easy-to-understand illustrations by overlaying the original images with additional visual layouts generated from transparency techniques. There existed two main visualization-based methods: (1) Visualizing pixel-attribution maps: These maps may be generated using gradient-based importance analysis104 first encoded images to deep features and then clustered samples based on these deep features for prediction or image grouping tasks. Feature importance was also well-studied to identify features that are most relevant for a specific class by feature perturbation106 and gradients107,108 identified and removed features with less importance for final prediction through feature ranking.Feature-based methods directly analyze encoded features in an attempt to make the models transparent. Various feature-based transparency method were proposed for transparent learning111 and healthy-looking sub-regions112 was used to find the most informative and relevant sub-regions for classification and detection tasks.As an alternative to measure feature contribution, input region importance was also analyzed to reveal sub-region relevance to each prediction class. Image occlusion with blank sub-regions58 pruned the architecture according to the degree of scale invariance at each layer in the network113 created ten branches with shared weights for ten ultrasound images to mimic the clinical workflow of liver fibrosis stage prediction114 aggregated information from all three views of mammograms and used traditional methods to detect nipple and muscle direction, which was followed by a grid alignment according to the nipple and muscle direction for left and right breasts115 proposed to learn representations of the underlying anatomy with a convolutional auto-encoder by mapping the predicted and ground truth segmentation maps to a low dimensional representation to regularize the training objective of the segmentation network.Other approaches modified the network architecture according to relevant clinical knowledge to make the whole system transparent116 used similar-looking images (nearest training images in feature space) to classify testing images with majority votes. Causal inference with plug-in clinical prior knowledge also introduced transparency directly to automatic systems119. Confidence calibration and uncertainty estimation methods were also used to generate additional confidence information for end users122.Some other methods used the training image distribution to achieve transparency in classificationn = 30). From the remaining 38 articles, 17 articles implied that the envisioned end users would be clinicians, while the remaining 21 did not specify the envisioned target users. Articles that were more explicit about their end users were more likely to rely on clinical prior knowledge (Level 2 evidence) in model design. In total, 47% of articles that specified or implied clinicians as end users implemented clinical prior knowledge in the transparent systems while only 18% of articles without end user information use clinical prior knowledge.A striking observation was that none of the selected articles aimed at building transparent systems for users other than care providers. Less than half of the articles explicitly specified clinicians as the intended end users of the system . For instance, some articles evaluated the localization ability of post-hoc explanations by defining an auxiliary task, such as detection88 or segmentation112 of anatomical structures related to the main task. They then contrasted relevant regions identified by the model with ground truth annotations. These quantitative measures  allowed for further comparisons with traditional explanations methods. Similarly116, defined a multi-task learning framework for image classification and retrieval, evaluating retrieval precision and providing a confidence score based on the retrieved neighbors as an attempt to check the learned embedding space. Capturing relevant features consistent with human intuition was proposed in ref. 106 by measuring the fraction of reference features recovered, which were defined according to a guideline. Overall, the evaluation of explanations through auxiliary tasks required additional manual efforts to get the necessary ground truth annotations.The second approach attempted to quantify the quality of explanations for a specific purpose , completeness (fraction of the training set covered) and compactness (size in bytes) were measured in ref. 87. A measure related to completeness was defined in ref. 88 and aimed to capture the proportion of training images represented by the learned visual concepts, in addition to two other metrics: the inter- and intra-class diversity and the faithfulness of explanations computed by perturbing relevant patches and measuring the drop in classification confidence. Other articles followed a similar approach to validate relevant pixels or features identified with a transparent method; for example, in ref. 64 a deletion curve was constructed by plotting the dice score vs. the percentage of pixels removed and ref. 55 defined a recall rate when the model proposes certain number of informative channels95 proposed to evaluate the consistency of visualization results and the outputs of a CNN by computing the L1 error between predicted class scores and explanation pixel-attribution maps. In summary, while the methods grouped in this theme are capable of evaluating how well a method aligns with it’s intended mechanism of transparency, they fall short of capturing any human factors-related aspects of transparency design.Properties of the explanation itself were also quantified as their usefulness to identify risky and safe predictions at a voxel-level for the main task by thresholding on their predictive uncertainty valuesn = 40) by showing pixel-attribution visualizations overlaid with the input image or rankings of feature relevance, along with narrative observations on how these visualizations may relate to the main task. These qualitative narratives might include comparisons with other visualization techniques in terms of the highlighted regions or the granularity/level of details. Furthermore, following a retrospective analysis, the consistency between the identified relevant areas/features and prior clinical knowledge in a specific task was a common discussion item in 37% of all the articles (n = 25); refer to articles117 for examples. While grounding of feature visualizations in the relevant clinical task is a commendable effort, the methods to generate the overlaid information have been criticized in regards to their fidelity and specificity124. Further, as was the case for methods that evaluate the fidelity of transparency information, these methods do not inherently account for human factors.The third, and most common approach, involved a qualitative validation of the transparent systems  to understand the AI decision process. Users’ understanding was evaluated as the accuracy to predict the AI’s diagnosis for a target image and a binary judgment on whether they certify the AI for similar images (and justify using multiple-choice options). Users’ agreement with the AI’s predictions was measured as well. The empirical evidence suggested that explanations enabled radiologists to develop appropriate trust by making an accurate prediction and judgment of the AI’s recommendations. Even though radiologists could complete the task by themselves, a comparison with the team performance was not included, nor the performance of the AI model in standalone operation. An alternative evaluation of example-based explanation usefulness was performed in ref. 121, in which pathologists (14 study participants) determined the acceptability of a decision support tool by rating adjectives related to their perceived objectivity, details, reliability, and quality of the system. Compared to a CNN without explanations, the subjective ratings were more positive towards the explainable systems. However, neither the team (expert + AI) nor expert baseline performance was evaluated. The benefit of involving a dermatologist to complete an image grouping task was demonstrated in ref. 102, in which domain knowledge was used to constrain updates of the algorithm’s training, resulting in a better grouping performance than a fully automated method. The user evaluation only measured the task performance. These studies that explicitly involve target users to identify whether the envisioned human factors engineering goals were met stand out from the large body of work that did not consider empirical user tests. It is, however, noteworthy that even these exemplary studies are based on very small sample sizes that may not be sufficiently representative of the target users. Careful planning of the study design  that allows to properly evaluate whether the system achieves the intended goals by adding transparency to the ML system is fundamental, especially considering the resources needed and challenges involved in conducting user testing in the healthcare domain.Lastly, transparent systems can be directly evaluated through user studies on the target population, in which the end users interact with the developed ML system to complete a task based on a specific context. In ref. n = 49) or did not discuss the transparency component at all (n = 9). Task performance was evaluated in the majority of the articles, 91% (n = 62), and most of them contrasted the performance of the transparent systems with a non-transparent baseline (n = 41). Of those, 36 works (88%) reported improved performance and 5 (12%) comparable results.Even though there were articles that assessed human factors-related properties of the transparency mechanism, a striking majority of articles did not report metrics beyond performance in the main task (n = 28) directly implemented these priors into the model structure and/or inference, while only 68% articles with computer vision priors (n = 40) provided transparency by the model itself and/or the inference procedure.We differentiate two types of priors that can be used as a source of inspiration to devise transparent ML techniques: (1) Priors based on documented knowledge, and especially clinical guidelines considering the unvaried end user specification identified above; and (2) Priors based on computer vision concepts. Most (93%) articles that incorporated clinical knowledge priors  in cardiac MRI79 and mean diameter, consistency, and margin of pulmonary nodules80 were commonly computed to establish transparency. For problems with a well-established image reporting and diagnosis systems, routinely-used clinical features, e.g., Liver Imaging Reporting and Data System (LI-RADS) features for Hepatocellular carcinoma (HCC) classification84 or BI-RADS for breast mass82 suggested that the ML systems may be intuitively interpretable to experts that are already familiar with these guidelines. Human-understandable features relevant to the task domain were extracted from pathology images, e.g., area and tissue structure features70. Radiomic features were also computed to establish the transparency of ML systems125.A direct way to include clinical knowledge priors was through the prediction, extraction, or use of human-understandable features. Morphological features, n = 11) mimicked or started from clinical guidelines and workflows to construct the ML systems114 followed the clinical workflow to encode multiple sources of images and fused the encoded information for the final prediction. Other works followed the specific clinical guidelines of the problems to create transparent systems65 split brain MRIs into 96 clinical meaningful regions as would be done in established clinical workflows and analyze all the regions separately. Some other clinical knowledge priors were also presented126 established a hierarchical label structure according to clinical taxonomy for image classification71 leveraged the transparency from the correlation between the changes of polarization characteristics and the pathological development of cervical precancerous lesions. Clinical knowledge from human experts was used to refine an image grouping algorithm through an interactive mechanism in which experts iteratively provided inputs to the model102.Besides human-understandable features, clinical knowledge can be used to guide the incorporation of transparency within a model. Some articles  proposed transparent ML algorithms for classification and detection problems, such as image classification and abnormality detection. Three-dimensional (3D) radiology images (n = 24) and pathological images (n = 15) were the most popular modalities involved in the development of transparent algorithms. The complex nature of both 3D imaging in radiology and pathological images makes image analysis tasks more time consuming than 2D image analysis that is more prevalent in other specialities, such as dermatology, which motivates transparency as an alternative to complete human image analysis to save time while retaining trustworthiness. In detail, classification problems in 3D radiological images and pathological images included abnormality detection in computed tomography (CT) scans106, MRIs117, pathology images121 and positron emission tomography (PET) images63. Mammography dominated the 2D radiology image applications125, mainly focusing on breast cancer classification and mass detection. For other 2D radiology image applications118, aimed at pneumonia and pneumothorax prediction from chest X-rays and113 created a transparent model for liver fibrosis stage prediction in liver ultrasound images. Classification and detection tasks were explored in other clinical specialities, including melanoma85 and skin lesion grade prediction87 in dermatology, glaucoma detection from fundus images97 and retinopathy diagnosis111 in ophthalmology, and polyp classification from colonoscopy images in gastroenterology120.Various types of medical image analysis tasks were explored in the included articles. Most of the articles (n = 9). Research about transparency mainly focused on segmentation problems for brain and cardiac MRIs115. Other segmentation problems included mass segmentation in mammograms76, cardiac segmentation in ultrasound115, liver tumor segmentation in hepatic CT images, and skin lesion segmentation in dermatological images58. There also existed other applications, e.g., image grouping in dermatological images102 and image enhancement (super resolution task) in brain MRIs122 and cardiac MRIs115.Segmentation was another major application field (n = 60). A much smaller sample of articles (n = 4) aimed to build transparent systems for much more difficult tasks where no human baseline exists, e.g., 5-class molecular phenotype classification from Whole Slide Images (WSIs)88, 5-class polyp classification from colonoscopy images120, cardiac resynchronization therapy response prediction from cardiac MRIs83, and super resolution of brain MRIs122. The remaining articles (n = 4) did not include explicit information on whether human baselines and established criteria exist for the envisioned application, e.g., magnification level and nuclei area prediction in breast cancer histology images58, age estimation in brain MRIs60, AD status in Diffusion Tensor Images (DTIs), and risk of sudden cardiac death prediction in cardiac MRIs79. As previously mentioned, tasks that are routinely performed in clinical evidence may have robust human baselines and clinical guidelines to guide transparent ML development. Applications that are beyond the current possibilities, however, require a more nuanced and human-centered approach that should involve the target end users as early as possible to verify that the assumptions that drive transparency are valid.Most of the application tasks were routinely performed by human experts in current clinical practice (Supplementary Information of the main article