N = 333 errors, with N = 167 uniquely occurring within the interface. Learning Designers identified errors at greater frequencies  = 28.86 per expert) than other evaluator groups . Patterns in severity and error types were also observed between reviewer groups. The findings suggest that Learning Designers are skilled in detecting interface errors, which benefits developers assessing usability when access to end users is limited. Whilst not offering rich narrative feedback generated by user-based evaluations, Learning Designers complement healthcare professionals’ content-specific knowledge as a ‘composite expert reviewer’ with the ability to generate meaningful feedback to shape digital health interfaces.User-based evaluation by end users is an essential step in designing useful interfaces. Inspection methods can offer an alternate approach when end-user recruitment is problematic. A Learning Designers’ usability scholarship could offer usability evaluation expertise adjunct to multidisciplinary teams in academic settings. The feasibility of Learning Designers as ‘expert evaluators’ is assessed within this study. Two groups, healthcare professionals and Learning Designers, applied a hybrid evaluation method to generate usability feedback from a palliative care toolkit prototype. Expert data were compared to end-user errors detected from usability testing. Interface errors were categorised, meta-aggregated and severity calculated. The analysis found that reviewers detected  For developers of online health resources, some inherent difficulties are identifying and recruiting representatives of the intended audience to participate in usability evaluations , comparePalliative care is one such domain. Palliative care is provided to patients who have been diagnosed with a non-curable life-limiting condition, illness, or disease, as a family-centred care model supporting the quality of life of the dying person until death, and provides help for carers and families during the illness and bereavement ,6. ThereFor developers applying a user-centred approach, accessing patients receiving palliative care or their carers could influence the likelihood of usability being evaluated. Involving patients and carers in the process is challenging on three fronts: (1) identifying potential volunteers from within the community , specialDemographically, carers are likely to be engaged with the online environment  and reprWhere representative carers or family members as end users may not be available to participate in evaluations , alternaFor digital health resources developed from research activities, multidisciplinary development teams within academic settings are likely to be non-specialists in designing and developing online or digital health products. Subject-matter experts and academic development teams will likely adopt a collaborative approach to supplement their deficiencies in the product development cycle through contracted partnerships with individuals offering expertise in areas including technical development, design, evaluation, or marketing and promotion. As content specialists, healthcare professionals and clinicians are vested in translating their research outcomes into a meaningful experience for their patients, carers and family members. However, the involvement of usability experts is unlikely due to a lack of funds to support this expertise, along with the recognition of the need to engage in formative assessment to optimise interface design for end users.In higher education institutions, Learning Designers consult with teaching academics to guide development, incorporate learning pedagogy, and construct measures or instruments assessing the effectiveness of materials to scaffold learners’ knowledge . In an eIn contexts where representative end users may not be available to participate in evaluations , alternaAs usability novices , healthcHeuristic evaluation and cognitive walkthroughs are other inspection UEMs where experts detect potential usability issues during sessions. Heuristic evaluations require experts to assess the interface’s usability against a set of guidelines or principles ,23. CognExpert peer review also considers end users’ knowledge and awareness of the subject domain and specific content-based errors detected by domain specialists based on the understanding and context of the objective and application of information provided within the resource . All thrFor all development teams, usability testing is considered the ‘gold standard’ , a methoPragmatically, the likelihood of multidisciplinary academic development teams conducting usability testing depends on funding, time available, or the availability or experience of project staff to recruit, conduct, interpret and report outcomes within constrained processes. Moreover, unlike commercial developers with a specific audience, the generalist nature of health resources forces additional complications upon developers to discern range and prioritise selection, accounting for a diverse audience’s knowledge, skills, and abilities.In comparison, expert-based usability inspection methods can offer a high rate of return for a relatively small investment of time and money ,31 in coThe difficulties experienced by multidisciplinary development teams underpinned by usability evaluation inexperience and practical constraints surrounding development processes could be offset by the involvement of an ‘in-house’ composite double expert. For example, accessing Learning Designers already employed within higher education settings could offer development teams the ability to examine interface flaws objectively, comparing content to domain knowledge to judge quality, reliability, and accuracy. Further offering feedback as a subjective assessment of the interface for technical errors associated with operational or functional aspects that decrease usability and impede end-user interactions. How this feedback compares to the quality or efficacy of end-user-derived usability feedback is unknown.As the literature has not previously reported recruitment, participation, or outcomes from usability evaluations of digital health information resources undertaken by Learning Designers during development, this current study will explore the following research questions:RSQ1. How does usability feedback differ between Learning Designers and healthcare professionals when completing an expert review of a palliative care resource?RSQ2. Is there a role in usability evaluations of digital health interfaces for Learning Designers: as a sole expert evaluator group, in combination with healthcare professionals, or a tripate with healthcare professionals and end users?Traditional usability practise requires developers to recruit three to five heuristic double experts  with domComprehensive usability evaluation was undertaken on an early prototype of the Australian Carer Toolkit for Advanced Disease (known here as the CarerHelp Toolkit or ‘the Toolkit’). The Australian Government Department of Health funded the Toolkit project. The CarerHelp Toolkit is an online resource designed to support family carers of relatives or friends with the advanced disease living at home within the community. This resource aims to increase family carers’ knowledge and confidence to support the provision of end-of-life caregiving at home through evidence-based information, educational activities to build skills or knowledge, and access to how-to guides in the form of vignettes, interactive activities, and videos.Usability evaluation approaches from the Web Development Model for Healthcare Consumers (WDHMC)  were appUsability evaluations were conducted on a prototype of the CarerHelp Toolkit; information was structured across two levels consisting of four sections: Carer Pathways, Carer Voice, Carer Library and About CarerHelp. User-based usability testing was undertaken on an earlier version of the prototype accessed by experts due to the timing of evaluation within the development cycle. These prototypes were similar, although the earlier version had fewer interactive features and limited visual representations within the interface compared to the later version. Screengrabs are provided in https://www.carerhelp.com.au/.The reiterated post-release version of the CarerHelp Toolkit is freely available online at the URL (accessed on 10 January 2023): This study received ethical approval from the Flinders University Social and Behavioural Research Committee (Project Number 8347).Two groups of experts, healthcare professionals (HCP) working in palliative care as subject-matter experts and Learning Designers (LD) as practising designers of digital education , were invited to participate as reviewers within this study. Suitability for the study was limited to professionals currently practising and there were no limitations placed on levels or years of experience. Instead, inclusion criteria focused on availability to be involved around regular working hours, access to a device with Internet connectivity, and the willingness to share feedback in written and verbal formats (using online conferencing software). Between 4 and 8 reviewers were sought for each review group to ensure reviewer diversity . As this was an exploratory study, a smaller sample for each group was considered acceptable, especially given the difficulties in identifying and recruiting representatives from both professions . Expert reviewers could not participate in the study due to a lack of flexibility in their work schedules rather than disinterest.Local, state and national palliative care organisations/services were approached to identify potential HCPs who had previous experience or were currently supporting the palliative care needs of patients and their carers living within a community setting. Palliative care specialists, general practitioners, nurses, and allied health professionals were invited to participate in the usability sessions through their services. These included Southern Adelaide Palliative Service, Palliative Care Queensland and the Australian and New Zealand Society of Palliative Medicine. Four HCPs who were actively involved in palliative care practise consented to review the Toolkit. Participants included two directors of palliative care services , a social worker and nurse practitioner.The professional organisation for online designers  working within the private and higher education sector, the Australasian Society for Computers in Learning in Tertiary Education (ASCILTE), assisted by widely promoting the study to their membership. Seven LDs from across Australian higher education institutions self-nominated and consented to be an expert reviewer.Before commencing the review process, participants provided informed consent after the researcher had explained the research protocol and perceived risks involved. Each expert was then asked for descriptions of their professional credentials  and to self-assess their level of technical ability using the Internet by responding to the following question:An avoider of everything onlineA novice or learner or beginnerMostly confident—having intermediate skillsAn expert who is confident in finding and using online informationI am:Evaluators completed the review process in two stages; the first was a digital document providing a structure and guiding interaction with the prototype refer to . ProfessOnce the review document was completed and returned to the researcher, participants undertook the second stage of the review process by remotely debriefing their findings during a 30 min online interview session and providing an opportunity to explain their written feedback. In addition, the functionality of the conference software  demonstrUser-based feedback on issues and errors within the prototype was generated through formal usability testing methodology, with participants representing the intended audience of the CarerHelp Toolkit.Family or primary carers of patients with palliative care needs living at home within the community were invited to participate in the study. Carers were sought from the wider community and specialist palliative care services within the southern areas of Adelaide, South Australia. A cohort of active and bereaved carers (6–8 months post-death) was identified by the Network Facilitator at the Laurel Hospice Caregiver Network . The researcher contacted carers who were interested and eligible to participate, and the study protocol was explained. Perceived risks were clarified, and formal consent to participate in usability testing was provided. A sample size of six was calculated by applying the probabilistic model of problem discovery  to identAll participants customised the device and peripherals, reflecting their natural Internet interaction environment experienced at home . All online interactions were documented and digitally recorded using conference software to ensure that all audio commentary, facial expressions, and accompanying cursor movements were captured for post-session analysis.The following section will not comprehensively describe the usability testing protocol or methodology. Instead, detailed explanations of scenario development, task descriptors, interface satisfaction assessment, and outcome performance measures are described in the formal usability report informing reiteration of the interface refer to .Before beginning the session, the researcher ensured that each carer was aware of the Toolkit content and any perceived risks and emphasised that the session could be stopped at any time if individuals were experiencing distress. After providing formal consent, carers completed questionnaires describing their online behaviours, self-rating their technical ability and level of health literacy.Each of the six participants then completed eight scenario-based tasks within the CarerHelp Toolkit interface using the Concurrent Think Aloud technique, where individuals verbally describe their thoughts or feelings when completing each task. Participants completed an identical set of tasks during the sessions; each task had a completion limit of 3 min. Completion was when the target information was located within the allowed time. Failures were registered if the target was not found within time, if the incorrect target was identified or if the participant abandoned the task.Methods undertaken within this study follow traditional formative usability evaluation processes requiring small groups of evaluators to provide reiterative feedback to improve interface design. Therefore, it is acknowledged that this study is under-powered and unlikely to detect statistically significant differences between evaluator groups due to the small sample sizes. However, trends in descriptive usability error detection patterns and exploration of types or severity of usability errors can assist in understanding the potential roles LDs can play in optimising digital interface designs for palliative care resources .After deidentifying data, a qualitative meta-summary of content findings was generated from the written feedback document and other narratives from the debrief interviews from both reviewer groups. Quantitative logic was then applied to aggregate error types between participants, and this provided a process to assess error frequency and identify problems, missing resources or content, and opportunities or suggestions for interface improvements. Types of errors describe problems experienced by evaluators relating to specific functional aspects of user interaction at the user interface level.Further analysis of content-specific errors identified within written information included frequency-based analysis of the types of content errors detected by reviewers by applying a modified coding schema to accommodate the interface’s online environment and technological aspects to error data as described in All types of errors from all three reviewer groups (USER (carers), HCP and LD) were collated into a single list, with a bottom-up approach to meta-aggregation adopted to collate the differences from summarising types of usability errors detected within the Toolkit interface across all evaluators.Aggregation provided an opportunity to compare commonalities or differences in error types identified by specific reviewers, further providing the capacity to highlight interface problems discovered exclusively by a single reviewer group or perceived (shared) across more than one reviewer group. Error types were grouped into classes categorised by approaches, development considerations, content level or structural elements of the user interface. These are considered groups of errors relating to global aspects of user interface design.Content-specificDesign or content constructionInformation flowNavigationEmbedded resource or activityPedagogy or educational strategiesMinor typographical or grammatical errorsMajor typographical or grammatical issuesClasses of usability error included:Frequency of the occurrence within the interfaceImpact of the error (if it occurs) on users to overcomeThe persistence of the error within the interface continuously affects evaluator interactionsThe severity of interface errors is then assessed on three factors  influencIn considering these factors, each error was assessed and, using Nielsen’s Severity Rating scale , receive0 = I do not agree that this is a usability problem at all1 = Cosmetic problem only: need not be fixed unless extra time is available on the project2 = Minor usability problem: fixing this should be given low priority3 = Major usability problem: important to fix, so should be given high priority4 = Usability catastrophe: imperative to fix this before a product can be releasedIn rating interface issues, errors persistently having a significant impact on evaluators received the highest severity rating.Of the four HCPs recruited for the study, two were from Palliative Care Queensland and one from the Australian and New Zealand Society of Palliative Medicine and Southern Adelaide Palliative Care Service. The HCPs had a minimum of seven years specialising in palliative care . Three of the four self-rated themselves as having expert technical skills, with one self-rated their ability as being of intermediate level. Characteristics of the HCPs are presented in Seven Learning Designers were recruited in total, all members of the ASCILTE organisation. All participants were employed within the university sector and held positions in institutions within five different states of Australia. All LDs were self-assessed as experts in using technology whose combined experience spanned 24 years post-qualification . In addition, two participants were managing academic units; although they had extensive experience as educational technologists, the other five LDs actively practised the design of educational materials. Characteristics of participants working in higher education as professional designers of digital learning experiences working are summarised in The differences between feedback generated through expert review UEM by subject-matter (HCP) and LD with digital expertise were examined. The types, frequency and severity of usability errors detected by each evaluator group were examined to explore areas of strengths or weaknesses when detecting problems within the interface. Additionally, content-specific and types of qualitative descriptions of errors were compared between groups to supplement the analysis of usability problems detected by each set of evaluators.In total, both expert evaluator review groups identified 279 errors within the Toolkit prototype interface . Those HCPs who had been practising palliative care for 6–10 years identified the highest number of errors with a medium–low severity compared with other cohorts. In addition, 79.2% of interface errors were detected by HCPs who were self-rated experts with technology compared to intermediate-skilled HCP reviewers (20.8%).For LDs, years of experience designing online positively influenced error detection frequency. Although LDs with greater than 16 years of experience identified an equivalent number of errors as those with less experience , LDs with increased practical experience on average detected 52 errors compared with M = 19.6 errors for LD with less than 16 years designing experience. LDs also identified a higher frequency of errors rated highly severe than errors found by HCPs, with 71.4% of the most severe errors detected by LDs with between 11–20 years of experience. Errors with the highest frequency were proportional to LD and HCP reviewer groups when calculated as a percentage of total errors identified. Specific content errors constituted over 50% of total errors detected by HCP and 38.1% for LD. The frequency of navigation issues was comparable between 14–15% of total errors per group, and LD identified a greater proportion of errors impacting information flow  than the HCP reviewer group .Analysis of the qualitative feedback was undertaken by the primary researcher (AA) and categorised the content based on the modified Sayoran’s schema provided in Whilst LDs detected a greater frequency of errors than HCPs overall , average errors per reviewer were similar across groups . Types of content errors with the greatest frequency within the interface by experts explicitly referenced issues within embedded resources or learning activities  in “Need to make sure that this toolkit provides information for carers on how to improve and sustain person’s quality of life when at home” (HCP reviewer),descriptions of errors requiring rephrasing with examples of revisions statements (18.3%) such as:‘Caring for someone who is dying is the end of a journey of caring …—could be something like: Caring for someone dying also means that your role of carer will come to an end after the person has died. These resources help you to be prepared for dealing with the end-of-life care’ (LD Reviewer),an explicit reference to problems within the interface (16.7%), including:‘I think language is okay, but there are just too many words’ (HCP Reviewer),Or:‘…appropriate to also insert a link here to take the users back to the first page, rather than telling them to go to and use the menu (where is that?) to get back to the main page.’ (LD Reviewer).HCPs and LDs detected equivalent numbers of error descriptions with a provision of evaluation statements and assertions on the learnings from the text. LDs identified a greater frequency of errors describing grammatical or spelling errors (3 HCP:8 LD), were more likely to provide suggestions on applying strategies to content (4 HCP:9 LD) and were more forthcoming with the provision of alternate text through revision statements compared with healthcare professionals (1 HCP:21 LD). Unsurprisingly, HCPs identified specific issues or errors within the written content of the Toolkit webpages and were skilled at detecting content mistakes in-text to provide feedback based on statements of their knowledge of palliative care.Analysis examined the frequency and types of errors identified by each evaluator group. The role of LDs as evaluators in designing digital palliative care resources was explored through comparisons of LD usability data across evaluator groups, both as an independent reviewer group and in combination with the other evaluators. Shared  and exclusive errors were mapped to understand the influence that different evaluators have on usability data collected as a pragmatic reflection of the availability of these groups to participate in evaluations during the development process undertaken within academic development settings.Similarities and differences in the error type and frequency of detection by reviewer groups involved within the Toolkit usability evaluation approaches were explored by meta-aggregation of HCP, LD, and USER data to characterise error occurrences within the interface. Errors were categorised by reviewer, type, level of exclusivity or inclusivity and whether the errors are unique or co-existing across the reviewer group. Unique errors are discernible occurrences identified by single or multiple reviewers whose accretion decreases total interface error counts compared to the overall error number.Seventeen reviewers identified a total of 333 errors that did not occur exclusively for any single reviewer group, and these are co-existing within the interface and identified by all three reviewer cohorts. Further analysis found 167 unique errors occurring once within the interface , whilst the USER group detected 10.78% of unique errors, although only 16.22% of the overall errors within the interface. LDs identified the greatest number of errors overall and unique errors at the highest frequency within the interface  compared to the other reviewer groups. LDs also detected higher frequency on average per reviewer in both inclusive (MLD = 28.86: MHCPs = 19.25: MUSER = 9.0 errors) and uniquely occurring issues (MLD = 15.29: MHCP = 10.0: MUSER = 3.0 errors). Differences in the rate of error identification between the USER and expert reviewers widened when combing error counts into a single EXPERT cohort (HCP + LD). The EXPERT group attributed 279 errors  and 133 uniquely occurring errors  at an average of 12.09 errors per expert.N = 333). Further data consolidation into combinations of reviewer groups where each error instance is assigned a single identifying group decreased the overall unique error count from N = 167 to N = 143. Similarly, the analysis found that LDs identified over 50% of the total unique errors compared with HCPs (16.08%) and 5.59% for the USER review group. Consequently, review groups in combination with LDs were more likely to identify an increased proportion of unique errors compared to other HCP or USER group combinations  = −2.460, p = 0.031), USER (t(10.521) = 2.983, p = 0.031), HCP + USER (t(13.25) = 2.545, p = 0.024), LD + USER (t(20) = 2.144, p = 0.044) and HCP + LD + USER (t(20) = 2.747, p = 0.012).This pattern is analogous to counts of overall co-existing errors. LD was also more likely to identify errors or issues with site or platform performance and accessibility for end users. Five distinct accessibility issues were detected by just the LD reviewers and the LD + USER group. Differences in error detection rates between groups were analysed using independent t-tests , significance was indicated when p = 0.048), USER (t(10.707) = 2.864, p = 0.016), HCP + LD (t(11.402) = 2.392, p = 0.035), HCP + USER (t(10.555) = 2.573, p = 0.01) and LD + USER (t(10.662) = 2.788, p = 0.018) and HCP + LD + USER (t(10.032) = 3.191, p = 0.01). Significant differences were also observed between the average number of overall errors detected by the HCP reviewer group compared to HCP + USER (t(10.55) = 2.573, p = 0.027) and HCP + LD + USER groups (t(10.309) = 2.712, p = 0.021). There was only a single error type that all three reviewer groups identified. An error relating to grammatical or spelling errors specific to content within the pages or activities appeared in fourteen instances across the interface and was mutually inclusive to all reviewers participating in the evaluation process.LDs could also identify a significantly greater number of unique errors within the interface than all other reviewer groups: HCP (t(12.043) = −2.204, For digital health or medical information resource developers, there are inherent difficulties in undertaking inquiry-based usability evaluations that generate crucial expert or user-based feedback to inform interface reiterations. Factors such as identification, access, and availability of suitable experts or representative end-user recruitment influence evaluations’ likelihood of integration into typical development processes. The involvement of double experts who are equally knowledgeable of clinical subject matter and skilled in usability to evaluate digital health interfaces heuristically is appealing to development teams as it can alleviate the need for end users ,42 to paAccess to LDs, a potentially rich source of technically skilled professionals with background knowledge of user-centred design and an understanding of usability evaluation approaches, would be advantageous, especially for inexperienced, under-resourced development teams. Universities engage LDs to work across interaction, visual and education design. Alignment between designing educational materials, Toolkit instructional components  and evalIn this current study, there was an opportunity to explore the potential value to multidisciplinary teams in having access to ‘composite heuristic experts’ to improve interface designs for diverse audiences by assessing a palliative care prototype. Inviting LDs to be involved in evaluations provided a unique opportunity to compare types, frequency and severity of errors identified between evaluator groups through a hybrid UEM process  to assess usability within an academic setting. The interface of the palliative care digital Toolkit was assessed by HCPs (as subject experts), LDs (interaction experts), and representatives end users (carers). The potential role in evaluation and suitability as a double heuristic expert was explored by analysing the types, frequency and severity of usability errors identified by LDs, then comparing these to those detected by subject-matter experts and users. Areas of overlap and exclusion were mapped to understand the composition of evaluators and end users required to be involved to optimally shape the interface design for a resource supporting a diverse audience. Firstly, it is important to highlight the continuing difficulties in recruiting end users and evaluators, including LDs, to participate in usability evaluations.Difficulties in recruiting representative end users from health-specific domains were a primary driver in exploring the relative ease of identifying experts involved in the process compared to end users, for this study being carers providing palliative care within the community. Although approaches were similar, expert reviewers were equally challenging to recruit, not through lack of interest or willingness to participate but rather an availability constrained by time due to work commitments. Just as the unmoderated and remote review increased opportunities for participation in both groups of experts , this method could also limit participation given the diversity in user characteristics impacting interactive behaviours, especially within HCPs. Interestingly, HCPs with lower technical or digital competency levels should be encouraged to participate as a reviewer for developers to understand some of the difficulties or barriers their users could face when interacting with their interface.Conversely, end users were extremely difficult to recruit. Promotion through carer peak body organisations electronic communication channels failed to identify any potential carers interested in participating in the study from within the local carer community. A gate-keeper advocate provided access to interested carers from the local palliative care community service. No fewer than twelve active and recently bereaved primary carers were approached from within the carer network to satisfy the usability quota of six participants. Those who did not want to be involved acknowledged that their willingness to participate was tempered by anxiety and fear of reliving painful or sad experiences rekindled by viewing the Toolkit content. These feelings are offset by the need to assist developers in improving the resource to help carers have all the missing information that could have been of value during their own experience. Active carers expressed that time and availability due to their caring duties were also reasons to consider participating in the study. The difficulties in accessing suitable participants within vulnerable populations, including from within the palliative care domain, subsequently influenced the ability to recruit without impacting the timeliness of the development process. Once recruited, both evaluators and end users were pleased to be involved in the evaluation process, offering valuable feedback on the usability of different aspects of the Toolkit prototype interface. Finally, the analysis of verbal feedback demonstrated different approaches undertaken by healthcare professionals and LDs when providing expert feedback on the errors or problems encountered within the interface.Unsurprisingly, subject-matter experts (HCPs) were more likely to identify content-specific errors provided as statements of their palliative care knowledge in supporting carers’ needs in their professional practise. HCPs view the structure of sentences, text organisation and the relationship to how the text relates to a specific purpose or audience by applying process knowledge  with metacognition to review text for issues  to ensurLearning Designers could also identify content-based errors, although their feedback offered alternatives or rephrased content as revision statements described in their review documentation. Narratives offered an insight into lay understandings of palliative care that were less detailed than the commentary offered by HCPs and should be privileged. This information potentially improves interface reiterations creating understandable content similarly levelled at an audience with a limited understanding of palliative care as part of their lived experience. Feedback provided represented a comparison of reviewers pre-existing knowledge to the overall text from a generalist viewpoint compared to the HCP process, who approached the review process through a sequential, step-by-step method to formulating feedback as directed by their domain knowledge .Qualitative descriptions offering valuable experience-based context to perceiving why errors are considered a problem for a reviewer is but one facet of understanding barriers to interface usability. The identification of explicit errors can highlight critical interface areas impeding end-user interaction. For example, the similarities and differences in the types, frequency and severity of errors detected by each group were explored to map strengths and perhaps weaknesses when comparing LDs to HCPs and end users.Meta-aggregation and applying quantitative logic to the analysis highlighted errors in the interface that were shared or discrete to reviewer groups. Experts, on average, were equally skilled at identifying high-frequency content errors; however, as reviewers were evaluating the Toolkit interface, there was a trend for LDs to be more sensitive to errors affecting the user experience of the information. Errors were more likely associated with information flow between and within pages, navigation devices or scripted hyperlink text, and interactions between the site and the user. Typical usability error types were more pronounced within the interface for LDs; HCPs identified a sub-set of these error types, perhaps reflecting common issues that they, as typical users, had previously experienced during their interactions with online technologies. HCPs and LDs identified examples of all categorised error types. As a reflection of their usability knowledge and professional practise, LDs detected four discrete error types that can improve interactions for users who face barriers to using or accessing health information, including visual representations, utility, error recovery, and accessibility.As a single reviewer group, LDs demonstrated the ability to detect errors at a greater frequency than HCPs and carers whilst, on average, having improved efficiency in identifying errors per evaluator. Rates of error identification across the interface of the CarerHelp prototype indicated that LDs detected similar error quotients as heuristic double expert evaluators  and, in Study observations and qualitative narratives suggest LDs are well suited to review online health toolkits due to technical skills and awareness of building within online platforms or programs, integrating activities to create interactive experiences, and understanding interface features contributing to functionality, including navigation.Conceptually, online health toolkit interfaces ,53 are lThe combination of HCPs and LDs could effectively generate usability feedback where heuristic double experts are absent or, in many cases, unavailable. Both reviewer groups balanced the types of feedback provided to developers and the variety of error types identified within the interface. The data indicated that each group complemented the other, especially in covering the deficits in identifying errors that are unsighted or not recognised by one group tend to be detected by the other. LDs bring their expertise, knowledge of interaction design and, although usability evaluations are not a standard component of their everyday practise, an awareness of usability principles and their personal ‘baggage’ . An indiThe necessary recruitment of representatives for both review groups to participate in the process was equally problematic. Reviewers, regardless of the profession, were enthusiastic about being involved and whilst remote facilitation enabled involvement geographically, difficulties identifying suitable and available professionals were further compounded by complex development processes.For development teams designing and building online health information resources, these two groups of experts could replace users within the evaluation process, as experts are more adept at identifying errors across both skills, technical  and ruleQualitative narratives provide a powerful mechanism to reiterate the interface, improve the content, and shape interface functionality. This narrative also adds weight to identifying veritable errors, an alternate perspective to expert opinions, and improving usability and experience within the interface.It is important to recognise the exploratory nature of this study as this research approach follows user-centred design principles involving methodologies to generate formative data to inform the reiteration of the interface. Pragmatically, the small evaluator sample sizes reflect typical usability evaluation practise. It is acknowledged that on analysis of usability data, small, subtle, or nuanced differences in the between-group comparisons are unlikely to be detected as the sample size was small and statistically underpowered ,37. It iOutcomes from this study suggest that there is potential value in LDs and HCPs being involved as composite double experts to generate valuable feedback in the development and design of palliative care digital resource.Further investigation is required to understand the comparative differences or similarities in the type, frequency and severity of the usability errors identified by composite experts and those detected by trained, heuristic double experts. Areas of commonality between the two evaluator groups could assist multidisciplinary development teams in deciding whether investing time and money to either access or train in-house heuristic experts is worthwhile. This is especially so given the low investment and high return of utilising HCPs and LDs to generate valuable feedback to supplement end-user narratives of the lived experience.Future research could investigate the types of usability information that might support healthcare professionals to become heuristic double experts. In addition, there is scope to develop, implement and evaluate education or practise guidance that could support healthcare professionals to rapidly expand and develop their existing technical abilities to include scholarship in user-centred design principles and an understanding of the need to consider end-user experience in interface design. The question of composite expert diversity is also an important area to explore, especially given HCPs are both diverse in their abilities, backgrounds, and experiences and in their specialty areas. For example, types of feedback generated by different HCPs such as specialist palliative care physicians and nurses compared to those of generalised specialties who are caring for patient’s clinical and psycho-social symptoms at the end of life. Findings could support the wider application of this approach to develop generalised health online resources or m-health applications supporting health promotion, monitoring, or encouraging behaviour change in the wider population.Recruiting end users is difficult, and for multidisciplinary teams creating digital resources for carers of palliative care patients is further complicated by the perceived sensitivity of the subject domain, design inexperience, and working within an increasingly complex development environment. Within a palliative care context, usability evaluations involving LDs as expert evaluators were explored to understand their potential to support HCPs in shaping health interface designs when end users are difficult to recruit. Comparative analysis of content-specific errors found that HCPs were more likely to offer feedback by applying process knowledge to their understanding of the palliative care interface information and carers as end users. LDs identified similar content errors; however, they could offer narratives from their perspective of someone who has experienced death or a loved one dying and were equally adept at identifying high error frequencies. Importantly, LDs were increasingly sensitive to errors impacting end users who face barriers to accessing and using digital health information.Our research suggests that through their professional aptitude as digital interaction designers and ability to reflect on their lived experiences, LDs have and can provide a unique and valuable perspective as evaluators during usability evaluations. However, feedback generated from carers as end users was highly contextualised to their needs and reflected their experiences as patients, carers or families interacting with healthcare, information, or systems. For developers of digital health interfaces, the involvement of experts cannot replace the insights of the lived experience and the influence of diversity of user characteristics on the types of feedback generated by end-user participation. Recruiting end users for usability evaluations remains a crucial step in creating meaningful and useful interactive experiences whose designs can support our communities’ health information needs.


Hardware removal is a common procedure in orthopedics, but it can be challenging and time-consuming. Difficulties in screw removal may arise due to bone growth or cement covering the screw heads, leading to screw damage, and increased surgical time. In this article, we describe a simple and inexpensive technique utilizing bone wax to protect screw heads from bone growth or cement, facilitating future implant removal. The application of bone wax over screw heads acts as a barrier, preventing bone growth or cement from engaging with the screw heads. Hardware removal is a common procedure in orthopedics and can be indicated for several situations, including hardware irritation, infection, hardware breakage, revision for nonunion or malunion, and patient demand -3. In soHerein, we describe a simple inexpensive method for protecting the head of screws from bone growth or bone cement in various surgical situations, which facilitates future implant removal using bone wax.We use bone wax as a mechanical barrier to protect head screws from bone formation or cement in various situations, such as isolated lag screws or cancellous screws, screws in plates, and screws over acetabular cage before cementation, preventing cement engagement to the head of the screw.We apply a small quantity to cover the head of each screw before acetabular cup cementation in case the acetabular cage is used for total hip arthroplasty Figure .Additionally, bone wax is utilized to protect screw heads during cement use in tumor surgeries that involve curettage and filling with bone cement. It is also employed in the Masquelet technique for treating bone defects and in trauma cases following plate fixation Figure .This technique eases future implant removal and saves time during future implant removal by decreasing the time of removing bone growth or cement from the head of screws before screwdriver application, decreasing the risk of stripped head screws.Bone wax is a nonabsorbable material that was first described by Henri Ferdinand Dolbeau in 1864 and then popularized by Sir Victor Horsley in 1892 . Bone waAlthough bone wax is generally regarded as safe and has shown its advantages, it is important to be aware of potential complications that may arise from its use. One significant concern is the risk of infection -13. As bIn our department, bone wax is commonly used to protect the heads of screws during revision total hip replacement surgeries when an acetabular cage is required as well as in tumors and trauma cases. This technique prevents cement from getting into the screw heads, making future removal easier. We have not encountered any complications reported in previous studies. Our hypothesis is that the lack of complications in our cases may be attributed to our specific technique of applying bone wax. In our technique, we only applied the bone wax to inert materials and in small amounts. Moreover, in hip arthroplasty cases, the bone wax was covered by cement, providing an extra layer of protection. This approach differs from previous reports where bone wax was directly applied to the bone surface -14. We sThe application of bone wax as a protective barrier for screw heads in various orthopedic scenarios offers a simple and cost-effective method to facilitate future implant removal. By preventing bone growth or cement from engaging with the screw heads, this technique effectively reduces the risk of stripped heads and simplifies the removal process. Despite potential complications associated with bone wax, such as infection and foreign body reaction, the benefits of this technique outweigh the risks when used appropriately. Ultimately, the utilization of bone wax has the potential to save time, reduce surgical complexity, and assist orthopedic surgeons in facilitating future implant removal.


Objectives: Over 3 years of the COVID-19 pandemic, and intense societal and governmental response, a wealth of research has examined risk perceptions and public risk mitigation behaviours. The vast majority of this inquiry has focused on health risks. Nevertheless, as a “total social fact” influencing nearly every aspect of quotidian life, the pandemic engenders a wide range of risk perceptions.Methods:Via a survey  of representative samples of the general public in five European countries , we explore perceptions of a range of personal/public health, economic, and societal risks. We also investigate the effects of perceptions of official governmental risk communication in one’s country on risk perceptions and risk mitigation behaviours.Results: Structural equation modelling reveals that whilst perceptions of effective risk communication directly increase behaviours that mitigate COVID-19 health risks, these same perceptions indirectly decrease behaviour frequency via a mediated relationship with societal risk perceptions.Conclusion: The findings highlight the import of governmental authorities analysing and communicating about the range of risk perceptions citizens might have about a “total social fact” such as COVID-19. We ran the survey  through the online survey panel provider Qualtrics from 1 April—4 May 2021 in five European countries . The samAlthough there certainly were differences between the five countries in how members of the public perceived risks and responded to government communication, herein we focus on the similarities. Nuanced differences can be gleaned from Our analysis herein focuses on three primary groups of variables, beyond demographic covariates. First, our exogenous attitudinal items of interest all relate to public perceptions of official governmental risk communication about COVID-19: 1) perceptions of the extent to which information needs were met, 2) extent to which the communication was clear and understandable, and 3) consistency in government recommendations. Complete wording of all survey items mentioned in this article is available in the Our second primary group of variables measured risk perceptions of 15 outcomes due to COVID-19. Three of these related to personal health risks, three to public health risks, three to personal economic risks, and six to societal risks . Exploratory factor analyses and reliability scaling reveals that the 15 items pool well onto these four domains. We discuss specific items further in the results section.Our final set of key variables are behavioural outcomes. Because we are interested in the extent to which perceptions of official governmental risk communication shape actions that could mitigate the spread of COVID-19, we asked about frequency of two behaviours that were widely recommended by public health experts throughout the pandemic in each of the five countries in our study: “keeping the required social distance” and “getting tested when having symptoms.”via basic descriptive statistics, repeated measures ANOVAs to compare across risk perceptions, and bivariate correlations examining relationships between risk perceptions and risk mitigation behaviours. We answer the next three research questions is via a series of structural equation models (SEMs) that combine confirmatory factor analyses with structural regression pathways. We chose this approach due to our theoretically constructed latent variable of societal risk perceptions (composed of six measured variables) and the multi-stage mediated regression pathways needed to answer our questions. We chose to focus on the role societal risk perceptions, because the literature review revealed that research to date on COVID-19 risk perceptions has given the least attention to these risks. We explore our first research question We ran separate models for the respondents from each country in our study, and then one combined model for the full sample . We ran this set of six models for each of the two dependent variables of interest: respondents’ frequency of “keeping the required social distance” and “getting tested when having symptoms”; therefore, twelve SEMs in total. The direct pathway in In addition to the SEMs, we include some basic independent samples t-tests, bivariate correlations, chi-square tests on crosstabs, and ANOVAs to explore socio-demographic predictors of perceptions of risk communication and to assess the relationship between the multiple types of risk perceptions we measured in the survey.We measured four broad content foci for risk perceptions: personal health risk, public health risks, personal economic risks, and societal risks . Initial exploratory factor analyses revealed clear pooling of the fifteen measured items into the four aforementioned categories (see 2 effect size = 0.11). Perceptions of public health and societal risks on average fell between moderate and significant levels of risk; the lowest risk estimate was for more people dying in one’s country compared to elsewhere, and the highest estimate for the pandemic being hard on small and medium size businesses. The risk perceptions in 2 = 0.15).To examine our first research question, we first ran repeated measures ANOVAs to compare the means of the personal health and personal economic risk items , and theTo answer our second research question , we chose to include the risk perceptions that have been least discussed in academic literature on COVID-19. Initial exploratory correlations also revealed moderately strong negative correlations between perceptions of risk communication and societal risk perception—suggesting an intriguing relationship opposite to that of risk communication perceptions on health risk perceptions .Our subsequent analyses all rely on the “societal” sub-set of risk perceptions. In We ran twelve SEMs to assess our research questions. In the SEMs of the full survey sample ; nevertheless, the indirect is significant.The direct and indirect effects of beliefs about official authorities’ communication on COVID-19 mitigation behaviours mean that concurrently positive beliefs about communication are There are clear differences cross-nationally in the goodness-of-fit of the SEMs, and in the coefficient values and statistical significance of key structural regression pathways in the models. Norway, Switzerland, and the UK have good overall model fit, whereas Germany and Sweden do not [see notably the CFI ]; this aAlthough there are some variations in which pathways between variables are significant across the countries, the risk communication variables have fairly consistent effects in Norway and Switzerland on societal risk perceptions and on both of the COVID-19 mitigation behaviours. In Sweden, beliefs about risk communication are relevant for predicting societal risk perceptions, but not notably so for predicting COVID-19 mitigation action. The reverse is true in the UK and Germany—beliefs about risk communication are relevant for predicting COVID-19 mitigation actions, but not notably so for predicting societal risk perceptions.Importance of demographics varies more widely between countries. The effects of socio-demographics on societal risk perceptions and risk mitigation behaviours are reflected in Being male was the most common predictor cross-nationally of viewing governments as providing no useful information. Political orientation was the most common predictor of perceived consistency of governmental communication, although the direction of effect varied—right-leaning associated with greater consistency in Norway and the UK (governed by Conservative parties during April 2021 survey administration) and left-leaning in Sweden and Switzerland . Switzerland was governed by a coalition during survey administration, but Alain Berset, the Head of the Federal Department of Home Affairs—a leading spokesperson for the Swiss response to the pandemic, is a Social Democrat.via decreased societal risk perceptions.Our finding that most advances literature of risk perceptions and risk mitigation actions in relation to COVID-19 is the nuanced relationships in the mediated pathway from perceptions of risk communication to societal risk perceptions then to behaviours. Perceptions of good risk communication  directly increases risk mitigation behaviours. In other words, perceptions of good quality risk communication leads to support for sanctioning in response to the total social fact of COVID. Indirectly, however, such perceptions lessen risk mitigation behaviours, mediated In this latter instance, put in the concepts of the total social fact, the reduced risk perceptions seem to, indirectly, reduce perceptions of the external threat to certain valued objects, and thus perhaps necessitate less sanctioning. Additionally, the heavy focus on health in most risk communication on COVID from governmental authorities could be seen as reducing the “totality” of the total social fact, because there is not strong focus on it cutting across multiple domains of everyday life. Furthermore, thinking government communication is good quality could foster social norms to support mitigative actions , but socThe mediated indirect reduction in risk mitigation behaviours highlights the importance of providing attention in risk communication to risks other than solely health risks. One feasible interpretation of our data is that effective government communication about health risks crowded out other concerns. Societal risk perceptions, however, also associate positively with risk mitigation behaviours, suggesting that efforts to highlight those risks could be further beneficial for government entities seeking to promote risk mitigation actions. This is consistent with recent research revealing that social histories and myths concerning masking in societies have heavily influenced the political nature of relevant mitigation actions . DiscourBeyond seeking to influence specific behaviours, one might argue from a normative perspective that a key role of government is to provide its citizens with the resources to make informed decisions broadly, and that a more holistic treatment of risks from a “total social fact” such as COVID-19 would help facilitate such decision making . Social Our series of structural equation models reveal some overarching patterns, but also clear variation cross-nationally. Our core messages herein about relationships between governmental risk communication, societal risk perceptions, and risk mitigation behaviours apply most clearly in our Norway, Switzerland, and UK samples. For Germany and Sweden, additional empirical research and/or revised theoretical models are needed to better understand the weak relationships between these variables.In summary, our five-nation survey reveals the import of governmental authorities remaining cognisant of and vigilant to the range of risk perceptions citizens might have about a “total social fact” such as COVID-19. The pandemic, the societal response, and the ways in which individuals reacted touched nearly every aspect of quotidian life. We have not systematically analysed governmental risk communication; neither are we seeking to critique any governmental authority for neglecting specific risks. We are, rather, cautioning that risk communication attending exclusively to individual and/or public health risks might foster unintended consequences for risk mitigation behaviours. A total social fact could clearly impact upon a range of risks and concerns, extending well beyond the six societal risks included in our models. Yet, these provide a start for conversations on how to expand the suite of COVID-19 relevant risks.


We demonstrate that the values-based process objectives defined by the participant group are an effective tool for evaluating participatory success. This paper highlights that participatory approaches can be effective even in resource-constrained environments when the process is adapted to fit the decision-making context.Environmental flows (e-flows) management takes place within a complex social-ecological system, necessitating the involvement of diverse stakeholders and an appreciation of a range of perspectives and knowledge types. It is widely accepted that incorporating participatory methods into environmental flows decision-making will allow stakeholders to become meaningfully involved, improving potential solutions, and fostering social legitimacy. However, due to substantial structural barriers, implementing participatory approaches can be difficult for water managers. This paper assesses the effectiveness of an e-flows methodology that combines elements of structured decision-making and participatory modeling, whilst constrained by project resources. Three process-based objectives were identified by the group at the start of the process: improving transparency, knowledge exchange, and community ownership. We evaluated the success of the approach according to those objectives using semi-structured interviews and thematic analysis. In evaluating how well the participatory approach achieved the process objectives, we found that at least 80% of respondents expressed positive sentiment in every category ( Water is valued by people in many ways, ranging from the biophysical to the metaphysical  River in Victoria, Australia to evaluate the success of a participatory process. The participatory process integrated participatory modeling into a structured decision-making framework to contribute to the development of e-flows recommendations  River is a large, lowland river located within the Northern Victorian region of the Murray Darling Basin (MDB) in Australia Fig. . The KaiDue to the presence of several socially and culturally significant species , the Kaiela River has been identified as a high priority waterway (Victorian Waterway Management Strategy The stakeholder participants were recruited with guidance from the Goulburn-Broken Catchment Management Authority (GBCMA) and included four distinct stakeholder groups: local Indigenous organizations, community members, water agency representatives, and discipline-based researchers. Community member participants were recruited primarily from the GBCMA’s Environmental Water Advisory Group (EWAG). For the purposes of data analysis within this paper, Indigenous representatives from Yorta Yorta and Taungurung Aboriginal corporations were categorized as community members to protect their anonymity given the small participant group. Researcher stakeholders are members of the expert panel recruited to support the model development and provide a discipline-specific knowledge base for the development of flow recommendations. They participated in the workshop series as stakeholders and provided guidance for the development and quantification of supporting models. Water agency representatives from local, state, and federal levels included environmental management and infrastructure operators.Before the first workshop, we conducted semi-structured interviews with members of the stakeholder participant group. Semi-structured interviews are comprised of both specific and open-ended questions, which allows researchers to investigate subjective perspectives and experiences utilizing a consistent framework . Questions from the introductory interview targeted the participants’ pre-existing relationship with and knowledge of the catchment, including their experience with participatory engagement and decision making. The introductory interviews were used to create a baseline understanding of the participants’ knowledge and values.The workshop series was the primary mode of engaging with stakeholder participants for the e-flows assessment. While ideally these workshops would have occurred in person, the COVID pandemic and strict lockdown requirements in Australia mandated that two of the four workshops be held online. The meetings included a mix of formal presentations, open Q&A, and breakout groups focused on specific activities. A breakdown of the workshop series including participants can be seen in Table Three process objectives were elicited from and defined by stakeholders in Workshop 1 based on the objectives framework defined within the structured decision-making literature Keeney . ProcessExit interviews were conducted at the end of the project, after the final report and recommendations were submitted. These interviews took place over the phone or online and were digitally recorded. The same process was used to generate transcripts for these interviews. In total, 15 exit interviews were obtained: 5 community members, 5 water agency representatives, and 5 expert panel members. Not all participants from the introductory interviews participated in exit interviews. Exit interviews were more difficult to obtain once the project had ended.Exit interview questions were also semi-structured and participants were encouraged to elaborate on their answers. However, questions were designed to target the process objectives developed in the first workshop. The exit interview questions were clearly divided into four sections, each with 4–6 questions aimed at each process objective and a final section about their overall experience.Both introductory and exit interviews were analyzed using thematic analysis in NVIVO 12 Pro . Questions related to community ownership targeted the following themes identified during the introductory interview and workshops:Responses across these four themes indicate that the participatory approach used in the project supported community ownership and input legitimacy See Fig. . particiThere was a strong positive response from 13 of the 15 participants when asked whether they felt the workshops and project had helped foster relationships amongst stakeholders. Respondents across all stakeholder groups emphasized the importance of being present together and participating in conversations and noted that COVID restrictions impacted later stages of the project. Community members expressed that they felt their views had been included in the final report, though one noted that some technical aspects were difficult to engage with. Researchers were more likely to express neutral or mixed sentiments regarding community ownership, remarking that they felt they were unsure about this aspect of the project Table .Table 10Comfort expressing perspective and knowledgeAdequate opportunity for discussion/questionsCommunication of technical project elementsUse of “best available science”Questions related to knowledge exchange targeted the following themes identified during the introductory interview and workshops Fig. :Comfort All community members felt there was adequate opportunity to discuss the topics and issues with other stakeholders, while agency representatives and researchers had mixed responses. Three respondents noted the difficulty of discussing the project through the online platforms necessitated by the COVID pandemic. When asked whether the scientific elements were adequately communicated, all five community members responded positively. Most community members expressed that while they may not have completely understood all the technical material, they felt they understood the process on a conceptual level. This sentiment was further reflected by three water agency representatives, who noted that model development could be complex and difficult to understand. There was a sense that detailed understanding of technical elements is less important than understanding the process at a high level Table .Table 11Appropriate briefing and adequate updates during projectReport represents methodologyUnderstanding of e-flows decision makingUnderstanding of future implementation of e-flowsQuestions related to transparency and trust targeted the following themes identified during the introductory interview and workshops Fig. :Appropri“Do you have a good understanding of how these environmental flow recommendations will be applied in the future?” and “Has your understanding of environmental flows decision making improved?” Both expert panel (researcher) members and community members believed that future application and implementation was dependent on “politics”. They describe a lack of transparency around internal decision making at these higher organizational levels. Community members, who started with a relatively lower understanding of these institutional dynamics, reported that their understanding had improved through their engagement with the project. However, another community member said that while the CMA and local level was very clear and transparent, there is a lack of clarity around decision making beyond the local level (Table Negative responses regarding transparency arose primarily from two questions: el Table .Table 12When asked about their overall experience, 13 participants responded that they had a positive experience through their engagement with the project. All community members and water agency representatives expressed positive opinions about the process. General feedback about the project was positive with water agency representatives and community members expressing satisfaction with the process and final product. Researchers were divided, with three expressing that they had positive experiences and enjoyed interacting with other stakeholders and developing the models. The negative responses from researchers were motivated by the expert-opinion elicitation process, which was not considered part of the project’s participatory process.“more efficient, effective, and enjoyable… [than online]. You don’t chat with people over coffee, or over lunch. And often, that’s where the most interesting interactions occur” (Table We asked participants to reflect on how the project was impacted by COVID and whether the project team responded adequately. All stakeholder groups expressed disappointment that the number of in-person workshops had to be limited. Several participants expressed that online workshops were not conducive to free-flowing conversation and that digital spaces were more tiring, necessitating shorter workshops. The online format limits a participant’s ability to engage with others and makes it more difficult to focus on complex material. One researcher described face-to-face interactions as r” Table .Table 13There are three primary motivations for adopting participatory approaches: the normative, substantive, and instrumental Stirling . Within While community ownership was the idealized goal of the engagement process, it is an ambiguous and difficult-to-define objective (Lachapelle In our interviews, we saw major differences in how trust and transparency were articulated, reflecting divides between the stakeholder groups and their motivations. Researchers almost never mentioned the issue of trust unless explicitly asked. Trust is difficult to define, yet widely recognized as critical to building social legitimacy for e-flows (Stern and Coleman procedural trust, this is not the only kind of trust that needs to be considered within engagement (Stern and Coleman affinitive trust or the trust a participant has for an individual representative’s motivations and concerns. Affinitive trust is built through iterative interactions with individuals perceived as having integrity (Stern and Coleman While transparency and accountability build towards Communication and engagement are impacted by the scale of decision-making, and this is reflected in the trust community participants have for different institutional organizations (O’Donnell et al. Engaging stakeholder participants throughout the e-flows assessment resulted in significant differences in both the process and the outcomes compared to other traditional assessment approaches. Most importantly, how the project objectives were developed. Besides the inclusion of process-based objectives, which are not typically part of a traditional e-flows assessment, new fundamental ecological objectives were also selected. Driven by participant interest, the project included out-of-scope flows that would connect the floodplain to the river. Water representatives highlighted during interviews the inclusion of out-of-scope flows as beneficial to their overall planning and have used the results of the assessment to support negotiations aimed at relaxing flooding constraints. Participants were able to meaningfully influence decision-making within the project because water agency representatives were open to an exploratory process. This emphasizes the importance of including non-agency stakeholders in conversations around project scope and limitations. Allowing stakeholders to interrogate these boundaries can potentially reveal novel management strategies. Open, flexible processes that allow stakeholders to influence the decision-making process and substantially impact outcomes foster community acceptance of programs and build social legitimacy.Within this participatory process, we found that conceptualization of knowledge and participant roles in knowledge exchange varied between stakeholder groups. While community members and agency representatives valued community knowledge about the river, researchers were unlikely to express this sentiment. Furthermore, in introductory interviews, some researchers expressed hesitation about including community members in some aspects of the project, citing concerns that community members might not have adequate background or that the ecological objectives would not be a high priority. Given that researchers often have considerable experience with technical, top-down approaches to e-flows assessments, it will take time to shift the culture around how different types of knowledge are valued and expand the role of community participation (Rosendahl et al. While our analysis suggests that the participatory process was generally successful, several components of the project could have been improved. Due to a tight timeline, participant recruitment was largely drawn from the CMA’s pre-existing advisory group. While this ensured that most participants had a baseline knowledge of e-flows, there is also the possibility that the group was biased, and some perspectives were not represented. We would recommend dedicating more time to recruitment and casting a wide net to find potential participants. This may require additional resources as project team members would need to raise awareness around the project and have conversations within the community. Within this project, one team member was responsible for designing the participatory process and conducting all participant interviews. Upon project completion, agency representatives noted that this work was instrumental for achieving a high level of participation. Many projects would not have the staff capacity to dedicate to this process. Designing and implementing participatory approaches require specific skills and resources but can be effectively included through adequate resourcing and planning.Working as a group to define the process-based objectives created the space to explore and articulate shared values and establish a baseline for communication amongst stakeholders. It also gave us a valuable tool for measuring participatory effectiveness with an internally defined metric. These objectives were clearly defined, communicated within the group, and presented in intermediate and final reports. Self-reflective participatory objectives are not a standard procedure in traditional e-flows assessments. By contrast, using process-based objectives to evaluate the success of an e-flows project enables self-reflective learning and can improve engagement strategies over time. It is essential that these objectives are defined by the participant group themselves and reflect the project context and group values. We recommend working collaboratively to clearly articulate these objectives, including definitions and methods of measurement.Our project demonstrates that participation can be effective, even when subject to the budget and time constraints that typically limit the potential for meaningful stakeholder engagement. In our evaluation, respondents across stakeholder groups expressed positive sentiments regarding project outcomes for all three process-based objectives. Agencies and e-flows practitioners need to be aware of the differing motivations for a participant’s involvement in a project. If community stakeholders want to have a substantive impact on a project, then agencies need to facilitate these kinds of opportunities and approach participation with flexibility and openness. Collaborative development of process-based objectives opens these kinds of discussions and provides a self-reflective evaluation tool that can improve engagement over time. E-flows programs that include intentional, reflective participation approaches will allow stakeholders to build trust, explore new management strategies, and ultimately foster social legitimacy.Supplementary materials


The COVID pandemic exposed the critical role T cells play in initial immunity, the establishment and maintenance of long term protection, and of durable responsiveness against novel viral variants. A growing body of evidence indicates that adding measures of cellular immunity will fill an important knowledge gap in vaccine clinical trials, likely leading to improvements in the effectiveness of the next generation vaccines against current and emerging variants. In depth cellular immune monitoring in Phase II trials, particularly for high risk populations such as the elderly or immune compromised, should result in better understanding of the dynamics and requirements for establishing effective long term protection. Such analyses can result in cellular immunity correlates that can then be deployed in Phase III studies using appropriate, scalable technologies. Measures of cellular immunity are less established than antibodies as correlates of clinical immunity, and some misconceptions persist about cellular immune monitoring usefulness, cost, complexity, feasibility, and scalability. We outline the currently available cellular immunity assays, review their readiness for use in clinical trials, their logistical requirements, and the type of information each assay generates. The objective is to provide a reliable source of information that could be leveraged to develop a rational approach for comprehensive immune monitoring during vaccine development. Human immune responses include a range of cellular and soluble factors. To optimally assess immune responsiveness an integrated approach that measures both cellular and soluble factors is required. However, only the antibody response was measured at scale throughout clinical development of the first generation COVID vaccines, in particular titers of neutralizing antibodies and total immunoglobulin G (IgG). Cellular immunity was partially characterized, and only at the early-stage trials –9. This in vitro antigen-specific stimulation prior to sample analysis, have established standardization procedures, require the same level of Clinical Laboratory Improvement Amendments (CLIA) complexity, and can be rapidly scaled with existing laboratory infrastructure. ELISpot has been the primary cellular immunology assay deployed in COVID vaccine clinical trials, reflected in its proportional increased use since 2020 . There is no further characterization of the responding cells. ICS is a multi-parametric flow cytometry assay which can also measure the frequency of cytokine-secreting cells, but also determines the responding cells’ lineages, activation states, and proportion of multi-functional responders, which defined as cells that make more than one cytokine following antigen-specific stimulation and which are highly desirable after vaccination . AIM assBox 1ELISpot: Enzyme-linked immune absorbent spot assay. A method for measuring cytokines secreted by activated cells following antigen stimulation  compatible formats. changes . The assTCR sequencing: T cell receptor sequencing assay. A version of scRNA analysis focused on measurement of expressed T cell receptor genes at the single cell level  and T cell receptor (TCR) sequencing have seen a rapid rate of adoption in clinical trials in the last 5 years Box 1. SA comparison of the cellular immunity assays discussed here is presented in Multiple methods are available for high resolution immune monitoring Figure 2The type of information provided by the cellular immunity assays is going to be essential for evaluation of next generation COVID vaccines, especially to define critically needed correlates associated with duration of protection . Phase IELISpot provides the least in-depth characterization of the cellular immune response, whereas scRNA would provide the most in depth characterization. Multi-parametric flow cytometry as represented by the ICS and AIM assays falls somewhere in between, but nonetheless is still able to provide the critical information needed to evaluate the type of cellular immunity induced by the vaccination, as well as assess cross reactivity and risk of adverse effects. Judging from the overall operational costs involved, including sample procurement, processing, laboratory operations and subsequent data analysis, multi-parametric flow cytometry may be the most cost-effective application for the monitoring of vaccination-induced cellular immunity and identification of cellular immunity correlates.ELISpot and the flow cytometry assays need live cells, which is a major source of logistical concern and also a source of data variance. Inconsistent cell preparations not only affect the yield of PBMC per tube of blood collected, but can also affect responsiveness in subsequent testing. Effective deployment of any of these cellular assays requires the establishment of properly trained sample collection and processing sites. Alternatives to PBMC preparation and cryopreservation such as whole blood ICS, whereby the blood is directly stimulated with the required antigens  could heSeveral studies have demonstrated that relatively limited number of subjects  are sufficient to demonstrate significant differences between vaccinated/treated groups and placebo control or baseline using ELISpot or ICS , 54, 55.The incremental costs of these studies could be argued that they will discourage investment in vaccine development or restrict innovative approaches to vaccine development. A mitigating strategy might be to simply require the acquisition and retention of appropriate clinical samples during early developmental efforts while allowing deferral of the investment analyzing those samples until a later phase of clinical trials but prior to FDA approval. We would also recommend a deliberately agile process to determining what clinical samples should be retained and what subsequent assays should be performed as we learn more about which studies are most relevant in general, and more specifically which assays are more relevant to different infectious agents. A collateral benefit of systematically including T cell assays in the assessment of vaccines is that focusing more attention on these assays will likely accelerate simplification and cost-reduction for these tests and generate novel technical approaches to obtain comparable information.Finally, it is important to note an emerging, but completely different technologic approach to tracking individual immune response - wearable biosensors, also known as wearable biometric devices Box 1. BCOVID has brought increased public attention to the consequences of insufficient immune protection, and the limitations of insufficiently monitoring cellular immunity. We have an opportunity to integrate cellular immunity and antibody testing to inform more effective vaccination strategies and develop immune monitoring tools that should be adopted for every therapy that involves the immune system. A monolithic, homogenized approach to understanding vaccine effectiveness at a total population level overlooks the opportunity to better address the specific needs of large and growing cohorts of elderly and immunocompromised sub-populations. We can now better leverage the abundance of biotech innovation to extend our ability to understand vaccine effectiveness in different subcohorts, which itself is the first of many steps towards a future of more personalized vaccinology.The original contributions presented in the study are included in the article/supplementary materials. Further inquiries can be directed to the corresponding author.All authors listed have made a substantial, direct, and intellectual contribution to the work, and approved it for publication.


Recovery housing is an important resource for those in recovery from substance use disorders. Unfortunately, we know little about its relationship to key community health risk and protective factors, potentially limiting the role it could play as a broader health resource. Leveraging county-level data on recovery residences from the National Study of Treatment and Addiction Recovery Residences (NSTARR), this study used multilevel modeling to examine Community COVID Vulnerability Index (CCVI) scores as well as availability of COVID testing and vaccination sites in relation to recovery housing. CCVI composite scores were positively associated with recovery housing availability. Analyses using CCVI thematic sub-scores found that population density and number of churches were positively associated with recovery housing availability, while epidemiological factors and healthcare system factors were negatively associated with recovery housing availability. In counties with recovery housing, there also was a positive association between CCVI and both COVID testing and vaccination availability. Recovery residences tend to be located in areas of high COVID vulnerability, reflecting effective targeting in areas with higher population density, more housing risk factors, and other high-risk environments and signaling a key point of contact to address broader health issues among those in recovery from substance use disorders. Early in the coronavirus disease (COVID) pandemic, persons with substance use disorders (SUD) were identified to be at increased risk for SARS-CoV-2 infection , 2. HoweN = 1,492) found that equivalently large majorities of women and men reported that the COVID pandemic had not affected their recovery at all , a shorter length of time in recovery was associated with increased risk of return to use during the pandemic among women [In addition to risk for COVID infection, the pandemic also may have increased risk for return to use among those in recovery from SUD, as general population studies have highlighted increased levels of alcohol  and subsng women . Changesng women , 17.Safe and supportive housing is critical to recovery from SUD . RecoverBy providing housing and other support, recovery residences are a key component of comprehensive, recovery-oriented systems of care –26; theyTo better understand these risks, this study examines community-level COVID vulnerability in relation to US recovery residences. Specifically, we explore whether COVID vulnerability is associated with recovery housing availability and density at the county-level as well as which factors driving COVID vulnerability are most strongly related to availability and density. To better understand the availability of health resources in the communities where recovery residences are located, we also present data on availability and accessibility of COVID testing and vaccination services and explore how this is related to overall COVID vulnerability and the factors that comprise it.Using geolocated data on recovery residences from the National Study of Treatment and Addiction Recovery Residences (NSTARR) database and the Community COVID Vulnerability Index (CCVI), this secondary data analysis study used multilevel modeling to examine whether the CCVI and other community resources were related to recovery housing availability and density. In counties with at least one recovery residence, we also used multilevel modeling to examine whether the CCVI and other community resources were associated with density of and mean distances of recovery housing from COVID testing and vaccination sites at the county level using US Census Bureau’s 2019 County Business Patterns data. The NSTARR COVID Supplement study was reviewed, approved, and monitored by the Public Health Institute Institutional Review Board, which determined it to be exempt from review under Category 2 of 45 CFR 46.104. Details pertaining to our data sources and analytic approach are provided below.Recovery housing availability was operationalized as an indicator of any recovery housing (vs. none) in a given county and recovery housing density was a count of recovery residences in each county that had any recovery housing.Recovery housing data came from the NSTARR project, which collected information on locations of recovery residences during 2020. We used an NSTARR database export from spring 2021 containing information on 10,358 residences operated by 3,628 providers in all 50 states . RecoverThe 2020 CCVI is a composite measure of seven social determinants of health, encompassing modified themes from the Centers for Disease Control and Prevention’s Social Vulnerability Index in combination with COVID risk factors to identify communities in need of additional support during the COVID pandemic. There is an overall score, as well as thematic scores for each domain . Each county was ranked on a scale from 0 = least vulnerable to 1 = most vulnerable on each of 40 variables, which then were summed by theme, with scores ranging from 0–1 , 33.. The crowd-sourced data were collected between March 2020 and November 2021. All data collection, cleaning, and validation tasks were completed by a group of over 300 GISCorps volunteers. We removed duplicates and selected non-private testing  and vaccination  sites for our project, as these would be accessible to people regardless of health insurance provider or coverage status. The data on testing and vaccination sites came as a geocoded shapefile.As has been used in other studies examining access to COVID testing and vaccination sites –36 we utTo supplement the testing and vaccination site data, we appended county-level counts of community resources from the US Census Bureau’s 2019 County Business Patterns dataset. We included count data on two specific types of healthcare providers likely to provide testing and vaccination for COVID: physicians’ offices  as well as pharmacies and drug stores . We also included schools and churches because public testing sites and vaccination efforts often were conducted at elementary and secondary schools , community colleges  and churches  to expand the reach of public health efforts during the pandemic , 38.Locations of all recovery residences were geocoded using ArcGIS , with 86N = 3,142 counties) and basic count variable of recovery housing density were specified as outcomes in separate regression models, with the density models limited to counties with at least one recovery residence . The series of multilevel regression models began with unadjusted models using the continuous overall CCVI composite score as the independent variable for both outcomes. Adjusted models then simultaneously tested all seven CCVI theme domain scores and included density of community resources as continuous independent variables. All models included a fixed effect for state and adjustment of the standard errors to account for clustering at the state level. Another set of negative binomial regression models followed the same series, using as the outcome variable the county-level counts of testing and vaccination sites in those counties with at least one recovery residence . These were followed by a final set of linear regression models using mean distances to testing and vaccination sites from recovery housing . All regressions were run in Stata 16.1 [Bivariate t-tests for preliminary analyses compared COVID vulnerability scores, density of testing and vaccination sites, and other community resources in counties with and without recovery housing. Associations between COVID vulnerability and community resources with recovery housing availability and recovery housing density were examined using multilevel logistic and negative binomial regression models. The dichotomous indicator of recovery housing availability  and recovery housing density in counties with at least one recovery residence . When the separate thematic sub-scores were entered simultaneously along with availability of community resources, population density and the number of churches were positively associated with both measures of recovery housing availability, while epidemiological factors and healthcare system factors were negatively associated with both measures of recovery housing availability. Housing risk factors and high-risk environments were positively associated with presence of any recovery housing. The minority status and language score was positively associated with recovery housing density, and the number of drug stores and pharmacies was negatively associated with recovery housing density. No associations were observed between recovery housing with the socioeconomic status score, the number of physicians’ offices, or the number of educational institutions (neither elementary and secondary schools nor community colleges) in the county.Figure p < 0.001). The composite scores also were negatively associated with average distance to testing and vaccination sites , with shorter mean distances indicative of greater accessibility of COVID testing and vaccination.Figure When the separate thematic sub-scores and availability of community resources were entered simultaneously, population density, socioeconomic status, minority status and language, and the number of churches were positively associated with density of both testing and vaccination sites, while healthcare system factors and high-risk environments were negatively associated with density of both testing and vaccination sites. Additionally, the score for epidemiological factors was negatively associated with density of non-private testing sites, and the number of physicians’ offices was negatively associated with density of non-private vaccination sites.In the final set of models including the separate thematic sub-scores and availability of community resources, population density also was strongly negatively associated with both average distance to nearest testing site and distance to nearest vaccination site. Housing type and household composition score was negatively associated with average distance to nearest testing site, while healthcare system factors and the number of churches were associated with greater average distance to COVID testing. Finally, in contrast to the model for accessibility of COVID testing, when accounting for population density, healthcare system factors also were negatively associated with distance to nearest vaccination site.Using a national database on recovery housing, this study documented that indicators of higher vulnerability to COVID were positively associated with recovery housing availability and the number of recovery residences in counties with at least one recovery residence. Although our findings underscore potential risks faced by individuals living in recovery housing, they also highlight effective targeting of recovery housing in areas that have higher population density, more housing risk factors and other high-risk environments related to the spread of an infectious disease such as COVID. Some research suggests that rates of COVID infection and mortality may have been lower among recovery housing residents than rates in the general population . While mIn addition to examining the association between overall COVID vulnerability and the availability of recovery housing, we also examined additional components of the CCVI corresponding to key SDOH, as well as other community resources. In bivariate analyses, counties with recovery housing were found to have lower epidemiological and healthcare system factors scores but more resources, like physicians’ offices, drug stores, educational settings, and churches. Prior work examining availability of recovery housing found that residences were more likely to be in urban areas , and it While it is heartening that recovery housing can be found in more densely populated counties, greater attention should be paid to ensuring that recovery housing can also be accessed by individuals in less densely populated areas, which may have fewer recovery and healthcare services . FurtherThis study also examined factors associated with the COVID response, using geocoded data on counts and mean distances to COVID testing and vaccination sites in counties with recovery housing. In these areas, greater COVID vulnerability was associated with greater numbers of testing and vaccination sites, as well as shorter distances to these resources. Again, this finding highlights potentially beneficial targeting of recovery housing to areas with greatest need, but this also could be heavily influenced by the increased availability of recovery housing in densely populated areas. Indeed, in models including the CCVI’s separate thematic sub-scores and availability of community resources, population density was negatively associated with both average distance to nearest testing site and distance to nearest vaccination site, suggesting greater accessibility in more densely populated areas. Further, when accounting for population density, healthcare system factors were positively associated with distance to nearest testing site, yet again suggesting that recovery housing may be relegated to under-resourced neighborhoods within urban areas. Density of churches also was positively associated with density of testing and vaccination sites; as such, churches may be an important, albeit less-traditional, resource to consider when addressing spread of infectious diseases among communities with recovery housing.This study represents the first examination of COVID vulnerability and recovery housing across the US, but a number of key limitations to this work should be noted. Information on recovery residences came from a database of over ten thousand residences across the country, but this information was collected in 2020 and may not represent an exact count of all residences due to underreporting and inclusion of residences that may have subsequently closed . FurtherRecovery residences tend to be located in areas highly vulnerable to COVID. While this may present a higher risk for people in recovery, these communities could benefit from the support and resources provided by recovery residences, which may help offset individual-level COVID risks. Additionally, recovery housing residents may benefit from proximity to local COVID testing and vaccination resources, particularly those in more densely populated areas with higher socioeconomic vulnerability and more racial and ethnic diversity. Future research should explore the extent to which residents in recovery residences located in vulnerable areas were affected by the COVID pandemic, as well as the role of supportive community resources and the relative contributions of individual measures of key SODH in addressing the needs of individuals in recovery housing.