It is the most common of the hemophilias, with an incidence of 1 in 5,000 male births, and about two-thirds have severe disease.4 However, the costs and burdens of prophylaxis with factor replacement are high, and it does not maintain patients at normal factor VIII levels. Furthermore, some patients with severe hemophilia A who receive factor VIII concentrates develop neutralizing antibodies known as "inhibitors,"5 rendering factor VIII ineffective for prophylaxis and on-demand treatment in these patients.The use of factor VIII concentrates as on-demand treatment and prophylaxis has dramatically improved the management and clinical course of patients with hemophilia A. To reduce bleeding risk, patients with severe hemophilia A typically receive factor VIII concentrate intravenously multiple times per week.6 Emicizumab is administered subcutaneously and may be dosed weekly every 2 weeks or every 4 weeks.Emicizumab , a bispecific monoclonal antibody that targets activated factor IX and factor X, the role normally played by activated factor VIII in the clotting cascade, was approved by the US Food and Drug Administration (FDA) as a prophylactic treatment for hemophilia A patients who have inhibitors to factor VIII in 2017 and later in 2018 for patients without inhibitors.7Valoctocogene roxaparvovec  is an investigational gene therapy for hemophilia A that uses adeno-associated virus serotype 5 (AAV5) vector, which carries a liver-directed gene that results in factor VIII production. BioMarin submitted a biologics license application for valoctocogene roxaparvovec to the FDA in December 2019 and received a Complete Response Letter (CRL) rejecting approval in August 2020, specifying the need for longer-term results to substantiate the duration of benefit.https://icer.org/wp-content/uploads/2020/10/ICER_Hemophilia-A_Final-Report_112020.pdf.The Institute for Clinical and Economic Review (ICER) conducted a systematic literature review and cost-effectiveness analysis to evaluate the health and economic outcomes of emicizumab and valoctocogene roxaparvovec. All results related to valoctocogene roxaparvovec in the ICER review were considered highly preliminary given the FDA CRL determination. Here, we present the summary of our findings and highlight the policy discussion with key stakeholders held at a public meeting of the New England Comparative Effectiveness Public Advisory Council (CEPAC) on October 30, 2020. The detailed report is available on the ICER website: We did not identify any trial directly comparing emicizumab and valoctocogene roxaparvovec to each other. Because of major differences in study design and study characteristics, we evaluated the 2 interventions separately and did not perform a quantitative indirect comparison through network meta-analysis (NMA).8 HAVEN 3 enrolled patients aged 12 years and older with severe hemophilia without factor VIII inhibitors; 89 who had not been on prophylaxis were randomized to receive open label emicizumab or no prophylaxis, and 63 who had been on factor VIII prophylaxis were treated with emicizumab and compared in a before/after methodology. We identified 1 randomized trial of factor VIII vs no prophylaxis (SPINART)9 that was sufficiently similar to HAVEN 3 to allow us to conduct an NMA of emicizumab vs factor VIII prophylaxis.The key trial for emicizumab was HAVEN 3, a randomized trial that had a primary outcome of annualized bleeding rate (ABR) for treated bleeds.10 In this study, among 39 children without inhibitors, all of whom had been receiving factor VIII prophylaxis, fewer treated bleeds were observed in the 6 months after initiating emicizumab  compared with the pre-emicizumab period .NMA results showed a nonsignificant lower rate of treated bleeds  and treated joint bleeds  with emicizumab prophylaxis compared with factor VIII prophylaxis. The before/after comparison conducted in HAVEN 3 showed a 68% reduction in treated bleeds with emicizumab prophylaxis compared with the pre-emicizumab period when patients were on factor VIII prophylaxis . A benefit of reduced bleeding was also supported by 1 observational study conducted in patients with a median age of 8.6 years.The most common treatment-related adverse event (AE) with emicizumab was injection site reaction, which occurred in 25% of patients on emicizumab prophylaxis in HAVEN 3. Similar patterns of AEs were observed in other emicizumab trials in this population, with very few serious AEs and those that occurred deemed not to be related to emicizumab.16 neither of which had a control arm. The key phase 1/2 multiyear study enrolled 15 adults with severe hemophilia A without inhibitors, of which 7 received the 6x1013 vg/kg dose anticipated to be used clinically. All 7 participants achieved the prespecified primary endpoint of factor VIII activity levels of 5 IU/dL or more at week 16. A nonhemophilic range is considered to be factor VIII levels > 40 IU/dl. At the end of 1 year, the mean factor VIII activity level in the 7 participants was 64 IU/dl .Evidence to inform our assessment of valoctocogene roxaparvovec gene therapy was derived from a single phase 1/2 trial and 1 ongoing phase 3 trial,However, over the course of the second to the fourth year of follow-up, factor VIII levels decreased in all participants. Mean factor VIII expression dropped to 36 IU/dl in year 2, 33 IU/dl in year 3, and 24 IU/dl in year 4. At the end of the first year, 6 of the 7 participants were in the nonhemophilic range, and 1 was in the mild hemophilic range (> 5 IU/dl). By the fourth year, only 1 participant was still in the nonhemophilic range; 4 participants were in the mild hemophilic range, 1 participant was in the moderate hemophilic range (1-5 IU/dl), and 1 participant was back in the severe hemophilic range (< 1 IU/dl). The mean ABR for treated bleeds in the 7 participants dropped from a baseline of 16.3 events per year to a cumulative mean of 0.8 per year after 4 years of follow-up, representing a 95% reduction.Although only limited data are available, valoctocogene roxaparvovec did not appear to be as successful in the ongoing phase 3 trial. Of the 16 participants who had reached 26 weeks at the time of an interim analysis, only 7 had achieved the prespecified factor VIII levels of 40 IU/dl or greater.The most common treatment-related AE in the phase 1/2 study was elevation of the alanine aminotransferase level, a marker of liver inflammation, occurring in 86% of patients. All participants in the study developed anti-AAV5 antibodies.For emicizumab, the best randomized control trial (RCT) evidence of factor VIII prophylaxis that was most comparable to the emicizumab trial used doses of factor VIII prophylaxis lower than typically used today in the United States. Additionally, the effect of emicizumab on inhibitor development are currently unknown. Finally, RCT evidence may overestimate adherence to a burdensome therapy such as factor VIII, and this could incorrectly characterize the relative benefits of emicizumab vs factor VIII prophylaxis in the real world.13 vg/kg. Second, the duration of follow-up is currently limited, and factor VIII levels are declining over time, leading to uncertainties in the duration of benefit. Third, interim data from the phase 3 trial suggest lower rates of success in achieving factor VIII. Fourth, there is uncertainty about the long-term safety of this intervention, and the development of anti-AAV5 antibodies may preclude the ability for patients to take a subsequent gene therapy should a more effective option emerge.The evidence base for valoctocogene roxaparvovec has many limitations that have created uncertainties. First, there are currently very few patients studied in the likely dose of 6x1017 Both models used a lifetime time horizon, and costs and outcomes were discounted at 3% per year.We developed 2 de novo decision analytic models for patients with hemophilia A without inhibitors to factor VIII. In the first model, we evaluated the cost-effectiveness of emicizumab vs factor VIII in patients of all ages eligible for prophylactic therapy. The second model evaluated the cost-effectiveness of valoctocogene roxaparvovec relative to factor VIII only in adult patients with severe hemophilia. The second model used the adapted methods under the ICER ultra-rare disease framework, but we selected only the health care sector perspective as the base case, given that the societal perspective was not substantially different.18 Upon reaching a PS of 28, the base-case model assumed that patients had joint replacement surgery and return to a PS of 1. Transitions through the PS states in the models were based on the expected frequency of joint bleeds associated with the treatments and subsequent expected increases in the PS. In each cycle, the expected number of treated nontarget joint bleeds and treated target joint bleeds across treatments were modeled along with related costs and impacts on patient utilities.Both models focused on acute bleeds as the best available outcome measure from trials that could be translated into changes in quality of life through their relationship to long-term joint damage caused by joint bleeds and the potential need for joint replacement surgery. Joint damage was measured using Pettersson scores (PS) that ranged from 0 to 28 and increased with joint bleeds.19 For factor VIII, we used doses consistent with current clinical practice. As such, we opted to use bleed rates for factor VIII from a recently published study that included self-reported bleed rates from patients with severe hemophilia A or B being treated in US hemophilia treatment centers.20 We viewed this rate as an evidence-based lower bound of bleed rates associated with factor VIII at currently representative doses.The models were informed by key clinical trials, real-world evidence, previous relevant economic models, other published studies on hemophilia A, and stakeholder input. Dosing levels and efficacy for emicizumab and valoctocogene roxaparvovec were determined from the key clinical trials previously described. Treated bleed rates for valoctocogene roxaparvovec were modeled based on projected factor levels across time.https://icer.org/wp-content/uploads/2020/10/ICER_Hemophilia-A_Final-Report_112020.pdf.The outcomes of interest included the cost per quality-adjusted life-year (QALY) gained, cost per life-year (LY) gained, and cost per treated bleed avoided. Cost per equal value of a life-year gained (evLYG) was not calculated separately given that without a mortality benefit the QALY and evLYG would be identical. Full details on ICER’s cost-effectiveness analysis and model are available on ICER’s website at As shown in 21As previously noted, the FDA issued a CRL for valoctocogene roxaparvovec, so our analysis of this intervention is deemed preliminary. The bleed rates for valoctocogene roxaparvovec were based on a very small number of patients and had to be projected over time. As already noted, dosing levels and efficacy for factor VIII were taken from patients in US treatment centers, while those for emicizumab and valoctocogene roxaparvovec were from clinical trials. If those doses or efficacies are substantially different in practice, it could change the results. Finally, we also did not incorporate inhibitor development into the model, since we received conflicting clinical opinions about which regimen would lead to more inhibitor development, and it has already been shown that emicizumab is a dominant treatment for patients with inhibitors.The New England CEPAC is one of the independent appraisal committees convened by ICER to engage in the public deliberation of the evidence on clinical and cost-effectiveness of health care interventions. The New England CEPAC comprises medical evidence experts, including practicing clinicians, methodologists, and leaders in patient engagement and advocacy. Their deliberation includes input from clinical experts and patient representatives specific to the condition under review, as well as formal comments from manufacturers and the public. A policy roundtable concludes each meeting during which representatives from insurers and manufacturers join clinical experts and patient representatives to discuss how best to apply the findings of the evidence to clinical practice, insurance coverage, and pricing negotiations.The ICER report on emicizumab and valoctocogene roxaparvovec for hemophilia A was the subject of a New England CEPAC meeting on October 30, 2020. Following the discussion, the CEPAC members deliberated on key questions raised by ICER’s report. The panel voted 15 to 0 that the evidence was adequate to demonstrate a greater net health benefit of emicizumab over prophylaxis with factor VIII. The panel did not vote on the net health benefit of valoctocogene roxaparvovec due to the CRL issued by the FDA.The CEPAC panel also voted on "other potential benefits" and "contextual considerations" as part of a process intended to signal to policymakers whether there are important considerations when making judgments about long-term value for money not adequately captured in analyses of clinical and/or cost-effectiveness. The results of these votes are shown in As described in ICER’s Value Assessment Framework, questions on long-term value for money are subject to a value vote when incremental cost-effectiveness ratios for the interventions of interest are between $50,000 and $175,000 per QALY in the primary base-case analysis. Because the FDA issued a CRL for valoctocogene roxaparvovec and the results of cost-effectiveness modeling showed emicizumab to be cost saving and health enhancing at its current price, no votes were held on the long-term value of money for either treatment.https://icer.org/wp-content/uploads/2020/10/ICER_Hemophilia-A_Final-Report_112020.pdf. Several key policy recommendations for valoctocogene roxaparvovec and emicizumab are as follows:The policy roundtable discussion explored how best to translate the evidence and additional considerations into clinical practice and pricing and insurance coverage policies. The full set of policy recommendations can be found in the Final Evidence Report on the ICER website: Pricing of factor VIII in the United States is far higher than in other developed countries and represents a failure of competition. The price is far too high, even in light of factor VIII’s substantial benefits for patients; this pricing structure creates financial toxicity for patients, their families, and the health systems and builds a platform for pricing of treatments such as emicizumab and for potential cures such as gene therapy that will only exacerbate these problems.Trials of gene therapies for hemophilia need to be long enough to assess whether the benefits are durable enough to outweigh the risks, particularly since patients may be unlikely to be able to receive a second gene therapy using the same viral vector.Payers should explore innovative approaches to covering high-impact single-time therapies such as gene therapies for hemophilia.Considering the evidence of equivalent to improved comparative effectiveness, relative convenience, and lower overall cost, emicizumab will be the preferred agent for prophylaxis for many patients. Payers should ensure appropriate access to emicizumab and may wish to share information with clinicians and patients regarding its potential advantages over factor VIII prophylaxis.


TMAC Savings of 90% for Heartburn Drugs in the United States Therapeutic maximum allowable cost (MAC) is a managed care intervention that establishes a defined benefit dollar amount per therapeutic procedure or indication. TMAC can be established for any medical procedure  or any pharmacological indication . Heartburn is ideal for TMAC, particularly since there is ample evidence that the PPIs are therapeutically indistinguishable. The potential cost savings that can be realized from a TMAC intervention for PPIs can be estimated from the data in Table 2, for community and mail-service pharmacy claims for the 90-day period ended April 30, 2006. The average price per standardized 30-day supply, excluding manufacturer rebates, varied by about 7-fold, from $26 for omeprazole OTC to $179 for brand-name omeprazole. These are the actual prices for average daily dosing; i.e., the average drug cost per day and cost per 30-day supply combine all dosages and strengths of the drugs as actually used and paid by plan sponsors and as copayments by drug plan beneficiaries. A small self-insured employer in the United States implemented a TMAC program for heartburn drugs on January 1, 2006. 9 This employer had approximately 300 employees and 650 total beneficiaries in its drug benefit plan in the month of implementation of the TMAC program. The defined benefit amount was $0.67 per day ($20 per month) of heartburn drug therapy. There were no prior authorizations or other exceptions permitted for this defined benefit amount. Actual drug utilization and prices for this employer group for the first 4.5 months of 2005 through May 15 are shown in Figure 1; the member costshare amounts are shown to model the expected outcomes when members used each of the drugs after the TMAC implementation. Motivation for this employer to implement the TMAC program arose from the large proportion (15.7%) of total drug benefit spending attributable to the heartburn drugs, nearly twice the average.10 Operationally, the Medi-Span Generic Product Indicator (GPI) was used to measure preperiod and postperiod costs and administer the TMAC intervention, using the programming logic, GPI begins with 492. This GPI captures all of the PPIs and all of the histamine-2 blockers . As illustrated in Figure 1, the member cost-share is $0 for any heartburn drug that has an allowed charge of $20 or less per 30-day supply .


Although allogeneic hematopoietic stem cell transplantation (HCT) can be a curative therapy for hematologic disorders, it is associated with treatment-related complications and losses in cardiorespiratory fitness and physical function. High-intensity interval training (HIIT) may be a practical way to rapidly improve cardiorespiratory fitness and physical function in the weeks prior to HCT. The primary aim of this study was to assess the feasibility of implementing a pre-HCT home-based HIIT intervention. The secondary aim was to evaluate pre to post changes in cardiorespiratory fitness and physical function following the intervention.2peak) and physical function  between baseline and sign-off. Pearson correlations were used to determine the relationship between intervention length and changes in cardiorespiratory fitness or functional measures.This was a single-arm pilot study with patients who were scheduled to undergo allogeneic HCT within six months. Patients were instructed to complete three 30-minute home-based HIIT sessions/week between the time of study enrollment and sign-off for HCT. Sessions consisted of a 5-minute warm-up, 10 high and low intervals performed for one minute each, and a 5-minute cool-down. Prescribed target heart rates (HR) for the high- and low-intensity intervals were 80–90% and 50–60% of HR reserve, respectively. Heart rates during HIIT were captured via an Apple Watch and were remotely monitored. Feasibility was assessed via retention, session adherence, and adherence to prescribed interval number and intensities. Paired t-tests were used to compare changes in fitness . 30-second sit to stand and SPPB chair stand scores significantly improved in adherent participants. Improvements in 30-second sit to stand (13.8±1.5 to 18.3±3.3 seconds) and 6MWT (514.4±43.2 to 564.6±19.3) exceeded minimal clinically important improvements established in other chronic disease populations, representing the minimum improvement considered meaningful to patients.Thirteen patients (58.8±11.6 years) participated in the study, and nine (69.2%) recorded their training sessions throughout the study. Median session adherence for those nine participants was 100% (IQR: 87–107). Adherence to intervals was 92% and participants met or exceeded prescribed high-intensity HR on 68.8±34.8% of intervals. VOFindings demonstrate that implementing a pre-HCT home-based remotely monitored HIIT program is feasible and may provide benefits to cardiorespiratory fitness and physical function. Allogeneic hematopoietic stem cell transplantation (HCT) is a treatment approach for a variety of malignant and non-malignant hematologic disorders . The intPoor cardiorespiratory fitness has been associated with a longer hospital stay during HCT, greater number of hospitalized days before Day 100, and higher risk for mortality . Lower fHigh-intensity interval training (HIIT) is a type of training that consists of repeated short bouts of exercise performed at or close to maximal levels of exertion interspersed by a low-intensity recovery phase . In addiThis was a single-arm pilot study to evaluate the feasibility of a pre-HCT, mobile-health supported, home-based, HIIT program for patients prior to receiving allogeneic HCT. The trial was registered with clinicaltraials.gov (NCT03823651). Participants provided written informed consent, which was approved by the Duke Health Institutional Review Board (IRB # Pro00092963). Following consent, participants completed a demographics questionnaire and baseline assessments at the Duke Adult Bone Marrow Transplant (ABMT) clinic, followed by an introductory HIIT session. After the introductory session, participants completed three home-based HIIT sessions per week on their own, with support by weekly check-ins and video conferencing as needed. Post assessments were completed in person at sign-off, just prior to transplant. Aside from the introductory session, all study procedures were conducted remotely or coincided with regular clinical visits.Patients were recruited from the ABMT between June 2019 and May 2022. Adults between the ages of 18–80 years who were scheduled to undergo allogeneic HCT for cancer or a non-cancer illness within six months were eligible to participate. Exclusion criteria included an impairment that precluded them from exercise, an absolute contraindication to exercise , or inabSince this was a single-arm study, all participants received the HIIT intervention. The intervention was developed as a home-based training program due to the following reasons: 1) many patients live far from the ABMT, and thus home-based training is a more realistic option for patients; 2) due to patients’ compromised immune system, attending a fitness and wellness center may not be safe for exercise training; and 3) we have the ability to track exercise adherence to the training intervention using mHealth technology.2peak during the CPET. Interval intensity was calculated as a percentage of HR reserve (HRR) using the following equation: ((maximum HR–HR rate) x training intensity) + resting HR. Heart rates for the high- and low-intensity intervals were prescribed at 80–90% and 50–60% of HRR, respectively. Participants were instructed to complete three HIIT sessions per week for a total of 30 minutes per session, beginning with a 5-minute warm-up at 40–50% HRR, followed by 20 minutes of interval training, and ending with a 5-minute cool-down at 40–50% of HRR. The 20-minute interval portion consisted of 10 total intervals at a work-to-rest ratio of 1:1, which corresponded to one minute of high-intensity (80–90% of HRR) followed by one minute of low-intensity (50–60% of HRR) exercise. Although participants completed the HIIT sessions on the days and times that were convenient to them, they were encouraged to perform the HIIT sessions on non-consecutive days with at least 48 hours between sessions.The intensity of the intervention was individualized to each participant with prescribed HRs for the high- and low-intensity intervals determined from the maximum HR achieved at VOFollowing baseline assessments, participants were introduced to the HIIT protocol by a certified exercise physiologist who was trained on the study protocol and intervention administration procedures. During this in-person introductory session, the exercise physiologist reviewed all aspects of the training program with the participant including prescribed frequency, intensity, duration of the sessions, and timing of intervals. Following the review, the participant performed a practice HIIT session under the supervision of the exercise physiologist using a mode closest to the participant’s self-selected mode for the home-based sessions . The practice session included a warm-up, 3–6 intervals at the prescribed intensities, and a cooldown to familiarize participants with the protocol. Heart rate was monitored continuously throughout the session by the exercise physiologist via the Polar OH1 HR monitor . Rating of perceived exertion was assessed using the 6–20 Borg scale . FollowiParticipants were instructed to maintain the prescribed frequency, intensity, session time, interval ratio, and interval time throughout the duration of the study. While participants were encouraged to start slowly and work up to the 10 intervals over the first 1–2 weeks, no further progressions were implemented since exercise progression for HIIT is inherent to the program: as participants’ cardiovascular fitness improves, the workload required to reach the prescribed HRs will most likely increase. Therefore, participants were encouraged throughout the intervention period to maintain the prescribed HRs for the high- and low-intensity intervals. The length (in weeks) of the intervention varied among participants and was dependent on the length of time between the baseline assessments and sign-off for transplant.Participants were provided two mobile devices throughout the duration of the study that allowed for tracking and monitoring of HRs: 1) an Apple Watch  to wear daily and during the training sessions so that participants could monitor their HRs throughout the workout; 2) an iPhone with an mHealth application that transferred HR data recorded by the Apple Watch to a protected server for researchers to access. The mHealth application installed was either Technology Recordings to better Understand BMT (TRU-BMT)  or SplenWith the exception of the introductory sessions, and per request as needed, all of the training sessions were unsupervised; however, the exercise physiologist was available to provide additional training or support through phone calls or video conferencing using the iPhone provided to participants. The exercise physiologist checked in with participants on a weekly basis via phone calls or text messages (based on participant preference) to provide encouragement and support as well as answer participants’ questions. In addition, HR data from the HIIT sessions were downloaded by the exercise physiologist on a weekly basis and graphed to identify adherence to prescribed interval number and HR zones. The exercise physiologist provided participants with feedback and guidance based on whether the exercise intensity was higher or lower than the prescribed HR zones.Feasibility was assessed via recruitment, retention, session adherence, exercise prescription adherence, mHealth utility, and adverse events. Recruitment was calculated as the percentage of approached participants who consented to participate in the study. Based on recruitment rates from previous cancer trials and overall healthcare exercise trials, we considered a recruitment rate ≥50% as sufficient . Partici2peak recorded as the highest 30-second value elicited during the CPET. Physical function was assessed within three weeks prior to the intervention and coincided with clinical visits. Physical function assessments included the Short Physical Performance Battery (SPPB), 30-second sit to stand, and 6MWT. The SPPB was conducted first and consisted of three tests: balance, gait speed, and five repeated chair stand test [Secondary outcomes included changes in cardiorespiratory fitness and physical function. Cardiorespiratory fitness was assessed just prior to the start of the intervention on a cycle ergometer using a medically supervised CPET, 12-lead electrocardiogram, and breath-by-breath metabolic analysis . The test utilized a ramp protocol (15–20 Watts/min), with VOand test . For theand test . The disThe primary hypothesis for safety and feasibility was that at least 50% of enrolled patients would complete ≥50% of their HIIT sessions and prescribed intervals. Therefore, we planned to enroll a sample size of 35 patients, which would be able to estimate a completion rate of 50% to within a 95% confidence interval of ±19%. For higher completion rates, such as 75%, we would be able to estimate to within a 95% confidence interval of ±17% . ParticiOf the 24 eligible patients who were contacted or approached by the research team regarding potential participation in the study, fourteen patients (58.3%) consented to participate . ReasonsOf the 13 participants who were prescribed the HIIT intervention, ten participants recorded and reported exercise session dates and times to the study team. Three participants did not record or report any exercise session dates despite multiple efforts by researchers to contact them. Uploaded data for those three participants indicated that one participant never wore the Apple Watch, one wore the watch for the first 13 of the 118-day period between enrollment and sign-off, and the third participant wore it for the first 4 of the 84-day intervention period. No HIIT sessions were identified from the data during those wear days, suggesting that the participants did not start the intervention. Therefore, adherence to training sessions and intervals for those three participants could not be determined.2) compared to the four who did not (33.6 kg/m2). There were no differences for age, cardiorespiratory fitness, or functional measures.Of the ten participants who reported any training sessions, nine consistently reported their weekly training sessions and participated in the intervention throughout the duration of the study: seven participants participated in the weekly HIIT sessions from the time between baseline assessments and sign-off and two participants who did not undergo transplant completed the weekly training sessions until they were withdrawn from the study (did not complete sign-off assessments). One participant reported exercise sessions for the first eight weeks, but did not report sessions for the next 24 weeks up to sign-off. Therefore, only nine (69.2%) participants participated in HIIT and followed reporting protocols for the intervention while four (30.8%) did not report sessions or participate in the intervention. A comparison of baseline age, body mass index (BMI), cardiorespiratory fitness, and functional measures  showed that the nine participants who participated/reported sessions had a significantly lower BMI  and women (15.4 mL/kg/min) [Average maximum HR for the high-intensity intervals was 90 ± 21% of HRR with seven of the nine participants averaging a maximum HR that met or exceeded the prescribed intensity and two participants averaging lower than prescribed intensity. Among the nine participants who adhered to the program, baseline VO/kg/min) , demonstThere were no adverse events related to HIIT. One participant experienced abnormally high resting HRs related to a history of atrial fibrillation during one week of the study, but was cleared to continue exercising by his physician. Five of the nine participants reported missing an average of 4.2 ± 1.8 training sessions throughout the study due to cancer- and/or treatment-related effects; these reasons included fatigue from chemotherapy treatment , hospital admission , pain , and a blood clot . These five participants completed 2.4 ± 0.9 make-up sessions in subsequent weeks to account for those missed sessions. The program was well tolerated, with modifications made for only one participant who self-selected longer interval durations due to challenges with HR recovery during the low intensity intervals.2peak from baseline to sign-off . The CPET and introductory training session were completed on the same day, just prior to the start of the intervention (mean time between CPET/introductory session and start of intervention: 2 ± 1 days). Post assessments were completed on all patients regardless of adherence to the study protocol. Post CPET assessments were completed at the time of the sign-off appointment for HCT (prior to transplant) and functional assessment coincided with the closest clinical visit prior to sign-off. Mean time between the last training session and sign-off CPET and functional assessments were 4 ± 2 days and 0 ± 10 days, respectively. Of the ten participants who underwent HCT, seven completed the post CPET and nine completed functional assessments. Reasons for not completing the post CPET assessment was the start of the COVID-19 pandemic (n = 2) and fatigue (n = 1). One participant did not complete the functional assessments due to time constraints from clinical appointments. Among all participants who completed post assessments, there was a significant improvement in VO2peak, 30-second sit to stand, and five repeated chair stand time significantly improved from baseline to sign-off , 30-second sit to stand decreased by 1.7 repetitions and 6MWT distance increased by 10.9 meters. Of those three participants, only one completed the post CPET assessment, with an increase in VO2peak of 4.1 mL/kg/min.There were no changes in SPPB total score, individual SPPB components , 30-second sit to stand, or 6MWT distance. In a secondary analysis that included only the baseline and sign-off scores for the patients who participated in the training program for the duration of the study and underwent HCT (n = 6), VOsign-off . Among t2peak, 6MWT distance, and 30-second sit to stand measures for each individual participant from baseline to sign-off based on intervention length. There were no significant associations between intervention length (in weeks) and change in cardiorespiratory fitness, SPPB scores or individual components, or 6MWT. Among participants who participated/reported session, there was a significant negative correlation between intervention length and changes in 30-second sit to stand scores ) and the relationship between intervention length and time for both the five repeated chair stands and gait speed from the SPPB were approaching significance (five chair stands: r = -.80 (p = 0.057); gait speed: r = -.77, (p = 0.075)).To our knowledge, this is the first study to demonstrate the feasibility of implementing a remotely monitored HIIT program in patients prior to HCT. Although the feasibility and effectiveness of implementing a pre-HCT home-based interval training program has been previously investigated , 24, theAlthough only nine of the 13 participants who were prescribed the HIIT intervention consistently completed and reported their training sessions, data from those participants demonstrate that a home-based HIIT program is feasible in this population in regard to patients’ ability to complete three HIIT sessions/week, achieve intensities close to maximal levels of exertion during the high-intensity intervals, and sustain multiple intervals over time. Adherence to the number of training sessions and session duration among the nine participants was high, with five participants averaging ≥ 3 sessions/week while completing an average of 9.2 of the prescribed 10 intervals per session . Session adherence for all participants who were enrolled in the study, including those who did not log any sessions, was 73 ± 53%, which still exceeded previously reported exercise adherence reported in this population . ParticiAlthough adherence was high among the nine participants who took part in the intervention and logged/reported their training sessions, three participants did not report any of their sessions or wear the Apple Watch during the majority of the intervention period and one participant only reported the first eight of 32 weeks. Reasons for the lack of exercise participation in these participants is unknown as participants did not respond to repeated attempts by the exercise physiologist to contact them about their training sessions. Although completion may be related to response to exercise, the data retrieved from two participants who did not report sessions, but wore the watch for a several days at the beginning of the intervention, indicate that no HIIT sessions were attempted. For the participant who reported only the first eight weeks of the intervention, data were not able to be retrieved, which may have impacted continued participation for this patient as the study team was unable to provide feedback and coaching based on the HR data for his HIIT sessions. Given that participation in the intervention was mostly dichotomous with either high adherence among patients (69.2%) or little to no participation at all (30.8%), future research is needed to determine factors that predict exercise participation and strategies to promote exercise adherence during the pre-HCT period. We did identify BMI as a potential factor, which suggests that higher bodyweight may be a barrier to participation in HIIT protocols in this population. In addition, HCT recipients experience health concerns, high infection burden, and barriers to exercise that are unique compared to other cancer and chronic disease populations , which mAnother challenge with implementation of a pre-HCT intervention is the enrollment of patients who may not eventually proceed to HCT. In this current study, three patients did not undergo transplant due to cancer relapse. While their data were included in the feasibility analysis, they did not complete sign-off assessments since they did not proceed to HCT, thus limiting pre- and post-intervention data and generalizability of results to patients who will ultimately undergo HCT. The expected attrition during the pre-HCT period may pose limitations in future trials, especially for those examining the effect of pre-HCT interventions on post-HCT outcomes, and thus both dropout and exclusion of participants who do not ultimately undergo HCT should be considered when determining sample size.Heart rate data from the training sessions were successfully retrieved from 72.5% of reported sessions, demonstrating that the Apple Watch along with an mHealth App can be practical tools for remotely monitoring aerobic training and HIIT. Heart rate data during the sessions were retrieved approximately every five seconds, allowing for a comprehensive assessment of HR for the entire HIIT session. Graphs for each session allowed for analysis of HR data to determine the number of intervals completed and how many intervals met the target HR zones. Although we demonstrate that using the Apple Watch with an mHealth App is feasible for remotely monitoring HIIT sessions, we did have issues with missing HR data for 27.5% of reported sessions. This occurred mainly from improper positioning of the watch on participants’ wrist, participants forgetting to wear the Apple Watch, and issues with HR data transferring from the watch into the database. Therefore, to minimize loss of data, strategies including training participants on how to use the Apple Watch, ensuring the wrist band is the appropriate size for the participants’ wrist, frequent check-ins with participants to ensure they are wearing the watch, consistent troubleshooting when data fail to transfer to the database, and making improvements in application and database performance to ensure data are consistently transferred.Despite some of the technical challenges related to the Apple Watch and the retrieval of the data, patients persevered in working with the platform and the research team with 9 of the 13 participants continuing to exercise until transplant or until it was determined that their disease relapsed and they were no longer eligible for transplant. Furthermore, all patients who participated in the program and were evaluable experienced improvements in cardiorespiratory fitness and performance on the chair stand tests and 6MWT. This suggests that despite technical challenges, patients participated and improved their function, supporting feasibility of a remotely monitored home-based HIIT intervention.2peak significantly improved from baseline to sign-off, with an average improvement of 3.1 mL/kg/min (21%). This is consistent with findings from Wood et al. [2peak following a prehabilitative interval training program in HCT recipients. We also observed an improvement in VO2peak in one participant who did not adhere to the intervention protocol. Since this participant did not report his training sessions or wear the Apple Watch, we do not know if he completed any exercise sessions or increased his overall physical activity since the start of the study, thus limiting our ability to draw conclusions from his data. Considering it has been previously shown that every one MET (3.5 mL/kg/min) increase in exercise capacity is associated with a 12 percent improvement in survival [2peak ≤ 16 mL/kg/min was associated with a longer hospital stay during initial transplant, a higher number of hospitalized days before Day 100, and a higher risk for mortality compared to patients with a VO2peak > 16 mL/kg/min. Five of the seven participants in our study who completed the baseline and post CPET started off with a VO2peak that was less than 16 mL/kg/min. Following the intervention period, three of the five participants had increased their VO2peak to above 16 mL/kg/min, suggesting that HIIT may result in cardiorespiratory improvements with a potential benefit on clinical outcomes. In addition, there was no relationship between changes in VO2peak and intervention length with comparable VO2peak increases observed among patients with shorter intervention length and longer intervention length. This suggests that even a short HIIT intervention period prior to HCT may be enough to gain cardiorespiratory improvements.Although our small sample size limits us in regard to statistical analysis and power, our preliminary analysis of changes in cardiorespiratory fitness and physical function suggest that a pre-HCT HIIT program has beneficial effects on cardiorespiratory fitness. In those who participated in the intervention, VOd et al.  who reposurvival , our ressurvival , found tAmong adherent participants, scores for the 30-second sit to stand and five repeated chair stand time significantly improved, which suggests that HIIT was beneficial for leg strength and power. While clinically meaningful changes have not been established in HCT recipients, the 4.5 repetition increase in 30-second sit to stand exceeds minimal clinically important improvements established in other chronic disease populations , 37. Lik2peak during the CPET to tailor the program to each individual participant. Another strength is the use of mHealth technology to motivate participants, monitor training intensity, and assess the feasibility of the intervention. Since a home-based training program is a safer and more practical option for this population, the use of technology allowed researchers to remotely monitor the participants’ exercise sessions in real time to provide feedback on HR values and assess adherence to the HIIT prescription. Therefore, findings regarding feasibility for this study can be used to inform future trials on the use of technology for assessing exercise program intensity, adherence, and fidelity of home-based interventions.A strength of this study includes the individualized HIIT program that was based on resting HR and maximum HR achieved at VO2peak and physical function improvements could be obtained in those who participated in the intervention for a shorter length of time.Limitations to this pilot study include the non-randomized design, small sample size, and a small number of participants who logged/reported their training sessions and wore the Apple Watch for analysis. In addition, of the 24 participants who were screened for eligibility and approached for the study, 18 were male and 6 were female, with 10 males and 3 females ultimately participating in the study. Given that the majority of patients who were approached and participated in the study were male, results may be less generalizable to female patients. We also did not obtain information regarding prior exercise experience or other activities participants completed outside of the HIIT sessions, which may have influenced exercise adherence and results. The start of the COVID-19 pandemic also limited recruitment and prevented the completion of sign-off CPETs for two participants and three participants never underwent HCT. As a result, we were limited in our approaches for data analysis and ability to interpret the data. Furthermore, data were not retrieved on 27.5% of sessions, which limited our ability to analyze data from those HIIT sessions. Due to the technical challenges in obtaining some of the exercise HR data from the Apple Watch, we also cannot be sure that data are missing at random. Given that this was the first trial where we tested the utility of the Apple Watch and the mHealth Apps to collect the HIIT intervention data, we encountered issues early on in collecting and retrieving the Apple Watch data that we were able to resolve more quickly as the trial went on, thus improving our ability to collect and retrieve the data. Thus, we anticipate a lower percentage of missing data in future trials. Since it has been previously suggested that <5% of missing data is acceptable while >20% may pose threats to validity , more reAlthough findings demonstrate that participants were able to perform HIIT with high adherence to the exercise prescription, it is also important to acknowledge that there is heterogeneity among the HCT population and thus a standardized remotely monitored program like HIIT may not be realistic for all patients. Although the HIIT prescription was individualized to each patient and the mode was self-selected, other types of exercise programs that begin with lower intensities as well as the use of additional behavior change strategies may be more practical options for patients who may be limited by factors such as high fatigue, obesity, and/or functional impairments. Long-term sustainability, affordability, and accessibility of this intervention should also be considered as trained staff is needed to implement the intervention and provide motivation, costs associated with the Apple Watch and iPhone, and necessity of WiFi access so that Apple Watch data can be transferred to researchers. Future research should explore options for automated tracking of HIIT intervals and feedback and cost-effective strategies for implementation.Findings from our study demonstrate that implementing a home-based remotely monitored HIIT program in patients prior to undergoing HCT is feasible. In addition, we demonstrated that mHealth technology can be used to remotely monitor exercise adherence and training intensity. Although analysis of changes in cardiopulmonary and physical function was limited due to the small sample size, our preliminary findings support the need for larger randomized control trials to determine if HIIT can be an effective approach for improving these outcomes prior to HCT. Future studies are also needed to determine barriers to participation in a remotely monitored exercise intervention and strategies for promoting retention and exercise adherence in this population. Furthermore, future development and improvements of phone applications and databases to transfer Apple Watch data to the research database are needed to ensure HIIT session data can be retrieved by the research team, which will improve interpretation of results and provide a better indication of the Apple Watch’s ability to consistently capture HR data during HIIT.S1 File(PDF)Click here for additional data file.S2 File(PDF)Click here for additional data file.S3 File(XLSX)Click here for additional data file.


Persons experiencing homelessness (PEH) or rough sleeping are a vulnerable population, likely to be disproportionately affected by the coronavirus disease 2019 (COVID-19) pandemic. The impact of COVID-19 infection on this population is yet to be fully described in England. We present a novel method to identify COVID-19 cases in this population and describe its findings. A phenotype was developed and validated to identify PEH or rough sleeping in a national surveillance system. Confirmed COVID-19 cases in England from March 2020 to March 2022 were address-matched to known homelessness accommodations and shelters. Further cases were identified using address-based indicators, such as NHS pseudo postcodes. In total, 1835 cases were identified by the phenotype. Most were <39 years of age (66.8%) and male (62.8%). The proportion of cases was highest in London (29.8%). The proportion of cases of a minority ethnic background and deaths were disproportionality greater in this population, compared to all COVID-19 cases in England. This methodology provides an approach to track the impact of COVID-19 on a subset of this population and will be relevant to policy making. Future surveillance systems and studies may benefit from this approach to further investigate the impact of COVID-19 and other diseases on select populations. Persons experiencing homelessness (PEH) or rough sleeping are likely to be disproportionately impacted by infectious diseases such as coronavirus disease 2019 (COVID-19) due to poor underlying health, undiagnosed health conditions increasing the risk of complications, limited access to sanitation and vaccinations and the sometimes crowded conditions of shared accommodation –5.Routine surveillance of health outcomes in PEH is challenging, in part because surveillance records rarely contain information about individual's housing status and may not be kept up to date . The impTo identify a case in the UKHSA national COVID-19 dataset who was likely to be experiencing homelessness or rough sleeping a ‘case phenotype’ was developed by seeking and incorporating the feedback of experts in the field.A reference list of addresses designated to providing temporary accommodation and shelter to PEH was collated by the Office of Health Improvement and Disparities. The list was compiled using data provided by NHS England, the Greater London Authority, Homeless Link and Housing Justice. The data included addresses permanently designated for accommodation for PEH, such as hostels, as well as addresses temporarily used for this purpose, such as night shelters and hotels procured for accommodation under the ‘Everyone In’ campaign. For some addresses, specific information was available, such as size, target demographic and active dates.In a process originally developed during the COVID-19 pandemic to identify vulnerable cases, such as care home residents and those in prisons , UKHSA rThe UKHSA National COVID-19 case dataset was queried, and cases were linked to the reference list of addresses using UPRN. Data were included from 24 March 2020 when the UK government announced the first ‘lockdown’ in England to 07 March 2022.Any test-confirmed COVID-19 case, where the residential address matches to a known homeless shelter or accommodation using UPRN.ORAny case with ‘nfa’ or ‘no fixed abode’ in their address fieldORAny case with an NHS pseudo postcode of ‘ZZ993VZ’ which are supplied by The Office for National Statistics (ONS) and used in the NHS to denote in patient records when a case has no fixed abode.Once this cohort was identified, the following cases were excluded through manual data cleaning:Any case with a specimen date that did not coincide with the time frame that the accommodation they matched to was active (if known), for example certain temporary accommodations such as hotels used for the ‘Everyone In’ campaign which were reopened to the public following the easing of national restrictions.Cases with addresses associated with a prison BLPU code  or any case with address information related to a foreign address, who can sometimes be assigned NHS pseudo postcodes.Cases where patient sex was unknown.The study cohort was enhanced with data UKHSA holds on COVID-19 cases, including ethnicity and mortality. Ethnicity is derived from test information, and where missing or conflicting from Hospital Episode Statistics database. Further details on surveillance data that UKHSA holds are described comprehensively elsewhere [‘A death in a person with a positive severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) test and either died within 60 days of the first specimen date of the most recent infection or died more than 60 days after the first specimen date of the most recent infection, only if COVID-19 is mentioned on the death certificate’.A comparison between the cases identified as PEH and all cases in England for the specified time period was carried out to highlight demographic differences between these populations.Following linkage, the criterion for inclusion were:lsewhere . For thiA total of 1835 cases were identified as PEH. Of these 1649 cases were identified via address matching. A further 127 were identified as having no fixed abode via an NHS pseudo ZZ postcode, and 39 were identified as having ‘no fixed abode’ or ‘nfa’ in their address. A total of 237 records did not have an NHS number (12.9%). Cases matched to a total of 397 unique reference address UPRNs. The greatest number of cases identified at a given UPRN over the study period was 40, with a median of 7.Approximately a third (29.8%) of PEH cases we identified had a residential address based in London, followed by the North West (14.4%).There were 1152 males (62.8%) and 683 females (37.2%) identified; the median age was 35 (IQR = 23) and 23 (IQR = 16) years old respectively. There were 77 cases under the age of 18. Most cases identified were of White ethnicity (63.8%), followed by Black ethnicity (12.6%). Ethnicity was unknown for 203 cases (11.1%).There were 25 deaths within 60 days of earliest specimen date (1.4%) in PEH during the study period; 19 in males and 6 in females.The distribution of PEH cases over time displays correlation with the national case trend at various stages of the pandemic  and indiBeing a vulnerable population, it is important to highlight the impact of COVID-19 on PEH in England to inform policy responses; however, studies using national data are currently lacking.This study has identified a cohort of 1835 COVID-19 cases who were likely to be PEH between March 2020 and March 2022. We report that most PEH cases were young males, and that the number of cases in PEH was greatest in London. We also find that the majority of PEH cases were of White ethnicity. Our findings broadly align with the expected demographic characteristics of PEH cases which has been described in England . This mevs. 15.9%). The opposite was seen in other regions; for example the East Midlands (4.7% vs. 8.6%) and the South East (7% vs. 15.2%). It is unclear whether transmission of COVID-19 among PEH was greater in London, though it was expected as the number of PEH is highest in London as reported by the homeless charity Shelter and ONS [We identified most cases in London 29.8%) and the North West 14.4%) and the fewest in the North East (2.9%). Relative to national case data, we identified a greater proportion of PEH cases in London (29.8% 9.8% and .4% and tvs. 3.5%). This may reflect a higher risk of COVID-19 infection in PEH of Black ethnic groups, however this cannot be concluded with certainty without PEH denominators overall and by ethnicity.When further comparing demographic characteristics of PEH to the national COVID-19 case data in England, there are some key differences concerning to the relative impact COVID-19 has had on PEH. Although the majority of PEH cases were of White ethnicity (63.8%), the proportion of PEH that are of an ethnic minority background was generally higher in PEH than in the national case data. For example, the proportion of cases of Black ethnicity in PEH was suggestively higher compared to the national case data  suggests poorer COVID-19 health outcomes in PEH which is an area of potential concern. Further analysis and consideration would be needed to understand these findings in more details based on differences in testing, cohort sizes as well as differences in vaccination status, sex and co-morbidities which were outside the scope of the development of these methods. Due to deductive disclosure via small numbers it was not possible to display mortality by age-bands.Additionally, the proportion of deaths, based on an unadjusted comparison, in PEH (1.4%) These findings provide important context to public health policy measures  that werA more in-depth assessment of outbreaks within shared accommodation settings and shelters is needed to provide specific details on the risk of COVID-19 transmission. Further studies could use this methodology to investigate the severity of COVID-19 in PEH through linkage to hospital records, as well as the uptake of vaccines which will be key to policy and inclusion health outcomes in this area. Similar methodologies could also be applied to other infectious diseases that disproportionately affect PEH such as Hepatitis, HIV and other respiratory diseases to identify cases based on address information.It is not possible to assess the scale of the potential limitations associated with this methodology, as there is no relevant national reference data set for comparisons to be made.These findings will likely be an underestimate of COVID-19 in PEH, primarily due to the low sensitivity of the methods to identify broader homelessness situations, and should patient address information be out of date in laboratory and electronic medical records. We attempted to address this limitation by using NHS pseudo postcodes, which are used in patient records when a case has no fixed abode, although its completeness in COVID-19 testing and hospital records is unknown.PEH face challenges in accessing care, which affects the quality and timeliness of health records  and it iThis study presents evidence that PEH have been impacted by the COVID-19 pandemic. The methodology and phenotype for this population has been shown to be practical to implement and demonstrates that it could be applied to further surveillance and research on COVID-19 in PEH, and potentially also to other diseases.


The proliferation of health care data in electronic health records (EHRs) is fueling the need for clinical decision support (CDS) that ensures accuracy and reduces cognitive processing and documentation burden. The CDS format can play a key role in achieving the desired outcomes. Building on our laboratory-based pilot study with 60 registered nurses (RNs) from 1 Midwest US metropolitan area indicating the importance of graph literacy (GL), we conducted a fully powered, innovative, national, and web-based randomized controlled trial with 203 RNs.This study aimed to compare care planning time (CPT) and the adoption of evidence-based CDS recommendations by RNs randomly assigned to 1 of 4 CDS format groups: text only (TO), text+table (TT), text+graph (TG), and tailored (based on the RN’s GL score). We hypothesized that the tailored CDS group will have faster CPT (primary) and higher adoption rates (secondary) than the 3 nontailored CDS groups.Eligible RNs employed in an adult hospital unit within the past 2 years were recruited randomly from 10 State Board of Nursing lists representing the 5 regions of the United States  to participate in a randomized controlled trial. RNs were randomly assigned to 1 of 4 CDS format groups—TO, TT, TG, and tailored (based on the RN’s GL score)—and interacted with the intervention on their PCs. Regression analysis was performed to estimate the effect of tailoring and the association between CPT and RN characteristics.P=.01). The CPT in the TG CDS format was associated with age (P=.02), GL (P=.02), and comfort with EHRs (P=.047). Comfort with EHRs was also associated with CPT in the TT CDS format (P<.001).The differences between the tailored (n=46) and nontailored  CDS groups were not significant for either the CPT or the CDS adoption rate. RNs with low GL had longer CPT interacting with the TG CDS format than the TO CDS format  interacted on the web with the intervention on their PCs. Future studies applying our innovative methodology are recommended to cost-effectively enhance the understanding of how the RN’s GL, combined with additional factors, can inform the development of efficient CDS for care planning and other EHR components before use in practice. Despite the length of time that electronic health records (EHRs) have been around, there continue to be reports of unintended consequences from poor design and inadequate testing -5 advers23(b)(4) ).F1,119=13.0; P<.05). No significant differences in nurses’ perceived workload were found. In another systematic review of visualizations of patient-reported outcomes, clinicians’ preferences for line graphs, heat maps, or bar charts and objective accuracy of interpretations were mixed [A systematic review of 28 studies targeting hospital nurses’ decision-making underscored the promise of CDS, including improved care processes in 7 out of 7 studies, improved usability in 4 out of 4 studies, and improved patient outcomes in 3 out of 4 studies . Howeverre mixed .In a study of visual data displays in 2 homecare facilities, nurses understood graphs most easily, followed by tables, line graphs, and spider graphs. Furthermore, nurses with low numeracy and GL had poorer comprehension of the information displayed across all formats. High GL appeared to enhance the comprehension of data regardless of numeracy . In a saIn our prior laboratory-based preclinical simulation RCT, also funded by the National Institute of Nursing Research, we compared the care planning time (CPT) and rates of selecting evidence-based best practice nursing diagnoses, interventions, and outcomes for RNs. The 60 RNs were randomly assigned to 1 of 4 different CDS format conditions: text only (TO), text+table (TT), text+graph (TG), or no CDS . We founThe primary aim of our nationally representative, web-based simulation RCT reported here was to examine the effects of tailoring  by comparing 203 RNs randomly assigned with CDS format groups  on CPT and the adoption of evidence-based best practices. We hypothesized that the tailored CDS group would have faster CPT (primary) and higher CDS recommendation adoption rates. This study is part of our team’s larger research agenda focused on generating standardized nursing data gathered in EHRs, automatically analyzing these data, and returning relevant evidence to the point of care in a CDS format supporting efficient and effective RN decisions. This study is part of a research agenda that began in 1998 with the creation of a research-based nurse care planning system. A synopsis of the team’s work to date is shown in Study Measures section. The study involved 3 phases: recruitment, screening, and consent—session 1; testing intervention; and postsurvey—session 2  of the University of Florida (IRB201902611) and the University of Iowa (201910213).Our study was guided by the Cognitive Load Theory (CLT) . The CLTIn our study, we recognized that RNs spend much of their day processing large amounts of data  to make clinical decisions. The data processing time combined with multiple competing demands on nurses’ time leads to a high cognitive workload. A suboptimal display of CDS that does not accommodate individual nurses’ learning differences  is expected to increase the cognitive load of RNs and diminish their understanding of the data and the efficiency of decision-making in terms of time .As depicted in We devised an innovative approach that enabled us to conduct a cost-effective national study to validate the findings of our pilot study comprising RNs from a large Midwest US metropolitan area . To do tThe first year of the study was spent reprogramming and iteratively testing the new web-based version of the intervention. Reprogramming was performed by our team programmer and initial testing was conducted by the team. In the final phase of development, the entire procedure for accessing, orienting, and interacting with the intervention was pretested with a sample of 31 RNs . This ensured that the intervention operated as intended for a broad range of user setups (internet and computer), enabling a diverse set of RNs to participate from their homes.The intervention, described in detail elsewhere , began wThe hypothetical patient scenarios were developed by the investigators (DW and GK) using real patient histories and simulating realistic patient trajectories across time. The CDS recommendations in the study were based on evidence generated by our research team on previous data collected with HANDS under real-time conditions and peer-reviewed literature. An example of Mr Taylor’s history  is shown in The data reported for this intervention study were collected from RN participants between June 2019 and January 2022 in 2 sessions . To ensuThe 10 SBON lists contained addresses and phone numbers of RNs, with 5 states  also providing email addresses. Potential RN participants from SBON lists without email addresses were contacted through the mail-only approach, and those with email addresses were contacted by either the mixed (mail and email) or email-only approaches. All RNs contacted were randomly selected from these lists and sent recruitment messages with follow-up messaging as per the survey recruitment methods by Dillman et al . The posNurses were eligible if they (1) were aged ≥18 years; (2) possessed an RN license; (3) worked on an adult medical-surgical unit in a hospital setting within the last 2 years; and (4) met technical requirements—access to a tablet, laptop, or desktop computer (<4 years old) with a screen size ≥9” on the diagonal; an internet connection of ≥25 Mbps; had Windows 10 operating system, Apple macOS 10, or newer systems; and had the latest versions of Zoom and Google Chrome or Safari. The exclusion criteria were as follows: (1) did not meet technical requirements and all other inclusion criteria; (2) were unable to complete the study owing to physical or cognitive impairment; or (3) were unable to speak, read, or write English. To further enhance generalizability, quotas were set to ensure the representation of important subgroups: men , racial and ethnic minority group participants , and nurses with Associate’s Degree in Nursing as their highest degree . We set out to recruit approximately 20% from the 5 US regions.pwr package, we determined that a sample of 200 RNs would provide sufficient (>80%) power to detect the differences between the tailored CDS group and the 3 nontailored CDS groups.Our primary aim was to estimate the effect of tailoring on RN CPT. Therefore, we focused on the comparison between the tailored CDS group and each of the 3 nontailored CDS groups: TO, TT, and TG. To achieve a family-wise type-1 error of 5%, we set the cutoff for significance at 0.016 using the Bonferroni adjustment. Using the findings from our pilot study on the association between GL and RN CPT , we estiStratified by GL , block randomization was used to assign RNs to 1 of the 4 CDS format groups . All 4 groups accessed the same 2 patient scenarios. Randomization was performed by the software automatically. Study staff recruiting and interacting with the RNs and the RN participants themselves were blinded to the group assignment.Data collection occurred in 2 sessions. The amount of time needed to complete both sessions was approximately 1 to 2 hours.Session 1: Screening and consent—this session involved screening, consenting (using a web-based process), collecting participant demographics, and assessing the RN participant’s GL score. Data were collected and securely stored in the REDCap  database hosted by the Clinical and Translational Science Institute at the University of Florida, which is Health Insurance Portability and Accountability Act compliant and approved by the IRB of the University of Florida and privacy office for the collection and storage of protected health information. After completing the GL assessment in session 1, eligible RNs in categories under the quota limit were scheduled for session 2 (CDS intervention).Session 2: Experimental CDS intervention—in this session, data related to the experimental intervention and the postsurvey were collected. These data were securely stored in a SQL database server at the University of Florida. No personal information was collected in this session. Before the start of session 2, the project staff verified that each RN participant had access to the required technology, including the Zoom app and an internet browser compatible with the study care planning system. The session began with a standardized video orientation to the user interface and the care planning terminologies, nursing diagnosis , nursingThe study intervention occurred across 3 simulated shifts and began with screens showing the current care plans for 2 hypothetical patients. The research assistant verbally delivered the histories of the 2 hypothetical patients. The RN was directed to update the 2 patients’ care plans based on the patient histories and other information that appeared during the intervention. The research assistant observed but did not interact with the RNs as they updated the care plans. After the RN completed the first simulated shift, the intervention included directions for a second and third shift in which the patients’ outcomes and CDS were automatically updated to reflect the RN’s decisions in the prior shift. The RNs viewed the updated outcomes and were asked to apply their clinical judgment when processing the information and prompts displayed . We hypothesized that the RNs assigned to the tailored CDS group  will have faster CPT and higher CDS suggestion adoption than those assigned a fixed CDS format independent of their GL.Demographics and comfort with EHRs were collected in session 1—gender ; race ; Hispanic (yes or no); nursing degree programs completed ; years of experience; and state of RN license. Comfort with EHRs was assessed using a single item rated on a scale of 1 to 10.The primary outcome of this RCT was CPT, defined as the total amount of time (in minutes) the RN spent interacting with the care planning software over the 3 simulated shifts. The secondary outcome was the CDS adoption rate, defined as the percentage of CDS recommendations adopted. Each action that the RN took while interacting with the CPS was automatically time-stamped and recorded by our application.GL is a 13-item objective tool with one right answer per item [r=0.44). The total score is the number of items answered correctly (range 0-13). In our study, GL was assessed on all RNs before randomization and used to assign tailored CDS group RNs to format conditions as follows: RNs with scores up to 10 were presented with the TO format, those with scores of 11 were presented with the TT format, and the RN with scores ≥12 were presented with the TG format [per item . The tooG format . This taG format .Numeracy, the ease with which individuals can process basic probability and mathematical concepts, was measured objectively with the Numeracy Scale [cy Scale . The NumCognitive workload, the perceived amount of mental processing required to perform clinical tasks in the EHR [ the EHR ,45, was  the EHR ,46,47. I the EHR , includi the EHR -51.2-sided α of .016 for comparison of the primary outcome between the tailored CDS group and the nontailored CDS groups and .05 for other analyses.Session 1 data were stored in REDCap, whereas session 2 data were stored in the SQL databases. Data were imported into the statistical software R  for the analysis. Descriptive statistics, including mean, SD, frequency, percentage, and IQR (25th percentile and 75th percentile) were obtained. ANOVA and Fisher exact test were used to examine the variation of variables across groups. Linear regression analysis was used to compare CPT (primary outcome) and CDS adoption rate (secondary outcome) between the tailored CDS group and the 3 nontailored CDS groups and to examine their association with various predictors. For the handling of the small amount (<1%) of missing data, we used the approach of multiple imputations, where multiple completed data sets were generated using the fully conditional specification. Inference was performed on each completed data set and then aggregated using the Rubin rule . CPT wasA total of 38,700 RNs were randomly selected from the SBON lists and sent recruitment messages using 1 of the 3 methods: mail only , mixed , and email only . The response rates were low as expected, with the mixed approach having the highest response rate. r=0.26; P<.001). The test reliabilities for these 2 scales were 0.70 and 0.82, respectively. The RN completers were aged 22 to 69 years with a mean age of 40.8  years . Of the 203 participants, 20% (n=41) were men, 9.4% (n=19) were Asian, 10.3% (n=21) were Black, 12.8% (n=26) were Hispanic, 74.9% (n=152) were White , and 5.4% (n=11) were other races. Overall, 73.9% (150/203) of the participants had a Bachelor of Science in Nursing. On average, the participants had 12.3 (SD 10.5) years of RN experience, with 55% (112/203) having up to 10 years, 27% (55/203) having 11 to 20 years, 9% (19/203) having 21 to 30 years, 7% (15/203) having 31 to 40 years, and 1% (2/203) having >40 years of experience, and identified themselves as very comfortable  working with the EHR. RNs in our study had relatively high GL and numeracy scores, with a mean of 11.0 (SD 1.6) and 8.9 (SD 1.8), respectively. The 2 scores were weakly correlated  in this sample for either the CPT or the CDS adoption rate. Regression analysis, including RN characteristics  as covariates yielded the same conclusion.Outputs of the regression analysis comparing the 3 groups assigned a fixed CDS format with the tailored CDS group are presented in P=.01). Other comparisons of CPT under the presumed optimal (GL-matched) format and alternative formats did not show significant differences . Each GL category had 3 groups, those who interacted with the format matched to their GL and those who interacted with one of the other 2 formats, for example, the 3 low GL groups included those who interacted with the TO (matched format) and those who interacted with TT or TG format. An RN could be exposed to the matched GL format by being assigned to the tailored CDS group or to one of the other 2 formats through random assignment. We hypothesized that RNs assigned to the matched format would have a lower CPT. For RNs with low GL, CPT was higher for those given CDS in the TG formats than in the matched format .Finally, we compared the perceived cognitive workload of using the CDS system reported by the participants in the 4 CDS format groups. The mean and SD by group for each item and the overall workload are presented in In the current national RCT with 203 hospital RNs, we found that our hypotheses were not supported. We expected the tailored CDS group to outperform the nontailored CDS groups on time spent on care planning and adoption of CDS evidence-based recommendations, but this did not occur. Our exploratory analysis found that comfort working with EHR was associated with less time spent on care planning. Older age and lower GL were associated with longer decision time when evidence was presented in a TG format, whereas such associations did not hold when evidence was presented in a TO or TT format. There were also no significant differences found across groups when comparing the RNs’ perceived cognitive workload of using the assigned CDS. Perhaps the most important outcome, however, is that this study demonstrated the feasibility of an innovative approach for conducting a national RCT on a prototype EHR component via the internet. This success provides a viable approach for pretesting EHR components before integration into practice with truly diverse representative samples of potential users under simulated conditions, thereby avoiding unintended risks to patients.Currently, >96% of general acute care hospitals use EHRs  with nurAs noted, we did not find significant differences in CPT between the tailored group  and those assigned randomly to these 3 CDS formats. When examining CPT for RNs by GL, however, we found that RNs with low GL had significantly higher CPT when interacting in the TG format versus when interacting with the TO format . There were no significant differences found for RNs with medium or high GL between hypothetical optimal formats and other formats. These findings provide insight into why the tailored CDS group did not perform better. However, for those with a medium or high GL, other factors may be more important when tailoring CDS formats. In our previous study, we found that higher subjective numeracy and GL were associated with less time spent on care planning for the TT and TG conditions, respectively . This stWe also know from our team’s earlier research that GL varies among RNs ,55. OtheAnother central finding was that, on average, participants adopted the vast majority of the CDS recommendations offered. This is consistent with previous evidence suggesting that clinicians are willing to adopt evidence-based recommendations suggested to them via CDS . We hypoIf adoption rates are similarly high when evidence-based CDS is implemented in clinical practice, one would expect patient care outcomes to be improved . Future A key innovation of our research was conducting a national RCT remotely using internet-based communication tools . This allowed us to securely test the intervention with a diverse, representative sample of RNs in their homes on their PCs. Although we are not the first to use internet-based tools in an RCT -64, we aAs generalizability was a priority for establishing the potential broad applicability of our findings, our team translated the pilot laboratory version of the intervention into a web version that could be tested using a national sample of RNs from their homes. The intervention was successfully translated and delivered consistently, ensuring fidelity of the intervention throughout the study. Our recruitment strategy attracted a balanced and diverse national sample of RNs with recent experience in adult hospital units. The RNs contacted for potential participation were randomly selected from 10 state board lists, representing 5 regions of the country. Furthermore, we intentionally oversampled important minority groups  within the profession that are expected to expand in the years to come. As a result, our sample had a higher proportion of these groups than those found in the 2020 Nursing Workforce Survey  among alAs noted earlier, reports of unintended consequences from poor design and inadequate testing -5 of EHRBuilding on our pilot findings, we set out to determine whether tailoring assignment to a CDS format by GL would further improve the documentation time and the potential quality of care plans  in a national RCT of 203 RNs. We purposefully sought to ensure that our findings would represent the diversity of RNs in the United States employed in hospital medical-surgical units. To do this, we implemented an innovative recruitment method and deployment of our intervention. We used 10 SBON lists to randomly identify RNs eligible for our study . The pilAlthough our RCT hypotheses were not supported, this study replicated our previous findings that low GL was associated with higher CPT in the TG condition. The robustness of these associations, given the high quality of our sample, suggests the need to consider the GL of RNs when designing and integrating visual displays such as graphs into care planning CDS. Our findings are supported by others who also found an association between higher GL and reduced decision time ,31,56,57In summary, although our hypotheses did not hold, the results of this study offer important knowledge for future directions. Our findings suggest a clear need for further research to establish the combination of factors to consider when using visual displays in the EHRs for specified purposes. Although others have examined clinician format preferences and the relationship of visual display type to the accuracy of some types of data, research has yet to establish the conditions  that collectively affect the achievement of desired outcomes. Although our tailoring scheme did not reduce RN documentation time for care planning, we confirmed the relationship between GL and documentation time in our pilot study. In the future, we will continue using our innovative preclinical recruitment and national testing methods to further isolate the factors ,70 that Our study has some limitations that should be considered as study findings are interpreted. First, although our rigorous recruitment processes produced a nationally diverse population of RNs and important subgroups within, our overall response rate was low indicating potential selection bias. The age range of our participants was lower than the national statistics. The participating RNs may also be atypical in their interest in research in general and health information technology in particular. However, our participants may well represent the workforce 5 years from now and thus our outcomes once further validated will apply to the workforce of the day. Second, this study was conducted during the COVID-19 pandemic. The very high workload and stress placed on RNs might have had an impact on the response rate and on the representativeness of the participating RNs. Third, testing novel EHR features in a simulation environment provides important insight but is only one step toward improving EHR in a clinical setting. Fourth, RNs might have evaluation anxiety that affected how they interacted with the CDS system even though we took great care in the study to mitigate such anxiety. Fifth, RNs’ comfort with EHR was assessed with a single item and found to impact documentation time. Future studies are needed to validate this finding and to investigate the effect of an RN’s prior experience with CDS, which might be more relevant than comfort with EHR in general. Finally, we focused on matching CDS formats to RN characteristics. In clinical practice, it will also be important to take into account other characteristics that we have highlighted above when choosing data visualization formats.