ObjectiveThe aim of this research was to determine if medical students' use of the active study strategy of working practice questions is associated with improved performance on the United States Medical Licensing Examination (USMLE) Step 1 exam when compared to students who used the passive study strategy of watching educational videos.MethodsThe study used a correlational design. Participants were students from two cohorts in a United States medical school (N=164 and N=163) who completed their first two years and took the USMLE Step 1 exam. Data collected retrospectively included the number of practice questions completed, educational videos watched, Step 1 exam scores, average scores on in-class exams, and scores on the Medical College Admission Test (MCAT).ResultsThe number of videos watched was negatively and significantly correlated with the Step 1 score for cohort 2022  and cohort 2023 . The number of practice questions worked was positively and significantly correlated with the Step 1 score for cohort 2022  and cohort 2023 (r=0.143 though not significant). The number of practice questions was a significant positive predictor of Step 1 score for cohort 2022  and cohort 2023 . Videos were significant negative predictors for cohort 2023 .ConclusionsAnswering practice questions appears to be a more effective study method than passively watching videos. Though other studies have supported the use of active learning methods, this study is unique in finding a negative correlation between test scores and the number of educational videos watched. Medical students should be urged to make the most effective use of study time by incorporating working practice questions and limiting watching educational videos. Helping medical students who are struggling with academic difficulties is a challenging task for medical educators. As of yet, there is not a clear and consistent approach in the literature on how to help struggling medical students -3. ReasoChallenges for medical students include performing well on standardized exams within their coursework and on board exams required for academic progression. Performance on the United States Medical Licensure Exam (USMLE) Step 1 and Step 2 Clinical Knowledge (CK) board exams can affect students' residency selections and career choices; therefore, a major challenge for medical schools in the United States of America (USA) and Canada is to help students improve course exam scores and performance on the USMLE Step 1 and Step 2 examinations -7.Active study methods, including retrieval practice, spaced repetition, and the use of practice questions and question banks (a pool of practice questions), have been shown to be effective means of improving students' learning and long-term retention -11. PracVarious educational videos from different sources have been shown to be the most popular e-learning modality . InteresSettingThe University of Arkansas for Medical Sciences (UAMS) College of Medicine (COM) is a four-year public medical school located in the southern region of the United States of America (USA). The curriculum is organized into two pre-clinical years followed by two clinical years .Data indicated that students in UAMS COM who underperformed in their first and second years of medical school had a higher failure rate on the USMLE Step 1 exam. Based on data showing that academic coaching and peer tutoring helped struggling students, the UAMS COM implemented a policy in 2019 requiring students who score 75% or less on three or more major course exams to meet with a learning specialist in the Student Success Center and develop an individualized study plan to support academic success. A student's individualized study plan encourages evidence-based active study methods and the use of practice questions and may include regular academic coaching and peer tutoring.Study design and participantsThe study, which used a correlational design, was approved by the University of Arkansas for Medical Sciences Institutional Review Board for Clinical Research Administration (CLARA) as not human subjects research (IRB number 273939). Study participants were medical students from two cohorts, the cohort graduating in 2022 (N=164) and the cohort graduating in 2023 (N=163), who successfully completed their first and second  years and took the USMLE Step 1 exam.These two cohorts had similar demographic characteristics and similar scores on the Medical College Admission Test (MCAT) entry exam. Differences between the cohorts included the fact that the 2022 students faced random delays in taking their Step 1 exams due to COVID-related issues, and the 2023 students attended their second year of medical school entirely online .AnalysisStatistical analysis was performed using SPSS . Data were analyzed using Pearson correlations to assess associations between Step 1 scores and the number of questions completed and videos watched. Multiple linear regression was used to assess the prediction of the dependent variable, USMLE Step 1 score, by a set of independent variables (predictors): MCAT scores, first-year in-class exam scores, second-year in-class exam scores, number of questions completed, and number of videos watched. All probability (p) values were two-sided, with the alpha level set at 0.05.Descriptive statistics of cohorts and correlationsCohorts 2022 and 2023 were similar in terms of first-year in-class exam scores, the number of practice questions completed, videos watched, and MCAT scores Table  but diffPearson correlations for cohort 2022 among the first- and second-year averages of in-class exams, number of practice questions completed, number of videos watched, and USMLE Step 1 score are presented in Table Pearson correlations for cohort 2023 among first- and second-year averages of in-class exams, number of practice questions completed, number of videos watched, and USMLE Step 1 score are presented in Table For both cohorts, working on more practice questions was associated with higher Step 1 scores, while watching more videos was associated with lower Step 1 scores. For cohort 2022, the number of practice questions completed was positively correlated with Step 1 score r=.176, α=.05, and the number of videos watched was negatively correlated with Step 1 score r = -.294, α=.01. For cohort 2023, the number of practice questions completed was positively, though not significantly, correlated with Step 1 score r=.143, and the number of videos watched was negatively correlated with Step 1 score, r= -.175, α=.05.Multiple linear regressionA simultaneous multiple regression was performed to investigate the contribution of each independent variable  in predicting the dependent variable (Step 1 score) for cohorts 2022 and 2023. For both cohorts, the model explained a statistically significant amount of variance in the Step 1 score. For cohort 2022, R2=.537, F=36.17, p<.001, and for cohort 2023, R 2=.571, F=41.81, p<.001.For cohort 2022, the second-year average of in-class exams was the strongest predictor of the Step 1 score . MCAT was the second predictor , and the third predictor was the number of questions worked . Watching videos was a negative predictor.  but not statistically significant Table .For cohort 2023, the second-year average of in-class exams was the strongest predictor of Step 1 score . MCAT was the second predictor , and the third predictor was the number of questions worked . Watching videos was a negative predictor and was statistically significant for this cohort  Table .In summary, for both cohorts, the second-year average of in-class exams was the strongest predictor of Step 1 scores. MCAT was the second predictor of Step 1 scores for both cohorts. The number of practice questions was a significant positive predictor of Step 1 scores. Videos were significant negative predictors for cohort 2023; for cohort 2022, videos were also a negative predictor but not statistically significant. First-year averages of in-class exams were not statistically significant predictors for Step 1 for both cohorts.The purpose of this study was to investigate if students utilizing the active study strategy of using practice questions was associated with earning higher scores on the USMLE Step 1 board exam compared with students using the passive study strategy of watching educational videos. The findings were that the active study strategy of working practice questions was associated with higher Step 1 scores, and passively watching educational videos was associated with lower Step 1 scores.Other studies have addressed the importance of retrieval practice in preparing for exams ,12,15,21The findings from this study can help inform study preparation for any medical knowledge tests, even though other medical schools may use different board exams. This study looked specifically at Step 1 board exam scores, which were reported as three-digit scores until January 2022. Step 1 is now reported as pass/fail only, but the importance of maximizing test performance remains ,17,18.Using a question bank with high-quality questions prompts cognitive retrieval practice and students' critical thinking, which are important for all medical students. The quality of questions in a question bank is vital to helping students prepare for in-class exams and board exams. Whether questions are designed by faculty or are in a commercial question bank, of chief importance is that the questions are well-written, constructively peer-reviewed, and contain content that prompts critical thinking and analysis .In comparison, educational video content varies in that some videos contain optional self-assessments while others do not. Additionally, it is not possible to know if or how students are engaging in the video material. Furthermore, passive listening and watching may not offer the same learning advantage as active study methods such as working questions . It may Results of the multiple regression analysis indicate that the strongest predictor of the Step 1 score is students' performance on the second-year in-class exams, which suggests that engaging students in the curriculum should be prioritized. Not surprisingly, the MCAT score is a predictor of the Step 1 score, as previous preparation and test-taking skills may support the transition to medical school. The number of practice questions worked is also a positive predictor of Step 1 scores, which suggests that students should be encouraged to use active study strategies such as working questions. The number of informational videos watched was a negative predictor of Step 1 scores, which suggests that students should be urged to limit watching videos and instead opt for more active study strategies.One limitation regarding data on students watching videos is that there is little research about students' attention and engagement while videos are playing. Additionally, little is known about what other resources students may be using to study beyond those recommended by the medical school.A strength of the study was the specific focus on active versus passive study strategies and their potential impact on preparation for a critically important board exam. Additionally, the study habits were not from self-reported survey results but were observed by student usage data of resources that were equitably available to all students. It should be noted that students' educational experiences were impacted by the COVID-19 pandemic, and future research may investigate the long-term effects of educational disruption, physical illness, and mental stressors due to the pandemic. The results of this study support the premise that answering practice questions was an effective study method, but add a notable distinction regarding active versus passive study methods employed by students. Unique to this study was the finding that watching educational videos had a negative correlation with Step 1 scores. Medical students should be urged to make the most effective use of study time by limiting passively watching educational videos and instead actively incorporating the regular use of practice questions into their study.


Photoacoustic (PA) imaging can be used to monitor high-intensity focused ultrasound (HIFU) therapies because ablation changes the optical absorption spectrum of the tissue, and this change can be detected with PA imaging. Multi-wavelength photoacoustic (MWPA) imaging makes this change easier to detect by repeating PA imaging at multiple optical wavelengths and sampling the optical absorption spectrum more thoroughly. Real-time pixel-wise classification in MWPA imaging can assist clinicians in monitoring HIFU lesion formation and will be a crucial milestone towards full HIFU therapy automation based on artificial intelligence. In this paper, we present a deep-learning-based approach to segment HIFU lesions in MWPA images. Ex vivo bovine tissue is ablated with HIFU and imaged via MWPA imaging. The acquired MWPA images are then used to train and test a convolutional neural network (CNN) for lesion segmentation. Traditional machine learning algorithms are also trained and tested to compare with the CNN, and the results show that the performance of the CNN significantly exceeds traditional machine learning algorithms. Feature selection is conducted to reduce the number of wavelengths to facilitate real-time implementation while retaining good segmentation performance. This study demonstrates the feasibility and high performance of the deep-learning-based lesion segmentation method in MWPA imaging to monitor HIFU lesion formation and the potential to implement this method in real time. High-intensity focused ultrasound (HIFU) is a minimally invasive technique for in situ tissue destruction . In HIFUMRI has been used to monitor HIFU for breast fibroadenoma and uterine fibroids ,17. It pWhile MRI mainly monitors the thermal effect caused by HIFU, US imaging focuses more on the mechanical effect, which produces a hyperechoic spot at the focal point of the HIFU transducer . This reSince both thermal and mechanical effects fade gradually after the cessation of HIFU exposure, an imaging modality that focuses on monitoring the permanent tissue property change caused by HIFU could greatly supplement or even replace MRI and US imaging. Photoacoustic (PA) imaging is a hybrid imaging modality that combines the contrast of optical absorption with the resolution and penetration depth of US imaging ,31,32,33Semantic segmentation is a task of simultaneous object recognition and pixel-wise classification . The pasA programmable imaging system , a personal computer (PC) , a linear US transducer array , an HIFU transducer , and a programmable laser system  with a customized dual-output laser fiber were used in the experiment . The laste frame . In ordeEx vivo bovine tissue (sirloin) was cut into blocks (approximately 4.0 cm × 3.0 cm × 2.5 cm) as imaging specimens. The tissue block was then placed in a water bath on top of an agar backdrop at room temperature for imaging. Placing the sample on this agar block eliminates possible image artifacts due to possible reflections from the back side or light absorption in the background. The system performed the first round of MWPA and US data acquisition . Next, HMWPA and US images were reconstructed from the acquired data with the standard delay-and-sum algorithm . The imaThe ablated tissue was cut open after the data acquisition process was finished, and then color photographs of the cross-sections of the ablated tissue were taken. By visually matching the features in the color photographs and the US images and leveraging the co-registration of the US and MWPA images, ablated and non-ablated pixels were manually labeled to generate training and test examples for the learning algorithms . The redT, and The amplitude of PA pressure waves  formula ,38:(1)p0N is the system noise.Assuming the imaging system is linear , the pixThe optical fluence lated as :(3)Φ:(4)v:(4)v3):,,6), we cThe MWPA images acquired from the 19 blocks of bovine tissue were used as training and test examples for the learning algorithms. In total, 15 were used as training examples and 4 as test examples.A total of four machine learning algorithms including quadratic discriminant analysis (QDA), neural network (NN), support vector machine (SVM), and random forest (RF) were used for lesion segmentation. Scikit-learn  was usedEach pixel in the MWPA image was processed independently. Since the ablated portion was always smaller than the non-ablated portion in each MWPA image, there was an imbalance between the numbers of ablated and non-ablated pixels. Machine learning algorithms tend to produce unsatisfactory classifiers when faced with imbalanced datasets. Therefore, the non-ablated pixels were under-sampled to make the training examples balanced. For each MWPA image, all the ablated pixels were used as positive training and test examples, and an equal number of non-ablated pixels were selected randomly as negative examples. This approach eliminated the imbalance between positive and negative examples in the training data set.F1 score [F1 score, a commonly used metric to evaluate the performance of a classifier, is the harmonic mean of recall and precision. Assuming that in the segmentation result on a MWPA image, Neural network, support vector machine, and random forest involved some parameters that needed to be customized. To find the optimal parameter value, the training set (15 blocks) was divided into two groups, one group  for training and the other group  for validation. Tentative models with different parameters were trained on group A and tested on group B to calculate the model’s F1 score , and theF1 score is calculated as:The F1 score of a segmentation algorithm is the average value of the algorithm’s F1 scores on the MWPA images used as test examples.The F1 score would change as the number of hidden layer nodes increased. Based on the F1 score vs. number of hidden layer nodes curve  curve  curve b, C was F1 score would change as the number of trees increased. Based on the F1 score vs. number of trees curve  as the min_samples_leaf value increased from 100 to 5000  were tr to 5000 d, and so to 5000 e. In ordThe CNN was composed of 6 layers . Layers imension , and theLayers 4 and 5 did not contain max-pooling layers and their convolution kernel size was 1 × 1 , so layeThe CNN was trained by minimizing a loss function. The loss function calculated a weighted average cross entropy between the output of the network and the human labeling. For non-ablated pixels, the weight was 1. For ablated pixels, the weight was the ratio of the number of non-ablated to ablated pixels. For unlabeled pixels, the weight was 0. This strategy helped minimize the impact of the imbalance between ablated and non-ablated pixels. An Adam optimizer  was usedData augmentation was used to generate more training examples for the CNN because it required significantly more training examples than traditional machine learning algorithms. Each image was blurred with a 3 × 3, 5 × 5, and 7 × 7 window and also corrupted with 2 versions of random Gaussian noise . AdditioScanning 53 wavelengths with a 20-Hz repetition-rate laser system would result in a frame rate smaller than 0.5 frames per second (FPS), even when system channels could address all transducer elements concurrently and no signal averaging is employed. To facilitate real-time implementation, wavelength selection was conducted to reduce the number of wavelengths that were needed to segment HIFU lesions in MWPA images. The built-in sequential feature selection function in MATLAB   was usedF1 score of each algorithm is calculated for the test examples  require longer training times and thus it is difficult to identify parameter values that are comprehensively optimal. Further data analysis is necessary in order to extend beyond the application level and explore what features distinguish ablated and non-ablated tissue apart.HIFU lesion formation can also be monitored by measuring temperature in the image view. MRI has superb temperature-mapping capability and is widely used for temperature measurement during HIFU therapies. Due to the linear dependence of the PA signal amplitude on temperature and the linearity of the PA imaging system, temperature can be measured indirectly from PA imaging with proper calibration . A drawbWhen running on a PC with two 16-core CPUs, the inference time of quadratic discriminant analysis, neural network, random forest, and CNN is less than one second . So, all3) [Compared with the other algorithms, the inference time of quadratic discriminant analysis is significantly shorter because it assumes that the distribution of each category of pixels is a single Gaussian distribution, and this model is computationally simpler. The inference time of support vector machine is significantly longer. This is because its inference time complexity is O(n3)  where n In our experiment, a commercial transducer probe (L7-4) is used for PA data acquisition. In order to increase the SNR, multiple (100) acquisitions are averaged for each frame. A 2D CMUT with integrated electronics has been shown to achieve high SNR in PA imaging  and is aThe MWPA images were not fully labeled since we labeled the ablated pixels manually mainly by visual inspection of the tissue cross-section after completing the experiment. With this information, it is not possible to label every pixel with complete confidence. The difficulty lies not only in distinguishing ablated from non-ablated tissue in the tissue picture but also in co-registering the tissue picture with the PA and US images since deformation is inevitable when cutting the tissue blocks. Furthermore, we could not guarantee that the cut plane perfectly overlaps with the imaging plane. An imaging modality that can provide a gold-standard reference (such as MRI) on lesion segmentation would improve the experiment and provide stronger validation of our approach.In this paper, we presented a CNN-based approach to segment HIFU lesions in MWPA images. Traditional machine learning algorithms were also trained and tested to compare with the CNN-based approach, and the results show that the performance of CNN significantly exceeds traditional machine learning algorithms. Feature selection was conducted to reduce the number of wavelengths to facilitate real-time implementation. The performance of the CNN with 2 wavelengths exceeds the traditional machine learning algorithms with 53 wavelengths. This work demonstrates the high performance of CNN in HIFU lesion segmentation and the potential to operate it in real time. Further work includes running the CNN with accelerating hardware to assess its capability in monitoring HIFU lesion formation in real time and potentially in vivo, as well as further improving the segmentation performance with more sophisticated neural networks.


The COVID-19 pandemic has affected the entire global healthcare system, including oncological care. This study investigated the effects of the COVID-19 pandemic on the diagnosis, stage, and treatment of esophagogastric cancer in the Netherlands.Patients diagnosed in 2020 were divided into 5 periods, based on the severity of the COVID-19 pandemic in the Netherlands, and compared to patients diagnosed in the same period in the years 2017–2019. Patient characteristics and treatments were evaluated for esophageal cancer (EC) and gastric cancer (GC) separately.The number of esophagogastric cancer diagnoses decreased prominently during the first 2 months of the COVID-19 pandemic. During this period, a significantly higher percentage of GC patients was diagnosed with incurable disease . We observed a significant reduction in the percentage of patients with potentially curable EC treated with resection and neoadjuvant chemoradiotherapy . Also, patients diagnosed with incurable GC were treated less frequently with a resection  in the second half of 2020.Compared to previous years, the number of esophagogastric cancer diagnoses decreased in the first 2 months of the COVID-19 pandemic, while an increased percentage of patients was diagnosed with incurable disease. Both in the curative and palliative setting, patients were less likely to be treated with a surgical resection. The online version contains supplementary material available at 10.1007/s00535-023-02009-3. Since December 2019, the Coronavirus Disease 2019 (COVID-19) pandemic caused by the SARS-CoV-2 virus has greatly impacted health care systems around the world. Many resources were preferentially dedicated to COVID-19 patients, forcing clinicians to make difficult triage decisions. In the Netherlands, the first COVID-19 patient was diagnosed on February 27, 2020 . In ApriA diagnostic delay is of particular concern in cancers of the esophagus, esophagogastric junction, and stomach, together also known also as esophagogastric cancer. Esophagogastric cancer has a poor prognosis and accounted for 13.2% of all cancer deaths worldwide in 2020 . PresentFor inclusion, patients diagnosed with esophageal cancer (EC) or gastric cancer (GC) between January 1, 2017 and December 31, 2020 were selected from the Netherlands Cancer Registry (NCR). The NCR is a nationwide population-based cancer registry that covers the entire Dutch population of more than 17 million people. The NCR is directly linked to the national pathological archive (PALGA) wherein all histologically confirmed cancer diagnoses are registered . Every pThe year 2020 was divided into 5 periods based on the severity of the COVID-19 pandemic in the Netherlands. Period 1 spanned weeks 1–8 ; period 2 included weeks 9–12 (after the first COVID-19 case and before the lockdown); period 3 week 13–17 (during lockdown); period 4 weeks 18–26 ; and period 5 encompassed weeks 27–52 . Patients were categorized into one of the five time periods based on the date of diagnosis for all analyses.Esophagogastric cancer was classified into early carcinoma ; potentially curable disease , palliative disease (cT4B or cM1), and unknown disease severity according to the TNM staging system for analysis of changes in disease stage . PatientIn the Netherlands, treatment of esophagogastric cancer is selected based on the guidelines of the Dutch Federation of Medical Specialists , 17. ForFor patients with potentially curable GC, treatment selection is based on tumor stage, performance status, operability of the patient, and patient preference . TreatmeSurvival in patients with incurable disease and a proper performance status can be improved with chemotherapy combined with targeted therapy. Surgery, short-course radiotherapy or placement of stents may be considered to palliate symptoms , 17. PatTime to start treatment was defined as the time from diagnosis until the start of any treatment indicated, including chemo (radio) therapy and resection. The time until direct resection was defined as the time from diagnosis until primary surgery in patients without any (neo) adjuvant treatment. The time until surgery was defined as the time between the end of neoadjuvant treatment and resection.p < 0.05 was considered statistically significant. All analyses were performed using R version 4.1.1 software [Comparisons were performed between the five periods in 2020 and the corresponding periods in 2017–2019. Tumor characteristics and treatments were displayed as counts and percentages and chi-squared tests were used to compare the percentages between groups. Treatments were compared separately for the potentially curable setting and the palliative setting for both EC and GC. The time between diagnosis, neoadjuvant treatment, and resection was represented as the mean with the standard deviation (SD). Times were assessed separately for the potentially curable and the palliative setting for both EC and GC. Two-sided unpaired Wilcoxon tests were used for comparison between 2017 and 2019 and 2020 for all periods. Two-year overall survival was analyzed using Cox regression per group  and correcting for sex, age, comorbidities, and performance status. Survival data was censored after 731 days to ensure that the follow-up data of all patients were comparable. A software .). Age, histology, stage, and comorbidities were evenly distributed between patients diagnosed in 2020 and patients diagnosed in 2017–2019. In 2020, there were significantly more female EC patients (p = 0.05) and significantly more GC patients had a worse performance status (p = 0.03). Additional patient characteristics for all periods separately are shown in Online Resource 1.In total, 15,715 patients were included in this study, of whom 9,959 EC patients and 5,756 GC patients (Table 1n = 2523 average in 2017–2019 and n = 2391 in 2020), whereas the incidence of GC diagnoses remained stable (n = 1442 average in 2017–2019 and n = 1431 in 2020). When analyzing the five separate periods, the percentage of GC patients diagnosed with incurable disease increased significantly  after the first COVID-19 patient was diagnosed in the Netherlands . No significant difference was found in the distribution of stages of disease at time of diagnosis over the entire year 2020 compared to 2017–2019.The absolute number of diagnoses decreased most prominently in the periods before and during the social lockdown Fig. . The totp < 0.001) and a significant increase in the percentage of patients treated only nCRT . The difference in nCRT-ER and nCRT was most prominent in patients diagnosed during and after the lockdown . This increase was seen both in patients with histologically diagnosed esophageal squamous cell carcinomas and adenocarcinomas (28.8% in 2017–2019 compared to 35.7% in 2020 and 8.6% in 2017–2019 and 14.7% in 2020 respectively). No difference in the percentage of patients treated with dCRT was observed over the whole year of 2020 compared to 2017–2019 (p = 0.10). For the patients with potentially curable GC, there was no significant difference in treatment over the entire year. However, a similar decreasing trend in CT-GR was observed after the first COVID-19 patient was diagnosed in the Netherlands  with a concurrent increase in CT or GR alone . A similar pattern was seen in the group of GC patients in the palliative setting with relatively more patients being treated with chemotherapy in the second half of 2020 . There was no difference in the time until direct resection for patients suffering from GC or EC in any of the periods in 2017–2019 as compared to 2020.The time to start treatment was significantly shorter during and after the COVID-19 outbreak in the Netherlands both in the potentially curative setting and in the palliative setting for EC Table  and GC pp = 0.04). GC patients diagnosed with curable disease in the period just before the lockdown had a significantly worse 2-year overall survival compared to patients diagnosed in the same period in the previous years . There was no difference in 2-year overall survival for EC patients diagnosed with potentially curable disease or GC patients diagnosed with incurable disease in any of the periods or when comparing the whole year 2020 to 2017–2019 (Online Resource 2).EC patients diagnosed with incurable disease in the period after the lockdown had a significantly better 2-year overall survival compared to patients diagnosed in the same period in previous years  trial, where EC patients with a clinical complete response after nCRT underwent active surveillance  . The steThe differences in treatment during the COVID-19 pandemic did partially influence survival. Patients diagnosed with incurable EC were treated more frequently with chemotherapy in the months after the social lockdown. A concurrent decrease in EC patients receiving other treatments such as best supportive care in the palliative setting was found. This shift in treatment toward chemotherapy resulted in a significantly better 2-year overall survival for EC patients in the palliative setting. As mentioned before, patients diagnosed with incurable GC in the second half of 2020 were less likely to undergo a palliative gastric resection. We observed a concurrent increase in the percentage of incurable GC patients treated with chemotherapy. This decrease in palliative resections did not impact overall survival, as a gastrectomy for advanced gastric cancer is known to have no added survival benefit . For patWe also observed differences in the time to and between treatments. Interestingly, we found that the time between diagnosis and start of treatment was significantly shorter for most patients in 2020. Possibly the decrease in the number of patients diagnosed with esophagogastric cancer allowed for faster scheduling in the outpatient clinic. The clinical relevance of starting treatment 1 week sooner is debatable, as our results also show that overall survival was not different for almost all subpopulations.While time between diagnosis and start of treatment was significantly shorter, time to surgery after neoadjuvant treatment was significantly longer in the second half of 2020. Since the effect of longer time to surgery was not present during the first half of 2020, it is again unlikely that the SANO trial is the only explanation for the increased time to surgery. Probably, scheduling difficulties occurred in the second half of 2020 due to upscaling of healthcare. In line with the existing literature, we found no impact on overall survival for a longer time to surgery after nCRT –33.This study is the first to show the effect of the COVID-19 pandemic on the incidence, stage, and treatment of esophagogastric cancer. It allowed for a nationwide investigation of not only the incidence and stage of patients with esophagogastric cancer but also the types of treatment that were given. A limitation may be the relatively low number of reference years, which posed some statistical difficulties when comparing the absolute number of diagnoses and resections. We chose these reference years, since they were most comparable due to the changing epidemiology of esophagogastric cancer in the previous years . This apIn conclusion, the incidence of esophagogastric cancer diagnoses decreased significantly during the COVID-19 pandemic in the Netherlands. Relatively more incurable patients with worse performance scores were diagnosed. In addition, we found evidence that fewer surgical resections were performed.Supplementary file 1.Below is the link to the electronic supplementary material.


In low- and middle-income countries (LMICs), hospitals can rarely utilize their own antimicrobial resistance (AMR) data in a timely manner.To evaluate the utility of local AMR data generated by an automated tool in the real-world setting.From 16 December 2022 to 10 January 2023, on behalf of the Health Administration Division, Ministry of Public Health (MoPH) Thailand, we trained 26 public tertiary-care and secondary-care hospitals to utilize the AutoMated tool for Antimicrobial resistance Surveillance System (AMASS) with their own microbiology and hospital admission data files via two online meetings, one face-to-face meeting and online support. All meetings were recorded on video, and feedback was analysed.Escherichia coli (CREC) was 129 (range 0–1204), by carbapenem-resistant Klebsiella pneumoniae (CRKP) was 1306 (range 0–5432) and by carbapenem-resistant Acinetobacter baumannii (CRAB) was 4472 (range 1460–11 968) per 100 000 patients tested for hospital-origin BSI. The median number of all-cause in-hospital deaths with hospital-origin AMR BSI caused by CREC was 1 (range 0–18), by CRKP was 10 (range 0–77) and by CRAB was 56 (range 7–148). Participating hospitals found that the data obtained could be used to support their antimicrobial stewardship and infection prevention control programmes.Twenty-five hospitals successfully generated and shared the AMR reports with the MoPH by 28 February 2023. In 2022, the median frequency of hospital-origin bloodstream infections (BSIs) caused by carbapenem-resistant Local and timely AMR data are crucial for local and national actions. MoPH Thailand is inviting all 127 public tertiary-care and secondary-care hospitals to utilize the AMASS. Using any appropriate analytical software or tools, all hospitals in LMICs that have electronic data records should analyse and utilize their data for immediate actions. In March 2023, only the cumulative antibiogram from each participating hospital up to 2020 is available online.5 The cumulative antibiograms are not stratified by infection origin, do not include other parameters  and could not be easily compared between hospitals or areas.In low- and middle-income countries (LMICs), there is some national coordination for an AMR surveillance system; however, having a system that is able to generate local and reliable information on AMR and feedback to local hospitals in a timely manner is still a challenge. For example, in Thailand, 92 surveillance sites participated in the National Antimicrobial Resistance Surveillance Thailand (NARST) and 10 surveillance sites provided data to Global Antimicrobial Resistance Surveillance System (GLASS) in 2020.6 The automatically generated reports stratify by infection origin into community origin and hospital origin based on the recommendations of WHO’s GLASS,7 and provided additional reporting on mortality (%) and total number of deaths with AMR and non-AMR bloodstream infections (BSIs). We recently reported the difference of frequency of AMR BSI between tertiary-care hospitals (TCHs) and secondary-care hospitals (SCHs) using the AMASS v2.0 and retrospective data from 2012 to 2015.8The AutoMated tool for Antimicrobial resistance Surveillance System (AMASS), an offline application to generate standardized AMR surveillance reports from routinely available microbiology and hospital data files, was developed and independently tested in seven hospitals in seven countries.9 In addition, the data of patients with AMR infection can be used as measures/indicators to monitor and guide the antimicrobial stewardship (AMS)10 and IPC programmes.11 Therefore, we consider that the availability and timeliness of local AMR data is one of the most important aspects for local hospitals, national authorities and policymakers. Here, we aim to evaluate the utility of local AMR data generated by the AMASS in the real-world setting in Thailand.The national action plan on AMR in Thailand aims to reduce AMR morbidity by 50%, and the analysis of baseline data from target hospitals is included in the plan.2. Health regions in Thailand were divided into 12 groups of provinces plus Bangkok, totalling 13 regions, using the concept of decentralization.12 Health Administration Division, Ministry of Public Health (MoPH) Thailand, supervised 127 public TCHs and SCHs in health regions 1 to 12 countrywide. These included 35 advanced-level referral hospitals , 55 standard-level referral hospitals  and 37 mid-level referral hospitals .13 Based on the WHO definitions, class A hospitals are generally TCHs and referred to as regional hospitals,14 and classes S and M1 are generally SCHs and referred to as provincial or large district hospitals. All class A and S hospitals, and most of class M1 hospitals, are equipped with a microbiology laboratory capable of performing bacterial culture using standard methodologies for bacterial identification and susceptibility testing provided by the Bureau of Laboratory Quality and Standards, Department of Medical Sciences, MoPH, Thailand.In 2022, Thailand had a population of 66.1 million, consisted of 77 provinces, and covered 513 120 kmFrom 16 December 2022 to 10 January 2023, on behalf of the Health Administration Division, MoPH, we invited and trained 26 TCHs and SCHs to utilize the AMASS with their own microbiology and hospital admission data files via two online meetings, one face-to-face meeting and online support. These 26 hospitals were invited because they were highly active in previous activities with the MoPH; 1 to 4 hospitals were selected from each health region, and the MoPH aimed to have them train other hospitals to utilize the AMASS in the next phase.6 and (iii) informing participants about data files that they needed to export from their own hospital information system (HIS) and laboratory information system (LIS) to be used in the next session.The first online meeting focused on (i) informing participants about the AMASS; (ii) teaching participants to be able to download, unzip and successfully run the AMASS with the example datasets provided in the AMASS;The second online meeting focused on supervising hospitals to run the AMASS with their own hospital datafiles; including how to (i) complete the data dictionary files of the AMASS; (ii) verify the log files to understand whether the AMASS automatically imported and analysed the data accurately or not; and (iii) read the AMR surveillance reports automatically generated by the AMASS. We emphasized that hospitals should not transfer raw patient-level data files to the study team.Escherichia coli (CREC), carbapenem-resistant Klebsiella pneumoniae (CRKP) and carbapenem-resistant Acinetobacter baumannii (CRAB) with other participating hospitals; (iii) and discussion on how to utilize the local and timely AMR reports generated by the AMASS for local actions.The face-to-face meeting focused on (i) validation of the generated AMR reports; (ii) comparison of multiple AMR parameters  of each microbiology laboratory and HIS to participate. For the face-to-face meeting, we additionally requested representatives from AMS and IPC teams to participate.In the second online and face-to-face meetings, we asked participants whether the data in their AMR reports generated by the AMASS were proportional to their workload  and their AMR situation  they observed in their own hospitals. We also asked whether they found the AMASS and the AMR reports useful or not.15In the face-to-face meeting in January 2023, we asked each participating hospital to generate the AMR surveillance report using their own hospital admission and microbiology laboratory data files of the year 2022 and the AMASS v2.0.This was a retrospective study describing the epidemiology of AMR among participating TCHs and SCHs in Thailand, and the utility of local and timely data generated by the AMASS. All meetings were recorded on video, and feedback was analysed by P.K. and K.T.7 Patient hospital number (HN) was automatically used as a record linkage between the microbiology and hospital admission data files. A deduplication process was automatically conducted in which only the first isolate of a species per patient per specimen type per survey period was included.AMASS analysed the data and generated reports based on the recommendation of the WHO GLASS.E. coli (3GCREC) and K. pneumoniae (3GCRKP), CREC, CRKP, carbapenem-resistant Pseudomonas aeruginosa (CRPA) and CRAB, which are in the WHO GLASS priority list of AMR bacteria and are locally important. Only blood culture isolates were included in the analysis of the AMASS v2.0 and in this study.For AMR infections, we analysed the following organisms: MRSA, third-generation cephalosporin-resistant Community-origin  BSIs were defined for patients in the hospital within (longer than) the first two calendar days of admission when the first blood specimen culture positive for a pathogen was taken, with calendar day one equal to the day of admission. Patients were considered at risk for hospital-origin BSI after they stayed in the hospital for more than two calendar days.6 and per 100 000 admissions during the reporting period. The frequency of hospital-origin AMR BSI was calculated as the total number of new patients with hospital-origin AMR BSI per 100 000 patients tested for hospital-origin BSI ,6 per 100 000 admissions and per 100 000 patient-days at risk for hospital-origin BSI during the reporting period. All-cause in-hospital mortality was defined using the discharge summary regularly completed by the attending physicians and reported to the MoPH.The proportion of AMR (%) was calculated as the percentage of patients with new AMR BSI over all patients with new BSIs for the pathogen of interest during the reporting period. The frequency of community-origin AMR BSI was calculated as the total number of new patients with community-origin AMR BSI per 100 000 patients tested for community-origin BSI P values were given to two significant figures, and no longer than four decimal places (e.g. P < 0.0001). Box and dot plots were used to illustrate the distribution of data. We used STATA  for the final statistical analyses and R version 4.0.5 for figures.We compared all continuous variables using the Mann–Whitney test. https://doi.org/10.6084/m9.figshare.22339933.The anonymous AMR surveillance reports generated from participating hospitals are open access and available at Ethical permission for this study was obtained from the Institute for the Committee of the Faculty of Tropical Medicine, Mahidol University (TMEC 23-011). Individual consent was not sought from the patients as this was a retrospective study, and the Ethical and Scientific Review Committees approved the process.Of 26 hospitals invited to the early implementation of the AMASS, all generated and shared their AMR surveillance reports for the year 2022 with the MoPH by 28 February 2023. However, the AMR report from anonymous hospital no. 1 was excluded because the data did not pass validation by the study team. The AMR reports of the other 25 hospitals were validated and included in further analysis  and 10 (42%) were classified as TCHs and SCHs, respectively. Of 10 SCHs, 9 and 1 were classified as class S and M1 by the MoPH Thailand. The median bed number was 860 (range 527–1387) in TCHs and 414 (range 282–535) in SCHs (JAC-AMR Online).In 2022, for community-origin BSI, the median proportion of MRSA was 6% (range 0%–23%), of 3GCREC was 34% (range 18%–49%), of 3GCRKP was 22% (range 10%–46%), of CREC was 1% (range 0%–25%), of CRKP was 7% (range 1%–20%), of CRPA was 14% (range 0%–79%) and of CRAB was 50% (range 13%–100%). The median proportion of AMR was not different between TCHs and SCHs for all pathogens , of 3GCREC was 58% (range 35%–79%), of 3GCRKP was 67% (range 0%–92%), of CREC was 7% (range 0%–42%), of CREKP was 37% (range 0%–72%), of CRPA was 36% (range 0%–100%) and of CRAB was 87% (range 61%–94%). The median proportion of AMR for 3GCRKP, CREC and CRKP was higher in TCHs than in SCHs , by 3GCREC was 1039 (range 333–2525), by 3GCRKP was 260 (range 79–527), by CREC was 50 (range 0–325), by CRKP was 75 (range 19–265), by CRPA was 41 (range 0–1556) and by CRAB was 137 (range 34–294). The median frequency of AMR BSI was not different between TCHs and SCHs for all pathogens Figure .The median frequency of hospital-origin BSI caused by MRSA per 100 000 patients tested for hospital-origin BSI was 132 (range 0–857), by 3GCREC was 1269 (range 427–2877), by 3GCRKP was 2363 (range 0–8045), by CREC was 129 (range 0–1204), by CRKP was 1306 (range 0–5432), by CRPA was 447 (range 0–1331) and by CRAB was 4472 (range 1460–11 968). The median frequency of AMR BSI for 3GCRKP, CREC, CRKP and CRAB was higher in TCHs than in SCHs  and anonymous hospitals no. 16 and 25 (among SCHs) had the highest frequency Figure . We alsoWe also estimated the frequency of hospital-origin AMR BSI per 100 000 admissions and per 100 000 patient-days at risk for hospital-origin BSI (data not shown), and similar findings were observed.P > 0.10), except that the median all-cause in-hospital mortality of patients with community-origin CRKP BSI was higher in TCHs than in SCHs .The median all-cause in-hospital mortality (%) of patients with hospital-origin AMR BSI caused by MRSA was 42% (range 0%–100%), by 3GCREC was 33% (range 0%–67%), by 3GCRKP was 55% (range 24%–100%), by CREC was 50% (range 0%–100%), by CRKP was 57% (range 10%–100%) and by CRAB was 64% (range 26%–100%). Difference in the all-cause in-hospital mortality (%) of patients with AMR BSI caused by each AMR pathogen was not observed between TCHs and SCHs  nurses, 7 pharmacists, 7 information technology staff, and 2 medical doctors participated in the training and provided feedback. All microbiology laboratory staff responded that the cumulative antibiograms obtained from the AMASS would be useful for their reports to their AMS and IPC teams and to national authorities. All medical doctors, IC nurses and pharmacists agreed that the summary data reported by the AMASS could be used to support their AMS and IPC programmes.This study demonstrates the utility of local and timely AMR data generated by the AMASS for local and national actions in the real-world setting. The utility of the data is shown by the ability to compare epidemiology data of participating hospitals in a timely manner and by feedback from the participating hospitals. The MoPH Thailand is inviting all 127 public TCHs and SCHs to utilize the AMASS in 2023 and planning to update the AMR dashboard by using data generated from the AMASS nationwide.The AMASS empowers hospitals to analyse, generate and share their own cumulative antibiograms, together with their total number of patients and deaths with AMR BSI, in a quick manner (i.e. within 2 months after the end of the year) and with minimal resources required . The AMR data obtained highlight the significant burden of AMR infections in participating hospitals in 2022 and the urgency to step up actions against AMR infections countrywide.10 and IPC programmes.11 The observed numbers of patients and deaths with AMR BSI in their own hospitals last year can be used to communicate with all levels of healthcare workers, including the directors and executive boards of the hospitals, better than the modelling data, national estimates or global estimates.16 Some participating hospitals are generating their own AMR reports independently using the data from previous years , discussing the findings among staff in their own hospitals, and sharing those reports with the MoPH.At the local hospital level, the frequency of hospital-acquired AMR BSI and other data of patient outcomes  can be used as measures/indicators to monitor and guide their AMSAt the national level, the ability to obtain and compare reliable AMR data between hospitals in a quick manner is crucial for designing interventions and resource allocation. If the AMASS were utilized by all TCHs and SCHs in Thailand, the policymaker could understand the situation of the AMR in the country, in each region and in each hospital much more clearly. Utilization of the AMASS can also support participating hospitals to join the WHO GLASS Thailand.9 the National Drug Policy and National Drug System Development Strategy,17 the Rational Drug Use Service Plan,18 and the national plan to achieve the sustainable development goals (SDGs) in Thailand.19The future version of the AMASS aims to include automatic analysis of electronic drug prescription data and generate reports on diagnostic and antibiotic use practices. The availability of data of AMR burden, diagnostic practice and antibiotic use practice at the local hospital level nationwide will support the national action plan on AMR,20 The all-cause in-hospital mortality reported by the AMASS21 can be used by modelling studies together with studies using recommended methods for estimating attributable mortality of AMR BSI.22–25At the international level, the data from hospitals of all sizes in LMICs can be helpful for modelling studies estimating the global burden of AMR.8 The frequency observed in this study is relatively higher than that previously observed.8 The change of the AMR burden over time will be studied separately using the data of 127 public TCHs and SCHs in 2022.The higher frequency of hospital-origin AMR BSI in TCHs compared with SCHs observed in this study is consistent with the previous study using the data of 49 hospitals from 2012 to 2015.Escherichia coli, E. coli, ESBL E. coli and ESBL-producing E. coli). For AMASS users, we emphasize the importance of completing the dictionary files and validating the AMR reports with log files, as well as comparing the data obtained in the report with their previously known workload , their previous AMR reports (if available) and manual data calculation.26The AMASS has some common issues that users face. Firstly, users need to change the Windows region settings from Thailand to either US or UK because the AMASS cannot run properly with the Windows region setting as Thailand. Secondly, dates must not be exported as date time variables. Users need to export dates as date variables or generate new variables in the Excel files as date variables prior to running the AMASS. Lastly, users need to complete the dictionary files using the log files as guidance. The completion of the dictionary files is crucial because it is not uncommon for each bacterial species to be recorded in the LIS with multiple names  reported in this study was all-cause in-hospital mortality, and could be much lower than the true all-cause mortality because a preference to die at home is high in some regions in Thailand.27 The comparisons between mortality among patients with AMR BSI, antimicrobial-susceptible BSI and ‘no infection’22–25 are not within the scope of this study. Fourth, VRE was not included in the analysis because the proportion and frequency of VRE BSI in our study hospitals were still relatively low during the study period (https://doi.org/10.6084/m9.figshare.22339933). Fifth, comparing AMR data between hospitals will need to be performed carefully due to the difference in classes, types or levels of the hospitals, case-mix, blood culture utilization and other confounding factors.29 In addition, the total number of cases and deaths with AMR BSI in each hospital depends not only on AMS and IPC practices, but also on the size of the hospital . Therefore, the frequency and mortality rate  should be calculated and used to compare the AMR burden between hospitals with similar levels of care and between areas. Sixth, formal qualitative and quantitative studies for feasibility of the AMASS and utility of the data were not performed. These will be conducted in the next phase of the AMASS implementation.The study has some limitations. First, the AMASS requires users to validate the autogenerated reports as recommended for any analytical software.In conclusion, local and timely AMR data are crucial for local and national actions. Empowering local hospitals to analyse, validate and utilize their own AMR data in a timely manner and with minimal effort, by using analytical software such as the AMASS, is strongly encouraged.dlad088_Supplementary_DataClick here for additional data file.


Was the voluntary use of an information technology tool with forced structured entry of relevant clinical attributes associated with improved completeness  of radiologist recommendations for additional imaging?In this cohort study involving 1008 radiologist recommendations for additional imaging, implementation of a voluntary closed-loop communication tool that forces inclusion of time frame, modality, and rationale was associated with a significant 3-fold increase in complete recommendation content.This study suggests that supplementing radiologist free-text dictation with voluntary use of a structured entry tool for clinical attributes was associated with increased recommendation completeness, which may enable improvement initiatives for timely performance of clinically necessary recommendations. It is challenging to ensure timely performance of radiologist-recommended additional imaging when radiologist recommendation language is incomplete or ambiguous.To evaluate whether voluntary use of an information technology tool with forced structured entry of recommendation attributes was associated with improved completeness of recommendations for additional imaging over time.This cohort study of imaging report data was performed at an academic quaternary care center in Boston, Massachusetts, and included consecutive adults with radiology examinations performed from September 12 to 13, 2019 , October 14 to 17, 2019 (before intervention), April 5 to 7, 2021 (1 week after intervention), and April 4 to 7, 2022 (1 year after intervention), with reports containing recommendations for additional imaging. A radiologist scored the 3 groups  of 336 consecutive radiology reports (n = 1008) with recommendations for additional imaging.Final implementation on March 27, 2021, of a voluntary closed-loop communication tool embedded in radiologist clinical workflow that required structured entry of recommendation attributes.P < .05.The a priori primary outcome was completeness of recommendations for additional imaging, defined in a taxonomy created by a multidisciplinary expert panel. To validate the taxonomy, 2 radiologists independently reviewed and scored language attributes as present or absent in 247 consecutive radiology reports containing recommendations for additional imaging. Agreement was assessed with Cohen κ. Recommendation completeness over time was compared with with 1-sided Fisher exact tests and significance set at P < .001) 1 year after intervention; completeness in the corresponding free-text report language increased from 14% (46 of 336) before the intervention to 25% (85 of 336) (P < .001) 1 year after the intervention.Radiology-related information for consecutive radiology reports from the 4 time periods was collected from the radiology department data warehouse, which does not include data on patient demographic characteristics or other nonimaging patient medical information. The panel defined 5 recommendation language attributes: complete , ambiguous , conditional , multiplicity (multiple options), and alternate (language favoring a different examination to that ordered). Two radiologists had more than 90% agreement (κ > 0.8) for these attributes. Completeness with use of the tool increased more than 3-fold, from 14% (46 of 336) before the intervention to 46% (153 of 336) (This study suggests that supplementing free-text dictation with voluntary use of a structured entry tool was associated with improved completeness of radiologist recommendations for additional imaging as assessed by an internally validated taxonomy. Future research is needed to assess the association with timely performance of clinically necessary recommendations and diagnostic errors. The taxonomy can be used to evaluate and build interventions to modify radiologist reporting behaviors. This cohort study evaluates whether voluntary use of an information technology tool with forced structured entry of recommendation attributes was associated with improved completeness of recommendations for additional imaging over time. Two initial challenges emerge. First, reducing subjectivity and variability of recommendations for additional imaging requires modification of radiologist behavior. For example, we know that incomplete recommendations for additional imaging, such as those that do not include a recommended time frame, have a lower likelihood of being followed.8 Second, identifying and tracking recommendations for additional imaging is a challenge when radiologist language is incomplete or ambiguous.9Radiologist recommendations for additional imaging are often subjective and variable.10 Radiologist recommendation language can lack specificity and clarity.11 This lack of specificity and clarity is partly associated with radiologist diagnostic uncertainty.12 Attempts to follow the American College of Radiology initiative have focused on shifting radiologists toward more definitive reporting.15 Commonly accepted behavior theory suggests that we could increase effectiveness by also identifying and relieving diagnostic uncertainty and other forces that may be associated with radiologists’ variable and potentially ambiguous reporting tendencies.16The American College of Radiology has launched an initiative to improve the communication and follow-through rates of these recommendations for additional imaging.18 These tools generally rely on consistent and comprehensive radiologist free-text reporting, yet we know that radiologist language is variable and potentially ambiguous.12 In 1 recent study of radiology reports, 62% of recommendations still did not include a time frame,19 making it challenging to assess the timeliness of recommendation completion. Furthermore, although validated taxonomies have been developed to analyze pathology reporting20 and medical error or incident reporting,22 for example, validated taxonomies have not been developed to categorize and analyze radiology reports. Taxonomies are designed to define and identify associated factors that may provide explanatory power,21 enable recognition of diversity and similarity,21 and enhance systematic organization learning.22 They are a mechanism for categorizing the narratives in reports.22 A taxonomy for recommendations for additional imaging would allow us to evaluate the completeness and ambiguity in the recommendations for additional imaging. This evaluation mechanism would enable validated and reproducible studies of the language of these recommendations and the effectiveness of tracking tools. It would also help identify radiologist- and examination-specific patterns in the completeness and ambiguity of the recommendations for additional imaging, allowing for the identification of the forces that underlie the incomplete and ambiguous language regarding these recommendations.Several tools to track the completion of recommendations for additional imaging have been developed.In this study, our primary aim was to evaluate whether voluntary use of a closed-loop communication tool that forces structured entry of recommendations for additional imaging was associated with improved overall completeness over time. Secondarily, we aimed to evaluate whether the communication tool occasioned completeness that was not provided in the associated free-text radiology report. To achieve these aims, we developed and internally validated a taxonomy to categorize and analyze the actionable vs confusing attributes of these recommendations for additional imaging.STROBE) reporting guideline for reporting observational cohort studies. Given the multiple concepts and components of this study, a study flow diagram with summary study results is included in the This retrospective cohort study of imaging report data was conducted at a large academic quaternary care center. It was approved by the Mass General Brigham institutional review board with a waiver of informed consent due to the large number of retrospectively reviewed charts and deidentification of data prior to analysis and was performed in accordance with Health Insurance Portability and Accountability Act requirements. This report followed the Strengthening the Reporting of Observational Studies in Epidemiology  iteratively constructed a taxonomy classification designed to identify the language choices for recommendations for additional imaging that cause confusion due to lack of specificity or clarity. Identified issue categories were unclear imaging modality, time frame, or rationale (not complete); equivocal, vague language (ambiguous); qualifying language ; multiple options without delineation of the best option (multiplicity); and language dismissive of the ordered examination in favor of a different examination . Conversely, we thus considered the recommendations for additional imaging to be readily actionable if they were complete, unambiguous, and unconditional and without multiplicity or dismissive alternative language. As with other taxonomies, each category was designed to be independent from the others, and each category could be independently scored yes or no. A recommendation could be scored yes in all categories, no in all categories, or any combination of yes and no across categories. Additional details are provided in The taxonomy was validated on a set of 247 consecutive radiology report impression sections that contained recommendations for additional imaging in examinations conducted September 12 and 13, 2019. This sample size was calculated based on prior unpublished data showing a detection rate as low as 3% for some of the taxonomy categories, a null of κ = 0.4 indicating that an agreement of less than 0.4 would be considered essentially random, and a target κ of 0.8 that would be considered excellent agreement.23 and the pretrained BERTBASE model. Radiology reports from the following subspecialty divisions were included in each search: abdominal imaging, emergency radiology, musculoskeletal radiology, neuroradiology, noncardiac vascular imaging, nuclear medicine, thoracic radiology, and ultrasonography. Reports from other divisions  used unique reporting language and were therefore excluded. If different recommendations were provided for more than 1 finding in the same report, only the first recommendation was considered in this study for ease of data management, given the data were collected on a report-level basis.Radiology reports containing recommendations for additional imaging were identified using an internally developed radiologist recommendation extraction algorithm based on BERT , a deep learning framework introduced by Google in 2018,Two board-certified diagnostic radiologists with 7 years (N.K.) and 3 years (J.P.G.) posttraining experience individually scored (yes or no) each of the taxonomy categories for the 247 recommendations for additional imaging. The interobserver agreement rate and the Cohen κ correlation coefficients were calculated for each taxonomy category in RStudio, version 2022.02.3 build 492 using R, version 4.2.1 , and the IRR package.9 was developed as an expansion to a critical results closed-loop communication system26 to deliver clear radiologist recommendations for additional imaging and to track the acceptance and fulfillment of these recommendations. An early version of ARRC was trialed in our thoracic imaging group for the reporting of pulmonary nodules beginning October 2019.11A hospital-level quality improvement initiative called Addressing Radiologist Recommendations Collaboratively (ARRC)In brief, radiologists are highly encouraged to use the ARRC closed-loop communication tool (the ARRC tool) to communicate recommendations for additional imaging to referring clinicians. The ARRC tool interface requires the radiologist to specify a discrete time interval, imaging modality, and rationale for additional imaging ; April 5 to 7, 2021, the week after the release of the first division-level summaries (time point 2); and April 4 to 7, 2022, 1 year later (time point 3). Information on whether these recommendations were filed with the ARRC tool was obtained from the ARRC database. A board-certified diagnostic radiologist with 3 years posttraining experience (J.P.G.) scored (yes or no) each of the taxonomy categories for the 3 sets of 336 recommendations for additional imaging.Radiologists can generate a free-text recommendation that does not contain all of the complete attributes, yet they can file the recommendation in the ARRC tool, which forces inclusion of all the complete attributes of the recommendation. In other words, if a free-text recommendation lacks 1 or more of the complete attributes but the same recommendation is filed with the ARRC tool, it is by definition complete because of the ARRC tool forcing function. We therefore assessed both the change in completeness of free-text recommendations (complete free text) and the overall change in completeness, including recommendations with complete free-text language and/or complete via filing with the ARRC tool .P < .05.To assess the primary outcome measure of whether the ARRC tool was associated with improved completeness of recommendations for additional imaging, 1-sided Fisher exact tests were performed to compare rates of reports with complete recommendations for additional imaging at time point 1 with time point 2 and to compare time point 1 with time point 3. Significance was set at P < .05. Statistical analyses were performed in RStudio, version 2022.02.3 build 492 using R, version 4.2.1.In addition, a secondary outcome measure was the qualitative change in radiologist language associated with the ARRC tool. One-sided Fisher exact tests were performed independently for each taxonomy category , again comparing time point 1 with time point 2 and comparing time point 1 with time point 3. Mathematical correction was not made for multiple planned comparisons given these were exploratory secondary outcomes. An additional secondary outcome measure, whether the forcing function of the ARRC tool was associated with improved completeness of the recommendation compared with free-text recommendation language, was assessed with 1-sided Fisher exact tests and raw percentage differences at time point 2 and time point 3. Significance was set at Radiology-related information for consecutive radiology reports from the 4 time periods was collected from the radiology department data warehouse, which does not include data on patient demographic characteristics or other nonimaging patient medical information.The 2 radiologists had excellent agreement, with more than 90% agreement and a Cohen κ greater than 0.8 for all categories, thereby validating the taxonomy . Selected illustrative examples of the application of the taxonomy are provided in P < .001) and 46% (153 of 336) 1 year after implementation (P < .001).The percentage of recommendations for additional imaging categorized as complete overall increased significantly from 14% (46 of 336) before implementation to 34% (113 of 336) 1 week after implementation (P = .048) and 25% (85 of 336) 1 year after implementation . The percentage of recommendations for additional imaging categorized each as ambiguous, conditional, alternate, and multiplicity did not change significantly. However, there appeared to be a downward trend in the latter 3 categories  before implementation to 20% (66 of 336) 1 week after implementation  and 1 year after the intervention . Of the 113 complete recommendations for additional imaging in the 1-week postimplementation period, 87 (77%) were filed in the ARRC tool. Only 40 of the 87 recommendations for additional imaging (46%) contained complete language in the associated free-text report. Of the 153 complete recommendations for additional imaging in the 1-year postimplementation period, 124 (81%) were filed in the ARRC tool. Only 56 of the 124 recommendations for additional imaging (45%) contained complete language in the associated free-text report. In both periods, ambiguous, conditional, and multiplicity was present in the free-text reports associated with recommendations for additional imaging that were filed with the ARRC tool, including ambiguous language in 32 of 87 of these free-text reports (37%). Results are presented in Use of the tool increased the completeness rate compared with free-text report language 1 week after the intervention  was associated with a significant increase in completeness of recommendations for additional imaging  from 14% of recommendations before implementation to 46% of recommendations 1 year after implementation. Furthermore, the taxonomy clarified that the forcing function within the ARRC tool increased the number of complete recommendations for additional imaging relative to the associated free-text reports. However, nearly one-third of the complete recommendations filed with the ARRC tool still contained ambiguous language in the associated free-text radiologist reports.20 and on a taxonomy designed for categorizing and evaluating family medicine error reports.21 The concept of using a taxonomy to categorize and analyze recommendations for additional imaging has been recently explored.19 That taxonomy was not validated, and the taxonomy presented here is less complex because it does not contain multiple domains. In this study, we applied our taxonomy to track the association between an intervention and the recommendations for additional imaging. However, the taxonomy would also be suitable for exploring what situations are associated with unclear recommendations . The taxonomy would also be suitable for identifying specific individuals who frequently provide unclear recommendations .Our approach to building this taxonomy was based on a taxonomy designed for categorizing and evaluating pathology reportsIn this study, we found that the free-text language of these recommendations often lacks completeness even when the radiologist has simultaneously filed a recommendation with the ARRC tool, which forces the radiologist to include the recommended examination modality, time frame, and reason. The free-text dictation and the ARRC closed-loop communication tool exist separately, and therefore the dictation language and the recommendation filing in the tool are both subject to radiologist discretion. This finding demonstrated that free-text radiology reports cannot be relied on to contain adequate information for optimal patient care without some type of function that forces completeness . As such, natural language processing and artificial intelligence tools that search free-text reports are unlikely to be fully effective in identifying and tracking appropriate recommendation completion or fulfillment, unless the reports are autopopulated by a tool with a specificity forcing function.8 We postulate that the rate of fulfillment of recommendations for additional imaging will therefore be higher for complete recommendations as well as unambiguous recommendations than for incomplete and ambiguous recommendations. Further research will be needed to assess whether the fulfillment rate does truly increase, either due to increased complete reporting or due to reduction in overall recommendations due to decreased frequency of recommending additional imaging when the recommendation would have included ambiguous language. Further research will also be needed to evaluate whether a higher resulting fulfillment rate will be beneficial for patients. Conditional recommendations may have value in certain circumstances, but we suspect that they are likely overused. Recommendations that contain alternate language are challenging because they imply that the ordering clinician ordered the wrong test. Recommendations with a multiplicity of choices defer the choice of imaging examination to another clinician, who may not know the pros and cons of the imaging examination options, and making it difficult to monitor the fulfillment and appropriateness of the recommendation.9It has been previously shown that explicitly mentioning the recommended interval as part of a recommendation for additional imaging is associated with a 3-times-higher likelihood of the recommended additional examination being performed.27 even that yield can be low, and such data are not available for most clinical presentations. Our taxonomy provides a mechanism that will allow us to study more informed recommendation practices.With varied use of language, the radiologist deflects the decision on when and whether to obtain more imaging back to the ordering clinician. In our discussions, ordering clinicians interpret this behavior in various ways, including feeling frustrated due to perceived ambiguity. Similarly, while we know that radiologist recommendations have a clinically important yield in diagnosing specific abnormal results, such as chest computed tomography recommended for outpatient chest radiograph abnormal results,This study has some limitations, primarily its single-site design and retrospective nature. We did not establish or evaluate the clinical necessity of recommendations, and we do not know which recommendation or how the recommendations in each category were fulfilled. Furthermore, 1 year after the intervention, half of recommendations for additional imaging were still not complete; multifaceted quality improvement initiatives  may be needed to enhance the adoption of this intervention by radiologists.Using a newly validated taxonomy that enables analysis of radiologist recommendations for additional imaging, this cohort study has shown that the voluntary use of a closed-loop communication tool was associated with a significantly increased completeness  of recommendations for additional imaging and that a forcing function is likely required beyond free-text dictation to ensure completeness. Furthermore, ambiguous free-text recommendation language persists even when complete recommendations are filed in the closed-loop communication tool.


Geriatric patients are often frail and may lose independence through a variety of mechanisms including cognitive decline, reduced mobility, and falls. Our goal was to measure the effect of a multidisciplinary home health program that assessed frailty and safety and then coordinated ongoing delivery of community resources on short-term, all-cause emergency department (ED) utilization across three study arms that attempted to stratify frailty by fall risk.Subjects became eligible for this prospective observational study via one of three pathways: 1) by visiting the ED after a fall ; 2) by self-identifying as at risk for falling ; or 3) by calling 9-1-1 for a “lift assist” after falling and being unable to get up (121). The intervention consisted of sequential home visits by a research paramedic who used standardized assessments of frailty and risk of falling (including providing home safety guidance), and a home health nurse who aligned resources to address the conditions found. Outcomes of interest were all-cause ED utilization at 30, 60, and 90 days post-intervention compared with subjects who enrolled via the same study pathway but declined the study intervention (controls).Subjects in the fall-related ED visit arm were significantly less likely to have one or more subsequent ED encounters post-intervention than controls at 30 days ; 60 days ; and 90 days . In contrast, participants in the self-referral arm had no difference in ED encounters post-intervention compared to controls at 30, 60, or 90 days . The size of the 9-1-1 call arm limited statistical power for analysis.A history of a fall requiring ED evaluation appeared to be a useful marker of frailty. Subjects recruited through this pathway experienced less all-cause ED utilization over subsequent months after a coordinated community intervention than without it. The participants who only self-identified as at risk for falling had lower rates of subsequent ED utilization than those recruited in the ED after a fall and did not significantly benefit from the intervention. The preservation of autonomy and the ability to live independently is a major focus of geriatric medicine.8Approximately three million individuals are treated for falls each year in emergency departments (ED).16Multiple significant barriers have limited frail elders’ acceptance of home healthcare assessments and delivery. One observed barrier for high-risk populations has been patient reluctance to admit home health personnel into their homes; however, once EMS professionals have been trained in “community paramedicine” techniques, they were able to achieve patient trust and have made significant contributions to various public health aims.Providing the elderly who are frail with the ability to maintain independence and live safely at home is of paramount importance to preserving their quality of life.What do we already know about this issue?Elderly falls at home requiring EMS response were associated with repeat 9-1-1 calls and transport to an ED over the following 30 days.What was the research question?Can a coordinated health and safety visit by a paramedic and a home care nurse decrease all-cause ED utilization over 30–90 days?What was the major finding of the study?Among frail elderly, the intervention reduced the proportion of repeat ED visits significantly at 30, 60, and 90 days .How does this improve population health?The complementary skills of EMS and home care nurses can enhance the health and safety of elders, reducing their reliance on emergency medical care.We sought to measure the effect of a coordinated frailty assessment and home safety intervention by research paramedics with follow-up visits by community-based home health nurses on subsequent, all-cause ED utilization at 30, 60, and 90 days post-intervention. Ultimately, the goal was to improve the safety of enrolled subjects and enhance their ability to live independently.The Paramedic Referrals for Increased Independence and Decreased Disability in the Elderly (PRIDE) program was a prospective observational study, conducted between March 2015–April 2018. Subjects resided in the geographic catchment area of 15 towns in south-central Connecticut.Study subjects were recruited into one of three enrollment populations: 1) those who were seen in the ED after falls; 2) individuals who responded to public messaging and perceived themselves to be at risk for falling; and 3) those referred by EMS agencies after they called 9-1-1 for a “lift assist” or help getting up after a non-injury fall at home. Subjects were recruited into the ED arm by research associates (RA) stationed in the ED of a large, urban, tertiary care hospital with over 100,000 ED visits per year.On assigned schedules that generally covered day and evening shifts seven days a week, these RAs monitored patient locator boards for chief complaints suggestive of falls by seniors. When appropriate, they approached the patient and/or family, explained the study, and if eligible, invited them to participate. Following informed consent, the participants were enrolled as study subjects. Those who were interested in participating but were admitted to the hospital on that ED visit were contacted again by the RAs near the time of discharge to facilitate entry into the study.Subjects who perceived themselves as elderly and at risk of falling, were recruited through information events and public messaging. Recruitment efforts included tables at senior centers, senior housing complexes, churches, and other venues, which were staffed by research paramedics wearing PRIDE logos who answered questions and distributed brochures. Radio spots and billboards describing the program and providing contact information were also used.Subjects were recruited into the EMS referral arm at the time of a “lift assist” call if they or the responsible family member at the scene agreed that a study representative could subsequently call and invite them to participate in the study. If the patient consented to the follow-up call, his or her name and telephone number were forwarded by the EMS responders to study personnel. Following informed consent, all subjects who did not wish to participate in the study intervention were given the option of declining.In all arms of the study, eligibility was restricted to adults living at home or in assisted living facilities within the geographic catchment area. Participants living in long-term care facilities were not eligible to participate. There was no explicit age requirement, but participants were primarily over 65, likely due to use of the term “elderly” in the program title. We defined the intervention group as those who agreed to participate in the intervention. Participants comprising the control group consented to have their subsequent ED utilization followed but chose not to participate in the intervention. Each participant received a $10 gift card to a local supermarket for enrolling, and a $15 gift card for completing the interventions.The intervention consisted of a visit by a Connecticut-licensed paramedic serving independently of the EMS system and trained and identified to the public as a research assistant for this project. The research paramedic performed a home safety check , obtained a list of current medications, and employed standard instruments to assess degrees of frailty.Following the research paramedic’s visit, there was a pre-arranged house call by a nurse from one of several participating home health agencies. The nurse reviewed the findings of the research paramedic’s assessment, performed medication reconciliation, and confirmed access to currently prescribed medications. The nurse also determined needs for durable equipment and ongoing services such as physical or occupational therapy and arranged for delivery. Research paramedics and visiting nurse staff were formally trained for the intervention, including didactic training and opportunities to ride along with their counterparts in the care team and to shadow case managers and care coordinators in the ED. Further details on the training curriculum for paramedics and nurses can be found in Finally, after the interventions were completed, a brief satisfaction survey was mailed to each participant. This survey was adapted for PRIDE from the Centers for Medicare & Medicaid Home Health Care Consumer Assessment of Healthcare Providers and Systems, which was beta-tested on an early subset of subjects representing all three enrollment populations and did not require revision before deployment.Data obtained from the participants directly at the time of enrollment, during the interventions performed by the research paramedics and visiting nurses during the home visits, and participants’ responses to the post-completion satisfaction survey, were collected and maintained using REDCap electronic data capture tools hosted at Yale University.We measured ED utilization that occurred 30, 60, and 90 days after enrollment in the control group, and after the home health nurse visit was completed in the intervention group. Study subjects were considered part of the control group until both visits outlined in the intervention (research paramedic and visiting nurse) were completed. For example, subjects whose study intervention was completed over 60 days following enrollment, had 30-day and 60-day data included in the control group. The date of completion of the second visit was considered day 0 for the intervention group. Any EMS use at 30 days was also measured and published separately.13Primary outcomes were subsequent all-cause ED utilization.We conducted a generalized estimating equation (GEE) analysis using SAS analytic software 9.4  to compare the proportions of participants that had at least one ED visit during the 30, 60, or 90 days following enrollment in the control group or following completion of the visits in the intervention group. The GEE was used to accommodate repeated assessments from the participants, some of whom were sequentially included in control and then intervention groups. We similarly compared data across the three enrollment populations  to determine whether the intervention appeared more or less effective among these groups. We also conducted a multivariable analysis with covariate adjustment including age, gender, and insurance type.Additional supportive analyses were performed on the ED-enrolled subject populations to evaluate sensitivity. First, to further evaluate for any effects related to having some data from the same subjects in both intervention and no-intervention groups, we removed all the data from the crossover subjects from the dataset and only those who had never received the intervention throughout the study were compared by logistic regression with those who did. Second, to address potential bias due to variable delays between the time of enrollment in the ED and the time of the intervention, we looked at our population of crossover subjects (those who had outcomes recorded both before and after the study intervention). We performed paired analysis using GEE to compare the no-intervention phase vs the intervention phase of their study participation. The crossover subjects thus served as their own controls.P<0.05, two-sided.We performed person-time analysis using generalized Poisson regression to further evaluate the intervention’s effect on healthcare utilization. The statistical significance was defined as There were 5,665 individuals enrolled in the PRIDE study: 121 from 9-1-1 calls; 2,757 from ED visits; and 2,787 via self-referral. Of these, full 90-day follow-up data were available for 5,439 (96%) of enrolled subjects. P-values significant at <0.001). The adjusted P value reflects demographic covariates including age, gender and insurance type(s). Individuals who entered this study through the self-referral mechanism did not have significant reductions in subsequent ED encounters . Those who enrolled as a result of 9-1-1 referrals also showed no apparent benefit, although the numbers in this arm were too small for reliable comparison.P-values <0.001.The results of the analysis for the ED-enrolled subjects excluding the intervention crossovers are presented in P-values reflecting the smaller numbers included in these subsets of study participants.In analyses of crossover participants only , all of the subjects had at least 30, and some up to 90 days, of outcomes data prior to receiving the intervention. As the data shown in P<0.001). The incidence rates of first ED visit among the 9-1-1 lift-assist and self-referral groups showed no significant difference between the intervention and no intervention groups. Incidence rates of total overall visits also demonstrated a significant difference among subjects recruited from the ED: 6.27 visits per 1,000 follow-up days in the intervention arm vs 7.16 visits per 1,000 follow-up days in the control arm (P<0.01), but not among subjects recruited via 9-1-1 lift assist or self-referral.We also performed a person-time analysis to initial ED visit and an event-time analysis for all ED visits. The results are displayed in The following question was asked as part of the participant satisfaction survey that was mailed to every subject who completed both the PRIDE paramedic and nurse visits: “Using any number from 0 to 10, where 0 is the worst healthcare experience possible and 10 is the best healthcare experience possible, what number would you use to rate your experience with the PRIDE program?” The participants were provided self-addressed, stamped envelopes for returning their surveys. A total of 3,806 surveys were mailed to participants and 1,952 were returned, for a response rate of 51%, although 77 individuals (3.9%) left this question blank. To our knowledge, this is the largest study of an intervention aimed at reducing short-term morbidity while maintaining independence among frail, community-dwelling older adults. Based on promising earlier studies, EMS personnel coordinated with home healthcare agency nurses and primary care physicians to address gaps in home support services and to define the individuals who were most likely to benefit from the interventions. 13This study demonstrated decreased subsequent all-cause ED utilization at 30, 60, and 90 days after a home assessment intervention among ED patients who presented with falls, but not among individuals who self-referred. There were substantial demographic differences between the self-referred and ED-enrolled arms, as seen in The significantly lower ED utilization among subjects receiving the PRIDE intervention within the ED-recruited population but not in the self-referral population further suggests that falls are a useful marker for frailty, and that the associated high risk of short-term illness and injury may be modifiable by the right set of interventions. Indeed, for patient populations not specifically restricted to falls, home visit interventions have been found to be more effective on higher risk patients.41Recent research has shown that emergency physicians fail to identify risk factors for falls in the ED.In contrast, the difficulty enrolling subjects into the 9-1-1 lift-assist arm attests to the regulatory and workflow challenges for EMS personnel to enroll individuals in the same intervention. Several towns within the geographic catchment area were reluctant to allow EMS personnel to perform this enrollment, and unlike in the ED, EMS did not have additional staff helping with patient enrollment. These practical considerations are unlikely to be unique to this study and may represent reasons ED enrollment may be preferable to EMS agencies recruiting subjects on scene.This was an observational cohort study that provided participants the choice whether to receive the intervention, rather than being a randomized controlled trial. Without randomization it is unknown whether selection bias is present and a contributor to the differences in outcomes between the control and intervention arms. Nevertheless, analyses of those that were observed during both control and intervention periods  provided similar results. Another limitation in analysis of the intervention is that the efficacy of the PRIDE intervention was assessed in aggregate. The effect size or direction of independent components of the multifactorial intervention  could not be determined from this study.Outcomes reporting of ED visits was limited to within the Yale New Haven Health System (YNHHS); therefore, repeat ED visits to other health systems are not reflected in our analysis. However, the vast majority  of ED visits and 88% of the inpatient beds in the study’s catchment area are at YNHHS facilities.Research paramedic and visiting nurse home visits were associated with lower rates of subsequent all-cause ED utilization among subjects who presented to the ED after falls but not among subjects who self-enrolled by identifying themselves as at risk for falling, nor among subjects who contacted 9-1-1 for lift assists. These findings suggest that individuals who present to the ED after falls can efficiently be enrolled and are likely to benefit from a program involving standardized home assessment of frailty and safety by specially trained paramedics and follow-up visits by home health nurses to arrange for appropriate, ongoing medical and community resources. By targeting this vulnerable group with a focused intervention, the autonomy of these patients and their ability to live independently may be enhanced and potentially preserved.