Although previous research has made substantial progress in developing high-performance artificial intelligence (AI)–based computer-aided diagnosis (AI-CAD) systems in various medical domains, little attention has been paid to developing and evaluating AI-CAD system in ophthalmology, particularly for diagnosing retinal diseases using optical coherence tomography (OCT) images.This diagnostic study aimed to determine the usefulness of a proposed AI-CAD system in assisting ophthalmologists with the diagnosis of central serous chorioretinopathy (CSC), which is known to be difficult to diagnose, using OCT images.For the training and evaluation of the proposed deep learning model, 1693 OCT images were collected and annotated. The data set included 929 and 764 cases of acute and chronic CSC, respectively. In total, 66 ophthalmologists  participated in the observer performance test. To evaluate the deep learning algorithm used in the proposed AI-CAD system, the training, validation, and test sets were split in an 8:1:1 ratio. Further, 100 randomly sampled OCT images from the test set were used for the observer performance test, and the participants were instructed to select a CSC subtype for each of these images. Each image was provided under different conditions: (1) without AI assistance, (2) with AI assistance with a probability score, and (3) with AI assistance with a probability score and visual evidence heatmap. The sensitivity, specificity, and area under the receiver operating characteristic curve were used to measure the diagnostic performance of the model and ophthalmologists.The proposed system achieved a high detection performance (99% of the area under the curve) for CSC, outperforming the 66 ophthalmologists who participated in the observer performance test. In both groups, ophthalmologists with the support of AI assistance with a probability score and visual evidence heatmap achieved the highest mean diagnostic performance compared with that of those subjected to other conditions (without AI assistance or with AI assistance with a probability score). Nonretina specialists achieved expert-level diagnostic performance with the support of the proposed AI-CAD system.Our proposed AI-CAD system improved the diagnosis of CSC by ophthalmologists, which may support decision-making regarding retinal disease detection and alleviate the workload of ophthalmologists. Computer-aided diagnosis (CAD) is a software system that assists in the diagnostic decision-making of clinicians . CAD sysWith recent advancements in computer vision and deep learning techniques, deep neural networks have been reported to achieve expert-level performance in clinical diagnoses -10. ThisFollowing age-related macular degeneration (AMD), diabetic retinopathy, and branch retinal vein occlusion, central serous chorioretinopathy (CSC) is the fourth most prevalent vision-threatening retinopathy and is characterized by serous detachment of the neurosensory retina at the posterior pole ,12. MostIn ophthalmology, OCT is a noninvasive, rapid, and accurate test that produces highly reproducible outcomes ,16,17. IHerein, we propose an AI-CAD system that can alleviate the heavy workloads and improve the diagnostic performance of retinal disease for ophthalmologists. We tried to find out whether AI could really help ophthalmologists’ diagnostic activities through a CAD system in the field of ophthalmology, and we selected CSC, one of the representative macular diseases, and built a CAD system. In particular, the proposed AI-CAD system may support ophthalmologists in distinguishing the subtypes of CSC. To investigate the effectiveness of the proposed system, we conducted a within-subject user study involving 66 ophthalmologists.This study was conducted in accordance with the 1964 Declaration of Helsinki guidelines. The Ethics Committee of Hangil Eye Hospital approved the research protocol (IRB 21018) and its implementation and waived the requirement for informed consent as this study was retrospective and observational in nature and used medical records to extract the required data.To train and evaluate the proposed deep learning model, 1693 OCT images of patients who visited Hangil Eye Hospital between June 2017 and June 2021 were collected and annotated. This study aimed to construct an AI-CAD system that identifies CSC subtypes.All CSC cases were diagnosed by independent retinal specialists using fundus examinations, fluorescein angiography (FA), indocyanine green angiography (ICGA), and OCT images. On all CSC cases, FA and ICGA were performed simultaneously using a Heidelberg Retina Angiograph (Heidelberg Engineering) confocal scanning laser ophthalmoscope. Other potentially conflicting retinal pathologies such as AMD, polypoidal choroidal vasculopathy, pachychoroid neovasculopathy, and pachychoroid pigment epitheliopathy were excluded from our analysis.Acute CSC was diagnosed based on the presence of serous retinal detachment involving the macula, as demonstrated by OCT, and the leakage at the level of the RPE on FA ,17,21. IThe proposed AI-CAD system formulated in this study  compriseThe AI probability panel displays the probability score for each retinal disease (acute or chronic CSC). These scores are generated from the last fully connected layer of the proposed deep learning model using the softmax activation function and allow users to measure the confidence of the AI model with its decision. The probabilities are illustrated with progress bars to enable users to intuitively perceive the model’s confidence.The evidence heatmap panel reveals important regions in the OCT image while the model classifies the target label . Gradient-weighted class activation mapping was adopted to highlight the important regions . The actThe status panel displays the patient information of the current sample. Patient information included identification number, sex, and age. Users were able to identify the demographic information of a patient while analyzing a given image.To automatically classify a given OCT image into 2 different CSC subtypes, we use CNN-based architecture, VGG-16 . The conTo train and evaluate the AI model, the data were randomly split into the training, validation, and test sets in an 8:1:1 ratio. The validation set was exclusively used to tune the hyperparameters of the model, and the test set was singularly used to evaluate the final performance of the model. We trained the proposed model using batch sizes of 64 and 30 epochs and Adam optimization  (learninNo AI). In the subsequent step, an AI probability panel was provided to the observers . The AI probability panel shows the probability score of each retinal disease (acute or chronic). At the end of the step, both the AI probability panel and AI evidence heatmap panel were added to the system to provide a visual explanation to the observers .To investigate whether each component of the proposed system can assist in improving the diagnostic performance of ophthalmologists, a web-based experiment was conducted in which each participant was instructed to classify CSC subtypes from a given SD-OCT image. The experimental procedure comprised 3 steps . In the In each step, all observers had to determine whether the given OCT image reflected acute or chronic CSC by selecting a button on the web system. The same OCT image was used in the 3 steps. As 100 SD-OCT images were used in our experiment, each participant assessed 300 cases  in total. The 100 images were randomly extracted from the test set that was not used to train our model. The step-by-step user interface for the observer performance test is illustrated in The study recruited 66 participants, including 36 retina and 30 nonretina specialists. The retina specialists were medical doctors who had completed 1-2 years of the retina fellowship training program. In contrast, nonretina specialists were board-certified ophthalmologists who were not specialized in the retina. The detailed information of the 66 participants is summarized No AI, AI Prob, and AI Prob+Evid) [t test was used to compare diagnostic performance between the nonretina and retina specialists.Receiver operating characteristic analysis was conducted to evaluate the performance of the proposed model and ophthalmologists in classifying the CSC subtypes. Thereafter, the receiver operating characteristic curve with the true-positive and false-positive rates was plotted to measure the area under the receiver operating characteristic curve (AUROC) score. Friedman  test, foob+Evid) . A 2-taiThe proposed model exhibited high accuracy, sensitivity, and specificity values of 96.3%, 97.1%, and 95.7%, respectively (n=163). The model achieved 98.4% (n=163) of the AUROC which outperforms Resnet-50  and VGG-19 . The model failed to accurately predict 6 cases only in our test set. Importantly, the training AUROC of 95.6% (n=163) indicates that our model strikes a balance between avoiding overfitting and underfitting, further affirming its reliability.The diagnostic performance of the AI-CAD system was compared with that of ophthalmologists. The AUROCs were calculated to evaluate the AI-CAD and human predictive abilities for 100 images randomly extracted from the test set. The AI-CAD system  outperformed both retina  and nonretina  specialists.No AI, AI Prob, and AI Prob+Evid; df=2; P<.001) and nonretina  specialists. In particular, the retina specialists who were provided with the AI probability panel and AI evidence heatmap panel  achieved the highest mean diagnostic performance  compared with those subjected to other conditions . The nonretina specialists also displayed their best performance  with numerical and visual information compared with when they were subjected to other conditions .The retinal-disease detection performance of the 36 retina and 30 nonretina specialists were evaluated under 3 different conditions  and retina specialists not supported by the AI-CAD system . This finding demonstrates that nonretina specialists can achieve expert-level diagnostic performance with the support of the proposed AI CAD system  of the misclassified cases under the No AI condition were accurately classified in the AI Prob+Evid condition  and negative  changes that could be observed between the ondition . In partThis study proposed the development of an AI-CAD system to assist ophthalmologists in distinguishing chronic from acute CSC. In particular, the proposed system provides (1) the probability of retinal disease and (2) visual evidence to effectively assist ophthalmologists in their clinical decisions. To evaluate the effectiveness of the proposed AI-CAD system in enhancing ophthalmologists’ clinical decision-making, a within-subject user study involving 66 ophthalmologists was conducted. The extensive experiments demonstrated that the proposed AI-CAD system effectively assists ophthalmologists in improving their diagnostic performance for retinal disease.The proposed deep neural network in the AI-CAD system achieved a high retinal-disease detection performance of 99.5% (n=100) of the AUROC, outperforming all 66 ophthalmologists who participated in the experiment. The high performance of the proposed AI-CAD model implies that it can lessen the heavy workloads and reduce potential errors by clinicians ,9. The qThe observer performance test revealed that the proposed AI-CAD system can effectively help ophthalmologists diagnose retinal disease. The experimental results revealed that the diagnostic performance of the retina specialists and nonretina specialists increased by 3.5% n=36) and 4.4% (n=30), respectively . This si and 4.4%Deploying a clinical decision support system (CDSS) in real-world health care settings presents a set of practical challenges. Chief among these concerns is the system’s susceptibility to errors, which can undermine trust in AI-driven solutions. To address this, a dynamic training model becomes indispensable. In the realm of academia, researchers have embraced the “Human-In-the-Loop” paradigm to tackle this issue -32. ThisAnother pivotal concern pertains to the system's security. Safeguarding patient data and upholding the integrity of the CDSS is paramount within the health care domain. This demands the implementation of robust encryption, stringent access controls, and regular security audits to protect sensitive information and prevent unauthorized access or data breaches. Furthermore, continuous monitoring and timely updates to the CDSS are essential for addressing emerging security threats and vulnerabilities, ensuring a high level of security in real-world applications.By concurrently addressing error mitigation and security, CDSS developers and health care professionals can collaborate in creating a more dependable and trustworthy system that serves the best interests of both patients and medical practitioners.This study has some limitations. First, all images were acquired from a single OCT device located at a single academic center. Although the data set was sufficient to train and validate the proposed model for distinguishing between CSC subtypes, external validation with a different center is needed. Second, experiments were conducted using the web-based AI-CAD system developed in this study. Thus, the environment was relatively different from that of actual clinical practice. However, we attempted to design and develop a user-friendly AI-CAD system under the supervision of retina specialists. Third, considering that the model's training data comprises images taken exclusively by the Heidelberg Spectralis device (Heidelberg Engineering Inc), its performance might be insufficient when dealing with images from different devices. Future studies should prioritize (1) extending the proposed AI-CAD system to other retinal diseases, such as AMD and diabetic retinopathy; (2) developing strategies to improve the reliability of doctors when using the AI-CAD system; and (3) exploring the application of transfer learning techniques to address the challenges arising from variations in devices.This study has several implications. First, to the best of our knowledge, this study is the first attempt to develop and evaluate an AI-CAD system for the detection of retinal disease using OCT. Prior studies have developed AI-CAD systems for the detection of pulmonary disease and evaluated their effectiveness ,33,34. HSecond, the proposed AI-CAD system is potentially useful for small or local medical care centers where retina specialists are unavailable. Unlike in large-scale medical care centers or hospitals, retina specialists are rarely found in small or local centers. Diagnosing subtypes of retinal diseases (ie, acute vs chronic CSC  and polyTo the best of our knowledge, this study is the first attempt to design, develop, and evaluate an AI-CAD system for the detection of retinal disease using OCT. First, an AI-CAD system was developed with a high-performance deep learning model. Thereafter, an observer performance test was conducted with the proposed system to determine the ability of the system to assist ophthalmologists in diagnosing retinal diseases. The results indicated that the proposed AI-CAD system can provide retinal expert-level diagnostic performance and help ophthalmologists improve their diagnostic performance in detecting CSC subtypes. Thus, the proposed AI-CAD system can alleviate the heavy workload of ophthalmologists and help in the decision-making process involved in detecting CSC subtypes. As a base study, this study demonstrates the usefulness and effectiveness of using an AI-CAD system in detecting retinal diseases, particularly CSC subtypes. In the future, the proposed AI-CAD system may be easily extended to the detection of other retinal diseases, such as AMD, diabetic retinopathy, and branch retinal vein occlusion.


Migraine is a common neurological disease that can have a substantial impact on patients’ lives and on society. Erenumab, a fully human monoclonal antibody that targets the calcitonin gene-related peptide receptor, was specifically developed for migraine prevention. The efficacy of erenumab has been established in several clinical trials; however, the real-world comparative effectiveness of erenumab has not been fully investigated.To evaluate the real-world impact of erenumab and onabotulinumtoxinA on acute medication usage and health care resource utilization (HCRU) among patients with migraine in the United States.This retrospective US claims analysis (Optum’s deidentified Clinformatics Data Mart Database) evaluated patients aged at least 18 years diagnosed with migraine who initiated erenumab or onabotulinumtoxinA between May 1, 2018, and September 30, 2019 (index date: first erenumab/onabotulinumtoxinA claim). Cohorts were matched 1:1 using the propensity score (PS) method . Stratification was performed based on gender, chronic migraine without aura diagnosis, onabotulinumtoxinA use, and acute/preventive drug use. The impact of erenumab and onabotulinumtoxinA on acute medication usage and HCRU was assessed in the 6-month post-index period. An exploratory analysis assessed the impact of erenumab and onabotulinumtoxinA on a composite endpoint of: (1) outpatient visit with a migraine diagnosis and associated acute medication claim, (2) hospital admission with a primary migraine diagnosis, or (3) emergency department visit with a primary migraine diagnosis. PS-matched data were used for comparative analyses; logistic regression with covariate adjustment was used for dichotomous variables, and a negative binomial model was used for count variables, with odds ratios or rate ratios (RRs) and 95% CIs calculated.P = 0.0069), although the difference in the number of claims for triptans and barbiturates was statistically nonsignificant. The adjusted average number of all-cause and migraine-specific visits per person to health care providers was generally lower in the erenumab cohort compared with the onabotulinumtoxinA cohort. Patients in the erenumab cohort had a significantly lower number of composite events . Similarly, the adjusted proportion of patients with any of the 3 composite events was lower in the erenumab cohort .Following stratified PS matching, 1,338 patients were included in both cohorts. At 6 months, the adjusted average number of claims per person for any acute medication was significantly lower in the erenumab cohort  and HCRU to a greater extent than onabotulinumtoxinA. What is already known about this subjectMigraine is a common neurological disease that affects approximately 39 million people in the United States.The use of acute medication in the treatment of migraine is often associated with limited effectiveness and poor tolerability, while the overuse of acute medication can lead to disease progression, adverse comorbid medical consequences, and the development of medication overuse headache.The health care resource utilization and cost associated with migraine impose a substantial economic burden on the United States.What this study addsThis study provides comparative effectiveness data on the impact of erenumab and onabotulinumtoxinA as migraine preventive therapies on acute medication usage and health care resource utilization.Erenumab reduced acute medication usage  and health care resource utilization to a greater extent than onabotulinumtoxinA in the 6-month post-index period.Erenumab treatment may be associated with a lower frequency of migraine attacks compared with onabotulinumtoxinA as assessed using a composite endpoint as a proxy to evaluate migraine attacks.1 Depending on the frequency and regularity of symptoms, migraine can either be categorized as episodic migraine (EM), defined as fewer than 15 headache-days per month, or chronic migraine (CM), defined as at least 15 headache-days per month for more than 3 months, of which at least 8 days have features of migraine.2 Migraine imposes a significant burden on patients in terms of pain, disability, reduced quality of life, and reduced work productivity, and it consistently ranks among the top 10 leading causes of years lived with disability worldwide.3 Migraine also imposes a heavy economic burden, with the majority of direct costs due to greater utilization of health care resources compared with matched individuals without migraine.6Migraine is a common, debilitating neurological disease that affects approximately 39 million people in the United States and 1 billion people worldwide.9 Acute treatment consists of both migraine-specific medication, such as triptans and ergots, and more recently, lasmiditan and gepants, as well as nonspecific medication, such as nonsteroidal anti-inflammatory drugs (NSAIDs). Opioids and barbiturate-containing medications are not recommended for the treatment of migraine but are still frequently prescribed.10 The ultimate goal of acute treatment is to provide a sustained pain-free response upon onset of an attack.9 However, acute treatments are often associated with limited effectiveness and poor tolerability,12 and many of these medications have certain contraindications in different patient subgroups.8 Overuse of acute medication is also a well acknowledged problem in migraine management15 and is associated with the transition from EM to CM, with greater pain intensity and with the development of medication overuse headache (MOH).18Pharmacological treatment of migraine involves both acute and preventive therapy, with their use depending on the severity and frequency of attacks, as well as patient characteristics and preferences.19 Preventive treatment has typically involved the use of medications such as beta blockers, anticonvulsants, and antidepressants that were originally developed for other conditions and subsequently repurposed for migraine.21 A substantial number of patients discontinue such treatment due to lack of efficacy and/or poor tolerability.24In contrast, the goal of preventive treatment is to reduce the frequency, severity, and duration of attacks in order to improve patient quality of life, as well as to reduce the excessive use of acute treatment.28 making it the only therapy approved by the FDA specifically for that indication. Since 2018, the FDA has approved 4 monoclonal antibody (mAb) drugs that target the calcitonin gene-related peptide (CGRP) pathway , which were purposely designed for the prevention of migraine.37 Of those, erenumab  is the first and only fully human mAb designed to specifically block the CGRP receptor, which plays a key role in migraine pathophysiology.38 The clinical efficacy and safety of erenumab were established in several placebo-controlled studies in EM and CM40; however, the real-world comparative effectiveness of erenumab has not been fully investigated.In October 2010, the US Food and Drug Administration (FDA) approved onabotulinumtoxinA (Botox) injection therapy for the preventive treatment of CM,The aim of this study was to evaluate the effectiveness of erenumab and onabotulinumtoxinA in a real-world setting by examining acute medication usage and health care resource utilization (HCRU) among commercially insured patients with migraine using a US administrative claims database.Data were extracted from Optum’s deidentified Clinformatics Data Mart (CDM) Database, a database of administrative health claims of beneficiaries from commercial and Medicare Advantage health plans. Data are derived from claims submitted by providers and pharmacies to obtain payment for health care services rendered, track plan membership for premium billing, and track participating physicians who have contracts with health plans to provide services. Such data provide a key source of information for a variety of research efforts, including research related to health care costs and resource utilization, as well as quality and effectiveness.This study was a retrospective, exploratory, treatment effectiveness, noninterventional claims database analysis that aimed to evaluate the real-world impact of erenumab and onabotulinumtoxinA on acute medication usage and HCRU  among patients with migraine 6 months after treatment initiation.41 and the ethical principles laid down in the Declaration of Helsinki. This study was exempt from institutional review board approval, since only deidentified patient records were used.This study was conducted in accordance with the Guidelines for Good Pharmacoepidemiology Practices of the International Society for Pharmacoepidemiology 2016, the Strengthening the Reporting of Observational Studies in Epidemiology guidelines,Eligible patients were selected based on having at least 1 migraine diagnosis between May 1, 2017, and September 30, 2019; having at least 1 prescription for erenumab or onabotulinumtoxinA between May 1, 2018, and September 30, 2019; having continuous medical/pharmacy coverage in the 12-month pre-index and 6-month post-index periods; and being aged at least 18 years as of the index date.In the erenumab cohort, patients were required to have at least 3 claims of erenumab in the 6-month post-index period, including the index date. OnabotulinumtoxinA has a quarterly dosing schedule, and therefore patients were required to have at least 1 prescription for onabotulinumtoxinA and an associated migraine diagnosis on or before 14 days of the medication claim. The index date refers to the first erenumab/onabotulinumtoxinA claim.Patients were excluded from the erenumab cohort if they used other anti-CGRP therapies during the 12-month pre-index or 6-month post-index period. Patients were excluded from the onabotulinumtoxinA cohort if they had a prescription for onabotulinumtoxinA for any reason or if they used any anti-CGRP therapy, including erenumab, in the 12-month pre-index period, or if they used any anti-CGRP therapy, including erenumab, in the 6-month post-index period.Data from the 12-month pre-index period for both the erenumab and onabotulinumtoxinA cohorts were used to determine most baseline characteristics . Data from the 6-month pre-index period were used to determine baseline acute medication use.Since some of the acute and preventive medications are nonmigraine-specific and approved for other conditions, a claim associated with a migraine diagnosis was required. Use of nonmigraine-specific acute medication  required a migraine diagnosis on or before 7 days of the medication claim. Use of nonmigraine-specific preventive medication  required a migraine diagnosis on or before 14 days of the medication claim, with a supply of at least 28 days.In order to evaluate the effectiveness and real-world impact of erenumab and onabotulinumtoxinA on acute medication usage, the number of claims per person for acute medication at a specific class level , the proportion of patients who used acute medication, and the number of type of acute medications used in the 6-month post-index period were assessed.In order to evaluate the effectiveness and real-world impact of erenumab and onabotulinumtoxinA on HCRU, the number of all-cause and migraine-specific ED/inpatient visits, office visits, neurologist or headache specialist visits, and other outpatient visits per person at 6 months following treatment initiation of erenumab or onabotulinumtoxinA were assessed. The proportion of patients with all-cause and migraine-specific visits at 6 months following treatment initiation of erenumab or onabotulinumtoxinA were also evaluated. Diagnosis positions 1-5 were used for migraine-specific office visits, including neurologist or headache specialist visits and other outpatient visits. Diagnosis position 1 (primary diagnosis) was used for migraine-specific ED/inpatient visits.An exploratory objective was to explore the composite endpoint of: (1) outpatient visits with a diagnosis of migraine and an associated acute medication claim, (2) hospital admissions with a primary diagnosis for migraine, or (3) ED visits with a primary diagnosis for migraine. Any events that occurred on or fewer than 3 days apart were counted only once.Data analyses were performed using SAS 3.8 (SAS Institute). Erenumab and onabotulinumtoxinA cohorts were matched 1:1 using the propensity score (PS) method with stratification. Greedy nearest neighbor matching with caliper = 0.1 was used. Stratification was performed based on gender, diagnosis of CM without aura, onabotulinumtoxinA use, number of preventive drugs used during the 12-month pre-index period, and number of acute drugs used during the 6-month pre-index period. The variables for PS matching included age, gender, insurance type, region, Charlson Comorbidity Index (CCI), selected comorbidities, acute/preventive drug use, and HCRU. Bivariate analyses of baseline characteristics on pre- and post-PS matching were performed, with the standardized mean difference (SMD) reported to assess the heterogeneity of the 2 cohorts.Supplementary Materials, available in online article). When the P value was < 0.05, groups were considered to be significantly different.PS-matched data were used to assess the comparative effectiveness of erenumab and onabotulinumtoxinA; logistic regression with covariate adjustment was used for dichotomous variables, and a negative binomial model with covariate adjustment was used for count variables, with odds ratios (ORs) or rate ratios (RRs) and 95% CIs calculated. A proportional odds model was used for ordinal variables. Sensitivity analysis was performed with the inverse probability of treatment weighting (IPTW) model using pre-matched data . Patients with at least 1 migraine diagnosis between May 1, 2017, and September 30, 2019, were identified from the Optum’s deidentified CDM database. In total, 3,171 patients met the criteria for inclusion in the erenumab cohort and 3,100 for the onabotulinumtoxinA cohort (Supplementary Table 1 (available in online article). In the post-matched data, the SMD was mainly kept below 0.1, a threshold that has been recommended for declaring balance.42An overview of the study design is shown in P = 0.0069; Supplementary Figure 2, available in online article).At 6 months, the adjusted average number of claims per person for any acute medication was significantly lower in the erenumab cohort . The adjusted average number of claims per person for opioids in the 6-month post-index period was significantly lower in the erenumab cohort . The adjusted average number of claims per person for NSAIDs was also significantly lower in the erenumab cohort . The adjusted number of claims per person for barbiturates was numerically lower, but statistically nonsignificant, in the erenumab cohort . Similarly, the adjusted proportion of patients who used barbiturates was numerically lower, but statistically nonsignificant, in the erenumab cohort . The results for ergot use in the 6-month post-index period are not reported due to insufficient data.The adjusted average number of claims per person for triptans in the 6-month post-index period was numerically lower, but statistically nonsignificant, in the erenumab cohort  and for the proportion of patients who used acute medication  in the 6-month post-index period.The adjusted number of different types of acute medications used  at drug class level and generic drug level was similar for both cohorts (data not shown). Sensitivity analysis using the IPTW model demonstrated similar findings for the number of claims per person for acute medication use .By 6 months, the adjusted average number of all-cause office visits per person was significantly lower in the erenumab cohort  (data from IPTW model), while the adjusted proportion of patients with migraine-specific visits to a neurologist or headache specialist was 58.8% in the erenumab cohort compared with 77.4% in the onabotulinumtoxinA cohort .The adjusted average number of all-cause visits per person to a neurologist or headache specialist in the 6-month post-index period was lower in the erenumab cohort , as was the adjusted number of migraine-specific other outpatient visits .The adjusted average number of all-cause other outpatient visits per person in the 6-month post-index period was lower in the erenumab cohort , while numerically fewer patients in the erenumab cohort had a migraine-specific ED/inpatient visit .The adjusted average number of all-cause ED/inpatient visits per person in the 6-month post-index period was not significantly different between cohorts; however, the adjusted number of migraine-specific ED/inpatient visits was lower in the erenumab cohort  and for the proportion of patients with allcause and migraine-specific visits to health care providers  in the 6-month post-index period.Sensitivity analysis using the IPTW model demonstrated similar findings for the number of all-cause and migraine-specific visits to health care providers .Patients in the erenumab cohort had a significantly lower number of composite events than the onabotulinumtoxinA cohort  and for the proportion of patients with a composite event  in the 6-month postindex period.Sensitivity analysis using the IPTW model demonstrated similar findings for the number of composite events per person  to a greater extent than onabotulinumtoxinA over a 6-month period, triptan and barbiturate use remains similar with both migraine preventive medications. Whether a longer duration of treatment (or a bigger sample size in the case of barbiturates) is needed to observe a difference between cohorts remains to be seen. A separate publication will provide a robust assessment of the effectiveness of erenumab on acute medication use and HCRU in patients with migraine 6 months before and after treatment initiation.Depending on the severity of attacks, triptans or NSAIDs are generally recommended as first-line options for the acute treatment of migraine.5 Health care and lost productivity costs associated with migraine are estimated at $36 billion annually,1 with the majority of direct costs due to higher utilization of health care resources in migraine patients compared with demographically matched controls.5 Headache is consistently the fourth- or fifth-leading cause of ED visits,54 with migraine prompting at least 1.2 million ED visits in the United States annually.55 Previous studies have shown that the use of preventive therapy alongside acute treatment can reduce resource utilization in patients with migraine.57 In this study, patients in the erenumab cohort made significantly fewer all-cause and migraine-specific office visits compared with the onabotulinumtoxinA cohort. In particular, the number of migraine-specific visits to a neurologist or headache specialist, which may be indicative of greater disease severity, was lower in the erenumab cohort, as was the number of migraine-specific ED/inpatient visits. We recently demonstrated that outpatient visits, including office and other outpatient visits, are the main driver of HCRU-associated cost.58 Since erenumab reduced the number of these visits to a greater extent than onabotulinumtoxinA, HCRU-associated costs would also likely be lower with erenumab; however, further studies are needed to confirm this.Migraine imposes a substantial direct and indirect cost burden on the United States.Composite endpoints such as the one reported here have the potential to be used as a proxy to evaluate migraine attacks, although further validation is warranted. In this study, the number of composite events per person and the proportion of patients with any of the 3 composite events in the 6-month post-index period were significantly lower in the erenumab cohort compared with the onabotulinumtoxinA cohort. These findings suggest that preventive treatment with erenumab may be associated with a greater reduction in the frequency of migraine attacks than treatment with onabotulinumtoxinA, which in turn could lead to an overall reduction in the burden of the disease.29 By using PS-matched cohorts, any inherent differences between these treatment groups are removed. Furthermore, a sensitivity analysis using the IPTW model produced similar findings.In clinical practice, onabotulinumtoxinA is approved for use in CM only, while erenumab can be used for the preventive treatment of patients with EM and CM.International Classification of Diseases coding, not use of the International Classification of Headache Disorders, 3rd Edition; therefore, some diagnoses may be missed, different professional types may have different coding patterns, and not all coding may be accurate.Using retrospective data from administrative claims has several limitations that should be noted. Due to the observational design of the study, the analysis may be affected by unobserved confounding factors—eg, data contained in administrative claims databases may be inaccurate or subject to human or technical error. US claims data are also dependent on professional Clinical information is also lacking beyond diagnosis codes. Optum’s deidentified CDM database only includes data on insured patients in the United States; therefore, study findings are not generalizable to the entire population. Furthermore, patients with migraine frequently use over-the-counter acute medication, which is generally not included in the claims database and is thus unaccounted for in this study.Patients receiving erenumab from the free drug program are also not captured in this database; thus, it is impossible to ascertain whether the first erenumab claim is truly indicative of the first time erenumab is used by a given patient.Another potential limitation is that onabotulinumtoxinA has a quarterly dosing regimen administered by a health care professional, while erenumab is self-administered by the patient. Although onabotulinumtoxinA users may have had at least 1 more office visit than erenumab users during the 6-month follow-up time period in order to facilitate onabotulinumtoxinA administration, it is likely physicians required a follow-up visit with erenumab users within 3 months of treatment initiation since this was a relatively new biologic from a new drug class at the time.Finally, it should be noted that the absence of an event, condition, or medication in the database is not confirmatory that it did not occur, while prescriptions filled at a pharmacy are not confirmatory of the medication being consumed/self-administered by the patient.The findings from this retrospective real-world study indicate that erenumab reduces the use of acute medications  to a greater extent than onabotulinumtoxinA. Furthermore, erenumab was associated with lower HCRU and a lower number of composite events. Together, these findings suggest erenumab may be more effective as a migraine preventive therapy in this study population during this period of time.


We further developed a method that integrates the associations of cis- and trans-expression components of genes with traits to identify causal genes at even small-effect loci and construct regulatory networks. We identified 36 putative causal genes of SPP, including SDT (MIR156j) and OsMADS17, and inferred that OsMADS17 regulates SDT expression, which was experimentally validated. Our study reveals the impact of regulatory variants on rice panicle architecture and provides new insights into the gene regulatory networks of panicle traits.Panicle architecture is a key determinant of rice grain yield and is mainly determined at the 1-2 mm young panicle stage. Here, we investigated the transcriptome of the 1-2 mm young panicles from 275 rice varieties and identified thousands of genes whose expression levels were associated with panicle traits. Multimodel association studies suggested that many small-effect genetic loci determine spikelet per panicle (SPP) by regulating the expression of genes associated with panicle traits. We found that alleles at  Panicle architecture significantly determines rice grain yield. Here, the authors investigate the transcriptome of young panicles from 275 rice varieties and propose a method to identify causal genes for panicle traits and construct regulatory networks. Oryza sativa) is an important food crop that feeds billions of people worldwide. Panicle architecture is a key factor in determining rice grain yield, therefore identifying genes involved in panicle development, exploring their natural variation, and understanding their regulatory mechanisms are essential for breeding high-yielding rice varieties1. Based on mutant analyses or bi-parental populations, scientists have cloned several genes involved in panicle development, such as OSH12, LAX13, Gn1a4, OsSPL146, DEP17, and FZP8. However, their regulatory networks are not fully understood, and for most of them, the favorable alleles remain unexplored9. In recent years, genome-wide association studies (GWAS) have been widely used in plants to explore the abundant natural variations, and GWAS for panicle traits have been reported in rice13. However, only a few genes under quantitative trait loci (QTLs) have been identified by GWAS as the large linkage disequilibrium (LD) decay distance of the rice genome and the complexity of the genetic basis of panicle traits. Therefore, it is necessary to utilize advanced genomic technologies and develop appropriate methods for a more comprehensive and effective understanding of the regulatory mechanisms of rice panicle traits.Rice  project collects and analyzes transcriptome data of multiple tissues from different individuals, demonstrating the effectiveness of population transcriptome data in identifying genetic variations that can explain differences in gene expression among individuals, known as expression quantitative trait loci (eQTLs). Currently, we are running a Rice GTEx project that aims to build a comprehensive resource to study tissue-specific gene expression and regulation in rice. This will help identify causal genes and understand the molecular processes through which genetic variations affect agronomic traits.Gene expression variation regulated by non-coding OsMADS17 as a causal gene that affects the number of spikelets per panicle (SPP) and regulates the expression of another causal gene, SDT. Our study provides a valuable resource for understanding the regulatory mechanisms of rice panicle traits and potential targets for molecular breeding.This study is part of the Rice GTEx project and focuses on developing analysis strategies appropriate for crops and understanding gene regulation related to rice panicle traits. In this study, we collected young panicle samples and conducted a transcriptomic study on 275 representative rice varieties Fig. . We iden18 and collected panicle traits including SPP, number of primary branches (NPB), and panicle length (PL) . This result, combined with the results of association analysis of SPP and NPB (lack of significant major effect loci but many peaks below the significance threshold), suggests that SPP and NPB may be more regulated by small-effect loci. To confirm this speculation, we used LASSO21 to estimate the phenotypic variances explained by GWAS loci identified at different thresholds (“Methods”). We found that the variances of SPP and NPB explained by GWAS loci were relatively small at the threshold of 10−7 . Further permutation tests confirmed that the phenotypic variances explained by the small-effect loci of GWAS were significant  and is a YABBY transcription factor reported to control seed shattering and undergo parallel selection during domestication in multiple cereals23. Another YABBY is OsYABBY1  which is specifically expressed in the palea and lemma from their inception and controls spikelets development24. The six MADS transcription factors include OsMADS1 , OsMADS7 , OsMADS8 , OsMADS3 , and two AGAMOUS-LIKE6 genes . These MADS genes have been found to be important in controlling spikelet initiation and spikelet organ development31. Our results show that the expression of these genes is negatively correlated with SPP, which is also consistent with the view that inhibiting the transformation of spikelets prolongs the development time of branches and increases SPP32. These results indicate the important value of our data in uncovering the regulatory direction of genes for panicle traits.We carried out transcriptome-wide association studies (TWASs) by associating the expression values of each expressed gene in the population with panicle traits. At a false discovery rate (FDR) of 0.05, 4175, 5844, and 6839 significantly associated genes were detected for SPP, NPB, and PL, respectively , OSH1 , OsPID , and PLA1  were in the list  and MIR529a are negatively associated with IPA1  and SPP . In addition, we also identified other miRNAs associated with panicle traits, such as miR39635  and miR17233  which have been reported to play an important role in panicle development.We further surveyed the known panicle development-related genes among the top 500 most significantly associated genes and found ist Fig. . The idep value = 2.39 × 10−4) and genes that were progressively down-regulated in panicle development , while the negatively associated genes (NAGs) enriched for genes that were progressively up-regulated in panicle development  of SPP enriched for genes that were preferentially expressed in young panicles . However, for the SPP and NPB, no pQTL were found to be associated with more than 10 TWAS significant genes at the suggestive threshold, and pQTLs obtained using the 275 varieties were similar, implying that the expression variation of TWAS significant genes might not be regulated by the GWAS-significant loci, which is different from the results reported in other crops39. On the other hand, genetic correlation analysis suggests that a large part of the correlation between TWAS significant genes and panicle traits could be explained by genetic causes  at the relaxed pQTL thresholds . It could be that there is still some heterogeneity between the 529 varieties and the 275 varieties, although the 275 varieties were selected as representative of the 529 varieties. Therefore, we used the GWAS results of the 275 varieties panel if not specifically stated in the later analysis.To confirm that speculation, we used LASSO to predict the expression of the top 500 TWAS significant genes in different varieties based on lead variants of pQTLs identified at different thresholds and then calculated Pearson’s correlation coefficients between the predicted gene expression values and panicle traits (“Methods”). We found that the correlations were low at the pQTL threshold of 10lly Fig. . Furtherp value <10-4 for both pQTLs and eQTLs). A pQTL is defined as a pQTL-eQTL hotspot if it is an eQTL for many genes and these genes are significantly enriched for more than 10 TWAS significant genes . We identified 37, 31, and 176 pQTL-eQTL hotspots for the TWAS significant genes of SPP, NPB, and PL, respectively  of lead variants of cis-eQTLs. We observed that as DAF increased, the proportion of variants belonging to lead variants of cis-eQTLs gradually increased, suggesting that cis-regulatory variants play an important role as targets of natural or artificial selection in rice , GWAS signals of cis-eQTLs for genes with higher cis-eQTL DAF had a greater departure from an expected p value distribution , GWAS signals of cis-eQTLs for genes with lower cis-eQTL DAF had a greater departure from the null distribution , which is positively associated with SPP  for this gene up-regulates the gene expression , the expression of this gene is negatively associated with SPP  down-regulates the expression of this gene , while the variation of gene expression caused by distal variants is defined as the trans-expression component (trans-EC). If Gene B is expressed variably among individuals and its cis-EC is associated with variation in a phenotype, then Gene B may be a causal gene affecting the phenotype, but it may also be a false positive simply due to LD between cis-regulatory variants of Gene B and a nearby causal variant of the phenotype. In contrast, if not only the cis-EC of Gene C is associated with the phenotype, but also the trans-EC of Gene C is associated with the phenotype, i.e., Gene C may be in a regulatory network that affects the phenotype, and not only its cis-regulatory variants may affect the phenotype, but it is also regulated by other regulators  affecting the phenotype, then Gene C is more likely to be a causal gene compared to Gene B.GWAS can rapidly locate QTLs affecting phenotypes, but further identification of candidate genes within the QTL regions remains a challenge. This problem is particularly serious in rice due to the large LD decay distance. Based on the population transcriptome data, we proposed a method to combine information on cis-EC of each gene using GCTA based on variants within 100 kb of the gene, and then subtracted cis-EC from the gene expression and used the residuals as trans-EC (Methods). At the threshold of 1.62 × 10−6 , 14,392 genes with significant cis-genetic variance were identified and used for subsequent analyses. We then associated cis- and trans-ECs with panicle traits separately  and prioritized genes with both cis- and trans-EC associated with phenotypes as candidate genes. Consistent with the aforementioned results that panicle traits were primarily regulated by small-effect loci, we found only two genes whose cis-EC was significantly associated with SPP and none for NPB (FDR <0.05). At a relatively relaxed threshold , we identified 288, 259, and 477 genes with cis-EC associated with SPP, NPB, and PL, respectively. Finally, by considering the results of trans-ECAS simultaneously (FDR < 0.01), we identified 36, 48 and 99 putative causal genes (referred to as cis- and trans-ECAS genes) significantly associated with SPP, NPB and PL, respectively  located within 5 kb upstream of the transcription start site (TSS) was significantly associated with the expression of SDT. Besides, we found a variant (vg0626555871) in the second intron of SDT, which is in LD with SDT-V1 but less significantly associated with SDT expression and SPP. By checking 33 high-quality rice genome sequences44, we found that this variant actually is an ~2.7 kb insertion rich in “CTAT” repeats. Moreover, we noticed that two SNPs  at ~24 kb and 20 kb upstream of the TSS were significantly associated with SDT expression but not in LD with SDT-V1. Since SDT-V2 was in LD with SDT-V3 and more significantly associated with SDT expression than SDT-V3, we finally prioritized SDT-V2 as well as SDT-V1 as candidate causal variants. The three haplotypes formed by SDT-V1 and SDT-V2 showed significant differences in cis-EC of SDT and SPP  MADS-box gene, and its paralog OsMADS6 was found to play an important role in regulating rice floral organ identity and floral meristem determinacy45, while their function on SPP has not been reported. We found that both the cis- and trans-EC of OsMADS17 were negatively associated with SPP , probably because their expressions were both regulated by more than one independent cis-variant. The fitted cis-EC combined the effects of multiple cis-variants and thus had a higher power to detect associations compared to the associations between individual variants and phenotypes. These results demonstrate the benefit of using population-level transcriptome data to identify causal genes.The associated SDT and OsMADS17 prompted us to ask whether the cis- and trans-ECAS genes tend to have stronger GWAS signals for panicle traits. We found that indeed the GWAS signals of cis-eQTLs for the cis- and trans-ECAS genes had a great departure from the null distribution compared to the GWAS signals of all variants  the GWAS signals of the cis-eQTLs of the TWAS significant gene. This result suggests that the cis-regulatory variants of the cis- and trans-ECAS genes have a stronger GWAS signal compared to the TWAS significant genes and cis- and trans-ECAS genes are more likely to be causal genes affecting panicle traits.The examples of nts Fig. . Althougcis-regulatory variants on phenotype is relatively weak, we wanted to know whether these cis-regulatory variants of cis- and trans-ECAS genes could explain the variation in phenotypes. We first defined alleles of cis-eQTLs that up-regulate PAGs or down-regulate NAGs as “favorable alleles” and then counted the number of favorable alleles in each variety to predict phenotypes in the remaining 254 of the 529 varieties for which the transcriptome data were not available. The results show that the number of favorable alleles of the cis and trans-ECAS genes in each variety for all three traits are highly correlated with the corresponding phenotypes  than the correlations between the phenotypes and the number of favorable alleles for the same number of randomly selected TWAS significant genes with significant cis-genetic variance  and associate it with all genes' cis- and trans-EC to identify putative upstream regulatory genes of the target gene. Alternatively, we can also examine the association between the cis- and trans-EC of a regulatory gene with the expression of all other genes to identify putative downstream genes. In this way, we can construct causal gene regulatory networks, which utilize information from genetic variation and are therefore more reliable than co-expression networks.Similar to the identification of candidate genes affecting phenotypes, it is still a challenge to identify regulatory genes affecting gene expression. In the above sections, we demonstrated that OsMADS17 as an example. ECAS analysis indicated that it may be a causal gene that negatively regulates SPP , we found that OsMADS17 might be an upstream regulator of 64 TWAS significant genes  with the expression of SDT (as dependent variable), the cis- and trans-EC of OsMADS17 ranked 26th and 1st, respectively, in the ECAS results. These results suggest that OsMADS17 might be a regulator of SDT and negatively regulate SPP by up-regulating SDT expression.Taking SPP Fig. , but itsSDT, we collected young panicles of three varieties with different haplotypes on SDT to perform the assay for transposase-accessible chromatin using sequencing (ATAC-seq). From the ATAC-seq results, we noticed that the chromatin accessibility of a distal open chromatin region (dOCR) 4.2–4.4 kb upstream of the TSS of SDT  was consistent with expression variation of SDT for the three varieties  further confirmed that OsMADS17 can directly bind to this dOCR  results of old Fig. , making SDT, OsMADS17 may also negatively regulate RFL and OSH1  compared the knock-out line CR-osmads17 to wild type, while no significant difference was found in NPB  and the other (vg0429308367) is a 65-bp long sequence inserted at the start codon of the ORF, potentially shifting the ORF backwards by 15 bp. However, these two variants are not associated with the SPP . We also identified 12 variants within 5 kb upstream of the TSS of OsMADS17 that were significantly associated with its expression level . Based on all 14 variants, we identified three main haplotypes in the OsMADS17 region  coincides to the three haplotypes and is most significantly associated with the expression level of OsMADS17 . This variant also has the highest functional impact score among the 12 non-coding variants predicted by a deep learning-based approach in RiceVarMap46, suggesting that it may be a functional variant that leads to changes in OsMADS17 expression. At this site, we observed that both types of deletions were associated with down-regulation of OsMADS17 and an increase in SPP  derived from the tropical japonica variety IRAT109 into a breeding line 1035 with temperate japonica background, which carries the reference genotype of OsMADS17. We selected a line heterozygous on OsMADS17-V1 in a BC4F3 population and examined the phenotypes of its segregating progenies based on the genotypes of OsMADS17-V1. The results showed a 25.0% increase in NSB and an 18.5% increase in SPP for lines with the 39-bp deletion on OsMADS17-V1 compared to lines with the reference genotype  and lower frequencies in temperate japonica and indica  which have larger planting areas. Similarly, the superior haplotype of SDT (G_G) has a higher frequency in tropical japonica (0.687) but a lower frequency in other subgroups (<0.1). These results suggest that the superior alleles of OsMADS17 and SDT have not been widely used in breeding and have great potential for enhancing grain number in the main cultivars.We examined the distribution of the 39-bp-deletion allele of 47. Meanwhile, which of these associated genes are causative and the regulatory networks among these genes are unclear, appropriate methods are needed to overcome these limitations.In this study, we identified a large number of genes whose expression levels were associated with panicle traits in rice for the first time. Although association is not causation, these information provide new insights and resources to further resolve the regulatory networks of panicle development. Among those genes associated with SPP, NAGs are enriched in negative regulators of SPP and might provide abundant targets for improving rice yield using genome editing techniques such as CRISPR-Cascis-component of gene expression and the phenotype48. However, genes identified by this approach are also prone to be false positives due to LD between cis-regulatory variants and phenotypic causal variants50. We developed a method to integrate the association of cis- and trans-components of gene expression with phenotype to screen for causal genes affecting phenotypes or gene expression. This method is particularly useful for crop research, as it is relatively easy to obtain expression data and phenotype data for the same population simultaneously. To enhance the power of identifying causal genes, we recommend: (1) integrating phenotype data across years or locations; (2) obtaining phenotype data from samples grown in the same environment as transcriptome samples; (3) acquiring high-quality transcriptomes from as many varieties as possible.In human medical studies, several methods have been developed in recent years to identify causal genes affecting phenotype based on the correlation between the cis-variants of SDT and OsMADS17 affect panicle traits and further demonstrated that OsMADS17 regulated the expression of SDT. Since SDT encodes MIR156j which is the negative regulator of IPA16, our study reveals a regulatory mechanism by which OsMADS17 up-regulates SDT and thus negatively regulates IPA1 to reduce SPP. The superior alleles of OsMADS17 and SDT might provide a useful resource for breeding. In addition, despite the weak GWAS signals of SDT and OsMADS17 , the results of cis- and trans-EC associations strongly support that they are causal genes, demonstrating the ability of our method to identify causal genes in small-effect loci. The identified cis- and trans-ECAS genes are worth further experimental validation.We found that 52, but whether cis-regulatory variants affecting gene expression are under selection during domestication or breeding, and how the selection affects agronomic traits in crops has rarely been studied. We found that the variants with higher DAF are more likely to be cis-eQTLs of genes, indicating that cis-regulatory variants of gene expression were subject to positive selection, which is consistent with our previous study finding that high-impact regulatory variants in rice are subject to positive selection46. However, unlike the study in maize finding that domestication and breeding processes prefer to increase gene expression41, we found that the selection direction of the derived alleles of cis-eQTL is differentiated and related to gene function. The derived alleles with high-frequency prefer to up-regulate the expression of PAGs or down-regulate the expression of NAGs, i.e., tend to have a positive effect on SPP. In contrast, derived alleles of cis-eQTL which have a negative effect on SPP tend to have lower frequencies. This difference in selection is further supported by the difference in GWAS signals for lead variants of cis-eQTL with positive and negative effects of the derived alleles on the SPP phenotype. This indicates that domestication and improvement rewired gene expression to increase grain numbers in rice.The genomic footprint of domestication or breeding processes in crops has been widely studiedIn conclusion, our study provides insights into the complex genetic structure and regulatory networks of panicle architecture and has potential implications for molecular breeding in rice. In addition, the methods and strategies developed in our study can efficiently identify causal genes affecting agronomic traits, even those at small effect GWAS loci, which are expected to facilitate research in other crops as well.53. The varieties were selected to represent both the genetic diversity in Asian cultivated rice (O. sativa L.) and their usefulness in breeding. Based on genetic distance among varieties, we further selected 300 varieties from them to obtain young panicles. We first selected a fixed set of varieties  and then iteratively selected the variety with the maximum distance to any variety in the current set of varieties. This process continues until 300 varieties have been selected. These varieties were planted in four batches in the experimental field of Huazhong Agricultural University during the summer at Wuhan , China. One variety, Zhenshan 97, was planted in all four batches and its young panicles were collected each time. Total RNA was extracted from young panicles (1–2 mm) of 275 rice varieties using the TRIzol reagent (Invitrogen). Totally 278 paired-end libraries (one library for each of 274 varieties and four libraries for Zhenshan 97) were constructed and sequenced using the Illumina HiSeqX platform at Novogene to obtain >9 Gb of PE150 sequencing data for each library. The high correlations between the four replicates of Zhenshan 97 indicate that the effect of planting batch on the transcriptome is small  were calculated to correct for differences in library size across samples. The TPMs of different transcripts belonging to the same gene were combined as the gene’s TPM. Then the TPMs were added by one, and log2 transformed as gene expression levels. We screened genes with expression levels greater than 0.5 in more than 14 varieties (5% of the 275 varieties) as expressed genes and obtained 30,869 expressed genes. To control system bias, we used PEER57 to exclude the first three factors, and the residuals were used as corrected gene expression levels. Then a rank-based approach was used to transform the distribution of expression levels of each gene in the population into a normal distribution.We first counted the read numbers of RGAP (Rice Genome Annotation Project) annotated genes12, We used the lmer function in the R package lme458 to fit the best linear unbiased prediction model (BLUP) with both year and variety as random effects, and then used a rank-based approach to transform the phenotypic values of the BLUP to a normal distribution.The 275 rice varieties for which young panicle transcriptome data were obtained were also investigated for panicle phenotypes. Phenotypic data for each variety were obtained from five independent plants. The main panicles of five independent plants were collected, the primary and secondary branches were separated with tape and pasted on white paper, and RGB pictures were taken and saved by a camera. Further statistics were conducted on spikelets per panicle (SPP), number of primary branches (NPB), and panicle length (PL). To integrate published phenotypic data from Hainan 2013, Wuhan 2013, and Wuhan 201446, and variants with minor allele frequency (MAF) greater than 0.05 and the number of minor alleles greater than six were used for subsequent analysis, with a total of 10,567,425 variants in the 529 varieties panel. We used linear mixed models for association studies, EMMAX59 was used to implement the calculations, and all variants with MAF greater than 0.05 were used to calculate the kinship matrix by PLINK60. The genome-wide significance threshold for GWAS was determined by the significance threshold corrected for multiple hypothesis testing, and the GEC61 showed a total of 1,970,782 independent variants in 529 varieties panel, and the number of independent variants in 275 varieties panel for which transcriptome data were obtained was quite similar , thus the genome-wide significance threshold was set at 2.54 × 10−8 and the suggestive threshold was set at 5.07 × 10−7.Genotype data were obtained from RiceVarMap V2The panicle traits were associated with the expression levels of each expressed gene to uncover panicle development-related genes. The following linear mixed model were used:59 was used to implement the calculations. Genes with FDR less than 0.05 were considered to be significantly associated with the trait.Here, 20. All common variants were used to calculate the genetic relationship matrix (GRM), with the parameter “--autosome --maf 0.05 --make-grm-inbred”. Then, the restricted maximum likelihood method was used to estimate SNP heritability. Genetic correlations between the expression of genes and panicle traits were also estimated using GCTA, with the parameter “--reml-bivar 1 2 --reml-bivar-nocove --reml-bivar-lrt-rg 0”. To estimate the phenotypic variances explained by GWAS loci identified at different thresholds, we first screened all lead variants which association p values smaller than 10−3. Then regression models for phenotypes were constructed using variants exceeding different p value thresholds , respectively. The regression models were fitted by LASSO and implemented using the cv.glmnet function in the glmnet package62, and the 10-fold cross-validation R2 was employed to represent the explained phenotypic variance. To evaluate the significance of phenotypic variances explained by the GWAS loci identified at different thresholds, we permuted the phenotypes of panicle traits 500 times and performed GWAS for each permuted trait. Then, the same method was used to estimate the variance of the permuted phenotypes explained by the GWAS loci identified at different thresholds. To demonstrate that the expression variations of TWAS significant genes affecting the panicle traits are mainly contributed by small-effect GWAS loci, we predicted the expression of the top 500 TWAS significant genes in different varieties using the GWAS loci identified at different thresholds . The regression model was fitted by LASSO for each gene at each GWAS threshold. Then, the R2 (square of Pearson’s correlation coefficient between predicted gene expression and panicle trait) was used to represent the genetic-caused expression variation of TWAS significant gene affecting panicle trait. For the permutation test, TWAS was also performed for each of the permuted trait, and then the same analysis as for the panicle trait was performed using the top 500 TWAS genes and the GWAS loci of the permuted trait.SNP heritability for panicle traits was estimated using GCTA36. We calculated the ratio of the average expression in young panicles to the average expression in other tissues for each gene, genes with a ratio greater than 0.95 quantiles of all genes were considered to be preferentially expressed in young panicles. The genes preferentially expressed in young panicles in ZS97 and MH6343 and Nipponbare63 were merged. Gene sets with progressively up-regulated and progressively down-regulated expression in young panicles were from the CREP43. The list of transcription factors is from PlantTFDB64, and the list of chromatin modification-related genes is from Ensembl BioMarts65. Fisher’s exact test was used to examine the TWAS significant genes enriched transcription factor family. GO enrichment analysis and TF binding motif enrichment analysis of TWAS significant genes were performed on the web service of Plant Regulomics37.Gene Set Enrichment Analysis (GSEA) was performed using the GSEApy packagep value <10−4) with the expression level of all expressed genes. We then analyzed the enrichment of associated genes  with TWAS significant genes for each pQTL. The pQTLs whose associated genes were significantly enriched for TWAS significant genes were defined as pQTL-eQTL hotspots .To identify pQTL-eQTL hotspots that affect phenotype and regulate multiple TWAS significant genes, we performed an association analysis of GWAS pQTLs  were removed. For the remaining variants, a total of 1,275,135 variants located within 10 kb of the TSS of genes were used for cis-eQTL analysis. The most significantly associated variant for each gene was taken as the lead variant of cis-eQTL, and a total of 13,877 genes were identified with cis-eQTL under the threshold of 10−5.To assess the effect of rice domestication and breeding improvement processes on gene expression, we obtained the derived alleles and allele frequencies of variants from RiceVarMap V220 to obtain the gene expression levels predicted by cis-genetic variations . We did not estimate trans-expression component  directly, but used the residuals of gene expression after subtracting cis-EC as trans-EC, since estimating trans-EC of genes may be imprecise66. And the plant materials for our transcriptome data of all varieties were planted in the same field, thus the variation in gene expression after removal of cis-EC was mainly from trans-EC and random errors. At the threshold of 1.62 × 10−6 , 14,392 genes had significant cis-genetic variance and were used for subsequent analyses. We then used the following linear mixed models to associate the panicle traits with the cis- and trans-EC of each gene, respectively:We propose a novel strategy to identify causal genes that affect phenotype. We first calculated the genetic relatedness matrix (GRM) using variants within 100 kb of the gene. Then, based on the GRM, BLUP was performed for the expression levels of each gene using GCTAcis-EC of a gene across varieties, trans-EC of a gene across varieties, and cis-EC and trans-EC were associated with the phenotype were prioritized as candidate genes .Here, cis- and trans-ECAS, to identify upstream regulators of genes. The expression values of each TWAS significant gene (FDR <0.01) were used as molecular phenotype (e-trait). We associated the e-trait with the cis- and trans-EC of all other genes with significant cis-genetic variance, and the genes for which both cis- and trans-EC were significantly associated with e-trait  were prioritized as possible upstream regulators.We applied the same method as for identifying causal genes for panicle traits, OsMADS17 were amplified with primer pair NONE-OsMADS17-F/R and then cloned into NONE to generate NONE-OsMADS17 as the effector. The SDT truncated promoters were amplified with primer pair SDT-promoter-4kb-F/R, SDT-promoter-4.6kb-F/R, and SDT-dOCR-F/R, and then cloned into 190-LUC, respectively, to generate 190-LUC-SDT-Pro 4kb, 190-LUC-SDT-Pro 4.6kb, 190-LUC-SDT-dOCR, as reporter. The sequences of primers are listed in Supplementary Data The coding sequences of osmads17 mutant, the sgRNA-CAS9 plant expression vectors were constructed as described previously67. Two target sequences  for 30 min at room temperature, meanwhile, HIS protein was used as a negative control. In competition reaction, 2 pM (100×), 4 pM (200×) and 8 pM (400×) un-labeled probes were mixed with 20 fM biotin-labeled probes, the mixture incubated with HIS-OsMADS17 protein in the binding buffer for 30 min at room temperature. The DNA-protein complex was subjected to 6% native polyacrylamide gel electrophoresis at 4 °C. After gel electrophoresis separation, the biotin-labeled probes were detected using the Chemiluminescent Nuleic Acid Detection Module (Thermo Fisher Scientific) according to the manufacturer’s protocol. All the primers for EMSA were listed in Supplementary Data For EMSA, two complementary oligonucleotides were synthesized, labeled with biotin at 5’ end, and annealed to form DNA probes. The corresponding unlabeled DNA probes were used as competitors. EMSA was performed using the LightShift Chemiluminescent EMSA Kit (Thermo Fisher Scientific). In the experimental group, 20 fM biotin-labeled probes were incubated with HIS-OsMADS17 in the binding buffer  of rice using the TRIzol reagent (Invitrogen) according to the manufacturer’s instructions, and first-strand cDNA was synthesized from 1–5 μg total RNA with HiScript III 1st Strand cDNA Synthesis Kit (+gDNA wiper) . qRT-PCR was performed with the Applied Biosystems 7500 real-time PCR detection system using SYBR Green MasterMix (Applied Biosystems). The data were analyzed using the relative quantification methodFurther information on research design is available in the Supplementary InformationPeer Review FileDescription of Additional Supplementary FilesSupplementary Data 1-14Reporting SummarySource Data


Since the beginning of the COVID-19 pandemic, there has been a growing literature on COVID and work –4 and reClinicians have also observed patients who suffer from persistent symptoms following COVID-19, often called “long-COVID” . The WorThe consequences of such unexpected persistence of disease on the health of the workforce are large. Most of the post-COVID symptoms found in the general population were similar for workers: in a recent review including 60 cohort studies of working age adults and 10 cases studies, the most frequently reported post-COVID-19 symptoms were fatigue (92%), shortness of breath (82%), muscle pain (44%) and joint pain (35%) . In a reAben et al  conducteStrategies promoting return to work for those with post-COVID-19 conditions will need to be implemented and could be comparable to programs developed for other chronic conditions, since post-COVID conditions share some similarities to post-intensive care syndrome, fibromyalgia, and chronic fatigue syndrome , 33, 34.In conclusion, COVID-19 remains an important topic for the occupational health research agenda, including acute and post COVID conditions. Although there is still debate about the definition of what a ‘post-COVID condition’ entails, the sheer number of patients who are not returning to work in a timely manner or returning to work with limitations, and the lack of research interventions available should lead occupational health practitioners and researchers to work not only to prevent infection but to prevent or reduce work disability resulting from the COVID-19 pandemic and future pandemics.


Background: Little research to date has examined the quality of data obtained from resident performance evaluations. This study sought to address this need and compared inter-rater reliability obtained from norm-referenced and criterion-referenced evaluation scaling approaches for faculty completing resident performance evaluations.Methods: Resident performance evaluation data were examined from 2 institutions , with 4 evaluation forms: 2 criterion-referenced  and 2 norm-referenced. Faculty inter-rater reliability was calculated with intraclass correlation coefficients (ICCs)  for each competency area within the form. ICCs were transformed to z-scores, and 95% CIs were computed. Reliabilities for each evaluation form and competency, averages within competency, and averages within scaling type were examined.Results: Inter-rater reliability averages were higher for all competencies that used criterion-referenced scaling relative to those that used norm-referenced scaling. Aggregate scores of all independent categories  for criterion-referenced scaling demonstrated higher reliability  than norm-referenced scaling . Moreover, examination of the distributions of composite scores  suggested that the criterion-referenced evaluations better represented the performance continuum.Conclusion: Criterion-referenced evaluation approaches appear to provide superior inter-rater reliability relative to norm-referenced evaluation scaling approaches. Although more research is needed to identify resident evaluation best practices, using criterion-referenced scaling may provide more valid data than norm-referenced scaling. Graduate medical education programs are required to collect faculty evaluations of resident trainees on a regular basis, but the mechanism by which residents are evaluated is left to the discretion of individual programs. Thus, programs have employed a wide variety of approaches. However, many programs seem to have been influenced by changes in accreditation reporting requirements. For example, in 2013, the Accreditation Council for Graduate Medical Education (ACGME) launched the Next Accreditation System (NAS) that transitioned the focus of resident performance reporting practices toward assessing specific behavioral criteria linked to an underlying competency continuum . Biannually, programs must indicate each trainee's performance on the milestones, and these milestone reports oftentimes heavily rely on faculty evaluations of resident performance. Some performance management systems facilitated milestone reporting by creating default evaluation scales  that likely encouraged programs to use these scales. Furthermore, many individuals may have elected to use the same scaling as the milestones to simplify mapping in general.3 or on the feasibility of reporting,4 or they have used data gathered through the NAS to stratify competency-based achievement by level of training.5 But these studies have not addressed the reliability of evaluation data gathered by programs to support the NAS. This research effort sought to provide some preliminary inferences in this regard.Thus, resident evaluations from many programs similarly transformed to mirror the biannual ACGME milestone reporting scales in which residents are mapped to where they fall on the ability continuum (criterion-referenced). Prior to this transformation, most programs focused on identifying stragglers or those excelling and used comparison-type (norm-referenced) scales. Little published evidence compares these 2 types of scaling approaches. Studies evaluating the NAS have focused on evidence of the validity or construct validity of milestones ratings6In the context of evaluation creation, there are many potential considerations. First, you need to make sure that evaluators will be able to interpret the content in the scales . You need to ensure that the numbers you ask raters to use—the scale you have selected—are appropriate for the content being assessed and the proposed application. Scaling drives the translation from an abstract construct  to a tangible number. But that number can be more or less accurate depending on several factors .6 Thus, resultant ratings indicate whether an individual is better, worse, or about the same as a normative referent (such as the average resident).Norm-referenced evaluation scales require raters to make assumptions regarding where individuals reside on average relative to other similar individuals without specific regard to the competency continuum.7 Certain standards have to be met, and only via aligning trainee performance to these standards can educators ensure that trainees are adequately prepared. Thus, criterion-referenced scales assume an underlying continuum of skill to which resident performance can be mapped .Glaser and Klaus proposed that norm-referenced scaling was not sufficient for educators’ needs because the actual proficiency level of trainees needs to be identified.8 Norm-referenced evaluations may also be easier for nontrained individuals to develop, and they may be more familiar to inexperienced raters.9Arguably, the criterion-referenced scales may remove some ambiguities regarding the comparison group ; however, making norm-referenced comparisons may be less cognitively demanding than making absolute (criterion-based) judgments.10 Effective criterion referencing necessitates an understanding of how individuals on average perform to develop the required standards of performance. Thus, norm referencing is inherent in developing the standards in criterion-referenced assessment. Interestingly, Lok et al also argued that norm-referenced scaling can allow for more freedom for evaluators. For example, some trainees can be forced to fail in a norm-referenced model regardless of the overarching quality of performance. Conversely, with a criterion-referenced model, as long as trainees achieve the standards, they pass. Thus, criterion-referenced approaches may increase grade inflation and passing and may not effectively identify the worst performers.10 However, others have pointed out that norm-based evaluation approaches may lead to standards being set below where they would with criterion-based approaches.11Guidelines to date have mixed inferences at best, with arguments supporting both sides. In the context of education in general, Lok et al suggested that the 2 approaches are to some extent intertwined.12 Using 1-way random effects intraclass correlation coefficients (ICCs), they found absolutely no agreement between residents and faculty on these ratings and low agreement between faculty. Even faculty did not highly agree when rating residents using milestones in an ideal testing context of a standardized simulation. Conversely, Scielzo et al found that resident self-reports and faculty reports of resident criterion-assessed competencies were consistent.13 This study, however, used a correlational design . Pereira and colleagues, arguing that undergraduate medical education needs to move away from norm-referenced assessment, suggested that current norm-based approaches are not predictive of later performance and that undergraduate medical education needed to transition to criterion-referenced assessment.14 Interestingly, they also noted the dearth of research assessing the impact of criterion-referenced vs norm-referenced assessment in graduate medical education.In the context of medical education, little research has examined the impact of scaling type on assessments. For example, Wittels et al examined the correlation of resident self-assessment and corresponding faculty ratings of clinical competence using milestone ratings in the context of a highly standardized training simulation.Thus, as criterion-referenced evaluations are now widely used in resident education, examining this form of evaluation in comparison to its predecessor is imperative. Given our duty to society to ensure that physician trainees are ready for independent practice upon graduation and to ensure that our inferences in this regard are as accurate as possible, examining reliability is an important first step.6 Reliability in evaluation refers to the reproducibility of data or scores over time, event, or items designed to measure the same constructs.15 Reliability is an important consideration in evaluating the utility of a given measurement; without reliability, inferences are not stable. In other words, without reliability, you cannot have validity.17 Imagine your home scale giving the following 3 readings in a 3-minute period: 87 lbs, 205 lbs, and 147 lbs. Because the scale cannot systematically infer a weight, the weight results clearly are not valid, and it is probably time to discard the scale. Appendix B provides general information about reliability.We elected to examine inter-rater reliability to make some initial inferences regarding the quality of the information gained from the evaluation scaling types. Inter-rater reliability, the consensus or degree of agreement among raters, provides one indicator of the psychometric utility of a scale.In the case of resident assessments, low inter-rater reliability might suggest that the evaluation questions were hard to understand, that raters did not actually have enough opportunities to observe rating behaviors, that the items did not provide a good representation of the underlying constructs, or that the scaling approach may not be ideal for the behaviors being measured.In this study, we sought to determine the effect of using norm-referenced vs criterion-referenced scales on the reliability of evaluation data. Specifically, we examined the impact of evaluation scaling type  on the inter-rater reliability of faculty members completing resident performance evaluations.We compared the inter-rater reliability of 4 different evaluation forms. These evaluations came from 3 graduate medical education residency training programs at 2 universities: Dartmouth-Hitchcock and the University of Texas Southwestern (UTSW). Two of the evaluations were criterion-referenced and 2 were norm-referenced. One of the criterion-referenced evaluations had an additional norm-referenced item that we also included in our analysis.Program I was the internal medicine program at Dartmouth-Hitchcock. Program II was the internal medicine program at UTSW. Program III was the general surgery program at UTSW. Program I had both norm-referenced  and criterion-referenced  evaluations. The norm-referenced evaluations were collected in 2012-2013 prior to the ACGME transition, and the criterion-referenced evaluations were collected in 2013-2014. Program II used an evaluation form (n=196) that had predominantly criterion-referenced items (referred to as criterion-referenced 2), but also had 1 norm-referenced item (referred to as norm-referenced 2). Program III had a norm-referenced evaluation . We elected to include all of these different evaluations to try to overcome some of the limitations of this study, such as temporal differences  and having the exact same raters and individuals being rated . Appendix C presents the specific items used in each form and how each was linked to competencies.This study was an archival data analysis, an examination of end-of-rotation data collected for educational purposes . Thus, faculty evaluation behaviors were not influenced by knowing that their scores would be examined. Ethical approval was obtained from the Dartmouth College Institutional Review Board that declared the study exempt from formal review.Data were obtained for residents in postgraduate years (PGY) 1 to 3 in Programs I and II, and from residents in PGY 1 to 5 in Program III. Evaluation items had been mapped during evaluation creation to their relevant ACGME competencies: Practice-Based Learning and Improvement, Patient Care, Professionalism, Medical Knowledge, Interpersonal and Communication Skills, and Systems-Based Practice. A few items assessed overall perceptions of a resident's standing .If multiple items were associated with a competency, an average was computed (see Appendix C for more detail). As an example, if we had 2 items assessing Patient Care, we averaged these 2 items to represent that construct for each evaluation administration .The evaluation for Program II was designed with the goal of minimizing the time to complete while retaining as much important information regarding performance as possible. After careful analysis from the development team, we decided that broader items tapping multiple competencies were preferred. Thus, this evaluation contained 2 items that each represented 2 constructs . Thus, the same reliability index was used in the analysis for these competencies  for Program II.18 This assessment enabled us to examine the correspondence of scores across random raters (any of our faculty members may make these ratings). Similarly, we needed to evaluate whether our raters were in agreement, not just consistent. We also wanted to make estimates for a larger sample of raters consistent with our most important decisions .Inter-rater reliability was assessed with a 1-way random effects model ICC.19 based on an average of 10 assessments per resident, the minimum annual number of evaluations per resident, to provide an estimate of the accuracy of the data used for milestone reporting. This formula allows us to make more accurate estimates of reliability based on applied utilization, rather than just relying on the lower reliability inferences from the 2 observations.The first 2 observations for each trainee were used, with a resultant ICC . These values were then applied to the Spearman-Brown prediction formulaWe examined the average for all criterion-referenced ICC values within a competency compared to the average for all norm-referenced values. Thus, all ICCs were averaged within competency and scaling type. To examine the average reliability based on scaling type, we averaged all independent reliability values within scaling type after removing duplicate values for Program II. Appendix B provides general information about ICCs.z-scores. The 95% CIs were computed around each z-score using 1/sqrt(N-3) for the standard error. Inter-rater reliability estimates were calculated using SPSS Statistics version 27.0 (IBM Corporation),20 and Spearman-Brown estimates and z-score confidence intervals were calculated in Microsoft Excel (Microsoft Corporation).To facilitate comparison and the interpretation of CIs, these effect sizes were translated to Finally, all items were averaged for each individual being rated to provide an estimate of their overarching standing based on these evaluations. Thus, for resident X, all items were averaged for both evaluators, creating a composite estimate of overall ability. Then, these composite scores were broken into 5 equal intervals and mapped back to the underlying scales used to rate the trainees, enabling us to examine the distribution of scores for each program with multiple competencies (excluding the 1 norm-referenced assessment item in Program II) and providing another indicator to examine the quality of the data obtained. Graphs were created for each scaling type in Microsoft Excel.z-score for the average reliability for each scaling type by competency, higher reliability averages were observed for criterion-referenced evaluation scales relative to norm-referenced scales for all competencies (z-score of 1.16 (CI 1.05-1.27) for our criterion-referenced evaluations vs an average z-score of 0.76 (CI 0.63-0.89) for the norm-referenced evaluations. Neither z-score falls within the CI of the other, and thus, with 95% confidence is statistically different. Furthermore, the criterion-referenced item that assessed overall competence was statistically higher than the average of the 2 items using norm-referenced scaling (z=1.83 vs 1.31).First, we examined the extent of the impact of scaling type (norm-referenced vs criterion-referenced) for each competency. Examining the etencies . For exaz=1.37, CI 1.26-1.48) had statistically higher reliability than norm-referenced scaling   plus the question assessing overall competence were averaged. For Program II, the 4 independent competencies  plus the question assessing overall competence were averaged. The norm-referenced average included all 6 competencies plus the question assessing overall competency for Program I, the item assessing overall competence for Program II, and all 6 competencies for Program III. Overall, criterion-referenced scaling , but more research is needed.Future research should examine other validity and reliability estimates, such as the correspondence of ratings with in-service/board examination scores and/or other indicators of performance. Future research needs to evaluate faculty satisfaction with these rating types . Given the importance of our evaluation efforts, much more research is needed to identify best practices for creating the most psychometrically and practically useful evaluation tools possible.We found that criterion-referenced scaling results in higher inter-rater reliability relative to norm-referenced scaling for faculty evaluations of resident performance. Although more research is needed, criterion-referenced scaling appears to provide better inter-rater reliability and, in turn, more valid inferences for deriving milestone data. The results of this study suggest that programs should consider adopting criterion-referenced scaling approaches to ensure that programs are making the most accurate inferences possible when assessing residents’ skills.