The combination of cough and fever/chills has 4.2-fold amplification in COVIDpos patients during the week prior to PCR testing, in addition to anosmia/dysgeusia, constitutes the earliest EHR-derived signature of COVID-19. This study introduces an Augmented Intelligence platform for the real-time synthesis of institutional biomedical knowledge. The platform holds tremendous potential for scaling up curation throughput, thus enabling EHR-powered early disease diagnosis.Understanding temporal dynamics of COVID-19 symptoms could provide fine-grained resolution to guide clinical decision-making. Here, we use deep neural networks over an institution-wide platform for the augmented curation of clinical notes from 77,167 patients subjected to COVID-19 PCR testing. By contrasting Electronic Health Record (EHR)-derived symptoms of COVID-19-positive (COVID The clinical course and prognosis of patients with COVID-19 varies substantially, even among patients with similar age and comorbidities. Following exposure and initial infection with SARS-CoV-2, likely through the upper respiratory tract, patients can remain asymptomatic with active viral replication for days before symptoms manifest . The platform utilizes state-of-the-art transformer neural networks on the unstructured clinical notes to automate entity recognition , quantify the strength of contextual associations between entities, and characterize the nature of association into ‘positive’, ‘negative’, ‘suspected’, or ‘other’ sentiments. We identify specific sensory, respiratory, and gastro-intestinal symptoms, as well as some specific combinations, that appear to be indicative of impending COVIDneg patient diagnoses and 2,317 COVIDpos patient diagnoses. The COVIDpos cohort had a mean age of 41.9 years (standard deviation = 19.1 years) and was 51% male and 49% female while the COVIDneg cohort had a mean age of 50.7 years (standard deviation = 21.4 years) and was 43% male and 57% female. Only 11 (0.5%) of COVIDpos and 196 (0.3%) of COVIDneg patients were hospitalized 7 days or more prior to PCR testing, indicating that the vast majority of patients were not experiencing serious illness prior to this time window. During the week prior to PCR testing, 135 (5.8%) of the COVIDpos and 5981 (8.0%) of the COVIDneg patients were hospitalized. Additionally, the frequencies of ICD10 diagnosis codes for these cohorts were found for the week prior to PCR testing, with unspecified acute upper respiratory infection appearing in over 20% of both cohorts  test approved for human nasopharyngeal and oropharyngeal swab specimens under the U.S. FDA emergency use authorization (EUA) . This PC cohorts . In orde cohorts . For the cohorts . As a nepos over COVIDneg patients in the week preceding PCR testing  while appearing in 10,171 out of 77,167 patients (13.2%).Altered or diminished sense of taste or smell (dysgeusia or anosmia) is the most significantly amplified signal in COVIDidelines  these sypos patients for the week preceding PCR testing  and myalgia/arthralgia has a 2.0-fold amplification . Rhinitis is also a potential early signal of COVIDpos patients that requires some consideration . Finally, dysuria was included as a negative control for COVID-19, and consistent with this assumption, 0.69% of COVIDpos patients and 0.97% of COVIDneg patients had dysuria during the week preceding PCR testing.As may be expected, respiratory difficulty is enriched in the week prior to PCR testing in COVIDpos versus COVIDneg patients in the week prior to the PCR testing date . Examining the other 300 possible pairwise symptom combinations, excluding the altered sense of smell of taste, reveals other interesting combinatorial signals. The combination of cough and diarrhea is noted to be significant in COVIDpos over COVIDneg patients during the week preceding PCR testing; that is cough and diarrhea co-occur in 8.0% of COVIDpos patients and only 2.8% of COVIDneg patients, indicating a 2.8-fold amplification of this specific symptom combination  in the COVIDpos patient cohort over the COVIDneg patient cohort by 5.2-fold on day −7 , 4.31-fold on day −6, 3.6-fold on day −5 , and 3.1-fold on day −4 . The diminishing odds of cough as a symptom from 7 to 4 days preceding the PCR testing date is notable and this temporal pattern could potentially suggest that the duration of cough and other symptoms, in addition to their presence or absence, is a useful indicator of infection. Similarly, diarrhea is amplified in the COVIDpos patient cohort over the COVIDneg patient cohort for days furthest preceding from the PCR testing date, with an amplification of 2.2-fold on day −7  and 1.8-fold on day −6 . Likewise, fever/chills and respiratory difficulty both show matching trends, with significant amplification in the COVIDpos cohort on days −7 to −4 and days −7 to −5, respectively. However, unlike diarrhea, cough, fever/chills, and respiratory difficulty, we find that change in appetite may be considered a subsequent symptom of impending COVID-19 diagnosis, with significant amplification in the COVIDpos cohort over the COVIDneg cohort on day −4 , day −3 , and day −2 . The delay in the onset of change in appetite/intake compared to the other aforementioned symptoms indicates that this change only appears after other symptoms have already manifested and thus could be secondary to these symptoms rather than directly caused by infection.We further investigated the temporal evolution of the proportion of patients with each symptom for the week prior to PCR testing . AlteredThis high-resolution temporal overview of the EHR-derived clinical symptoms as they manifest prior to the SARS-CoV-2 PCR diagnostic testing date for 77,167 patients has revealed specific enriched signals of impending COVID-19 onset. These clinical insights can help modulate social distancing measures and appropriate clinical care for individuals exhibiting the specific sensory , respiratory , gastro-intestinal , and other  symptoms identified herein, including for patients awaiting conclusive COVID-19 diagnostic testing results (e.g. by SARS-CoV-2 RNA RT-PCR).pos over COVIDneg patients in the days preceding confirmed diagnostic testing. This requires leveraging a privacy-preserving, federated software architecture that enables each medical center to retain the span of control of their de-identified EHR databases, while enabling the machine learning models from partners to be deployed in their secure cloud infrastructure. To this end, seamless multi-institute collaborations over an Augmented Intelligence platform, which puts patient privacy and HIPAA-compliance first, are being advanced actively over the Mayo Clinic’s Clinical Data Analytics Platform Initiative (CDAP). The capabilities demonstrated in this study for rapidly synthesizing unstructured clinical notes to develop an EHR-powered clinical diagnosis framework will be further strengthened through such a universal biomedical research platform.While PCR testing is the current diagnostic standard of COVID-19, identifying risk of a positive diagnosis earlier is essential to mitigate the spread of the virus. Patients with these symptom risk factors could be tested earlier, undergo closer monitoring, and be adequately quarantined to not only ensure better treatment for the patient, but to prevent the infection of others. Additionally, as businesses begin to reopen, understanding these risk factors will be critical in areas where comprehensive PCR testing is not possible. This study demonstrates how such symptoms can be extracted from highly unstructured institutional knowledge and synthesized using deep learning and neural networks . Such aupos individual, whereas now PCR testing is more widespread. In each case, the delay between symptom onset and PCR testing would be expected to change over time for a given patient population.There are a few caveats that must be considered when relying solely on EHR inference to track symptoms preceding the PCR testing date. In addition to concerns regarding testing accuracy, there is an inherent delay in PCR testing, which arises because both the patient and physician must decide the symptoms warrant PCR testing. More specifically, to be tested, the patient must first consider the symptoms serious enough to visit the clinic and then the physician must determine the symptoms present a possibility of COVID infection. The length of this delay could also be influenced by how well-informed the public is of COVID-19 signs and symptoms, the availability of PCR testing, and the hospital protocols used to determine which patients get tested. Each of these factors would be absent or limited at the beginning of a pandemic but would increase or improve over time. This makes synchronization across patients difficult because the delay between symptom onset and PCR testing changes over time. For example, patients infected early in the pandemic would be less inclined to visit the clinic with mild symptoms, while those infected later have more information and more cause to get tested earlier. Similarly, in the early stages of the COVID-19 pandemic when PCR testing was limited, physicians were forced to reserve tests for more severe cases or for those who were in direct contact with a COVIDpos and 372 (0.5%) of the COVIDneg patients had notes from all 7 days prior to PCR testing. Moreover, the number of patients with notes prior to PCR testing tends to decrease with each day prior to the PCR testing date for both cohorts  of the COVID cohorts . With repos and COVIDneg clinical symptoms across diverse patient populations to the multi-omics signals from patient-derived bio-specimen will help advance a more holistic understanding of COVID-19 pathophysiology. This will set the stage for a precision medicine approach to the diagnostic and therapeutic management of COVID-19 patients.As we continue to understand the diversity of COVID-19 patient outcomes through holistic inference of EHR systems, it is equally important to invest in uncovering the molecular mechanisms  and gainpos) patients. First, we read through the charts of 100 COVIDpos patients and identified symptoms, grouping them into sets of synonymous words and phrases. For example, ‘SOB’, ‘shortness of breath’, and ‘dyspnea’, among others, were grouped into ‘shortness of breath’. For the SARS-CoV2-positive patients, we identified a total of 26 symptom categories , 'No' (ruled out phenotype), 'Maybe' (suspected phenotype), and 'Other' , with examples of each classification shown in pos patients and 74,850 COVIDneg patients. Each patient is counted only once. Once they have a positive SARS-COV-2 PCR test, they are considered COVIDpos. If a patient were to test negative and then positive subsequently, that day of positive PCR testing is considered day 0 for that patient. We then focus on the notes from seven days prior to the SARS-CoV-2 diagnostic test  were manually verified. Sentences arising from templates, such as patient education documentation, accounted for 10.2% of sentences identified. These template sentences were excluded from the analysis. The true positive rate, defined as the total number of correct classifications divided by the number of total classifications, achieved by the model for classifying all symptoms was 96.7%; the corresponding false positive rate was 6.1%. The model achieved true positive rates ranging from 93% to 100% for the major symptom categories of Fever/Chills, Cough, Respiratory Difficulty, Headache, Fatigue, Myalgia/Arthralgia, Dysuria, Change in appetite/intake, and Diaphoresis. Classification performance was slightly lower for Altered or diminished sense of taste and smell; here, the true positive rate was 82.2%. Detailed statistics are displayed in pos and COVIDneg patients that had positive sentiment for that symptom in at least one note between 1 and 7 days prior to their PCR test. We additionally computed the ratio of those proportions to determine the prevalence of the symptom in the COVIDpos cohort as compared to the COVIDneg cohort; we then computed 95% confidence intervals around these ratios. A standard 2-proportion z hypothesis test was performed, and a p-value was reported for each symptom. A Benjamini-Hochberg adjustment was then applied on these 26-symptom p-values to account for multiple hypotheses. To capture the temporal evolution of symptoms in the COVIDpos and COVIDneg cohorts, the process was repeated considering counts and proportions for each day independently. . Pairwise analysis of phenotypes was performed by considering 325 symptom pairs from the original set of 26 individual symptoms. For each pair, we calculated the number of patients in the COVIDpos and COVIDneg cohorts wherein both symptoms occurred at least once in the week preceding PCR testing. With these patient proportions, a Fisher exact test p-value was computed. A Benjamini-Hochberg adjustment was applied on these 325 Fisher test p-values to account for multiple hypothesis testing.For each synonymous group of symptoms, we computed the count and proportion of COVIDStudy of COVID-19 patient characteristics with augmented curation of Electronic Health Records (EHR) to inform strategic and operational decisions’. All analysis of EHRs was performed in the privacy-preserving environment secured and controlled by the Mayo Clinic. nference and the Mayo Clinic subscribe to the basic ethical principles underlying the conduct of research involving human subjects as set forth in the Belmont Report and strictly ensures compliance with the Common Rule in the Code of Federal Regulations (45 CFR 46) on Protection of Human Subjects.This research was conducted under IRB 20–003278, ‘ In the interests of transparency, eLife publishes the most substantive revision requests and the accompanying author responses.Acceptance summary:The concept of using EHR system and machine learning in COVID and finding anosmia as a specific clinical sign without a specific trigger for doctors to ask this sign is a very nice example of this system to help filter out new signatures in new or old diseases. The manuscript shows how to use big data to elucidate relevant clinical clues in practice.Decision letter after peer review:eLife. Your article has been reviewed by two peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Jos van der Meer as the Senior Editor. The reviewers have opted to remain anonymous.Thank you for submitting your article "Augmented Curation of Unstructured Notes from a Massive EHR System Reveals the Signature of Impending COVID-19 Diagnosis" for consideration by The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.eLife". Please let us know if you would like to pursue this option. As the editors have judged that your manuscript is of interest, but as described below that it needs revision before it is published, we would like to draw your attention to changes in our revision policy that we have made in response to COVID-19 (https://elifesciences.org/articles/57162). We are offering, if you choose, to post the manuscript to bioRxiv  along with this decision letter and a formal designation that the manuscript is "in revision at Summary:The manuscript provides a deep neural networks study over an institution wide EHR platform for the augmented curation of 15.8 million clinical notes from 30,494 patients subjected to COVID-19 PCR testing during the SARS-CoV2 pandemic. The studies focused on potential earlier diagnosis of COVID-19 by identifying specific gastro-intestinal, respiratory, and sensory phenotypes, as well as some of their specific combinations. Overall, the research question is interesting, and would contribute to the understanding of COVID-19 early diagnosis. This part of the manuscript is strong and justifies its publication.However, the manuscript overall needs some reorganization, as there are sections from the Materials and methods and Discussion that belong in the Results, and the overall details about the data, methodology and discussion need better clarity and expansion.The data on RNAseq and treatment strategies  do not belong in the context of this paper.Essential revisions:1) The cohort itself is not well-defined and should be better described (see below).2) Materials and methods section should be written more clearly (see below).3) Remove RNA seq data and treatment figures and discussion about these topics. It is distractive and does not make the manuscript stronger.Cohort description:How many patients had an admission 7 days before getting a swab taken? How many individuals do the authors have a full week of information on prior to the PCR? Why was there a delay in PCR testing of these patients? Who are the 30,000 patients, and why were they admitted to a hospital? Or were they outpatients? The setting and patient population needs to be described. Completely basic information like sex and age are missing. The authors report that they use 15.8 million clinical notes from 30,494 patients, so about 500 notes/patient. Are these from the 7-day period, so more than 70 notes written per day? Or from a longer non-disclosed time-span?Materials and methods section:In this study patients are analyzed until date of diagnosis (test), it seems to be an analysis of when in the natural course one chooses to test? As it is formulated in the manuscript, "temporal patterns" first of all indicate, that the population converge towards day of test, so patients "progress towards same phenotype" and it is unclear how this relates to COVID-19 progression? What is the link between the analysis and the references to chloroquine/hydroxychloroquine?The presentation in Materials and methods, also has many unclear aspects. For example, what was the output from also curating disease and medication? It seems, that only symptoms are presented in the manuscript? How do symptom categories and phenotypes differ? The iteration for optimization of the model seems a little unclear and how were the 18,500 sentences in the test set selected? What was the indication for COVID-testing in these patients? Were they all hospitalized for different conditions? Were all 30,494 under suspect for COVID-19 or were some tested simply because they were hospitalized in the period of the pandemic (i.e. routine/screening test)? And what were the diagnoses of the COVIDneg cases? Where there notes on all patients from index -7 to 0 as mentioned above? What are the demographics of these patients? And were symptoms handled as chronic or temporary conditions? Why was altered or diminished sense of taste and smell (anosmia/dysgeusia) included in Results despite a classification performance of 64.4%? Not sure why there are two F scores. Why calculate F scores on sentences labeled as "not present" – how is recall not undefined in such a calculation? How were the sentences in step 2 and 3 chosen? Why is the sum of the true positive rate and the false positive rate not 100%? Confidence intervals would help the interpretation of the data. It would be great if the authors would provide number of tests or the significance level to help interpret the Benjamini-Hochberg correction.2) The authors state that the platform utilizes state-of-the-art transformer neural networks. But they used BERT  indeed. Bert is not the state-of-the-art transformer model. XLNet and RoBERTa are the state-of-the-art models. For name entity recognition of clinical note data, there have been some specific BERT-based variations or pre-trained models, such as BioBERT.The authors did not provide details of the implementation of BERT configuration, like how many layers, heads? How they train BERT, like how many epochs? what is the optimizer setting? Did they use a pre-train (on which corpus) BERT? etc. All these details need to be provided for reduplication. My suggestion is better to provide several examples regarding what the input looks like ,i.e., the unstructured clinical notes, and corresponding structured output, so that readers can understand how powerful the model is.3) The dataset split (90%/10%) is not usual. "70%/10%/20%" or "60%/10%/30%" are more common for fair evaluation in deep learning . Could the authors provide the reason why they split dataset in this way or provide a reference that applied this strategy?4) In the Results section, the first paragraph is about details of data collection, it would be more appropriate in part of the Materials and methods, and the final paragraph would be more appropriate in the Discussion. In Table 1, the proportion columns would be better to combine with the patients-count columns.5) In terms of the Discussion, it would be important to emphasize and discuss the main findings. "By contrasting EHR-derived phenotypes of COVID pos (n = 635) versus COVID neg  patients during the week preceding PCR testing, we identify anosmia/dysgeusia (37.4-fold), myalgia/arthralgia (2.6-fold), diarrhea (2.2-fold), fever/chills (2.1-fold), respiratory difficulty (1.9-fold), and cough (1.8-fold) as significantly amplified in COVID pos patients" This statement in the Abstract should be the key finding, but the authors did not emphasize the statistical method and make discussion properly.6) The pairwise analysis of phenotypes considered only the combination of 2 phenotypes. How to process the combination of multiple phenotypes?7) As the NLP-based entity recognition can bring errors, the following statistic analysis could be biased by these errors. The authors should emphasize this point. I would suggest the authors to use the manually curated data from the first 100 patients to perform the same analysis to see if it can generate the same results.eLife. Your article has again been reviewed by two peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Jos van der Meer as the Senior Editor. The reviewers have opted to remain anonymous.Thank you for resubmitting your article "Augmented Curation of Clinical Notes from a Massive EHR System Reveals Symptoms of Impending COVID-19 Diagnosis" for consideration by You have definitely made a number of changes that have improved the readability and consistency of the manuscript. Although the paper essentially only presents already known symptoms of the disease (e.g https://doi.org/10.1136/bmj.m1996), it is now clearer that the paper aims to characterize when and how long prior to PCR-diagnosis a symptom is overrepresented.However, one reviewer comments that there are still many inconsistencies in the manuscript that at best makes it difficult to read but also bring uncertainty about the validity of the results. Materials and methods section paragraphs are still hard to follow to the extent that the study would be hard to reproduce.Specific comments:1) Altered smell is as mentioned not a new finding, the authors also report this in relatively small numbers (6.3%) for covid19 patients, it seems unclear how much of an impact this would have clinically. The authors should comment on how this differs from other studies. In the context Tim Spector's work using app may also be relevant.2) I could not follow the number of symptoms (26 or 27?) and how were these 26 or 27 selected? Who selected these and what was their backgroundsWhy not search for overrepresented symptoms in general and include way more symptoms? Currently only suspected/known symptoms were included. "Examining the other 325 possible pairwise symptom combinations" which were the first 26 for example. In Table 1, why is respiratory failure and dysuria marked in a light grey color?3) Why is there automated entity recognition of "drugs"? Drug are not otherwise mentioned in the paper? There is still a part about scRNA-sequencing in the results, but no results about RNA-seq presented.4) Why was the method applied to the 35 million notes? When only notes from 7 days are analyzed in the study… I don't agree with deleting information about the number of notes, it does not solve the problem from the original submission. It would be nice for the reader to be able to evaluate the number of notes or words written about the included patients. Also, how many notes are from day 0? Are notes recorded at day 0 included or not, there seems to be inconsistency here throughout the manuscript. The BERT validation seems to be done on notes including day 0, whereas the actual proportion analyses seem to be excluding day 0 which is the day one must expect to have most notes concerning covid19.5) Many basic aspects are still unclear. How many data points could one patient contribute with? Could one patient tested multiple times contribute more than once? If a patient was tested negative and then positive, how was this handled?6) It a considerable limitation that there is no separation between chronic and acute symptoms. If the condition is chronic it wouldn't help at all in the diagnosis, only that this population for some reason is more prone to be covid19 positive.7) It still remains unclear why patients with obvious covid19 symptoms like "respiratory difficulty" and "fever" were not tested for sars-cov-2 earlier. It is very unlikely with patient delay from this point since the symptoms were recorded in the notes. These cases should be manually evaluated.The method was trained on "cardiovascular, pulmonary, and metabolic diseases and phenotypes", so not gastro-intestinal? How were symptoms like diarrhea validated?Still confusing to me that "negative sentiment classification" can have recall. I don't understand this, are there true positives in the negative classification? If so why? Essential revisions:1) The cohort itself is not well-defined and should be better described (see below).We now describe the cohorts in better detail in terms of demographics and comorbidities, as discussed below in more detail.2) Materials and methods section should be written more clearly (see below).Based on this feedback, we now elaborated our Materials and methods section, as discussed below in more detail.3) Remove RNA seq data and treatment figures and discussion about these topics. It is distractive and does not make the manuscript stronger.We accept this suggestion and have now removed figures and discussion about the RNAseq data and treatment.Cohort description:How many patients had an admission 7 days before getting a swab taken?This was computed and the number of patients who were admitted 7 days or more prior to the PCR testing date are shown below.COVIDpos : 11 COVIDneg : 196 How many individuals do the authors have a full week of information on prior to the PCR?This was computed and the number of patients who have at least one note in all 7 days prior to PCR testing are shown below.COVIDpos : 6 patients (0.3%)COVIDneg : 372 patients (0.5%)Why was there a delay in PCR testing of these patients?This inherent delay is caused by that fact that both the patient and physician must consider the symptoms enough to warrant PCR testing. More specifically, the patient must first consider the symptoms serious enough to visit the clinic and the physician must then determine that the symptoms present a possibility of COVID infection to order the test. We hope that the results presented in this manuscript, as well as more widely available testing, will help to shorten this delay in the future.Who are the 30,000 patients, and why were they admitted to a hospital? Or were they outpatients? The setting and patient population needs to be described.Expanding on the hospitalization counts 7 days or more prior to PCR testing (shown above), we have also computed the number of patients who were admitted between day -7 and day 0. The remainder of the patients were outpatients.COVIDpos : 135 patients (5.8%)COVIDneg : 5,981 patients (8.0%)Completely basic information like sex and age are missing.We have provided details regarding the age and sex of each cohort in the main text and present them below.COVIDposAge: Mean: 41.9 Std: 19.1Sex: M: 50.6% F: 49.4%COVIDnegAge: Mean: 50.7 Std: 21.4Sex: M: 43% F: 57%The authors report that they use 15.8 million clinical notes from 30,494 patients, so about 500 notes/patient. Are these from the 7-day period, so more than 70 notes written per day? Or from a longer non-disclosed time-span?We agree that the number of notes presented could be a point of confusion, as the numbers presented were the total number of notes for the entire medical history of these patients at the Mayo Clinic, not just day -7 to day 0. To prevent confusion for readers, we have removed the note counts from the manuscript and instead highlight the number of patients used for this analysis.Materials and methods section:In this study patients are analyzed until date of diagnosis (test), it seems to be an analysis of when in the natural course one chooses to test? As it is formulated in the manuscript, "temporal patterns" first of all indicate, that the population converge towards day of test, so patients "progress towards same phenotype" and it is unclear how this relates to COVID-19 progression?We agree with the reviewer that synchronization is integral to the analyses described in this manuscript. As described previously in this rebuttal, there is an inherent delay between when COVID-19 infection occurs and when the patient and physician deem testing necessary based on the symptoms present. This delay could be influenced by how well-informed the public is regarding COVID-19 signs and symptoms, the availability of PCR testing, and the standardization of hospital protocols for which patients get tested. Each of these factors would be limited at the beginning of a pandemic, but improve over time. As is the case with real-world data, there will be outliers, but increasing the size of the COVIDpos cohort 4-fold has increased the statistical significance of many of the signals presented herein. The fact that many of the COVIDpos patients progress toward the same set of symptoms demonstrates that synchronization using the PCR testing date is apt for such analysis. By understanding the temporal progression of symptoms prior to PCR testing, we aim to reduce the delay between infection and testing in the future.What is the link between the analysis and the references to chloroquine/hydroxychloroquine?We agree that these references were tangential and have now removed the references to chloroquine/hydroxychloroquine.The presentation in Materials and methods, also has many unclear aspects. For example, what was the output from also curating disease and medication? It seems, that only symptoms are presented in the manuscript?We agree that including medications in the Materials and methods was unrelated to the analyses presented and have now removed medications from the manuscript.How do symptom categories and phenotypes differ?We have clarified the distinction between symptoms and phenotypes throughout the manuscript. All analyses described were performed using COVID-19 symptoms. Phenotypes, i.e. other diseases/conditions, were only present in the 18,490 sentences used for training the initial SciBERT model, which was later supplemented with 3,188 sentences containing COVID-19 symptoms. Thus, “phenotype” is only used in the Materials and methods for the description of this model.The iteration for optimization of the model seems a little unclear and how were the 18,500 sentences in the test set selected?We have clarified how the model was optimized in the Materials and methods section. As we now describe, the 18,490 sentences selected for the test set were randomly selected from a pool of sentences containing 250 different cardiovascular, pulmonary, and metabolic diseases and phenotypes.What was the indication for COVID-testing in these patients? Were they all hospitalized for different conditions? Were all 30,494 under suspect for COVID-19 or were some tested simply because they were hospitalized in the period of the pandemic (i.e. routine/screening test)? And what were the diagnoses of the COVIDneg cases?We have included hospitalization information in the responses above and in the main text of the manuscript. Additionally, we have included the most frequent ICD10 diagnosis codes for both cohorts between day -7 and day 0 in the Supplementary information.Were there notes on all patients from index -7 to 0 as mentioned above?Yes, all patients in both cohorts had at least one note between day -7 and day 0.What are the demographics of these patients?We have provided details regarding the age and sex of each cohort in the main text and present them below.COVIDposAge: Mean: 41.9 Std: 19.1Sex: M: 50.6% F: 49.4%COVIDnegAge: Mean: 50.7 Std: 21.4Sex: M: 43% F: 57%And were symptoms handled as chronic or temporary conditions?No distinction was made between chronic and acute symptoms. We are currently developing models to extract the temporal context around a symptom, but here the model only classifies a symptom as Yes (confirmed), No (ruled out), Maybe (possibility of), or Other .Why was altered or diminished sense of taste and smell (anosmia/dysgeusia) included in Results despite a classification performance of 64.4%?Our initial analysis was run using a model trained on 18,490 sentences containing 250 different cardiovascular, pulmonary, and metabolic diseases and phenotypes. After directly examining sentences containing COVID-19 symptoms, we came to the conclusion that this training data did not adequately capture the context in which these symptoms were described. Thus, we manually labeled 3,188 sentences containing COVID-19 symptoms and used these in combination with the 18,490 sentences to train a new model. This new model was used for all of these analyses described herein, and resulted in an overall accuracy of 85.6% of altered or diminished sense of taste and smell.Not sure why there are two F scores. Why calculate F scores on sentences labeled as "not present" – how is recall not undefined in such a calculation?We apologize for the confusing wording used previously. We have revised the text and included tables for all models, which contain the F1-Scores for each classification.How were the sentences in step 2 and 3 chosen?We have revised the Materials and methods regarding model development to more accurately reflect the workflow. As we now describe, 18,490 sentences were selected randomly from a pool of sentences containing 250 different cardiovascular, pulmonary, and metabolic diseases and phenotypes. Similarly, 3,188 sentences were selected randomly from a pool of sentences containing COVID-19 related symptoms.Why is the sum of the true positive rate and the false positive rate not 100%?We believe the reviewer meant to ask why the true positive rate and false negative rate do not equal 100%, as they should. This was a labeling error, which has been fixed in Supplementary file 1H and now shows the true positive rate and false positive rate.Confidence intervals would help the interpretation of the data. It would be great if the authors would provide number of tests or the significance level to help interpret the Benjamini-Hochberg correction.We have added confidence intervals to both Table 1 and Supplementary file 1D. We have also clarified that the Benjamini-Hochberg correction was applied on 351 Fisher test p-values.2) The authors state that the platform utilizes state-of-the-art transformer neural networks. But they used BERT  indeed. Bert is not the state-of-the-art transformer model. XLNet and RoBERTa are the state-of-the-art models. For name entity recognition of clinical note data, there have been some specific BERT-based variations or pre-trained models, such as BioBERT.The authors did not provide details of the implementation of BERT configuration, like how many layers, heads? How they train BERT, like how many epochs? what is the optimizer setting? Did they use a pre-train (on which corpus) BERT? etc. All these details need to be provided for reduplication. My suggestion is better to provide several examples regarding what the input looks like ,i.e., the unstructured clinical notes, and corresponding structured output, so that readers can understand how powerful the model is.We have addressed these concerns in the Materials and methods and Figure 1—figure supplement 1-2, showing the SciBERT architecture, training configuration, and example sentences from the unstructured text of clinical notes with their classifications.3) The dataset split (90%/10%) is not usual. "70%/10%/20%" or "60%/10%/30%" are more common for fair evaluation in deep learning . Could the authors provide the reason why they split dataset in this way or provide a reference that applied this strategy?At the time of training, since we had limited tagged examples, we wanted to maximize the size of the training and test sets. Thus, when choosing our models, we used the test set accuracy to determine the optimal number of epochs to stop at.4) In the Results section, the first paragraph is about details of data collection, it would be more appropriate in part of the Materials and methods, and the final paragraph would be more appropriate in the Discussion. In Table 1, the proportion columns would be better to combine with the patients-count columns.We have revised the first paragraph of the Results section to focus on characterization of the COVIDpos and COVIDneg cohorts. We have also revised the final paragraph to better summarize the Results prior to the Discussion. Finally, we have combined the columns for the patient counts and proportions as suggested by the reviewer.5) In terms of the Discussion, it would be important to emphasize and discuss the main findings. "By contrasting EHR-derived phenotypes of COVID pos (n = 635) versus COVID neg  patients during the week preceding PCR testing, we identify anosmia/dysgeusia (37.4-fold), myalgia/arthralgia (2.6-fold), diarrhea (2.2-fold), fever/chills (2.1-fold), respiratory difficulty (1.9-fold), and cough (1.8-fold) as significantly amplified in COVID pos patients" This statement in the Abstract should be the key finding, but the authors did not emphasize the statistical method and make discussion properly.We have revised the Discussion to focus on the implications of identifying an early symptom signature for COVID-19 prior to testing.6) The pairwise analysis of phenotypes considered only the combination of 2 phenotypes. How to process the combination of multiple phenotypes?We have also considered combinations of 3 and 4 symptoms. However, as we increase the number of symptoms, the results become more and more under-powered. Thus, we have chosen to only include those for combinations of 2 symptoms.7) As the NLP-based entity recognition can bring errors, the following statistic analysis could be biased by these errors. The authors should emphasize this point. I would suggest the authors to use the manually curated data from the first 100 patients to perform the same analysis to see if it can generate the same results.We have now emphasized the potential sources of error in the Materials and methods. Performance on the manually curated data is represented in Supplementary file 1H.You have definitely made a number of changes that have improved the readability and consistency of the manuscript. Although the paper essentially only presents already known symptoms of the disease (e.g https://doi.org/10.1136/bmj.m1996), it is now clearer that the paper aims to characterize when and how long prior to PCR-diagnosis a symptom is overrepresented.However, one reviewer comments that there are still many inconsistencies in the manuscript that at best makes it difficult to read but also bring uncertainty about the validity of the results. Materials and methods section paragraphs are still hard to follow to the extent that the study would be hard to reproduce.We thank the reviewers and the editors for their feedback and address them below.Indeed, as the editors have suggested, our study characterizes when and how long prior to PCR-diagnosis a symptom is overrepresented. We would like to highlight that there are other key differentiators too from other other studies such as the BMJ paper by Argenziano et al. This BMJ paper is a manual curation effort to comb through 1000 COVID-19 patients EHRs. Please see this statement from the paper(https://doi.org/10.1136/bmj.m1996): “An abstraction team of 30 trained medical students from the Columbia University Vagelos College of Physicians and Surgeons who were supervised by multiple clinicians and informaticians manually abstracted data from electronic health records in chronological order by test date.”We would like to point out that our augmented curation technology premised on neural networks is highly scalable and efficient, whereas the manual curation approach used by the above study, continues to be the current state-of-art despite the associated disadvantages of scalability. Clearly, there is a need to validate any augmented curation methods, which we have now conducted and included in the context of COVID-19 phenotypes.The BMJ paper and others like it typically look at severe disease or critically ill patients from the ICU . Our manuscript, on the other hand, looks at the whole spectrum of COVID-19 patients, including from outpatient visits.Finally, it is noteworthy that our manuscript contributed to flagging anosmia  as an early indication of SARS-CoV-2 infection, despite a majority of physicians at the time of this paper’s submission not explicitly asking patients regarding a newfound sense of loss of smell. It may further be noted that our preprint on medRxiv and the subsequent conversations that the Mayo Clinic COVID-19 task force led with the state and federal governments contributed towards the CDC altering its guidelines to explicitly add altered or loss of smell or taste to its new guidelines for SARS-CoV-2 management:(https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/symptoms.html).Specific comments:1) Altered smell is as mentioned not a new finding, the authors also report this in relatively small numbers (6.3%) for covid19 patients, it seems unclear how much of an impact this would have clinically. The authors should comment on how this differs from other studies. In the context Tim Spector's work using app may also be relevant.We would be happy to address this comment. In the paper by Tim Spector and colleagues focused on self-reporting of symptoms, it is unclear how many days prior to PCR-based COVID-19 diagnosis the reporting of symptoms is available. It is possible that many of the symptoms were noted after diagnosis too.While the study by Tim Spector and colleagues indeed does point to 65.03% of the SARS-CoV-2-positive patients experiencing a loss of smell or taste, it needs to be taken into consideration that 21.71% of the SARS-CoV-2-negative patients also reported this phenotype (https://www.nature.com/articles/s41591-020-0916-2). Hence, this only represents about a three-fold enrichment of anosmia/dysgeusia in COVIDpos over COVIDneg patients. However, our study identifies a much higher enrichment (27-fold) of anosmia/dysgeusia in COVIDpos over COVIDneg patients.Finally, it also needs to be considered that the app explicitly asks the question of whether an altered sense of smell or taste was encountered by each participant, which is different from the real-world clinical setting, particularly during the early phases of the pandemic when anosmia/dysgeusia was not part of the CDC guidelines.2) I could not follow the number of symptoms (26 or 27?) and how were these 26 or 27 selected? Who selected these and what was their backgrounds Why not search for overrepresented symptoms in general and include way more symptoms? Currently only suspected/known symptoms were included. "Examining the other 325 possible pairwise symptom combinations" which were the first 26 for example. In Table 1, why is respiratory failure and dysuria marked in a light grey color?We thank the reviewer for pointing out the mistyped number of symptoms, which has been fixed in the revised version of the manuscript.eLife in ~6 months, at which point the patient counts would have also significantly increased, thus amplifying the overall statistical confidence of the day-by-day enriched signals preceding COVID-19 diagnosis.During the week preceding PCR testing, 100 COVIDpos patients’ clinical notes were examined to select the over-represented symptoms. The list of symptoms considered was examined and approved by the COVID-19 taskforce at Mayo Clinic, and physicians from multiple specialties that are actively undergoing patient care for COVID19 patients across a broad spectrum of phenotypes. Conducting further holistic phenotypic scan will be the topic of follow-up studies upon publication of this manuscript – potentially as a Research Advance to The light gray shading for the dysuria and respiratory failure rows in Table 1 was an artifact of a previous draft and has been removed in the revised manuscript. We thank the reviewer again for helping us clarify this.3) Why is there automated entity recognition of "drugs"? Drug are not otherwise mentioned in the paper? There is still a part about scRNA-sequencing in the results, but no results about RNA-seq presented.The reference to “scRNAseq” and “drugs” were in the context of broader discussion and introduction points. Since they appear to be distracting, we have now removed them in the revised manuscript. We thank the reviewer for pointing out.4) Why was the method applied to the 35 million notes? When only notes from 7 days are analyzed in the study… I don't agree with deleting information about the number of notes, it does not solve the problem from the original submission. It would be nice for the reader to be able to evaluate the number of notes or words written about the included patients. Also, how many notes are from day 0? Are notes recorded at day 0 included or not, there seems to be inconsistency here throughout the manuscript. The BERT validation seems to be done on notes including day 0, whereas the actual proportion analyses seem to be excluding day 0 which is the day one must expect to have most notes concerning covid19.The model was indeed applied to over 35 million notes at the onset of this study, which represents the entire medical record of the patients considered for this analysis. However, as the reviewer correctly points out, only 7 days prior to PCR testing date was ultimately decided to be of immediate interest from a SARS-CoV-2 diagnostic standpoint. In order to make this clear, the number of patients and associated notes on each day of the week preceding SARS-CoV-2 PCR testing has now been included explicitly in Supplementary file 1E.Day 0 was mentioned in the table for reference and is not part of any analysis barring the model training. Obviously, day 0 presents no predictive insight as to the outcome of the SARS-CoV-2 PCR test which is also conducted on that same day, and hence is not included for any further interpretation.5) Many basic aspects are still unclear. How many data points could one patient contribute with? Could one patient tested multiple times contribute more than once? If a patient was tested negative and then positive, how was this handled?Each patient is counted only once. Once they have a positive SARS-COV-2 PCR test, they are considered COVIDpos. If a patient were to test negative and then positive subsequently, that day of positive PCR testing is considered day 0 for that patient. We now mention this clearly in the Materials and methods section of the current manuscript.Each patient is counted only once. Once they have a positive SARS-COV-2 PCR test, they are considered COVIDpos. If a patient were to test negative and then positive subsequently, that day of positive PCR testing is considered day 0 for that patient.”“6) It a considerable limitation that there is no separation between chronic and acute symptoms. If the condition is chronic it wouldn't help at all in the diagnosis, only that this population for some reason is more prone to be covid19 positive.Based on currently available literature and current CDC guidelines , there are indeed multiple specific underlying risk factors associated with COVID-19 diagnosis . However, to our knowledge, there are no known associations to chronic versions of symptoms described in our manuscript . Furthermore, the day-to-day variability of the phenotypes also clearly argue in factor of the considered phenotypes not being the result of a generic chronic underlying condition but rather that of SARS-CoV-2 infection onset and early COVID-19 progression. Finally, at the level of the population, the signals that we present are fairly robust given the large cohorts of COVIDpos and COVIDneg patients considered.eLife in ~6 months, at which point the patient counts would have also significantly increased and provide an extended time-line to examine chronic aspects associated with COVID-19 as well.We do intend to continue monitoring and reporting the updated statistics, as noted above, via potentially a Research Advance to 7) It still remains unclear why patients with obvious covid19 symptoms like "respiratory difficulty" and "fever" were not tested for sars-cov-2 earlier. It is very unlikely with patient delay from this point since the symptoms were recorded in the notes. These cases should be manually evaluated.In this “real world evidence” based study, there are multiple factors that could contribute to the time before the test. The early stages of the pandemic did not necessarily involve patients getting PCR tests at the very first semblance of many of these common phenotypes, and furthermore there was the limited availability of the PCR tests themselves during such early days of the pandemic. During the further stages of the pandemic, there has been a large push to self-isolate upon symptom manifestation, which might have delayed patients from coming into the hospital to get tested, despite discussions either over tele-medicine or in-person visits with physicians that didn’t culminate in a PCR test at the first juncture.Furthermore, it is quite possible that negative PCR test results were indeed produced in some patients during the week preceding the eventual first positive PCR test result. It is also important to note that, many of these patients in whom PCR may come our negative during early days of the infection, may well present with mild symptomatology such as anosmia. Indeed, there is indeed emerging evidence in the scientific literature that does show that PCR testing results being positive may be delayed depending on the nature and extent of the nasopharyngeal swab conducted, the ability to gather sufficient replication-competent virions in such swab specimen, as well as other factors remaining to be understood that constitute some of the caveats of false negative PCR testing. Regardless, it is a fact that PCR testing continues to be the gold standard for positive SARS-CoV-2 diagnosis till date and is further unlikely to be substituted anytime in the near future.In the manuscript, we have the following text that clarifies these points:pos individual, whereas now PCR testing is more widespread. In each case, the delay between symptom onset and PCR testing would be expected to change over time for a given patient population.""There are a few caveats that must be considered when relying solely on EHR inference to track symptoms preceding the PCR testing date. In addition to concerns regarding testing accuracy, there is an inherent delay in PCR testing, which arises because both the patient and physician must decide the symptoms warrant PCR testing. More specifically, to be tested, the patient must first consider the symptoms serious enough to visit the clinic and then the physician must determine the symptoms present a possibility of COVID infection. The length of this delay could also be influenced by how well-informed the public is of COVID-19 signs and symptoms, the availability of PCR testing, and the hospital protocols used to determine which patients get tested. Each of these factors would be absent or limited at the beginning of a pandemic but would increase or improve over time. However, this makes synchronization across patients difficult because the delay between symptom onset and PCR testing changes over time. For example, patients infected early in the pandemic would be less inclined to visit the clinic with mild symptoms, while those infected later have more information and more cause to get tested earlier. Similarly, in the early stages of the COVID-19 pandemic when PCR testing was limited, physicians were forced to reserve tests for more severe cases or for those who were in direct contact with a COVIDThe method was trained on "cardiovascular, pulmonary, and metabolic diseases and phenotypes", so not gastro-intestinal? How were symptoms like diarrhea validated?The original diagnosis model was trained using cardiovascular, pulmonary, and metabolic diseases/phenotypes (Supplementary file 1G). Because this model was not trained on COVID-related symptoms, e.g. gastrointestinal symptoms, additional sentences were manually labeled containing COVID-19 symptoms, such as diarrhea. To make a new, more generalizable model, these sentences were included in the training and validation of a new model composed of sentences from the former model and these additional COVID-related symptoms (Supplementary file 1H). Validation of this model was performed on 4001 sentences containing COVID-related symptoms (Supplementary file 1I).In fact, a custom model for each organ/disease is not the ideal way to develop the neural network based augmented curation system. The authors are of the belief that a model that is trained on a broader range of phenotypes and indications would generalize better to many diseases and phenotypes that the model hasn’t actually encountered in the past. These views are being explored through a detailed follow-up study comparing models trained on specific corpora of knowledge as opposed to a “core corpus” that includes multiple diverse indications not biased/defined by current siloes.Still confusing to me that "negative sentiment classification" can have recall. I don't understand this, are there true positives in the negative classification? If so why?Please consider the following example of a hypothetical sentence from a provider’s clinical note, “The patient does not have diarrhea today”. This sentence will be classified by our model as the result: “NO”, for the symptom: “diarrhea”. Such a negative sentiment would be regarded by our model as a “true positive” for the lack of diarrhea. Thus, a “negative sentiment classification” can have recall computed from these true positives.


Radiotherapy is often used as a neo-adjuvant treatment in locally advanced rectal cancer. While treatment generally induces an improvement in the outcome, some patients show resistance to treatment for reasons that still have to be elucidated. In this work, we report an in vitro and in vivo model based on patient-derived cancer stem cells. This model is able to predict individual responses to radiotherapy. The results indicate that cells found to be radiation-sensitive in vitro generated radiation-sensitive tumor xenografts upon subcutaneous implantation. Analogously, cancer stem cells (CSCs) that did not respond to in vitro radiation treatment generated radiation-resistant tumor xenografts. Moreover, radioresistant CSCs were generally isolated via biopsies of patients with poor responses to neo-adjuvant radiotherapy. This suggests that a cell-based in vitro test may itself be sufficient to predict outcomes in donor patients.Neo-adjuvant radiotherapy is frequently employed in the therapeutic management of locally advanced rectal cancer (LARC). Radiotherapy can both reduce local recurrence and improve the success of surgical procedures by reducing tumor mass size. However, some patients show a poor response to treatment, which results in primary resistance or relapse after apparent curative surgery. In this work, we report in vitro and in vivo models based on patient-derived cancer stem cells (CSCs); these models are able to predict individual responses to radiotherapy in LARC. CSCs isolated from colorectal cancer biopsies were subjected to in vitro irradiation with the same clinical protocol used for LARC patients. Animal models, generated by CSC xenotransplantation, were also obtained and treated with the same radiotherapy protocol. The results indicate that CSCs isolated from rectal cancer needle biopsies possess an intrinsic grade of sensitivity to treatment, which is also maintained in the animal model. Notably, the specific CSCs’ in vitro and in vivo sensitivity values correspond to patients’ responses to radiotherapy. This evidence suggests that an in vitro radiotherapy response predictivity assay could support clinical decisions for the management of LARC patients, thus avoiding radiation toxicity to resistant patients and reducing the treatment costs. Colorectal cancer (CRC) is the third most common cancer (10.2% of tumor diagnoses) in terms of incidence and the second most common in terms of mortality, for both sexes combined, in western countries , obesityCancers of the rectum account for 30% of all diagnosed CRCs. Although rectal cancers are histologically similar to colon cancers, considerable biological and clinical evidence indicates that rectal cancer must be considered a distinct entity ,9. TotalDespite improvements in treatment, the prognosis of CRC remains unfavorable. Although several important side effects can occur, negatively affecting patient quality of life , radiothCancer Stem Cells (CSCs) are a small subset of cells that possess self-renewal potential and persistent tumorigenic capacity. Indeed, it has been widely demonstrated that CSCs are able to initiate and sustain tumor growth and, moreover, to recapitulate cancer cell heterogeneity. The existence of CSCs was first demonstrated in acute myeloid leukemia  and was Conventional anti-cancer therapies target only differentiated and fast proliferating cells, while sparing CSCs that are quiescent and undifferentiated. For this reason, CSCs are often drug-resistant, leading to tumor recurrence and metastasis . ConsequIn this study, we developed an in vitro and in vivo model of radiotherapy, based on patient-derived CSCs, for the prediction of treatment efficacy to support clinical decisions. Our data indicate a concordance between in vitro and in vivo CSC sensitivity to radiotherapy, suggesting that the CSC model may be sufficient to assess the suitability of therapeutic regimens, thus providing a feasible translational approach for the prediction of efficacy. The proposed approach may be useful for the timely identification of resistant patients who can be spared from side effects of ineffective radiotherapy, thereby supporting more appropriate clinical decisions for therapeutic options.CD44 and CD133 molecules are two common surface markers used for the identification and isolation of colorectal CSCs . Flow cyIncreased aldehyde dehydrogenase (ALDH) activity has been described in cancer stem cells from different carcinomas ,39,40,41A pre-requisite of putative CSCs is their capacity to develop tumors that have the same phenotype as human parental cancer when transplanted in recipient mice.CSCs derived from CRC biopsies were injected subcutaneously into the flanks of thymic immunocompromised mice. This procedure resulted in 100% tumor formation efficiency within a few weeks (2–4 weeks) after injection for all four cell lines. The tumor mass explanted from the animal was processed for histologic and immunohistochemical analysis to verify that its phenotypical features reproduced those of the original human tumor.As shown in The establishment of CRC stem cell cultures may facilitate the direct evaluation of radiation cytotoxic activity on the putative cells responsible for tumor growth and spread, which represent optimal cellular targets for successful therapy. As a fractioned 25 Gy dose administered daily (5 Gy/Day) is the most commonly updated protocol used before proceeding with curative surgery for rectal cancer , the samp values, respectively, of 0.113 and 0.233). Accordingly, the apoptosis assay did not show any significant difference in the percentage of cell vitality between the treated and untreated (NT) samples at any dose rate or at any time point considered . Finally, the single cell cloning assay exhibited only a slight reduction for both lines  in their ability to give rise to new cell clusters 7 days after treatment (data not shown).To evaluate the effects of the radiotherapy treatment in vitro, different assays, including proliferation, apoptosis, and single cell cloning assays, were performed on irradiated cells at different time points . Since radiation can be administered using different dose rates during the daily sessions, the effects of the dose rate in terms of cell growth arrest and apoptosis induction were investigated. In vitro cultures were treated using a 5 Gy daily dose at 600, 1400, and 2400 MU/min, and the MTS and Annexin V results were compared. No statistically significant difference was observed among the different dose rate treatments in all the lines assessed A–H. On tp values both below 0.001 with no significant differences between dose rates). The Annexin V assay confirmed these data, already showing a reduction in cell vitality of about 80% at 14 days after treatment . Although all assays were performed on irradiated cells at five different time points , data are reported only for the most informative time points . In accordance with the radiobiology of the treatment, the cells start to die only a few days after the treatment because they were unable to repair DNA damage and accumulated mutations. At the earlier time points, the induction of cell death was not noticeable, even when already triggered.On the contrary, the other two CSC lines (Lines 2 and 3) were shown to be highly sensitive to radiotherapic treatment. The proliferation assay revealed a significant reduction in cell proliferation for both lines at all dose rates tested  compared to the NT samples. In particular, after 14 days, almost all the cells were shown to have lost their proliferative potential in all treated samples .These in vitro results demonstrated that the CSCs derived from different patients had diverse sensitivity levels to radiation treatment, although no differences were found among the different dose–rate protocols tested.Graphical representations of the proliferation  and apoptosis assay  24 h and 14 days (14D) after treatment with a 5 Gy dose radiation supplied for 5 consecutive days in combination with different dose rates  in comparison with the non-treated sample (NT) for CSC Line 1 , Line 2 , Line 3 , and Line 4 .The individual responses of CRC patients to neoadjuvant radiation may vary in relation to the unique characteristics of each individual tumor, differently affecting the response to radiotherapy in terms of mass reduction and risk of recurrence.In vitro experiments already demonstrated that the cells isolated from different colorectal cancer patients present different sensitivity levels to radiotherapeutic treatments. To evaluate the in vivo responses to radiotherapy, animal models were exposed to the same radiotherapy protocols used in the in vitro experiments.3, the animals were subjected to a TC scan to better estimate the tumor dimensions and exactly define the zone to expose to radiation. Mice were treated with 5 Gy radiation for 5 consecutive days during the same hour and under the same experimental conditions. Tumor growth was evaluated by external caliper measurements every 4 days for 4 weeks. At the end of this period, the mice were subjected to a final TC scan to exactly evaluate their tumor mass reduction in comparison with the initial TC scan. The in vivo results were fully consistent with those obtained in vitro. In particular, the radiation treatment was ineffective for Line 1- and 4-derived tumor grafts, as shown in p values below 0.001). The same trend was observed in the CT scan images at the beginning (0D) and end (30D) of the treatment  composed of three animals and a treated group containing eight animals. Once the tumors reached a diameter of about 100–150 mmreatment D,G.Ex-vivo histological analysis confirmed tumor regression in the treated explants derived from Lines 2 and 3. In these tissues, no viable tumor cells were observed. On the other hand, explanted tissues derived from treated tumors derived from Lines 1 and 4 presented viable tumor cells.Graphical representations of the in vivo tumor growth (%) measured at three different time points  after treatment with a 5 Gy dose of radiation supplied over 5 consecutive days in combination with a dose rate of 2400 MU/min compared to the non-treated sample (NT), for CSC Line 1 (A), Line 2 (B), Line 3 (E), and Line 4 (F).  Representative CT scan images taken on the day of the last irradiation (D0) and 30 days after treatment for Lines 1, 2, and 3. Line 4 produced fast growing refractory tumors with a consequent risk of ulceration. All mice implanted with Line 4 had to be sacrificed on day 20 because they reached the tumor mass limit. For this reason, CT scan images are not available. Representative images of ex vivo measures for Line 4 (H).In the clinic, all four CRC patients whose CSCs were isolated and used for the in vitro experiments were subjected to neo-adjuvant radiotherapy. In this way, we compared the in vitro responses with the clinical outcomes.Our data show that the patients achieved a good clinical response when their isolated CSCs were sensitive to in vitro treatments, as shown in The European Society of Medical Oncology (ESMO) guidelines  recommenAs the evaluation of the complete pathological response alone is not sufficient to identify the cases that, despite an apparent initial response to the treatment, may be affected by recurrence events, thus compromising the clinical outcome, the identification of more effective strategies for the prediction of treatment efficacy is desirable. On the other hand, radiotherapy is not free from undesirable effects, as it increases the risk of serious anorectal and genitourinary complications, which may negatively impact a patient’s quality of life. Numerous radiotherapy-related side-effects have been described, including surgical site infection, anastomotic leakages, wound dehiscence, and functional disorders, such as low anterior resection syndrome and genitourinary dysfunction. The severity of these effects is affected by individual susceptibility, the dose of administered radiation, and the accuracy with which the radiation is precisely delivered to the target area. For the above-mentioned reasons, it is very important to accurately identify both the patients who are likely to benefit from neoadjuvant radiotherapy and the most suitable therapy (dose or protocol) for each individual to optimize patient management and support clinical decisions to minimize morbidity and improve clinical outcomes.Our data suggest that heterogeneity exists in the individual sensitivity levels of patient-derived CSCs to different doses of radiation. We demonstrated that CSCs have specific individual dose thresholds, above which they are triggered to stop proliferation, activate the apoptotic pathway, and lose the stem-cell-like phenotype. The concordance between the treatment effects under both in vitro and in vivo irradiation indicates that this individual sensitivity is not affected by experimental conditions and seems to be related only to the intrinsic cellular phenotype. While the underlying cellular and molecular mechanisms responsible for these different responses are yet to be elucidated, the in vitro model itself may be proposed as an investigational tool to predict the therapeutic response of the donor patient. The experimental procedure evaluated in this work was developed to potentially fit a clinical scenario in an effort to define a protocol for personalized medicine in radiotherapy. On the one hand, we demonstrated that CSCs can be isolated from scarce materials like the biopsy tissue collected from patients before any intervention. To evaluate the feasibility of a translational protocol able to predict responses to neo-adjuvant therapy, the only material available for CSC isolation is provided by a biopsy. According to our experience, the isolation of CSCs from solid tumors  may be aOn the other hand, to be effective, a predictive protocol must provide the clinician with relevant information quickly enough to treat the patient. While CSCs are isolated, in vitro expansion and in vitro treatment can be completed in a relatively short time, while the development of an in vivo xenotransplantation model is much more time demanding. To develop patient-derived mice models, 2.5 × 106 cells were obtained (5 × 105 cell for five mice). Then, after the subcutaneous cell implantation, the CSC-derived tumor masses required from 3 to 9 weeks to reach a sufficient volume for the subsequent experimental procedure. After the mice were treated, they were observed for about one additional month to evaluate the final effect of the treatment. In all the evaluated cases, we observed perfect concordance between the results of the matched in vivo and in vitro treatments. For each of the patient-derived CSC lines, both the CSC culture and the patient-derived mouse model were shown to possesses the same dose sensitivity threshold. While this observation is preliminary, this concordance suggests that an in vivo procedure may not be required to identify the lowest effective dose of radiation. Finally, we demonstrated that every time a patient-derived CSC culture exhibited a certain level of radio resistance, the corresponding donor patient featured a poor reduction in tumor mass following neo-adjuvant radiotherapy, while the tumor biopsies of patients who showed an optimal response to the radiotherapy produced CSC cultures with a low radiosensitivity threshold.Hence, the in vitro data could help adapt the neoadjuvant radio treatment approach to improve clinical outcomes and reduce toxicity for patients with CRC.Furthermore, in most CRC cases, in vitro radiotherapy response tests can be quickly performed, demonstrating the ability of this model to produce treatment suggestions in a clinically acceptable time frame. In conclusion, the assessment of the in vitro based model predicted CRC patient responses to radiotherapy treatment and could be developed as a powerful diagnostic tool for CRC treatment.A complete study diagram is included as Newly diagnosed T3 or T4 colorectal cancer patients who were eligible for neo-adjuvant radiotherapy and subsequent surgical resection were enrolled at the Mediterranean Institute of Oncology . T2 patients eligible for neo adjuvant therapy, according to ESMO indication  were also enrolled.Rectal cancer biopsy samples were obtained before the administration of neo-adjuvant radiotherapy. Eight weeks after neo-adjuvant radiotherapy, the individual responses to therapy were evaluated via CT-scan and Magnetic Resonance, and the patients were classified.The study is compliant with all relevant ethical regulations involving human participants and was approved by the Institutional Review Board protocol . Informed consent was obtained from all subjects.2. Every 2/3 days, half of the culture medium was refreshed. In these stringent culture conditions, immature cells grew slowly and formed non-adherent clusters, called tumor spheres, while non-malignant cells or differentiated cells died. Tumor spheres became evident after a variable length of time, ranging from 5 to 7 days to 3 weeks. Regular culture splitting (1:2) was usually needed after 3–6 weeks from isolation. Spheroids were weekly subjected to mechanical or enzymatic dissociation via incubation for 10 min at RT with the Accutase enzyme (GIBCO).Seven fresh human colorectal cancer biopsies were obtained in accordance with the standards of the ethics committee on human experimentation of the IOM. Biopsy samples were collected before the administration of neo-adjuvant radiotherapy. CSC isolation from the biopsies was performed as previously described ,47. BrieFrom a total of 7 surgical samples, 4 CSC lines were established and validated to determine their ability to generate tumors in mice, and 3 samples did not generate growing in vitro cultures .The expression of stem cell markers was evaluated by flow cytometry analysis using FACSAria II (Becton Dickinson (BD), Franklin Lakes, NJ, USA). Single cells dissociated from spheroids were incubated with the appropriate dilution of a specific antibody: anti CD133-PE  and anti CD44-FITC . Unstained cells were used as the negative control.Cell stemness was also evaluated by using an ADEFLUOR KIT  according to the manufacturer’s instructions. The ALDEFLUOR™ reagent system is a non-immunological method to identify stem/progenitor cells by their aldehyde dehydrogenase (ALDH) activity.CSCs were seeded into 35 × 10 mm dishes  and reached approximately 80% confluency at the time of irradiation.A system for the in vitro irradiation of CSCs and custom-designed irradiation geometry were developed. This system utilizes the same equipment used for patient treatments. To simulate the flow of radiation beams through human tissues before reaching the target tumor area, a single dish containing cultured CSCs was inserted into a custom-built phantom made of plexiglass, a material similar to water, which is the main component of human tissue. The dish was housed within a niche created inside the phantom to place the cells at the radiation isocenter.2 defined at the machine isocenter located at the center of the niche containing the plate.The phantom containing the colorectal CSCs was irradiated through a Varian Novalis-TrueBeam STx linear accelerator, which is able to perform stereotactic treatments with very high precision. This radiotherapy equipment uses the high dose rate Flattening Filter Free (FFF) technique and a High Definition Multilamellar Collimator (MLC); with a minimum leaf size at the isocenter of 2.5 mm, this device is specifically designed to treat small lesions. A fractionated dose of 25 Gy (5 daily fractions of 5 Gy) was administered  in combi2 for a further 24 h, 48 h, 72 h, 7 d, and 14 d, respectively, and then analyzed for cell viability, apoptosis, and the ability to give rise to new clones.After irradiation, cells were incubated at 37 °C in a humidified atmosphere of 5% CO® Aqueous One Solution Cell Proliferation Assay Kit  according to the manufacturer’s instructions. The fluorescence signal was detected with Synergy HT  at 24 h, 48 h, 72 h, 7 d, and 14 d.The cell viability assay was performed using a CellTiter965 cells for each sample were stained at 24 h, 48 h, 72 h, 7 d, and 14 d, with Annexin V fluorescein isothiocyanate (FITC) at a final concentration of 0.375 μg/mL , according to the manufacturer’s instructions. To distinguish between early apoptotic cells with intact cellular membranes and necrotic or late-apoptotic cells, 1 μg of propidium iodide (PI) was added to each sample. Cytometric analysis was performed with a FACS-Aria II flow cytometer . For each measurement, 1 × 104 cells were counted, and the results were analyzed. Three replicates were analyzed for each CSC line in each condition assessed.Annexin V staining of the phosphatidylserine (PS) in the outer surface of the cellular membrane is a widely used assay for studying cellular apoptosis, as an increase in PS staining is directly connected with early apoptosis. Here, 1 × 10CSCs were dissociated with Accutase  and then resuspended in a fresh medium to generate a single-cell suspension with a density of 10 cells/mL. Then, 200 μL of the single-cell suspension was dispensed into each well of a 96-well non-treated plate. The day after plating, only wells that contained 1 viable cell were selected, excluding wells with no cells or more than one cell. Single-cell cultures were maintained in a medium and were checked after 14 days to ascertain their clonogenic potential.scid Il2rgtm1Wjl/SzJ (NSG) mice .All animal procedures were performed according to the Italian national animal experimentation guidelines (D.L.116/92) upon approval from the experimental protocol by the Italian Ministry of Health’s Animal Experimentation Committee. This study used 4-to-6-week-old female NOD. Cg-Prkdc5 cells were resuspended in 100 μL of a 1:1 growth medium/Matrigel  solution, and the cell suspension was injected subcutaneously into the flank of the animal. For each CRC cell line, 5 replicates of xenotransplants were performed. For all 4 lines, a tumor mass was detectable within 3–5 weeks in at least 3 out of 5 mice. As soon as the tumor mass reached a diameter of 10 mm, xenografts were explanted, and one-half of the mass was formalin-fixed, paraffin-embedded, and processed for histology to evaluate the tumor phenotype in comparison with the parental human tumor. The other portion of tumors was dissociated into single cells that were seeded in a tumor sphere medium and expanded to be assayed once again for stemness markers .The main feature of stem cells is their ability, once implanted in a recipient mouse, to reproduce a tumor with the same phenotype of the original one. To this end, 5 × 10Tumors were fixed with 10% formalin and paraffin-embedded for histological analysis. Three-micrometer-thick sections were cut with a microtome and automatically stained with hematoxylin-eosin .The presence of colon adenocarcinoma was also evaluated via immunohistochemical analysis. Three-micrometer-thick sections were cut from the FFPE blocks. Anti-human CK20 (clone SP33) and CDX2 (clone EPR2764Y) rabbit monoclonal primary antibodies  were used for the analysis. Slides were incubated using the BenchMark ULTRA platform, and an OptiView DAB IHC Detection Kit  was used to detect protein expression. Tissues were counterstained with Hematoxylin II  for 4 min. The control of immunostaining specificity was performed by omitting the primary antibody.5 cells were injected subcutaneously into the flank of the animal [2 × D, where d and D represent shorter and longer tumor measurements, respectively. When tumors reached a dimension of 100–150 mm3, the mice were randomly assigned to the control (3 mice/group) and treatment groups (8 mice/group). On the day of the radio treatment, mice belonging to the treatment group were moved, one by one, from the cage into a plexiglass box, where an anesthetic gas containing Vetflurane  was insufflated. All the following procedures, including image acquisition, contouring, elaboration of the treatment plan, and the same radio-treatment, were performed with the mice inserted and immobilized inside the plexiglass cage. Here, mice were first subjected to a computed tomography (CT) scan, and the CT images were the sent to the treatment planning system (TPS) dedicated to stereotactic radiotherapy treatments. Here, contouring of the volumes of interest was performed, including the target volume, spinal cord, heart, lungs, and bowels. At this point, the treatment plan was elaborated. This plan consisted of two non-coplanar dynamic conformal arcs with the optimized opening of MLC leaves based on dose constraints established during planning. Plan evaluation was performed carefully while observing the dose distributions on each CT image and the dose–volume histograms to assess the radiation dose that reached the target and the neighboring organs. To set-up the verification, we used the image-guided radiotherapy system “ExacTrac X-Ray 6D”, through using which it was possible to carry out pre-positioning through the infrared system and positioning using the X-Ray imaging system. Irradiation of the mice was performed by delivering 5 Gy for 5 consecutive days, for a total dose of 25 Gy at a maximum dose rate of 2400 MU/min using the 10 MV FFF photon beam produced by the Varian Novalis-TrueBeam STx linear accelerator.4- to 6-week-old female NSG mice  were randomly assigned into 4 groups, one group for each CSC line. Each group was formed from 11 mice. For each line, the CRC-SCs were resuspended in 100 μL of a 1:1 growth medium/Matrigel, and 5 × 10e animal . Tumor gControl animals were inoculated but not treated. Thirty days after treatment, a CT scan was performed to verify the tumor dimensions.This study was performed in accordance with the ethical statement established by Italian law . This study was authorized by the Italian Ministry of Health with the code 0D183.2.The patients enrolled in this observational study, whose isolated cells were used for in vitro and in vivo studies, were treated according to the actual therapeutic protocol. At the end of the treatment, the individual responses to therapy were evaluated via CT scan, Magnetic Resonance, and colonoscopy, and the data were compared with the in vitro results.The effectiveness of the treatment was also assessed using the tumor regression grade on surgical specimen by histological analysis. Tumor regression grading (TRG) is used to categorize the number of regressive changes after the treatment, estimating the percentage of residual tumor in relation to the previous tumor site.https://www.R-project.org/).Quantitative endpoints  measured at different timepoints were evaluated between groups of the treatments and controls using a repeated measures ANOVA (RMAN). Statistical analysis was performed using the R statistical environment (R Core Team (2019). R is a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL In this study, we demonstrated the intrinsic individual sensitivity of CSC to radiotherapy. This sensitivity profile was conserved under both in vitro and in vivo treatments in all the assessed samples, thus indicating that the subcutaneous tumors inherited this feature from their parental cells. Sensitive CSC were observed to be obtained by patients showing an optimal response to the therapeutic protocol. On the other hand, cells isolated from the biopsies of radioresistant tumors give rise to radioresistant CSC cultures. This suggest that the effect exerted by the administration of radiotherapy in vitro may be useful to predict the outcome of the treatment in donor patients. CSC cultures may be rapidly established by most of the samples obtained by LARC patients. These cultures can be rapidly expanded and employed for their sensitivity by using the described in vitro treatment protocol. The possibility to employ primary sphere-forming cultures without the need to use mice models facilitates the translation of the model, thereby reducing both the time and costs needed. We also observed that some samples did not yield CSC cultures. The reason for this requires additional investigations. The limited size of the patient cohort also highlights the preliminary nature of the presented study. Nevertheless, the novelty of the approach may help deepen the suitability of the model for translation into a clinical setting through a broader study. The proposed approach may be suitable thanks to its time frame, which may fit the current therapeutic settings for LARC. In conclusion, assessment of the in vitro based model predicted CRC patient responses to radiotherapy treatment, so this model could be developed as a powerful diagnostic tool for CRC treatment.