DUB Portal and promise to improve understanding of DUBs as a family as well as the activities of incompletely characterized DUBs (e.g. USPL1 and USP32) and those already targeted with investigational cancer therapeutics .Deubiquitinating enzymes (DUBs), ~100 of which are found in human cells, are proteases that remove ubiquitin conjugates from proteins, thereby regulating protein turnover. They are involved in a wide range of cellular activities and are emerging therapeutic targets for cancer and other diseases. Drugs targeting USP1 and USP30 are in clinical development for cancer and kidney disease respectively. However, the majority of substrates and pathways regulated by DUBs remain unknown, impeding efforts to prioritize specific enzymes for research and drug development. To assemble a knowledgebase of DUB activities, co-dependent genes, and substrates, we combined targeted experiments using CRISPR libraries and inhibitors with systematic mining of functional genomic databases. Analysis of the Dependency Map, Connectivity Map, Cancer Cell Line Encyclopedia, and multiple protein-protein interaction databases yielded specific hypotheses about DUB function, a subset of which were confirmed in follow-on experiments. The data in this paper are browsable online in a newly developed  Deubiquitinating enzymes (DUBs) are a family of ~100 proteases (in humans) that cleave ubiquitin from protein substrates . They arA growing body of literature shows that DUBs are dysregulated in many disease settings including cancer, chronic inflammation, and neurodegenerative diseases  and thatOnly a few DUB inhibitors have entered clinical testing, and none are as-yet approved . The curIn this paper, we take a combined and relatively unbiased computational and experimental approach to investigating the functions of DUBs, with an emphasis on cancer cell lines in which DUB inhibitors have been most extensively studied. First, we use high-throughput RNA-seq to measure the transcriptome-wide impact of an 81-member DUB CRISPR-Cas9 knockout library and also of seven small molecule DUB inhibitors chosen for high selectivity. As a complementary source of phenotypic and molecular information, we mine several large omics datasets , includiThe DepMap aims to identify genes that are essential for cell proliferation based on a genome-wide pooled CRISPR-Cas9 knockout screen conducted in more than 700 cancer cell lines spanning multiple tumor lineages. Each cell line in the DepMap carries a unique barcode to enable parallel analysis, and each gene knockout is assigned a ‘dependency score’ on a per cell-line basis. The dependency score quantifies the rate at which a cell carrying a particular CRISPR-Cas9 guide RNA is out-competed (‘drops out’) in a specific cell line; the more negative the dependency score, the stronger the impact on proliferation and thus, the higher the rate of guide disappearance from the population . The depThe CMap is a database of ~1000-gene mRNA signatures obtained from cells treated with small molecule drugs or in which individual genes have been knocked down using RNAi or overexpressed; data are available for ~3000 different genes and ~5000 small molecules. Each mRNA signature involves the measurement of a representative subset of the transcriptome using a bead-based (Luminex) assay . The CCLPPI datasets were obtained from BioGRID, IntAct, and Pathway Commons PPIDs, as well as the NURSA dataset that is focused on interactions among proteins involved in transcription; in these datasets, interaction was assessed using a variety of methods including affinity capture followed by mass spectrometry, affinity capture followed by western blotting, and assembly of reconstituted complexes from purified recombinant subunits . When coTo characterize the DUB family of enzymes, we combined laboratory experiments and data mining , Table 1Next, we leveraged the DepMap dataset to investigate DUB essentiality. It has been observed that genes having similar DepMap scores across large panels of cell lines are likely to have related biological functions , a propeExperimental support for functional associations obtained from data mining was sought by knocking out the DUB and several co-dependent genes and then assaying phenotypic similarity by mRNA profiling and other means. We also compared CRISPR-Cas9 knockout phenotypes with phenotypes induced by treatment of MDAMB231 or MCF7 cells with one of seven recently developed small molecule DUB inhibitors. We selected these inhibitors based on their reported selectivity for specific DUBs. We then compared drug-induced transcriptomic profiles with those obtained by CRISPR-Cas9-mediated gene knockout.GO cell division category), as expected for a DUB essential for proteasome function and thus, cell cycle progression  genes in MDAMB231 cells at a false discovery rate (FDR) adjusted p-value of p<0.05 included ten DUBs, one deSUMOylating enzyme (SENP2), and one ubiquitin-like protein (UBL5) . To detegression . When CMgression . In the gression . Both DUgression . Moreovegression . We inteDifferences in the expression of DUBs in normal tissues and malignant tumors has been described previously , but ther=–0.32, p-value =1.6 x 10–6, r>0.2), meaning that the most sensitive cell lines had the lowest DUB expression levels , except in the case of TNFAIP3, which was more highly expressed in more sensitive cell lines . Similar results were obtained when the top 5, 7, or 10 co-dependent genes were analyzed by GO enrichment analysis, but setting the threshold at seven genes yielded the largest number of GO terms for well-characterized DUBs that were consistent with literature knowledge. We therefore used a threshold of seven co-dependent genes for the remainder of our analysis. For 35 of 65 DUBs examined, we could identify at least one significantly enriched GO term for co-dependent genes. To enable easy visualization of these and related data, we summarized them as a series of concentric rings ; the rinGO: endopeptidase complex), and co-dependent genes for CYLD and TNFAIP3 included members of the NF-κB pathway   . We thergnaling) . An addiAmong the 69 DUBs whose knockout had little or no detectable impact on transcription in MDAMB231 cells (<20 DE genes) in our studies, 55 had little or no impact on proliferation of MDAMB231 cells in DepMap data and 7 were absent from the dataset . In no cTo search for evidence of physical interaction between genes scored as similar in function to DUBs based on DepMap and CMap data, we mined protein interaction datasets. First, we compiled the protein-protein interactions for each DUB from four PPIDs: BioGRID, IntAct, Pathway Commons, and NURSA . These dWe performed GO-enrichment analysis on DUB interactors and significantly co-expressed proteins and compared the resulting set of significantly enriched GO terms to the GO terms enriched in the co-dependent genes in the DepMap. This comparison enabled identification of GO terms that are significantly enriched in multiple datasets, providing corroboration for GO terms enriched in the DepMap co-dependent genes. We identified 35 DUBs whose co-dependent genes in the DepMap were associated with one or more significantly enriched GO terms. For 26 of these DUBs, at least one GO term was also enriched in the co-expressed proteins or PPID interactors for the relevant DUB . OverallDUB Portal (see methods for full URL). To further increase confidence in results obtained from data mining, we tested specific hypotheses by direct experimentation.From these data, we conclude that DepMap co-dependencies, CMap signatures, CCLE proteomic data, and PPIDs provide complementary and consistent data on the likely functions and physical interactors for the great majority of DUBs. These data are summarized in the different rings in Since most well-annotated DUBs have many reported substrates, we sought to use DepMap data to identify which function(s) of DUBs or their substrate(s) might be responsible for cell-essential phenotypes. For example, UCHL5 is known to interact with both the INO80 complex  and with the proteasome . UCHL5 hr = 0.25), which is one of the primary sources of ubiquitin in mammalian cells  implying that USP8 may have a role in recycling cytokine receptors as well as growth factor receptors. We also observed similarity between the mRNA profiles for USP8 knockout and overexpression of NF-κB signaling proteins such as TNFRSF1A and BCL10 . ESCRT complexes recognize ubiquitinated transmembrane receptors and facilitate their transport to lysosomes for degradation . USP8 inTo test this hypothesis, we used CRISPR-Cas9 to knock out three ESCRT proteins  co-dependent with USP8 in DepMap data . We founUSP7 has been the focus of many small molecule inhibitor campaigns  and seveTo investigate this hypothesis, we knocked out USP7 and C16orf72 in MCF7 cells using CRISPR-Cas9 and performed 3’DGE-seq after four days. We found that knockout of either gene resulted in a similar mRNA expression profile: in both cases, TP53 pathway genes were upregulated . Publish). Exposure of parental MCF7 cells to XL177A resulted in 737 DE genes, which strongly correlated with DE genes elicited by MDM2 inhibition with nutlin3a as well as knockout of USP7 in parental MCF7 cells  to isogenic MCF7 cell lines that were WT for TP53 or that had TP53 stably knocked down using shRNA ; we thenFigure 4dF7 cells . These fGO response to ionizing radiation, FDR =7.0 × 10–3 and 3.0 × 10–3 for USP38 and USP28 respectively), supporting the hypothesis that these DUBs are involved in DNA repair  (Copy number loss of USPL1 is frequent in cancer cell lines and predictive of increased sensitivity to CRISPR-Cas9-mediated knockout of USPL1 in the DepMap  and 3. I = 0.03) . The str = 0.03)  suggesti = 0.03) .To investigate these inferred connections, we used CRISPR-Cas9 to knock out members of the LEC  in MDAMB231 cells followed by 3’DGE-seq to score phenotypes. We found that the USPL1 knockout expression signature clustered with signatures for knockout of several of the LEC genes we tested and was most similar to knockout of ICE1 : in bothRetrograde Transport Endosome to Golgi. The co-dependent genes for USP32 include VPS52, VPS54, and RAB6A, which are known to be involved in endosomal sorting to the Golgi apparatus. RAB7 was not a codependent gene, however  (r = 0.776) which is consistent with previous data that shows that OTUD5 regulates UBR5  and ubiquitin conjugating enzymes (E2 enzymes) ( x 10–3) . DUBs ar x 10–3) . In this x 10–3) . This netes UBR5 . Many prtes UBR5 .There is growing interest in developing small molecule DUB inhibitors for use as human therapeutics  but the Our findings were consistent with three scenarios: (i) in the case of USP14 inhibition by I-335, the mRNA signature was very similar to that of target knockout, (ii) in the case of USP7.Hallmark P53 pathway, q-value=3.2 x 10–29) and cell cycle pathways , which are GO terms also enriched in USP7 knockout DE genes. This suggests that XL177A affects the same TP53 signaling pathway as USP7 knockout but to a greater degree. This difference might reflect differences in time point (24 hr for inhibition vs. 96 hr for knockout), incomplete knockout of USP7 by CRISPR-Cas9, or the existence of additional XL177A targets (some of which could include DUBs other than USP7 shown above to regulate TP53). Not all the effects of XL177A on cells were TP53-mediated however: exposure of TP53 KD cells to XL177A upregulated cell cycle genes, such as genes in the G2 checkpoint  as well as histone genes  .In contrast, inhibition of USP7 with XL177A resulted in 737 DE genes whereas knockout generated 48 DE genes. The two signatures were strongly correlated and querying CMap with the USP7 inhibitor signature also returned USP7 knock down signatures . Genes t 0.0066) .Exposure of cells to the OTUD7B inhibitor I-145, the USP30 inhibitor MF-094, USP19 inhibitor I-124, or the CSN5 inhibitors Compound 6 or CSN5i-3 resulted in strong perturbation of transcription  whereas CRISPR-Cas9-based knockout of OTUD7B, USP30, USP19, or COPS5 resulted in far fewer DE genes  and the small molecule and CRISPR-Cas9 signatures were not significantly correlated . HoweverDUB Portal.Rapid growth in publicly available and ‘functional genomic’ datasets affords an opportunity for extensive analysis of large gene families such as human DUBs. A total of nine different public resources measuring transcript signatures following gene perturbation (CMap), gene essentiality , protein co-expression (CCLE proteomics), and protein-protein interaction  were mined for data, in most cases starting with a CRISPR-Cas9 knockout or small molecule signature we collected in our laboratories. Comparison of enriched gene sets or GO terms made it possible to bridge different types of data. We observed substantial and encouraging consistency among datasets. For example, genes identified as co-dependent with DUBs in DepMap data frequently exhibited similar transcript signatures, were co-expressed across cell lines, and were physically associated. Overall, our analysis yielded three types of information: (i) potentially new or more precisely specified functions for several well-characterized DUBs, including a UCHL5, USP7, USP8, and USP14 (ii) potential pathways or functional roles for understudied DUBs, including a role for USPL1 in the Little Elongation Complex and UCHL3, USP38, VCPIP1, and USP42 in the regulation of TP53 signaling; (iii) insight into the DUB family as a whole, including evidence that 23 DUBs play a role in stabilizing 33 E3 ubiquitin ligases, most likely by antagonizing their auto-ubiquitination activities. 52 DUBs were found to be essential for proliferation in at least five cancer cell lines, and 34 of these DUBs affected the proliferation of cell lines from one tumor type more than cell lines from all other tumor types, potentially providing insight into disease context. These data are summarized in a simple graphical form in In several cases, our studies yielded unexpected hypotheses about the functions of DUBs that have already been well studied. For example, USP14 is a component of the proteasome and considered to be a promising therapeutic target in cancer due to the clinical success of other proteasome inhibitors . HoweverOne of the most promising potential uses of DUB inhibitors is to indirectly regulate the levels of disease-associated genes that are not conventionally considered to be druggable such as transcription factors and scaffolding proteins. This strategy has been most actively pursued for USP7, which is a regulator of MDM2, the E3 ligase for the TP53 tumor suppressor protein: inhibition of USP7 increases the levels of ubiquitinated MDM2, promoting its degradation and thereby increasing TP53 levels. Our data on USP7 are consistent with this hypothesis: we find that the top DepMap co-dependent gene for USP7 is MDM2, and knockdown of TP53 largely rescues the transcriptional phenotype observed for USP7 inhibition in TP53 wildtype cells. We find that USP7 has at least one additional substrate, C16orf72, that may also be a TP53 regulator. Moreover, the USP7 inhibitor XL177A has a phenotype that is independent of TP53 and involves upregulation of histone genes and genes involved in the G2M cell cycle checkpoint. We speculate that this may reflect the reported involvement of USP7 in the regulation of polycomb complexes , althougThe relationship between USP7 and MDM2 does not appear to be the only instance of a DUB regulating an E3 ligase. DepMap co-dependent genes for DUBs were strongly enriched and positively correlated with E3 ligases and other ubiquitin or ubiquitin-like transferases; in many cases, DepMap data were supported by PPID or co-expression data (e.g. the VCPIP1 DUB and HUWE1 E3 ligase). Overall, we identified 23 DUBs with at least one co-dependent E3 ligase, and eight of these DUBs had a co-dependent E3 ligase also supported by PPID or co-expression data. Selected DUBs have previously been reported to stabilize E3 ligases; for example, USP7 stabilizes MDM2, CYLD stabilizes TRAF2, and OTUD5 stabilizes UBR5 . HoweverUsing transcript profiling we compared seven small molecules reported by their developers to be highly selective inhibitors of specific DUBs to CRISPR-Cas9-mediated knockout of their targets. DUB inhibitor signatures were significantly similar to knockout signatures in only two cases: USP14 inhibition with I-335 and USP7 inhibition with XL177A. We conclude that these compounds are selective, although the signature of USP7 inhibition was substantially stronger than that of USP7 knockout. This was true in general, with exposure of cells to DUB inhibitors resulting, in all cases, in significantly more DE genes than knockouts. In the case of XL177A, our studies cannot determine whether this difference reflects the time at which the measurements were made, the degree of USP7 inhibition by drug or mRNA depletion by CRSPR-Cas9, or the existence of off-target effects. In the cases of the COPS5, OTUD7B, USP19, and USP30 inhibitors, the lack of a significant correlation between weak knockout phenotypes and the strong drug-induced phenotypes suggest substantial off-target activity. Small molecules targeting multi-protein families via competitive inhibition at the active site commonly exhibit some degree of polypharmacology . For example, seven DUB knockouts impact MDAMB231 viability in the DepMap but do not elicit a strong transcriptional phenotype following CRISPR-Cas9 knockout. We speculate that these discrepancies are due to differences in time point (4 days in the transcriptomics and three weeks in the DepMap screen) or incomplete knockout . More study is required to understand the origins and significance of discrepancies between DepMap and CMap. We also found that many physically-interacting and co-expressed genes are not co-dependent in DepMap. Such differences are not unexpected from a biological perspective, given the many different ways in which genes can interact, but further work focused on distinguishing technical errors from functional differences will be important.Because we analyzed whole gene knockouts rather than point mutations or protein deletions, the results in this paper do not distinguish among catalytic and structural functions for DUBs. This will be an important next step, particularly for clarifying the therapeutic utility of competitive small molecule inhibitors. For example, copy number loss was predictive of increased sensitivity to USPL1 deletion, so we hypothesize that a USPL1 inhibitor may be useful in this context; however, one study suggests that USPL1 is important for Cajal body architecture independent of its catalytic activity .Our studies provide a diverse set of data on the DUB family as a whole, as well as new insight into many individual DUBs, including several that have been studied intensively. One theme that emerges is that for genes with multiple proposed functions (USP7 and UCHL5 for example), a combination of profiling CRISPR-Cas9 knockouts or drug-induced perturbations with systematic mining of functional genomic databases makes it possible to distinguish among essential and non-essential phenotypes. A second is that more DUBs than anticipated have non-redundant roles in the tumor suppressor and oncogenic pathways, most notably TP53 regulation, suggesting new approaches to undruggable targets. The approaches described in this work are also directly applicable to other gene families and therapeutic targets.The Cas9-Flag was a generous gift from Andrew Lane at the Dana Farber Cancer Institute.All compounds were quality control checked using LCMS and NMR. CSN5i-3 was purchased from MedChemExpress. XL177A, I-335, Compound 6, I-145, I-124, and MF-094 were synthesized according to published methods and compound characterization data matched published data  .2 incubator at 37 °C, they were identity-validated by STR profiling (MDAMB231 (ATCC cat no. CRM-HTB-26) was maintained in DMEM media (Corning 10–017-CV) with 10% FBS (Life Technologies 26140–079) and 1% penicillin/streptomycin (Corning 30–002 Cl). MCF7 (ATCC cat no. HTB-22) was maintained in EMEM media with 10% FBS and 1% penicillin/streptomycin. Isogenic MCF7 and MCF7 stable shRNA p53 were a generous gift from the Galit Lahav lab. Cell lines were maintained in a 5% COrofiling  and veriCell lines stably expressing Cas9 were generated by lentiviral infection with pCRISPRV2- FLAG-CAS9 (Addgene #52961) followed by puromycin selection and monoclonal population generation by limiting dilution in 96 well plates. Monoclonal populations with the highest observed knockout efficiency of individual DUBs (USP10 and USP7) were selected.MDAMB231 Cas9-Flag and MCF7 Cas9-Flag expressing cells were seeded in 96 well plates (4000 cells/well) and allowed to adhere for 24 hr. crRNAs were resuspended in 10 mM Tris-HCl Buffer pH 7.4 (Dharmacon (Horizon Discovery) B-006000–100) and four crRNA guides per DUB were pooled to increase knockout efficiency. Guide transfection was performed according to the recommended manufacturer protocol with optimized conditions as follows. Cells were transfected with 25 nM crRNA and 25 nM tracr RNA using 0.2 µL/well Dharmafect 4, triplicate transfections were performed per condition. The media was replaced 16–18 hr post transfection.In parallel, MDAMB231 , MCF7 parental , and MCF7 TP53 shRNA  were seeded in 96-well plate format and allowed to adhere for 24 hr. DUB inhibitors and DMSO were dispensed into the 96-well plate using a d300 digital dispenser (Hewlett-Packard).Cells were lysed 96 hr post crRNA transfection or 24 hr post inhibitor treatment; the plates were washed one time with PBS on a plate washer (BioTek). The PBS was removed , and 30 µL/well 1 X lysis buffer  was added. The plates were incubated for 5 min at room temperature to aid cell lysis, and then frozen at –80 °C until RNA extraction.https://www.protocols.io/view/3-39-dge-high-throughput-rna-library-preparation-bumynu7w). All automated liquid handling steps described below were performed at the ICCB-Longwood Screening Facility. The cell lysates were mixed and 10 µl was transferred from each well of the 96 well screening plates to a well in a clean 384-well PCR plate, consolidating samples from up to four 96 well plates into a single 384-well plate for RNA extraction. SPRI (solid-phase reversible immobilization) beads, prepared as described previously . The pipeline removes any barcodes that differ by more than one base from an expected barcode and uses unique molecular identifiers (UMIs) to identify unique reads and remove PCR duplicates. RapMap was used to align reads to the transcriptome (GRCh38). The R package DESeq2 (version 1.30.0) was used for differential expression analysis, and the R package gseaMultilevel was used for gene set enrichment analysis of all genes sorted by the log2 fold change  using Molecular Signatures Database (MSigDB) gene sets.The data was separated by well barcode and the reads were converted to counts using the bcbio-nextgen single cell RNA-seq analysis pipeline .The Broad Institute Dependency Map dataset (CRISPR AVANA dataset 2020 Q3) was analyzed to determine the impact of DUB knockouts on cancer cell lines . The recTo determine co-dependent genes for each DUB, the CRISPR AVANA dataset was used to calculate Pearson correlations between each DUB and all other gene knockouts in the dataset. We limited this analysis to DUBs that had at least three dependent cell lines. To find the pathways and complexes significantly enriched in the strongest co-dependent genes for each DUB, the R package ClusterProfiler was used for gene set overrepresentation analysis of the top 5, 7, or 10 co-dependent genes for each DUB using MSigDB GO gene sets. The overall results were similar, but because the top seven co-dependent gene analysis yielded the largest number of expected GO terms for the well-studied DUBs, the top seven results were used.In order to extract the associations between DUBs and ubiquitin transferase enzymes, the GO gene set GO Ubiquitin Like Protein Transferase Activity was used to subset DUB co-dependent genes that are ubiquitin or ubiquitin-like transferases.To compile rich protein-protein interaction data for each DUB, interactions from multiple sources were compiled: IntAct, BioGRID, PathwayCommons, and NURSA. The R paThe normalized protein abundance data for CCLE cell lines was analyzed to determine genes co-regulated with each DUB. PearsonTo determine which associations between DUBs and genes have support across multiple analyses, the DUB-gene pairs were integrated across the four analyses: top seven DepMap co-dependent genes, CMap (Broad recommended threshold of Score >90), CCLE proteomics coexpression  and |z-score|>2), and PPIDs . Each DUB-gene association was given an evidence count score computed as the sum of the number of analyses that interaction was significant in. Thus, evidence scores range from 0  to 4 . The complete table integrating all of these datasets is provided so individual DUBs can be explored .To make the data and results presented in this paper available in a reusable form, we also generated online data resources.SCR_022476, version 2, https://labsyspharm.github.io/dubportal/), for exploring the most notable results from the experimental and computational analyses for each DUB. The page for each DUB first lists the standard identifier for the related gene, protein, and orthologs in model organisms. It also shows the significantly differentially expressed genes resulting from its knockout in the CRISPR-Cas9 screen described above as well as the significant gene sets calculated by gene set enrichment analysis (GSEA) over the MSigDB., which integrates existing large-scale datasets with new knock-out and inhibition experiments. The combined data confirm known DUB functions and, importantly, correct several current assumptions and highlight potential new functions of DUBs. The data are made available through an online portal, providing a useful resource for investigators interested in DUB functions or considering DUBs as drug targets. public reviews designed to be posted alongside the preprint for the benefit of readers; ii) feedback on the manuscript for the authors, including requests for revisions, shown below. We also include an acceptance summary that explains what the editors found interesting or important about the work.Our editorial process produces two outputs: i) Decision letter after peer review:eLife. Your article has been reviewed by 3 peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Volker Dötsch as the Senior Editor. The following individual involved in review of your submission has agreed to reveal their identity: Benedikt M Kessler (Reviewer #3).Thank you for submitting your article "Integrating multi-omics data reveals function and therapeutic potential of deubiquitinating enzymes" for consideration as a Tools and Resources article by The reviewers have discussed their reviews with one another, and the Reviewing Editor has drafted this to help you prepare a revised submission.eLife, a better presentation of the data , as well as clearer explanations on steps taken and thresholds applied will be needed.The compilation of all the available DUB information in one resource was seen as a useful contribution to the field by the majority of the reviewers. To be acceptable for Essential revisions:1. A major question that arose was how quickly the DUB portal is going to be outdated. Surely, for example, the number of papers column will be outdated in the near future. Are the underlying databases and the statistical analysis linked to the portal in a way that any new entries to these databases or any new relevant large datasets will be reflected in the DUB portal? This would make it a reliable resource for the community in the long term.2. Better presentation of the data:– The "synthesis" of data that the paper promises, and which to some extent is covered in the paper in Figure 3, is not well reflected in the online portal. Could there be a more innovative way, rather than tables, to present the data? A real benefit of this paper is seeing correlation or consistency among multiple data sources – so this should be better highlighted.– Columns in the online portal should be explained and defined within the portal.– The "About" page in the DUB Portal seems undeveloped.– Figure 1 was experienced as confusing by several reviewers. The figure implies that data is entered into the databases, when mostly data is taken out of databases. Please make it match with your workflow in a more intuitive way. This figure is important for readers to be able to follow the strategies taken.– For a reader of the paper, it would be interesting to know to what extent the new data (knockout/RNA-seq of 81 DUBs) has changed the conclusions or enabled conclusions that were not possible with only the mining of the existing databases.– Figure 3B: USP5 is represented twice, and one of these instances should probably instead be USP53.– Please provide the GEO accession number for the new data.– The abstract could better reflect the findings. A current focus is on c-myc, where it is somewhat unclear what new biology has been discovered.– In the discussion, page 30, "… by a molecule that inhibits multiple TP53-regulating DUBs". Do multiple TP53-regulating DUBs have highly similar binding pockets? And wouldn't such nonspecific inhibitors have more off-target effects? Is there prior evidence that makes this a viable strategy?3. Clearer explanations on steps taken and thresholds applied:– Knockout/RNA-seq data and Figure 2A: Why were those 81 DUBs chosen?eLife. The manuscript would benefit from an effort to provide simplified explanations for the rationale and what can be extrapolated in biological terms.– Many terms used are highly specialized and not easily accessible to the general reader of – Please provide information on how thresholds were selected. For example, genes whose knockout resulted in 20 or more DE genes – why 20? Page 24: why top 7 co-dependent genes?  What do the authors consider a significant "normalized enrichment score", what a good "CMap tau score"?4. Since your paper is considered for the "Tools and Resources" section, we won't make this a prerequisite for publication, but the confidence of the community in the mined data would be greatly enhanced by providing at least one targeted follow-up experiment on one of the novel hypotheses generated – for example substantiating novel findings presented in Figure 6. Essential revisions:1. A major question that arose was how quickly the DUB portal is going to be outdated. Surely, for example, the number of papers column will be outdated in the near future. Are the underlying databases and the statistical analysis linked to the portal in a way that any new entries to these databases or any new relevant large datasets will be reflected in the DUB portal? This would make it a reliable resource for the community in the long term.In response to this concern we have implemented a new, automated workflow that updates the information in the DUB portal on a regular basis. We are grateful for this suggestion and feel that the updates will help to make it a useful resource in the medium to long term. We describe below the specific elements of the website that have now been automated.The DUB Portal website is hosted on GitHub and therefore has version control. We used GitHub Actions to create a continuous integration service that automatically rebuilds the DUB Portal on a weekly basis. The build date and the versions of the resources used for the current DUB Portal are now shown on each page. Each build has an associated commit in version control, so it is always possible to retrieve and re-deploy an old version of the DUB Portal if desired. Each build of the DUB Portal now includes information derived from the following actions:– Querying PubMed for the number of publications associated with a given DUB gene. As the reviewers pointed out, this is almost certain to increase over time.– Querying the INDRA Database for the latest statements associated with a given DUB gene (the INDRA Database itself is updated nightly). INDRA includes the protein-protein interaction databases, so new entries to these databases will get added to the DUB portal. These databases are always expanding, so this automation will also add data moving forward.– Querying Reactome for the latest pathway annotations.About page.– Retrieving explanations for DUB dependencies derived from integrated analysis of BioGRID, IntAct, PathwayCommons, has also been automated. Data from BioGRID, IntAct, and Pathway Commons are re-queried on a weekly basis. NURSA is a static data source  so therefore it does not update and does not need to be repeatedly queried, Furthermore, BioGRID and PathwayCommons are included within INDRA, so new additions to these datasets will be included in the Evidence column of the appropriate DUB Portal table along with a link to a page that explains which datasets provided a specific piece of INDRA evidence. We describe these new features of the DUB Portal in an updated version of the As described above, we have also integrated the graphics of the manuscript with the tables in the Portal to make access to the underlying information more intuitive.2. Better presentation of the data:– The "synthesis" of data that the paper promises, and which to some extent is covered in the paper in Figure 3, is not well reflected in the online portal. Could there be a more innovative way, rather than tables, to present the data? A real benefit of this paper is seeing correlation or consistency among multiple data sources – so this should be better highlighted.In response to this suggestion, we have made numerous changes to the DUB Portal and the way it generates data. However, for the Portal to be updatable, it requires that data we read into the portal have a structured format such as tables with hyperlinks. This is not easily converted to the more visually appealing but inherently unstructured, static figures we have created for the publication itself. While an expert in data visualization might be able to program a site in which tables are used to automatically generate a new version of Figure 3 (the circular plot), this is not a function found in tools such as R-Shiny. Such a complex task seems to us beyond the scope of the current paper and our own expertise in data visualization.evidence column that provides information on correlation or consistency among multiple data sources. This is an important type of information for integrating data across types and sources.We note that even if the tabular presentation of data in the DUB Portal is less visually striking, it includes additional information that is not included in the manuscript itself such as an .– Columns in the online portal should be explained and defined within the portal.About page.This is an excellent suggestion. Each header field in the tables now has hover text explaining its meaning. A more detailed explanation for each field is provided on the – The "About" page in the DUB Portal seems undeveloped.About page to provide detailed documentation of the DUB Portal and each of its components. The About page also contains metadata on the version of different resources used to build the portal and links to Github where issues can be submitted.We have significantly extended the – Figure 1 was experienced as confusing by several reviewers. The figure implies that data is entered into the databases, when mostly data is taken out of databases. Please make it match with your workflow in a more intuitive way. This figure is important for readers to be able to follow the strategies taken.We have generated an updated version of Figure 1 that more closely matches the actual workflows. In this revised figure datasets are placed above one arrow that conveys their use in achieving the results below the arrow. We hope that the reviewers find this representation more intuitive. If not, we are happy to consider additional changes.– For a reader of the paper, it would be interesting to know to what extent the new data (knockout/RNA-seq of 81 DUBs) has changed the conclusions or enabled conclusions that were not possible with only the mining of the existing databases.This is an interesting point that cannot be answered precisely since it is a counterfactual scenario. However, we found that collecting a set of original  data on DUB knockouts was essential for us to gain traction with online resources, particularly the Connectivity Map. To query the Connectivity Map, an RNA-seq signature is required. There are not many DUB knockdowns in the Connectivity Map, and thus, we could not query the gene family directly. Moreover, signatures in the Connectivity Map were collected using an unusual L1000 Luminex bead-based assay that does not allow for conventional gene set enrichment. Thus, with respect to DUB biology the transcriptional knockout signatures we have collected are potentially more broadly useful than those in the Connectivity Map. One limitation in our study is that we did not use a CRISPR-Cas9 library for all 94 DUBs that were subsequently included in our bioinformatic analysis; however, as mentioned above, Dharmacon is interested in using data in the current paper to enhance its library.Discussion: this involves a comparison of CRISPR knockout RNA-seq signatures with signatures for small molecule inhibitors. Such an analysis would not have been possible without newly acquired data since no database of which we are aware has the necessary information on CRISPR and small molecule perturbations.We now highlight a specific use case of newly acquired RNA-seq dataset in the Moreover, in the section of the results describing USP8 we discuss use of our RNA-seq data and the DepMap to generate hypotheses about USP8 biology. The RNA-seq data gives insight into the downstream pathways that are perturbed (e.g. cytokine expression in the case of USP8 knockout) which provided additional insight beyond the functions predicted by the databases . Many receptors are trafficked through endosomes, so predicting that cytokine signaling would be the dominant pathway perturbed by loss of ESCRT machinery would not have been intuitive.– Figure 3B: USP5 is represented twice, and one of these instances should probably instead be USP53.Thank you for catching this typographical error. We updated Figure 3B to correct the instance of USP5 that is supposed to be USP53.– Please provide the GEO accession number for the new data.The GEO accession number (GSE187008) has been added to the Data Availability section.– The abstract could better reflect the findings. A current focus is on c-myc, where it is somewhat unclear what new biology has been discovered.Thank you for this suggestion. We removed the reference to c-myc and instead highlighted the individual DUBs that are the focus of the follow up studies in the paper.– In the discussion, page 30, "… by a molecule that inhibits multiple TP53-regulating DUBs". Do multiple TP53-regulating DUBs have highly similar binding pockets? And wouldn't such nonspecific inhibitors have more off-target effects? Is there prior evidence that makes this a viable strategy?Thank you for highlighting this issue. Our study suggests it would be worth targeting the redundancy of the DUB p53 regulatory mechanism. However, we were actually imagining that it might be necessary to develop a combination therapy to achieve this. The possibility that a multi-targeting compound might be developed is less certain, for the reasons the reviewer describes. To make it clear that multiple, selective DUB inhibitors may be the easiest way to address redundancy we modified the sentence in the Discussion to:“This could potentially be achieved by developing a small molecule inhibitor active against multiple TP53-regulating DUBs or by using a combination of selective compounds.”3. Clearer explanations on steps taken and thresholds applied:– Knockout/RNA-seq data and Figure 2A: Why were those 81 DUBs chosen?At the time this part of our study was performed, it seemed logical to use a commercially available guide RNA collection, in part because it would then be easy for others to repeat relevant aspects of our study. We therefore used a library from Dharmacon that is predominantly focused on DUBs. To our knowledge, there is no specific reason that the 81 DUBs in this library were selected out of ~100 in the whole genome. One factor is that, at the time that the library was constructed, annotation of DUBs was incomplete. For example, there are no guides in the library targeting the genes in the recently discovered MINDY and ZUP1 DUB families. To make this point clear we have modified the main text to read:“To generate knockouts, we used a commercially available arrayed CRISPR-Cas9 library targeting 81 out of ~100 DUBs and 13 additional proteins in the ubiquitin-proteasome system, including ubiquitin-like proteins…”We have written to Dharmacon to recommend that they update the collection in the DUB focused CRISPR library to make it more comprehensive and up to date, and they stated they are now considering this possibility. We anticipate that publication of this manuscript will increase their enthusiasm.– Many terms used are highly specialized and not easily accessible to the general reader of eLife. The manuscript would benefit from an effort to provide simplified explanations for the rationale and what can be extrapolated in biological terms.The reviewer makes an excellent point! In response, we have made two changes to the manuscript. First, we have added a new Table (Table 1) that briefly describes specific data types as well as their key features, parameters, and abbreviations. Second, we have revised the text to ensure that a rationale is included for each of the datasets collected and resources used. This rationale is included in the first section of the results so the reader will understand the approach before detailed information on individual DUBs is presented.For example, we expanded the rationale for collecting knockout RNA-seq data and using the CMap, which now states:“As a first step, we sought to leverage CMap to identify genes that, when silenced with RNAi, or overexpressed, had similar transcriptional effects as DUB knockouts. Perturbations that act on the same pathway often share similar transcriptomic signatures. ”The rationale for the DepMap analysis is:“It has been observed that genes with similar DepMap scores across cell lines are more likely to have related biological functions , a property known as co-dependency. More specifically, co-dependent genes are frequently found to lie in the same or parallel pathways (as defined by gene ontology (GO), for example) or to be members of the same protein complex. We identified co-dependent genes for DUBs and then ran GO enrichment analysis to identify pathways in which they were likely to be active.”The rationale for the protein-protein interaction analysis and co-expression analysis is:“Second, we used protein-protein interaction databases (PPIDs) such as BioGRID, IntAct, Pathway Commons, and NURSA to ascertain whether co-dependent genes might interact physically with one another. Third, we searched CCLE proteomics data for proteins whose expression levels across ~375 cell lines strongly correlated with the level of each DUB; it has previously been observed that proteins in the same complex are often co-expressed to a significant degree across a cell line panel .”– Please provide information on how thresholds were selected. For example, genes whose knockout resulted in 20 or more DE genes – why 20? Page 24: why top 7 co-dependent genes?  What do the authors consider a significant "normalized enrichment score", what a good "CMap tau score"?These are all good points. In response we have added the text to clarify threshold our selections, while also admitting that the specific values are sometimes somewhat arbitrary. Our goal was to set them at a level that emphasized strong phenotypes that we might attempt to explain biologically. For example, for co-dependent genes we played with a range of threshold values and found using a large number had little impact on the conclusions: when more weaker phenotypes were included in the analysis, few additional insights were obtained (see below). We have provided the complete dataset so other researchers interested in exploring the data more deeply can easily apply different thresholds.There were also some boundaries imposed by the methodology. For example, the CMap tool requires at least 10 differentially expressed genes for a signature query, but not all gene names are included in the CMap analysis because the tool uses only ~10,000 genes that are either measured or judged to be reliably inferred from L1000 signatures. Thus, we selected for analysis only knockouts that resulted in >20 significantly differentially expressed genes. All raw RNA-seq data are available on GEO (accession number: GSE187008) and differential expression results are provided on Synapse (syn25008205), so the readers can easily explore subtler phenotypes as well.To explain our choice of 7 top co-dependent genes for the DepMap analysis, we added the following sentence to the Results section: “When the top five, seven, or ten co-dependent genes were analyzed with GO enrichment analysis, the overall results were similar, but because the top seven co-dependent gene analysis yielded the largest number of anticipated GO terms for the well-characterized DUBs, we used the top seven co-dependent genes.” These details are reiterated in the Methods section. We uploaded the code for analyzing the top co-dependent genes in the DepMap to GitHub (https://github.com/lauradohertyws/DUBcodependencies) so that the reader can easily run the code with different thresholds.We did not use the Normalized Enrichment Score as a threshold for significance. Instead, we used the adjusted p-value to identify significant gene sets . We therefore added the adjusted p-values in parentheses along with the Normalized Enrichment Scores to demonstrate the significance of the results.The Broad recommends that CMap tau scores larger than 90 be considered significant. The following sentence is included in the Results section:“We used the recommended threshold of tau similarity score > 90 to determine significantly similar perturbations.”4. Since your paper is considered for the "Tools and Resources" section, we won't make this a prerequisite for publication, but the confidence of the community in the mined data would be greatly enhanced by providing at least one targeted follow-up experiment on one of the novel hypotheses generated – for example substantiating novel findings presented in Figure 6.We certainly appreciate this feedback but are not sure that we understand the criticism. From our perspective, the majority of the manuscript is made up of follow-up experiments designed to test the veracity of conclusions from database analyses. Admittedly, these do not have the depth one would expect for a manuscript focused on a single gene, but we have arrived at a number of new and potentially significant conclusions. The Discussion and DUB Portal now discuss how our “validation” studies could be followed up further. Specific examples of hypotheses from data-mining that were confirmed experimentally include the following:1. By comparing the KO of USP14 to inhibition of USP14 with a small molecule inhibitor we confirmed that loss of USP14 enzymatic activity results in selective upregulation of the ubiquitin gene. This provides experimental evidence that the correlation between USP14 and ubiquitin in the DepMap is meaningful. Given the many reported functions for USP14, the resulting hypothesis that it plays a key role in ubiquitin recycling, and this is likely to be its primary non-redundant function is unexpected.2. Using additional CRISPR knockouts we compared the impact of loss of USP8 on cytokine transcription to the impact of loss of ESCRT components to determine whether the phenotype was likely due to perturbed ESCRT function. The ESCRT members we selected to test were chosen based on the analysis of the DepMap.3. Based on the DepMap analysis, we predicted that C16orf72 would be a novel regulator of TP53 signaling. When we knocked out C16orf72 we observed an impact on TP53 target genes. To our knowledge (at least at the time of our experiment) no connection between C16orf72 and TP53 existed in the literature.4. Given the strong correlation between MDM2 and USP7 (as well as more recent literature on USP7), we predicted that knockdown of TP53 would rescue the impact of a USP7 inhibitor, and we tested this experimentally.5. Analysis of the DepMap suggested that USPL1’s key function is in the Little Elongation Complex. Although USPL1 has been connected to the Little Elongation Complex in the literature, USPL1 has not been well-characterized, so the systematic comparison of USPL1 loss to loss of other components of the Little Elongation Complex is a valuable test of data derived from integrated DUB analysis. We conclude the pleiotropic phenotype associated with following loss of USPL1 is mediated by the Little Elongation Complex rather than another, as-yet undiscovered function of USPL1.6. We also provide confidence in the mined data via systematic comparisons with a large body of existing DUB literature. This should provide confidence to readers less familiar with DUB biology.


Patients with rheumatoid arthritis require frequent consultations to monitor disease activity and intensify medication when treatment targets are not met. However, because most patients are in remission during follow-up, it should be possible to reduce the number of consultations for them. Electronic patient-reported outcomes (ePROs) could be used to identify patients who meet their treatment goal and who could therefore be eligible to skip their visit.The aim of this study was to assess the probability that patients with low disease activity scores on their ePROs do not need a disease-modifying antirheumatic drug (DMARD) or steroid intensification in the first 2 weeks after completion of the ePROs.This medical-records review study compared results of ePROs answered during routine care with DMARD or steroid intensifications collected from anonymized electronic medical record at Reade. The primary outcome was the positive predictive value (PPV) of having a low disease activity score on an ePRO for not receiving a DMARD or steroid intensifications within 2 weeks. The 3 studied ePROs (and respective low disease activity outcome) were the Routine Assessment of Patient Index Data 3 (RAPID3) (score <2), Patient Acceptable Symptom State (PASS) (yes), and the flare question (no). The secondary aim of the study was to assess which combination of ePROs resulted in the best PPV for DMARD or steroid intensifications.Of the 400 randomly selected records, ultimately 321 were included (302 unique patients). The PPV of a RAPID3 <2, being in PASS, and a negative answer on the flare question were, respectively, 99%, 95%, and 83% to not receive a DMARD or steroid intensification within 2 weeks. The combination of a RAPID3 <2 and a negative flare question resulted in a PPV of 100%; this combination was present in 29% (93/321) of the total study population.The RAPID3, PASS, and flare question have a high diagnostic accuracy to identify individuals who will not receive a DMARD or steroid intensification in the following 2 weeks. The combination of the RAPID3 and flare question yielded the best combination of diagnostic accuracy and highest percentage of patients who could be eligible to skip a visit. These results suggest that accurate identification of patients who meet their treatment goal with ePROs is possible. Therefore, the current system results in a vast amount of  unnecessary consultations. These consultations consume so much health care costs and time that the current system will not be sustainable in the future: the projected number of rheumatologists needed to sustain the system is twice the number of available rheumatologists by 2030.4 Therefore, ways should be sought to lower the number of outpatient clinic visits for patients in remission or low disease activity.The disease activity of patients with rheumatoid arthritis (RA) is frequently monitored at the outpatient clinic following treat-to-target principles, with the goal to achieve remission or low disease activity.If we could identify patients who meet their treatment target (low disease activity or remission) before consultations, we could let them skip their visit. For this to work, we would need to (1) assess disease activity without a consultation and (2) be confident that patients with low disease activity assessments truly are in remission. Disease activity assessment with electronic patient-reported outcomes (ePROs) could meet both requirements, as ePROs are scored independently by patients and tend to overestimate disease activity. This overestimation should lead to a low number of patients who are falsely classified in remission.5 the Patient Acceptable Symptom State (PASS),6 and the flare question “Are you having a flare of your RA at this time?” might be suitable to identify patients who are in low disease activity.7 This is because patients who are in RAPID3 remission or low disease activity are in more than 90% of the cases also in DAS28 low disease activity or remission.9 Furthermore, patients have a relatively low pain threshold  to classify their symptoms as not acceptable in the PASS.6 This makes it likely that patients who classify their symptoms as acceptable in the PASS truly have low disease activity. And lastly, patients in remission have a high agreement (κ >0.73) with their physician regarding their nonflare status.8 However, whether identification of patients who meet their treatment target is possible with these ePROs is unknown.Several ePROs such as the Routine Assessment of Patient Index Data 3 (RAPID3),11 Therefore, the relationship between the ePROs and actual disease-modifying antirheumatic drug (DMARD) or steroid intensification (rather than doctor-measured disease outcomes) will be key in the assessment whether accurate identification of patients in low disease activity is possible. Very little is known about the relationship of the RAPID3, PASS, and flare question with treatment intensifications. This study will evaluate the relationship between the results of 3 ePROs  and receiving a DMARD and/or steroid intensification, as a preparatory step toward identifying patients in remission or low disease activity before outpatient clinics to let them skip their visits. We hypothesize that patients who report to be in remission or low disease activity according to their RAPID3 score, PASS, or flare question answer will receive very few DMARD or steroid intensifications.So far, research has focused on the relationship between ePROs and clinical composite measures. In practice, however, the decision to intensify treatment when arthritis is active is more complex than not meeting a DAS28 or Clinical Disease Activity Index threshold.This was a medical records study performed at Reade, a secondary care facility for rehabilitation and rheumatology in Amsterdam, the Netherlands. All patients in routine care who are scheduled for a telephone call or outpatient clinic visit with a rheumatologist or a rheumatology nurse received an invitation by mail to complete the RAPID3, PASS, and flare question in an online portal before their consultation. The results of the ePROs and the corresponding patients' demographics  of each patient were extracted from the electronic medical records. Records (results of the ePROs) were included if they were completed between March 1, 2020, and January 15, 2021, and they regarded adult (aged ≥18 years) RA patients. Records were excluded if they regarded a patient with a disease duration less than 1 year (newly diagnosed patients were not considered to be eligible to skip their visits) or if no consultation took place within 2 weeks of the completed ePROs (to minimize the risk of changes in disease activity between completing ePROs and the consultation). Records that were completed within a 3-month follow-up time frame of an already included ePRO record of the same patient were also excluded. All eligible records were ranked using a random-number generator in SPSS , and the first 400 ePRO records were included in this study.Disease-modifying antirheumatic drug or steroid intensifications were looked up in anonymized patient files extracted from the electronic medical records of Reade by one researcher (S.J.V.). Cases were discussed with a second researcher (B.F.S.) in case of doubt. Both researchers were unaware of the patients' ePRO results during this process. Disease-modifying antirheumatic drug or steroid adjustments were recorded as “direct” (within 2 weeks) or “during follow-up” (within 3 months). The 2-week time frame was chosen as patients who are initially called and suspected to be flaring are seen within 2 weeks at the outpatient clinic for an in-person follow-up.The procedures were performed according to legislations of the Medical Research Involving Human Subjects Act and Dutch ethical standards for research involving humans. Because this was a study with anonymized medical files, no approval from the ethics committee was needed.The primary outcome of this study was the positive predictive value (PPV) of a low RAPID3 score, being in PASS, or not reporting a flare for direct DMARD or steroid intensifications. This result can be interpreted as the probability that a patient with a low RAPID3 score, who is in PASS, or who reports not to be flaring does not receive a DMARD or steroid intensification.5The RAPID3 categorizes patients into remission (0–1.0), low (1.1–2.0), moderate (2.1–4.0), or high disease activity (>4). These categories were dichotomized into a “low” (≤2) and “high” (>2) RAPID3 score.6The PASS is a single yes or no question: “Considering all different ways RA is affecting you, if you were to stay in this state for the next 3 months, do you consider your current state satisfactory?” Patients who answered “yes,” indicating they were satisfied with the current status of their RA, were defined as patients “in PASS.” Patients who answered “no” were defined as patients “not in PASS.”flare question is a single yes or no question from the RA Flare Core Domain set.7 Patients were asked if they experienced a flare: “Are you having a flare of your RA at this time?”The Disease-modifying antirheumatic drug or steroid intensifications were defined as an increase in dosage, adding or switching prednisone , conventional synthetic, biologic, or targeted synthetic DMARDs. In addition, intentions to intensify DMARDs were recorded when they were formulated by a rheumatologist, but the intensification was not executed because of refusal by a patient. Both actual intensifications and intentions to intensify were considered as DMARD or steroid intensifications in the analysis.Medication reductions were defined as a decrease in dosage or stopping medication. If medication was temporarily halted in case of an infection or before an operation, this was not registered as a reduction. Tapering of short-term high-dose prednisone was also not counted as a reduction.2 test, or if one of the cells had a frequency <5, a Fisher exact test. The α was set at .05. No sample size calculation was performed as no accurate method exists to calculate the power for this type of study.12 Following the Bland-Altman method, a 100-subject sample usually considered a good sample size to evaluate agreement. Therefore, we included a 100-subject sample per variable  to assess the agreement with DMARD intensifications. We chose to assess an extra 100  to account for excluded records.In case of normally distributed data, data were summarized using means and SD. Dichotomous data were summarized using frequencies and percentages. In case of not normally distributed data, data were summarized using medians and interquartile range. For these dichotomous data, we used a χTest measurement properties  with 95% confidence intervals (95% CIs) were calculated. This was done for both direct DMARD or steroid intensifications and during follow-up. A RAPID3 ≤2, being in PASS, and reporting not to be flaring were considered as a positive diagnostic test result; henceforth, not receiving a DMARD or steroid intensification was considered as the criterion standard. Electronic patient-reported outcomes with a high specificity can be used to identify the presence of a “condition,” in this case low disease activity. The PPV is the chance of having this condition when the ePROs show a positive test. To test the accuracy of the ePROs, receiver operating characteristic curves were calculated, yielding area under the curve (AUC) values.As secondary outcomes, all combinations of the ePROs were evaluated with a cross-tabulation to assess which combination had the best PPV and AUC. In addition, for each ePRO and the best combination of ePROs, the intensification-free survival was displayed in a Kaplan-Meier curve. All statistical analyses were performed in SPSS version 23 (SPSS Inc).Between the March 1, 2020, and the January 15, 2021, the RAPID3, PASS, and flare question were completed 3688 times. Of the 400 randomly selected ePRO records, 79 were excluded for the following reasons: an inconclusive RA diagnosis, despite the RA treatment code (n = 7); disease duration <1 year (n = 48); no consultation within 2 weeks of the completed ePROs (n = 21); and ePRO records within 3 months of a previously completed ePRO record of the same patient (n = 3). The 321 records that were included were completed by 302 patients (19 patients with 2 entries); characteristics of the study population are shown in Table p < 0.001; during follow-up: 5% vs 39%, p < 0.001), in patients in PASS compared with patients not in PASS , and in patients who reported not to be flaring compared with patients who reported to be flaring . The respective sensitivity, specificity, NPV, and PPV of each ePRO are displayed in Table The percentage of patients who received a direct DMARD or steroid intensification or during the follow-up was significantly less in patients with a low RAPID3 compared with patients with a high RAPID3  of the study population; adding the PASS to this combination did not improve the test characteristics (lower AUC 0.682 vs 0.672) and reduced the size of the group that would have been eligible to skip their visit. The results of the 2 × 2 cross-tabulations of all combinations can be found in Appendix 1, The DMARD or steroid intensification-free survival during the 90 days of follow-up is displayed in Figure. In the group of patients with a low RAPID3 and that reported not to be flaring, the first DMARD or steroid intensification was prescribed after 40 days. The intensification-free survival curves of the separate RAPID3 and RAPID3 combined with the flare question were very similar.Disease-modifying antirheumatic drugs were reduced within 2 weeks in 8% (24/321) of the total study population; 6 additional reductions were registered in the remainder of the follow-up period. Respectively, 12 of 24 (50%) and 13 of 30 (43%) of these reductions were registered in the low RAPID3 and “no” flare group.To the best of our knowledge, this was the first study in patients with RA that investigated the predictive value of the RAPID3, PASS, and flare question for DMARD or steroid intensifications. Independently, a low RAPID3 had the best PPV (99%) for not receiving a direct DMARD or steroid intensification, but the PPVs of the PASS (95%) and flare question (93%) were also high. A low RAPID3 combined with not reporting a flare further increased the PPV to 100% and remained high during the 3-month follow-up (96%). This indicates that if patients with the combination of no (self-reported) flare and a low RAPID3 were to skip their visits, no patients who would need treatment intensification would be missed in the first 2 weeks.13 indicated that monitoring disease activity with the RAPID3 alone captured changes in objective inflammatory signs insufficiently. This premise is supported by the low correlation of high RAPID3 scores with the DAS28. Nevertheless, our results suggest that the RAPID3 can be of value to identify stable patients, because patients who report a low RAPID3 score truly are in a low disease activity/remission state according to their DAS. This is demonstrated by the high predictive value of a low RAPID3 for low disease (according to the DAS28) by Wiegel et al.8 and Boone et al.9 and the low number of DMARD or steroid intensifications these patients receive in our study. Therefore, the RAPID3 could be used to identify patients in remission, especially when combined with the flare question. The combined results of the RAPID3 and the flare question demonstrated perfect (100%) results (in terms of PPV) within 2 weeks and excellent (96%) results after 3 months. This was further strengthened by the intensification-free survival curve that illustrated that in the first 40 days no DMARD or steroid intensifications would have been missed.Our results show that approximately one-third of the RA population has a low RAPID3 score and does not report a flare; these patients may be eligible to skip their visit on the basis of their ePRO results. Previously, Boone et al.15 As a result of the low sensitivity and NPV, there are still many patients who may be eligible to skip their visit, which have not been identified. There may be ePROs with a higher sensitivity and NPV that could identify a bigger proportion of patients who could be eligible to skip their visit. However, the priority is to not miss any patients who need a DMARD or steroid intensification. Therefore, the likely consequence for any PROM with a high specificity and PPV is that some patients who will not receive a DMARD or steroid intensification will also be called up. This could also have a beneficial effect, as the patients who have indicated some form of disease activity on their ePROs will be able to discuss it (and prevent further deterioration).The sensitivity and NPV of the ePROs  were low, which indicates that many patients with high disease activity scores on their ePROs did not receive DMARD or steroid intensifications. This may be due to (1) the discrepancy between the patient and doctor assessed disease activity or (2) a failure to treat to target.16 In addition, identifying stable patients could be used in other strategies to reduce unnecessary visits, lower health care costs, and reduce manpower needed per patient. For instance, aside from letting patients skip consultations completely, an alternative strategy would be to replace the physical consultation with a rheumatologist with a nurse consultation by phone. This would decrease the workload of rheumatologists and has been shown to be just as good.19Based on our data, it could be argued that the RAPID3 on its own misses only very few DMARD or steroid intensifications. However, we suggest combining the RAPID3 with the flare question. The flare question takes only a few extra seconds to answer, and combining it with the RAPID3 slightly increased the PPV without reducing the group size of patients. If these results are validated prospectively, ePROs could be used to allocate outpatient clinics visits according to need. The potential of this strategy is also suggested by the results of the SEMORA study that combined patient-initiated follow-up with telemonitoring with ePROs.20 Because of the COVID-19 pandemic, there were more consultations by phone than usual. We do not believe this has significantly influenced our results, as the number of DMARD or steroid intensifications was in line with numbers reported by Saraux et al.17 (20%) and Dougados et al.14 (15%). This study has several limitations. First, the need for a consultation was defined as receiving a DMARD or steroid intensification. However, there may be other reasons for having to consult a rheumatologist, such as monitoring adverse events or reducing medication (dosage). However, these nonprescribing medical activities such as monitoring adverse events could also be performed by rheumatology nurses. Based on these data, 50% of the reductions would take place in the low RAPID3 and no-flare group. Therefore, it may be possible to actively screen this group for the possibility to reduce medication (dosage) with an additional question. In a prospective study, medication reductions should be a point of consideration. In addition, despite not receiving a DMARD intensification, disease activity might still be moderate or high. This might be considered a limitation, although previous studies have shown that the agreement of PROs with the DAS28 in the low disease activity categories is also high.9 Second, rheumatologists were not blinded from the ePRO records, which may have led to more treatment intensification in patients with undesirable ePRO results. However, this resembles the routine clinical practice in which the intervention would also be implemented and was therefore not considered as an issue. Third, identification of stable patients can only work if patients complete their ePROs. Our results are therefore applicable only to patients who complete their ePROs. In our setting, approximately half of the ePRO questionnaires are completed. Persuasive principles and optimization of the ePRO process could improve the completion rates.21The characteristics of this study population were comparable to characteristics of RA populations in Western countries.The RAPID3, PASS, and flare question have a high diagnostic accuracy to identify patients who will not receive a DMARD or steroid intensification in the next 2 weeks. With the combination of a low RAPID3 and flare question, none of the patients received a DMARD or steroid intensification in the following 2 weeks. These results suggest that accurate identification of patients who meet their treatment goal with ePROs is possible.