We complement these resources with the largest dataset of age-dependent high-level cognitive assessment of mouse models of Alzheimer’s disease, expanding knowledge of affected cognitive domains from male and female mice of three strains. We envision that these new platforms will enhance sharing of protocols, data availability and transparency, allowing meta-analysis and reuse of mouse cognitive data to increase the replicability/reproducibility of datasets.Open Science has changed research by making data accessible and shareable, contributing to replicability to accelerate and disseminate knowledge. However, for rodent cognitive studies the availability of tools to share and disseminate data is scarce. Automated touchscreen-based tests enable systematic cognitive assessment with easily standardised outputs that can facilitate data dissemination. Here we present an integration of touchscreen cognitive testing with an open-access database public repository ( The public nature of research and increased rigor applied to research outputs have encouraged new approaches to enhance transparency, data sharing, and reproducibility . Over thRecently, there have been several attempts to build databases of rodent behaviour data. The Jackson Laboratory has developed the Mouse Phenome Database, a repository of mouse data taken from several studies . AdditioConventional cognitive assessments in mouse models are subject to large variation , which mEfforts to address these important gaps in data collection, automation, and translational research led to the development of the rodent touchscreen testing method . This temousebytes.ca). We reveal longitudinal heterogeneity as well as commonalities in cognitive function between the various strains modelling AD. For example, 3xTG-AD mice, males and females, present early attention deficits (at 3–6 months of age) when compared to their age matched controls, demonstrating reproducibility of earlier results. Overall, our cognitive assessment suggests which mouse models can be used to model cognitive phenotypes consistent with Alzheimer’s disease.Here we used data obtained with male and female mice from three distinct mouse lines commonly used in Alzheimer's disease (AD) research to highlight the use of a new repository and Web-based software, MouseBytes (mousebytes.ca), so they can pre-process, run automated quality control scripts, store, visualise, and analyse their data alone or alongside other researchers’ stored data. Moreover, researchers can use a knowledge sharing tool https://touchscreencognition.org to disseminate community-driven information, including standard operating procedures (SOPs) and protocols. We foresee this repository for touchscreen data as a major step towards increasing the availability of datasets, including negative results, that can serve to evaluate reproducibility, decrease publication bias and to bring high-level cognitive assessment into the Open Science era.MouseBytes is available to researchers worldwide ] , behaviomousebytesQC). The rules and codes for the QC are available for download and modification in GitHub  (mousebytes.ca) and integrated to the analytics TIBCO Spotfire to generate an interactive visualisation platform for 5-CSRTT, PAL, and PD . Briefly, to navigate through the data in mousebytes visualisation (Spotfire) the users should select the cognitive task in the dropdown menu. After the selection of the cognitive task, corresponding features are selected . A glossary with the description of the training and probe phases is found in MouseBytes (mousebytes.ca_description). Moreover, the user can check or uncheck the filter boxes on the right side of the page to define the data to be visualised and export or analyse specific graphs using the side tabs. This allows features selected, such as site, mouse strain, genotype, sex and age for example to be quickly compared .Following completion of individual experiments, Extensive Markup Language (XML) files were generated using the Animal Behaviour Environment Test II  software. XML files were uploaded into MouseBytes and screened using an automated quality control (QC) procedure which is a tool available at MouseBytes.ca (ed-data)  . Files Our pipeline enabled collection of extensive amounts of data from different ages, mouse strains, and sex. In total, we tested 652 different mice and generated 62,411 xml files . Importantly, by scanning the files through our automated QC procedure, we identified 487 files (0.8%) that did not meet the QC system criteria. 62,393 xml files (99.2%) passed the automated QC criteria and were transferred to the database. After QC, files that did not meet the criteria, or could not be fixed, were automatically discarded and were not used for analysis .Due to the amount of data we generated in this work, classical graphical format to visualise all information contained in these datasets would require close to 30–40 figures with 6–10 panels each, depending on the kind of comparisons being performed. This situation underscores the need for online and on-the-fly data assessment, multidimensional visualisation, and comparison using online visualisation tools , such asGitHub codes; Currently, our system has been optimised for the intake of data from the Bussey-Saksida Operant Chamber systems . There are alternative commercial touchscreen systems available (i.e. Med Associate K-Limbic touchscreen operant chambers), as well as several open source alternatives . In orde5-CSRTT MouseBytes data link, PAL: PAL MouseBytes data link, and PD: PD mousebytes data link. One of the features of this open-access database is the possibility of downloading a standardised dataset (using a hyperlink generated by MouseBytes) related to particular experiments (i.e. linked to a particular figure of a paper) to perform customised analyses (A series of videos is available on the website that demonstrates how to use MouseBytes: MouseBytes-Guidelines).The sample sizes for all experiments/tasks can be found in Statistical analysis of the performance of distinct AD mouse models was conducted using R, taking advantage of the fact that CSV files can be generated for specific datasets using MouseBytes, which facilitates the use of open-source statistical packages. A summary of the split-plot ANOVA of all behavioural measures for each genotype can be found in Variability of mouse performance in behavioural tests across different laboratories is an important issue for replicability . The usea#). Interactions with test site that were significant typically had a small effect size and lacked consistency across behavioural domains and mouse strains (a#). Overall, the evidence suggests low site-to-site variability and high replicability for touchscreen test performance. For example, low variability was observed between sites when we compared longitudinally the performance of wild-type female mice (B6129SF2/J) and their AD-mouse model counterpart, 3xTG-AD, in the 5-CSRTT task (attention). We observed a difference in accuracy (0.6 s stimulus duration) in wild-type females at 3–6 months of age between the two sites (dataset 1). Other than that, no statistically significant differences were found for either accuracy or omission for both B6129SF2/J females  or 3xTG females  between the two sites. We observed similar reproducible results for mouse touchscreen performance across the other AD models and their wild-type counterparts (see mousebytes.ca for more comparisons). As an important feature, MouseBytes allows the generation of dataset hyperlinks to easily identify and download the raw data used to generate each figure panel .To directly assess potential site variability in the current dataset stored in MouseBytes, we included site as a factor in our 5-CSRTT analyses. The 5-CSRTT task was chosen due to the larger cohorts of mice used in these experiments at the two sites. Throughout the analyses of 5-CSRTT measurements , no consistent pattern of main effects or interactions emerged for site between age, sex, genotype, strain, or measure (stimulus length) . In addition, we also tested female mice and similar to the males, 3xTG-AD female mice also presented lower accuracy (dataset 11) and no difference in omissions (dataset 12) when compared to the wild-type controls. Moreover, both male and female 3xTG-AD mice that were tested starting at 4 months of age also presented lower accuracy  and no difference in omissions (dataset 16) when compared to controls (b#). We also examined vigilance (the ability to maintain concentrated attention over a prolonged period of time), which was also previously reported to be affected in this mouse line (c#) or 3xTG- AD female mice . No differences were observed for omissions . 3xTG-AD mice did not show any sign of deficits in visual discrimination learning or behavioural flexibility in PD , as this was the original background in which this mouse line was generated , and it neration  (see Mat did not . Moreove did not .e). Initial training on the PAL task did not generate robust results as both 5xFAD and their controls  were poor performers . No deficits in behavioural flexibility or visual discrimination learning were observed for 5xFAD mice when compared to their respective controls . Furthermore, an early deficit in behavioural flexibility was observed for female APP/PS1 mice at four months of age in the PD reversal task, which is interesting from the point of the translational utility of this mouse model, as behavioural flexibility deficits are not typically associated with AD at early stages of the disease progression (h#).APP/PS1 mice  did not show any sign of attentional deficits in the 5-CSRTT task at any age  – Tab 3 gression   and males (dataset 18) B6129SF2/J presented higher levels of accuracy on 5-CSRTT at 3–6 months of age, but not at 11–13 months of age , when compared to the other two wild-type strains tested (B6SJLF1/J and C57BL6/j). Moreover, both male and female B6SJLF1/J mice (3–6 and 11–13 months of age) were found to engage in more premature responses than the B6129SF2/J and C57BL6/j lines . This suggests a general phenotype of impulsiveness inherent to these B6SJLF1/J mice. We envision that with multiple users depositing their data in MouseBytes, it will be relatively easy to make comparisons of performance for thousands of mice from different strains. Ultimately, these overarching analyses could help to inform the background strains to be used for new mouse lines to investigate specific high-level cognitive domains, for example, models that can now be generated using new genome-editing techniques such as CRISPR/Cas.The choice of background strain for mouse models of disease can have major implications for cognitive assessment . Howeverdataset 25) or 11–13 months of age (dataset 26). Similarly, no differences were found when 3xTG-AD males and females were compared at 3–6 (dataset 27) or 11–13 months of age (dataset 28). These results suggest little or no difference for high-level cognitive performance between male and female mice. To the best of our knowledge, the dataset presented here and deposited in MouseBytes provides the most extensive evaluation of performance of female mice in touchscreen tests.Recognition that behavioural rodent research is biased towards using male mice has led funding institutions to establish specific guidelines in the choice of animals for research . Severalkml3d  and low numbers of correction trials . Mid-perIn the PD tasks, behaviour of high performing mice was characterized by high response accuracy (% correct) and low number of correction trials (errors) . The typIn the 5-CSRTT, high performing mice were characterised by high accuracy (% correct), low omissions, and higher perseverative responses, i), suggesting the potential that these mice present lack of motivation or abnormal motor function at the ages tested, which has been described previously . Further information and requests for resources and reagents should be directed to and will be fulfilled by the Lead Contact, Marco Prado.All cognitive data are deposited in MouseBytes  and its age-matched control mice (B6129SF2/J RRID:MGI:101045); 5xFAD  and control mice B6SJLF1/J  and control mice C57Bl/6J (RRID RRID:MGI:000664). All mice used in this study were bred at the Jackson Laboratory in Bar Harbor, Maine and shipped to Canada. Procedures were conducted in accordance with approved animal protocols at the University of Western Ontario (2016/104) and the University of Guelph (3481) following the Canadian Council of Animal Care and National Institutes of Health guidelines. The N values for each group of animals can be found in The choice of AD mouse lines considered mice with different rates of accumulating pathology, their use by a variety of researchers, and commercial availability from a single source. Three AD mouse lines were tested: 3xTG-AD , while the other half was housed at The University of Western Ontario . For all the 5-CSRTT experiments, an equal distribution of male and female mice was kept at both sites, allowing for comparison and reproducibility studies. For PD and PAL experiments, all male mice were tested at The University of Western Ontario and all the females were tested at the University of Guelph. At each site, mice were housed in a single colony room maintained on a standard 12 hr light cycle . All the experiments were conducted during the light phase of the cycle. The colony rooms were typically maintained at a temperature of 22–26°C. Mice were initially housed in groups of two mice per cage (16 cm x12cm x 26 cm) and they had their tail tattooed with a unique animal identification number. During each experiment, mice were maintained on a restricted food diet to ensure adequate levels of motivation and to maintain their body weight at 85% of their original weight. Male mice were fed on average 2.5 g of chow, while female mice were given 2.0 g per day (Tekland Chow – Harlan). We set a maximum initial weight of 25 g and if a mouse was over that weight we slowly decreased its weight to 25 g and then set this as a 100% . Mice were weighed every other day to ensure maintenance of body weight at 85% of original weight and water was available ad libitum throughout the course of the experiment. After each session, mice received food according to their body weight. Mice are social animals, but it has been described previously that group housed mice can exhibit aggressive behaviour towards their cage mates and establish social hierarchy. This can also affect, in the non-dominant mouse, gene expression and induces depression and anxiety-like behaviours . After fSWE), microtubule-associated protein tau (P301L) and presenilin1 (M146V)  (M146V knocking (129/C57BL6 background) mice, were co-microinjected with human mutant tauP301L and the double mutant APPSwe (MK670/671 NL). Both are under the control of Thy1.2 expression cassette  . Brieflycassette . Age, secassette .SWE), I716V (Florida - APPFL), and V717I (London - APPLON) mutations. In addition, these mice express the M146L and L286V mutations in human PSEN1. Transgene expression is driven by the mouse neuron specific Thy1 promoter . Polymerase chain reaction (PCR) was done using the Bio-Rad T100 Thermal Cycler  with a 500 bp x 40 cycle schedule (94°C x 3 min followed by 40 x [94°C x 30 s] + 60°C x 30 s + 72°C x 30 s then 72°C x 2 min). The tubes were held at 10°C until use. The following reagents were used for each sample: 5 μl of 2x premix, 0.5 μl of retinal degeneration (RD) three oligonucleotide primer , 0.1 μl of RD4 oligonucleotide primer  and 2.9 μl of RD6 oligonucleotide primer . RD3 and RD4 amplify a 0.55 kb PCR product from the rd1Pdeb mutant allele, while RD3 and RD6 amplifiy a 0.40 kb PCR product from the WT allele , an inbred strain of mouse known to be homozygous for the rd1Pdeb mutation. This mouse was purchased from the Jackson Laboratory . The control for the WT allele of Pdeb for this gel was ear tissue obtained from a B6SJLF1/J mouse.The generation of the 5XFAD mice by Oakley and colleagues in 2006 has been described previously . Brieflypromoter . These fpromoter . 5xFAD mpromoter . 5xFAD mmosome 5 . Thus, Feriments . This saeriments . To confT allele . The PCRAPP/PS1 are double transgenic mice expressing a chimera of mouse/human APP (Mo/HuAPP695swe) and a mutant human presenilin 1 (PS1-dE9). Generation of this mouse line has been previously described . TransgeAll the behavioural tests were conducted in the automated Bussey-Saksida Mouse Touchscreen Systems model 81426  . Mice weDay 1 (Habituation 1), the mice were placed in the testing chambers for 10 min with the house lights off, with no stimuli displayed and no reward presented. On Days 2–4 (Habituation 2), the mice were placed into the testing chambers for 20 min (Days 2 and 3) and for 40 min (Day 4). On Days 2–4, the reward tray light was turned on and the reward  was presented and paired with a sound (tone) for 280 ms every 10 s. During this phase, the mice should strengthen the association between the reward tray light and tone with the reward; however, no performance criteria were in place at the habituation phase.Mice (10–12 weeks of age) experienced several pre-training stages (shaping) prior to probes in each task. For the first four days of pre-training, the mice were habituated to the testing chambers (Habituation schedules). On Following habituation, the mice were subjected to the ‘Initial Touch’ schedule (Phase I), which involves pairing the reward with the presentation of stimuli (random images for PAL and PD and a white square for 5-CSRTT) on the touchscreen. In this phase, a single stimulus appears randomly in one of the windows. After 30 s the stimulus is turned off and the illumination of the reward tray light is paired with a tone and delivery of the reward (7 μl of strawberry milkshake). If the mouse touches the screen during the time that the image is displayed, a reward is immediately presented with a tone. A new trial starts when the mouse collects the reward, and Initial Touch sessions are repeated daily until the subject completes 30 trials within 60 min.SOP2) and PAL (SOP3) and other touchscreen tasks can be found in the Touchscreen Cognition (www.touchscreencognition.org).The next stage of pre-training, ‘Must Touch’ (Phase II) involves displaying a stimulus randomly in one of the windows, as before. However, in this phase, differently from Phase I, the mouse is required to touch the stimulus on the screen in order to receive the reward paired with a tone. If the mouse touches any window other than the one in which the stimulus is present, it receives no reward. Daily sessions are repeated until the mouse completes 30 trials in 60 min. The next phase of shaping introduces the animals to the initiation procedure, a schedule called ‘Must Initiate’ (Phase III). At the beginning of each trial, the reward tray is illuminated, and the mouse is required to initiate the stimulus delivery by a nose poke into the reward tray. Successful initiation extinguishes the tray light, and a stimulus is presented in one of the windows on the screen. After touching the stimulus and collecting the reward, the mouse is subjected to a 5 s inter-trial interval  before the illumination of the reward tray light signals the beginning of the next trial. A criterion of 30 correct trials within 60 min must be met in order for the mouse to proceed to the next phase. The last pre-training phase (Phase IV) is called ‘Punish Incorrect’ and requires the mouse to both initiate and touch the stimulus but if an incorrect choice is made, it receives a 5 s timeout, during which the lights are turned on and no reward is delivered. The mice continue performing this phase of shaping until they are able to obtain at least 80% of trials correct within 60 min for two consecutive days. Intertrial intervals of 5 s for 5-CSRTT and 20 s for PD and PAL were used in all the phases. All the Standard Operant Procedures (SOPs) for 5-CSRTT , PD (SOP+ correct image and S- incorrect image) (S4A). If a correct response is made, reward is delivered, but if an incorrect response is made, a 5 s time-out is initiated with activation of the house light. Following the time-out period, mice are required to complete the same trial repeatedly until a correct response is made. These correction trials do not count towards the overall trial count. To move to the next phase a criterion of 24 correct out of 30 trials (2 days in a row) is required.After reaching the criteria on Punish Incorrect schedules, mice were trained on PD Acquisition sessions. In this task, mice must initiate the trials by poking their head into the reward tray after a light signal is displayed. Immediately after exiting the reward tray, two different images appear on the screen. Mice are required to learn that a rewarded response is determined by the correct visual image , mice were trained on the 5-CSRTT, which requires responses to brief flashes of light pseudo-randomly displayed in one of the five response windows on the touchscreen chamber, as described previously . All micOnce mice reached criterion on the 2 s variation of the 5-CSRTT, they were exposed to a series of probe sessions. During each probe session, one of four test stimulus durations was used: 1.5 s, 1.0 s, 0.8 s, and 0.6 s. Each mouse completed two consecutive days of probe trials with each of the stimulus durations. Following each probe session, mice were returned to the 2 s stimulus duration version for two consecutive baseline days before beginning the next probe session (two days with a different stimulus duration). The order of presentation for probe sessions was counterbalanced across all mice. In order to assess attentional performance longitudinally (which to the best of our knowledge has not been performed in AD mice), the same mice were tested on probe sessions at 4, 7, and 10–11 months of age. To ensure maintenance of sufficient baseline performance on the task between probe sessions, mice were given a single day of training with the 2 s stimulus duration each week until the next set of probe sessions. Prior to the start of the 7 month and 10–11 month probe sessions, mice were given 5 days of 2 s 5-CSRTT sessions in order to re-baseline them on the task. The same reduced stimulus durations  were used for probe sessions at each testing age, and the order of stimulus duration sessions was counterbalanced between mice at each age as well as for each mouse across each of the testing ages.After reaching the criteria on Punish Incorrect schedules (Phase IV of pre-training), mice were trained on PAL Acquisition sessions. In this task, mice must initiate the trials by poking their head into the reward tray when the light on the feeder turns on. Immediately after exiting the reward tray, two different images appear in two of three positions on the screen. Mice must learn that each specific visual image is associated with only one correct spatial location on the touchscreen, and only one image per trial is presented in the correct location. If a correct response is made a reward is delivered (7 μL of strawberry milkshake), but if an incorrect response is made, a 5 s time-out is implemented with activation of the house light. Following the time-out period, mice are required to complete the same trial repeatedly until a correct response is made. These correction trials were not counted towards the overall trial count. The criterion for this training phase is the completion of 36 trials in 60 min (per day). All the mice from both groups and all the time points were able to reach this criterion in the first session.different PAL (dPAL) or on same PAL (sPAL) tasks. 3xTG-AD and APP-PS1 were tested in dPAL at 4 and 11 months of age. Due to a poor performance in dPAL at 4 months of age, 5xFAD mice (10–11 months of age) were tested in sPAL, which is relatively easier test when compared to dPAL. In both tasks, a mouse initiates the task by touching the reward tray, which triggers the display of both S+ and S- on the screen. As described above, S+ refers to the stimulus presented in the correct location and S- refers to a stimulus presented in the incorrect location. In this task, the mice are required to learn to associate a stimulus with its correct location. Similar to the PAL Acquisition phase, if a correct response is made (S+), the reward is delivered. However, touching the S- stimulus results in a 10 s time-out and illumination of the light in the chamber (10 s). Following the punishment period, mice are required to initiate the same trial repeatedly until a correct response is made; these correction trials did not count towards the overall trial count. The only difference between dPAL and sPAL is the fact that on dPAL S+ and S- stimuli are different images and on sPAL S+ and S- are the same image  once a week until the commencement of the second set of probe trials that started at 7 months of age. The images used for the training and maintenance were removed from the probe trial database and were not displayed to the mice during the probes for PAL or PD tasks. This was done to ensure that the mice did not forget the basic aspects of touchscreen task performance and hence would not require retraining prior to the subsequent 7- and 11 month trials. Upon completion of the second set of trials, mice were put back on a weekly maintenance schedule until the commencement of the third set of trials at 11 months of age.For PD and PAL, different sets of images were used at each testing age. All the images were randomly selected from the ABET II imaging databank and, according to the manufacturer, the images present the same number of pixels. Image sets I, II and III were used for testing mice on PD at 4, 7 and 10–11 months of age, respectively (S4). Image sets IV, V and VI were used to test mice on dPAL or sPAL at 4 and 10–11 months of age (S4). It is well known that rodents prefer some stimuli over others . To evalMice were anaesthetized using ketamine (100 mg/kg) and xylazine (25 mg/kg) in 0.9% sodium chloride solution, and then transcardially perfused with 1x phosphate buffered saline  for 5 min. For each mouse, one harvested hemibrain was post-fixed in 4% paraformeldahyde overnight and subsequently used for immunostaining, while the other harvested hemibrain was stored at −80°C for biochemical analyses.The hemibrains for immunohistochemistry were cryopreserved using increasing concentrations of sucrose , embedded in optimal cutting temperature (OCT) compound and frozen at −80°C. Sagittal sections (10 μm) were cut using the Cryostat (Leica Biosystems), directly mounted and frozen. All slides were immersed in 70% ethanol for 1 min followed by distilled water for another minute before they are stained as described below.Slides were stained with filtered 1.25% Thioflavin-S solution in 50% ethanol for 8 min at room temperature. Slides were then washed twice with 80% ethanol, once with 95% ethanol, and then three times with distilled water before mounting.AB_2564652) diluted 1:200 in TBS 1x. Following two TBS 1x washes, slides were incubated at 4°C in 488 goat-anti-mouse secondary antibody (RRID:AB_2564652) diluted 1:1000 in TBS 1x, 1% HS and 1% BSA. Nuclei were stained with To-Pro-3-Iodide  diluted in PBS1x (1:1000) for 15 min. Slides were then rinsed three times with TBS 1x and mounted.Slides were washed twice with Tris Buffered Saline (TBS) 1x and then permeabilised with 1% Triton X-100 (tx) in 1x TBS for 15 min. Non-specific binding was prevented by incubating the slides for one hour in 2% horse serum (HS) and 2% bovine serum albumin (BSA) in TBS 1x with 0.3% Triton-x. Slides were then stained overnight at 4°C with 6E10 primary antibody (RRID:(1-42) .For Aβ (1-42) quantification we used food-restricted and non-food restricted 3xTG-AD and 5xFAD  by six months of age. Hippocampus fractionation was performed as described previously . ELISA wMounted slices were visualised by confocal microscopy using Leica-TSC SP8 or SP5  . Images were analysed using ImageJ . For each mouse, the cortex and hippocampus  of 3–4 slices were imaged and quantified in terms of percentage area. The experimenter was blind to genotype during image acquisition and quantification.touchscreencognition.org. Monthly conference calls were made, and reports were exchanged between the researchers to assure the maximum standardisation of the procedures across the sites. After automated collection, the data were automatically saved and backed-up onto two different servers at The University of Western Ontario and the University of Guelph. For data protection, the database and back-up were strictly controlled and logged. ABETT II files were converted to XML files (each XML file corresponds to the data from a unique mouse ID session/day of training or testing) and uploaded into mousebytes.ca. The XML files were then automatically checked by automated quality control (QC) algorithm and the codes are available for free download and modification in GitHub   were standardised between the two sites before the start of experimentation and the SOPs can be found in Control)  Files wiControl) .mousebytes.ca/spotfire and mousebytes.ca/tutorial.The processed data were transferred to the open-access application . The comAccuracy – percentage of correct trials; Omissions – percentage of trials on which no response is made; Correct Response Latency – reaction time for correct response; Reward Collection Latency – reaction time to collect the reward on correct trials; Premature responses – number of responses made prior to the stimulus presentation; Perseverations – number repeated responses at a previously rewarded window before onset of the next trial. Omissions and Premature responses did not count towards Accuracy. Analysis of the probe trial data was conducted using a 3 (age) x 4 (stimulus duration) x 2 (research site) x 2 (genotype) x 2 (sex) split-plot ANOVA. In addition, vigilance in the 5-CSRTT was analysed with a 4 (stimulus duration) x 5 (block) x 2 (genotype) split-plot ANOVA. To characterise genotype-specific effects, we a priori decided to conduct 4 (stimulus duration) x 2 (genotype) split-plot ANOVA analysis between wildtype and transgenic mice for each sex, strain, and age.In all pre-training stages for the 5-CSRTT task (phases I-IV), the number of sessions to reach criterion were analysed to determine any differences in learning of the task. A similar analysis was performed for both 4 s and 2 s stages of 5-CSRTT training. For the probe sessions, several parameters were analysed as an average between each set of two probe sessions with each stimulus duration: Accuracy – percentage of correct trials; Correction trials – number of trials until a correct choice is made. Correct Response Latency – reaction times for correct response; Reward Collection Latency – reaction time to collect the reward on correct trials. To analyse PD reversal data, a 10 (session) x 2 (genotype) x 2 (sex) split-plot ANOVA was conducted. We decided a priori to investigate genotype-specific effects within each mouse line, so we additionally conducted 10 (session) x 2 (genotype) split-plot ANOVA within each sex, strain, and age.For the pre-training stages for the PD task (Phases I-IV) and acquisition sessions, the number of sessions to reach criterion were analysed to determine any differences in task learning. In addition, several behavioural parameters were analysed for the PD and reversal phases: Accuracy – percentage of correct trials; Correction trials – number of trials until a correct choice is made. Correct Response Latency – reaction times for correct response; Reward Collection Latency – reaction response to collect the reward on correct trials. Data from the 45 sessions (days) of dPAL/sPAL were binned in intervals of 5 sessions or one week of testing . Data for the PAL training was analysed with a 9 (bin) x 2 (sex) x 2 (genotype) split-plot ANOVA. All data were analysed separately at 4 months and 11 months of age and for males and females. To characterise genotype effects specifically, we a priori decided to conduct an additional 9 (bin) x 2 (genotype) ANOVA models separating by sex, strain, and age.In the pre-training stages for the PAL task, the number of sessions to reach criterion were analyzed to determine any differences in basic touchscreen acquisition. In addition, several behavioural parameters were analysed for the dPAL/sPAL and retention phases: In order to categorise the overall performance of the mice over the duration of the tasks we employed a k-means clustering approach. Given the longitudinal nature of the behavioural tasks, repeated performances across multiple trials, a three-dimensional version of the analysis is required. To do so, we used the kml3d R package . This alFor all immunofluoresecence studies, a total of 8 B6SJLF/1  and 8 5xFAD  mice were used. Immunofluorescent analysis was conducted on four slices from each brain sample. Comparison of 6E10 and thioflavin-S expression between genotypes was conducted using two-tailed independent samples t-tests.A total of 6 B6129SF2/J , 8 B6SJLF/1 , 6 3xTG-AD , and 8 5xFAD  mice were used for the ELISA analysis of Aβ. In order to quantify differences between wildtype and transgenic mouse Aβ expression, two-tailed independent samples t-tests were used.https://mousebytes.ca data repository. The code used for analysis of these data can be found in https://github.com/srmemar/Mousebytes-QualityControl  data is becoming increasingly important.Decision letter after peer review:eLife. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by a Reviewing Editor and Huda Zoghbi as the Senior Editor. The following individuals involved in review of your submission have agreed to reveal their identity: John G Howland (Reviewer #2); Yogita Chudasama (Reviewer #3).Thank you for submitting your article "An open-access high-throughput pipeline and database for rodent touchscreen-based cognitive assessment" for consideration by The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.Summary and Essential revisions:All three reviewers expressed enthusiasm for efforts to standardize and share data generated by this increasingly popular behavioral testing paradigm. The major concerns raised at this stage for the authors to address relate to: 1) the ease of 'adoptability' of the analysis across laboratories that are not working in collaboration with the authors, and 2) the strength of the narrative justifying the collation of the analysis methods with the example empirical data in this one paper. In my opinion, these issues should be addressable in a compelling rebuttal to the reviewers and a carefully revised version of the manuscript.Reviewer #1:With more and more researchers adopting touchscreen systems for high-level cognitive assessment, there is certainly a need for a well maintained repository to allow for sharing of published datasets, large meta-analysis, and importantly, but not mentioned, the dissemination of null datasets. Co-authors and institutions involved in the current effort are leaders in the field of using this technology and these tools have the potential to lead to advancements in the field. However, the structure of the manuscript, and in particular the lack of mention of major difficulties in comparing datasets across labs, make the current results unclear, and the tools, such as the K-mean classification approach, difficult to assess.- The authors repeatedly stress the strength of the touchscreen systems is the standardization of outcomes. However, even within the 3 tasks the authors use here, there are an incredibly amount of variables, including stimulus choices, inter-trial intervals, trials and sessions per day, reward type and amount, stimulus durations for 5-CSRTT etc. that can vary across experiments and labs. Much more discussion and description needs to be given to how these factors can be controlled for, or at least noted in order for Mousebytes users to make well-designed comparisons across datasets.- The current study stresses the high-degree of collaboration and communication across sites including pre-standardization, use of the same systems and software, monthly conference calls and collaboration on QC. This can't possibly be expected to happen with an open access database. How will the system account for data collected on behavioral systems from other providers, or open source built systems? Gurley, J. Exp Analysis of Beh 2019; Butler and Kennerly, Behav. Res. Methods 2018 A discussion of how the system can or cannot deal with files using different category labels, different orders of variables, etc. needs to be added.- The authors state both in the Introduction and Materials and methods that during QC "files" with potential errors are flagged by the procedure, fixed manually, or discarded and not used for analysis if not fixable. What is a "file"? Is this a daily data file for an individual animal? If so, how does the software account for the "hole" in the data? Are those data points simply ignored, wouldn't this decrease the total errors, alter the percent correct for that animal?- While the authors correctly note the huge number of graphs and figures that can be generated from this dataset, the data included in Figure 2 seem arbitrary. Site comparison is included for 3-6 M/Males but not 11-13. Strain is shown for males and not females. Given only 4 figures are included in the main document, there is room to add these panels and aid the reader in the logical assessment of the variables.- Application of unbiased assessment via machine learning is exciting and potentially important, but the approach is not clear. If the authors wish to unbias the approach, why were sex and age specifically conducted separately? Does this analysis require the assumption of independent samples? Inclusion of each animal as a separate data point at each age would violate this.- The data presented in Figure 3 (or 4?) do not seem to support the authors' conclusions in the text. The statement "5xFAD and 3xTG-AD transgenic mice consistently are lower performers than their WT counterparts in the 5-CSRTT" is confusing, Based on the figure, it seems that WT 3xFAD males and females are consistently worse at most ages. APP/PS1 results are not discussed, although females show almost no differences (except at 7 months) and in males, WT seem to be consistently worse than TG.- Follow up motor testing on the 5xFAD males is potentially interesting but detracts from the main point of the manuscript. It is unconvincing that increased reward collection latency "uncover(ed) unexpected phenotypes in mouse models" as 1) general phenotyping of mouse models for characteristics  that would alter touchscreen results should be done prior to testing and 2) this phenotype was previously reported as the authors note.- Similarly, the inclusion of the lack of effect of AB burden based on ad lib vs. food restriction is potentially interesting, but here just adds more density to an already overpacked manuscript- The longitudinal analysis is potentially powerful, but also raises questions. Some discussion of how previous training might alter later performance needs to be included. There is a strong case to be made that touchscreen testing could be considered enrichment. How does previous experience alter later cognitive performance in these models versus standard housed non-tested mice?Reviewer #2:Beraldo and colleagues present a novel open source analysis pipeline for data coming from a suite of tasks used in touchscreen-equipped operant conditioning chambers. They use this pipeline to analyze the behaviour of 3 mouse lines used to study Alzheimer's disease. The pipeline and on-line interface provide an exciting opportunity for combining research from different laboratories to increase understanding of cognition using touchscreen-equipped operant conditioning chambers. Results from the mice tested in the present experiments show results related to the reliability of touchscreen-based assessments, the effects of genetic background, sex, and age on cognition, and also provide an example of using K-mean clustering to analyze behaviour of hundreds of mice in the 3 tasks. Overall, the study is important, exciting, and will provide a useful resource for behavioural neuroscientists studying cognition in rodents. Below are a series of queries for the attention of the authors.The title of the paper indicates that the pipeline will be used to study "rodent touchscreen-based cognitive assessment". However, the manuscript deals exclusively with mice. The absence of discussion of rats (or other species) appears to be a missed opportunity in this regard. Will the MouseBytes site support datasets from researchers studying rats, or other species?https://labrigger.com/blog/2018/03/09/custom-touchscreen-behavior-systems-for-rodents/). Will the data coming from these systems be compatible with the MouseBytes pipeline?The Introduction of the manuscript provides a compelling case for the importance of open science and the researchers should be lauded for pushing forward with this pipeline related to cognitive assessment. However, a more detailed discussion of the hardware used to collect the data should be provided. As the hardware used at both sites was the Bussey-Saksida Mouse Systems, it would of considerable interest for other open source hardware options were discussed  the introduction of MouseBytes; and 2) the results of assessing the AD-related mice. I found that the specifics of the findings related to the AD-mice were lost in places, particularly the summary and end of the Introduction. How do the authors' experimental results advance our understanding of the 3 mouse models and what specific future research questions can/should be answered with them?Reviewer #3:This paper is essentially a methods paper. The authors have taken a bold step in putting together a much warranted open-access database that enables transparency, collaboration and potential likelihood of reproducibility. The paper is well written with clear detail of the database repository, mousebytes.ca.I have only two comments which are essentially suggestions that I think might help make the paper stronger.1) The question is how far can mice carry Alzheimer Disease research or any other disease related research, and is data sharing and transparency the answer to translation? The growing disillusionment with murine models of human disease especially in drug discovery, have questioned mouse models constructed with highly penetrant alleles of human disease because of the large number of compounds which apparently 'cure' the mouse but not the human. This discrepancy is repeatedly argued to be due to failure in data/experimental reproducibility causing journals to emphasize more statistical details as if statistics will provide greater translation. Seems to me that the mousebyte.ca database is one approach that could feasibly enable a better mouse to human translation or at least work in this direction. The truth of the matter is that it is virtually impossible to replicate data even in the same lab, let alone between labs for a multitude of reasons. Can mousebytes.ca facilitate translation?2) At the moment, the paper focuses on three mouse lines thought to model Alzheimers disease (AD) to highlight the strength of the database. The database is also specific to data acquired using the operant touchscreen platform because non-automated methods, the authors argue, are subject to large variations. Are their plans to expand the database to a much larger cohort of mice that are not necessarily tested for cognitive behaviors tested on a touchscreen platform? The large array of autistic mouse models come to mind for which social communication and interaction, motor, cognitive and emotional behaviors are all variable leaving a very confused state in the field. Summary and Essential revisions:All three reviewers expressed enthusiasm for efforts to standardize and share data generated by this increasingly popular behavioral testing paradigm. The major concerns raised at this stage for the authors to address relate to: 1) the ease of 'adoptability' of the analysis across laboratories that are not working in collaboration with the authors, and 2) the strength of the narrative justifying the collation of the analysis methods with the example empirical data in this one paper. In my opinion, these issues should be addressable in a compelling rebuttal to the reviewers and a carefully revised version of the manuscript.We thank the editors for summarizing the main concerns and the reviewers for the positive evaluation. We have addressed these two concerns below in the response to reviewers.Regarding point 1, the intrinsic features of MouseBytes, and particularly those now added in response to the reviewers’ comments, are specifically designed to promote ease of use for all touchscreen users, especially those with whom we do not collaborate directly. MouseBytes allows users to enter Metadata that facilitates comparison and combination of data from different sites and also different touchscreen apparatuses (see reply to reviewers’ comments below). It is also important to emphasize that MouseBytes is by no means immutable; it will continue to evolve based on user’s feedback.Thanks to the feedback provided by the reviewers, we made substantial changes to the manuscript, and also to the software. We now can collect even more Metadata on, e.g., experimental conditions. We have also developed codes to easily convert output files from any touchscreen apparatus to the MouseBytes format.Regarding point 2, we improved the narrative concerning the empirical data, and in particular made it clearer how the data illustrate the advantages of MouseBytes. This is now explained more clearly in the Abstract and Introduction.Reviewer #1:With more and more researchers adopting touchscreen systems for high-level cognitive assessment, there is certainly a need for a well maintained repository to allow for sharing of published datasets, large meta-analysis, and importantly, but not mentioned, the dissemination of null datasets. Co-authors and institutions involved in the current effort are leaders in the field of using this technology and these tools have the potential to lead to advancements in the field. However, the structure of the manuscript, and in particular the lack of mention of major difficulties in comparing datasets across labs, make the current results unclear, and the tools, such as the K-mean classification approach, difficult to assess.We thank the reviewer for raising these important points. We have now mentioned throughout the manuscript the potential difficulties in comparing datasets. In particular, we now emphasize collection of Metadata by MouseBytes as a way to greatly facilitate comparison of datasets. Users depositing data can now specify in detail all conditions of the experiments, and further are able to link datasets to published manuscripts which yield further details. Examples include added fields in the repository to account for potential variations in conditions between different laboratories such as light-dark cycle, and group v single housing. The greater availability of these details will increase the power of meta-analysis using MouseBytes.We have also improved the discussion and explanation about the k-mean analysis. We clearly state now in the manuscript both in the Introduction and Discussion the utility for depositing null-datasets.- The authors repeatedly stress the strength of the touchscreen systems is the standardization of outcomes. However, even within the 3 tasks the authors use here, there are an incredibly amount of variables, including stimulus choices, inter-trial intervals, trials and sessions per day, reward type and amount, stimulus durations for 5-CSRTT etc. that can vary across experiments and labs. Much more discussion and description needs to be given to how these factors can be controlled for, or at least noted in order for Mousebytes users to make well-designed comparisons across datasets.The reviewer is right regarding the potential for many variables in any task to be changed using touchscreens. At the moment, major variations in these parameters are identified by the QC codes, and therefore experiments that deviate from standard operating parameters are flagged and can only be uploaded after communication between the user and administrator (subsections “Open-Access Database and repository” and “Touchscreen operant platform” describe these procedures). When experiments deviate from standard parameters, users are requested to provide information in the experiment field and also link the datasets to the published manuscript or preprint providing further details regarding the experimental conditions. Datasets that differ from standard protocols will still be uploadable in MouseBytes and can then be identified and curated for meta-analysis by using the information provided by the experimenter and the QC codes can be also modified by users. This is mentioned in the Results and Discussion and Materials and methods .- The current study stresses the high-degree of collaboration and communication across sites including pre-standardization, use of the same systems and software, monthly conference calls and collaboration on QC. This can't possibly be expected to happen with an open access database. How will the system account for data collected on behavioral systems from other providers, or open source built systems? Gurley, J. Exp Analysis of Beh 2019; Butler and Kennerly, Behav. Res. Methods 2018 A discussion of how the system can or can not deal with files using different category labels, different orders of variables, etc. needs to be added.https://github.com/srmemar/Mousebytes-An-open-access-high-throughput-pipeline-and-database-for-rodent-touchscreen-based-data/blob/master/XML_Output.ipynb). This is now described in the Results and Discussion.We would like to thank the reviewer for this suggestion. It has always been our intention to allow Mousebytes to incorporate data not only from Bussey-Saksida touchscreens systems, but from any variant of touchscreen operant chambers as long the output data can be made to match the Mousebytes specifications. We have now developed codes to do just that, and have added this MouseBytes feature to the manuscript. These codes can be downloaded and modified by different users to convert XML files from any touchscreen system output to files that can be readable by MouseBytes  from a unique mouse ID. We have now addressed this point in the subsections “Open-Access Database and repository” and “Touchscreen operant platform”. If a file is flagged and cannot be fixed for some reason, these data are not transferred to MouseBytes and will not be part of the downloadable dataset or used for visualization. In the specific datasets already in MouseBytes, it is unlikely that the discarded files would affect the final results reported in the manuscript. For example, for the 5-choice data less than 0.6% of files were terminally flagged and not used to compose the datasets, and only 0.09% were from probe trials . Hence, the system will not use these data, but it should not have affected the final analysis in the manuscript.- While the authors correctly note the huge number of graphs and figures that can be generated from this dataset, the data included in Figure 2 seem arbitrary. Site comparison is included for 3-6 M/Males but not 11-13. Strain is shown for males and not females. Given only 4 figures are included in the main document, there is room to add these panels and aid the reader in the logical assessment of the variables.We thank the reviewer for this suggestion. We now provide several new panels comparing the performance of mice longitudinally, between sites, sex and strains.  to the current version of the manuscript which is now described in the Results and Discussion- Application of unbiased assessment via machine learning is exciting and potentially important, but the approach is not clear. If the authors wish to unbias the approach, why were sex and age specifically conducted separately? Does this analysis require the assumption of independent samples? Inclusion of each animal as a separate data point at each age would violate this.In the k-mean clustering approach that we present in the manuscript, we did not separate sex in the primary clustering algorithm to define Low, Mid and High performers. However, following the k-mean analysis, we determined if group memberships were significantly different by utilizing Fisher’s test. In this secondary analysis, we did separate the data by age and sex. We did this to prioritize looking at group membership and genotype in the Fisher’s Exact Test. In order to address the multiple tests created by separating age and sex, we used a False Discovery Rate correction analyses.In the k-mean analysis, we assumed that performance could change with age, so age was treated as an independent between factor, where each animal had two or three cluster points for each of the ages of assessment. The k-mean clustering algorithm does not require independence for the initial clustering. The Fisher’s exact test we conducted had independent data points as part of our statistical analysis.The decision to treat animals at different ages as separate data points was done in order to allow for mice to switch k-mean categories, based on the assumption that performance would change with age. In a given experiment, wildtype mice may naturally have reductions in performance on cognitive tasks related to age. We were interested in detecting if transgenic mice would shift from a category of “high performers” into a category of “low performers” or vice-versa as they were assessed at later time points. More specifically, we were interested in if this shift from high to low performers was larger for transgenic mice than wildtype mice.Future efforts with the MouseBytes data system will certainly take advantage of other unbiased classification systems  with larger datasets to begin to better determine how to classify behavioural data from mouse models.- The data presented in Figure 3 (or 4?) do not seem to support the authors' conclusions in the text. The statement "5xFAD and 3xTG-AD transgenic mice consistently are lower performers than their WT counterparts in the 5-CSRTT" is confusing, Based on the figure, it seems that WT 3xFAD males and females are consistently worse at most ages. APP/PS1 results are not discussed, although females show almost no differences (except at 7 months) and in males, WT seem to be consistently worse than TG.These data are presented now in Figure 5. Analysis of the dataset shows that indeed, the 3xTG AD and 5xFAD AD mice are shifted to the lower performance group in the 5-Choice Task. Otherwise, their WT counterpart mice are shifted to the higher and moderate performers. For the APP/PS1 mice we did not observed any difference in the performance of females (comparing WT with the TG) in 5 choice. However, interestingly, APP/PS1 male mice shifted their performance to the high and moderate performers while WT shifted to lower performers in the 5-Choice Task. The presentation and discussion of this information are in the Results and Discussion.- Follow up motor testing on the 5xFAD males is potentially interesting but detracts from the main point of the manuscript. It is unconvincing that increased reward collection latency "uncover(ed) unexpected phenotypes in mouse models" as 1) general phenotyping of mouse models for characteristics  that would alter touchscreen results should be done prior to testing and 2) this phenotype was previously reported as the authors note.We agree with the reviewer and accordingly removed the dataset from the manuscript.- Similarly, the inclusion of the lack of effect of AB burden based on ad lib vs. food restriction is potentially interesting, but here just adds more density to an already overpacked manuscriptWe thank the reviewer for the suggestion. However, given that environmental enrichment and decreased food consumption could affect pathology in mouse models, we find these results important for readers to understand that lack of change in cognitive domains is not related to changes in pathology. We have decreased as much as possible the discussion on these issues and only provide information for readers to get the main message.- The longitudinal analysis is potentially powerful, but also raises questions. Some discussion of how previous training might alter later performance needs to be included. There is a strong case to be made that touchscreen testing could be considered enrichment. How does previous experience alter later cognitive performance in these models versus standard housed non-tested mice?This is an important point raised by the reviewer. We elaborated on this point in the manuscript (Results and Discussion), but the present work did not directly address this question.However, one possible source of insight into this question might be in the data from mice performing the 5-Choice serial reaction time task. In our manuscript, we present data from the same mice tested at 3 ages, with final timepoint at 11-13 months, when mice would have been probed twice in this task. Romberg and colleagues, 2011, also tested a 3xTG-AD mice previously with the 5-Choice task at 10 months of age, however, these mice were naïve when tested at this age. The data and results between these two experiments are very similar, suggesting that prior cognitive training may not have a dramatic effect on subsequent touchscreen behavior in the 5-Choice test. Moreover, comparison of performance of mice longitudinally in the 5-Choice task suggests that accuracy does not change with age, although omission is improved after repeated training .We have also analyzed the performance in pairwise visual discrimination and reversal with distinct images in different ages and found that depending on the image mouse performance varied, but previous training did not seem to affect the performance. This is now discussed in the Results and Discussion. We anticipate that with increased datasets being deposited, this important question will be addressed in the future for multiple tasks.Reviewer #2:[…] Below are a series of queries for the attention of the authors.The title of the paper indicates that the pipeline will be used to study "rodent touchscreen-based cognitive assessment". However, the manuscript deals exclusively with mice. The absence of discussion of rats (or other species) appears to be a missed opportunity in this regard. Will the MouseBytes site support datasets from researchers studying rats, or other species?https://labrigger.com/blog/2018/03/09/custom-touchscreen-behavior-systems-for-rodents/). Will the data coming from these systems be compatible with the MouseBytes pipeline?The Introduction of the manuscript provides a compelling case for the importance of open science and the researchers should be lauded for pushing forward with this pipeline related to cognitive assessment. However, a more detailed discussion of the hardware used to collect the data should be provided. As the hardware used at both sites was the Bussey-Saksida Mouse Systems, it would of considerable interest for other open source hardware options were discussed . These codes can be easily adapted by users with data obtained in other touchscreen systems. This is now mentioned in the Results and Discussion.As described in response to reviewer 1, we have created a way for Mousebytes to incorporate data not only from Bussey-Saksida touchscreens systems, but from any variant of touchscreen operant chambers. To do that, we have developed codes that can be downloaded and modified by different users to convert XML files from any touchscreen system output to files that can be readable by MouseBytes (There are two important components to this study: 1) the Introduction of MouseBytes; and 2) the results of assessing the AD-related mice. I found that the specifics of the findings related to the AD-mice were lost in places, particularly the summary and end of the Introduction. How do the authors' experimental results advance our understanding of the 3 mouse models and what specific future research questions can/should be answered with them?These issues have now been addressed in the Introduction and the Discussion. We identify common age-dependent attentional deficits in 3xTG-AD and 5xFAD mice. We also found that behavioral flexibility did not seem to be consistently affected in most mouse lines tested. This dataset provides the foundation to test new drugs to correct attentional deficits, which were consistently reproduced in 2 out of 3 mouse lines.Reviewer #3:[…] I have only two comments, which are essentially suggestions that I think might help make the paper stronger.1) The question is how far can mice carry Alzheimer Disease research or any other disease related research, and is data sharing and transparency the answer to translation? The growing disillusionment with murine models of human disease especially in drug discovery, have questioned mouse models constructed with highly penetrant alleles of human disease because of the large number of compounds which apparently 'cure' the mouse but not the human. This discrepancy is repeatedly argued to be due to failure in data/experimental reproducibility causing journals to emphasize more statistical details as if statistics will provide greater translation. Seems to me that the mousebyte.ca database is one approach that could feasibly enable a better mouse to human translation or at least work in this direction. The truth of the matter is that it is virtually impossible to replicate data even in the same lab, let alone between labs for a multitude of reasons. Can mousebytes.ca facilitate translation?The reviewer summarizes one of our motivations to create a repository for touchscreen datasets. Databases such as MouseBytes can provide more transparency, to allow others to analyze datasets and confirm experiments. We hope that our database can serve as an interface for researchers interested in studying replication and reproducibility.Additionally, we are hoping to provide a platform for researchers to identify critical features in the lab environment that may contribute to behavioural variation. We have recently begun to address this by incorporation of new meta-data into the MouseBytes experiment records for factors such as colony light cycle, visual stimuli selection, and type of animal housing. We plan on expanding our meta-data system to include several more factors, but these can also be informed in manuscripts that are linked to datasets via their DOI. Eventually, with enough labs and experiments in the system, we can begin to systematically identify which factors may be contributing to experimental variation, and address the issues of translation and reproducibility.2) At the moment, the paper focuses on three mouse lines thought to model Alzheimers disease (AD) to highlight the strength of the database. The database is also specific to data acquired using the operant touchscreen platform because non-automated methods, the authors argue, are subject to large variations. Are their plans to expand the database to a much larger cohort of mice that are not necessarily tested for cognitive behaviors tested on a touchscreen platform? The large array of autistic mouse models come to mind for which social communication and interaction, motor, cognitive and emotional behaviors are all variable leaving a very confused state in the field.: We appreciate the desire and need to expand open platforms of behavioural sharing to non-touchscreen behaviour. At this time, we are not considering expanding our system to non-touchscreen behavior, because most of these conventional methods are not automated and have no standardized outputs. We have chosen to focus on touchscreen behaviour with the MouseBytes data, due to the inherent benefits of these systems for replication, experimental control, and translational validity. We are however interested in expanding the touchscreen repertoire to include tasks that are more emotional/social in nature and could probe different types of deficits in other mouse models. Indeed, we are aware of other researchers who are using videos in touchscreens to assess murine social behaviour. We also anticipate the contribution of datasets by the community of researchers using touchscreens using mouse models of developmental disorders.Our view would be that although touchscreens are ideally suited for databasing applications, the hope is that other behaviours will eventually join the open science/datasharing movement. We think that MouseBytes could help to lead the way.


Deploying accurate computable phenotypes in pragmatic trials requires a trade-off between precise and clinically sensical variable selection. In particular, evaluating the medical encounter to assess a pattern leading to clinically significant impairment or distress indicative of disease is a difficult modeling challenge for the emergency department.This study aimed to derive and validate an electronic health record–based computable phenotype to identify emergency department patients with opioid use disorder using physician chart review as a reference standard.A two-algorithm computable phenotype was developed and evaluated using structured clinical data across 13 emergency departments in two large health care systems. Algorithm 1 combined clinician and billing codes. Algorithm 2 used chief complaint structured data suggestive of opioid use disorder. To evaluate the algorithms in both internal and external validation phases, two emergency medicine physicians, with a third acting as adjudicator, reviewed a pragmatic sample of 231 charts: 125 internal validation (75 positive and 50 negative), 106 external validation (56 positive and 50 negative).Cohen kappa, measuring agreement between reviewers, for the internal and external validation cohorts was 0.95 and 0.93, respectively. In the internal validation phase, Algorithm 1 had a positive predictive value (PPV) of 0.96 (95% CI 0.863-0.995) and a negative predictive value (NPV) of 0.98 (95% CI 0.893-0.999), and Algorithm 2 had a PPV of 0.8 (95% CI 0.593-0.932) and an NPV of 1.0 (one-sided 97.5% CI 0.863-1). In the external validation phase, the phenotype had a PPV of 0.95 (95% CI 0.851-0.989) and an NPV of 0.92 (95% CI 0.807-0.978).This phenotype detected emergency department patients with opioid use disorder with high predictive values and reliability. Its algorithms were transportable across health care systems and have potential value for both clinical and research purposes. EHRs are system . Yet at e system . Integrae system -6. An EHpository ,7. Phenopository . They arpository .An estimated 2.1 million people in the United States have opioid use disorder (OUD) , and ovePhenotyping could be used as a clinical tool to identify patients likely to benefit from ED-initiated buprenorphine or other interventions and as a research tool to identify patients who should be included in large-scale intervention studies of OUD interventions. We will conduct a multi-system pragmatic trial of user-centered clinical decision support to implement EMergency department-initiated BuprenorphinE for opioid use Disorder (EMBED) across 20 EDs in 5 health care systems . EHR pheThis phenotype was created for the purposes of identifying patients with OUD who could benefit from ED-initiation of buprenorphine in a subsequent trial or quality improvement initiatives. Therefore, the phenotype only included ED patients who were discharged from the hospital , were not currently prescribed buprenorphine, methadone, or naltrexone as medication treatment for OUD, and were not pregnant . This study was performed within the XXXX Health System in YYYY and XXXX Health System in YYYY by identifying a cohort of adults (>18 years of age) with ED encounters between November 1, 2017, and October 31, 2018, in the EHR. The 2 health care systems use different billing companies, but the same EHR vendor . Data were extracted from the EHR of each hospital using local Epic Clarity databases . These data comprised information available within the EHR on the date of service of the ED visit in question. Approval for this study was provided by the Institutional Review Boards of the respective institutions .Although psychiatric evaluation is the gold standard for diagnosing OUD, within the emergency medicine (EM) context, diagnosis if performed is based on the Diagnostic and Statistical Manual of Mental Disorders, 5th Edition (DSM-5) criteria . The DSMThe computable phenotype algorithm was developed based on data elements from available primary care OUD phenotypes ,23 with heroin, opiate, opioid, or narcan were included in their chief complaint for the ED visit. However, as naloxone is often used in ED patients with undifferentiated altered mental status or overdose, patients with narcan in their chief complain who did not have an OUD-related final diagnosis were excluded. Upon preliminary testing of the algorithm, the 2 most frequent false-positive diagnoses were alcohol- and benzodiazepine-related. Visits with these chief complaints but alcohol- and benzodiazepine-related final diagnoses were removed by excluding patients with the words alcohol or benzodiazepine in their final ED diagnosis.Algorithm 2 identifies patients who have not been captured by Algorithm 1 but have information in their ED chief complaint suggestive of OUD. This algorithm flagged patients if the words After the 2 algorithms were reviewed, finalized, and approved by the investigative team, individual elements of each algorithm were converted into structured query language (SQL). The computable phenotype algorithm was written to be deployed in the Epic EHR systems across both the health systems. Data structures within Epic were mapped to each of the concepts  by a clinical informatics expert (HP). This eliminated the necessity of translating concepts to local codes within hospitals. Sample queries were run, and HP verified charts for accuracy. The SQL query and data dictionary . Charts were selected at random from the cohort with ED visits from April 10, 2018, to August 1, 2018, across the health systems and reviewed during August 2018 to October 2018 for the internal validation phase and December 2018 to January 2019 for the external validation phase. As the chart reviewers were given access to the patient’s full chart, the time window for the charts was deliberately narrow to avoid postvisit information  confounding the accuracy of the chart review of the ED visit.Following the implementation of the computable phenotype algorithm, internal validation was performed using a sample of 125 charts retrieved from the XXXX health system EHR by a clinical informaticist (HP). A total of 75 charts were intended to be representative of the resulting OUD phenotypes with 50 of these charts meeting Algorithm 1 criteria and the other 25 meeting Algorithm 2 criteria. The other 50 charts were The external validation cohort was constructed by a clinical informaticist (WKR) with 20,000 randomly sampled ED visits occurring between November 1, 2017, and October 31, 2018 across the XXXXX health system. We picked this number of charts given the rate of phenotype-positive charts in the internal validation cohort with a goal of estimating sensitivity of the phenotype based on prevalence in this random sample. A total of 55 charts met Algorithm 1 criteria. Of those not positive for Algorithm 1, 1 chart met Algorithm 2 criteria. Of the remaining negative cases, a 0.25% (50/200) random sample produced 50 charts for review. Cases positive for Algorithm 1 or 2 were combined owing to the low yield of a single chart identified as Algorithm 2 positive.OUD-positive or OUD-negative based upon a review of EHR data available up to and on the date of the ED visit (but not after the ED visit), their clinical judgment, and the DSM-5 OUD diagnostic criteria which were presented to them with each case at the time of review [OUD-positive, reviewers were then prompted to select at least 2 of the 11 DSM-5 criteria that informed their diagnosis.Each chart was reviewed independently and separately by 2 EM physicians  blinded to the results and the algorithms and the decision of the other reviewer. All cases of disagreement were adjudicated by a third EM physician reviewer  also blinded to the results of the algorithms and the decision of the other reviewers. Reviewers were asked to diagnose patients as f review . For casPhenotype performance was assessed using descriptive statistics. A standard 2×2 confusion matrix  was confAmong ED visits resulting in discharge from November 1, 2017, to October 31, 2018, across the 13 EDs in the 2 health care systems, a total of 474,176 unique ED visits (discharged patients only) with an average of 36,475 ED visits per year per site were identified. A total of 2294 of these visits were phenotype-positive with an average of 176 (median 104) phenotype-positive visits per site. Site visit by volume is presented in In the internal validation cohort of 125 charts, reviewers disagreed on the classification of 3 charts , with the adjudicator identifying the 2 discordant Algorithm 1 cases as not having OUD and the 1 discordant Algorithm 2 case as having OUD. Algorithm 1 had a positive predictive value (PPV) of 0.96 (95% CI 0.863-0.995) and a negative predictive value (NPV) of 0.98 . A total of 3 of the 8 discordant cases were phenotype-positive, of which the adjudicator determined 2 as having OUD. Of the 5 discordant cases that were phenotype-negative, the adjudicator identified 3 as having OUD. The combined phenotype had a PPV of 0.95 (95% CI 0.851-0.989) and an NPV of 0.92  information from patients’ charts.In both the internal and external validation chart reviews, the most common DSM criterion selected by the reviewers was “opioids are often taken in larger amounts or over a longer period than was intended.” In the internal validation phase, the second most common criterion was “recurrent opioid use in situations in which it is physically hazardous,” whereas the second most common criterion in the external validation phase was “craving, or a strong desire or urge to use opioids.” Although desire and effort to cut down or control opioid use are specific diagnostic criteria, they were inconsistently applied by the reviewers. As these specific criteria are not explicitly documented in the routine emergency care, the reviewers instead had to infer which criteria to apply to cases using available documentation. In both chart review phases, the least frequently identified criteria were those describing failures in social behavior as they pertained to the use of opioids. This could be because of the fact that ED billing requirements do not require detailed documentation of social history, and the impact of opioids on social behaviors usually has limited value for assisting clinicians in making a diagnosis during emergency care .Given the limitations of ED documentation, our phenotype benefited from incorporation of available structured data elements from the data dictionaries created in previous work to develop EHR phenotypes for primary care patients on chronic opioid therapy at risk for problematic opioid use ,23. GiveA strength of our study compared with previous EHR phenotype work is the external validation of the phenotype’s performance via chart review in the second health care system. For example, the HIV EHR phenotype developed by Paul et al  performsThe phenotype described here will be used as part of a subsequent pragmatic trial to be deployed across multiple health care systems to identify patients who may have been candidates for ED-initiated buprenorphine—these cases will form the denominator of a measure to assess what proportion of those potentially eligible actually received buprenorphine. As most patients evaluated using the phenotype will screen negative, our approach would likely result in a high number of false negatives in a true epidemiologic evaluation. As we were trying to maximize specificity for a pragmatic trial, the phenotype’s classification performance will meet the trial’s needs to screen patients for eligibility for ED-initiated buprenorphine with high specificity. Furthermore, the goal was not to definitively determine a diagnosis of OUD for each patient. For clinical practice, any patient identified as having OUD by this phenotype would require confirmation using an in-person assessment. As the capacity and expectation of EDs to treat OUD expands, so also does the value of an accurate EHR phenotype that could be used to identify patients who might benefit from treatment including ED-initiated buprenorphine and referral for ongoing medication treatment for OUD.The primary limitation of this study is the use of retrospective ED chart review as a reference standard for the diagnosis of OUD. Our chart review process was robust and included all clinical documentation up to date of the ED visit. However, a full diagnostic assessment by a psychiatrist or addiction medicine specialist would be the gold standard to establish a diagnosis of OUD. If available, it is possible that such an assessment would differ from chart review alone.External validation in an outside health care system strengthens the evidence for the generalizability of our phenotype. The external system uses the same EHR vendor but a different billing and coding company. It is unknown how the EHR phenotype would perform in systems using other EHR platforms. In addition, transportability issues have been discovered in preliminary estimates from a third health system because of differences in structured data capture of the chief complaint. In the external validation phase, the second algorithm did not identify a substantial number of cases. This suggests that there are likely local practice patterns in documentation or coding that may have affected the transportability of this EHR phenotype . AlthougIn future work, the efficacy of the phenotype algorithms will be tested in the EMBED trial, and the question as to whether these algorithms can function in a pragmatic ED setting will be answered. Statistically, the cohort selection and chart review performed here did not obtain a patient population reflective of the true prevalence of the disease, and as such, sensitivity and specificity calculations would not provide an accurate reflection of the phenotype’s performance in a true ED population. Assessment of sensitivity for events with low base rates is inherently unreliable. A very large sample size would be necessary for a precise estimate of sensitivity. Therefore, the assessment of sensitivity in this analysis is limited. To address this limitation, we report only PPV and NPV here. Estimating sensitivity and specificity for the external validation study can be done by inflating the phenotype-negative row numbers in the external validation confusion matrix by the sampling factor  to repreIn the upcoming trial, further evaluation of the phenotype algorithms’ performance will begin to address the intra- and intersite population sensitivity and specificity. To further refine the algorithm’s ability to discern between true and false positives, logistic regression is planned to predict future OUD-coded diagnoses given information from previous visits, such that variables can be removed given their performance in the regression model. Future work will also attempt to quantify the rate of false negatives through extended manual review and to determine whether changes to the algorithm improve sensitivity. The long-term goal of future work is to standardize the representation of the algorithms such that they can be portable beyond Epic to other EHR vendors as well as explore additional information retrieval techniques . A more An EHR phenotype derived and internally and externally validated for the purposes of a pragmatic trial to test the effectiveness of user-centered clinical decision support to increase the adoption of ED-initiated buprenorphine performed reliably and accurately to identify ED patients with OUD. The 2 algorithms comprising the phenotype were transportable across health care systems and have potential value for both clinical quality improvement interventions as well as research endeavors. Standardization of the phenotype will support efforts to use clinical phenotyping as an evidence-based tool at the front line of clinical practice.


Are reported rates of incorrect surgery changing in the US Veterans Health Administration (VHA)?This quality improvement study found that VHA-reported surgical adverse events have continued to trend downward from 1.74 to 0.47 per 100 000 procedures between 2000 and 2017. In this context, dentistry, neurosurgery, and ophthalmology remain a challenge.The VHA is holding the gains with preventing incorrect surgical procedures while continuing to work in selected areas of continued challenge. This quality improvement study examines the rate of surgical adverse events in US Veterans Health Administration medical centers from 2010 to 2017 and compares it with rates from previous studies. Reducing wrong-site surgery is fundamental to safe, high-quality care. This is a follow-up study examining 8 years of reported surgical adverse events and root causes in the nation’s largest integrated health care system.To provide a follow-up description of incorrect surgical procedures reported from 2010 to 2017 from US Veterans Health Administration (VHA) medical centers, compared with the previous studies of 2001 to 2006 and 2006 to 2009, and to recommend actions for future prevention of such events.This quality improvement study describes patient safety adverse events and close calls reported from 86 VHA medical centers from the approximately 130 VHA facilities with a surgical program. The surgical procedures and programs vary in size and complexity from small rural centers to large, complex urban facilities. Procedures occurring between January 1, 2010, and December 31, 2017, were included. Data analysis took place in 2018.The categories of incorrect procedure types were wrong patient, side, site (including wrong-level spine), procedure, or implant. Events included those in or out of the operating room, adverse events or close calls, surgical specialty, and harm. These results were compared with the previous studies of VHA-reported wrong-site surgery (2001-2006 and 2006-2009).Our review produced 483 reports . The rate of in–operating room (in-OR) reported adverse events with harm has continued to trend downward from 1.74 to 0.47 reported adverse events with harm per 100 000 procedures between 2000 and 2017 based on 6 591 986 in-OR procedures. When in-OR events were examined by discipline as a rate, dentistry had 1.54, neurosurgery had 1.53, and ophthalmology had 1.06 reported in-OR adverse events per 10 000 cases. The overall VHA in-OR rate for adverse events during 2010 to 2017 was 0.53 per 10 000 procedures based on 3 234 514 in-OR procedures. The most common root cause for adverse events was related to issues in performing a comprehensive time-out (28.4%). In these cases, the time-out either was conducted incorrectly or was incomplete in some way.Over the period studied, the VHA identified a decrease in the rate of reported adverse events in the OR associated with harm and continued reporting of adverse event close calls. Organizational efforts continue to examine root cause analysis reports, promulgate lessons learned, and enhance policy to promote a culture and behavior that minimizes events and is transparent in reporting occurrences. Accordingly, the National Quality Forum expanded its definitions to be more inclusive, recognizing the need to continue to address this challenge.10Health care continues to strive to improve patient safety. One of the most challenging issues is preventing wrong-site surgery,4 However, these estimates vary, and such rates are difficult to measure precisely.5 Some rates are based on clinician incident reporting, while others came from closed claims data,11 both of which may be limited in accuracy. Furthermore, even with indications of the prevalence of wrong-site surgery, we still need more evidence on why these errors continue to occur and how to prevent them.4Such events have been estimated to occur at a median (range) rate of 0.09 (0-3.9) events per 10 000 cases.13 Both systems reconcile these data quarterly to increase the accuracy of wrong-site surgery measurement. The last published2 (2006-2009) VHA rate of in–operating room (in-OR) adverse events was 0.4 adverse events per 10 000 procedures. That study also included specialty specific rates and root causes of such errors. The current study contains data on incorrect surgery from the VHA up to 2017. This study is based on data from the largest integrated health care system in the United States with 2 complementary systems reporting for increased accuracy of capturing surgical adverse events. Therefore, this study provides one of the most comprehensive reports on wrong-site surgery, spanning a rate review of 17 years. It not only includes event rates but also provides in-depth information on why such events may still occur and recommendations on how to prevent similar future events.The Veterans Health Administration (VHA) National Center for Patient Safety (NCPS) has a national centralized database for reporting patient safety events. The VHA National Surgery Office (NSO) also has a centralized reporting system specifically for surgical patient safety errors.SQUIRE) reporting guideline14; it describes several years of an initiative to improve surgical safety care. We provide the context and the overall interventions of systemwide efforts to prevent incorrect surgery, results, limitations, and discussion. The Research and Development Committee of White River Junction VHA Medical Center approved this study and the Committee for the Protection of Human Subjects of Dartmouth College considered it exempt. All data were deidentified.This is a quality improvement study based on the Standards for Quality Improvement Reporting Excellence  and probability of occurrence (frequent to remote), with ratings from 1 to 3 for both actual events and close calls. Events that receive an SAC score of 3 are a combination of the most severe and frequent events.12 The most severe cases undergo an RCA, and less severe events are entered as safety reports. Root cause analysis teams examine what happened, why it happened, and how to prevent it from happening again. The data reviewed were limited to information derived from RCAs and safety reports submitted to NCPS and were not abstracted from medical records.Reported surgical adverse events in the VHA are reviewed at the local facility. Patient safety managers rate the severity and probability of harm related to events using a safety assessment code (SAC) matrix, developed by the VHA to prioritize adverse events for the root cause analysis (RCA) process.13In addition to the RCA and safety reporting, in 2010 the VHA established the Critical Incident Tracking Notification (CITN) process to ensure timely notification of surgical adverse events reported to the NSO and NCPS. The NSO and NCPS reconcile CITN events with RCA and safety report data on a quarterly basis and educate surgical staff in the field by sharing redacted summary reports of these events with the intent of preventing similar events. The CITN-based reporting and lessons learned processes have been described in detail elsewhere.1 For example, close calls were defined as events in which the patient did not undergo a procedure  that was incorrect. Specific examples of close calls include the wrong patient given a consent form, the wrong procedure written on a consent form, the wrong site marked, or the wrong side of the patient prepared for surgery. As long as the error was discovered and no skin puncture or injection was made, it was a close call.Two of us (J.N. and C.S.) coded 20 reports and reached a coefficient κ of 0.83. Thereafter, the cases were coded independently, and consensus was reached on any items for which the coders had questions. Reports were coded into major categories such as type of event and body segment. They also coded the root causes reported by the teams into groupings such as education and communication. The specific definitions for the coding have been previously described.2A descriptive analysis was conducted in which we summarized the types of adverse events and close calls that occurred, their location, and in which specialties they occurred for the 8-year period between 2010 and 2017. We calculated the rate of in-OR reported adverse events per 10 000 procedures . The numerator was the number of reported in-OR adverse events over the 8 years and the denominator was the number of surgical procedures performed cumulatively and by surgical specialty in the OR over the study period. We compared adverse event rates with those reported in a previous study (2006-2009).We performed a trend analysis by modeling the count data of the number of surgical events with a SAC score of 3 using the Poisson distribution  to examine the decrease in the rate of actual events with a score of 3 over time in the VHA. The independent variable was calendar year (modeled as a continuous variable). Rate ratios (RRs) and accompanying 95% confidence intervals were calculated to represent the strength of association between time (year) and incorrect surgery rates, estimated using the Poisson regression. In general, a simple Poisson model has the following general form: log (rate) = intercept + slope × independent variable.In our trend analysis, we calculated the rate of reported adverse events with the most harm (those with an SAC score of 3) as the numerator over the denominator of the total number of surgical procedures in the OR per year and multiplied by 100 000 for a whole number.P values are 2-sided, at a significance level of .05.All analyses were performed using SAS statistical software version 9.2 (SAS Institute Inc). All reported Our review of the most recent period (2010-2017) produced 483 reported cases over 8 years from 86 different VHA medical centers . This ti2 The overall VHA in-OR reported adverse event rate was 0.53 per 10 000 procedures based on 3 234 514 in-OR procedures. This rate is slightly higher than the overall rate of reported in-OR adverse events in the most recent previous study (2006-2009), which was 0.4 per 10 000.2The rate of in-OR adverse events with the most harm (SAC score of 3) decreased from 1.74 to 0.47 reported adverse events with harm per 100 000 procedures between 2000 and 2017   based on1 2006 to 2009, and 2010 to 2017.2 We show a downward trend (2001-2017) in reported adverse events with the most harm, with continued reporting of close calls. This supports that improvements are likely not due to decreased reporting overall. Continued close call reporting also indicates a culture of safety, supporting learning from events.This study focused first on the current update period (2010-2017) and then compared overall rates with serious harm over time during all 3 study periods: 2001 to 2006,2 For example, this study reported 86 wrong-implant events over 8 years, whereas the prior study (2006-2009) identified 16 wrong-implant events over 5.5 years.2 An event such as implanting an expired lens may previously have gone unreported, but the CITN process clearly mandated it as a reportable event. Accordingly, the overall rate of reported in-OR adverse events increased from the level shown in the 2006 to 2009 study2 (0.4 to 0.5 per 10 000 procedures) in the most current period. However, even this rate remains low and well within the private-sector range of 0 to 3.9 reported surgical adverse events per 10 000 procedures.4Beginning in 2010, the NSO collaborated with the NCPS in establishing the CITN process for immediate reporting of surgical adverse events. The number of reported surgical adverse events increased in comparison with previously reported VHA studies on this topic.15 directive added greater specificity regarding the intraoperative requirements for the attending physician performing spine procedures to confirm the placement of spine marker with a fixed radiograph to locate the appropriate vertebral body or intervertebral space for the procedure. The policy is currently being modified to recommend that the location of the spinal marker must be confirmed with a radiologist if any member of the surgical team questions the location of the spinal marker. Devine et al5 wrote extensively about the challenges that clinicians have with wrong-level spinal surgery.When reported in-OR adverse events were analyzed as a rate, we found that dentistry and neurosurgery had the highest rates of reported adverse events, followed by ophthalmology. Ten of the 13 neurosurgery adverse events were spine cases. The July 2013 update of the VHA Ensuring Correct Surgery and Invasive Procedures2 In the last review, dental events were one of the top 5 specialties to report adverse events, and dentistry was in the top 4 this time; this rate may not signal a change in the frequency of dental events. However, we are working closely with VHA dental leaders to clarify that sites must be marked using digital imaging, radiographs, or intraoral charts.We were able to calculate a specific rate for in-OR reported dental events in this review because we had denominator data specifically for oral surgery, which we did not have in the last review (2006-2009).2 Fifty-two of the 86 cases of wrong-implant adverse events were due to completed implantation of an expired lens. The Joint Commission does not include expired implants as a type of wrong-site surgery.6 However, because of the risk of causing harm, the VHA includes expired implants in its definition, and these 52 events are a warning sign indicating failure to follow policy. Specifically, the VHA modified directive will include a requirement to specifically state the implant expiration date with a read-back for lens implants during the time-out and just prior to implantation.Ophthalmology had the second highest rate of reported in-OR adverse events. This is similar to our previous reports (2001-2006 and 2006-2009).17 and discussed further possible actions during a VHA summit to prevent incorrect surgical adverse events in December 2017. Others have also shared similar challenges with incorrect eye surgery.18Reducing eye surgery adverse events has been a challenge. The cases are short, the time pressure is high, and there are multiple steps in the process from measuring the needed lens strength in the clinic to implanting a nonexpired correct lens in the OR. We issued a patient safety alert on this topic, conducted an in-depth assessment of the challenges,19Overall, the most common root cause for incorrect surgery was time-out–related issues. This differs from the last VHA report, in which a lack of standardization of clinical processes was the top root cause. This may signal progress, as sometimes in RCAs the team will write that although a time-out was done, it was not done mindfully. This signals a stronger commitment to safety rather than just ambivalent compliance.13 Each quarter we jointly review all reported adverse events for surgery, develop lessons learned, share these on a conference call with surgical leadership from the VHA’s 18 regions (Veteran Integrated Service Networks), and distribute summary information to staff.13There are many possible reasons for the improvements in this report and for the challenges that still exist. There has been a steady focus on OR safety in the VHA. We have continued to produce and share the surgical lessons learned and to collaborate closely with the NSO.This study has limitations. First, these data are based on self-report and therefore we cannot be sure all incorrect surgical adverse events were included, particularly those that occurred outside the defined structure of the OR. We also do not have demographic data about patients because the RCAs and safety reports are deidentified. In addition, there were times the reports did not contain enough information to categorize the cases fully. The coding of harm is done at the local facility and there may be variation in the application of the SAC scoring system.We have demonstrated an overall decrease in the number and severity of incorrect surgical procedures in the VHA. Telling people to follow the policy is not enough to prevent incorrect surgery. We need continued leadership support and peer-to-peer communication such as sharing lessons learned. We need to ensure the use of standardized procedures and to initiate specialty-specific safety measures in areas such as for eye and spine surgery. Most importantly, we need to do more than go through the motions of a time-out. We need to mindfully conduct time-outs, use the universal protocol, and use human factors engineering to work toward eliminating incorrect surgical adverse events.