Depression is a common comorbid condition in individuals with chronic back pain (CBP), leading to poorer treatment outcomes and increased medical complications. Digital interventions have demonstrated efficacy in the prevention and treatment of depression; however, high dropout rates are a major challenge, particularly in clinical settings.This study aims to identify the predictors of dropout in a digital intervention for the treatment and prevention of depression in patients with comorbid CBP. We assessed which participant characteristics may be associated with dropout and whether intervention usage data could help improve the identification of individuals at risk of dropout early on in treatment.Data were collected from 2 large-scale randomized controlled trials in which 253 patients with a diagnosis of CBP and major depressive disorder or subclinical depressive symptoms received a digital intervention for depression. In the first analysis, participants’ baseline characteristics were examined as potential predictors of dropout. In the second analysis, we assessed the extent to which dropout could be predicted from a combination of participants’ baseline characteristics and intervention usage variables following the completion of the first module. Dropout was defined as completing <6 modules. Analyses were conducted using logistic regression.2: OR 1.55, 95% CI 1.18-2.04) were significantly associated with a higher risk of dropout. In the analysis that aimed to predict dropout following completion of the first module, lower and higher age , medium versus high social support , and a higher number of days to module completion  predicted a higher risk of dropout, whereas a self-reported negative event in the previous week was associated with a lower risk of dropout . A model that combined baseline characteristics and intervention usage data generated the most accurate predictions (area under the receiver operating curve [AUC]=0.72) and was significantly more accurate than models based on baseline characteristics only (AUC=0.70) or intervention usage data only (AUC=0.61). We found no significant influence of pain, disability, or depression severity on dropout.From participants’ baseline characteristics, lower level of education  and both lower and higher age (a quadratic effect; age: OR 0.62, 95% CI 0.47-0.82, and ageDropout can be predicted by participant baseline variables, and the inclusion of intervention usage variables may improve the prediction of dropout early on in treatment. Being able to identify individuals at high risk of dropout from digital health interventions could provide intervention developers and supporting clinicians with the ability to intervene early and prevent dropout from occurring. Chronic back pain (CBP) is a major global health concern with lifetime prevalence rates of 60%-70% . CBP is treatment gap in mental health care [Psychological interventions have been demonstrated to be effective in the treatment of depression for individuals with CBP . Accordilth care .Digital interventions, which deliver psychotherapeutic components via a web browser or smartphone app, have been proposed as a scalable, cost-effective way to meet the growing demand for psychological treatment and address many of the challenges associated with accessing traditional face-to-face therapy -15. TherDespite the demonstrated efficacy, adherence to digital interventions remains a major challenge. A recent meta-analysis on digital interventions for depression identified that, on average, little more than half of the participants completed the full intervention and only 25% of the participants completed the full intervention when the intervention was delivered within routine health care settings . The samSeveral studies have assessed predictors of dropout in digital mental health interventions ,32-34. IHowever, to the best of our knowledge, no research has been published to date that assesses which factors may predict dropout in a digital intervention for depression in patients with comorbid somatic illness. This question may be particularly pertinent for these individuals as chronic pain and depression are often associated with reduced motivation to initiate or complete goal-directed tasks ,36. As tKnowing which patient characteristics predict a higher likelihood of dropout may be valuable for identifying individuals in advance of treatment that might benefit from alternative care pathways ,40. Howechurn-prediction models to identify if a user is likely to churn , similar models and principles of human-computer interaction may be valuable to predict dropout and maximize adherence within the field of digital health. For example, in a digital intervention for a chronic lifestyle disease, Pedersen et al [Although the field is still nascent, there is a wealth of data generated from digital interventions that may aid the prediction of dropout once a patient has started treatment. In the same way that many digital apps outside of health care use en et al  were ablen et al  identifiAs digital mental health interventions are being increasingly adopted by health care systems worldwide ,47, the Drawing from 2 recent studies on a guided digital intervention for the treatment and prevention of depression in individuals with comorbid CBP, we asked the following research questions:Can we predict intervention dropout from participant baseline characteristics? If so, which participant characteristics predict a higher likelihood of dropout?Can we develop an “early warning system” that identifies participants at risk of dropout early on in the intervention? Specifically, how accurately can we predict which participants will drop out after completion of the first module and what is the most accurate model for classifying at-risk individuals?Do predictors of dropout differ between patients clinically diagnosed with major depressive disorder (where the intervention is targeting treatment) and patients with subclinical depressive symptoms (where the intervention is targeting prevention)?This study was a secondary analysis of data from 2 trials that assessed the efficacy of a therapist-guided internet-based intervention for the treatment  and prevAll participants (N=253) assigned to the intervention arms of the primary studies were included in this analysis. The inclusion criteria of the primary studies were as follows: (1) age ≥18 years; (2) presence of depressive symptoms, either reported persistent subthreshold depressive symptoms in the past 3 months or meeting the Diagnostic and Statistical Manual of Mental Disorders, 4th Edition, criteria for a mild to moderate depressive episode or persistent depressive disorder; (3) diagnosed back pain chronicity of at least 6 months; (4) have German language skills; and (5) access to internet and PC. The exclusion criteria were as follows: (1) having ongoing or planned psychotherapy within the forthcoming 3 months, (2) being currently suicidal or having had suicidal attempts within the past 5 years, or (3) having had a severe depressive episode within the past 6 months. In the primary studies, participants were recruited during or following discharge from 1 of the 82 orthopedic clinics across Germany. They were recruited personally by a clinician or on the web using a flyer and information letters distributed by the clinic.The intervention is a guided internet- and mobile-based intervention for the treatment (eSano BackCare-D ) or prevIn this study, 8 baseline characteristic variables were assessed as potential predictors of dropout. Variables were chosen on the basis of previous research pointing to demonstrated or hypothetical relationships between the predictor variables and intervention adherence or dropout ,45,46,51Intervention usage measures included both active and passive measures. The active measures were the stress level reported by the participant at the start of each module (“Burden”) and the occurrence of any negative events experienced in the past 7 days, self-reported by the participant at the start of each module (“Negative Events”). Burden was assessed using a Likert scale from 0 to 10, where 0=“not burdened at all” and 10=“extremely burdened.” Negative events were dummy coded as 0=“no negative event in the past week” and 1=“at least one negative event in the past week.” For passive measures, we included the number of days taken to complete each module (“N Days to complete module”) and the number of minutes spent on the web completing each module (“Time spent online completing module”).Dropout was defined as completing <6 intervention modules, in accordance with the intervention developers ,50. It wTo assess whether participants’ baseline characteristics could predict dropout, analyses were conducted using logistic regression in 3 steps. First, we conducted a series of bivariate analyses to assess the odds ratios (ORs) of each baseline variable (bivariate “bivariate model”). Second, we repeated the analyses with all baseline variables simultaneously entered into the binomial model (the “complete model”). Finally, we built a “parsimonious model” in which we excluded nonsignificant predictors with no incremental predictive power from the complete model in a stepwise procedure.Akaike information criterion (AIC) and Bayesian information criterion (BIC) were used as measures of model fit and for model comparison. For nested models, likelihood ratio tests were used to directly compare whether 2 models were significantly different from one another . CollineAs this was an exploratory study, we did not adjust for multiple testing. The study was not powered for confirmatory analysis of the predictors, and alpha adjustment may have increased the likelihood of type II errors.To assess whether we could identify people at risk of dropout early on in the intervention, we first created a subset of the data available up until the point of module 1 completion . We then compared the performance of three separate logistic regression models using the constrained data set: (1) a model based on participant baseline characteristics only—the “baseline characteristics model,” (2) a model based on intervention usage variables only—the “intervention usage model,” and (3) a model combining all baseline characteristics and intervention usage variables—the “combined model.” The quality of the models was assessed using the area under the receiver operating characteristic curve (AUROC) and related measures of sensitivity and specificity . The optSensitivity analyses were conducted to assess whether predictors differed in the prevention and treatment studies. Here, study was included as a dummy-coded variable (0=PROD-BP for the prevention study and 1=WARD-BP for the treatment study) in all parsimonious models, first as an additional predictor to assess for a main effect of study type on dropout and then as an interaction term with other predictors in the model to assess whether the effect of a predictor differed across studies.To assess whether the number of modules completed influenced the relative risk of dropout, we conducted sensitivity analyses using Cox proportional hazards regression . In thisMissingness occurred in 111 out of 3084 (3.6%) data points and was assumed to be missing at random, indicating that missingness depended on observed data . To avoiAll analyses were conducted in R using R Studio . The pRThis study was a secondary of analysis of data from two RCTs—Sander et al  and BaumAmong the 253 participants, 149 (58.9%) were female and 104 (41.1%) were male. The age of participants ranged from 24 to 78 years, with a mean age of 51.1 (SD 8.88) years. Of the 253 participants, 171 (67.6%) reported having a low level of education. Of the 253 participants, 34 (13.4%) were single, 180 (71.1%) were in a relationship or married, and 39 (15.4%) were divorced or separated. The mean depression severity at baseline was 10.3 (SD 5.93), as measured by the HAM-D, and 9.94 (SD 4.41), as measured by the Patient Health Questionnaire-9. The mean level of pain disability was 31.3 (SD 14.7), as measured by the ODI, and the mean level of pain self-efficacy was 34.9 (SD 13.0), as measured by the PSEQ. On average, the participants completed 4.65 out of the 6 regular and 3 optional modules (SD 3.48). The participants took an average of 17.64 (SD 19.55) days to complete each module, and the mean time on the web taken to complete a module was 80.26 (SD 136.96) minutes. The mean self-reported burden was 4.55 (SD 1.97), and the mean number of self-reported negative events across the intervention was 0.80 (SD 1.33). r=0.63 and r=−0.73, respectively) and were not significant in the bivariate analyses, they were not included in the multivariate predictor models to prevent collinearity. The results of the bivariate analysis indicated that a lower level of education was significantly associated with a higher risk of dropout . None of the other potential predictors  were statistically significant at the level of P<.05 in the bivariate analysis.P=.03). When age was added as a quadratic term (age2) to the model to account for the nonlinear relationship between age and dropout, we found that both age  and age2  were significant predictors, such that both lower and higher age were associated with increased risk of dropout.In the complete model, being single was found to be an additional significant predictor of dropout  and age  remained significant predictors of dropout. Marital status, internet affinity, baseline depression severity, and baseline pain intensity were found to be nonsignificant after controlling for the other predictors.In the parsimonious model, where predictors were reduced stepwise to relevant predictors only, low education  and low education  were significant predictors of dropout. The AUROC for the model was 0.70, the sensitivity was 68%, and the specificity was 62%.P=.005), whereas a self-reported negative event in the previous week was associated with a lower risk of dropout . The AUROC for the model was 0.61, the sensitivity was 56%, and the specificity was 54%.In the parsimonious model using only intervention usage data, a higher number of days to module completion predicted a higher risk of dropout , medium versus high social support , and a higher number of days to module completion  all predicted a higher risk of dropout, whereas a self-reported negative event in the previous week was associated with a lower risk of dropout . The AUROC for the model was 0.72, the sensitivity was 76%, and the specificity was 59%.In the parsimonious model that combined participant baseline characteristics and intervention usage variables as predictors, higher and lower age .As shown in Sensitivity analyses assessing whether findings differed between the treatment (WARD-BP) and prevention (PROD-BP) studies found no significant difference between the two, either in terms of main effect or interaction effects with other predictors. Sensitivity analyses assessing whether findings differed when using Cox proportional hazards regression versus logistic regression found no difference in the significant predictors. This study aimed to identify the predictors of treatment dropout in a digital intervention for the treatment and prevention of depression in patients with comorbid CBP. From the participants’ baseline characteristics, we found that a lower education level and lower and higher age (a quadratic effect) predicted a higher risk of dropout. From the intervention usage variables, we found that a higher number of days to module completion predicted a higher risk of dropout, whereas the occurrence of a negative event in the previous week predicted a lower risk of dropout.Participants with lower education levels were more likely to drop out of treatment, which is consistent with a large body of research on adherence to both digital interventions and face-to-face psychotherapy ,51,65-67This study also demonstrated the feasibility of predicting dropout early on in the intervention based on data restricted to the first module and participant baseline characteristics. Our finding that the number of days taken to complete the first module significantly predicted dropout is consistent with the study by Bremer et al  that ideIndeed, a comparison of models using baseline characteristics and intervention usage variables revealed that a model that combined baseline characteristics and intervention usage data generated the most accurate predictions and was significantly more accurate than models based on baseline characteristics only or intervention usage data only. Moreover, in terms of clinical utility, the AUROC of 0.72 and sensitivity of 76% exceeded the accuracy threshold of 65%-70%, at which clinicians reportedly become willing to act on predictions . Implemebefore completion of the first module would be especially valuable as a significant proportion of individuals drop out before then [Notwithstanding the above, there is still significant room for improving model performance. In particular, the development of models that are able to predict dropout ore then ,69. Deveore then ), data sore then ), and adore then ,73. Armeore then ,46.Finally, as the findings of this study were specific to one intervention, future research would also benefit from assessing whether the predictors found herein are valuable for predicting dropout in other digital interventions, alternative disorders, and different populations. If it is consistently found that there are a set of variables such as “number of days to module completion” that are associated with higher risk of dropout, these predictors may then be used to inform the basis of models for other interventions in the future. In the same way that outcome feedback technology that identifies individuals at risk of deterioration during treatment has been shown to improve eventual treatment outcomes ,44, dropTo the best of our knowledge, this is the first study to examine the predictors of dropout in a digital intervention for depression in individuals with a comorbid somatic illness. This is also the first study to compare whether predictors of dropout differ when the intervention is aimed at prevention  versus treatment  and using study samples with clinically verified diagnoses at baseline. Finally, in contrast to most studies conducted to date, which have been based on efficacy trials with small sample sizes and convenient samples , this anDespite these strengths, we acknowledge several limitations of this study. First, the analyses were based on data specific to a prevention and treatment version of one intervention and one population, namely, individuals with depressive symptoms and CBP. As such, the predictors we found to be significant and the subsequent accuracy of the classification models may not generalize to other interventions or other populations. For example, dropout has been found to be significantly greater in unguided interventions than in guided interventions, and the mechanisms underlying dropout may differ between the two ,51. FutuThe high dropout rates associated with digital health interventions remain one of the biggest challenges to their successful implementation in real-world health care settings. Being able to identify individuals at high risk of dropout early on in treatment may provide clinicians and intervention developers with a valuable opportunity to intervene early and prevent dropout from occurring. Using a combined set of predictors from patient baseline characteristics and intervention usage data, we were able to identify individuals at risk of dropout early on in a digital intervention for depression in patients with comorbid CBP. Future research should explore ways of improving model accuracy and investigate the feasibility and efficacy of using these models directly within the interventions themselves to improve adherence.


Indoor housed dairy cows are highly motivated to scratch themselves using mechanical brushes. Many farms provide brushes to cows, yet no commercial brushes to date capture how the brushes are used by the cows in the pen. We developed an automated brush and tested how much cows use it at four different group sizes  and with different brush locations in the pen. We found that cows used the brush for longer when it was close to the feed and water and when they were housed in smaller groups. We suggest that future studies provide groups with multiple brushes to better understand the influence on brushing behavior of cows.Mechanical brushes are often provided on dairy farms to facilitate grooming. However, current brush designs do not provide data on their use, and thus little is known about the effects of group size and placement of brushes within the pen. The objectives of this study were to automatically detect brush use in cow groups and to investigate the influence of (1) group size and the corresponding cow-to-brush ratio and (2) brush placement in relation to the lying stalls and the feeding and drinking areas. We measured brush use in groups of 60, 48, 36, and 24 cows, with the brush placed either in the alley adjacent to the feed bunk and water trough or in the back alley. Cows used the brush for longer when it was placed in the feed/water alley compared to when placed in the back alley. Average brush use per cow increased when cows were housed in smaller groups, but the brush was never in use more than 50% of the day, regardless of group size. We conclude that brush use increases when availability is increased and when the brush is placed closer to the feed and water. Mechanical brushes are provided on dairy farms to facilitate grooming behavior, helping cows to remove dirt and parasites ,2. IncreDespite the growing interest from farmers to install mechanical brushes on their farms, and their recognized potential for providing welfare-relevant information , no commIn summary, the research problem is twofold: there is a lack of scientific knowledge about how cows use mechanical brushes in groups of different sizes and with varying locations, and there are no practical devices available to automatically monitor brushing behavior in dairy cow groups. To address this research gap, we aimed to establish an automated brush prototype and to investigate how the use of a single mechanical brush is affected by (1) increasing group sizes and (2) placement of the brush in relation to proximity to feed and water. We hypothesized that cows would use the brush more when housed in smaller groups and when the brush was placed closer to the feed and water.ad libitum access to water and were removed from the pen twice daily for milking at approximately 0730 h and 1730 h. We measured the brush use of pregnant lactating Holstein cows  housed at The University of British Columbia (UBC) Dairy Education and Research Centre in Agassiz, BC, Canada, between September 2021 and January 2022. Cows were housed in a free stall barn with lying stalls bedded with sand; the cow-to-stall ratio was 1:1. Fresh Total Mixed Ration  was delivered twice daily at approximately 0600 h and 1530 h; cows accessed feed from a post-and-rail feed barrier providing 0.6 m per cow . Cows haOne cylindrical-shaped mechanical brush  was placed in a pen designed to house 60 cows and adapted to this study by the inclusion of an additional alley . The bru2/cow) and stocking densities of cows to lying stalls and feeding areas for groups of 60, 48, 36, and 24 cows. We measured brush use when the brush was placed at two different locations: the feed alley adjacent to the feed and water, or the alley furthest away from the feed and water . Cows exited and entered the pen for milking from the back alley  for each group size and brush placement treatment using R 4.2.0 . For each treatment and replicate , we calculated the average daily brush use per cow across days and used this as a dependent variable in a linear model including brush location (two levels), group size, and replicate (two levels) as fixed effects. We did not detect any significant interactions so these terms were excluded from the final model. We also investigated the difference in the average daily temperature between replicates using a separate linear model. To understand brush use levels during different times of the day, we calculated the average hourly brush use during daytime (0600 h to 1800 h) and nighttime (1800 h to 0600 h) and investigated their difference using a linear mixed model with time of day  as a fixed effect and group  as a random effect. Model residuals were plotted and inspected visually to confirm homoscedasticity and normality.Group-level brush use ranged between 2.6 and 10.1 h/day. The mean ± SD of daily brush use for different group sizes was: 24 cows, 3.8 ± 0.6 h; 36 cows, 5.5 ± 1.3 h; 48 cows, 5.6 ± 1.4 h; 60 cows, 7.6 ± 1.8 h. Across treatments, brush use averaged 8.5 ± 2.0 min/day/cow . Cows used the brush more when it was placed in the feed alley  and brush use was lower during the second replicate . The average daily temperature was also lower during the second replicate . The average hourly brush use in the groups did not differ between daytime and nighttime.Average daily brush use per cow decreased with increasing group size  showed increased brush use, suggesting that individual brush use increases with increasing availability. However, questions remain as to the effects of larger groups given that some commercial farms employ much larger group sizes than those tested in this study . FurtheWe found higher brush use when the brush was placed close to the feed and water resources, as did previous work . AlthougPrevious work found a decrease in the frequency of brush use events with increasing heat load , which mIndividual variation between the cows enrolled in the two replicates may also explain some of the differences observed. Although we did not distinguish between individual cows in this study, previous findings indicated considerable between-individual variation in the daily brush use ,14. GrooWe altered the group size and the cow-to-brush ratio simultaneously and so are unable to determine which factor contributes to the differences in brush use across treatments. Cow behavior and resource use are known to be affected by both group size  and stoUsing an automated brush, we found that cows used the brush for longer when it was placed close to feed and water. Average brush use per cow was also higher in smaller groups with a lower cow-to-brush ratio. We suggest that future research assesses the use multiple automated brushes in each pen.


Training prehospital personnel in identifying patients with acute stroke is key to providing rapid treatment. This study aimed to investigate whether game-based digital simulation training is a feasible alternative to standard in-person simulation training.Second-year paramedic bachelor students at Oslo Metropolitan University in Norway were invited to participate in a study to compare game-based digital simulation (intervention) to standard in-person training (control). For 2 months, students were encouraged to practice the NIHSS, and both groups logged their simulations. Then, they performed a clinical proficiency test, and their results were assessed using a Bland-Altman plot with corresponding 95% limits of agreement (LoA).n = 23) spent an average (SD) of 42:36 min (36) on gaming and performed 14.4 (13) simulations on average, whereas the control group (n = 27) spent 9:28 min (8) simulating and performed 2.5 (1) simulations. Comparing time variables collected during the intervention period, the mean time for each simulated assessment was significantly shorter in the game group . In the final clinical proficiency test, the mean difference from the true NIHSS score was 0.64 (LoA: − 1.38 to 2.67) in the game group and 0.69 (LoA: − 1.65 to 3.02) in the control group.Fifty students participated in the study. Individuals in the game group . Timely prehospital identification of patients with suspected acute stroke is essential to ensure rapid treatment and improve outcomes , 2. ReduAs part of the Paramedic Norwegian Acute Stroke Prehospital Project (ParaNASPP), an e-learning program with digital classroom training  and a moApplying principles from game theory like rapid decision-making based on limited information has been proposed for acute stroke care . GamificGameStroke was developed by the Norwegian Air Ambulance Foundation in collaboration with stroke researchers, stroke clinicians and paramedics. The game simulates a paramedic character, performing the standardised NIHSS assessment of a simulated patient. The order of the 11 different NIHSS observations is determined by the application and cannot be deviated from. The simulated patient was programmed based on 12 unique simulated cases presenting symptoms of acute stroke based on a wide variety of NIHSS scores resulting in an algorithm with over 1000 variations. The game is divided into three difficulty levels: easy, medium and hard to make the game evolve and increasingly demanding. GameStroke features a range of game elements introduced to create engagement. The players can choose one of four avatars or upload a profile picture for a more personalised experience. The score per completed case is based 70% on the right answer and 30% on time incentives. They receive rapid feedback with an overview of the examination, right and wrong answers, time use and total score at the end of each game. The players can track the leaderboard, providing a social and competitive element; this can however be opted out of by playing offline. They can also earn badges and advance to more challenging levels as they master the game. GameStroke uses pictograms , were invited to participate. The study took place from allocation  until the clinical proficiency test .All students who agreed to participate signed an electronic written consent form and provided necessary background information.All enrolled students received a validated NIHSS training program for paramedics  consistiSix students had previous relevant clinical experience which could potentially affect the study results. To secure equal allocation of these students in the two study groups, the students were paired based on the extent and similarity of their clinical experience.Secondly, as the game currently only exists for iOS, students who did not own an iOS device were placed in the control group.Finally, the remaining students were randomly assigned to either the game or control group using the web-based ‘Research Randomizer’ . This waThe control group received a link to the online self-report form and the game group received instructions on how to download the GameStroke application, using a two-factor authorization to validate the players’ identification.NIHSS simulations training was voluntary for both groups. The control group was asked to fill out a short online self-report form if they did a simulation session to practice their NIHSS assessment skills. These simulations were arranged by the students without an instructor present. In addition to the number of simulations performed and time spent on each case, students in the control group reported the total amount of time spent simulating. This included time spent playing the role of ‘patient’ and observing fellow students’ examinations when waiting for their turn, since learning also takes place through observation .For the game group, simulation activities including the number of completed games, time and date and time spent per simulation were automatically registered in the GameStroke application. The data collected was linked to the student’s mobile phone number.After the 2-month period of voluntary simulation training, all study participants completed a clinical proficiency test to assess their skills in scoring NIHSS. They were tested simultaneously by observing and scoring video recordings of six NIHSS assessments, where a paramedic examined a fictitious patient acting different scripted stroke cases with a pre-defined NIHSS score. The video recordings have been used in previous validation studies in the ParaNASPP trials and a slight variance ± 3) was deemed acceptable based on these results , 8. In t was deemTo have comparable groups, students who did not attend the mandatory lecture were excluded prior to group allocation. Students in the intervention group who did not download the application or those who did not take the clinical proficiency test were excluded before analysing the results.The study was approved by the Norwegian Centre for Research Data (reference no. 543238). The consent form and self-reported training log for the control group were created online using an encrypted version of the Tjenester for sensitive data (TSD), a research platform for sensitive data. Data for the game group was automatically collected by the application. The students in the game group also accepted GDPR consent when logging onto the GameStroke application for the first time.U test was used to compare the skewed continuous variables in the game and control group, including differences in the number of simulations, time spent on simulation in total and time spent per simulation. Pearson’s chi-square test was used for categorical variables.Mann-Whitney U tests were performed using SPSS V27 . The Bland-Altman analysis was performed using R V4.0.3 .A Bland-Altman plot with 95% limits of agreement (LoA)  was usedOut of 58 invited participants, all students volunteered to participate, of which 50 were included for final analyses: 23 in the game group and 27 in the control group Fig. .Table 1Cp < 0.003). There was a significant difference in the total time spent on simulations. The intervention group spent an average (SD) of 42:36 min (36) on gaming, whereas those who simulated in the control group spent 9:28min (8) simulating. In total, the game group spent 16.3 h on game simulation, compared to the control group who only spent 2.4 h on active in-person simulation (p < 0.000). When including time spent passively observing fellow students’ examinations or playing the role of the stroke patient, the control group spent a total of 7.1 h on in-person simulation, which is still significantly less than the game group (p < 0.000).In the intervention group, 23 of the 28 students (82%) used the game application, whereas only 15 of the 27 (55.5%) in the control group self-reported that they had simulated (p < 0.000). The average time spent per simulation differed significantly, by almost a minute between the game group (2:57 min) and the control group (3:50 min) (p < 0.004).The intervention group completed a total of 333 games , whereas those who simulated in the control group completed a total of 36 simulations  , this was not associated with a difference in interrater agreement when tested in the proficiency test. It seems that paramedics trained in NIHSS have a high level of accuracy, but tend to overestimate the NIHSS score slightly, as seen in recent studies comparing prehospital NIHSS to in-hospital scores .5, this . The stuThese results are consistent with several previous studies which have found that gamification often leads to increased use and engagement  but infrGaming may be used to standardise both NIHSS training of health personnel and clinical assessment of stroke patients. With limited access to in-person simulation training, gamification opens new ways to acquire clinical skills and to provide further education and skill retention. It can also reduce operating costs , 26, altThe major strength of the study is the use of a control group with active learning and simulation . We alsoThe major limitations include the partial non-randomised design. The applications measure time spent down to the second, but there is likely a considerable margin of error in the self-reported times. Due to the use of video recordings, we were unable to test their NIHSS assessment time in the proficiency test. Technical difficulties may arise with application-based gamification, and a system for reporting errors has to be in place. It must be noted that this is a traditional Bland-Altman analysis  and doesParamedic students can learn to accurately assess NIHSS through game-based simulation training. Gamification of NIHSS seemed to give an incentive to simulate considerably more and to perform the assessment faster, compared to standard in-person simulation training. This feasibility study opens for further research on gaming as clinical skill training for healthcare personnel assessing stroke patients.


Artificial intelligence (AI) offers the potential to support healthcare delivery, but poorly trained or validated algorithms bear risks of harm. Ethical guidelines stated transparency about model development and validation as a requirement for trustworthy AI. Abundant guidance exists to provide transparency through reporting, but poorly reported medical AI tools are common. To close this transparency gap, we developed and piloted a framework to quantify the transparency of medical AI tools with three use cases. Our framework comprises a survey to report on the intended use, training and validation data and processes, ethical considerations, and deployment recommendations. The transparency of each response was scored with either 0, 0.5, or 1 to reflect if the requested information was not, partially, or fully provided. Additionally, we assessed on an analogous three-point scale if the provided responses fulfilled the transparency requirement for a set of trustworthiness criteria from ethical guidelines. The degree of transparency and trustworthiness was calculated on a scale from 0% to 100%. Our assessment of three medical AI use cases pin-pointed reporting gaps and resulted in transparency scores of 67% for two use cases and one with 59%. We report anecdotal evidence that business constraints and limited information from external datasets were major obstacles to providing transparency for the three use cases. The observed transparency gaps also lowered the degree of trustworthiness, indicating compliance gaps with ethical guidelines. All three pilot use cases faced challenges to provide transparency about medical AI tools, but more studies are needed to investigate those in the wider medical AI sector. Applying this framework for an external assessment of transparency may be infeasible if business constraints prevent the disclosure of information. New strategies may be necessary to enable audits of medical AI tools while preserving business secrets. Artificial intelligence (AI) and machine learning applications offer the potential to transform healthcare systems by assisting healthcare providers in diagnostic decision-making ,2. VarioAbundant guidance exists for providing transparency on clinical prediction models. These include guidance to report on the intended use; used datasets; development and validation steps ,16,17,18Our framework includes a survey to prompt structured reporting about the intended use, AI model development and validation, ethical considerations, and caveats for deployment based on the existing guidelines for the transparent reporting of prediction models. After obtaining the reports, we assessed the degree of transparency by rating each survey response with either 0, 0.5, or 1, indicating if the required information was not, partially, or fully disclosed. Additionally, we assessed if the provided reports met the transparency requirements for trustworthy AI according to the ethical AI guidelines ,12. For https://www.lamapoll.de/ (accessed on 22 August 2022)), because it is compliant with the General Data Protection Regulations (GDPR).Two of our team members with backgrounds in machine learning and epidemiology compiled a semi-open questionnaire to guide transparent model reporting based on the existing guidelines for standard reporting on clinical prediction models and ethical considerations for Here, we provide a summary of the survey questions. The full survey can be found in Respondents were recruited through an open call, which was shared via mailing lists and online social networks Linked-In and Twitter, as well as personal contacts from members of our team between 8 May 2021 and 30 September 2021. Respondents stated their interest to participate via e-mail and subsequently received the participation information, consent form, and the link to the survey. Use cases were included in the analysis if the reported model was a medical AI use case using learning-based machine learning. Use cases were excluded from the analysis if the respondent discontinued the survey. Among the selected use cases, our assessor team  collected their remaining questions about the reported information and clarified those in a follow-up teleconference of 45–60 min with each respondent. Respondents were informed that they can answer clarification questions during the teleconference with ‘Not able to disclose’. If new information was disclosed during the conference, the response was added to the survey-based report. After the teleconference, our team assessed if the report provided the transparency requirements for trustworthy AI. Respondents received exhaustive written feedback, including assessment results and recommendations to improve the transparency and trustworthiness of their AI model.After completing the teleconference session, we assessed transparency across 67 questions within the sections on intended use (Section 1) until caveats and recommendations (Section 6). Box 1Q9: Was the intended use specified for a specific clinical task?Q10: Is the tool assistive, i.e., designed to include human oversight by a medical expert?Q11 and Q12: Is the tool recommended for applications in any setting for the intended use or optimized for specific settings? If applicable anywhere, was the tool sufficiently validated in external validation settings?Q15: Was the AI model output specified, and is it appropriate for the intended use?Q16: Was the development in close clinical collaboration to ensure medical integrity and safety?Section (1) Intended use of the AI model:Q29, 34, 36, 38, 39, and 42: Were the training data source, the timeframe of the data collection, the number of samples in the total dataset and subclasses, instruments and settings, and medical image sizes transparently specified?Q30: Is the training data accessible for other researchers or regulatory bodies?Q43: Was cross-sectional metadata recorded and variables reported? Q44: Was missing data reported transparently?Q45: Were the inclusion and exclusion criteria reported transparently?Q50 and Q51?: Were the training data preprocessing steps, including splitting, reported transparently?Section (3) Training data information:Q54: Was the data anonymized and personal information protected?Q55 and Q56: Did individuals give consent that their anonymized data can be used to develop this AI model? If yes, was consent revocable?Q57: Were any stated ethical principles considered during product development?Q58: Did the model deliberately use sensitive attributes to make predictions?Q59: Did the report reflect a performed assessment of fairness (performance stratification among the subgroups)? If yes, which groups were investigated, and was the performance similar across them all?Q60: Was potential harm reflected and transparently disclosed?Q61: Was the risk of bias across the subgroups mitigated? (Can be scored with one point if the performances across the subgroups were investigated but no differences were found.)Section (4) Legal and ethical considerations:Q62: Was the model performance assessed on external data?Q63, 65: Were the sizes of the total test dataset and classes transparently reported?Q64: Were the inclusion and exclusion criteria for the test dataset transparently reported?Q66 and Q67: Were the results from the model assessment shared transparently, including performance plots?Q68–74: Was the model assessment done across the quality dimensions of bias, fairness, robustness, interpretability, human comparison, and cost efficiency?Section (5) Technical validation and quality assessment:Were the caveats for deployment  reflected and transparently reported?Were underrepresented groups in data transparently reported and for further performance investigation in those suggested?Section (6) Caveats and recommendations for deployment:Six respondents from different institutions and companies reported about their AI tools using our survey. Three of these tools were excluded as use cases from this analysis for the following reasons: One use case performed a spatiotemporal analysis without a prediction target; therefore, all questions regarding training and validation were not applicable. One use case did not apply learning-based machine learning and instead used precalculated odds ratios as the prediction parameters and therefore could not answer questions on model training. The third use case discontinued the survey, because it was challenging to report all the ensemble tasks from the applied ensemble model.The remaining three AI tools were included as use cases (UC1–3) in this analysis, as they were clinical prediction models that used leaning-based machine learning. The respondents answered all questions in the questionnaire and completed the questionnaire and subsequent teleconference alone within 45–75 min. The reports cannot be shared to preserve sensitive business information of the participating companies, but we provide a summary of the reported information.The three respondents had self-reported academic backgrounds in engineering (UC2), computer science (UC1 and UC3), and natural sciences (UC1). The respondents’ experience in machine learning ranged from 1 to 7 years, and all were involved as data scientists in developing the product for 1–4 years. UC1 was developed at a large (>500 employees) company in Germany, UC2 was developed at a small company in India (UC2), and UC3 was developed at a small (<50 employees) company in Germany. UC1 reflected a prediction model to detect one type of cancer in histopathology images. UC2 was a multiclass model to predict the correct placement of endotracheal or gastric tubes using X-rays. UC3 was a model intended for routine care screening that predicts one type of cancer on X-rays. UC1 and -2 were still in the validation phase and not yet available on the market at the time of reporting. UC3 was commercially available on the market. All three use cases applied deep learning-based prediction methods.We calculated the absolute and relative transparency and trustworthiness scores for all use cases in total and across all sections of the questionnaire . UC2 andWe summarized a set of observations that strengthened or reduced the transparency and trustworthiness scores among the three use cases separated by sections in the survey.Among the eight questions, UC3 achieved the highest level of transparency and trustworthiness (both 100.0%), followed by UC1 (transparency 75.0% and 66.7% trustworthiness) and UC2 (transparency 87.0% and 83.3% trustworthiness). The intended use and clinical considerations of UC2 and UC3 were clearly and transparently specified. We scored 0 points for transparency and trustworthiness for UC2 in Q11 and Q12, because the report disclosed that the tool can be applied anywhere for the intended use without giving enough evidence to support this statement. UC1 could only partially disclose the prediction target due to business constraints.The 12 selected questions in this section were considered for scoring transparency but not trustworthiness. UC2 reached the highest (91.7%) transparency score in this section, because most details on the model development were reported, and the source code could be shared. Details on the implemented machine learning methods and the source code could not be disclosed for UC1 (66.7%) and UC3 (45.8%) due to business constraints.UC1 achieved the highest level of transparency (68.8%) and trustworthiness (57.7%), followed by UC2  and UC3 . The training data for UC1 was mixed from a publicly available dataset and data purchased from a data broker. Information on the data collection  was only partially available. Information on the timeframe of data collection and laboratory procedures for annotating sample labels were unknown. Instrument types for image acquisition and cross-sectional metadata information could not be disclosed due to business constraints. For UC2, a preprocessed and open-source dataset was used for training. While information about the data provider and data sample sizes were transparently disclosed, details on the data acquisition such as timeframe, instruments for obtaining chest radiographs, annotation, and preprocessing steps were unknown. The dataset used in UC2 did not include cross-sectional metadata variables , which we scored with transparency and trustworthiness scores of 0 (Q43). For UC3, details on the acquired dataset  could not be disclosed due to business constraints. UC1 and UC3 reported the potential of domain- and label bias in the training data. Information on splitting and selecting the data for model training was transparently shared by all use cases.The highest level of transparency and trustworthiness was achieved by UC1 (both 62.5%), followed by UC3 (both 56.3%) and UC2 (both 37.5%). All use cases used deidentified data for developing their AI model (Q54) and no sensitive attributes  as predictors (Q58), which we scored with 1 point for transparency and trustworthiness. The company of UC1 had not yet assessed the potential of performance differences across different subgroups or harm and did not apply bias mitigation steps, which we scored with 0 points for transparency and trustworthiness (Q59–61). We further scored 0 points for transparency and trustworthiness for UC2 and UC3, because they did not consider any ethical guidelines during model development. The potential harm of UC2 was missed to report in the questionnaire but was transparently disclosed during the teleconference. We counted the response from the teleconference and scored 1 point for transparency and trustworthiness. UC3 reported potential harm in the questionnaire but stated that consent was not necessary as the data was anonymized, which we assigned with zero transparency and trustworthiness points. Performance differences across the subgroups were investigated, but none were found, and the details could not be disclosed, which we scored with 0.5 points for transparency and trustworthiness. For UC2, it was unknown to the company if consent was obtained from the individuals represented in the open-source dataset (Q55 and Q56), which scored 0 points for transparency and trustworthiness. Performance differences across the subgroups were not investigated, because the necessary metadata to form the subgroups were unavailable.UC3 received the highest level of transparency and trustworthiness (both 80.8%), followed by UC2 (both 61.5%) and UC1 (both 30.8%). All use cases underwent external validation obtained from cohorts that were different from those in the training data. The total sample sizes and selection criteria of the validation data were disclosed for all use cases. For UC3, the overall results were stated (Q66) but plots (Q67) could not be disclosed due to business constraints. Fairness of the predictions (Q68) was assessed across the clinical subgroups but not across the demographic subgroups (0.5 points for transparency and trustworthiness), because the necessary demographic metadata were not available due to data privacy (GDPR) regulations. Steps to assess the model uncertainty and saved costs were reported, but the results could not be disclosed due to business constraints, for which we scored 0.5 points. For UC2, the sample size per label class in the validation data was unknown. The respondent explained in the teleconference that some label classes were missing in the validation data, potentially due to different medical practices between the countries represented in the training and validation data. The performance results were disclosed in a report including plots. The report stated that the model uncertainty was investigated by reviewing false predictions together with a clinician, which we scored with 0 points, because we did not consider this a valid approach. A quality assessment by performance stratification across the subgroups and cost analysis was not yet completed. For UC1, the overall performance results and plots could not be disclosed due to business constraints. The applied methods for a feature analysis were disclosed (Q71); the participant explained that the results were challenging to interpret, because the diagnostic outcome was rare and only a few specialists worldwide are trained to detect the pathology, which we scored with 0.5 points. The rarity of the outcome also challenged the comparison of the AI model performance to a human medical expert, which was not yet conducted. The cost analysis results were only partially reported, because the analysis was not yet completed.UC3 scored 100%, UC2 50%, and UC1 0% for transparency and trustworthiness in this section. The report of UC3 reasonably outlined the caveats for deployment application constraints. For UC2, no caveats were initially reported. During the teleconference, the participant outlined that deployment was not recommended to predict one particular outcome class due to low performance during validation, and the tool should not be deployed for children, as the performance was not investigated among this group. The potential boundaries for medical applications were not yet assessed for UC1.Transparent reporting is a crucial requirement for trustworthy medical AI, but reporting gaps are common despite the many available guidelines. With the motivation to enhance transparency, we developed and piloted a novel survey-based assessment to quantify the degree of fulfilling the transparency and trustworthiness requirements with three medical AI use cases. We discuss our subjective experiences and anecdotal evidence from this assessment.The three respondents answered all questions within 45–75 min. The teleconference was useful to clarify answers and explain why certain information was unknown or not able to be disclosed. Reporting caveats for deployment in the survey was difficult for one participant, similar to a previous observation ,34, but Our survey requires respondents to have a solid understanding of the development and validation lifecycle of the use case to avoid reductions in transparency scores due to limited knowledge. In our pilot, all three respondents were data scientists who developed the use cases and were able to answer all questions in the survey alone. Answering the survey report alone, however, might introduce subjective reporting errors, recall bias, or reflect the respondents’ perspectives. In our pilot, we were unable to verify if the given responses are true for the use case, and we raised the question of how the truthfulness of the reporting information could be guaranteed.We conducted an external audit using our survey to ensure an unbiased assessment by independent assessors. Currently under debate is which professions should conduct external audits of medical AI tools ,33. SimiOne major obstacle for our external audit was business constraints, because they prevented reporting on the intended use, implemented machine learning technology, and used datasets and validation results for two use cases. Similar to a previous report , the resOur survey-based assessment helped to systematically pinpoint reporting gaps and give specific recommendations to increase compliance with the stated guidelines in a feedback report for participants. We share a set of observed transparency and trustworthiness gaps but would like to note that these cannot be generalized to the general field of medical AI due to the small sample size.per se. The overall trustworthiness score, however, was highest for the use case that was available on the market, because it had completed all the validation steps.The reporting gaps due to business constraints also reduced the trustworthiness scores, because they prevented disclosing information on the data used for training and validation. Limited insights about the collection process of the externally acquired data were an additional reason for reporting gaps for two use cases. Obtaining informed and revocable consent for using data from individuals is crucial for trustworthy AI ,12, but It is not possible to conclude if our provided feedback report with recommendations motivated companies to improve transparent reporting and increase compliance with the stated guidelines, which should be investigated in a future study.Our assessment included a three-point scale  to quantify if the reported information fulfilled the transparency and trustworthiness requirements. It is unclear if this simple three-point scale may have biased the calculations of the relative transparency and trustworthiness scores. Future works may consider a 5-point or 10-point scale for a more granular assessment. We acknowledge that the scoring was subjective to our team and may require adaptation to assessors with different expectations. We assumed that each medical AI product has application boundaries ,31 that We provided a survey-based framework to assess to which degree the transparent reporting and trustworthiness requirements are fulfilled by medical AI products. In our pilot, our quantitative assessment pin-pointed reporting gaps and limitations to fulfill the trustworthiness criteria and helped to give specific recommendations to participants to comply with them. We observed that business constraints and limited information about external data were obstacles to providing transparent information about the three use cases. The next step is to refine our survey by including feedback from a larger group of multidisciplinary stakeholders and administer the survey to a larger sample of companies to investigate if our observations persist in the wider medical AI sector. New strategies may be required to overcome business constraints and enable the disclosure of product information for external audits. It is important to note that our assessment cannot ensure the transparency and trustworthiness of medical AI tools alone. Policies that specify the minimum requirements of transparency for trustworthy medical AI are needed to fulfill the potential of assessments helping to increase the quality of medical AI.


As smartphone technology has become nearly ubiquitous, there is a growing body of literature suggesting that ecological momentary cognitive testing (EMCT) offers advantages over traditional pen-and-paper psychological assessment. We introduce a newly developed platform for the self-administration of cognitive tests in ecologically valid ways.The aim of this study is to develop a Health Insurance Portability and Accountability Act–compliant EMCT smartphone-based platform for the frequent and repeated testing of cognitive abilities in everyday life. This study examines the psychometric properties of 7 mobile cognitive tests covering domains of processing speed, visual working memory, recognition memory, and response inhibition within our platform among persons with and without bipolar disorder (BD). Ultimately, if shown to have adequate psychometric properties, EMCTs may be useful in research on BD and other neurological and psychiatric illnesses.A total of 45 persons with BD and 21 demographically comparable healthy volunteer participants (aged 18-65 years) completed smartphone-based EMCTs 3 times daily for 14 days. Each EMCT session lasted approximately 1.5 minutes. Only 2 to 3 tests were administered in any given session, no test was administered more than once per day, and alternate test versions were administered in each session.The mean adherence to the EMCT protocol was 69.7% (SD 20.5%), resulting in 3965 valid and complete tests across the full sample. Participants were significantly more likely to miss tests on later versus earlier study days. Adherence did not differ by diagnostic status, suggesting that BD does not interfere with EMCT participation. In most tests, age and education were related to EMCT performance in expected directions. The average performances on most EMCTs were moderately to strongly correlated with the National Institutes of Health Toolbox Cognition Battery. Practice effects were observed in 5 tests, with significant differences in practice effects by BD status in 3 tests.Although additional reliability and validity data are needed, this study provides initial psychometric support for EMCTs in the assessment of cognitive performance in real-world contexts in BD. Advances in digital technology and the increasing ubiquity of both internet access  and mobiEMA methods are increasingly being used to assess cognitive performance through brief, repeated neuropsychological tasks. In this context, we refer to the concurrent administration of EMA questions and mobile cognitive tests as ecological momentary cognitive testing (EMCT). Owing to the dynamic nature of cognitive processes with influence from a variety of situational factors , single-Bipolar disorder (BD) may be a particularly appropriate condition for which to implement EMCTs given the symptom fluctuation that characterizes the disorder . MoreoveThe goal of this study was to investigate the psychometric properties of 7 newly developed mobile cognitive tests administered within an EMCT platform (NeuroUX) in adults with BD and healthy comparison participants. These tests were designed to measure processing speed, working memory, visual memory, verbal recognition memory, reasoning, and inhibitory control. We examined adherence, practice and fatigue effects, intraindividual variability, reliability and validity metrics, and performance differences based on BD status.EMA adherence in BD, depression, and schizophrenia has varied across studies, with most reporting rates of at least 65% and sometimes >90% ,27-31. TPhone type differences were examined to address device variability, such as differences in response time latencies that have the potential to affect the recorded test performance ,35. To eParticipants were recruited through flyers and web-based recruitment portals and were either individuals with BD (n=45) or demographically comparable healthy volunteers (n=21). The inclusion criteria were as follows: (1) a diagnosis of BD on the MINI International Neuropsychiatric Interview version 6.0.0  or no psThis study was approved by the University of California San Diego Institutional Review Board (protocol #172120), and all participants provided written informed consent and demonstrated capacity to consent based on a brief screening measure . At the At the baseline visit, the participants completed self-report questionnaires and laboratory-based neuropsychological performance tests, including the NIH-TB-CB and Delis-Kaplan Executive Function System (D-KEFS) Color-Word Inference Test. The NIH-TB-CB iPad version was used, which includes a mix of self-administered and examiner-assisted–administered tests. An examiner is required to be present for the full battery, which takes approximately 30 to 45 minutes to administer. The D-KEFS Color-Word Inference Test is examiner-administered and takes approximately 3 minutes to administer. The participants could either use their own smartphones or borrow a study-provided Apple iPhone 7 for home-based EMCT. The participants were trained on the EMCT protocol, and all of them  completed a laboratory-based session of the 7 mobile cognitive tests with time for technical questions and troubleshooting. Participants also received an EMCT operating manual, which we developed for this study. The manual included information on when to expect the alerts to take the mobile cognitive tests, directions for how to complete the EMA surveys and mobile cognitive tests, important tips and reminders , troubleshooting tips, and frequently asked questions.The 14-day EMCT protocol began the day following the baseline visit and consisted of 3 SMS text message notifications per day, with each notification including a link to an EMA survey and 2 to 3 mobile cognitive tests . The timThe study staff contacted participants by telephone on the first day of the at-homeprotocol, as well as if participants missed >3 surveys in a row, to increase adherence and help troubleshoot any problems. Participants were provided with study staff information, who were available Monday to Friday (and often on Saturdays) to respond to questions and help troubleshoot problems. Following the 14-day EMCT data collection period, participants returned to the laboratory to complete the C-SSRS again and a study feedback questionnaire as well as to return the study smartphone (if one was borrowed). Participants were compensated for all study visits and were given a bonus compensation of US $1 for each EMCT session completed.Participants with BD were assessed for depression using the interview-rated Montgomery-Asberg Depression Rating Scale, with total scores ranging from 0 to 60 and higher scores indicating greater severity . SymptomThe NIH-TB-CB  consistsThe development of our EMCT platform was supported in part by the National Institute of Mental Health. The mobile cognitive tests were designed using a human-centered approach and gamified to increase engagement . Test stThe cognitive domain assessed is processing speed, and the time to complete is 90 seconds. In Matching Pair, participants are presented with a matrix containing tiles of varying colors and shapes. This tile matrix starts as a 2×2 grid (4 tiles) and gradually increases to a maximum grid of 4×4 (16 tiles). Participants select the 2 tiles that match in color as quickly as possible using 1 finger. Scoring is a weighted accuracy score calculated according to the grid size shown. Faster response times are associated with the possibility of higher scores. The grid size is multiplied  and added to the running score of the previous correct trial. For example, for a correct trial with a grid size of 3×3 and a previous trial score of 246, the trial score would be calculated as follows: 3 × 3 = 9 and 9 + 246 = a trial score of 255. If a pair is incorrectly selected, the score does not change. The reaction time for each response is also recorded.The cognitive domain assessed is visual working memory, and the time to complete is variable . In Memory Matrix, participants are presented with a matrix of tiles starting with 2×2 (4 tiles), which gradually increases to a maximum of 7×7 (49 tiles). A pattern of contrasting color tiles is presented, and participants are asked to learn the location of these highlighted tiles. The contrasting pattern disappears after 1.5 seconds, and participants touch the tiles that were previously presented. This sequence begins with 1 highlighted tile, and based on performance, it can increase to a maximum of 11 tiles. The number of highlighted tiles increases by 1 for each correct response and decreases by 1 for each incorrect response. The task terminates when the participant makes 3 incorrect responses. For each correct response, the number of highlighted tiles is added to the previous correct score. For example, if the previous trial score was 12 and a correct response is given in a trial with 4 highlighted cells, this trial’s new score would be calculated as 16. The score does not change if a trial response is incorrect. Reaction time is also recorded.The cognitive domain assessed is visual working memory (primary) and processing speed (secondary), and the time to complete is variable . In Odd One Out, the participants are presented with 6 symbols and identify which symbol differs from the others as quickly as possible. For example, a trial may have 5 pictures of squares and 1 picture of a rectangle. Participants must identify the rectangle as the correct choice. Each administration contains 9 trials. The task is scored by adding the number of correct responses. This total correct response score is recorded as a measure of working memory. The reaction time in seconds for each trial is also recorded as a secondary domain.yes or no). A total of 14 different lists were presented over the protocol period, and each list was presented only once during each mobile cognitive testing session. The development of the word lists has been described in previous publications [The cognitive domain assessed is recognition memory, and the time to complete is approximately 2.5 minutes in total . Participants are presented with a list of 12 words and given 30 seconds to learn the words. The list is then removed, and participants complete a distractor task . Next, participants are presented with 24 words, which are a mix of the 12 target words and 12 foil words in a recognition memory paradigm. Words are presented one by one, and participants indicate whether the word was on the original list  that is indicated as the trick  image is presented instead of the target image. This foil image, although different, appears similar to the target image , and both the target and trick are identified in the instructions before beginning the session. Each session contains a different target and trick combination. Participants respond by tapping the target when it appears as quickly as possible; they are instructed not to tap the foil. Similar to Quick Tap 1, either image is randomly presented within 1 to 5 seconds in each trial, and the next trial begins either immediately after the image is pressed or after 2 seconds of presentation of the image if the image is not selected. The probability of foil presentation for each trial is randomly generated and ranges from 30% to 60%. Quick Tap 2 always immediately follows Quick Tap 1. Each session contains 12 trials. Both Quick Tap 1 and Quick Tap 2 were presented 9 times each over the 14-day protocol period. For each correct response, either correctly tapping the target or not tapping the trick, the total score increases by 1 for a maximum possible score of 12. Reaction time is also recorded.The cognitive domain assessed is response inhibition, and the time to complete is approximately 1 minute in total for 12 trials. Quick Tap 2 is similar to Quick Tap 1 except that, in some instances, a foil or The cognitive domain assessed is visual working memory, and the time to complete is variable . Similar to the popular electronic game Simon, the participants are presented with a 2×2 matrix of colored tiles in a fixed position: red, yellow, blue, and green. The tiles briefly light up in a random order, and participants are asked to replicate the pattern by pressing on the colored tiles in the correct order. The number of tiles that light up begins at 1 and increases by 1 with each correct response. For a correct response, the next trial contains the same pattern as the previous trial plus 1 additional highlighted tile. When an incorrect response is made, the same sequence is presented again. Similarly, if no response is made after 20 seconds, the trial is marked as incorrect. There is no upper limit on the maximum number of tiles. The session ends after 3 incorrect responses. The task is scored by summing the number of correct trials. An additional task feature is that each color plays a distinguishable tone when highlighted if the phone volume is turned on.t tests or chi-square tests were used for continuous and categorical variables, respectively. Adherence was examined by calculating the percentage of completed tests for each participant. Average adherence was compared by diagnostic status using independent t tests. To understand what we call fatigue effects , a mixed effect logistic regression was used to examine whether the participants were more likely to miss a mobile cognitive test on later versus earlier study days. An interaction between diagnostic status and study day was included to examine whether fatigue effects differed by diagnosis. The scores on 71% (5/7) of the tests were normally distributed  and squared negatively skewed variables  to normalize them before creating the composite. We then created standardized t test was used to determine if intraindividual variability differed between participants with BD and healthy volunteers. Performance on each mobile cognitive test was aggregated within each participant across all administrations to examine the average differences in performance by diagnostic group, demographics, and phone type, as well as to examine convergent validity with laboratory-based cognitive tests. Independent t tests, ANOVAs with follow-up Tukey HSD pairwise comparisons, and Pearson r correlations were used for dichotomous , >2-level categorical , and continuous variables , respectively.The intraclass correlation coefficients by group are presented in P<.05, then diagnostic status and its interaction with the study day were removed from the model to simply estimate practice effects in the overall sample. When significant practice effects were identified, spline regression models were conducted to determine whether there was a point (study day) at which the performance stabilized. All statistical analyses were performed using R . Mixed effect models were conducted using the lme4 package [Linear mixed effect regression models examined the practice effects for each mobile cognitive test to understand whether performance improved as a function of study day. Mixed effect Poisson regression was used for each mobile cognitive test outcome . An interaction with diagnostic status was initially included in each mixed effect model to examine whether practice effects differed by diagnostic status. If the interaction was not significant at  package .There are several limitations to traditional tests of cognitive abilities, including the time, cost, and accessibility of neuropsychological services. This study provides initial research support for 7 newly developed mobile cognitive tests that can be easily self-administered on personal smartphones in real-world environments.Demographic and clinical characteristics by diagnostic status are shown in t64=0.97; P=.33). In the overall sample, participants were more likely to miss a test as the number of study days increased . This fatigue effect did not differ by diagnostic status , resulting in 3965 valid and complete mobile cognitive tests among all 66 participants. Adherence did not differ by diagnostic status . Among healthy volunteers, non-Hispanic or Latino participants had a greater Odd One Out score  than Hispanic or Latino participants . Non-Hispanic or Latino healthy volunteers had a faster CopyKat reaction time  than Hispanic or Latino healthy volunteers . There were no other significant differences in mobile cognitive test performance between Hispanic or Latino and non-Hispanic participants.Participants with BD were slower than the healthy volunteers on mean Quick Tap 1 performance; mean performance did not differ on the other mobile cognitive tests, although medium effect sizes were found for Matching Pair, Memory Matrix, and the VLMT . In eachP<.001 in each case). Adherence, practice effects, convergent validity with the NIH-TB Fluid Cognition Composite Score and individual tests, convergent validity with the D-KEFS Color-Word Interference Test, and intraindividual variability are provided below.Correlations between average mobile cognitive test scores and laboratory-based cognitive performance are presented in t64=0.96; P=.34). There was a significant difference in practice effect by diagnostic status  such that scores increased within persons across days among healthy volunteers  but not among persons with BD . Intraindividual variability did not differ by diagnostic status .The mean adherence to Matching Pair was 75.3% , and adherence did not differ by diagnostic status . There was a significant difference in practice effect by diagnostic status  such that (in contrast to Matching Pair) scores increased within persons among those with BD  but not among healthy volunteers . Intraindividual variability did not differ by diagnostic status .The mean adherence in the overall sample to Memory Matrix was 75.3% . Participants with BD had lower adherence than healthy volunteers , but the difference was not statistically significant . For the Odd One Out total correct variable, there was no practice effect in the overall sample (P=.15), nor was there a difference by diagnostic status (P=.80). For the Odd One Out response time variable, participants’ average response times increased across days in the overall sample, indicating worsening performance over time . This effect did not differ by diagnostic status (P=.97). Item difficulty did not vary across Odd One Out trials. There was no difference in this effect by diagnostic status , after which response times stabilized . The Odd One Out total correct score was unrelated to the NIH-TB Fluid Cognition Composite Score in the BD group , but these variables were strongly related in the healthy volunteer group . Conversely, the Odd One Out response time was strongly associated with the NIH-TB Fluid Cognition Composite Score in the BD group , but these variables were unrelated in the healthy volunteer group . Intraindividual variability of Odd One Out total correct score  and response time  did not differ by diagnostic status.The mean adherence in the overall sample to the Odd One Out mobile cognitive test was 69.9% , and adherence did not differ by diagnostic status . For the VLMT long delay, the mean adherence was 63.5% , which again did not differ by diagnostic status . In the overall sample, although there was no practice effect for the short delay, participants appeared to recall fewer total correct words at the short delay over time . This effect did not differ by diagnostic status , after which the scores stabilized . For the VLMT long delay, there was no significant practice effect for the VLMT total correct score in the overall sample , nor did this effect differ by diagnostic status (P=.24). The VLMT was not significantly related to the NIH-TB Fluid Cognition Composite Score in either group . This shows good discriminant validity given that the NIH-TB-CB does not include a verbal recognition test. Intraindividual variability did not differ by diagnostic status .The mean adherence to the VLMT short delay in the overall sample was 66.5% . Adherence did not differ by diagnostic status  or demographic characteristics of the participants, including age , sex , years of education , ethnicity , or White (vs other) race .We also examined within-person forgetting on the VLMT. From short to long delay, participants lost an average of 3.8 words . The average number of words lost from short to long delay was not related to diagnostic status . In the overall sample, there was a slight but significant practice effect for Quick Tap 1 response time. Response times decreased within persons across days . This effect did not differ by diagnostic status  and then significantly decreased after day 3 . Quick Tap 1 was strongly related to the NIH-TB Fluid Cognition Composite Score in both groups . Intraindividual variability did not differ by diagnostic status .The mean adherence to Quick Tap 1 was 72.4% in the overall sample , and adherence did not differ by diagnostic status . In the overall sample, there was no practice effect for the Quick Tap 2 total correct response score , and there was no difference in practice effect by diagnostic status (P=.63). Quick Tap 2 scores were unrelated to the NIH-TB Fluid Cognition Composite Score in both groups . Intraindividual variability did not differ by diagnostic status .Participants were 72.2% adherent to Quick Tap 2 on average ; this did not differ by diagnostic status . In the overall sample, there was no practice effect for the CopyKat total correct response score , and there was no difference by diagnostic status (P=.13). However, for CopyKat average reaction time, there was a significant difference in practice effect by diagnostic status  such that reaction time significantly decreased  among healthy volunteers over time  but not among participants with BD , after which reaction times stabilized . The CopyKat total correct response score was strongly related to the NIH-TB Fluid Cognition Composite Score in BD participants ; these variables were unrelated in the healthy volunteer group . Intraindividual variability for the CopyKat total correct response score did not differ by diagnostic status .In the overall sample, participants were 77.9% adherent to CopyKat on average . Adherence did not differ by diagnostic status  as well as a diversity of device factors . Given that these tests are intended to be self-administered, a combination of clear verbal and visual instructions was incorporated. The main findings of this study, which were broadly consistent with our hypotheses, include (1) adequate adherence to the study protocol, with participants completing an average of 70% of the 42 EMCT administrations over a 14-day period, and no group differences in adherence; (2) differences in sociodemographic factors and mobile cognitive test performance based on ownership of an iPhone versus an Android device ; (3) modThe only test in which we found a statistically significant difference between the groups was Quick Tap 1. In both groups, low intraindividual variability was observed in Quick Tap 1; this test may be a good candidate for studies or clinical trials looking for a simple processing speed test that is not overly variable and may be most sensitive to cognitive dysfunction detection. Furthermore, the mobile cognitive tests that showed the most promise for testing real-world cognitive performance among people with BD (based on the examination of effect sizes) were tests tapping the domains of processing speed (Matching Pair and Quick Tap 1), working memory (Memory Matrix), and recognition memory (VLMT). These cognitive domains are associated with disability and mood state effects in persons with BD and, thus, may be useful tools for examining cognition change over time or response to treatment.Of note, the correlations between our mobile cognitive tests and in-laboratory neuropsychological performance were less consistent in healthy controls than in participants with BD. This is likely related to the smaller sample size of the healthy control group and the more restricted range of scores on both mobile cognitive tests and in-laboratory neuropsychological tests among the healthy controls as limited variability restricts our ability to detect robust relationships between these measures. Future work should continue to validate these mobile tests in larger samples of healthy adults. Another finding we want to make note of is the unexpected results with the Odd One Out response times, in that participants did worse on this task (slower performance) as the number of study days increased. This could be due to fatigue or lower motivation to complete the task quickly after repeated administrations. Anecdotally, participants also reported to the study staff that they found this task “boring,” which may have affected test performance with repeated administrations. Nevertheless, adherence to this task was still high at an average of 69.9% (SD 21.8%), indicating acceptability among the participants. We also saw some evidence of ceiling effects for this task as well as Quick Tap 2, and further evaluation and iterations of these tasks are likely warranted to improve psychometrics and increase task engagement. Further work would also benefit from examining whether these tasks could be useful in other ways, such as measuring effort when completing fully remote ambulatory assessments.This study adds to the limited literature on foundational psychometric research of an EMCT platform. Many of the mobile cognitive tests available for download do not have accompanying psychometric data to guide their use . PreviouThis study is not without limitations. First, the sample size was relatively small, and the participants were of a specific psychiatric population, with only 32% (21/66) of healthy volunteer participants included. Our examination of intraindividual variability by diagnostic status was underpowered to detect meaningful differences; however, we included these data as they still provide valuable insights into performance in this sample. As previously mentioned, a recently published study provided validity data for the VLMT  among a large sample of persons with serious mental illness . FurtherZ scores) with the NIH-TB Fluid Composite Score and present relationships with individual NIH-TB tests, providing proof of concept for future studies. Our sample size was insufficient to create demographically corrected T-scores with our mobile cognitive tests, which is the best-practice metric for creating composites [Second, we were unable to examine the convergent validity of our tests with traditional laboratory-based cognitive domain scores, and the NIH-TB-CB does not include >1 test per domain to generate domain score data. However, we were able to demonstrate the overall construct validity of our mobile cognitive tests and our composite score , all of which have different data speeds and connectivity, and we did not examine differences by screen size , which may affect response times. Similarly, we did not collect smartphone metadata, which could create variance in test performance, such as accelerometry, touch sensitivity and latency , andA final limitation that applies to all remote mobile cognitive testing is that it is difficult to identify suspected cheating, such as whether the participant or someone else took the tests. One way to address this is to examine score distributions and flag outliers as potential instances of noncompliance. Other options would be to include the collection of biodata such as fingerprints or face IDs. Relatedly, it is difficult to assess effort in a mobile cognitive testing platform. Previous work has found a small effect of self-reported distractions and interruptions on mobile cognitive testing performance but also that convergent validity with laboratory-based tests was minimally affected by these factors . AnotherIn conclusion, the 7 mobile cognitive tests we have presented in this study may serve as useful tools for brief, frequent ambulatory cognitive testing in a person’s everyday environment. Our data show that people with a well-characterized psychiatric disorder can and will complete self-administered mobile cognitive tests with good adherence. The tests are automatically scored, can be integrated with EMA surveys, and are available for other investigators to use; thereby, they are poised for further psychometric (including norming) studies and scalability. As the field of neuropsychology continues to migrate toward precision medicine, there are several advantages to the availability of psychometrically strong mobile cognitive tests, including complementing traditional neuropsychological assessments by gathering data outside of the controlled clinic environment, examining intraindividual variability and establishing more reliable estimates of cognitive performance over time, improving sensitivity to detect change and reducing the number needed to treat in clinical trials, and ultimately having the capability to detect brain dysfunction and risk of cognitive decline earlier than is possible with traditional assessment methods.


Omega-3 fatty acids have known health benefits for humans, but the extent to which these benefits extend to animals is still being discovered. Any benefits derived depend on achieving specific blood omega-3 levels, and the metric often used in human studies is the Omega-3 Index (O3I). It is a valid measure of omega-3 status reflecting the eicosapentaenoic acid (EPA) and docosahexaenoic acid (DHA) content of red blood cell membranes. The O3I is often calculated from an analysis of whole blood dried on filter paper; and how to convert whole blood EPA + DHA into the O3I is well established in humans, but not yet for other species. This study analyzed blood samples from dogs and cats and developed equations to estimate the O3I in both of these species. Having a validated measure of omega-3 status in these animals will facilitate further research to define exactly what blood levels of omega-3 fatty acids are optimal for each species. These levels can then be used by veterinarians and pet owners alike to gauge the amount of supplemental EPA and DHA each animal needs to achieve an optimal O3I.2 between the DBS EPA + DHA value and the O3I was >0.96 (p < 0.001). The O3I was roughly 75% lower in dogs and cats than in humans. We conclude that the O3I can be estimated from a DBS sample, and the convenience of DBS collection should facilitate omega-3 research in these companion animals.The Omega-3 Index (O3I) is the red blood cell (RBC) eicosapentaenoic acid (EPA) plus docosahexaenoic acid (DHA) content expressed as a percentage of total RBC fatty acids. Although a validated biomarker of omega-3 status in humans, little is known about the O3I status of dogs and cats; species in which omega-3 fatty acids have known health benefits. The purpose of this study was to develop equations to predict the O3I in these species from a dried blood spot (DBS) analysis. Random blood samples from 33 dogs and 10 cats were obtained from a community veterinary clinic. DBS and RBC samples were analyzed for fatty acid composition. For both species, the R The Omega-3 Index (O3I) is the red blood cell (RBC) eicosapentaenoic acid (EPA) plus docosahexaenoic acid (DHA) content expressed as a percentage of total RBC fatty acids . It has The EPA + DHA content of whole blood (whether liquid or dried) is different from that of the RBCs or of the accompanying plasma, since whole blood is a mixture of both with each compartment having a different fatty acid profile . In humaMost of the previous studies examining the health benefits of a higher vs. lower O3I have been done in humans, but there is a growing interest in quantifying the omega-3 status in animals as well, since omega-3 supplementation has been shown to have beneficial effects on a variety of health indices in dogs ,17,18,19The primary purpose of this study was to derive equations to convert DBS EPA + DHA levels into the O3I in these two species. The secondary purpose was to do the same starting with plasma EPA + DHA levels as this has been the most commonly reported measure of omega-3 status in veterinary research. To have equations that can translate published plasma EPA + DHA levels into the O3I could help to harmonize earlier studies around a single omega-3 status marker. Human samples were included in these studies as a point of reference.The dog and cat samples used for this analysis were left over from routine care of animals being seen at the All-City Pet Care Veterinary Emergency Hospital . No dietary or demographic information about the animals was collected because the purpose of the study, as noted above, was simply to compare the EPA + DHA levels in different blood compartments; therefore, understanding why levels differed from animal to animal was not relevant to this investigation. The human data were from randomly chosen, deidentified, blood samples arriving at OmegaQuant Analytics  for routine fatty acid testing.The EPA + DHA content for RBCs , DBS 1212 and plMeans, standard deviations and regression analyses were calculated using Excel (Microsoft 365). 2 values > 0.88 , and all were statistically significant (p < 0.001). It is clear from For dogs the random collection of samples from a community veterinary hospital provided a wide range of omega-3 levels, presumably because some owners were feeding their pets omega-3 enriched foods and/or supplements. Considerably fewer samples were available for cats for reasons described in the Discussion. For these two species, comparisons of DBS EPA + DHA with the RBC-based O3I metric produced the conversion equations shown in The O3I calculated from a DBS EPA + DHA value of 4% (as an example), varied by species, being highest for humans and lowest for dogs and cats . In addiRelations between the O3I and plasma EPA + DHA [the latter expressed both as a percent of total plasma fatty acids and/or as a concentration (ug/mL)] for three of the species are shown in The purpose of this study was to develop equations that could be used to convert DBS omega-3 data into the RBC-based metric, the O3I, in two companion animal species: dogs and cats, as well as in humans. The data available to us were obtained from a community veterinary practice where pet dogs and cats are brought for routine care. Because blood is more commonly drawn on dogs than cats, we obtained only one-third as many samples from the latter compared with the former.  Because we have considerably less data in cats, we have less confidence in their equation compared to that from dogs, and further studies are clearly needed to confirm especially the cat equation. As noted above, we assumed that blood levels of EPA + DHA were at steady state for all three species studied here as blood samples were collected at random. Interestingly, the equations to convert DBS EPA + DHA to RBC O3I were relatively similar for dogs and cats, but these differed substandially from the equation derived from humans. For example, a DBS EPA + DHA value of 4% would translate into an O3I of 5.9% for humans, but 2.9% for dogs and cats. The DBS values are lower than the RBC values in humans but higher in the animals. This is because, as noted above, DBS (whole blood) samples are a mix of plasma and cells, and since the plasma percent EPA + DHA is higher than the RBC EPA + DHA in the animals (vice versa in humans), the DBS percent values are higher than the RBC values. The extent to which this is a coincidence or not, and for how many animal species it would hold, is unknown. Another consequence of blood being an approximately 55:45 mix of plasma and cells  is thatWhy dogs and cats would have a lower O3I than humans is not immediately clear. Perhaps domestication and feeding them grain-based food  may be at least part of the explanation. However, just eating raw meat would not raise the O3I unless it was oily fish .The rather wide distribution of omega-3 levels in the dogs suggests a variety of intakes of EPA and DHA from different kinds of dog foods. The same applies to cats who were, at one time, believed to be obligate carnivores owing to their reported inability to convert the 18-carbon essential fatty acids, linoleic and alpha-linolenic acids, to their longer-chain metabolites  . HoweverThis study had strengths and limitations. Among the former are the use on a single laboratory methodology to derive fatty acid levels from multiple sample types. This may also be viewed a limitation to the generalized translation of omega-3 levels in one blood compartment to another using other laboratory methods. As to limitations, we have already aluded to the rather small sample size for cats. In addition, we had no information regarding the sex and age of the animals, nor any data on current intake of fatty acids. We also excluded from this study plasma samples that were visibly lipemic, reflecting the recent consumption of a high fat meal. Since chylomicrons are very rich in triglycerides , the DBS omega-3 results would be artificially lower and less translateable to RBC EPA + DHA levels. Hence, our equations should only be used in non-lipemic samples . A final limitation would be our inability (due to limited resources) to repeat the study with a validation set to test the derived equations in a new group of animals. In this study we analyzed three blood sample types  in three species  in order to create equations that could be used to estimate the RBC EPA + DHA%  from the EPA + DHA% in the other two sample types, in particular from the DBS samples. An important finding was that the equation used for humans cannot be used (accurately) in dogs and cats. Future studies in other animal species will be needed to define this conversion equation for those species. Having a method to make these conversions could help facilitate future studies that seek to define the optimal range of the O3I for dogs and cats similar to what has been done in humans, where risk (for cardiovascular disease) is highest at <4%, intermediate between 4 to 8%, and low above 8% . It seem