Over half the world’s human populations are currently at risk from vector-borne diseases (VBDs), and the heaviest burden is borne by the world’s poorest people, communities, and countries. The aim of this study was to conduct a review on VBD interventions relevant to housing and hygiene (including sanitation and waste management) in urban areas.We conducted a scoping review, which involved systematically searching peer-reviewed and grey literature published between 2000 and 2016 using five scientific databases and one database for grey literature. Different data extraction tools were used for data coding and extraction. We assessed the quality of each study using the Mixed Methods Appraisal Tool and extracted descriptive characteristics and data about implementation process and transferability from all studies using the Template for Intervention Description and Replication and ASTAIRE  tools.We reviewed 44 studies. Overall, the studies were judged to be of high risk for bias. Our results suggest multifaceted interventions, particularly community-based interventions, have the potential to achieve wider and more sustained effects than do standard vertical single-component programs. The evaluations of multifaceted interventions tend to include integrated evaluations, using not only entomological indicators but also acceptability and sustainability indicators.This review highlighted the important need for higher quality research in VBDs and improved and standardized reporting of interventions. Significant research gaps were found regarding qualitative research and implementation research, and results highlighted the need for more interventions focus on sanitation and hygiene practices.The online version of this article (10.1186/s40249-018-0477-5) contains supplementary material, which is available to authorized users. Please see Additional file Over half the world’s human populations are currently at risk from vector-borne diseases (VBDs), and the heaviest burden is borne by the world’s poorest people, communities, and countries . Thus, VAedes aegypti  re-infested most of the countries where it had been previously eliminated . We included all possible associated keywords to each key concept and appropriate descriptors for each database  independently screened and evaluated the relevance of the titles and abstracts of 20 references. This enabled the development of post hoc eligibility criteria and ensured consistency between the two reviewers  in the selection of studies. These criteria were consistently applied during the full process of screening. After the independent title and abstract screening by two reviewers , the full texts of the included articles were screened by the same two reviewers. A third reviewer (SD) resolved any discrepancies at each stage of the selection process.The inclusion criteria were: 1) presents an intervention within a routine context, as opposed to an intervention in response to an outbreak; 2) presents an intervention focused on housing and/or on hygiene (including sanitation and waste management); 3) based in an urban context; 4) published between January 2000 and July 2016; and 5) language of publication: English, French, or Spanish.Articles were excluded if they: 1) included only epidemiological or prevalence data without a link to a specific intervention; 2) included only entomological surveillance without a link to a specific intervention; 3) used an experimental design to evaluate effectiveness of potential/new vector control measures (dose-effectiveness studies); or 4) were not available in full text versions.Items that were not original research  were excluded, but references lists were checked for potential relevant original studies.We defined key concepts to assist with applying the selection criteria. ‘Vector-borne diseases’ (VBDs) are illnesses caused by vectors such as mosquitoes, ticks, and lice that transmit infective pathogens  from one host  to another . We basehousing: defined as an intervention taking place in a housing unit, defined as “a place…intended for habitation by a single household, or one not intended for habitation but occupied as living quarters by a household” , in Puerto Rico, reported no effect on the density of adult mosquitoes resting indoors with the intervention consisting of source reduction of breeding sites and application of larvicide. The authors later showed that septic tanks , significantly contributed to the maintenance of dengue virus endemicity in the region, with an estimated productivity of 4.4  results .Ae. aegypti; 5) limited source reduction efforts; 6) lack of regular perifocal residual spraying with insecticides; and 7) lack of adequate, sustained community participation beyond mere acceptance of regular control measures, for which there were high levels [Papers describing multi-components interventions were more to include descriptions of the challenges the research team encountered, in contrast to the single component intervention studies, which is likely due to their complex designs. For Gürtler et al., despite positive indicators of effectiveness, the intervention failed to maintain larval indices below targeted levels, for which they suggested seven possible reasons: 1) incomplete surveillance coverage; 2) limited residual efficacy of temephos; 3) permanent sites for mosquito breeding due to lack of change in the management of large containers for permanent water storage; 4) very favourable climatic conditions for h levels . This ish levels . There ih levels . Sustainh levels .It cannot be expected that community participation in vector control interventions is simple. As discussed by Caprara et al., the social participation of subjects and groups is often heterogeneous and shaped by historical and present-day community dynamics . For exaIn addition to community participation, achieving and maintaining field staff motivation is a major challenge in vector control activities. As found by Ocampo et al. , resistaA unique challenge was reported by the only study with negative results. Those authors concluded that print education materials may have had unintended negative effects on residents’ attitudes and household management of mosquito production, which resulted in no behavioural changes and decreased concern around VBDs . This waThis review emphasizes the need for higher quality research and improved reporting of interventions for VBDs. The trend towards more multi-component and community-based interventions is promising for enhanced effectiveness and sustainability of vector control strategies, although such interventions present important challenges that need to be considered from the outset.Overall, the included studies were judged to be of high risk for bias, with the limited information provided. Context is a key as it is essential for understanding the elements needed to ensure successful interventions and to interpret the failures of previous interventions, which should be considered by researchers and implementers . In vectMost of the studies measured their success using entomological indicators with only 25% using human morbidity indicators, despite the uncertain relationship between entomological indicators and the relative human morbidity indicators . ReductiThe sustainability of vector control interventions is a key factor when attempting to scale-up research projects towards large-scale programs or policies . InterveThe multi-component community-based interventions included in this review showed promising results, with interventions producing larger effects than standard vertical vector control programs in terms of both reductions of mosquito populations and increased sustainability , 57, 58.Theoretical frameworks are essential when designing and implementing health education programs, given the need to understand psychosocial factors underlying individual and community-level decisions and behaviours . CommuniThe interventions were largely focused on solutions to minimize vector breeding sites with only three studies focused on sanitation interventions: domestic septic tanks (Turkey) , water sImportantly, only a few non-mosquito vector studies were included (sandflies and fleas) with dengue predominantly represented, revealing substantial gaps in VBD research in urban contexts. Another significant research gap is the under-representation of Africa as only two African-based studies were included: Madagascar  and TanzDespite our best efforts, we were unable to retrieve the full texts of 14 potentially eligible articles (based on title screening). We may have also missed relevant publications in languages not included in our review. In addition, the fact that our inclusion criteria focused exclusively on urban areas may explain the smaller number of studies included in our review from Asia and Africa compared to the Americas. It should be noted that the most urbanized regions are in North America (82% living in urban areas in 2014), Latin America and the Caribbean (80%), and Europe (73%). In contrast, Africa and Asia remain mostly rural, with 40% and 48% of their respective populations living in urban areas . The preSeveral knowledge gaps were identified in this review that need to be addressed in future research. Broad community participation and social mobilization are central to the success of complex health interventions and community-based interventions are promising and should be encouraged .However, the complexity of community-based interventions and questions of sustainability of community participation requires a comprehensive evaluation strategy with both quantitative and qualitative data collection. Given that few studies in our review addressed the long-term sustainability of community-based interventions, several questions remain and more research is needed , 86. IncThere is a critical need for researchers to report the methodology and the context of interventions clearly and completely to enhance the comparability of studies and the transferability of effective interventions to other locations and contexts. Checklists such as TIDieR  and ASTASeveral studies concluded that single-component interventions, such as use of insecticides, must be considered as one of the available measures for VBDs prevention, but not the only one . MultiplAs health education is a fundamental component, particularly in settings where literacy levels may be lower, careful consideration must be given to the educational approach and materials. These principles are not new in health education research  and undeHigher quality research and standard reporting of interventions are necessary if we are to successfully control in VBDs. The findings from this review included recommendations for devoting longer times to follow-up, combining human and entomological indicators in evaluating interventions, conducting more qualitative research, and using standardized tools to report intervention methods. More implementation research is needed to better understand what vector control interventions work in which contexts and, importantly, why and how. Interventions involving horizontal approaches, community participation, and social mobilization show potential, all precautions kept due to potential bias and limitations of the present review, and require sustained intersectoral collaborations between government sectors and communities to be successful.Conduct research to include sanitation and waste management.Include more systematically indicators of human morbidity, acceptability, sustainability and implementation indicators in intervention evaluations.Develop qualitative research and implementation research.Conduct research on ecological sustainable measures to control vector populations.Use and adopt high quality standards for reporting of interventions.Generate theory-based interventions and tools.Promote sanitation improvements as an integrated disease management strategy.Redirect vertical programs towards community-based programs and intersectoral partnership.Consider the time and resources required to successfully implement complex interventions.Ensure ownership and understanding of educational materials by the community and implement interventions based on relevant health education theory.Reduce dependence on chemical insecticides to curb growing resistance to insecticides by adopting an eco-health perspective.In the context of low resource settings, may need to prioritize the most vulnerable populations.Additional file 1:Multilingual abstracts in the five official working languages of the United Nations. (PDF 872 kb)Additional file 2:Complete search strategy. (DOCX 35 kb)Additional file 3:Data extraction grid. (XLSX 157 kb)


The ecological momentary assessment (EMA) of eating behaviors represents an innovative, detailed and valid approach to capture the complexity of food intake and to overcome limitations of traditional dietary assessment methods. Moreover, EMA studies might generate a large variety of data , thereby enabling to examine concurrent exposures and events. Due to the increasing number of studies in this field of research, here we systematically reviewed EMA methods for the assessment of dietary intake in epidemiological studies, and discussed implications and perspectives for future research. Our study summarized several protocols and platforms that may be applied to assess diet in terms of eating frequency, choices, and habits. Nearly 38% of studies used an event-contingent strategy by asking participants to report foods and beverages consumed in real-time at each eating occasion. Instead, approximately 55% of studies used a signal-contingent prompting approach that notified the participants to record their dietary consumption. The remaining studies used a combination of event- and signal-contingent protocols to compare their accuracy or to improve the assessment of dietary data. Although both approaches might improve the accuracy and ecological validity of dietary assessment—also reducing the burden for participants—some limitations should nevertheless be considered. Despite these limitations, our systematic review pointed out that EMA can be applied in various fields of nutritional epidemiology, from the identification of determinants of dietary habits in healthy people to the management of patients with eating or metabolic disorders. However, more efforts should be encouraged to improve the validity and the reliability of EMA and to provide further technological innovations for public health research and interventions. In an article published in 2018 in the British Medical Journal (BMJ), Mozaffarian and colleagues have described how the history of modern nutritional science has shaped our understanding of diet and health [In parallel with undeniably great strides in nutritional science, several methods for assessing dietary data—in terms of frequency of consumption, quality, and quantity—have been developed in the last decades. The common consensus is that biochemical and molecular biomarkers are the most objective surrogate estimates of dietary exposure since they are independent of memory and free of reporting biases . HoweverFor these reasons, further efforts are required to develop innovative, thorough and valid approaches to capture the complexity of food intake and to overcome limitations of traditional dietary assessment methods. In our opinion, the ecological momentary assessment (EMA)—described for the first time by Shiffman and colleagues in 2008 —has the Due to the increasing number of studies in this field, we aimed to systematically review EMA methods for the assessment of dietary intake in epidemiological studies and to discuss implications and perspectives for future research. To reach this aim, we conducted a systematic review of epidemiological studies using EMA tool for assessing dietary intake and behaviors. Next, we summarized different EMA approaches and potential applications in the field of nutritional epidemiology.The authors performed a systematic literature search in the Medline, EMBASE, Web of Science, SCIELO, KCI Korean Journal, and Russian Science Citation Index databases from inception to June 2019. The literature search was limited to articles written in English. The following terms were used:  AND (“diet” OR “food” OR “nutrient” OR “nutrition” OR “eating habits” OR “dietary habits” OR "dietary assessment" OR "nutrition assessment"). The Authors also searched the reference lists of selected articles to include all relevant studies. The preferred reporting items for systematic reviews and meta-analysis (PRISMA) guidelines were followed .The authors independently selected the retrieved studies if they were consistent with the following criteria: (i) observational or interventional epidemiological studies (ii) using EMA approaches (iii) to assess the characteristics of eating episodes  or the consumption of foods or beverages . By contrast, (i) systematic reviews, (ii) unpublished studies, (iii) and those that exclusively assessed eating disorders  were excluded. From each study, the authors independently extracted the following information using a structured review template: first author’s last name, year of publication, study name, aims of research, platform, and device, monitoring periods and duration of EMA, prompting strategy , and dietary data collected. For studies that applied signal-contingent EMA, the Authors also extracted information about prompting approach , frequency, and period of validity of each prompt. If several articles were published on the same study, data were extracted from all articles and reported as a single study. Inconsistencies between the two Authors in the study selection and data extraction phases were resolved through discussion. The extracted data were synthesized narratively and tabularized with the intent of summarizing EMA protocols for assessing diet and to discuss their potential applications in nutritional epidemiology. The PRISMA checklist is reported as n = 40).n = 29), others recruited subjects with overweight or obesity (n = 6), anorexia nervosa (n = 2), type II diabetes (n = 2), or psychological disorders (n = 1).The main characteristics of included studies are summarized in n = 25), 2 studies compared the reliability of a smartphone-based EMA approach with a paper-pencil recording, while the others used personal digital assistants (PDAs), web-based surveys or other devices. Almost all the studies monitored participants during one period of time (n = 37), while the remaining 3 studies included up to 6 waves of EMA. The duration of each monitoring period ranged from 3 days to 12 months, but about two-thirds of the studies (n = 24) monitored their participants from 7 to 14 days. With respect to EMA approaches, nearly 38% of studies (n = 15) used an event-contingent strategy by asking participants to report foods and beverages consumed in real-time at each eating occasion. Instead, approximately 55% of studies (n = 22) used a signal-contingent prompting approach that—through researcher-initiated signaled prompts—notified the participants to record their dietary consumption.Approximately 63% of studies used electronic EMAs administered on smartphones . The study by Hingle and colleagues was one of a kind, collecting consumption of foods and beverages using hashtags on Twitter [n = 15). The remaining studies asked their participants to define the type of each eating episode (n = 4), to report frequency of eating and snacking (n = 1), or to capture food images (n = 2). Notably, the latter studies used food images to assess nutrient or total energy intake.Overall, 8 event-contingent EMA studies asked their participants to define the type of each eating episode . Among these, 4 studies also asked to define the consumption of specific foods or food categories from a predefined list. Instead, another study exclusively recorded the consumption of predefined food types or food groups. The other 5 studies that used an event-contingent protocol asked their participants to capture images or videos of each eating episode, sometimes followed by a brief text/voice description  ,66. At sThe first thing we noticed was that few studies validated EMA protocols against current dietary assessment methods or nutritional markers. Among these, Chmurzynska and colleagues compared the feasibility of a signal-contingent EMA protocol for measuring the frequency of consumption of high-fat foods with standard retrospective methods, such as the Block Screening Questionnaire for Fat Intake . InteresAs mentioned above, women have a crucial role in food choice by providing meals for their families and making up the majority of the workforce in food-related jobs, health care and education . Thus, tThe TOPS recruited 277 mother–child pairs and applied a random signal-contingent assessment of child consumption of meals, snacks or drinks . In 2018This tool was also applied to examine mothers’ and children’s dietary intake during physical and sedentary activities . ExperieIn 2013, two independent research groups were the first to assess dietary habits in young people through an EMA approach. In particular, Spook and colleagues examined the feasibility, usability, and ecological validity of a mobile EMA application to assess physical activity and dietary intake among Dutch vocational education students . AlthougFood craving—an intense desire to eat a specific food—is certainly one of the main triggers of an exaggerated intake of energy-dense foods, such as snacks and dessert foods. Plausible explanations include ingredients of certain foods , low serotonin levels affecting the centers for appetite, or the release of endorphins as a result of a state of stress . To addrFood craving may also dangerously lead to uncontrolled overeating, especially in overweight and obese people. Despite this evidence, contextual factors related to maladaptive eating behavior in youth still remain poorly clarified. To examine internal and external cues related to perceptions of overeating and loss of control Goldschmidt and colleagues integrated different EMA approaches among 40 overweight or obese children . SpecifiFrom a broader perspective, some studies applied EMA strategies to evaluate the main determinants of unhealthy behaviors among students. In this context, to the best of our knowledge, one of the largest studies is the Social impact of Physical Activity and Nutrition in College (SPARC) study, which recruited 1450 first-year college students over an academic year . The maiMeal frequency and snacking vary dramatically across cultures, change with time, and may profoundly affect metabolic and cardiovascular health ,78. A reOther studies used EMA for understanding the main determinants of snacking and its relationship with health and diseases. Indeed, every day, we are faced with external stimuli  that prompt us to eat. In 2015, Schüz and colleagues provided the first attempt to describe environmental and affective factors that were associated with eating and drinking using an EMA approach . For insThe cognitive processes responsible for effortful behavioral regulation are known as executive functions. The Snacking, Physical activity, Self-regulation, and Heart rate Over Time (SNAPSHOT) study applied an EMA approach to examine the relationship between snacking, energy expenditure, and executive function . ResultsWhile several behaviors, such as physical activity and sleeping, have been shown to have stress-reducing and beneficial effects on wellbeing and health ,80,81, tUnderstanding the relationship between behaviors and emotions might be of particular interest in people with physical or mental health disorders. For instance, researchers of the Mobile Community Health Assistance for Tenants (m.chat) project used EMA to investigate the association of daily emotional states with physical activity, diet, social interaction, medication compliance, and tobacco smoking, among people with a history of chronic homelessness who participated in a health coaching program . HealthyThe first studies using EMA approaches in disease states were those related to anorexia nervosa, which increased our knowledge of different emotional, behavioral, and environmental features that affect disturbances in eating behavior. In 2014, Goldschmidt and colleagues asked women with anorexia nervosa to complete an event-contingent EMA of daily eating- and mood-related patterns . Their fYet, at the between-person level, people who commonly ate in the absence of hunger reported less overeating. Moreover, eating in the evening was associated with several contextual factors, such as overeating, alcoholic drinking, eating alone, eating because others are eating, and eating while watching television . Next, tThis raises the need for an integrated approach to understanding those factors that affect eating behavior, which may help us in improving preventive strategies against the obesity epidemic. Further studies used EMA to prepare obese patients for bariatric surgery  or to moWith respect to diabetes, a pilot study by Miller and colleagues aimed to monitor the adoption of a lower glycemic index diet among six patients with type II diabetes for six weeks . Using aOur systematic review summarizes current EMA methods used for the assessment of diet in nutritional epidemiology, and their potential applications for understanding the relationships between eating behaviors, health, and disease. When deciding to design an EMA study, several protocols and platforms may be applied to assess diet in terms of eating frequency, choices, and habits. We found that the majority of studies used smartphone-based EMA, even if other devices may be applied. The use of smartphones and other electronic devices definitively enhances the applicability and validity of EMA in nutritional epidemiology. However, it also poses some challenges for scientists and their funds. Indeed, EMAs administered through electronic devices are more expensive than other paper-and-pencil surveys, especially for those studies that provide participants with loaned devices. Whilst this ensures consistency in the administration of EMAs, it also may limit overall sample size. An alternative would be to use open source software, which allows tailoring EMA protocols to specific research topics. Moreover, several EMA protocols and prompting strategies might be applied, including event-contingent and signal-contingent approaches, which can be occasionally supported by an end-of-day survey. Although both might improve the accuracy and ecological validity of dietary assessment—also reducing the burden for participants—some limitations should nevertheless be considered. For instance, technical issues with the platform used or low user compliance might impair the assessment of dietary intake. The majority of studies included in the present systematic review did not report user response rate and compliance. However, it has been reported that compliance was higher in electronic groups than in people who received paper-and-pencil survey . By contDespite these limitations, our systematic review points out that EMA can be applied in various fields of nutritional epidemiology, from the identification of determinants of dietary habits in healthy people to the management of patients with eating or metabolic disorders. However, more efforts should be encouraged to improve the validity and the reliability of EMA, overcoming limitations of traditional dietary assessment methods, and to provide further technological innovations for public health research and interventions.


Methods: Eight Denver VAMC specialty care providers and three project nurses underwent telehealth training and obtained appropriate equipment. To identify in-person visits eligible for substitution of video visits, project nurses review charts of CNH Veterans, consult directly with Veterans, and obtain recommendations from staff. Project nurses serve as tele-presenters within the CNHs, while VA specialists provide care from the VAMC. After each visit, team nurses coordinate care with and deliver specialty care recommendations to CNH staff. Results: We assessed clinical, business, and technical domains of the Vet Connect project, and utilized process mapping to identify barriers and facilitators to implementation. Clinically, starting on 26 June 2017 through 1 June 2018, N = 203 video visits have been conducted with 11 different CNHs in three subspecialties: geriatrics, palliative care, and mental health. These visits generated 49 referrals for 37 Veterans. Fiscally, cost analyses indicate that per visit, the health care system saves an estimated $310. Technologically, the success rate was 83%. Process mapping helped identify facilitators and barriers to implementation of the telehealth program, including cultivating buy-in from key stakeholders , communication allowing for ongoing program adaptation, and building relationships. Conclusion: Subspecialty care delivery to nursing homes using video visit technology in the Vet Connect program is feasible using centralized organization to coordinate complex clinical, business and technical processes. Vet Connect has proved sustainable and has potential to expand within and outside of the VA.Veterans residing in Veterans Health Administration (VA) contracted Community Nursing Homes (CNHs) receive primary care from the CNH they reside in, but often travel to Veterans Affairs Medical Centers (VAMCs) for specialty care services. The Vet Connect project is a quality improvement project aiming to implement video technology to support access to specialty care.  Telehealth services are beneficial for those who reside in rural communities and those who find travel of any distance to a medical center difficult . TeleheaThese efforts are consistent with the VA Strategic Plan for 2013 to 2018 which states an overall objective to provide Veterans personalized, proactive, patient-driven health care . To accoWe implemented the current quality improvement project to provide VA specialty medical care and mental health support via telehealth to a subgroup of Eastern Colorado Health Care System (ECHCS) Veterans who reside in contracted Community Nursing Homes (CNHs). The few studies published providing elderly or nursing home care via telehealth suggest positive health outcomes, cost benefits, and staff satisfaction for patient glycemic management ; rehabilWhile literature shows benefits of telehealth to the geriatric population , and recThe aims of this paper are to: (a) Describe the implementation process of this quality improvement project delivering telehealth to geriatric Veterans; (b) explain facilitators and barriers encountered during implementation; (c) report utilization and cost measures; and (d) discuss the refined process and plans for sustainability of the Vet Connect program. The following materials and methods were utilized in implementation of Vet Connect; staffing, stakeholder engagement, administrative efforts, technology, and processes for visit preparation and commencement and post-visit follow-up. To illustrate materials and methods, our team created process maps A1 and A2 to clearly identify processes, barriers, and facilitators of program implementation; Vet Connect initially provided specialty care services to CNH Veterans within geriatrics and palliative care. We then identified mental health as a needed service, and providers in palliative care mental health and geriatric mental health were added to the program. Currently, Vet Connect partners with two geriatricians ; two psychologists ; three palliative care specialists; and one psychiatrist. Three part-time nurses act as tele-presenters during CNH visits. They coordinate care prior to and following visits by reviewing clinical issues with the CNH staff, obtaining a list of medications, and facilitating care coordination after visits. Importantly, all staff involved (except one psychologist and the project psychiatrist) previously worked at the Denver CLC prior to its closing, thus relationships between many staff and many Veterans within the program already existed. To secure buy-in from key stakeholders, we visited CNHs in person to ascertain needs and generate communication between CNH and VA staff. We also ensured the telehealth services offered filled gaps in care without duplicating existing services. In addition, we conducted meetings with VA staff, providers, social workers, and nurses to coordinate efforts and generate partnerships at multiple time points during the first year of Vet Connect. Telehealth leadership at the Denver VAMC assisted in the creation of appropriate clinics and consult notes and developing Memorandums of Understanding and Telehealth Service Agreements  between the VAMC and CNHs. Training and credentialing providers ensured appropriate implementation of services. In accordance with contracts between the VAMC and CNHs, primary care is provided by the CNH, while other specialty care is provided by the VAMC.Denver VAMC telehealth program specialists educated the project team on use of CVT tablets to conduct visits between the CNH and the VAMC. To initiate a visit, a team provider dials the specified CVT tablet number into their computer enabled with Jabber software . MeanwhiThere are several steps involved to prepare for a visit. Nurses identify suitable Veterans for a video visit . Suitable Veterans are defined as those Veterans who are in VA CNHs, who have or need a face to face visit that could be substituted for a video visit, and whose provider agrees to a substitution. Furthermore, as the project progressed, Veterans with cognitive impairments such as dementia were deemed inappropriate for video visits as communication was difficult. The process and preference for video over phone appointments, especially in building rapport and reading facial expressions, has been preferred in past research . After coordinating times between team nurses, provider, Veteran, and in some cases, the Veteran’s family, the visit time and date is scheduled. Team nurses will arrive at the CNH an hour prior to the scheduled visit to obtain patient vitals and medical information to best assist as the eyes, ears, and hands of the remote specialist. At the appropriate time, the nurse meets with the Veteran in the CNH, in a private and quiet setting, and the provider calls into the CVT tablet using Jabber software . First, a team nurses update the Veteran’s VA electronic medical records and input any referrals or follow-up appointment consultations that were generated during the visit. Second, the team nurses communicate to the CNH staff the recommendations from VA providers either in person or through fax. Lastly, visits and field notes are tracked in a secure database. We used micro-costing techniques to estimate the financial resources saved from delivering a video visit as compared to delivering a visit face-to-face . We engaWe defined outcomes in terms of the following domains: clinical utilization, business, and technical. We also created process maps to best assess pre- and post-visit processes, as well as provide guidance for future implementation efforts. Vet Connect model outcomes during the 11 months following implementation are described by domain below in N = 45). Referrals for ancillary services or other medical or mental health treatments totaled 49 referrals for 37 Veterans. Video visit times ranged from 10 to 60 min. We conducted 203 visits in 11 different CNHs, between 26 June 2017 and 1 June 2018 see . SpecialCost domains included: Staffing time to coordinate travel and schedule appointments; transportation costs; video visit technology including CVT tablets and provider video cameras; and office space. Based on these domains, the cost of delivering a visit face-to-face was approximately $187.34 for the VA and $171.01 for the CNH. The cost of delivering a video visit was approximately $30.67 for the VA and $5.70 for the CNH. Therefore, for every video visit substituted for a face-to-face visit, CNHs’ cost savings are estimated at $160, and the VA’s cost savings are estimated at $150. Thus, in total, the health care system saves an estimated $310 per video visit substituted for a face-to-face visit.Vet Connect visits to date have an 83.3% technical success rate . This is defined as a visit with zero reported technical difficulties: No dropped calls or reports of poor connectivity such as screen freezes and audio lag. Primary reasons for technology failure included poor connectivity using the tablet’s 4G data plan and Veteran difficulty hearing the provider. External headphones were added to enhanced volume for Veterans with hearing impairment. During 12 of the 34 visits with technical difficulties, nurses utilized a cellphone to call the provider and complete the visit instead of using the video visit technology. The specific steps involved in establishing and maintaining Vet Connect are outlined in process maps developed continuously throughout the implementation process see . We usedVA leadership support and informal feedback from and regular contact between VA and non-VA partners facilitated project implementation. Frequent team communication and collaboration was necessary to adapt program components on a weekly basis. A dedicated project coordinator centralized organization of tasks and communication. Denver VAMC leadership support was provided by: the director of palliative care, chief of psychology, the facility telehealth coordinator, and the VA-based CNH program social work supervisor. Buy-in from these key stakeholders allowed for guidance in understanding VA policies related to obtaining appropriate technology equipment, swift onboarding of staff, and garnering provider support.Frequent in-person, phone, and email conversations with CNH and VA staff facilitated program implementation by timely identification of gaps in access to subspecialty care and rapid program modifications to fill identified gaps. For example, an in-person meeting with a CNH transportation coordinator revealed the need for more timely specialty care referrals. In this case, we identified that newly admitted Veteran residents waited weeks to establish a primary care provider who could enter a subspecialty care referral. In fact, the need was only for the subspecialty care referral not a primary care provider because they already had a primary provider at the CNH. The Vet Connect program filled this need specifically by assigning a geriatrician as a primary point of contact who could make needed subspecialty care referrals and acts as a connection between the VA and CNHs, lending to faster referral times. This timely identification and response to identified needs facilitated rapid program implementation.Prior relationships between providers and team nurses, who had worked at the Denver VAMC CLC and Veterans who had resided there allowed for informal communication and feedback between VA provider stakeholders and project staff. Further, weekly team meetings provide essential time for problem solving, strategizing, and incorporating feedback received from both VA and non-VA partners. Administrative tasks including the development of a scheduling system for clinics proved complex and the timeline for building clinics took longer than initially expected. Continuous communication with VA telehealth and Health Administrative Systems (HAS) staff was necessary to successfully build clinics for each subspecialist. Furthermore, allotting appropriate workload credit to providers proved difficult in the Denver VAMC system for similar reasons, as adding or modifying new clinics for providers, again, took time and persistent follow-up. Given that workload credit directly contributes to provider performance reporting, accuracy is essential to accurately capture providers’ workload credit and thus proved to be a key step to clarify. As one of the first VA to non-VA facility telehealth programs in the Denver VAMC, novelty of the program necessitated increased need for team problem solving and collaboration with facility partners to develop tailored processes and documentation. Building relationships with the CNHs proved challenging, due to high turnover of staff managing multiple collateral duties leading them to be overburdened . One othVideo visits between CNHs and VA providers are feasible based on completion of 203 visits across three subspecialties in 11 different CNHs, with technical success in the majority (83%) of visits over the 11-month implementation period of the Vet Connect project. Given that cost savings for CNHs are estimated at $160 and $150 for VA, based on substitution of a video visit for an in-person visit, a focus on sustaining the program and overcoming the barriers identified is justified. As the program has proved, sustainability will be a critical next step. This is especially true because such video visit referrals are becoming more common in the delivery of health care nationally . SustainTechnical metrics indicate that CVT tablets utilizing Jabber software are a reliable medium to host video visits . Poor coLimitations of this project’s methodology include sample size and assumptions made in our initial cost analyses. Firstly, savings only occur if the video visit is substituted for an actual face to face visit, which may not always be the case. Secondly, Vet Connect visits often increased access to care, in that video visits increased the number of subsequent visits and referrals to the VA for CNH Veterans. It is important to note these assumptions in cost analyses. However, future research will attempt to answer whether Vet Connect saves costs longitudinally, given that increased access to care may cost more in the short-term and pay off in the long-term in lower rates of hospitalization . Lastly,Future directions for this project include conducting qualitative interviews with partnered VA specialty care providers, project team nurses and staff, and non-VA facility staff to better understand facilitators and barriers of the program from their perspectives. Future qualitative publications from these interviews will also shed light on potential for program sustainability and expansion. We also plan to disseminate lessons learned about facilitators and barriers of Vet Connect implementation to ideally influence best practices nationally. While Vet Connect illustrates many VA-specific steps, implementation lessons learned can inform non-VA health care entities, particularly those serving rural areas with multiple nursing homes or outpatient clinics who have interest in implementing a similarly focused geriatric telehealth model. Referencing our process maps can inform their efforts. Given the growing number of elderly Veterans and citizens in need of long-term care and an uptake in use of telehealth nationally, programs like Vet Connect will be useful in offering two, timely, and cost-effective specialty care while saving travel time to main medical facilities.


Cohorts of millions of people's health records, whole genome sequencing, imaging, sensor, societal and publicly available data present a rapidly expanding digital trace of health. We aimed to critically review, for the first time, the challenges and potential of big data across early and late stages of translational cardiovascular disease research.We sought exemplars based on literature reviews and expertise across the BigData@Heart Consortium. We identified formidable challenges including: data quality, knowing what data exist, the legal and ethical framework for their use, data sharing, building and maintaining public trust, developing standards for defining disease, developing tools for scalable, replicable science and equipping the clinical and scientific work force with new inter-disciplinary skills. Opportunities claimed for big health record data include: richer profiles of health and disease from birth to death and from the molecular to the societal scale; accelerated understanding of disease causation and progression, discovery of new mechanisms and treatment-relevant disease sub-phenotypes, understanding health and diseases in whole populations and whole health systems and returning actionable feedback loops to improve  existing models of research and care, with greater efficiency. In early translational research we identified exemplars including: discovery of fundamental biological processes e.g. linking exome sequences to lifelong electronic health records (EHR) (e.g. human knockout experiments); drug development: genomic approaches to drug target validation; precision medicine: e.g. DNA integrated into hospital EHR for pre-emptive pharmacogenomics. In late translational research we identified exemplars including: learning health systems with outcome trials integrated into clinical care; citizen driven health with 24/7 multi-parameter patient monitoring to improve outcomes and population-based linkages of multiple EHR sources for higher resolution clinical epidemiology and public health.High volumes of inherently diverse (‘big’) EHR data are beginning to disrupt the nature of cardiovascular research and care. Such big data have the potential to improve our understanding of disease causation and classification relevant for early translation and to contribute actionable analytics to improve health and healthcare. These record data are increasingly being used for research, beyond the primary purpose for which they were collected. ‘A new era of data-based and more precise medical treatment’resources exist for cardiovascular disease research?(i) What data challenges and barriers to realizing these opportunities?(ii) What are the early translational research including discovery science, drug development and repurposing, precision medicine?(iii) What is the potential of such data in late translational research including learning health care systems, real world evidence, citizen-centred and public health?(iv) What is the potential of such data in We review four areas in relation to big health record data:‘Big data’ are usefully characterized by ‘variety, volume, velocity, and value’ . EHR are intrinsically ‘big’ due to their complexity (‘variety’) and numbers of patients and amount of information on each patient (‘volume’) and are collected for a variety of purposes .Figure 1 illustrates the variety and volume of data showing the relation between scale (number of people) and depth of phenotypic and omics information in different settings: national population-based, hospital-based, and disease or procedure based registries. The amount of phenotypic information in hospital EHR is much greater than any single registry; but such deeper hospital EHR data, has been challenging for researchers to access at scale.Figure 1 makes an important distinction between those record resources with, and without genomic information. Boxes 1 and 2 provide further details of these resources which may be accessed for translational collaborative research. Biobanks and genomics consortia increasingly rely on EHR linkages for the ascertainment, validation, and phenotyping of not only specific disease outcomes but also the entire longitudinal phenome, as captured by an growing array of digital sources.,Boxes 1 and 2 are making increasing use of such data sources including the physical environment, consumer information, socioeconomic and behavioural factors,The resources illustrated in ,In order to exploit these resources for translational research there is an increasing use of computer science approaches to harness publicly available curated knowledge in different fields including: the medical literature (e.g. PubMed), catalogues of genetic variant-phenotype associations  and cross-national collections of EHR cohorts in 100 000 000.8 voxels and a clinical grade (×30) whole genome sequence provides 15 Gb of data.,Higher resolution enquiry of common and rare diseases , demands higher sample sizes: 5000 people in the Framingham cohort, 500 000 in UK Biobank,whole-system relevance when population and healthcare system records are used: for example, in countries with nationwide health record systems, EHR are the only way of obtaining large scale representative samples. The velocity of big data is an opportunity for real time analytics with intelligent feedback loops to improve healthcare systems and individual decision making. The exploitation of such rich big record data sources is more efficient and cost-effective compared with traditional researcher-led approaches since for example, in EHR cohorts the cost to research funders of baseline and follow up data collection is zero . The costs however of collating, cleaning and curating these data and meeting the challenges outlined below are substantial and are further elaborated below.The value of diverse, high volume data is already changing the way that health care is delivered and is yielding insights in early and late translation . There are many sources of value in big data, beyond the immediate scientific dimensions of scale and longitudinal phenotypic resolution. These include the In realizing the opportunities of such diverse, large volume data there are formidable challenges. These include: knowing what data are potentially available, information governance, models of data access (responsible data sharing), building and maintaining public trust, developing standards for defining disease, and developing tools for scalable, replicable science and equipping the clinical and scientific work force with new inter-disciplinary skills.Challenge: The quality of EHR data can be said to be ‘in the eye of the researcher’. In any given dataset the amount of missing data, often not missing at random, or inaccurate data, may prohibit valid inference for some but not all research questions. Linked EHR, subject to robust pre-processing and cleaning, have been shown to provide valid measures of risk factors and a wide range of diseases, and therefore offer a common scaffold on which to build specific research questions.Solution: A data mantra is ‘collect once, use many times’: and there are calls to make good quality clinical record keeping, as ‘research grade data’. It should be noted that accurate and complete recording, though desirable, does not replace appropriate study design or resolve limitations such as confounding by indication. Validity and data quality may be assessed in multiple ways including:Cross referencing multiple sources of data in the same individuals (each with their own strengths and limitations): e.g. for acute myocardial infarction linking four national population based sources  (CALIBER) shows the positive predictive value and prognostic validity of cases defined in different sources, and allows development and sharing of phenotypic algorithms.,International comparisons: for example, EHR cohorts in heart attack survivors using ICD codes from different versions  and different countries  demonstrated for 12 risk factors consistent relative risks associations with fatal and non-fatal long term outcomes.Genomic approaches to validating case definitions: across 1000s of hospital ICD codes (‘phenome-wide’), reproduce associations from genome wide association studies obtained one phenotype at a time,,Table 1, Denny et al.Figure 2).Challenge: ‘Genome browsers’ facilitate discovery in biological sciences, but currently the contents of the big data tapestry and whether they are suitable for a particular research purpose are hard to uncover within a researcher's own country, let alone across different countries . Solution: If big data are to disrupt current research models then there is a need for searchable catalogues of data, metadata, feasibility counts  and access arrangements. The creation of public, standards-driven metadata and data portals can assist researchers in locating the right dataset for their research question and obtaining up to date details on data availability and accessibility. For example, the IMI-funded European Medical Information Framework (EMIF) data catalogue contains information on over 300 data sources ranging from EHR, consented cohort studies, and surveillance datasets.Challenge: The information governance of big health data resources presents major challenges. The need for protecting privacy, confidentiality, discrimination and other potential harms is vital. However how the regulatory environment proportionately balances these concerns with the potential benefits of data sharing  is evolving.Solution: Broad consent models, such as those in UK Biobank, have an important role, recognizing that it is not possible to stipulate all the potential research uses of data, nor how they will change. Some have argued that a new social contract is required with trusted use of data under innovative, proportionate governance delivering benefits to patients and public.Challenge: Despite exhortation from funders, journals and the public to share data, all too often this does not happen. Once researchers have permissions to access data, the mode of data sharing may pose challenges to the researcher.Solution: Data sharing may involve: (i) material transfer agreements with data being physically shared e.g. UK Biobank; (ii) role-based secure remote access; (iii) distributed analyses where data remain stored in individual sources. The Global Alliance for Genomics and HealthChallenge: There is a lack of an international framework for defining, phenotyping, sub-phenotyping and discovering disease phenotypes in the context of health records. There are multiple controlled clinical terminologies and ontologies , but how these terms should be combined to define meaningful entities, let alone how they should be combined with research data is unclear. Currently many diseases lack internationally agreed criteria (preferably in a machine-readable format) for defining cases and non-cases; acute myocardial infarction, type 2 diabetes are exceptions. Current definitions of many diseases such as HF, AF and ACS span heterogeneous groups of patients and describe syndromes only rather than definitions based on understanding of molecular mechanism.Solution: Sharing, validating and refining replicable, scalable EHR phenotypic algorithms requires international efforts.,Challenge: There is a wide array of relevant approaches from quantitative disciplines  and from biological disciplines: until recently these have seldom been focused on big health record data.Solution: While there are 7 million hits per day on the European Bioinformatics Institute website; such national and international resources for health informatics are lacking. There is a need for organizations to be established which provide the analogous reference data, tools and methods in health informatics in general,Challenge: Few clinicians and health care professionals have had formal training in informatics, data science, (computer) coding, software development or other increasingly relevant skills. In many countries there are large shortfalls in the number of data scientists that have been trained.Solution: National efforts are likely to be important to substantially increase the number, and change the kind, of people required to deliver data-based medicine: hybrid professionals,, data scientists, data wranglers, and data-savvy health care professionals.,In this section we provide selected exemplars of the potential of big health record data arising from the variety, volume and value of the data being realized and how big data are contributing to scientific advance in cardiovascular medicine from discovery of underlying disease mechanisms, disease taxonomy, of treatment relevant sub-types of disease which underpin drug development, and precision medicine.,Table 1, Denny et al.,Figure 2).It is important to note that it is challenging to provide deep mechanistic insight in large scale EHR data resources given the limited availability of genetic information in sufficient depth. Bespoke, recallable investigator-led studies such as East London Genes & Health  have mainly been analysed with focus on a small number of diseases or using large-scale approaches without time consideration, exceeding a few years. Investigators performed a discovery-driven analysis of temporal disease progression patterns using data from an electronic health registry covering the whole population of Denmark. Utilizing the entire spectrum of diseases, they convert 14.9 years of registry data on 6.2 million patients into 1171 significant trajectories. Key diagnoses such as gout and chronic obstructive pulmonary disease (COPD) were identified as central to disease progression across many of these trajectories and hence important to diagnose earlier. Such data-driven trajectory analyses may be useful for predicting and preventing future diseases of individual patients.Table 1, Shah et al.,,,Most cardiovascular diseases  have syndromic descriptions and labels, which may span multiple underlying pathological disease processes.2.More drugs are required to prevent and treat cardiovascular diseases. Since 2000, the FDA has approved only two new classes of cardiac drugs with widespread application: P2Y12 receptor inhibitors  and novel oral anti-coagulants . Costly, late drug failures occurring within phase III trials have been recently seen for CETP inhibitors which raise HDL-cholesterol (HDL-C),Table 1, Stitziel et al.,EHR-DNA resources may play an increasingly important role in drug discovery, genomic drug target validation, marker validation and in drug repurposing. For example, NPC1L1  For example, examining 778 disease phenotypes based on ICD codes in the EHRSCN10A. This variant is associated not only with the anticipated arrhythmias, but (possibly) also with unanticipated diseases, here cholecystitis. Recent interest has been to scale this approach to systematically evaluate drugs against a wide range of untested diseases. To be successful this would require substantially larger EHR-DNA resources incorporating longitudinal disease trajectories from big record dataIdentifying novel disease indications for already approved drugs (repositioning or repurposing) has been successful for sildenafil,Drugs may fail in phase III trials because of the composition of primary endpoints. For example, the inclusion of myocardial infarction—which is not causally related to heart rate—in the trial of the heart rate lowering drug ivabradine. In trials of treatments in type 2 diabetes the primary endpoint often includes non-fatal MI, non-fatal stroke and death from cardiovascular diseases. Large scale record cohorts however demonstrate that the initial presentation of cardiovascular disease is commonly heart failure and peripheral arterial disease,,Once the ‘right drug, the right target and right endpoints’ have been evaluated, the next and most costly hurdle is to carry out the definitive experiment—the phase III trial. Twenty years ago the West of Scotland Coronary Prevention Study (WOSCOPS) statin trial study demonstrated the value of EHR linkage for long-term follow-up of clinical outcomes.,Table 1, Van Driest et al.Multi-scale biological data, when combined with these deeper phenotypes, underpin further dissection of disease. Whole genome sequencing is beginning to be implemented in clinical care, for molecular diagnosis, identification of risk of subsequent wide range of diseases, reproductive considerations and drug response.Table 1, Pasea et al.,One example of the need to individualize risk comes from prolonged dual anti-platelet therapy among patients who have survived 1 year after acute myocardial infarction. For example, Clinical record data are highly effective in distinguishing risk groups, for diverse diseases and in diverse settingsIncreasing costs, complexity of patients and fragmentation of healthcare systems are challenges to delivering high quality care with better outcomes and value. Far from a data-based health care system, all too often there is a largely data free (or data silo‘d) approach where the benefits of science and evidence, and experience of care are characterized by missed opportunities, waste and harm.,Table 2, Fröbert and James,,A trial of thrombus aspiration demonstrated the feasibility of randomizing a high proportion of patients at point of care in the setting of a national quality registry,,Early examples of data-based medicine are already here, with clinical data providing both the ‘brain’ to understand what needs fixing and the ‘spinal cord’ to help fix it. For example, analysis of health record cohorts provides understanding of the patient journey and cumulative missed opportunities of cardiovascular care over time,,A small but growing number of hospitals have a suite of readily modifiable information feedback loops to improve care.Table 2, Lependu et al.,,Mining EHR in real time with both coded and text data is an important source of safety information. For example, Table 2, Chung et al.,Nationwide, policy relevant comparisons of care and outcomes among people with CVDs across health systems have only recently been reported. For example, Table 2, Asaria et al.,Big data provide new opportunities in understanding the cost effectiveness of existing and new interventions. Because of the ability to assess baseline risks in unselected general populations , such ‘real world evidence’ is increasingly required by payers and the regulators. As more data sources are linked, greater granularity of the care data (e.g. 67 different types of primary care ‘consultation’) may provide more accurate and more complete resource use data. For example, People increasingly have more and different information than their doctor or researcher raising new possibilities of ‘disintermediation’, potentially disrupting current models of health care and research.,The heart and circulation are increasingly observable as a ‘sensed self’ with novel wireless devices for mobile monitoring, with huge new data streams.Table 2, Hindricks et al.Table 2, Free et al.Implantable devices such as pacemakers provide tele-monitoring data which might reduce the risk of fatal and non-fatal outcomes in patients with heart failure  (Table 2, Pujades-Rodriguez et al.Existing models of disease prevention are simple and often focus on one disease or one risk factor at a time. Big data invite a richer understanding of the importance of: multiple diseases co-occurring,Unlike many technological advances, big data may have a role in actionable understanding of, and reductions in, inequalities in health and healthcare in rich and poor countries. The opportunity to move to a neighbourhood with lower poverty may reduce obesity and diabetes.Table 2, Sims et al.,Big data can be used to evaluate the population impact of healthcare or public health interventions.Big health record data are beginning to disrupt the nature of cardiovascular research as well as models of care. Exploiting such data is beginning to improve understanding of cardiovascular disease causation and classification, and contributing actionable analytics to improve health and healthcare, but major challenges need to be addressed to realize more fully their potential.


Mentoring programs to enhance diversity in the HIV research workforce, using the research education grant mechanism (R25), were addressed to promote new investigator development in HIV-related behavior and social sciences. The utility and benefits of the R25 mechanism were discussed. Outcome data from publication history and funded grants of mentees from the major racial and ethnic minority backgrounds indicated the success of these programs in promoting HIV-related career development. Next steps and future challenges were addressed for further enhancing the HIV research workforce. Recently, there has been recognition of a compelling need for improved strategies to promote greater diversity in the biomedical, clinical, and behavioral research workforce and to eliminate disparities . These sOne pressing issue facing today’s research pipeline is the underrepresentation of individuals from diverse racial and ethnic backgrounds . MentorsPast work addressed the conceptual background and principles of the HIV research mentoring process for strengthening racial and ethnic diversity in the research workforce . This paEfforts to diversify the workforce are grounded in the notion that increasing workforce diversity will lead to decreased health disparities and health equity among racial/ethnic minority populations. Meeting the clinical and research needs of an increasingly diverse U.S. population is an issue of national concern, yet, data indicate that we are not currently benefitting from the talents and contributions of people who represent the diversity of all citizens . MoreovIn recent years, a substantial body of literature has explored the extent to which diversity in the composition of the health-care workforce may be used as a tool to enhance interactions and, therefore, reduce disparities in health and health care in the United States e.g., , 17). NI. NI17]).From 2006 to 2017, NIMH/Division of AIDS Research, received 98 new and renewal applications (36% success rate) in response to a series of five sequentially published funding opportunity announcements . Initially, this initiative was designed exclusively for mentee participants who are underrepresented in science and engineering  and was Mentees from three underrepresented, underserved populations  were selected for each of the R25 programs. These diverse mentees were at multiple career levels , with the greater majority at the postdoctoral or early career levels. Mentee home institutions were across the nation, but the majority of R25 programs were from the East Coast and West Coast. The criteria for mentee recruitment/selection, mentor–mentee matching procedure, and program evaluation were specific to the individual R25 programs.(the process of working collaboratively with groups of people) and transdisciplinary team science (disciplinary integration on methodologic and theoretic aspects): (i) Mentoring and career development by senior faculty, established researchers, and experienced mentors that included technical expertise, career advice, insight, and professional skills development to cohorts of culturally skilled scientists who were underrepresented in the biomedical sciences, to increase their potential to become competitive grantees. Mentoring activities included instruction in presentations skills, scholarly writing, and grantsmanship; promoting successful transitions from one career stage to another; providing opportunities for the development of leadership skills, and HIV topic-related didactics/courses. As indicated in Table Direct  research experiences in HIV thematic research areas, as noted in Table Mentees experienced two major activities that were structured around two central themes, community engagement Publications and external funding are considered the most commonly used definitions of academic success. To assess outcomes after R25 completion , a subset of mentees was selected from HIV R25 programs with (a) exclusive focus on diversity and (b) pilot research projects as research experiences. For publication data, 40 mentees (approximately 4–6 mentees from each of seven R25 programs) were selected and assessed according to Medline database described below. For grant data, 30 mentees (approximately five mentees from each of six R25 programs) were selected, based on the official input of mentee/trainee appointment data from X-Train into the NIH computer system, to assess publication history and funded grants. To allow time for publications and grants to appear after R25 completion, publications and grants were examined 2–5 calendar years after R25 completion.Results showed that 25% of the mentees published more than ten articles, 18% published between six and ten articles, and 48% published one to five articles . The most general conclusion that can be offered is that the R25 program may play an important role to welcome, engage, mentor, and promote the career of emerging HIV investigators.Looking ahead to “next steps,” it could be beneficial to further enrich some of the most important features and components of the current R25 mentoring programs. These next steps could potentially include: (i) extension and broadening of pipeline to include early academic/career development and trainees from a variety of academic backgrounds; (ii) improved recruitment through community-based participatory research and social networking approaches; (iii) establishing mentee proficiency in interdisciplinary competencies and overcoming challenges of interdisciplinary mentorship; and (iv) inter-institutional/consortia partnerships to further ensure collaborations for a research network.Some challenges and opportunities for diversity-focussed training programs have been identified by others . The folUnderstand the dynamics that produce successful scientists, the characteristics and factors that make investigators successful, including diversity among subgroups and how people make career decisions. What factors influence emerging scientists from diverse backgrounds to enter, exit, and sustain a behavioral and biomedical research career? How do minority mentees develop the requisite competencies at different training and career stages? These relevant factors and variables of career success might be analyzed through regression models.Not all individuals flow equally through the pipeline; rather, race, ethnicity and gender substantially influence the forward momentum through the pipeline at succeeding stages. Racial, ethnic, and sexual minorities are more likely to “leak” out of the pipeline, resulting in low numbers of minorities at every stage of career development . Thus, tFree-standing R25 mentoring program may be more effective when partnered with other existing, effective programs. Such partnering might involve embedding an R25 program within HIV clinical trial networks  or within cohort projects . Program integration may enable leveraging of existing resources to develop programs with more immediate impact, utilizing ongoing databases for preliminary mentee projects and access to senior, experienced mentors. Such integration promotes translational and transdisciplinary approaches and can lead to increased understanding of health disparities . AdditioSociocultural factors that affect recruitment and retention are critical regarding diversity workforce considerations for R25 programs and other relevant educational and training career development programs. Among the most important individual and institutional-based sociocultural factors are unconscious bias, stereotype threat, benevolent racism/sexism, microaggressions, belonging, cultural climate, and identity. For example, multiple studies have shown that implicit or unconscious bias  may contribute to health disparities and lead to negative health outcomes . It has Addressing these challenges will provide important information on the nature of mentoring interventions that may be successful in promoting the diversity and stability of the workforce. An understanding of dynamics may allow us to propose the development of asset models and leadership opportunities, rather than solely deficit models and remediation. By combining institutional policies that promote a research culture and climate for diversity with quality, evidence-based programs and networks of support, we can go beyond an individual skill-based approach to generate a sustainable research workforce that will inform policy. At an institutional level, a culture of mentoring needs to be fostered as a revered value. Finally, mentoring for diversity, alone, is not enough. A comprehensive approach to education and training of new emerging scientists is needed where mentoring operates together with other factors, such as infrastructure development, diversity inclusion/cultural competency, academic leadership and research management, experiential learning, and a contemporary curriculum.


Tuberculosis (TB) management can be challenging in low- and middle-income countries (LMICs) not only because of its high burden but also the prolonged treatment period involving multiple drugs. With rapid development in mobile technology, mobile health  interventions or using a mobile device for TB management has gained popularity. Despite the potential usefulness of mHealth interventions for TB, few studies have quantitatively synthesized evidence on its effectiveness, presumably because of variability in outcome measures reported in the literature.The aim of this systematic review was to evaluate the outcome measures reported in TB mHealth literature in LMICs.MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews were searched to identify mHealth intervention studies for TB (published up to May 2018) that reported any type of outcome measures. The extracted information included the study setting, types of mHealth technology used, target population, study design, and categories of outcome measures. Outcomes were classified into 13 categories including treatment outcome, adherence, process measure, perception, technical outcome, and so on. The qualitative synthesis of evidence focused on the categories of outcome measures reported by the type of mHealth interventions.A total of 27 studies were included for the qualitative synthesis of evidence. The study designs varied widely, ranging from randomized controlled trials to economic evaluations. A total of 12 studies adopted short message service (SMS), whereas 5 studies used SMS in combination with additional technologies or mobile apps. The study populations were also diverse, including patients with TB, patients with TB/HIV, health care workers, and general patients attending a clinic. There was a wide range of variations in the definition of outcome measures across the studies. Among the diverse categories of outcome measures, treatment outcomes have been reported in 14 studies, but only 6 of them measured the outcome according to the standard TB treatment definitions by the World Health Organization.This critical evaluation of outcomes reported in mHealth studies for TB management suggests that substantial variability exists in reporting outcome measures. To overcome the challenges in evidence synthesis for mHealth interventions, this study can provide insights into the development of a core set of outcome measures by intervention type and study design. Tuberculosis (TB) is one of the deadly infectious diseases that have claimed millions of lives worldwide. According to the World Health Organization (WHO), globally, there were 10.4 million new TB cases causing approximately 1.2 million deaths in 2016 . In factIn this context, using mobile devices for TB treatment has been recognized as an innovative approach for LMICs where mobile subscription rates have dramatically increased over the past decade. Mobile health  interventions involving mobile devices in the management of TB have the potential for reducing costs of information delivery and improving the quality of communication . mHealthDespite the potential of mHealth interventions for improving TB management, the empirical evidence on its effectiveness is mixed. Some studies have demonstrated the effectiveness and feasibility of the mHealth interventions for TB ,13, wherTo respond to this knowledge gap, this study aimed to systematically review previous mHealth studies for TB management and critically evaluate and categorize the outcome measures for different mobile technologies and study designs. The goal of this study was to provide researchers insights into the development of a core set of outcome measures for mHealth interventions intended to improve TB treatment adherence. In doing so, the study can facilitate the evidence synthesis of mHealth interventions for TB.Electronic databases  were searched to identify peer-reviewed studies of mHealth interventions for TB. The systematic search was supplemented by reviewing relevant review papers identified from the initial search. The search strategy for the study population includes key terms describing LMICs such as “resource poor” or “developing country.” The search strategy for the mHealth intervention combined multiple keywords such as “mHealth” and “text-messaging.” As for the target disease, “tuberculosis,” “TB,” “multi-drug resistant tuberculosis,” and “MDR-TB” were used as search terms. No restrictions were applied to the publication type or publication date, but the language filter was applied to identify studies published in English. The search included articles published up to May 2018. The full search strategy is available in Furthermore, 3 authors  independently reviewed the retrieved studies throughout the selection process. Each study identified from the databases was screened by 2 reviewers and then a full-text review was conducted for the potentially eligible studies. The disagreement on the selection process was resolved by the other authors who were not involved in the review of the specific study under discussion.The inclusion criteria for this systematic review were as follows: First, studies conducted in the context of LMICs, as defined by the World Bank’s income cutoffs ; second,The qualitative synthesis of evidence focused on the outcome measures reported in each type of mHealth intervention. Information about the study setting, mHealth technologies used, target populations, and types of outcome measures was extracted. To classify the diverse types of detailed outcome measures, the following categories were used: (1) treatment outcome; (2) treatment outcome as defined by WHO; (3) adherence; (4) process measure; (5) perception; (6) technical outcome; (7) health outcome; (8) quality of life; (9) knowledge; (10) cost-effectiveness; (11) cost; (12) psychosocial outcome; and (13) mortality. Some explanations about these measures are provided in the next few paragraphs.The treatment outcome includes any outcome measure that deals with the result of TB treatment, such as sputum smear conversion or microscopy test result. The treatment outcome following the WHO definition was separately categorized . The WHOThe adherence outcome includes medication adherence or treatment adherence. The process measure is any outcome measure related to treatment process, including the receipt of diagnostic test, attendance to appointments, or reporting of adverse events. The perception indicates any outcome measure that captures the user’s thoughts on mHealth for TB management. The technical outcome relates to the outcome measures that investigate the technical feasibility such as processing times or system installation. Health outcome, quality of life, knowledge , cost-effectiveness , cost , psychosocial outcome, and mortality outcomes are additional categories which are self-explanatory.To evaluate the quality of individual studies included for our review, risk of bias was assessed with the existing tools. As this systematic review includes various types of studies, it is important to have a coherent set of quality assessment tools for different study designs. Therefore, we used the modified version of the Critical Appraisal Skills Program (CASP) that provides the checklists specific to various types of studies ranging from RCTs and qualitative studies to economic evaluation . In caseAmong the 312 studies identified after removing duplicates, 260 articles were excluded during the screening process based on the titles and abstracts. Therefore, 52 articles were assessed for eligibility through a full-text review. Of those, 27 studies were included for the qualitative synthesis of evidence. A flow diagram for the selection process, based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, is provided in In terms of the outcome measures, there was a wide range of variants in their definitions even within each category of the outcomes. For instance, both Mohammed et al and Bediang et al defined treatment success as the primary outcome measure of SMS intervention for TB medication adherence in their RCT, but they defined treatment success differently ,22. MohaThis systematic review critically evaluated the outcomes reported in mHealth studies for TB management in LMICs. The reason why rigorous evidence synthesis is warranted is that recent literature for TB reports mixed results despite the rapid implementation of mHealth technology for TB management. The fragmented pieces of evidence on the effectiveness partly resulted from the wide variations in the definitions of outcome measures in TB mHealth interventions. Even though treatment outcome has been reported by many studies, they often did not adopt the standard definition recommended by the WHO .The WHO definition of TB treatment outcome is part of an effort to standardize outcome measures for TB at the global level. To promote the use of standardized sets of outcome measures for TB, WHO provided the standard definitions and classifications of TB in terms of diagnosis or treatment outcomes . The intOur findings also suggest that, to rigorously evaluate the effectiveness of mHealth interventions for TB, future studies should be carefully designed with regard to the selection of outcome measures. Indeed, using standard definitions for outcome measures within some commonly reported categories can improve comparability across different studies. As assessed in this study, examples of such categories include treatment outcome (preferably using the WHO definition), perception, process measure, adherence, and technical outcome.The value of this systematic review can be found in its potential to motivate and facilitate consensus on standard definitions of outcome measures used in mHealth interventions for TB so that such effort can guide more effective mHealth intervention designs for improving TB management. Although current literature shows considerable variability in the definition of outcome measures, discussion and coordination among researchers can promote standardized methods in measuring outcomes. Specifically, the outcomes should be comparable, promote transparent communication, and maintain consistency in terminology. Coordination at the global level is necessary to develop a core set of outcome measures for TB mHealth interventions by study design and technology type utilized.When choosing a core set of outcome measures with standard definitions, there are additional issues to consider. First, the time point for reporting outcome measures should be clinically meaningful and feasible . Second,Another issue to consider is related to evidence for cost. As this study suggested, insufficient evidence exists in terms of cost-effectiveness or cost of mHealth interventions for TB management. Those previous studies that have evaluated the cost-effectiveness or cost of mHealth interventions only considered SMS as mHealth channels and did not consider or evaluate other mHealth channels and technologies. However, other mHealth channels and technologies such as mobile apps or global positioning system are now available for TB patients . TherefoThis study has some limitations. First, the database used for identifying relevant studies is limited to the 3 most frequently cited sources, namely MEDLINE, EMBASE, and Cochrane Database of Systematic Reviews. Grey literature or other sources of information can supplement our findings. To complement this limitation, we attempted to identify additional related studies from relevant systematic reviews searched from our study. Second, the effectiveness of the mHealth interventions for TB was not quantitatively evaluated because of the heterogeneity of the outcomes reported.Despite these limitations, this study provides an overview of the currently reported outcome measures for mHealth interventions intended to improve TB management in the context of LMICs. The results from this review can be used as a starting point for discussion to adopt standardized definitions within different categories of outcome measures for future mHealth interventions for TB management in LMICs.This systematic review of mHealth studies for TB suggests that substantial variability exists with regard to the definitions of outcome measures across studies. Our review highlights that a standardized method for measuring the different outcomes is warranted to improve comparability of outcome measures across studies for a more rigorous and reliable evaluation of the effectiveness of mHealth interventions for TB. In doing so, the coordination among researchers and the development of a core set of outcome measures based on standardized methods would be necessary. Our study provides useful information for researchers to better assess the effectiveness of mHealth interventions for TB. In addition, the study provides insights into the possibility of developing a core set of outcome measures by intervention type and study design based on a standardized or coordinated set of methods.