In March 2020, the World Health Organization (WHO) declared the outbreak of Coronavirus disease 2019 (COVID-19) as a pandemic, which affected all countries worldwide. During the outbreak, public sentiment analyses contributed valuable information toward making appropriate public health responses. This study aims to develop a model that predicts an individual’s awareness of the precautionary procedures in five main regions in Saudi Arabia. In this study, a dataset of Arabic COVID-19 related tweets was collected, which fell in the period of the curfew. The dataset was processed, based on several machine learning predictive models: Support Vector Machine (SVM), K-nearest neighbors (KNN), and Naïve Bayes (NB), along with the N-gram feature extraction technique. The results show that applying the SVM classifier along with bigram in Term Frequency–Inverse Document Frequency (TF-IDF) outperformed other models with an accuracy of 85%. The results of awareness prediction showed that the south region observed the highest level of awareness towards COVID-19 containment measures, whereas the middle region was the least. The proposed model can support the medical sectors and decision-makers to decide the appropriate procedures for each region based on their attitudes towards the pandemic. Coronavirus disease 2019 (COVID-19) is a novel disease reported in Wuhan China in December 2019. In March 2020, the World Health Organization (WHO) declared COVID-19 a pandemic, as the virus spread worldwide . When itSentiment analysis (SA), or opinion mining, is defined as the process of using machine learning (ML) and natural language processing (NLP) to classify emotions in subjective information. It is considered one of the most popular areas of research in the field of natural language processing, as it provides a means to review and analyze opinions that are held by any number of individuals ,5. In orThe goal of this study is to collect and prepare a dataset that classifies an individual’s awareness in following precautionary measures during the quarantine period in the kingdom of Saudi Arabia, and then use the dataset to measure the awareness level of each region in the kingdom. The awareness will be identified based on the percentage of positive attitudes towards the precautionary measures expressed in an individual’s tweets in the targeted regions. The prediction of individual awareness will help medical sectors and the government to raise awareness, and make the right decisions (in the required procedures) to control the pandemic. The main contributions of this work are as follows: (1) to collect, preprocess, and prepare a dataset for Arabic sentiment analysis for COVID-19 precaution awareness in Saudi Arabia. The collecting and preprocessing steps discussed in this work can be applied to any Arabic tweet dataset. (2) Develop a machine learning model to classify and predict the individual’s awareness of the precautionary procedures based on tweets in the five regions of the Kingdom of Saudi Arabia. (3) To conduct a comparison between different machine learning techniques for classifying COVID-19 related tweets, and to select the best performing ones. (4) Finally, to apply the developed model, to predict the awareness of the individuals in Saudi Arabia’s five main regions, and to compare the level of awareness between these regions.The rest of this paper is organized as follows: Social media platforms, such as Twitter, are part of the dataset recourses that are widely used for sentiment analysis applications. The Twitter dataset provides a wide range of recent information related to user behavior, emotions, and opinions regarding events happening around the world . Since tOther works focused on characterizing Twitter comments related to public opinions toward quarantines that were imposed around the world due to the pandemic. In research conducted by Anderson , 80,000 The impact of using social media during the pandemic was also studied in the work presented by Shi et al. , which iStudying an individual’s emotions towards the COVID-19 pandemic in social media was discussed in ,11. The Focusing on the Arabic language, natural language processing  and sentiment analysis  is a challenging task . This chDespite the challenges, many works have been done for Arabic sentiment analysis. The focus of those studies mainly rely on data collection and preprocessing, besides training and testing several machine leaning classifiers with different datasets ,16. A wiAlomari et al.  processeSeveral machine learning classifiers were applied in the work presented in Al-Tamimi et al. , which aIn the study presented by Alhajji et al. , sentimeBehdenna et al.  used AraIn conclusion, sentiment analysis is proven to be a valuable source of information mining, especially for situations related to the need of analyzing a large amount of data related to the public, such as studying public behavior towards the COVID-19 pandemic, and its effect on public life. Moreover, it can be concluded that the three machine learning classifiers: SVM, NB, and KNN, are widely used in this field due to their high accuracy.In this section, the main components are discussed, including: data collection and preprocessing, main feature extraction using natural language processing techniques, and the development of several machine learning models using different classifiers. Since the targeted period is the quarantine, the Twitter dataset was chosen to be the data source due to the fact that Twitter was used extensively by individuals to express their feeling towards the situation. The dataset was downloaded from Twitter using TWINT which is an advanced Twitter scrapping tool. The Kingdom of Saudi Arabia was divided into its five main regions , and then tweets were collected for each region based on location. As this study targeted Arabic tweets only, the most related Arabic keywords and hashtags to the scope were used, such as (كوفيد_19# كورونا# عدم_الخروج_من المنزل# ارتداء_الكمامة# التعقيم# التعليم_عن_بعد # غسل_اليدين#التباعد_الاجتماعي #) . These keywords represent the main precautionary procedures, such as COVID-19, corona, wearing masks, hand washing, social distancing, stay home, etc. Furthermore, only tweets that lie in the quarantine period in Saudi Arabia, which was from 23 March 2020 until 21 June 2020, were collected. The total number of collected tweets was 242,525. Duplicated and unrelated tweets were excluded.In this stage, preprocessing techniques were applied in order to clean the tweets by removing unwanted symbols. The main aim of this stage was to prepare the collected data to be used with the developed models, for training and validation. To achieve this goal, several steps were applied as follows.Normalization is a preprocessing approach aimed at cleaning data, to have a sequence of texts with standard uniform . For theغررررريب”, were converted to their correct form, e.g., “غريب”, for the previous example. In addition, Arabic characters that had multiple forms, such as Alef, which is written in multiple ways “ أ– إ - آ “, were all normalized to their single form “ا” for a unified representation.Moreover, Taskhkeel (diacritics) and Tatweel (repeated letters) cases were handled in the normalization process, where all repeated characters, such as “لا’, ‘غير’, ‘دون’, ‘ألا’,’الا’) (except–no-none-without-with), were kept in the sentence due to their importance in identifying the meaning of it.Stop words are the words that do not affect the meaning of the sentence, such as conjunctions, prepositions, articles, and pronouns . In thisTokenization is the process of converting the text into tokens separated by white space. In tokenization, all special characters are removed, the word boundaries are determined; abbreviations and numbers are processed . For AraTweet annotation is the process of categorizing tweets based on the research scope . The datDifferent sentiment analysis models were built using different techniques, in order to identify which technique provides higher accuracy with the selected COVID-19 related twitter dataset for the Arabic language. The first step of the model development was the feature extraction process, aimed at finding a suitable set of features that improves the classification accuracy. For these tasks, different N-gram models were used. The second step of the experiment was to train the dataset with a number of machine learning classifiers that proved to provide high accuracy in sentiment analysis for similar cases. Three machine learning classifiers were used: Support Vector Machine, Naive Bayes, and K-Nearest Neighbor. Supervised classification was applied with three class labels for the data . The experiments aimed to find the feature extraction-classifier combination that worked better for classifying the set or Arabic tweets related to the awareness of the COVID-19 pandemic, and then used the best model to classify the tweets in order to predict the level of awareness for individuals in each region.N-gram model is one of the most popular natural language processing models that is widely used. By comparing different NLP models, N-gram is proofed to provide the best performance among other used models . N-gram,TF is the abbreviation for Term Frequency, which is the number of times the word appeared in the document. IDF is the abbreviation for Inverse Document Frequency, which reduces the weight of the word that occurs in different documents . This met in document d. N: total number of documents. n: number of documents that contains term t.tf: is the count of term The next step, after features extraction, was to build the machine learning model. The previously prepared dataset was used once with each of the three ML classifiers: SVM, NB, and KNN.In the research conducted in , SVM claSVM is a supervised machine learning algorithm that is used in classification and regression. It could be used for unsupervised machine learning as well. It works by plotting the data values on the n-dimensional space (n = number of features). The classification process is done by finding the hyper-plan that best splits the two classes . Kernel Many researchers have experimented with Naïve Bayes for solving sentiment analysis problems. The experiments conducted in  applied In another study discussed in , Naïve BNaïve Bayes is a supervised machine learning classifier based on the statistical method, used for classification problems, by finding the probabilities of attributes . It has ed model ,34. AmonKNN is used widely for sentiment analysis purposes due to its ability to support multi-labels. It is also used for its simplicity and efficiency in classification ,35.K-nearest neighbors (KNN) is a supervised machine learning algorithm, known also as instance-based learning . KNN worExperiments were conducted using the prepared dataset that contained positive, negative, and neutral classes. Moreover, 80% of the dataset was used for training, where 20% of it was used for validation with the implemented combinations of the feature extraction-classifier models. The proposed method involved applying different N-gram models with a set of classifiers. Features were extracted using four N-gram models: unigram, unigram TF-IDF, bigram, and bigram TF-IDF. These four feature extraction techniques were applied once with each of the classifiers: SVM, NB, and KNN.By comparing the performance of the three classifiers with the bigram TF-IDF, SVM obtained the highest accuracy at 85%, and scored 87%, 84%, and 82% for precision, F-score, and recall, respectively. The least accurate performance was obtained with the KNN, which scored 64%. Therefore, the best-performed model (SVM with bigram TF-IDF) was used to classify the collected tweets dataset into the polarity that matches its content. The percentage of each class for each region was calculated, the results are displayed in The percentage of the positive and negative tweets was used as an indicator of the increase or the decrease in awareness level towards the precautionary measures imposed to encounter the COVID-19 pandemic.The results revealed that the south region of the country showed the highest awareness level as it scored the highest percentage of positive tweets of 65%. On the other hand, the middle and west regions had the lowest percentage of positive tweets, with 54%. The east region came second (in terms of awareness level) with 60% positive tweets, and then the north region with 55% positive tweets.The high percentage of negative tweets was interpreted as carelessness towards the precautionary measures. The south and north regions scored the lowest percentage at 21% and 24%, respectively, which made them highly aware regions. Whereas the middle and east regions showed the highest percentage of negative attitudes towards the pandemic at 28%. The west region was ranked as the second least aware region as it scored 26% negative attitudes.Neutral tweets indicate both negative and positive attitudes. The east region had the lowest percentages of neutral tweets with 12%, followed by the south with 13%. The north and west regions had the highest percentages, with 21% and 20% neutral attitudes.The results revealed that the individuals in the south region showed the highest level of awareness and responsibility towards following the recommended precautionary measures to slow the spread of the coronavirus. On the other hand, middle region individuals showed a lower percentage of awareness compared to other regions of Saudi Arabia. To justify the results, the total number of COVID-19 confirmed cases in Saudi Arabia, during the same period of the study, were investigated . Figure To sum up, the results of this study align with the announced report of the confirmed cases, which supports the study and the obtained results.In conclusion, the developed model confirms its effectiveness at classifying Arabic Twitter data, to be used for prediction purposes based on sentiments. These results can help authorities in decision-making, regarding the appropriate actions based on the awareness level of each region. For example, since the south region showed a higher awareness level than others, the authorities may minimize restrictions in that area. In contrast, in the middle region, since individuals showed the lowest awareness level, the authorities may invest more time and methods to increase the awareness level in that region, or apply more restrictions to control the situation.This research aimed to study Saudi Arabian citizens’ awareness levels of the COVID-19 pandemic and their attitudes toward pandemic control. This work assists decision-makers, to help them set suitable precautionary procedures for each region based on its level of awareness. The goals were achieved by collecting and preparing a dataset from the five regions of Saudi Arabia: north, south, east, west, and middle. Since Twitter is a very popular platform in Saudi Arabia, tweets were the main source of data in this research. Tweet datasets were collected separately for each region, according to location, based on specific keywords that were predefined as related to COVID-19 and precautionary procedures. The data were cleaned, preprocessed, and annotated into three categories: positive, negative, and neutral.Three machine learning techniques were chosen for training the model. The results show that bigram TF-IDF with the SVM classifier produced the highest accuracy of 85%, which outperformed KNN and Naïve Bayes. The proposed model was used to predict the awareness of individuals per region; the south region showed the highest level of awareness at 65%, while the middle was the lowest among the regions.For future work, we recommend applying feature selection and parameter tuning while building the machine learning models, and experiment using deep learning models. Moreover, we recommend taking into consideration the number of infected people per region to analyze the awareness.


Novel strategies to re-engage and retain people living with HIV (PLWH) who are out of care are greatly needed. While mobile clinics have been used effectively for HIV testing and linkage, evidence guiding their use in providing HIV care domestically has been limited. To guide the development of a mobile HIV clinic (MHC) model as a strategy to re-engage and retain PLWH who are out of care, we aimed to explore stakeholder perceptions of barriers and facilitators to MHC implementation and use. From June 2019-July 2020, we conducted 41 in-depth interviews with HIV clinic providers, administrators, staff, legal authorities, and community advisory board members, PLWH, AIDS service organizations and city officials in Atlanta, Georgia, and domestic and international mobile health clinics to explore barriers and facilitators to use of MHCs. Interviews were transcribed, coded and thematically analysed. Barriers raised include potential for: breach of confidentiality with resulting heightened stigmatization, fractured continuity of care, safety concerns, staffing challenges, and low community acceptance of MHC presence in their locality. Participants provided suggestions regarding appropriate exterior design, location, timing, and co-delivery of non-HIV services that could facilitate MHC acceptance and address the concerns. In identifying key barriers and facilitators to MHC use, this study informs design and implementation of an MHC as a novel strategy for re-engaging and retaining PLWH who are out of care. Despite widespread availability of potent antiretroviral therapy (ART), less than half the 1.1 million people living with HIV (PLWH) in the United States (US) are virally suppressed , but is Some experts have proposed the use of mobile health clinics to facilitate access to HIV care for difficult-to-reach populations . Mobile Evidence from other settings suggests that mobile health clinics are highly acceptable and have several advantages over traditional clinics , 26, 36 This qualitative sub-study is nested within a larger mixed-methods pre-implementation study that aims to prepare for utilization of an MHC to re-engage and retain PLWH who are out of care. To explore perceived barriers and facilitators of MHC use, we conducted in-depth interviews with key stakeholders of a large Ryan White-funded infectious diseases clinic, surrounding agencies that provide HIV support services in Atlanta, Georgia, and existing MHCs from June 2019 through July 2020. The study was approved by the Emory University institutional review board and the Grady Research Oversight Committee. Written informed consent was obtained from each subject prior to participation for in-person interviews and oral consent was obtained for interviews conducted by videoconference.Potential participants were purposively sampled to explore a diversity of perspectives about the MHC. Sampling targets included 10 clinic staff and providers, 10 PLWH (5 out-of-care PLWH and 5 PLWH who were clinic Consumer Advisory Board [CAB] members), 10 staff from organizations providing community-based HIV clinical and support services, and 10 institutional and city-based regulatory, billing, and, medico-legal compliance experts . OrganizStudy protocols and interview guides were reviewed by the CAB and revised based on feedback. Interviews were conducted at the clinic or at a mutually convenient off-site private location , or by phone or Zoom video conference when participants were unable to meet in person. All interviews were audio-recorded and followed a semi-structured interview guide. Interview guides were tailored to each stakeholder group, and while each guide included open-ended questions about perceived facilitators, barriers, and strategies for using an MHC to deliver HIV care to PLWH who had fallen out of care, probes that followed were tailored to the unique expertise of each stakeholder group. For example, in follow-up to an open-ended question about perceived challenges with mobile HIV care delivery, providers and staff were probed about time, support, and technology challenges, while patients were probed about accessibility and location challenges. New insights from the interviews were shared at weekly research staff meetings and then probed by interviewers in subsequent interviews. The average interview duration was 47 minutes. Participants were compensated $50 for completing the interview.All interviews were transcribed verbatim. Transcripts were reviewed for quality and de-identified. We commenced coding alongside conducting the interviews, so that inductive codes arising from earlier transcripts could be probed further in subsequent interviews. The codebook was developed after the first three interviews and included deductive codes derived directly from the interview guide and inductive codes that emerged directly from the interviews. To establish consistency of coding and interpretation, definitions and examples of each code were included in the codebook and discussed at research team meetings, all transcripts were coded by two research team members using MAXQDA Plus 20.0.4, and discrepancies in coding were discussed until consensus was reached. When consensus could not easily be reached between the two coders regarding interpretation of a transcript and appropriate code, the code and text in question was presented at the weekly team meeting for further discussion until consensus was achieved. Thematic analysis on coded interviews was applied to examine barriers and facilitators to use of the MHC model.We conducted 41 interviews  with PLWPotential patient-level barriers to an effective MHC model include: concerns around confidentiality and stigma, physical and emotional comfort, patient hesitancy to trust a new clinic with possible new providers, accessibility, the possibility of patients being more likely to fall out of other necessary medical care, and potential for decreased patient responsibility and self-efficacy in care. Facilitators are described in relation to the respective barriers below.Given the persistent stigma associated with HIV, the potential for MHCs to breach patient confidentiality  was a concern brought up by almost every participant, and across stakeholder groups.I’d say the challenges and barriers are the stigma of being seen. If, you know, it is—if people recognize this mobile unit as an HIV treatment unit in the community, and then if a member of the community gets on to this mobile unit, people are automatically going to know possibly what this person’s HIV status is.-Mobile health clinic staffI mean I feel like it’s just more open. Like it’s more out there…like what, what if somebody who—who wants to know information about why that’s there in the community… I mean I could see it working for certain communities but umm I think it would be very convenient in certain, but for me because my family still doesn’t know and community and other people I know doesn’t know—don’t know, so it would be kind of like I wouldn’t do it.-PLWHOther patients discussed that potential breach of confidentiality was a tolerable risk if it meant receiving accessible care:Well, I’m comfortable regardless because I know this clinic is something to help me, so I wouldn’t be uncomfortable at all. That would be fine… because I know that once you go into the clinic, he [the HIV care provider] has to close the door so it’s privacy there just between me and him. So the person outside they can’t hear what’s going on, so I wouldn’t feel unsafe.-PLWHParticipants across all stakeholder groups also described numerous facilitators of patient confidentiality and privacy including incorporation of strategies that diminish chance of disclosure of the true intent of the van (i.e. through nesting HIV care among other services), that detract attention from the MHC in the community, and that limit the likelihood of patients coming across one another on the vehicle.If you could do like just a general medical bus, where they can get HIV care but also maybe they’re doing free checks or health checks, blood pressure checks, you know, um so you’re …pulling other people in.-PLWHI think if it’s in a generic area that blends in. Not on the street, but like a strip mall but over to the side where people are not, it’s not where you’d just see it. It kind of blends in with what’s going on in that neighborhood… You know what I mean? People are moving.-Social WorkerParticipants also raised concerns regarding patients’ physical and emotional comfort, specifically, whether patients would feel welcome, whether they would feel comfortable in the waiting area, and comfortable with the MHC location. These concerns were expressed primarily by PLWH, CAB members, providers and staff in the clinic, and those that worked at ASOs and on mobile health clinics. Facilitators were described by all stakeholders, and included a welcoming outside vehicle appearance, comfortable waiting area with amenities , culturally-competent and caring staff with diversity reflective of the community served, and a frequent MHC presence in the community.Spruce it up with some warm colors and comfortable seats, and …it feels better than a waiting room… Some people hate hospitals. And just making the care more comfortable, more approachable, less health care-ish.-ASO staff memberMaybe partner with [a community-based organization that provides meals] and then when they leave out of the clinic then they get a sandwich and something to drink…That’ll encourage and make them want to come because at least they know when they get there when they leave the doctor they got a sandwich, a dessert, a snack and something to drink, you know.-PLWHYou know what, have a greeter, have somebody at the door, “Hi, welcome,” and stuff. Don’t just have us come in and then they sit there and then you say, “yes ma’am.” No, greet, “Hi, how are you doing today?” That makes you feel so much more comfortable. It’s just the tone of the place, you know that I mean? It’s a happy place.-PLWHResponding to how one can make an MHC the most effective, acceptable, and safe place as possible, a mobile health clinic staff member described creating familiarity and comfort:We collaborate with other community-based organizations in the area, so maybe you [the patient] may not be familiar with [our clinic], but you’re familiar with another organization that we’re collaborating with at the time. Make it more comfortable, and I feel like, you know, more frequent presence in the community, you know, become more familiarized with us. So, that kind of stigma, you know, the stigma is reduced.-Mobile health clinic staffParticipants, primarily clinic providers and staff and ASO staff members, expressed that patients might be hesitant to trust an MHC model because of the unfamiliarity with receiving HIV care in a mobile setting, possible past negative experiences with HIV testing vans, and potentially new and unfamiliar service providers operating the van.Maybe just because it’s new, right? Just getting people to say, oh, wait, we are doing this in a van? Like, what are we doing? Because the last time somebody pulled up in a van, they were giving me $25 for a survey, right? So just trusting, like, hey, we actually work for [healthcare institution]…You can actually have a real appointment here if you want to, right?-Social WorkerCons would be, who are these people? Uh does it feel like I’m receiving care from a stranger?-ASO staff memberHowever, participants also expressed that trust in the MHC model would be facilitated by communicating the model through providers and peer navigators, word of mouth, patient trust in the larger healthcare institution, provision of quality care, and time itself.But I think also [healthcare institution] is a trusted brand. I mean, everyone knows the logo, everyone knows the name. It’s Atlanta. Like, you’re trusted to be like a powerful entity in the city of Atlanta with regards to healthcare that’s able to do things maybe other places aren’t… [healthcare institution] is a trusted name in healthcare, but also HIV care.-ASO staff memberI think patients once they see what it can offer, will really put the word out ourselves. So if we make it the best that it can possibly be and really put our best foot forward then our patients will be our best our advocates for it and keep it going.-Nursing staffThe convenience and accessibility of the van, including location and parking availability, were raised as key factors by clinic staff and providers, ASO staff members, CAB members, and PLWH, impacting the effectiveness of the MHC model.How are you going to target specific areas if, because Atlanta is so spread out, and I think our population is also kind of spread out. And though there might be some hotspots across Atlanta, I guess, how to be able to reach everyone that might need.-ASO staff memberWe have to find a good, convenient spot that all they got to [do] is if they drive or jump off the bus then they’re right there… They ain’t gonna come if it’s a long ways to walk, you see what I’m saying?-PLWHBringing HIV care and services to patients might have the unintended effect of reducing patient responsibility and self-efficacy in care, a concern expressed primarily by clinic providers and staff.Yeah I think it might create this sense for patients um that…that the provider is sort of always available to them. It might make them less inclined to um…be active in their care… The unwitting message that might be sent to the patient is ‘you are incapable of coming in right so, so you’re fragile so we need to um, we need to make sure you don’t leave and that we sort of attend to you in this way.-Mental health clinicianA second participant expressed concern that patients would fall out of other necessary, less convenient medical care,Many patients that we see have sort of multiple medical needs, so you know if they sort of need to sort of see a specialist would they be less inclined to then go someplace else to get their [other treatments]?-Mental health clinicianSome participants suggested that this barrier could be addressed if the MHC could be used successfully as a gateway to re-engagement in long-term care at a traditional fixed clinic.Yeah, we’re gonna be with you for a while, we’re gonna get you up on on your feet and then, you need to come to [the regular clinic].-Healthcare providerClinic-level barriers include: potential high cost, limited capacity, safety concerns, staffing and time limitations, ability to handle emergencies, adequate awareness of the MHC among the target population, ability to provide HIV care of equal quality to that of traditional clinic-based models, fragmentation of existing patient-provider relationships, and discontinuity of care. Facilitators to address clinic-level barriers are described in relation to the respective barrier.Some participants, primarily clinic staff and providers, institutional regulatory officials, and ASO staff members, expressed concern over MHC cost-effectiveness, specifically voicing high start-up costs, variable usage, capacity to treat only a few patients at a time, and that it could be a bad steward of providers’ time and clinic money in having substantive resources devoted to a few hard-to-reach patients.Are we looking for ten people to be on the mobile clinic and we only have five today? Is that still a successful thing? So, we have to have that conversation…What is the cost benefit factor?-Compliance officialOther participants, primarily including clinic providers, mobile health clinic staff, and ASO staff, thought it would be cost-effective:I mean honestly, it seems cheaper, more cost effective than having a brick and mortar.-ASO staff memberThey discussed that scheduled appointments rather than walk-in visits could reduce associated costs.There could be days where you have no one and it’s not a very good um steward of time or money…But I think it can be overcome with a schedule.-Nursing staffSome, primarily ASO staff members and clinic staff and providers, raised concerns about the limited capacity of an MHC model due to space constraints limiting the number of providers, services provided, and number of patients seen at a given time.How can you get all of the health care team, all the essential pieces into a little van? I don’t know if there would be a model, and this might be thinking way too far outside the box, but having a model instead of just a van, but doing a trailer, dropping off a trailer for, say, a month, and then having more space for these services, and then the next month being in another location and only kind of serving two to three miles radius every month.-ASO staff memberTo address the limited physical capacity of MHCs, participants suggested limiting HIV and non-HIV services to those most critical, integrating with other delivery models like telemedicine, and ensuring strong systems of referral and integration into the larger health care system.Participants, primarily clinic staff and providers, institutional regulatory officials, ASO staff members, and CAB members, expressed concern for the safety of providers, patients, and supplies, voicing that the MHC could be a target for theft of equipment, medications, and prescription pads, and stigma-related violence if associated with HIV care. Safety concerns were affected by the locations the van operated in as well as community acceptance.What would you do about security? Um, there’s you know, there’s, there’s a lot of um hateful people and you know, so you know those are some concerns I have. Um, you know I, I, I’d hate for a mobile clinic—I’d hate for any clinic to be a target of uh of you know any kind of violence or hatred.-CAB memberTo facilitate safety, participants, especially mobile health clinic staff and clinic staff and providers, suggested working with police to determine appropriate locations and area exit plans, notifying police of the schedule, providing staff and providers with safety training, adding security personnel or utilizing the driver as security personnel, limiting the medications on the vehicle, and installing a lockbox, panic button, camera, and signs to facilitate safety.The safety thing, it’s just a matter of access. Other than people having the misperception that it contains something that it doesn’t but that’s easily fixed by just saying, … “no more than [a certain amount of cash] is on hand anytime” disclaimer out there. There’s nothing on here, maybe a computer but yeah, just making its known. No medications, no narcotics, nothing of that nature is here.-Nursing staffParticipants, primarily clinic staff and providers and institutional regulatory officials, expressed concern with the MHC being staffed by existing clinic staff and providers, who were already being utilized at full capacity. Some, including staff of existing mobile health clinics, suggested staffing the MHC with advanced practice providers to preserve physicians’ limited time.With my schedule, I can’t just go on a mobile clinic… would it be like in place of a half-day of clinic, I’d go on the mobile van? Or, am I going to be asked to do it in addition to my clinic slots, which would be very difficult for me to do personally.-Healthcare providerTo facilitate handling of urgent and emergent situations, participants, primarily clinic staff and providers and institutional regulatory officials, stressed that the MHC model should include appropriate protocols, equipment, and adequate number of personnel.It’s like alright, um… somebody falls on the floor, they’re half dead, who’s gonna help the doctor?-Nursing staffThey further discussed a need for the MHC to integrate with the health care system to maintain continuity of care during urgent medical situations.Participants across all stakeholder groups raised concerns of inadequate awareness of the MHC among the target population leading to underutilization. They also expressed concern of being able to “market” and communicate about the model to PLWH without targeting patients or exposing it as an HIV-specific van.There’s going to have to be a great deal of effort on the part of the clinic staff to make patients aware and to truly make them aware … In my mind I just honestly believe that people should just drive and literally go and locate where people say they are living.-Nursing staffParticipants noted ways to facilitate awareness: through communication about the MHC option to patients in the fixed clinic before the patients fall out of care (i.e. during clinic orientation), through peer navigators that go into the community to reach patients who have fallen out of care, through flyers in the fixed clinic, through social media, and through word of mouth and visual reminders resulting from the MHC being seen in the community.Reach people through social media, especially with our current population, so… we let the public know where we will be on the mobile unit and when we’ll be there.-Mobile health clinic staffSome, including ASO staff members and clinic providers and staff, expressed concern about whether the MHC could provide care of similar quality to a fixed HIV clinic. While some participants voiced that all services provided at the fixed clinic should be available in an MHC model, others discussed that patients should not expect the MHC to provide the same level of care but rather, be used as a re-engagement tool with linkage and integration into the health care system.The care should be measurable, comparable [to the clinic]. They should be the same, you know what I mean? I, I shouldn’t be like, Dang, this? I gotta go in this van, it’s broken down, you know? Or I could go to a, the clinic and it’s you know. It should, it should be the same. It should be the same look, feel.-Social WorkerThere’s a stigma about being poor, and like getting crappy resources for everything…the way public health often looks is like look, we’re not going to give you anything, except we’re going to control this epidemic, right?… The danger to me is like, do we further alienate people and you know, when we don’t integrate things into a larger care model, like is it worth it? Like does it maybe have more long-term risks in our ability to like talk about HIV more productively?-ASO staff memberWhile clinic providers and staff and institutional regulatory officials raised potential fragmentation of existing patient-provider relationships, especially if a patient’s primary HIV physician was not on the MHC van, as a concern, they voiced this could be overcome by integration of the MHC into the wider healthcare system.I love my patients and I love my relationships with them and I don’t want to lose it because of, because of this… I would want the goal of the HIV um mobile clinic, that that goal is to get you back to me.-Healthcare providerAt the environmental level, barriers to an effective MHC model include community acceptance and safety. Facilitators to community acceptance and safety are also described here.Participants across all stakeholder groups except PLWH discussed that the success of the MHC van depended on community acceptance. Negative community perceptions of the MHC van could spur stigmatizing action towards patients, make the van a target for violence or hatred, and inadvertently deter patients from utilizing the services. Participants described that community acceptance could be facilitated by partnering with organizations who are trusted by the community, seeking support from local government, seeking and addressing community concerns before implementation, and embedding HIV care within other health services of benefit to the community’s wellness.If you get a neighborhood up in arms, they’re going to come in and meet with their council member and the next thing you know, there’s legislation coming out that regulates you.-City OfficialJust reaching out ahead of—before the fact—to the local governments, to the city and the other local governments and finding out you know, developing, establishing the relationships…developing partnerships with existing—with existing agencies, is probably a good way to do that, or a good way to get the support of the local governments.-City OfficialBy “bringing the clinic to the community,” MHCs have potential to revolutionize HIV care for out-of-care PLWH in the U.S. This exploratory study lays critical groundwork for the design of MHC models through examining key stakeholder perceptions about barriers and facilitators to MHC acceptance and use. Factors identified as critical to successful MHC uptake include: 1) adequacy of strategies to reach and engage PLWH who are out of care, 2) patient sense of physical and emotional comfort inclusive of confidentiality and privacy, 3) breadth of HIV and non-HIV services offered, 4) understanding of the community context in which the MHC is to be implemented, and 5) implementation costs inclusive of opportunity costs .Common strategies used by other mobile clinics to raise awareness of services among target populations  , are notUnderstanding of community context  was deemed in our study as key to community MHC acceptance, and has been accomplished by others through including community members in the planning . Lastly,The study strengths include its exploration of a new model of HIV care to reach out-of-care PLWH, its examination of a wide breadth of stakeholder perspectives to understand factors affecting MHC adoption, CAB involvement in study conceptualization, and transcript coding by two study team members. Study limitations include recruitment from a single clinic which may affect transferability of findings to other clinics, and the use of a broad definition for “out-of-care” PLWH. The definition unintentionally resulted in inclusion of some patients whose medical appointments were purposefully spaced to greater than 6 months as a result of the patients achieving longstanding viral suppression; thus, some perspectives may not have been reflective of patients who are truly “out-of-care.”In conclusion, this study lays the foundation for the development of a stakeholder-driven MHC to re-engage and retain PLWH. Key factors influencing use of the MHC include the extent to which patients feel physically and emotionally safe , the extent to which the MHC provides comprehensive integrated HIV and non-HIV services, the level of community acceptance of the MHC, cost-effectiveness of the model, and the development and use of effective strategies to reach and engage out-of-care PLWH. Next steps include development of the MHC model to address these factors with the continued participation of the various stakeholders, followed by MHC piloting. Future studies will evaluate the effectiveness of the resulting MHC in enhancing retention and viral suppression relative to traditional fixed-clinic HIV care. 23 Nov 2020PONE-D-20-30821Barriers and Facilitators to use of a mobile HIV care model to re-engage and retain out-of-care people living with HIV in Atlanta, GeorgiaPLOS ONEDear Dr. KalokheThank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.Please submit your revised manuscript by Jan 07 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at Please include the following items when submitting your revised manuscript:A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocolsIf applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: We look forward to receiving your revised manuscript.Kind regards,Bronwyn MyersAcademic EditorPLOS ONEJournal Requirements:When submitting your revision, we need you to address these additional requirements.1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found athttps://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf andhttps://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf2.  Please include additional information regarding the survey or questionnaire used in the study and ensure that you have provided sufficient details that others could replicate the analyses. For instance, if you developed a questionnaire as part of this study and it is not under a copyright more restrictive than CC-BY, please include a copy, in both the original language and English, as Supporting Information, or include a citation if it has been published previously.3. In the Methods, please discuss whether and how the questionnaire was validated and/or pre-tested. If these did not occur, please provide the rationale for not doing so.http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.4. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see In your revised cover letter, please address the following prompts:a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail  and who has imposed them . Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see We will update your Data Availability statement on your behalf to reflect the information you provide.[Note: HTML markup is below. Please do not edit.]Reviewers' comments:Reviewer's Responses to QuestionsComments to the Author1. Is the manuscript technically sound, and do the data support the conclusions?The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes**********2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: N/A**********3. Have the authors made all data underlying the findings in their manuscript fully available?PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.The Reviewer #1: Yes**********4. Is the manuscript presented in an intelligible fashion and written in standard English?PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.Reviewer #1: Yes**********5. Review Comments to the AuthorPlease use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. Reviewer #1: This article examined barriers and facilitators to the use of mobile HIV care clinics to re-engage people living with HIV who are out of care. This study fills an important gap by exploring the feasibility of mobile health clinics in the United States. The study strengths include the variety of stakeholders interviewed and engagement of a community advisory board.There are several areas that need to be strengthened. The methods need to be expanded to describe the sampling and analysis process used. The results need to more clearly indicate which barriers are of most concern to each stakeholder group. Finally, the discussion needs to provide a more a nuanced analysis of the main findings. I offer the following suggestions:IntroductionWell written!Methods• Line 67: Describe the sampling strategy used. Was there a target # for each stakeholder group?• Line 69: Describe the composition of the Community Advisory Board• Line 78: Was the same interview guide used for each stakeholder group?• Line 89 Provide more description of the analysis process. For the line, “...additional inductive codes added based on initial interviews” did coding commence while interviews were being conducted, or does this mean that codes were developed after reviewing a few initial transcripts? Describe the initial process of establishing consistency among coders and the interpretation process. It would be helpful to know how transcripts from each stakeholder group compared.Results• The results represent a variety of stakeholder groups which express different barriers and facilitators. For each section, specify which specific stakeholder group is having the concern  to help readers discern which are the most widely held barriers and facilitators for each group.• Remove Table 2 as it does not add to the results.• There are currently no quotes representing mobile health clinic staff. Integrate some quotes from this group as they may have the most direct experience with this care delivery model.• Overall, the results section is quite long. Cut back on the length of some of the longer quotes . Additionally, similar sections should be combined. The sections “potentially falling out of other medical care” and “Decreasing patients’ responsibility and self-efficacy in their care” are similar, as are the sections “community acceptance and safety” and “safety concerns.” Combined into one paragraph.Discussion• Line 364: the discussion of all the main findings are condensed in one paragraph. This paragraph should be broken down to add more nuance to findings. For example, the point about increasing security personnel may be in direct contrast to patient concerns about emotional and physical safety. The suggestion in the results that police be brought in may heighten concerns as some PLWH who are out of care may engage in criminalized behaviors. Additionally, there is no discussion about how trust can be built between MHCs and patients.• Line 389: Add to the limitations that “of the 41 interviews, only 5 were of PLWH.” It is a major limitation that they are underrepresented among study participants.• Line 393: In the conclusion, include a line or two synthesizing main barriers and facilitators.**********what does this mean?). If published, this will include your full peer review and any attached files.6. PLOS authors have the option to publish the peer review history of their article  digital diagnostic tool,  22 Dec 2020Response to ReviewerIntroductionWell written!Thank you.MethodsLine 67: Describe the sampling strategy used. Was there a target # for each stakeholder group?The study utilized purposive sampling to explore a diversity of stakeholder group perspectives. Our target goal was to achieve 10 clinic staff and providers, 10 people living with HIV , 10 organizations providing community-based HIV clinical and support services for PLWH , and 10 institutional and city-based regulatory, billing, and, medico-legal compliance experts. Sample size targets were based on Kvale’s estimate that 15+/-5 individuals from a group is sufficient to reach theoretical saturation.1Although we did not reach the target for the compliance/regulatory officials category, we concluded enrollment once we as a team concluded we had reached theoretical saturation. A description of the sampling strategy has been added to the Methods section on lines 69-72. Line 69: Describe the composition of the Community Advisory BoardThe Consumer Advisory Board (CAB) was made up of approximately 10 PLWH who receive HIV care at the study clinic. They were nominated to the CAB based on their interest in HIV advocacy and in enhancing the services provided within the clinic’s Ryan White Program. They represent various racial/ethnic, gender and sexual minority groups, and many endorse having had a period in their lives where they were out-of-care or struggled with their HIV diagnosis and appointment adherence and are able to describe strategies they used to overcome the challenges. A description of the composition of the CAB has been added to the Methods section on lines 80-85.Line 78: Was the same interview guide used for each stakeholder group?Different interview guides were used for each stakeholder group. While each guide asked open-ended questions about barriers, facilitators, and strategies for using a mobile HIV clinic (MHC) to deliver care, probes that followed were tailored to the stakeholder group. For example, in follow-up to an open-ended question about challenges with mobile HIV care delivery, providers and staff were probed about time, support, and technology challenges in addition to concerns around safety, privacy and confidentiality, while patients were probed about perceived challenges with MHC access and location . Clarification on the differences between interview guides has been added to the Methods section on lines 95-101.Line 89 Provide more description of the analysis process. For the line, “...additional inductive codes added based on initial interviews” did coding commence while interviews were being conducted, or does this mean that codes were developed after reviewing a few initial transcripts? Describe the initial process of establishing consistency among coders and the interpretation process. It would be helpful to know how transcripts from each stakeholder group compared.We commenced coding alongside conducting the interviews, so that inductive codes arising from earlier transcripts could be probed further in subsequent interviews. The codebook was developed after the first three interviews and included deductive codes derived directly from the interview guide and inductive codes that emerged directly from the interviews. To establish consistency of coding and interpretation, definitions and examples of each code were included in the codebook and presented and discussed at research team meetings, all transcripts were coded by two research team members, and discrepancies in coding were discussed until consensus was reached. When consensus could not easily be reached between the two coders regarding interpretation of a transcript and appropriate code, the code and text in question was presented at the weekly team meeting for further discussion until consensus was achieved. See below response regarding how transcripts from each stakeholder group compared. Additional description on the analysis process has been added to the Methods section on lines 107-117.ResultsThe results represent a variety of stakeholder groups which express different barriers and facilitators. For each section, specify which specific stakeholder group is having the concern  to help readers discern which are the most widely held barriers and facilitators for each group.For each section within the results, stakeholder groups that primarily brought up each barrier are described. Remove Table 2 as it does not add to the results.Table 2 is now removed.There are currently no quotes representing mobile health clinic staff. Integrate some quotes from this group as they may have the most direct experience with this care delivery model.Quotes representing mobile health clinic staff were integrated throughout the results section, within barriers: “confidentiality and stigma concerns” (line 140), “lack of physical and emotional comfort” (line 180), and “Adequate awareness of the MHC among the target population” (line 352). Overall, the results section is quite long. Cut back on the length of some of the longer quotes . Additionally, similar sections should be combined. The sections “potentially falling out of other medical care” and “Decreasing patients’ responsibility and self-efficacy in their care” are similar, as are the sections “community acceptance and safety” and “safety concerns.” Combined into one paragraph.Lines 318 and 269 were shortened. The header “Potentially falling out of other medical care” was removed and the text under this heading was abridged and moved to the header “Decreasing patients’ responsibility and self-efficacy in their care.” We chose to not combine the sections “safety concerns” and “community acceptance and safety" as the themes described under each section are different. The former describes clinic-level safety concerns (i.e. theft of supplies) and informs clinic-level strategies for enhancing safety , whereas the latter describes environment-level concerns (including safety concerns) that arise from inadequate engagement of the community (i.e. the neighborhoods where the MHC would be parked) in the process of MHC design and implementation and suggests environment-level strategies to address this concern .DiscussionLine 364: the discussion of all the main findings are condensed in one paragraph. This paragraph should be broken down to add more nuance to findings. For example, the point about increasing security personnel may be in direct contrast to patient concerns about emotional and physical safety. The suggestion in the results that police be brought in may heighten concerns as some PLWH who are out of care may engage in criminalized behaviors. Additionally, there is no discussion about how trust can be built between MHCs and patients. We fully agree with the reviewer and have expanded this paragraph for a more nuanced discussion of the implication of study findings in informing MHC design. Additions are on lines 436-438, 442-447, 450-455.Line 389: Add to the limitations that “of the 41 interviews, only 5 were of PLWH.” It is a major limitation that they are underrepresented among study participants.11 of the interviews were PLWH, including 5 who were out of care and 6 from the CAB. Language was added on lines 69-71 of the Methods section describing this sampling strategy. The team unanimously felt that we reached theoretical saturation with the 5 interviews with PLWH who were out of care and 6 PLWH who were on the CAB. Additionally, the qualitative findings presented in this manuscript are part of a mixed-methods study, which includes surveys (not published here) with 50 PLWH who are out-of-care that examine their relative preference for mobile HIV care versus several other community-based models of HIV care delivery and gauge their specific preferences for MHC implementation.Line 393: In the conclusion, include a line or two synthesizing main barriers and facilitators.The following line has now been added, “Key factors influencing use of the MHC include the extent to which patients feel physically and emotionally safe , the extent to which the MHC provides comprehensive integrated HIV and non-HIV services, the level of community acceptance of the MHC in their locality, cost-effectiveness of the MHC, and the development and use of effective strategies to reach and engage out-of-care PLWH. Next steps include development of the MHC model to address these factors with the continued participation of the various stakeholders, followed by MHC piloting” (lines 474-481).Additional References1. Kvale S. Doing Interviews. Thousand Oaks, CA: SAGE Publications; 2008.AttachmentPLOS ONE_response_submitted12.21.20.docxSubmitted filename: Click here for additional data file. 5 Feb 2021Barriers and Facilitators to use of a mobile HIV care model to re-engage and retain out-of-care people living with HIV in Atlanta, GeorgiaPONE-D-20-30821R1Dear Dr. Kalokhe,We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at onepress@plos.org.If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact Kind regards,Bronwyn MyersAcademic EditorPLOS ONEAdditional Editor Comments :Reviewers' comments:Reviewer's Responses to QuestionsComments to the Author1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.Reviewer #1: All comments have been addressed**********2. Is the manuscript technically sound, and do the data support the conclusions?The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: (No Response)**********3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: (No Response)**********4. Have the authors made all data underlying the findings in their manuscript fully available?PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.The Reviewer #1: (No Response)**********5. Is the manuscript presented in an intelligible fashion and written in standard English?PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.Reviewer #1: (No Response)**********6. Review Comments to the AuthorPlease use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. Reviewer #1: (No Response)**********what does this mean?). If published, this will include your full peer review and any attached files.7. PLOS authors have the option to publish the peer review history of their article (If you choose “no”, your identity will remain anonymous but your review may still be made public.Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.Reviewer #1: No 18 Feb 2021PONE-D-20-30821R1 Barriers and Facilitators to use of a mobile HIV care model to re-engage and retain out-of-care people living with HIV in Atlanta, Georgia Dear Dr. Kalokhe:I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. onepress@plos.org.If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact plosone@plos.org. If we can help with anything else, please email us at Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staffon behalf ofDr. Bronwyn Myers Academic EditorPLOS ONE


Research carried out in partnership with Indigenous youth at The Native Youth Sexual Health Network (NYSHN) demonstrates that Indigenous youth can (and do) develop and implement public health interventions amongst their peers and within their communities, when supported by non-youth allies and mentors.Together, NYSHN and Well Living House researchers co-designed a qualitative case study to demonstrate and document how Indigenous youth can and do practice their own form of public health implementation research (PHIR) in the realm of mental health promotion for 2SLGBTTQQIA and Gender Non-Conforming Indigenous youth. Academic and Indigenous youth researchers were: participant observers; conducted a focus group; and designed and implemented an online survey with Indigenous youth project participants. Governance, intellectual property, financial terms and respective academic and NYSHN roles and responsibilities were negotiated using a customized community research agreement. The data were thematically analyzed using a critical decolonizing lens that recognizes the historic and ongoing marginalization of Indigenous peoples while also highlighting the unique and diverse strengths of Indigenous communities’ knowledge and practice in maintaining their health and wellbeing.Analysis revealed how colonialism and intergenerational trauma have impacted Indigenous youth identity and the value of self-determination as it relates to their identity, their relationships, health and wellbeing. We also learned how knowing and doing about and for Indigenous youth needs to be youth determined – ‘nothing about us, without us’ -- yet also supported by allies. Finally, our analysis shares some promising practices in knowing and doing for and with Indigenous youth.This study provides a reminder of the need to centre Indigenous youth throughout PHIR in order to realize sustainable benefit from research, services and programming. It emphasizes the need to recognize Indigenous youth as leaders and partners in these initiatives, support their efforts to self-determine, compensate them as partners, and prioritize Indigenous youth-determined frameworks and accountability mechanisms.The online version contains supplementary material available at 10.1186/s12889-020-09863-3. Research, carried out in partnership with Indigenous youth at The Native Youth Sexual Health Network (NYSHN), demonstrates that Indigenous youth can (and do) develop and implement public health interventions amongst their peers and within their communities, when supported by non-youth allies and mentors –3. Key fInvestments in implementation science (IS) and more specifically, public health implementation research (PHIR), provide a possible avenue to remedy these challenges, in that the method “seeks to understand work within real world conditions” . As suchIn this paper, we present a qualitative case study, that applies an Indigenous case study method , 12 and The above-mentioned CIHR funded PHIR research project  was purpIndigenous community leadership and investment are critical mechanisms underlying successful health interventions for Indigenous infants, children, youth and their families , 15.Successful Indigenous health interventions are tailored to- and build on- local Indigenous values, skills, knowledge, and beliefs , 14.Relationality  reciprocity  and balance at the individual, interpersonal, collective, and socio-ecological levels are key concepts essential to successful Indigenous health services, research processes and capacity strengthening .A multi-generational perspective that is informed by one’s ancestors and takes into account the health of future generations is central to day to day analysis and decision making .Wisdom is achieved through the collective and iterative sharing, uncovering, and interweaving of iterative empirical observation, lived experience, dreams and linked metaphysical manifestations , 19.The theoretical framework underlying this work builds on an Indigenous reframing of the existing core steps of PHIR advanced by the senior author as the platform for a larger research program in which this case study was nested Fig. a and b. This framework challenges historic and ongoing marginalization of Indigenous peoples in the planning, implementation and assessment of Indigenous health research, services, and policies and the external imposition of non-Indigenous models premised on non-Indigenous epistemologies and knowledge assumptions. We are motivated in our desire to interrupt the systemic application of non-Indigenous approaches since this is known to be ineffective , 20, 21.The research team was comprised of Indigenous and allied Indigenous health scholars and NYSHN Indigenous youth researchers. Working together we applied an Indigenous research partnership approach that has been iteratively developed by the academic team at Well Living House (WLH) over the past decade , 22, 23,The Native Youth Sexual Health Network (NYSHN) is a grassroots organization, formed in 2010, by and for Indigenous youth, that works across issues of sexual and reproductive health, rights and justice throughout the United States and Canada. It is collectively led by and for Indigenous youth who are 30 years of age and under. In addition to its own membership, NYSHN is advised by 3 Youth Councils: National Indigenous Young Women’s Council, National Indigenous Youth Council on HIV/AIDS and National Native American Council on HIV/AIDS. The NYSHN model holds space for all members of NYSHN, youth councils, partner organizations as well as aunties, uncles and mentors. The model fosters sustainability because it has built-in core support for core youth members and a built-in model for working with those people who have history and memory with NYSHN.Well Living House (WLH) is an action research centre for Indigenous infants, children, and their families’ health and well-being. Our focus is on gathering, using, sharing, and protecting Indigenous health and well-being knowledge and practices. We draw on both Indigenous and public health knowledge to inform cutting edge scholarship and best practices. At the heart is an aspiration to be a place where Indigenous people can gather, understand, and share what it means to be a healthy child, family, and community – building a “Well Living House”.The study protocol was vetted and approved by NYSHN in keeping with the agreed-upon roles and responsibilities outlined in SMH-NYSHN Research, Data Sharing, and Publication Agreements  and our methods are primarily intrinsic according to case study methods more generally , 24. As documentation of Indigenous youth-determined conceptual models and practical strategies for research partnerships and platforms;identification of barriers to- and facilitators- of health and wellness promoting behaviours, including the role of underlying determinants of health including racism, family and community relationship and health literacy, and existing systems of health knowledge and practice;supporting the assessment of promising practices; and,sharing of best practices across Indigenous communities.Specific objectives for this NYSHN PHIR case study are:In keeping with qualitative case study methods , 24, then = 7) Indigenous self-identified 2SLGBTTQQIA youth (i.e. NYSHN youth leaders and members) and lasted 1 h and 20 min. The focus group was audio-recorded, transcribed verbatim and subsequently validated by NYSHN.The focus group was held in 2015 at the Youth-Governed Approaches to Mental Health Promotion and Suicide Prevention for 2SLGBTTQQIA and Gender Non-Conforming Indigenous Youth Gathering in Toronto, Ontario. It was co-facilitated by two Indigenous community members who were chosen based on their experience as facilitators and their ability to prioritize the comfort of the youth participants and provide peer support where requested. The focus group sample included seven (n = 5) invited members completed the survey, many of whom identify as 2S. Participants were asked to identify and reflect on barriers and facilitators to health and wellness. The survey took approximately 20–30 min to complete and was validated by NYSHN.In lieu of an additional planned focus group, an online survey was administered in 2016 by the NYSHN co-leads in order to accommodate participant schedules and capacities as well as varying comfort levels with research across the membership. Five NYSHN team members who were, at the time, actively involved in NYSHN organizational and research activities were invited to complete the anonymous, online survey. All five  into their practice to mark a transition into and out of youth.On the whole, participants considered youth to be a dynamic and transitional time. In particular, they described Indigenous youth as: resistant; survivors; inclusive; heterogeneous; and, intersectional.Participants in the focus group discussed the effects of intergenerational trauma on identity and how Indigenous youth are sometimes excluded from the narratives of colonialism. They shared their unique experiences with colonialism and how it acts to disconnect Indigenous youth from family and community, informing their identities.‘[ … ] the violence from these various systems happens from birth. Bad birth doctors’ separate newborns from their parents, then Child and Family Services steals them, and then educational institutes separate Indigenous youth from their parents, communities, nations.’ [S:2]Participants explained that as a result, Indigenous youth have been disconnected, oftentimes right at birth. As a result, we were told that meaningful and purposeful partnerships with Indigenous youth are integral to the development and implementation of sustainable and effective research, service and programs directed towards Indigenous youth. Partnering with youth, as advised by NYSHN and as represented in their governance, goes towards re-establishing intergenerational connections, in part, through inclusion, support and respectful relations.In general, the focus group, survey and documents all indicated that Indigenous youth health and wellbeing is complex, contradictory, dynamic, and subject to situational and environmental influences.Participants cautioned that Indigenous youths’ health and wellness has been poorly defined by the public health research community who disproportionately rely on research-based metrics, like height, weight, disease, etc. As noted by one participant, ‘[ … ] even if all these things were met, I’m not sure that it [health] would be defined as good, since it is dependent on what is going on in someone’s individual life; rather than the boxes that they ‘check’ of basic health and wellness or what someone else defines is going on for them.’ [S:1] These measures, we were advised, are too restrictive, focusing primarily on the individual and failing to recognize how, for Indigenous youth, health and wellness is interrelated with their communities and relationships, i.e. kinship networks.Participants in the focus group listed several aspects that, in their experience, influence Indigenous youth health and wellbeing, including: self-determination; harm reduction; intergenerational and peer support; validation; connection; and, relationships and cultural safety.http://www.nativeyouthsexualhealth.com/whatwebelievein.html]. Participants were quick to point out however, that in order to self-determine Indigenous youth still require support, skills and tools.According to many of the participants, poorly informed research contributes to ineffective, paternalistic public policy and service provision for Indigenous youth in Canada. Participants were clear that for Indigenous youth, the conversations can only begin by recognizing that health and wellness must be self-determined. For example, they explained that given the opportunity, Indigenous youth are more likely to focus on harm reduction over treatment services. Harm reduction is defined by NYSHN as a way of life, “that we, as well as our ancestors, have been keeping our communities safe and reducing harms long before the word ‘harm reduction’ came into the English language. It’s about reducing the many harms in our lives, not limited to just substance use  through the tools that work best for us, without stigma or judgement. We also don’t define what harm is for other people.” [“ … self-determination doesn’t mean totally leaving people alone (unless they explicitly say so) but holding space for their voice to matter and be heard.” [S:1]Figure Some participants struggled with defining how or whether they had ever experienced ‘good’ health and wellbeing. One participant described how ‘[ … ] Indigenous youth are in perpetual state of un-wellness.’ [S:3] This insight was shared by others and reflects how the persistence of colonialism and intergenerational trauma experienced by Indigenous youth in Canada has been normalized. Alternatively, one participant drew attention to hegemony, noting that who and how they define what qualifies as unhealthy for Indigenous youth should be approached with caution.‘I’m not sure what good health and wellness looks and feels like. I can get through the day I’m supported, when I’ve slept, when I’ve drank water, ate some food, and tried my best to listen and interpret what my body is saying.’ [S:4]‘[B] eing unhealthy or unwell is a completely natural response to the many violences constantly inflicted on our communities.’ [S:3]Participants from the online survey identified changes in behavior and disconnecting from friends and family as examples of early signs that peers may be unwell. Participants also identified youth kinship networks as youth-mediated means for identifying when other youth are unwell and in need of an intervention.Almost all participants shared how diverse relationship networks were critical to their health and wellbeing. Participants also spoke of how their relationships are often self-determined, and include a mixture of some or all of the following: chosen family; biological family; ancestors; land; and, service providers. Indigenous youth well-being is impacted when these relationships are disrespected through research and are prescribed, mandated or disrupted through social and public health services. Examples include early separation from bio-families and disrespecting or failing to recognize sexual identities or chosen kin.‘Disruption of family and community relationships, including and especially those of family of choice, ceremony and adoption, play huge roles in how young people see themselves and in particular their mental and spiritual health. Particularly for young people who have vocalized, in small or big ways, who family is to them or who their kinship network is they want to be with and are not allowed – either by law or by circumstances of lack of support.’ [S:1]These disruptions can begin early for Indigenous youth, as described by one participant who said, ‘the disruption of having people taken away from you too early is so harmful, i.e. your big supports pass away too early or youth are relocated or taken away by Child and Family Services.’ [S:2].The effect for some, as one participant pointed out, can be ‘traumatizing and sever folks from health supports.’ [S:5] Alternatively, however, the same participant went on to suggest that it can also be positive and liberating, in that disruption encourages self-determination. The participant noted, providing NYSHN as an example, that it might be ‘[ … ] a place where new families are made.’ [S:5] The ability of Indigenous youth to identify and access alternative relational supports is critical and depends mainly on the health and diversity of these networks.For these reasons, participants maintained that, supporting Indigenous youth to build and sustain these networks and acknowledging their choices are crucial to respecting their chosen identities.‘[I] t is just as important for young people to define what family and community is to them and who their peers are.’ [S:1]‘[ … ] supporting young peoples to make decisions about who their community and family is contributes to good health and wellness.’ [S:4]Participants noted that non-youth allies and mentors have a responsibility to ‘uphold these relationships and to protect them from all other colonial and systemic interferences that try to say they don’t matter as much.’ [S:1]The focus group, survey, and the NYSHN website and documents all promote that knowing and praxis in public health for and about Indigenous youth (i.e. IS and PHIR) must be youth-determined throughout the entire process. From priority setting to implementation and on through to health promotion and service delivery. Participants noted that, in their experience, many of the services and programs available to youth are, in fact, not youth-determined, and instead have the effect of further isolating Indigenous youth.‘I often find myself following things back to see if they are actually youth-led. And unfortunately, more often than not, they’re not.’ [F:1]Participants maintained that when knowing and praxis are instead Indigenous youth-determined, the results are diverse, inclusive, accessible, flexible, decolonizing, culturally safe and accountable to Indigenous youth. As an example, focus group and survey participants stated that priority setting should be led by- or undertaken in partnership with-, Indigenous youth. As put by one participant, ‘[ … ] having the agenda set by youth is, you know, key to everything.’ [F:3] Indigenous youth know what they need as well as the tools to get there and these are reflected in the NYSHN beliefs, principles and practice. However, as participants pointed out, priorities are mainly set by non-youth, resulting in an unequal division of power between those parties and Indigenous youth. Supporting youth-driven priorities requires the meaningful engagement of Indigenous youth in their communities, separate from the involvement of community leaders (i.e. chief and council).Systemic barriers in research were also identified as creating challenges to youth leadership. For the most part, participants viewed funding opportunities as biased to favour academics as opposed to youth and their communities. Participants stressed that funding for research should instead be more readily available and accessible to Indigenous youth. They noted that most direct funding opportunities that do exist for Indigenous youth are minimal and capped. As a result, participants described feeling pressured into having to develop relationships with researchers to pursue projects. One participant pointed out how, ‘all of our access to CIHR in the last ten years has had to be with academic institutions and the network (NYSHN) still can’t apply [ … ] our youth still can’t apply to research projects on their own.’ [F:7] Participants pointed to how these are difficult relationships to negotiate, particularly if there is a disconnect between what Indigenous youth are hoping to achieve versus what a researcher might be interested in. As an organization, NYSHN has structured itself to unsettle this by, ‘… trying to reverse that flow and say okay, this is what we’ve seen the need for, this is what youth are asking for and now how do we get the resources to make that happen?” [F:1]At the same time, focus group participants stressed that ‘[ … ] youth-led doesn’t mean that we put them on an island all by themselves like with no help or support, because I think that’s equally problematic.’ [F:6] Instead, participants pointed out that the role of adults and allies is to support and enable Indigenous youth leadership to facilitate them when setting their own priorities. As suggested by one participant, adults and allies need to ‘[ … ] challenge other adults to stop talking for, on behalf of, or making decisions for and on behalf of young folks.’ [F:2] The participant went on to say that, Indigenous youth need to be empowered to describe to their chosen allies  that ‘[ … ] this is the way I need your help.’ [F:2]Participants recommended that adults create safe spaces where Indigenous youth can share their concerns and ideas with each other and with non-youth. As explained by one participant, ‘just providing that safe space for them to talk and listen to them [Indigenous youth]’. [F:4] They also suggested that adults be more culturally safe. In many cases, research with and about Indigenous youth has been, and continues to be, unsafe, as it fails to acknowledge colonization and its associated traumas. Rather, research frequently reinforces trauma through linking the experience of violence to the cause of violence or repeatedly researching the cause of suicides as opposed to solutions. Participants suggested that research can be more culturally safe when it acknowledges and accounts for peer support systems and inter-generational trauma that is the result of colonization. Peer programming, we were told, should form an integral part of any project or service because it acknowledges ‘[ … ] that’s just as much a part of the process of a project or program is that we show up for each other because oftentimes, nobody else is showing up for us.’ [F:7] Another critical suggestion made was to stop all research programs that are rooted in reinforcing trauma.In Table There are ‘knowing and doing’ interventions that meaningfully engage and are viewed positively by Indigenous youth. These interventions are considered promising practices. One participant, for example, shared their experience with a food sovereignty and mental health garden project.‘It was for Indigenous youth of colour where youth didn’t have to prove their identity’, it was peer led with a youth circle and an aunties and uncles circle. It was about addressing mental health by participating in organic gardening for remedies that helped feed families and also harvested medicines for mental health itself. Over and over again, I heard young people say they felt they were being heard, they felt safe, and they felt like what they were doing mattered – to their own health, and their communities health.’ [F:1]http://www.nativeyouthsexualhealth.com/sexyhealthcarnival.html] The carnival also features interactive games, prizes and safer sex supplies, as well as content and age-appropriate activities for younger children so that their parents, siblings or other care-takers can participate in the carnival. It has become one of NYSHN’s most popular outreach tools, with many requests each year from Indigenous communities for the carnival to visit their community. Indigenous youth leaders conducted evaluation of the program in 2014 with funding from Ontario HIV Treatment Network (OHTN) and received the OHTN Community-based Research Award. They have also received a CIHR award to expand implementation and evaluation across additional communities. Critical to this example is how the community goals differ from those of research. Here Indigenous youth and their communities initiate their own Sexy Health Carnival and mentor youth to be able to do this work in comparison to research goals, one of which includes addressing gaps in the literature that connect sexual health to wellbeing, culture, etc.Others pointed to the Sexy Health Carnival: ‘Sexy Health Carnival is peer-led, provides accessible information, resources and space to learn to have fun and engage with information that made most sense to you.’ [F:4] An example of research practices that centre Indigenous youth’s leadership and ideas, as well as mentoring to strengthen Indigenous youth’s capacity, the Sexy Health Carnival was initiated by NYSHN youth facilitators, working with their community. The carnival works to break the barriers of fear, stigma and shame relating to issues that Indigenous youth face in their communities. It has created safer practices content that makes learning information about issues that affect our bodies more accessible and most importantly more fun to learn about for inspiring youth, community members and Elders. The idea to broaden the scope of the carnival came from Indigenous youth – including the creation of boards and supports. Various booths range from topics such as suicide, harm reduction, consent, sexual violence prevention, STI’s, birth control and masturbation.’  partnerships to enhance the success of a study’ (:6). In pThe participants in this study, similar to the Indigenous youth engaged in previous work , 29, areA key insight that emerged from this study was that participants did not differentiate between the stages of PHIR; public health research and health promotion activities were conceptualized one and the same. Knowing (research) and doing (service provision) were considered to be deeply interconnected and requiring the expertise and full involvement of Indigenous youth, as leaders and partners throughout. Knowing and doing with Indigenous youth, then, should not be approached as discrete activities. Colonialism has influenced the decontextualizing of knowledge from context and community, resulting in a schism between Indigenous knowing and doing . IndigenFinally, participants in this study also agreed that IS and PHIR need to be accountable to the Indigenous youth community. The individuals, agencies and institutions that are involved in directing and practicing research must be equipped to accept when they are told no, or to stop research when and if it is identified by youth as either harmful or lacking these goals, one possible accountability mechanism is for Indigenous youth organizations and allies to work in partnership and rate research, services and programs intended for Indigenous youth through audits. In a similar example, the Native Women Association of Canada has issued report cards for the Missing and Murdered Indigenous Women Inquiry . IndigenThe work described here is conducted and governed in full partnership with NYHSN – where NYSHN members and Indigenous youth were directly involved in- and central to- all stages of the research project . This process is realized and governed though a community research agreement. As a result, youth-determined concepts and experiences are well-represented in the work. This approach is critical to all public health research and programming with Indigenous youth.Data collection for this case study was limited to NYSHN leadership and NYSHN members and may be biased towards their experiences; the experiences of Indigenous youth more broadly, in different settings, are rich and diverse and will vary. As such, the research presented here can provide insight but is not necessarily generalizable. This limitation however, illustrates a larger concern which is the limited number of partnerships between Indigenous youth and/or their representative organizations with the larger public health research and services community.As emerging frameworks that prioritize context, end users, and community engagement, IS and PHIR are increasingly relied upon in the field of public health and amongst policy circles to improve population health. Similar to knowledge translation, implementation research more generally is viewed as a way to address the ‘know-do-gap’, forming a critical link between what should happen in theory and what actually happens in practice. In this respect, it attempts to address different aspects of implementation, including social and contextual factors , 32. WhiAdditional file 1.Additional file 2.Additional file 3.