The Covid-19 pandemic and subsequent national lockdowns resulted in drastic changes to the way that higher education was delivered. A mixed-methods research study was conducted to explore university students’ perceptions of online learning during the 2020/21 academic year. Students from across all Welsh higher education institutions were invited to participate. First, a series of focus groups (n = 13) were conducted to explore students’ experiences of online learning during the pandemic. Two were conducted in Welsh, the remaining eleven in English. Thematic analysis led researchers to develop eight key themes: Seeking the positives, Facilitators to learning, Barriers to learning, Lost sense of community, Let down by University, Workload, Assessment, and Health and well-being. These themes informed the design of a quantitative survey which was completed by 759 students. It was found that students were largely satisfied with the quality of online learning, however there were specific challenges associated with a lack of community, wellbeing concerns, and challenges with loneliness and isolation. Data from the focus groups and survey informed recommendations for practice in three key categories; teaching practice, institutional level recommendations, and student health and wellbeing considerations. The COVID-19 pandemic caused an unprecedented global disruption to everyday life . This inSurvey studies were deployed across the sector to capture the student experience of this transition to online learning, and the early indications were that the transition to online learning was associated with many negative experiences for students. For example, students reported dissatisfaction with a lack of peer and teacher interaction, and the challenges of engaging exclusively via a screen . MotivatThe transition was also a considerable challenge for university staff, many of whom were not experienced in online teaching and who were dealing with their own challenges caused by the pandemic. This resulted in a significant negative impact of staff wellbeing, increases in workload, and a further erosion of trust in university leadership , althougUniversities also experienced significant institutional challenges. There was a need to rapidly train both staff and students so that they could teach and learn online . DiffereDespite these challenges, a number of positives also emerged from the data. Students reported enthusiasm about the flexibility and convenience afforded by online learning . A ‘RemoOverall though, the experience appears to have been a net negative for both staff and students. However, one theme that is common to almost all the research cited here, is a feeling that the pandemic has forced changes to HE that will persist for a long time. For example, an increased proficiency in online delivery may reframe student’s choice about whether or not to actually attend a physical campus, particularly those students from deprived or rural backgrounds where the logistics are especially complicated . TherefoIn this study we adopted a pragmatic approach to try and capture the student voice through focus groups and a survey. Pragmatism is an approach to education research, that prioritises asking research questions whose answers are intended to be practically useful. In pragmatic research, epistemological and ontological considerations are of less significance to the methodology than the practical utility of the findings . The aimEthical approval was granted by Swansea University Medical School Research Ethics committee, stage 1 (reference number 2021–0037 26.4.21) and stage 2 (reference number 2021–0037). All participants provided written consent.A two-staged mixed methods research study was undertaken between March and September 2021. Students registered to study in any Welsh HEI were eligible to participate. There are eight HEIs in Wales, with a total of 81,670 registered undergraduate students (2019/20).Inclusion criteria for participating in either stage were:A student at a Welsh HEIAged 18 plusAbility to consentAccess to a WiFi/internet enabled device and (freely accessible) zoom software (for focus groups).Participants were invited to take part in a series of focus group discussions facilitated via Zoom between April and June 2021. The study advert and flyer –S4 FilesInterested participants provided consent and arranged a pseudonym prior to participation (if requested). Focus groups were facilitated in English and Welsh and video recorded via zoom software. Transcripts were anonymised prior to analysis. Participants were thanked with a £20 voucher.th June 2021 to 30th August 2021. On clicking the survey link, participants were first required to read the Participant Information Sheet. The survey then took them to a Consent page, and once consent had been given, the survey commenced. At the end of the survey, participants could click a second survey link to enter their e-mail address for the optional prize draw. These survey links were kept separately to ensure that it was not possible to link a participants e-mail address to their survey responses, therefore ensuring anonymity. There were 57 questions in the survey (including demographics), under the following sub-headings: About me, Quality of online learning, Community and collaboration, My workload, Online assessments, Accessing teaching and resources, Managing my health when learning online, Caring for dependents, and Thinking ahead. Questions were a mix of closed Likert questions and open, free text questions.The questionnaire was hosted online on Survey Monkey™  informed by the work of Braun and Clarke , 27. A sQuantitative survey data were analysed using descriptive statistics and visual analysis of graphed data. Likert-scale data were analysed by one-sample Wilcoxon test to determine whether the distribution of responses was significantly different to the mid-point of the scale. Statistical analysis and graphing were undertaken using Graphpad Prism 9 .Qualitative data from each free text question were analysed using a pragmatic content analysis , 29, infOne hundred and twenty-eight expressions of interest from seven different Welsh HEIs were received. A total of 54 participants took part in 13 focus group discussions, two of which were conducted in Welsh. The majority of participants were women . Mean age was 24 years old , . ParticiEach focus group included between two and five participants. Duration ranged from 1 hour 13 minutes to 2 hours 18 minutes.Seeking the positives, Facilitators to learning, Barriers to learning, Lost sense of community, Let down by University, Workload, Assessment, and Health and well-being. The themes are summarized in Through qualitative data analysis, the following eight themes were identified: Each theme is outlined briefly below, with participant extracts.Participants identified new learning opportunities encountered as a result of changes to the educational environment. These opportunities included learning new software and hardware which led to the development of new technical skills. Participants took time to learn or revise their digital skills and develop their confidence and digital literacy. Participating in extracurricular educational opportunities  and increased access to online conferences helped develop transferable skills. Students also reported that staff created networking opportunities. Participants reflected on what they gained from remote delivery of teaching, for example, inclusion of international voices across time zones, additional learning resources that had not previously been made available, reduced travel, ability to revisit learning materials, laboratory and practical skills being taught in a different way.“I’ve had to do a project on Zoom, which was a complete online production of Shakespeare’s The Tempest on Zoom, which was weird, but really good. So, it was just another learning experience for me, so it was positives and negatives to what I’m missing out on, and also the new experience that I’m gaining through all this technological voodoo stuff. So, you know, it’s branching out, learning new things and taking experience where you can”. “doing more things over Zoom has developed my IT skills because I can, like, understand how to go on Zoom and stuff and, I don’t know, do different stuff. So it has developed that element of things that maybe I wouldn’t have done” However, the way such experiences were discussed indicated that participants were searching for a positive outcome during the crisis period, and there were attempts to ‘make the best of a difficult situation’. Positive experiences were countered by comments expressing a desire to return to face-to-face teaching and learning, or a hope for hybrid environments in which peer and staff interaction are reinstated.This theme referred to aspects of remote and digital learning which supported student’s ability to learn. This theme is organised into three sub-themes.Sub-theme 1: Interaction and engagement. Interactive teaching methods and approaches which encouraged students to take part and contribute to live sessions aided ability to engage with teaching and learning in both synchronous (live) and asynchronous (pre-recorded) delivery modes.“multiple question tests at the end of each week, and it wasn’t for a mark, but still you can catch up and see what is going on, if you can understand the theory or not. So, that was really helpful”. “some modules I’ve got, the tutor does a poll halfway through and gets involved, gets us discussing it” Sub-theme 2: Flexibility and control. Remote learning afforded students increased personal control over the time and pace of their learning which helped some organise their time more effectively. For example, some participants explored how they had scheduled their own learning timetables around their home/personal/work and general life commitments. This flexibility was facilitated by the reliance on pre-recorded lectures, the removal of travel requirements and other life commitments (as a result of UK wide lockdown restrictions). Students highlighted that the flexibility and control afforded to them meant that they were better able to take part in learning at times which suited them.“I think the most valuable thing in the whole digital approach to learning this year was the flexibility and the availability of the learning” This was particularly prominent for those with pre-existing physical and mental health conditions, whereby health conditions had previously limited or affected their ability to participate in in-person teaching due to fatigue, illness, medical appointments or social anxiety.“I’ve got three children, I’ve got a fulltime job, and just that kind of family friendly–and being more flexible. I can do it from home while my daughter’s doing whatever she’s doing, you know. I think actually a positive from this move is going to be, moving forward, diversity and inclusivity and accessibility for people, and I think that can only be a good thing, you know, a really good thing, and actually a rationale for keeping an element of a blended approach”. Equally, international students highlighted the benefits of technological functions like, pause, speed control and subtitles which helped them better understand learning materials. For others availability of recordings supported active listening, in the knowledge that materials can be re-watched and repeatedly accessed.“they’ve been really useful, and I go back and like re-watch them constantly” Sub-theme 3: Positive communication. Effective and timely communication supported learning and ability to engage, while frequent and supportive communication from lecturers, academic mentors and personal tutors provided support.“my academic tutor, even though it’s just fifteen minutes, is always like, “Oh, how are you?” and like really checks in, which is really nice to have This theme is described via four sub-themes which highlight the range of experiences that had a negative impact on learning and ability to engage in learning.Sub-theme 1: Lost learning opportunities. Students identified a range of lost learning opportunities, these included, changes to programme delivery and the structure and format of modules available, the removal of course components such as laboratory experiments, fieldwork, practical workshops, and industry/workplace placements. Participants widely acknowledged their dismay and perceived this to be a barrier to learning which reduced their ability to develop required knowledge and transferable skills. This was particularly prominent for those undertaking scientific, healthcare, education and performing arts degrees.“Regards with online labs, I mean, it’s just ridiculous. I’m doing medical biochemistry. If I wanted to go into a research field and work in a lab, dragging a pipette with my mouse into a test tube is completely different compared to physically doing it. There’s so many more human errors. So, in that aspect, it’s ridiculous, but at the same time of course I understand the situation “We were meant to have a language clinic every Monday, and that was supposed to be for two hours, but we missed most”. Sub-theme 2: Technological barriers. A lack of access to technology (hardware and software), internet enabled devices and limited, poor or intermittent Wi-Fi presented a practical barrier to learning. Particularly evident for those living in shared accommodation where several students were required to negotiate access to Wi-Fi simultaneously or following a return to family homes where other members of the household were also using the Wi-Fi for their own studies or employment purposes. Instances were discussed where students had to rely on expensive 4G data to access live sessions or to avoid experiencing anxiety and worry arising from poor Wi-Fi. This was also a prominent issue for those in rural locations.“I’m always like to my housemates, “Can no one use the wi-fi for a bit?” And then I just have to plug in my headphones and my microphone, and just hope that that works, but it is quite worrying that it is all very temperamental, and even if you have good wi-fi, there are always going to be bad days where it could drop out”. Other technology related barriers included limited skill set, confidence and willingness to adapt to technology led learning environments. Further complicated by technical difficulties, poor functionality, inadequate equipment, poor usability and navigation issues of Virtual Learning Environments (VLE) , and digital tools which culminated in wasted time, frustration and/or inability to access required learning resources.“I would say basic structure–basic infrastructure or a layout, just to be sure that everything that will give you points, and ultimately credits, is in something called assessment and feedback, or assessment and practicals… another teacher would do it differently” Sub-theme 3: Learning environment. Participants described how their personal ‘learning environment’ often acted as a barrier to engagement. For example, many had to share IT devices with family (during the UK lockdown), experienced frequent interruptions by family/friends meant that study time wasn’t always available as planned and reduced ability to concentrate. Often participants needed to adapt their study routines and to access content at unusual hours while others did not have a private/safe place to study at all. Others felt embarrassed by their ‘learning environments’ which contributed to poor engagement in live remote teaching, and meant that cameras were not turned on which affected peer engagement and motivation.“I know a lot of people are hindered with disturbances. I’m quite lucky to be able to just say to my family, “Can I have a bit of quiet on the upper floor for my lecture?” It hasn’t stopped my mother coming in with a hoover when I’m in the middle of a presentation” “I know that there’s a few on my course who are embarrassed about their rooms and embarrassed about–you know, they don’t want people judging their rooms.”She said, “And there’s a few that just simply won’t go on screen unless they’ve got a full face of makeup on, because they just feel that they need to be ready, you know, eyelashes…” Sub-theme 4: Poor communication. Poor communication relating to timetables  meant that some students missed out on vital information. This was of particular concern following periods of unexpected illness; and poor and inconsistent communication regarding module content (such as availability of pre-recorded content).“we received email at 10pm the day before, actually, now it’s from three till five. Have a good night.…So I would say that the slightest mistake cost you a lot. You miss one email, you miss one notification, you click mark as read or ignore or letter…. And you could be losing. It’s a dangerous game” Participants described a real sense of lost community during online learning. In the main this resulted from the loss of all in-person interactions (peer to peer and peer to staff due to UK wide lockdown) and the resultant loss of the hidden curriculum, that is, the benefits associated with learning from and supporting each other academically and socially .“throughout my last two years, I’ve not actually known who are in my class because they just come up with little icons, so that’s why I’m really disappointed”. “you feel still quite alone because you are one of a 100 person–people, and you are alone in this group”. The lost sense of community had a detrimental impact on student mental health and wellbeing. Many talked about how their experiences of isolation and loneliness led to feelings of unwanted stress, anxiety and depression. Many shared their lowest points, their concerns moving forwards and their hopes for the future (see theme 2). There was a lack of opportunity to meet new people, make friends, form a social group and generally to find a place at university for themselves. The impact was significant, particularly for first year students who were only beginning to embed themselves in university life, form friendships and networks and establish themselves as independent learners.“this isn’t emotional, this is a screen, this is a computer, it’s like there’s no one here. So yeah, the university experience has not been with humans” However, this sense of loss was not restricted to new students. Participants across all levels of study talked at length about how they missed social interaction arising naturally via in-person interactions in a range of academic and social environments and the how this affected their learning and motivation to learn. For example, the sense of isolation and lack of community meant, for some difficulty engaging with lecture materials, exacerbated further by pre-recorded lecture content which built up over quickly over the semester and became overwhelming. Participants also noted the impact of zoom/screen fatigue, distractions and the monotony associated with studying in one single environment (mainly their university bedroom) with no access to wider university learning environments  which impacted on motivation.“The motivation, trying to get motivation behind–because being a student behind a screen is blooming hard work”. “I need to get up and get out of my bedroom. I can’t work in my bedroom from my bed. That’s where I sleep, not where I work.”Participants described being let down by the university (and for some the UK or Welsh governments) during the pandemic. Key issues included a lack of access to critical university resources and services which supported learning (including self-directed study) and independent living. For example, library and computer lab closures, removal of transport links (which impaired ability to attend teaching or move about the city), campus-based food halls and food supplies were closed . Participants felt there was inadequate provision of mental health resources or support systems for students (particularly Welsh language students where resources were not translated). Fees and monetary issues not addressed by the university which affected individual’s ability to access remote learning resources, and gave way to resentment and anger for those unable to afford technologies required (previously provided via computer labs and libraries). Limited consideration of training needs for themselves and staff. Students discussed examples of staff who were unable to effectively use new technology, who failed to adequately adapt their learning content for remote delivery and who wasted time during live virtual sessions through a lack of understanding of how to use the new technologies in situ. Combined, students felt a significant loss of the anticipated university experiences and important social and cultural rites of passage.“if anyone sort of said to me, “What is the one thing that you feel you’ve missed out on?” That would be it, just sort of going into the library and, yeah, like chatting to the staff and things like that, and asking them, you know, “Have you got any books on this?” ‘Cos I think, yeah, although I’m quite good with technology, I did sort of really struggle initially to use like Met Search and stuff … I do kind of miss that–just being in the library.”“with mental health there’s been no support offered by the University…. institutionally, I think there’s been systemic failure”. Two focus groups were held through the medium of Welsh. Welsh speaking students faced additional challenges, from a lack of Welsh language mental health support, a dearth of Welsh language provision for digital resources, to practical barriers of rural life with poor Wi-Fi connectivity.“finding resources or sources has been a bit harder, especially in Welsh. Like, if you need an old book, the chances are that there isn’t an electronic copy available, so you then have to go through the process of ordering the book from the library, see if they release it to you and then, obviously, go and pick that up. You’re okay if you live in [redacted], but when I was home, it was a bit more awkward. I had to plan a day so that I could pick the books up because the resources just aren’t really available.” Some participants perceived an increase in workload as a result of the increased time and energy required to adapt to new ways of learning, such as, using VLEs, learning new software, and accessing online resources. Participants reported that watching pre-recorded lectures was more time consuming than a traditional in-person lecture, as they required pausing materials, stopping for breaks, distractions, or to follow up on areas of interest. Students also found they would rewind and re-watch the pre-records to ensure understanding and language translation. The incessantness of the digital environments was challenging. There was also a need to develop new skills whilst simultaneously trying to keep up with the teaching content and assessments.“uni’s non-stop. Because we’ve got Canvas on our phones then, so that’s what we do our work through. And that combined with all the announcements I get on that and the WhatsApp group and everything, it’s just–there is no break from uni. Your life is now university” Experiences of online assessment were coupled with perceived higher workloads. The move to open-book examinations led to a reliance on Google™ and a change in the way students revised and studied. Some students noted that they found open-book exams harder and required additional time to prepare and complete the assessments. Others were concerned with technical failures and errors which affected time available to complete assessments. These were exacerbated by varying home learning environments which meant some individuals were disadvantaged.“my discipline is only decreasing …, I’ve found it that they’ve styled the question that I’m going to still need to look up the answer on Google, whether I’m confident or not, just because I can, and if I can, why shouldn’t I?”Remote learning impacted on students’ health and well-being. Students discussed their emotional responses to the challenges, changes and adaptations required of them and the support they received or wished they had received. Physical health was impacted by extended period of screen time, inadequate or a complete failure to schedule breaks between timetabled lectures. hidden health conditions were exacerbated by long periods of sedentary screen time, back pain, stiffness, and reduced ability to undertake self-care routines, whereas previously the educational setting had afforded opportunity to move, stretch, and take a break from screens and intense periods of concentration. Mental health was impacted via a lost sense of community and isolation (see theme 4).“Tuesdays last term I had lectures from about 9am or 9:30, all the way until 6pm. And the one until 6pm always overran because it was a practical. So, there’s no time to have lunch. I don’t know how we were expected to do that because every lecture ran until the end”. “after a year of being out of social circles, how are we going to fit in there again? How are we going to find our feet in our groups again? Maybe it’s going to be a new group, maybe it’s the same group, but, you know, lots of people will lose that thing that makes them fit into their group. So, I think it’s going to be difficult to fit in”. Seeking the positives, Facilitators to learning, Barriers to learning) were streamlined to focus on specific areas of learning . The final two survey subsections were included to identify the different situations that students noted impacted their ability to study and engage effectively (‘Caring for dependents’) and finally to reflect their hopes and views for the future of remote learning (‘Thinking ahead’).The qualitative themes discussed above directly informed the development of the survey for Part 2. Through triangulation and discussion amongst the research team, the themes were translated into distinct areas of enquiry which were used to structure the survey and develop the questions. Four of the qualitative themes were directly translated into survey subsection . Three of the learning related qualitative themes  participated in the survey. Participants were representative of all years of study, from those studying a foundation year, through to those who had been studying for 6–7 years, and came from a range of different degree types. Just over five per cent were foundation students, 72.15% were undergraduate students and 22.67% were postgraduate students. The full range of demographics is shown in Students were asked to rate their satisfaction with their experience of the different types of online learning during the 2020–2021 academic years. Results are shown in These data suggest that, overall, the majority of participants were satisfied with the quality of online teaching. Content analysis of free text data see  found thOn the basis of the Stage 1 data, we asked two separate specific questions about breakout rooms used in online teaching; students were asked to state, on a five-point scale, whether they found breakout rooms engaging or useful. For analysis, the five text points on the scale were converted to a numerical 1  to 5 (‘Very engaging/useful’) scale. The majority of participants responded that breakout rooms were not very engaging, with 32.28% selecting that they were ‘Not at all engaging’ and 22.83% selecting ‘A little engaging’ (22.83%). The majority also did not find them very useful with 32.64% responding that they were ‘Not at all useful’, and 20.37% selecting ‘A little useful’. A one-sample Wilcoxon test was used to test whether, for each online learning format the distribution of responses was significantly different from the midpoint of the scale. The distribution of responses was significantly different for both responses . These data indicate that, as a group, those participants who were taught using breakout rooms had a negative experience. Free text data revealed that participants found these awkward, and that other students were often reluctant to speak at all, making for an unsatisfactory learning environment.Students were then asked to rate their agreement with a series of statements regarding their experience with Community and Collaboration, Workload, and Online Assessments. Again, students were largely positive about their experience, except for online practical classes, and regarding the opportunity to meet and interact with their peers. Summary results are shown in Thirty-seven percent of participants agreed that they had sufficient opportunities to interact with other students, whereas 46% percent disagreed. When asked if they would have liked more opportunities for interaction, 73% said yes. Over 40% of the sample agreed they had experienced sufficient opportunities to work with other students, and 18% disagreed. Sixty-one percent would have liked more opportunities to work with others. Only 25% felt they had enough opportunities to meet their coursemates, with 53% disagreeing with this statement. A large majority of participants agreed that they were able to get answers from teaching staff (69.8%) and interact with staff (66.0%) during online learning. Content analysis of qualitative data revealed that many participants felt there was no sense of community, and that group work was difficult to conduct online. The majority of free text comments suggested that teaching staff were not very responsive to e-mail and that it was harder to interact with staff and ask questions, though some found it easier to interact with staff online. This however confounds with the quantitative responses see , which sFifty-four percent of students responded that online learning made them feel overwhelmed, whereas 29% disagreed with this statement. Additionally, fifty percent said that online learning made them feel anxious, 61% felt isolated, and 48% did not feel motivated. These findings highlight the impact that online learning had on mental health. The advantages of online learning however were found in the flexibility that it offered. Fifty-seven percent said that it enabled them to work at their own pace, and 62% reported that it enabled them to manage their time flexibly.Fifty percent of participants responded that online assessments were fair. The free text data (23% responses) suggested that Universities needed to make sure that the examination criteria were clear, but that participants largely wanted online assessments to continue. Participants also said online examinations were less stressful than in person exams, and open books exams were viewed positively. The possibility of internet issues during an exam was mentioned in 10% of comments, and 8% of comments were concerned with people cheating.Over 75% of respondents experienced issues with internet connection during teaching, and just over a third of students were not able to access a device when they needed it. These findings highlight the need for recordings to be made available so that students are still able to access teaching. With regard to accessing recommended resources, participants had mixed experiences. Almost 70% of free text comments reported that they needed better access to resources. Some participants said that they were able to easily access online texts and journals, however others said that only physical copies of key textbooks existed, therefore they could not access them due to libraries being closed or living too far away. Some respondents highlighted that online resources were available, but that they did not always know how or where to access them, and that they needed training in the different VLEs. A number of participants praised the institutional library for the service and support they provided. Data are shown in Figs Qualitative data revealed that many participants felt more anxious, depressed isolated, overwhelmed, and lonely during online learning. Participants also reported that these issues impacted their ability to access learning, as they found it harder to engage and felt unmotivated. Content analysis found that over 76% of comments stated that their physical and or mental health was negatively affected by online learning. Participants reported physical health issues including back pain, eye strain, sedentary behaviours, and weight gain. Some respondents however, found that online learning helped them better manage their pre-existing health conditions, due to the flexibility of learning and lack of travel.Twenty-four per cent of participants reported that they cared for dependents. Content analysis showed that 44% of comments reported online learning made caring for dependents harder, whereas 32% said it was easier. The impact that online learning had was mixed depending on individual circumstances. Participants with children often reported that online learning was harder, but some of these difficulties were due to school or childcare closures relating to the pandemic, rather than online learning per se. Participants with children found it challenging to find a quiet place to study, especially during live teaching, and found it more difficult to focus on their learning. Advantages of online learning for these participants was the lack of commute therefore being closer to a child’s school or dependent’s home; and the flexibility around when they studied due to recordings being available.Participants were asked how they would like to learn in future , with 39This research study explored student perceptions of learning across Welsh HEIs during the pandemic in the 2021/22 academic year. The intention of the research was to identify sector-wide recommendations for future online learning practice, informed by student views.Qualitative analysis developed eight broad themes. These subsequently informed the development of the online questionnaire which was structured into six domains. The key findings from each domain are summarised below.Quality of online learning: Overall, the majority of participants were satisfied with the quality of online teaching they received. However, students highlighted the need for additional interactive elements in both live and pre-recorded lectures. When asked about specific digital teaching techniques, breakout rooms were negatively perceived due to student feelings of awkwardness and limited engagement and interaction form other students, including non-use of cameras.Community and collaboration: Overall, students felt that the sense of community was missing during online learning. Survey data reported that only 37% felt they had had sufficient opportunities to interact with other students during their learning and the overwhelming majority (73%) wanted more interaction. However, a large majority of participants (70%) agreed that they were able to get answers from teaching staff and interact with staff during online learning, suggesting that staff were able to interact with students during the pandemic period, and that perhaps peer-to-peer interaction was most missed.My workload: Just over half of students surveyed (54%) indicated that online learning had made them feel overwhelmed. Additionally, half said that online learning made them feel anxious, isolated (61%), and un-motivated (48%). In contrast 62% indicated they benefitted from increased flexibility and (57%) the option to work at their own pace.Online assessments: Half the students surveyed felt online assessments were fair, and that they largely wanted online assessments to continue, citing reduced stress. Concerns were raised regarding the clarity of assessment instruction, assessment criteria, and information on how to handle internet issues arising during examination periods.Accessing teaching and resources: 75% of respondents experienced issues with internet connection during teaching, and just over a third of students were not able to access a device when they needed it.Managing my health when learning online: Survey respondents free text data indicated many felt anxious, depressed isolated, overwhelmed, and lonely during online learning which impacted their ability to engage with digital learning content. Content analysis found that over 76% of respondents stated that their physical and/or mental health was negatively affected by online learning. Participants reported detrimental physical health aspects including back pain, eye strain, sedentary behaviours, and weight gain.Caring for dependents: Almost a quarter (24%) of survey respondents had caring responsibilities. Content analysis showed that 44% of comments reported that online learning made caring for dependents harder. Participants with children found it challenging to find a quiet place to study, especially during live teaching, and found it more difficult to focus on their learning.Further analysis was not undertaken on the basis of specific demographics, e.g. gender and individual higher education providers. This is partly due to the overall size of the sample, along with a desire to make general, sector-wide recommendations rather than recommendations for specific universities or groups of students. However future work could examine these questions in more detailKey findings support previous research on remote learning which highlighted the challenges associated with navigating the new screen-based, remote, digital landscape . These iet al [Similar to the current study, Bastos et al  surveyedWu & Teets  reportedKulikowski et al.  highlighFinally, authors drew attention to student perceptions at baseline, for e-learning and the impact on outcomes. Specifically, a positive attitude towards eLearning could enhance student engagement, motivation and ability.Garris & Fleck  conducteThe general level of satisfaction with online learning during the pandemic is perhaps surprising. Media coverage of Higher Education, in the UK, during lockdown, contained many negative reports of students who were unhappy with their university experience, particularly when this extended into a second academic year . Some meOverall, whilst the findings of the present study reported benefits of online learning in terms of flexibility and time management, institutions need to be aware of some of the downfalls of online learning, to be able to mitigate these. The present study suggested that students were largely satisfied with the quality of online learning during 2020–2021, suggesting that much content can be successfully transitioned to an online format. Nevertheless, whilst quality can be maintained with careful lecture design, qualitative data from both the focus groups and the survey suggested that students missed many aspects of the face-to-face learning experience. Furthermore, there were negative impacts on mental health including feeling overwhelmed, lack of motivation, and increases in depression and anxiety. Some of these can also be mitigated through effective pedagogical design, for example by using shorter formal lectures, with the time replaced by more informal discussion-based sessions .On the basis of the data in the present study, some key recommendations for future practice were developed by the team and then finalised by three authors . The recommendations were focused on capturing lessons learning during the pandemic and turning them into positives aspects for online learning in a post-lockdown era. The recommendations were intended to be practically useful across the sector, but with a focus on the HE sector in Wales. They are categorised into three main areas: teaching practice, institutional level recommendations, and mental health and wellbeing.Use bite size lectures. The structure of a lecture is key to maintaining motivation and engagement. Pre-recorded lectures should be broken into smaller sections (approximately 15 minutes), and live lectures should include breaks. The lecturer should ensure that their camera is switched on to foster a connection between the lecturer and the student. Closed captions should be used.Ensure asynchronous access. Access to recorded content was a great success of lockdown teaching, and provides many obvious benefits to learning, some of which students now expect as basic requirements in return for tuition fees.Promote Interactivity and engagement. Interactive activities such as quizzes polls, and live word clouds were popular features of online learning, as these encouraged interaction and kept students engaged. It is important that opportunities for students to ask questions are included. When mostly pre-recorded teaching is used, students valued the opportunity to attend live Question and Answer sessions to complement these.Set clear expectations for online breakout rooms. Breakout rooms can be made more effective by ensuring that students are given clear guidance in advance about what tasks they will be expected to do. Ideally, students should be allocated to the same group each time or be able to sign up to a particular room with students that they know. This helps reduce some of the awkwardness and allows students to develop social networksPractical classes should remain in-person wherever possible. Practical classes did not translate well to an online medium. Some of the key learning can be achieved online, but the hands-on practical element is a key feature of in-person teaching.Ensure consistent communication. Students found it confusing when important information about their learning was communicated through different communication channels.Conduct a technology training needs analysis. Many students reported that staff were not able to use the technology required for online learning, which hindered the learning experience. Similarly, many students themselves felt unequipped to use this technology. We recommend conducting a training needs analysis for both staff and students, to identify knowledge gaps.Ensure access to IT support. One source of stress for students, particularly during online assessment periods, was when they had issues with technology but were unable to access help to resolve it.Address basic barriers to online learning. Many students reported that they did not have access to an appropriate device for online learning, did not have a suitable place to study, and had issues with WiFi. Institutions should therefore ensure that students can access shared study spaces, computer laboratories, and that suitable equipment is available to loan. The technological requirements should also be made clear when students are making study choices.Foster a sense of community. Many students reported that there were unique University experiences that they had missed due online learning during the pandemic. These consisted of rites of passage such as freshers’ week, the experience of campus based learning, meeting fellow students, and learning practical course skills. This should be factored in should students miss out on these events in future.Reduce student overwhelm. Many students reported that online learning was overwhelming. This can be reduced by providing clarity as to which components of a course are mandatory and which are optional and providing students with weekly checklists detailing which tasks need to completed.Provide first language mental health resources. Students that were first language Welsh speakers reported that they did not have access to mental health resources in Welsh. Welsh speakers reported that when talking about their mental health, it was more difficult conversing and expressing themselves in English.Schedule dedicated social opportunities at course or module level. Many students reported feeling isolated during online learning, therefore it is vital that social opportunities are timetabled to foster community.Timetabling breaks. All learners need regular breaks, particularly when learning online–both within a teaching session and between teaching sessions. This ensures they can take comfort breaks, have a screen break for their eyes, move around, and consume food and drink.Be aware of social anxiety online. Enforcing that cameras are switched on during teaching may induce social anxiety. Some students feel that having cameras on is an invasion of privacy, increases stress and means they cannot move around. Furthermore, some students feel like they are being watched which can make them uncomfortable. Neurodivergent students are more likely to experience some of these difficulties. In the interests of inclusivity and accessibility, switching cameras on should not be mandatory.S1 File(PDF)Click here for additional data file.S2 FileFocus group participant demographics.(DOCX)Click here for additional data file.S3 FileSurvey participant demographics.(DOCX)Click here for additional data file.S4 File(DOCX)Click here for additional data file.S5 FileThis is the raw data.(CSV)Click here for additional data file.


Rural, remote, and underserved communities have often struggled to provide adequate access to family physicians. To bridge this gap in Renfrew County, a large, rural region in Ontario, Canada, a community- based, hybrid care model was implemented, combining virtual care from family physicians and in-person care from community paramedics. Studies have demonstrated the clinical and cost effectiveness of this model but its acceptability to physicians has not been examined. This study investigates the experiences of participating family physicians.A mixed-methods study, combining physician questionnaire response data and qualitative thematic analysis of focus group interview data.n = 17 survey respondents and n = 9 participants in two semi-structured focus groups (n = 4 and n = 5 respectively). Physicians reported high satisfaction, driven by skills development and patient gratitude, and felt empowered to reduce ED visits, care for unattached patients, and address simple medical needs. However, physicians found it difficult to provide continuous care and were sometimes unfamiliar with local healthcare resources.Data was collected from This study found that a hybrid model of in-person and virtual care from family physicians and community paramedics was associated with positive physician experiences in two main areas: clinical impacts, especially avoiding unnecessary ED visits, and physician satisfaction with the service. Potential improvements for this hybrid model were identified, and include better support for patients with complex needs, and more information about local health-system services. Our findings should be of interest to policymakers and administrators seeking to improve access to care through a hybrid model of in-person and virtual care.The online version contains supplementary material available at 10.1186/s12913-023-09599-x. Robust primary care is crucial to a high-performing healthcare system , 2, and To address these challenges, some jurisdictions are exploring “virtual care,” which involves providing healthcare through internet and telephone services, as a treatment modality for rural, remote, and underserviced areas , 15. VirThis study examines an innovative, community based, hybrid model, the Virtual Triage and Assessment Center (VTAC), that combines virtual care from family physicians and in-person care from community paramedics. VTAC was established in Renfrew County, a rural county in Eastern Ontario, Canada, to provide all residents with access to COVID-19 assessment and testing and to prevent unnecessary visits to EDs. In keeping with the rapid changes to healthcare at the start of the pandemic, VTAC was developed and implemented at unprecedented speed, taking just 12 days from the initial planning phase to being operational –22. VTACPrevious research has demonstrated that VTAC may be an economical and effective way of delivering acute, episodic care in rural, remote, and underserved communities , 21, 23.A mixed methods study design was employed to investigate the experiences of family physicians who provided care to patients through VTAC. This was done in the context of the quadruple aim framework for evaluating innovations and change in healthcare, looking for improved clinical outcomes, lower healthcare costs, enhanced patient experience and enhanced provider experience . The clin = 22) were invited via e-mail to complete an online questionnaire  to August 11, 2021, able to understand and converse in English, and able to provide informed consent. All VTAC family physicians identified to meet the eligibility criteria  collected demographic information and included Likert scale-type questions used to assess physician experience with VTAC . n = 4 and n = 5, respectively) were conducted using a set of predetermined questions and follow-up probes to explore provider perceptions of VTAC. These interviews were conducted online via Zoom, audio recorded, and transcribed using Zoom auto-generated transcription. These transcripts were then cross-referenced with the respective audio files to correct any inconsistencies and prevent transcription error. Due to the limited availability for focus group participation, we were only able to conduct two interviews. However, after reviews of the transcripts, we found that the same or similar themes appeared across both focus groups and concluded that the data was sufficiently saturated to report our results [A total of two semi structured, focus group interviews . Further, some participants defined identifying when emergency care was required, as just as much of a success as being able to deal with an issue without resorting to additional care: “I feel good when I save ER visits, but when I would want them to go to the ER, I feel like that is a success as well” (P7).Participants frequently commented on how VTAC was able to help them positively change the lives of Renfrew County residents, especially those without family physicians, by providing them with a service that addressed their urgent health needs. Participants described how VTAC enabled them to save healthcare resources by reaching unattached patients. Many VTAC patients were seniors, often without a family physician and some with mobility restrictions. Participants in the discussion agree that the integration of Community paramedics into the VTAC has been a successful part of the program. It was “particularly helpful for linking paramedic care to the Community, especially for patients who are unable to travel and require complex care” (P4). Another physician recounted a specific instance when the Community paramedics used technology to enhance patient care, saying "I remember a visit where the paramedics were in the home, I was on the phone, they had a tablet with a camera so they were able to do sort of real-time video and pictures of this wound that the patient wouldn't have been able to manage…it worked really well." (P5). The option for VTAC physicians to augment their virtual care with in-person support from a paramedic was seen as an important part of this hybrid service, “I think that’s actually one of the most successful parts of VTAC” (P4).VTAC’s hybrid combination of virtual and in-person care was seen as an enabler for Several participants expressed “that a service like VTAC needs to stay around” because the care it offered to unattached patients would not be possible in “the Ottawa valley without VTAC at this point unless they get more family doctors” (P8). There was also discussion about adopting this model of care for other underserviced areas to supplement their primary care.improving patient care by interim management of chronic diseases and preventative care, and referrals to facilitate specialized care and follow up: “we are putting patients in touch with mammograms, FIT tests, referrals to gynecology” (P2). This also included review and renewal of prescriptions for patients. During virtual appointments, “there were a lot of prescription renewals that spilled into VTAC,” and they were successfully addressed (P2). In summary, nearly all participants described ways VTAC enabled them to increase their positive impacts on both patients and the healthcare system.Other participants described their experiences of personal fulfillment they experienced from giving patient centered care: “If I can help the patient in a way that they were hoping for, or that I think is appropriate, I think that seems to be the case for the most part” (P8). The gratitude of patients motivated participants at work and helped them see the value of a service like VTAC for underserved communities.Providing care through VTAC cultivated the participants’ clinical skills and patient feedback boosted their morale. Most participants touched on the professional development opportunity for participants to hone their skills which they could apply to “other areas of practice” (P3). Some participants explained that VTAC made them “a lot better at taking a really good history” due to limitations of the virtual encounter (P3). One participant that was in an urban center noted that they gained insight into treating patients from rural areas.VTAC also served as a delivering non-continuous care. This can be problematic for patients with multiple or complex issues requiring regular follow up: “In a 10-min period you can’t cover everything or be as detailed” and patients are “adamant about talking to the same doctor” (P7).Participants experienced some difficulties caring for patients with complicated issues, knowing about the full range of available local services in the patient’s community, and with video appointments. If a patient requires follow up after their first VTAC appointment, it is not possible to book with specific providers which leads to Participants also discussed how prescribing opioids and other narcotics posed particular difficulties.navigating community resources: “Sometimes I feel lost with patients, because I don't know what resources they have access to” (P9). Other participants had a hard time ordering and following up on medical investigations, especially when it involved using provincial computer systems outside of VTAC’s control: “it takes me forever to trace my own investigations… that drives me crazy and it's very time consuming” (P1). Most physicians did not experience these challenges. However, when these issues did arise, they could compound the concerns about non-continuous care described above; importantly, these problems had been recognized internally just prior to the focus groups, and swift action was initiated to remediate them.Participants that were located outside of Renfrew County encountered the added challenge of practicing via videoconference was a challenge for parts of Renfrew County, due to inequitable high speed internet access. One participant noted that it could be difficult to get a good internet connection with a patient, and “even if you can get it, it’s so broken or so pixelated it’s useless” (P1). This issue is compounded by the fact that “most of the patients or at least 50% that we see are older and not necessarily tech savvy” (P2).Participants found that In this mixed methods study, we examined physician perspectives of delivering care through VTAC during the COVID-19 pandemic. Our questionnaire identified that COVID-19 assessments, acute episodic care, and overall physician satisfaction were strengths of VTAC, while the challenges of supporting patients with complex care needs with continuous care were limitations. Focus-group interviews with VTAC family physicians were conducted to further explore these ideas, and our qualitative analysis found three main saturated themes within the data: physician impact, physician satisfaction, and physician challenges.Our results demonstrate that VTAC was associated with positive physician experiences in two major areas: (1) the clinical impacts family physicians were able to deliver, and (2) the satisfaction they derived from using the service.We also identified some areas for improvement, where physicians were less satisfied and felt that the program had the potential to provide even more benefits. In particular, physicians felt that their experience would have been enhanced with firmer guidance on controlled substance prescribing, better information about local health-system services, streamlined registration for external IT systems, and with more support to enable them to better care for patients with complex care needs. Through the research process, these issues have been raised with VTAC program management as quality- improvement opportunities, and have already resulted in improvements, such as enhanced protocols for prescribing controlled substances and improved workflows for safely prescribing long term medications .Based on our findings, we make two recommendations for those considering a local program of hybrid in-person and virtual care such as the VTAC model. First, since participating providers may be serving clients outside their normal practice geography, we recommend creating a centralized provider-friendly knowledge base of region-wide health-system resources, which would serve as a reference for providers. This could be as simple as a shared document containing addresses and phone numbers for different clinics and services, or as complicated as a "wiki" or database, as long as it provides a centralized and used-friendly store of local knowledge to assist remote providers.Second, we recommend having well-defined goals, regular opportunities to receive and act on feedback from providers, and a strategic development process that includes a deep knowledge of the target population, and cooperation and coordination with other healthcare providers and services in the regional healthcare system. VTAC is designed to provide acute, episodic care for patients who have urgent needs, but do not have a family physician or cannot access their usual primary care provider. Participants in our study were very satisfied with the level of care they could deliver to these patients in this context. VTAC is not designed to provide ongoing comprehensive care for patients with complex care needs, and participants in our study were less satisfied with the level of care they could provide to these patients.Renfrew County is working to expand VTAC's "sister program", Integrated Virtual Care (IVC), which provides attachment to a family physician and team-based, comprehensive primary care. VTAC is leveraged to identify patients suitable for admission to the IVC program. Innovative programs such as IVC, that provide permanent attachment to a family physician and comprehensive, team-based care are part of the longer-term solution to the problems of insufficient access to primary care. VTAC can act as a safety net for patients without access to a family physician, until such a time as they have access to comprehensive primary care.This study has several limitations that should be considered when interpreting its results. This study focused on the experience of physicians delivering care through VTAC and does not include an evaluation of the experience of paramedics. We interviewed about half of VTAC's participating physicians, but they may not be a representative sample of physicians overall. In addition, all participants worked in the province of Ontario, Canada, and results may not generalize to other countries or cultures. Lastly, focus groups were conducted remotely, so it is unknown what the impact of doing them in person might have been.Future work could extend this study in several directions. First, as this study's findings are translated into ongoing quality improvement initiatives, future work could help to understand physicians' perceptions of these initiatives and identify further opportunities for improvement. Another promising avenue would be to investigate the major gap physicians identified, the level of care they are able to offer to patients with complex needs. A study of these patients and their perceptions and needs could help to identify ways to better serve them through existing or new local initiatives, including new opportunities offered through virtual care. Further work is ongoing to garner a more fulsome picture of other providers’ experience within VTAC, particularly that of paramedics, as well as to evaluate the experience of patients accessing care through VTAC.Overall, VTAC’s family physicians felt empowered to reduce ED visits, care for unattached patients, and address acute, episodic medical needs. Physician satisfaction was driven by patient gratitude and skills development. However, physicians found it difficult to provide continuous care, and physicians from different locations were not always fully aware of all local community resources available to patients. Our findings suggest that a community based, hybrid model of in-person and virtual care can enhance the experience of family physicians providing care to patients who do not have a family physician or cannot access their regular primary care provider.Additional file 1. VTAC Physician Experience Survey.Additional file 2: Table S1. Preliminary Findings: Data Structure and Major Themes. Table S2. Recommendations and improvements.


While medications for opioid use disorder (MOUD) effectively treat OUD during pregnancy and the postpartum period, poor treatment retention is common. Digital phenotyping, or passive sensing data captured from personal mobile devices, namely smartphones, provides an opportunity to understand behaviors, psychological states, and social influences contributing to perinatal MOUD non-retention. Given this novel area of investigation, we conducted a qualitative study to determine the acceptability of digital phenotyping among pregnant and parenting people with opioid use disorder (PPP-OUD).This study was guided by the Theoretical Framework of Acceptability (TFA). Within a clinical trial testing a behavioral health intervention for PPP-OUD, we used purposeful criterion sampling to recruit 11 participants who delivered a child in the past 12  months and received OUD treatment during pregnancy or the postpartum period. Data were collected through phone interviews using a structured interview guide based on four TFA constructs . We used framework analysis to code, chart, and identify key patterns within the data.Participants generally expressed positive attitudes about digital phenotyping and high self-efficacy and low anticipated burden to participate in studies that collect smartphone-based passive sensing data. Nonetheless, concerns were noted related to data privacy/security and sharing location information. Differences in participant assessments of burden were related to length of time required and level of remuneration to participate in a study. Interviewees voiced broad support for participating in a digital phenotyping study with known/trusted individuals but expressed concerns about third-party data sharing and government monitoring.Digital phenotyping methods were acceptable to PPP-OUD. Enhancements in acceptability include allowing participants to maintain control over which data are shared, limiting frequency of research contacts, aligning compensation with participant burden, and outlining data privacy/security protections on study materials. In the past two decades, there have been substantial increases in the number of pregnancies affected by opioid use disorder (OUD) in the United States (US). From 1999 to 2017, the national prevalence of maternal OUD diagnoses at delivery increased from 1.5/1000 to 8.2/1000 deliveries . HoweverRisk factors for MOUD treatment non-retention during pregnancy and the postpartum period include non-White race, younger age, alcohol and drug use/use disorders, missing healthcare appointments, and MOUD-related factors, such as lower duration or dose of treatment prior to delivery –12. Qualin situ using data from personal digital devices,” is a novel approach for understanding social and behavioral dimensions of health and disease that overcomes many of the challenges inherent in traditional data collection methods , behaviors , and social influences  contributing to MOUD non-retention among pregnant and parenting people with OUD (PPP-OUD), which if understood could offer treatment providers a window of opportunity to intervene before MOUD cessation occurs. Increasing treatment retention among PPP-OUD is critical to support long-term recovery, reduce risk of return to drug use and potential overdose, and provide a stable and safe home environment for parent and child. To our best knowledge, there is no prior or published research on the use of digital phenotyping with PPP-OUD. Given the nature of this type of data collection, understanding attitudes and perspectives of PPP-OUD about the digital phenotyping approach is an important step that needs to be completed before it can be used in a clinical research study. We conducted qualitative research using a theory-driven framework to assess the prospective, or anticipated, acceptability of digital phenotyping among PPP-OUD participating in randomized clinical trial. While smartphone-based digital phenotyping encompasses active (surveys) and passive (sensor) data collection , our inqThis work was informed and guided by the Theoretical Framework of Acceptability (TFA) , which SThe parent study of the current project was Optimizing Pregnancy and Treatment Interventions for Moms 2.0 (OPTI-Mom 2.0) NCT03833245), a prospective clinical trial testing the efficacy of a patient navigator intervention to facilitate linkage to MOUD treatment engagement and psychosocial services among PPP-OUD . This study’s participants were informed that the interviews were being conducted as part of the parent trial and that participation was voluntary. All provided consent prior to qualitative data collection.We conducted this qualitative study using purposeful criterion sampling, which involves identification of a sample based on pre-determined criteria of importance . Our criTwo team members (KC and AW) supported recruitment by compiling a list of potential OPTI-Mom 2.0 study participants who met above criteria and invited them to participate in qualitative interviews through text message. Communications contained a description of the study and the first author’s (EC) contact information. Those who were interested to participate contacted EC through phone, text, or email. When possible, interviews were conducted immediately. No individuals refused participation. Recruitment and data collection were carried out simultaneously between March and May 2022.Data were collected by EC, a female, early career researcher with PhD-level training in qualitative inquiry and without any direct relation to research participants. Telephone interviews lasting approximately 30 min were conducted using a structured interview guide , which wAt the start of the interview, the interviewer informed participants of the study’s purpose and personal interest in the research topic. Field notes were taken during the interviews, and no other individuals were present besides the interviewer and participants. Participants received a $50 ClinCard payment as remuneration after interview completion. Upon observing informational redundancy during data collection, also known as data saturation , 27, we a priori based on TFA constructs from the interview guide. At the indexing stage, we linked the data to the thematic framework by applying deductive codes for TFA constructs. We also applied inductive codes for emergent themes. After coding three transcripts, EC and AW met to compare codes and discuss discrepancies in the application of the coding framework. Intercoder reliability was calculated for deductive codes with Cohen’s kappa, a measure of concordance that accounts for chance agreement  independently analyzed the qualitative transcripts by applying deductive framework analysis that included: (1) data familiarization; (2) identifying a thematic framework; (3) indexing; (4) charting; and (5) mapping and interpretation . First, greement , using Mgreement , both teCOREQ) checklist  explained:“I don’t think there’s anything wrong with it. I think you have to have data for everything. Whether it’s smartphone data or getting it from a computer…no matter where the source comes from, you still have to have it in order to make things better and advanced and move forward.”“I think the immediate is, what are they going to compensate me with, and then afterwards, if I might be actually helping other people.”The main reasons for interviewees’ interest in participating in digital phenotyping research included receiving personal benefits, such as remuneration, and benefitting society. As one participant (P9) shared, somebody hacking in and collecting certain information.” Other people reported apprehensions over data leakage, compromised personal information, receiving spam, and data being sold to third-party vendors or used for malicious purposes, such as identity theft. Interviewees also discussed their thoughts and opinions about which types sensor data collection would be acceptable. Content of text messages, phone calls, and photos were viewed as a “total invasion of privacy” (P4). Location/GPS data was acceptable to some but not to other individuals. As P10 commented, “I do not know really about that…where I’ve been going and everything that people would know… I mean, it’s kind of invasive I guess.”Despite overall positive attitudes, participants reported concerns about privacy and security related to digital data collection and storage. P6 expressed concerns about ““I kind of like my own shit. I would not want to get adjusted to another phone.” Those who favored a study phone provided numerous reasons for this preference, including having autonomy of the phone, not having a data plan to download apps, and having a second phone so that children could use their personal phone while the participant used a study phone. A few individuals did not have a preference and said that either option would be acceptable provided that they could keep the study phone at the end of the data collection period. As P11 explained, “If I had to give it [study phone] back, then I’d rather just keep my phone than go through all the trouble, honestly.” When probed about their preferred phone type, about half of participants preferred iPhone (n = 5) and the other half preferred Android (n = 6).Participants shared opinions about using their own phone vs. receiving a study phone for a digital phenotyping study. Those who preferred to use their own phone cited technical and personal familiarity as the main reasons for this choice. As P1 shared, “For me personally, it would probably just like be in the back of my mind, and…it would just be kind of a silent thing going on, on my phone.” They perceived no increased effort related to study participation during pregnancy, at the birth hospitalization, or at home after delivery, with the exception of P5 who expressed that study participation could be difficult during pregnancy due to morning sickness. Most people agreed that 12 months of participation after delivery would not be an excess burden. As P8 explained, “You always have a little bit of time when their sleeping and stuff. I mean the phone is not a problem. Everyone has a phone all the time, even when they have a baby. So that will not be a problem.”Interviewees reported low anticipated burden of participating in a digital phenotyping study. As P4 stated, “I feel like weekly would be almost too much unless it was like an email or a text message when I could answer it when I want it.” Most participants preferred monetary remuneration. While a few desired to have their phone bill paid as a participation incentive, this preference was situation dependent. As P9 explained, “I do not pay my cell phone bill because I’m on somebody else’s [plan]. So for me, personally, that [phone bill paid] would not be an advantage. I would rather get gas cards over that.”Interviewees made recommendations to limit check-ins with the research staff and provide appropriate compensation relative to burden for study participants. There was no consensus about the frequency of check-ins. Biweekly or monthly was described as appropriate by many participants, whereas weekly was viewed as too frequent for some but not others. P4 shared, Participants described ethicality of digital phenotyping in different ways, including the appropriateness of smartphone-based sensor data collection in the context of a research study or more broadly within society. Whereas they expressed broad support for digital phenotyping research with known, trusted individuals, they expressed frustration about companies and government agencies accessing smartphone data without their consent. One participant (P2) explained:“If it is not a study, it’s annoying to me and I don’t like it, because it causes a lot of things to pop up through your phone and phone calls and text messages. It’s very annoying. But for like a study purpose, I see it a little different because it’s specifically for one reason… I’ve been doing studies with them [the local university] for a while, and so I kind of feel comfortable with, you know, going in for a study, as opposed to how maybe the government collection watches you and stuff like that.”Participants overwhelmingly agreed that smartphone apps harvesting data to improve ad targeting is unethical. They likewise expressed concerns about government tracking of cell phone locations. One participant (P4) commented:“I don’t think it’s right that the government does that. No, I just, I don’t. I feel like that’s my privacy. And I mean, if you signed up for it, sure, because then you know about it. But I feel like a lot of the times people don’t know that they’re consenting to when they press accept on their phone.”“I would just have to, you know, hear exactly what kind of information they would be collecting, and everything like that, all the way down to the very bottom of it, you know what I mean.”Interviewees conveyed the importance of knowing specific study details, such as types of data to be collected and how information would be secured and stored, before agreeing to study participation. As P10 explained, “Everybody has their phone on them all the time.” P9, however, commented that keeping a phone charged may be challenging for individuals who are homeless: “They probably would not be able to…keep the battery life because they are charging their phone at McDonald’s versus in their home or in their car.” Despite participants’ confidence, over the course of the interviews they described instances of losing or breaking a phone, allowing children to use the phone, not having a data plan to download an app, and not having autonomy of the phone. As P3 shared, “I’m going through a divorce right now… My ex, like, took my kids. I have not seen them for 4 months, and then yesterday…he turned my phone off.” Another participant (P6) explained:Participants reported high self-efficacy to perform behaviors required to participate in a digital phenotyping study. None expressed concerns about keeping smartphones with them, charged, and turned on at all times. As P3 stated, “My kids tend to use the phone, like when I’m napping, or when we’re at a doctor’s appointment. I will give the phone to them so that they’re distracted, or like at church when I’m trying to keep them down. But other than that, I have my phone.”“I think we are going to learn a lot from it. We’re going to learn a lot of people’s behaviors, a lot of their habits, a lot of their freaking out.” Though not specifically asked, interviewees suggested how smartphone-based sensor data could be used to benefit study participants. Making data available for participants to track and improve their personal health behaviors, including sleeping patterns, screen time, and social media activity, was frequently mentioned. P2 suggested, “If you all would show me, like, this is how much you slept, then I would know, and I could maybe make some changes in my lifestyle.” One participant (P6) also proposed a mobile text messaging alert to remind participants to take their MOUD treatment: “If I had a little notification or you guys pinged me and were like, oh, it’s past your medication time… Then I would be like okay, you know, like a quick little reminder—then I would not forget it.”Participants perceived that digital phenotyping was a good opportunity to learn about their and others’ behaviors. A participant (P7) commented, This study applied the TFA to assess the acceptability of using digital phenotyping methods, specifically data collected passively from smartphone sensors, with PPP-OUD. We found that digital phenotyping approaches are generally acceptable to PPP-OUD when individuals accessing the data are known and trusted and participants maintain control over which data are shared.Participants’ attitudes and perspectives about digital phenotyping were influenced by the perceived universality of smartphones, their high self-efficacy for using smartphone technology and apps, and the type of sensor data to be collected. Approximately half of participants perceived that sharing location/GPS data was too intrusive. This is in line with findings from past research conducted among patients in drug treatment that reported 46 and 62% of participants were not comfortable with smartphone apps gathering location/GPS data , 36.In 2021, 85% of American adults owned a smartphone , and ratThe anticipated burden of participating in a digital phenotyping study during pregnancy and the postpartum period was low among study participants. Most people indicated that 12 months of participation after delivery was acceptable provided that remuneration for participation over time was appropriate and research contacts were minimized. While financial incentives were the favored method of remuneration, alternative compensation strategies, such as paying cell phone bills or allowing participants to keep the study phone, were popular and could be used in future studies to encourage participation from a wider pool of people. Limiting research contacts may also increase retention as some research suggests that contact frequency is associated with attrition in longitudinal studies .Participants believed that digital phenotyping was ethical from a research perspective, but they expressed concerns about data security and privacy, such as how data would be used, stored, and protected from being downloaded by or shared with outside entities. Notably, participants’ concerns were mitigated by having institutional trust and perceptions that research team members would ensure privacy and confidentially of their data. There is strong agreement among experts of various disciplines that privacy, security, and trust are central ethical issues in digital phenotyping . These iOur findings also provide initial insight into potential interventions that may be used to decrease perinatal MOUD non-retention. One participant perceived that text messaging would be valuable for facilitating MOUD adherence. Text messaging interventions can be simple and cost-effective in the promotion of medication adherence and improve health outcomes among people with SUDs and mental health disorders . One stua priori due to time and resource constraints. Finally, we coded qualitative data according to a priori codes based on TFA constructs, which have a considerable amount of overlap. Other investigators may have applied codes and constructed themes in an alternate manner.A primary strength of this study includes the use of a guiding theoretical framework to evaluate acceptability. As with all qualitative research, the main limitation of this study is generalizability. We used purposeful criterion sampling to select a narrow sample of PPP-OUD. While the sampling method was appropriate and improved the rigor of this research, we cannot extrapolate findings to PPP-OUD outside this sample. In addition, all participants in this study were of non-Hispanic ethnicity, and most of White race, and responses given by these individuals may not generalize to persons of other ethnic and racial groups. Another limitation is that many participants were still engaged with parent study and motivated to speak with the interviewer. Views of PPP-OUD who were lost to follow-up in the parent study, not interested to discuss digital phenotyping, or not eligible for the parent study may have had dissimilar perspectives and attitudes. Moreover, we did not conduct member checking with participants after transcribing the interviews in order to establish credibility of findings; however, this decision was made Digital phenotyping is a novel approach for understanding factors contributing to MOUD non-retention, however information has been lacking on whether it is suitable to PPP-OUD. This study provides initial evidence that digital phenotyping methods may be acceptable to PPP-OUD when people accessing the data are known and trusted and participants maintain control over which data are shared. Enhancements to acceptability include allowing participants to maintain control over which data are shared, limiting frequency of research contacts, aligning compensation with participant burden, and outlining data privacy/security protections on study materials.elizabeth-charron@ouhsc.edu.The datasets presented in this article are not readily available because due to the nature of this research participants of this study were not asked and did not agree for their data to be shared publicly. Requests to access the datasets should be directed to The studies involving human participants were reviewed and approved by University of Utah Institutional Review Board. The patients/participants provided their written informed consent to participate in this study.EC conceptualized the study, developed the methodology, conducted the investigation, and prepared the first draft of this manuscript. EC, AW, and KC recruited participants. EC and AW analyzed the data. EC, AW, KC, WA, JB, SP, MB, AG, EK, MS, and GC critically reviewed and revised the manuscript for important intellectual content. All authors accept responsibility for the integrity of the work reported and approved the final version of this manuscript.This study was supported by a grant from the Centers for Disease Control and Prevention . Cochran and Gordon are supported by a grant from the National Institute on Drug Abuse (1UG1DA049444–01 PI: Gordon/Cochran). The content is solely the responsibility of the authors and does not necessarily represent the official views of the funder.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.


In Vitro Assay to Evaluate Treatment Resistance in Patients Receiving Immunosuppressive Therapy” published on 25 January 2021. Specifically, concerns have been raised that three individuals, Veronica Sánchez Rodríguez, Marianna di Scala, and Javier Dotor, have been excluded from the author list. An investigation was conducted in line with COPE guidelines involving the authors’ institution and the funding body, but unfortunately no conclusion could be reached.With this notice, Frontiers states its awareness of concerns regarding the authorship of “The Immunobiogram, a Novel This Expression of Concern was approved by the Chief Editors of Frontiers in Immunology and the Chief Executive Editor of Frontiers. The authors do not agree to this Expression of Concern.