We asked for their experiences during COVID-19 confinement regarding everyday life, AN symptoms, and treatment. We used thematic analysis to interpret the data. The main themes identified from the patients’ interviews involved restrictions of personal freedom , interruption of the treatment routine , changes in AN symptoms (more exposure to triggering situations), COVID-19-related fears, and compulsions but also potential opportunities . The parents discussed changes in daily routines as negative  and positive . They expressed reservations about reduced outpatient monitoring and increased teletherapy and discussed challenges in keeping contact with the child and clinicians during inpatient treatment. Moreover, the parents discussed deteriorations and improvements in the patients’ psychopathology. Clinical implications from these in-depth insights include the importance of strengthening communication between changing staff cohorts, patients, and parents; motivational work; and joint weight monitoring with the therapist.COVID-19-related restrictions may have a serious impact on patients with eating disorders. We conducted semistructured interviews with female adolescent patients with anorexia nervosa (AN) ( Emerging evidence suggests that the coronavirus disease 2019 (COVID-19) pandemic has a significant impact on the mental health of the entire population, with individuals suffering from psychiatric disorders, such as eating disorders, being a particular vulnerable group . SeveralPotential pathways leading to this aggravation of eating disorder symptoms include increased social isolation and less distractions, increased exposure to stressful messages , food insecurity, less opportunity to exercise, and limited access to professional health care ,13,14,15Besides a negative impact of the COVID-19 confinement on adult eating disorder patients, also positive consequences have been reported in the literature, including greater connection with the family, more time for self-care, taking over responsibility for recovery, increased self-efficacy, and developing alternative coping strategies and resources facilitating recovery ,5,6,24. To date, research on the impact of COVID-19 measures on adolescent patients with eating disorders is scarce. However, it might be that the effects of COVID-19 measures reported on adult patients also apply—to some extent—to adolescent patients. Furthermore, research in general population samples of adolescents indicates that symptoms of depression and anxiety are on the rise ,26, whicCaregivers often play a key role in the treatment and recovery process of patients with eating disorders; however, the impact of the COVID-19 confinement on caregivers and their role in supporting their loved ones with an eating disorder has not gained much attention so far. A qualitative study involving caregivers of adult patients with AN showed that concerns about the provision of professional health care for patients, managing patients’ and families’ needs, and coping with patients’ aggravating eating disorder symptoms were most challenging for them . MoreoveOn 16 March 2020, the Austrian government announced restrictive measures due to the COVID-19 pandemic, including closures of school and universities, closure of shops not relevant for basic supply, and curfews. Restrictions ended on 30 April 2020. Additional measures were taken with regard to inpatient eating disorder treatment at the Department of Child and Adolescent Psychiatry, Medical University of Vienna, Austria, where this study was conducted. These include splitting hospital personnel into two cohorts who work alternately in a 2-week rhythm and restrictions for visitors of inpatient adolescents . Additionally, inpatients were confined to hospital grounds. Furthermore, face-to-face medical and psychotherapeutic eating disorder treatment for outpatients was reduced to a minimum, and remote interventions  were offered whenever possible.Current research indicates that COVID-19-related restrictions may have a serious impact on patients with eating disorders. However, little is known about the experiences of adolescent patients and their parents as well as how adolescents in inpatient treatment experienced COVID-19 restrictions. Moreover, the impact of confinement on eating disorder patients may vary based on the specific measures taken in countries and hospitals, which demonstrates the need for local and setting-based studies. Thus, this qualitative interview study aims to examine the impact of COVID-19 confinement on adolescent patients with AN and their parents in Vienna and its surrounding area with regard to eating disorder symptoms, inpatient and outpatient eating disorder treatment, and daily routines, including family and school life. In this study, we focused on patients with AN for the following reasons: First, our clinic is specialized on the treatment of adolescent AN, and thus, relatively more patients with AN are assigned to our clinic than patients with other types of eating disorders. Second, due to certain personality traits often seen in patients with AN , AN patients might be especially prone to loss of daily structures. Moreover, a previous study indicated that AN patients were less satisfied with remote interventions compared with patients with bulimia nervosa and binge eating disorder. Considering the often life-threatening course of AN, we prioritized our research on this group of patients.Participants were purposively recruited from the Eating Disorder Unit of the Department of Child and Adolescent Psychiatry, Medical University of Vienna, Austria. We approached adolescent patients aged between 13 and 18 years who were diagnosed with AN according to DSM-5 diagnostic criteria  and received inpatient or outpatient treatment. In detail, we approached all inpatients with AN who were admitted to our ward between end of April and May 2020 and who were able to participate in an interview . Outpatients were recruited via an ongoing project evaluating an outpatient cognitive behavioral treatment approach called MANTR-a (Maudsley model of anorexia nervosa treatment for adolescents and young adults)  and via The sample size for this study was derived based on data saturation, which we defined as the point whereby further data collection would yield no further major themes. This point was determined by mutual agreement of the interviewers. In total, interviews were conducted with 13 female adolescent patients with a mean age of 15.9 years (SD = 1.4). Two approached patients declined to participate without giving specific reasons. Nine patients were diagnosed with AN restrictive type and four patients with AN binge–purging type. At the time of data collection, eight patients received outpatient treatment, and five patients received inpatient treatment (at least for 2 months). Outpatient treatment included individual psychotherapy  as well as weekly or biweekly medical checks by a child and adolescent psychiatrist.The mean body mass index (BMI) was 16.7 (SD = 1.4), ranging from 13.7 to 19.3. Sex- and age-specific BMI percentile ranged between <1st and 25th percentile . The mean duration of the eating disorder was 23 months (SD = 16) ranging from 5 to 48 months. Psychiatric comorbidities were diagnosed in four patients . Eleven patients lived with both biological parents, and two patients lived with single parents (mothers). Eleven patients had at least one sibling living in the same household.Additionally, 10 out of the 13 parents  agreed to participate in the qualitative interview. Their mean age was 49.7 years (SD = 2.4). Seven parents had a university degree, two had an A-level degree, and one had a below A-level degree. Five parents were full-time, and 5 were part-time employed; however, 11 parents reported that the COVID-19 confinement has changed their working life . Of the parents who declined participation, two stated to have no time, and one expressed no interest in participating in the study.No participant (neither patients nor parents) reported a previous infection with COVID-19; two participants reported COVID-19 infections of relatives not living in the same household.The topic guide of the semistructured interview  experienced in the field of adolescent eating disorders . Two interviewers (those who performed 17 out of 23 interviews) already had experience with conducting semistructured interviews. All the interviewers participated in an interviewer training given by M.N. With the exception of one interview, the interviewers had no or minimal relationship with the interviewees prior to this study.®  But this whole exceptional situation with Corona has also somehow provoked this, because otherwise it would never have come so far. Otherwise, I would have continued to have medical checks in the hospital” (#12).(B2)Challenges and opportunities of teletherapy: In general, the psychological/psychotherapeutic online treatment was perceived as “rather satisfactory” (#07) by the patients whereby a bilateral video connection was preferred over audio connection only because eye contact, facial expressions, and gestures were regarded as important during the therapeutic process. However, face-to-face therapy was preferred in general as difficult topics stay in the doctor’s office: “For me, it was like, I go to my therapist, I leave my problems there, I go home. […] Now, many things take place in my room, and I have all these problems, these conversations right on the spot. This felt strange for me at the beginning.” (#09). While receiving therapy without having to travel was seen as advantage of teletherapy, privacy concerns when psychotherapy took place in one’s own room were discussed as main challenge: “I can′t really tell everything and be 100% honest, because I′m always afraid that someone is listening” (#01).(B3)Changes of staff cohorts and treatment offers in the inpatient setting: Due to the frequently changing staff cohorts in the psychiatry wards, it was perceived as difficult to build relationships with clinicians, which was additionally aggravated by wearing face masks. It was “difficult, because we could never concentrate on one person [doctor], who guides us through our whole inpatient stay, but we always had to switch, and that was a bit tedious” (#05). Furthermore, inpatients partly experienced insufficient agreements between staff from different cohorts. On the positive side, new therapists joined the team and offered new options for treatment and free-time activities like “hypnosis or yoga, which was somehow uncommon, but also pretty cool” (#03).This theme covers risks, challenges, and opportunities that are associated with changes in the treatment routine, both in the outpatient setting  and in the inpatient setting (change of staff cohorts and treatment offers). The following three subthemes were identified:(C1)Boredom and feeling of being observed, triggering eating disorder symptoms: Feeling bored and isolated at home, experiencing little distraction, and feeling observed by family members during eating and exercising triggered AN-related cognitions, restrictive eating behaviors , obsessive exercising, and binge–purging behavior: “She [my mother] is at home most of the time, my siblings too. Sometimes, I feel a bit observed, and then I feel bad when I’m eating so much” (#01). “I not only control my weight to the gram, but also my food intake. I′ve started to weigh food again, to count calories, and to track steps. […] This was because I′m always at home now. I′m not allowed to go to school because of my blood values […]. I spent lots of time in my “anorectic world” (#12). AN-related cognitions and behaviors were additionally boosted by social comparisons and exchange with peers: “Really every one of my classmates posted something like ‘Shit, quarantine, I’m getting fat now.’ And that triggered me so much” (#13).(C2)COVID-19-related fears and compulsions: This included worries regarding continuation of psychotherapy and medical checks, continuation of school lessons, professional and financial situation of the parents, and fear of a COVID-19 infection and infection of close others, particularly grandparents: “What I was very worried about was my grandparents […]; my grandma has cancer and is getting chemotherapy, and her immune system is very weak. And my biggest concern was that she will get infected” (#09). Furthermore, fear of not being able to buy certain foods and increased obsessional washing were discussed.(C3)Improvement in symptoms, indicating a normal treatment course: Other patients reported no direct effects of the COVID-19 pandemic on eating disorder symptoms or exercise behavior but mentioned general improvements in symptoms , indicating a normal treatment course: “I notice that it is getting better now, that I don’t think about food so much anymore, that after the meal I no longer have this mind spinning […] and yes, therapy helps me a lot” (#08).This overarching theme describes deteriorations and improvements in eating disorder symptoms and other psychopathology, which are related to COVID-19 measures and characteristics of a normal course of therapy. The following three subthemes were identified:(D1)Less stress allows to pay more attention to own needs: The patients highlighted that less stress regarding school, leisure, and social contacts allowed to pay more attention to one’s own needs. This has allowed them to better enjoy the time available and to follow their own needs: “I have escaped from this school stress a little bit and was able to better listen to myself [...] Maybe this was also a step that was necessary so that I could calm down myself a bit” (#11). Another patient told, “So I have paid more attention to my emotions and to my needs, what I wanted to do now, and not whether it fits into my schedule. So, I just have done what I really wanted to do and haven’t done things that I didn′t want to do” (#10). Although having more time for oneself bears the risk to get caught in AN-related cognitions, it also opened up the opportunity to “work towards the opposite direction” (#12), towards recovery. Experiencing lower stress levels was further associated with more flexible eating  and improved mood.(D2)Experiencing a more intensive time with the family: As many parents were working from home and siblings were at home due to homeschooling, there was more time for joint family activities and conversations, which facilitated growing together as a family. One patient described a barbecue event as follows: “We were all together, and it was a bit like in a storybook because we often don’t have this family idyll because I don’t eat with them most of the time” (#09). Another patient told that she has start talking with her sister about her eating disorder during this time: “Two or three days ago, it happened for the first time since I had this disease that I talked to my sister a little bit about it, and that was very relieving” (#02). Moreover, visits from parents at the ward were experienced “more consciously” (#03) ;“I could use the time with my mom more intensively because I spent my time alone with her every day” #08).(D3)Promoting autonomy and self-organization skills: Finally, patients acknowledged that the interruption in everyday and school routines enabled the promotion of autonomy and self-organization skills, including working autonomously on school assignments and planning of the day structure. For example, a patient positively highlighted “that I am a bit more on my own, that I have to look after myself, that I have to create a daily routine for myself, that the school doesn′t do that for me, that the parents can′t always do that for me” (#13).This theme described how the COVID-19 period opened up new opportunities to take care of one’s own needs and promote family cohesion.Three main themes were identified from the patients’ interviews, which are described below.(E1)Challenges regarding the organization and creation of a day structure: Parents of adolescents with AN faced various changes regarding daily structures and routines, which were challenging for them. These include changes regarding their job situation , family life (partner and siblings spent more time at home), and social life . Additionally, care responsibilities for the child with AN needed to be fulfilled. Parents felt that they needed to create new routines for their children’s lives : “They [the children] have needed help with their school stuff […]. Every two weeks, they have got new assignments. Each new assignment was a bit more of work. This was really a challenge. And at noon everyone is hungry. This has to be managed at the same time. And each of us has its own videoconferences […]. That caused me stress” (#06). In particular, managing the visits of the child in inpatient care was challenging for the one parent who was allowed to enter the hospital building. This was perceived as “exhausting, travelling to the hospital every day, all the responsibility that you have to bear alone” (#08).(E2)Experiencing more time with family: The changed family routines have enabled more family time, which was perceived as positive: “I have had all my children around me; it was very harmonious. We played a lot together, did a lot of things, even prayed, whatever the children wanted to do. Many things were initiated by them. From that point of view, we experienced a lot of positive things as a family out of the fact that we were all together” (#02). More family time has also fostered more intensive conversations between parents and the sick child, between siblings and between the parents, which has “brought us together even more” (#12). For example, one mother described the relationship between the sick child and her siblings as follows: “It’s simply a fact that she [the sick child] has managed to talk with her siblings, especially very intensively with her sister […]. I think this has made a big difference that she has improved so much recently” (#02).(E3)Slowing down for the whole family: COVID-19-related restrictions enabled the whole family “to slow down” (#13) and to strengthen self-care behavior. Less school and social stress due to school closures took the pressure off the child diagnosed with AN and provided the opportunity to work on recovery without the fear of missing something, “an opportunity that can be seized” (#05). A mother described, “What is good for her is that the school stress has gone. […] This ‘you have to study now’ and this self-imposed ‘you have to be good at school’ has dropped completely” (#10).This theme covers changes in the everyday routines of the parents and the entire family regarding the family, work, and school setting as well as related challenges and opportunities. The following three subthemes were identified:(F1)Reservations about reduced outpatient monitoring and increased teletherapy: Parents reported that outpatient face-to-face contacts did not take place at all or were significantly reduced, although they acknowledged the clinicians’ effort to keep them going. Nevertheless, face-to-face appointments would have been necessary as “it is important that they [the patients] are weighed and their urine is checked […] and clinicians see, how they look like, whether there is a relapse or things have improved. This cannot be seen online; this could be totally distorted” (#02). Remote therapy was seen as an important “back-up, that she has someone she can talk to […] However, it is, of course, quite different when it takes place face-to-face” (#10). Providing a place guaranteeing a confidential atmosphere for teletherapy was seen as a basic requirement so that teletherapy can work. Due to the reduced outpatient face-to-face appointments, parents reported to have listened more attentively to any changes in the child’s eating disorder symptoms  and felt that they should take over the doctors’ responsibility to a certain degree.(F2)Altered family contact with the child during the inpatient treatment: Parents and patients were burdened by the visiting regulations at the ward as the personal contact between the family and the child was strictly limited and “normal” family life was not possible. Particularly, visiting restrictions to the same one parent per week was experienced as burdensome: “It is difficult for me to understand that it was not possible for the parents to alternate. And for my daughter, this was twice the challenge. I always had the impression that she really suffered from the fact that only one of us could come and that she always had to decide who should come next week” (#04). A mother described a visit as follows: “I sometimes felt as if I were visiting her in prison. She also felt like that. She has sometimes stood in front of the windows and said, ‘The people here can go where they want, they can get out, and I′m trapped here.’” (#08). The short visiting hours and the insufficient availability of space allowing retreat in the visiting area and the obligation to wear face masks during visits were mentioned as burdensome by the parents.(F3)Maintaining contact between parents and treatment staff: Keeping regular contact between parents and clinicians/therapists during COVID-19 confinement to exchange relevant information was regarded as important. While some parents felt well informed, other parents pointed out that they had too little contact with the clinician after therapy has shifted to teletherapy or due to being not allowed to enter the ward during inpatient treatment of the child: “Sometimes, I would have liked more information as a parent, because I had no contact with the nursing staff at all, because I could not enter the ward. Issues remained unresolved” (#08). Furthermore, parents emphasized the importance of receiving parental support regarding how to deal with the eating disorder of the child.This topic covers the parental perspective regarding the outpatient and inpatient treatment of the children with AN during the COVID-19 confinement. Three subthemes were identified:“She is eating more regularly and faster than in the past. […] This more regular eating is because she is always at home now—during weekdays she had to eat in school together with her friends, and that hasn’t worked out so well. […] And eating together is good for her” (#10). Regarding physical activity, parents reported difficulties of patients to suppress the urge to exercise . However, most parents described the actual exercise behavior as moderate and not pathological. Regarding mood, some parents described their children as increasingly withdrawn, irritated, and depressive, which was partly explained by not being allowed to leave the hospital building for inpatients and missing daily routines and opportunities to pursue hobbies for outpatients: “At the beginning, it was difficult for her to deal with the situation, that she was at home all the time, that she could not follow her routines like the dance school, that she could not see her friends at school […] That has set her back. She was more irritated and withdrawn” (#02). However, other parents described the mood as more cheerful and relaxed, which they attributed to lower social stress and better family relationships.Changes in eating disorder and other psychopathology that were observed by the parents during COVID-19 confinement include deteriorations and improvements. The eating behavior was described as increasingly restrictive and selective, on the one hand, which was attributed to fewer social contacts and common family meals that were perceived as irritating and stressful. On the other hand, other parents reported more flexible and regular eating habits, which they also attributed to common family meals and lower social stress: To the best of our knowledge, this is the first qualitative study focusing on an adolescent population with AN exploring the impact of the COVID-19 confinement. We aimed to explore whether and how the COVID-19 pandemic affected eating disorder symptoms and AN therapy as well as family and school life. In this study, we also included the perspective of the parents as they play a key caregiving role, and their perspectives might differ from the perspectives of their children. Indeed, a general observation from the interviews was that the opinions of adolescents and parents largely coincided, which was true for challenges of the COVID-19 period  as well as for opportunities . This speaks for the validity of the obtained results.It must be noted that some of the themes identified in this study might relate not only to adolescents with eating disorders but also to adolescents in general. Particularly, outpatients might have—to some extent—experienced social distancing measures  in a similar way as their healthy peers. For example, the theme about feelings of being imprisoned and bored as well as missing close others identified in the present interviews resembles the general observation of increasing social isolation and loneliness among the general population of adolescents during this period . HoweverRegarding the impact on the patients’ eating disorder symptoms, the results of this study are consistent with those of a qualitative study including adult patients in the UK, which described the COVID-19 period as a catalyst for either exacerbation of symptoms or recovery . AdolescThe role of school closures in adolescents’ mental health is currently excessively discussed in the public media and scientific literature. While school closures and increased homeschooling are predominantly associated with elevated mental health concerns in adolescents ,41, the Furthermore, changes in the treatment and therapy routine may have had an impact on the course of the eating disorder. Outpatient face-to-face contacts were drastically reduced, which could have only been partly compensated by remote interventions. This is noteworthy because help-seeking behaviors of patients with eating disorders and their caregivers generally increased during the COVID-19 period . AdditioA key aim of this study was to explore how eating disorder treatment and therapy during the COVID-19 period was perceived and how treatment during this exceptional situation can be improved. Indeed, the findings from this study have implications for clinical practice. This is the first study that sheds light on how COVID-19 restrictions were perceived in adolescent patients receiving inpatient treatment. First, due to exit restrictions and boredom, leisure time at the ward was perceived as less positive. The clinic staff is therefore called upon to offer alternative activities and motivators that are not linked to external contacts that counteract boredom and increase treatment engagement. Second, the clinic management is challenged to find the right balance between strict visiting regulations to prevent COVID-19 infections and patient needs. This study undoubtedly demonstrates that the actual visiting regulations imposed great burden on the adolescent patients and their parents. Being able to see both parents regularly is a basic need of adolescents. Having been forced to decide whether the mother or father is allowed to come to visit was burdensome for patients and meant organizational challenges for parents. Thus, allowing alternate visits from both parents should be considered. Moreover, if visitors are not allowed to enter the patients’ rooms, a visiting area with enough space facilitating appropriate retreat for the family should be provided. Third, in order to ensure smooth treatment continuity for patients, it is important that the communication between clinicians of changing staff cohorts is sufficient and transparent. Rotations of staff cohorts should be reduced to a minimum.For patients in outpatient treatment, teletherapy was generally well accepted by the patients as it ensures therapy continuity and allows flexibility. However, some basic requirements must be met, including stable Internet connection, use of video instead of audio only, and privacy. These aspects are important for adolescent psychiatry patients in general . The treTwo further aspects should be considered for both inpatients and outpatients. Therapists should be aware that the motivation to work on recovery might be reduced. Thus, AN therapy during this time should particularly focus on motivational work and goal setting. Furthermore, the present study indicates that communication with parents may have somewhat been neglected. This may be one reason why parents expressed more reservations about teletherapy than patients themselves, a finding that has also been reported in a previous study . TherapiDue to hospital routines and COVID-19 regulations, interviews were conducted face-to-face and online. Moreover, in some online interviews, no bidirectional video connection was possible due to bad Internet connection and WiFi capacity. Different methods may have led to differences in the narrative flow and may not be fully comparable; however, the notes of the interviewers and observers do not provide any indication of this. As emphasized by Weissman et al. , traditiThe impact of the COVID-19 confinement on adolescent patients with AN is ambivalent. This period bears challenges and opportunities for the course of the disease, therapy, and family and school life. This study identified some risks of the current treatment practice and overall circumstances, like self-monitored weight or monitoring by parents, reduced motivation to work on recovery, restrictive visiting regulations at the ward, and partially neglected therapist–parent communication. The results provide useful implications on how to improve treatment during the pandemic. On the other side, positive consequences of the COVID-19 period must also be truly acknowledged, which include improved family relationships, being less stressed, and having more time for self-care. This, in fact, raises the question about the pressure society generally places on eating disorder patients and their families and society’s role in the development and maintenance of the eating disorder. Finally, it must be noted that this study focused on patients whose eating disorder started prior to the COVID-19 crisis only. Future studies may focus on adolescents who have newly developed an eating disorder during the COVID-19 period. Our clinical impression indicates that there was an increase in admissions of new-onset AN cases during the last months of 2020 and first months of 2021 with distinct phenomenology and etiology. This may encourage further research regarding the role of COVID-19 confinement in the devolvement of an eating disorder.


Increasingly, community-based models of doula care are receiving attention as possible interventions to address racial inequities in maternal health care experiences and outcomes. In 2018, community-based organization SisterWeb launched to provide free culturally congruent community doula care to advance birth equity for Black and Pacific Islander pregnant people, with funding from the San Francisco Department of Public Health. We conducted a process evaluation of SisterWeb’s first 1.5 years of existence to identify multilevel barriers and facilitators to implementation of their programs.Guided by the Equitable Evaluation Framework™, we conducted 46 in-depth interviews with individuals from 5 groups: SisterWeb leadership, doulas, doula mentors, and clients, and external stakeholders.Barriers included having diverse clientele groups with unique needs, an ineffective payment model, and simultaneously building an organization and developing and implementing programs. Facilitators included the presence of established strategic partnerships, positive reception of services due to an unmet need for culturally and linguistically congruent pregnancy and birth support among SisterWeb’s clients, a clear organizational vision and mission, and a unique doula cohort model.program versus an organization, including the required resources of each, the sustainability of payment models for community doulas, and the provision of culturally relevant, needed services within priority communities. Furthermore, strategic partnerships with maternal health stakeholders in birthing sites, particularly hospitals, are vital to the success of a community doula program.Our findings suggest developing community doula programs pay close attention to the difference between launching a  Founded in 2018, SisterWeb works towards its mission in two ways: by providing no-cost, high quality doula care to birthing people of color most at risk for adverse birth outcomes, and by providing their community doulas—who are from the communities they serve—with stable employment and the tools, job skills, and mentorship needed to succeed as professional birth workers.In their own words, SisterWeb works to build a San Francisco where “Black, Pacific Islander, and Latina/o/x families…are centered and uplifted in their reproductive journey to welcome their children with respect, dignity, joy and pride, leading to thriving families and communities and increased birth equity and justice” , and their model of care includes a total of 8 visits that include an initial intake visit and three prenatal visits, in-person labor and birth support, and four postpartum visits. Clients needing extra support are internally designated “Code Purple”; these clients have more interaction with doulas and are connected to additional community resources. Table We conducted a process evaluation of two of SisterWeb’s three programs, Kindred Birth Companions (KBC) and M.A.N.A. Pasefika, between 2018 and 2020. As the evaluation was funded to examine these two programs, the process evaluation did not include SisterWeb’s third program. The purpose of the process evaluation was to monitor the implementation of the programs—whether they were being implemented as designed, working well, accessible to clients and doulas, and supported by hospital staff. The full process evaluation included both qualitative and quantitative components; this paper reports on the qualitative component, which focused on implementation-related barriers and facilitators. All methods and results are reported according to the Consolidated Criteria for Reporting Qualitative Research  were invited to participate in interviews. While some individuals (n = 23) were unable to be interviewed, mainly due to nonresponse to recruitment emails, we were able to interview everyone who wanted to participate during the study period.Interviews (n = 46) occurred between September 2019 and July 2020. Interviews with doulas (n = 6) and mentors (n = 2) explored their experiences providing care and mentorship, and their perspectives regarding barriers to providing care and the sustainability of their work with SisterWeb. SisterWeb leadership interviews (n = 4) focused on program development, barriers and facilitators to program implementation, and any programmatic changes that have occurred since the organization’s launch. In client interviews, we investigated experiences with and reasons for program participation; we conducted interviews at two timepoints: after the first or second prenatal visit (n = 14) and after at least one postpartum visit (n = 9). Finally, we interviewed external stakeholders (n = 11) about their relationships and experiences with SisterWeb to further identify barriers and facilitators to program implementation; stakeholders included SisterWeb’s partners at local hospitals and city and social service agencies. All interviews were conducted via phone, videoconference, or in-person, and in English; interviewees may have spoken other languages, including Spanish and Samoan.Interviews were transcribed verbatim and lasted an average of 63 min. Members of the study team reviewed transcripts for errors and removed identifiers. We analyzed the data utilizing a Rapid Assessment Process (RAP), a team-based, rigorous analytic approach. RAP includes iterative analysis, additional data collection, and data triangulation in order to develop preliminary, actionable findings; the rapid turnaround of data findings makes this approach well-suited for qualitative research informing developing interventions as it allows for concurrent intervention modifications Beebe, . Despite“I think SisterWeb has done a really good job of making sure that anything that they apply for or get funding for is in line with their mission and vision, and not compromising who they are or their integrity and what kind of care they want to provide.”-StakeholderFrom inception, SisterWeb’s leadership outlined a vision and mission for building a community of professional birth workers and educators from and serving Black, Pacific Islander, and Latinx birthing people in San Francisco. The vision and mission served as a clear, uniting force that aligned SisterWeb staff, doulas and mentors, and community partners, which ultimately helped propel their work forward and implement their programs even when facing challenges.Our findings revealed that SisterWeb staff were mission-driven and deeply committed to addressing maternal health inequities in their community. Stakeholders, including healthcare providers and public health professionals, were impressed with Sisterweb’s authenticity, integrity, transparency, and commitment to their mission. Further, doulas and mentors strongly identified with the organization’s mission and passion for addressing maternal health disparities and cited this as a key reason that they enjoyed working with SisterWeb.I feel like [being a SisterWeb doula] has been life-changing… I have gained information and connections and mentorship that I would not have had, had I not been a part of this program.-DoulaOne aspect of SisterWeb’s program is the doula cohort model, where each client is assigned a cohort consisting of two or three doulas and a mentor. SisterWeb chose to use a cohort model based on their experiential knowledge and review of previous successful doula programs. They hypothesized that the cohort model would increase the sustainability of doula work, as doulas can rely on each other during family emergencies and illness, provide a way for doulas to balance their doula responsibilities with their personal and other professional commitments, and allow doulas to work in shifts during long births. SisterWeb incorporated mentors into the cohort to provide doulas with professional and emotional support, guidance, and coaching to support their confidence and ability to manage clients’ pregnancy and birthing needs, a benefit that doulas described wishing they had earlier in their careers.“[Having two doulas] makes me feel, you know, confident about the support that I'm getting from this program, that I'm going to have at least one person from this program to be there.”-Client“If I was, like, a doula on my own I would definitely need a partner. But the cohort system like, it kind of gives you like a built-in backup. You always know that someone has your back like if it goes too long, if you need a break, if your work gets, you know, you got work conflict like for prenatals and the birth. Like it’s honestly like, a huge, huge, huge relief.”-KBC DoulaThis model proved to be an essential ingredient of SisterWeb’s ability to implement their programs successfully. Interviews revealed it was effective at providing increased support for clients by ensuring they always had a doula available while also allowing doulas to share workloads and avoid burnout.“Championing our work within the hospital [contributes to] the longevity of fixing the problems…And the money again, it'd be easy to say that's the [most important] thing. Do we need it? Yes. To run it? Yes. But the real change is happening with the clients in the hospitals. It's like the secret power.”-LeaderRelationships with outside organizations and stakeholders, particularly at hospitals, are crucial to the delivery of doula care, especially the integration of doulas into labor and delivery units. Partnerships were also integral to SisterWeb’s work to ensure appropriate referrals to their program.We found that, overall, SisterWeb was positively regarded among the community stakeholders that they work with, and that this was key to launching and implementing their doula programs. SisterWeb leaders were strong at maintaining strategic partnerships within the community and with local institutions. One example of this is the Champion Dyad Initiative, a partnership between representatives at local hospitals and SisterWeb that was co-developed with key stakeholders, including Expecting Justice. As one leader said, Champion Dyads allows SisterWeb to “discuss what's going well with the doulas working at that particular hospital and what isn't going so well.” While we found there were some areas for improvement, such as streamlined communication between SisterWeb and referral sites, overall, these positive, established partnerships were critical to SisterWeb’s success. Importantly, in building these relationships, SisterWeb was able to leverage the advocacy and legacy of other birth justice organizations and doulas in San Francisco."Support system as far as like somebody that's communicating to me, somebody to be there for my labor... Especially with this being my first child. Like I just need a lot of – I won't say a lot of attention, but I just need a lot of people there worried about how I'm feeling and what I'm going through and stuff at the moment.”-ClientOverall, clients described a genuine need for the program and felt it provided an important service to their community. Importantly, some clients indicated a need for doula support based on their previous negative pregnancy and birth experiences, while others described needing extra support in addition to their support system. Notably, a few clients expressed concerns about the high rates of negative birth outcomes among women in their community, and many were drawn to the program to have an advocate during the birthing process and because they desired racially-concordant, community-focused support.“The attention and everything was great. The doulas there made me feel really comfortable doing something that I've never done before... I recommended the program to a friend that's currently pregnant, and she was accepted. So, she'll be doing it as well yeah. Very, very happy that it worked out.”-ClientClients were very satisfied with the services provided by SisterWeb. Clients offered few suggestions for improvement and reported they would recommend SisterWeb to others. This positive evaluation was with respect to not only the services directly provided by doulas  but also the entire process of becoming enrolled in the program, such as the intake and doula matching processes. Overall, SisterWeb’s model of services was extremely well-received by clients.“We did not have the time or the designated funding or staffing to create a planning process. So, it was like telling somebody, here's money to build a house, but we don't have time to hire an engineer or architect. So, carpenter, take it away. And you gotta start construction tomorrow.”-LeaderFrom the initial funding, SisterWeb was tasked with creating two programs to provide community doula care to Black and Pacific Islander pregnant people. SisterWeb as an organization did not exist at that point, and in order to launch the programs, they needed an organizational structure. Thus, SisterWeb simultaneously had to formalize as an organization, including obtaining a fiscal sponsor and onboarding staff, while also developing and implementing the specific doula programming. Their initial funding and planned timeline for the launch of the programs did not fully account for this complexity, which meant they were operating with limited time, funding, and administrative support.“I feel like the needs of our program are also a lot different, just because our community don't know a lot about doulas and having birth support… Pacific Islanders have like the highest rates of preterm birth in San Francisco, I feel like what we're doing right now is a lot more having to educate people about why Pacific Islander doulas are needed and what exactly that we do.”-M.A.N.A Pasefika DoulaThese limitations led to organizational challenges including slower internal and external communications and processes and disparate capacity for growth among SisterWeb’s programs, especially the Pacific Islander-serving program, which has unique community engagement and educational needs. These included a limited awareness about doula care and its benefits within the San Francisco Pacific Islander community, a potential preference for group- and community-based care instead of one-on-one support, and a need for materials to be translated into the many languages spoken by Pacific Islander people.“So, [doulas] get paid for the birth but don't get paid for a lot of the other stuff in between… we're asking [doulas] to do a lot. We're not just asking you to go sit and do your prenatal. We're asking you to track information, come to these meetings, come to your monthly business meeting, your monthly professional development.”-Doula CoordinatorSisterWeb initially used a per-birth, contractor payment model, wherein doulas are paid a lump-sum per birth supported. This payment model was due to the quick timeline in which the initial funding was awarded. SisterWeb leaders had a week to submit the proposal; they swiftly developed a payment model, combining popular approaches at the time, including private doula care models and those used by existing community doula organizations. However, leaders realized, from inception, that this payment approach was flawed and led to doulas not being paid an amount commensurate with their effort and hours worked, given that SisterWeb doulas were often providing services beyond what typical, private doulas offer. SisterWeb doulas work in cohorts of two or three doulas and go above and beyond for their clients to ensure they are supported in their pregnancies and births; doulas worked more hours than anticipated while receiving a fixed amount of pay, regardless of time spent with their clients, that they then had to split among cohort members. Furthermore, in addition to direct client services, doulas participated in regular professional development workshops and mentorship sessions.The initial payment model undermined SisterWeb’s program needs and values given their goal of a sustainable employment model and livable wages for their doulas. Notably, nearly all doulas worked multiple jobs to make ends meet and several were caretakers of children and family members. An in-depth examination of SisterWeb’s different compensation models has been published elsewhere . We believe our findings offer important considerations for the implementation of current and future community doula programs, their funders, and the hospitals where community doulas work.organization is different than launching a community doula program. Specifically, building a robust, nonprofit, mission-driven organization from the ground up, particularly when there are few similar organizations as examples, requires adequate time, effort, and funding outside of program development. In order to support grassroots organizations like SisterWeb meet its goals, funders should consider the necessary inputs for building an organization up from scratch, and provide funds and time needed for the critical planning phase. Funders should also consider the need for program evaluation and provide funding for researchers to engage in equitable evaluation. In doing so, researchers can work with developing community doula organizations to generate valuable data to inform in-real-time programmatic improvements and contribute to a shared best-practice guide for future community doula programs.Launching a community doula Considerable attention should be paid to doula payment models to ensure sustainable and livable wages, particularly for community doulas. Community doulas often provide services at low or no cost in order to increase access for those who need these services most, creating substantial barriers to the sustainability of this work. Contractor payment models based on births may perpetuate inequity in employment conditions for community doulas and access to community doula services for birthing people of color. In order to increase the sustainability of community doula work, we suggest organizations employing community doulas look beyond private doula payment models to hourly models of employment with the opportunity for benefits. Furthermore, policymakers should engage community doulas in advocacy efforts for Medicaid coverage of doula care, a potential, albeit logistically complicated, pathway to increasing equitable labor conditions for community doulas.Community doula programs and services must also be tailored to be culturally relevant to the communities served, and, as such, should be rooted in lived-experience, respect for culture, and community voice. Our findings suggest that community doula programs may benefit from time and funds to conduct a thorough needs assessment, led by and in partnership with their target communities, to determine community needs. However, efforts to do this should be inclusive of the communities to be served, including providing specific, culturally relevant training, engaging community members in the planning phase, and employing community leaders. Lastly, community doula organizations will be most successful when they are led by racially- and culturally-concordant community members, who share lived experiences with their clients and possess a deep knowledge of their needs.Our findings suggest strategic partnerships are critical to the success of community doula programs. SisterWeb leaders described their partnerships as a cornerstone of their program, providing them with opportunities to integrate into the hospitals where their clients give birth and receive referrals to the program. As such, stakeholders at hospitals where doulas work should partner with community doula organizations to advance health equity for birthing people of color and with low-incomes in meaningful, concrete, and measurable ways.


Post-analysis of predictive models fosters their application in practice, as domain experts want to understand the logic behind them. In epidemiology, methods explaining sophisticated models facilitate the usage of up-to-date tools, especially in the high-dimensional predictor space. Investigating how model performance varies for subjects with different conditions is one of the important parts of post-analysis. This paper presents a model-independent approach for post-analysis, aiming to reveal those subjects’ conditions that lead to low or high model performance, compared to the average level on the whole sample. Conditions of interest are presented in the form of rules generated by a multi-objective evolutionary algorithm (MOGA). In this study, Lasso logistic regression (LLR) was trained to predict cardiovascular death by 2016 using the data from the 1984–1989 examination within the Kuopio Ischemic Heart Disease Risk Factor Study (KIHD), which contained 2682 subjects and 950 preselected predictors. After 50 independent runs of five-fold cross-validation, the model performance collected for each subject was used to generate rules describing “easy” and “difficult” cases. LLR with 61 selected predictors, on average, achieved 72.53% accuracy on the whole sample. However, during post-analysis, three categories of subjects were discovered: “Easy” cases with an LLR accuracy of 95.84%, “difficult” cases with an LLR accuracy of 48.11%, and the remaining cases with an LLR accuracy of 71.00%. Moreover, the rule analysis showed that medication was one of the main confusing factors that led to lower model performance. The proposed approach provides insightful information about subjects’ conditions that complicate predictive modeling. The increasing volume of data collected and the expanding computational resources dictate current trends in data-driven modeling . It is nTo address the growing complexity of data-driven models and to understand the non-trivial logic behind their decisions, a number of methods for post-analysis have been proposed recently . Some ofIn this study, we addressed a particular question of model post-analysis, i.e., the conditions in the sample that lead to higher or lower model performance. To answer this question in the low-dimensional predictor space, error heatmaps can be used, since they allow the visualization and identification of subpopulations with high and low error rates considering one or two predictors . Then, dTaking into account the limitations of the existing studies, in the presented epidemiological example we revealed “easy” and “difficult” cases for the model when it operated in the high-dimensional predictor space. Particularly, we trained Lasso logistic regression (LLR) to predict cardiovascular death using data from the Kuopio Ischemic Heart Disease Risk Factor Study (KIHD) . After vThe data utilized in the current study were collected in 1984–1989 in the city of Kuopio and the surrounding area in Eastern Finland, whose population was recorded to have one of the highest rates of coronary heart disease ,18. The The baseline examinations 1984–1989) comprised 2682 randomly selected middle-aged men (42–60 years old), whose health state was carefully described with thousands of physiological, clinical, biochemical, psychological, and socioeconomic measurements. As a starting point, 950 predictors were preselected by the domain expert to perform predictive modeling. In our experiments, the outcome variable was “death from a cardiovascular disease by 2016” referring to codes I00–I99 of the 10th International Classification of Diseases (ICD 10) [84–1989 cn subjects described with m predictors each: Let us consider a dataset of In this study, we applied an LLR model, since in our previous experiments, it demonstrated the highest performance on the KIHD data . LLR desdel does . However,x(i)〉)) :(1)min ωIf To estimate the model performance on the test data, we executed 50 independent runs of a  library . In eachUsually, the model performance is described with statistics averaged over the whole sample. These aggregated values are helpful when comparing models, but they do not reflect the distribution of errors in the space of predictors. In other words, the model performance varies for different subjects and we might be more or less confident in the correctness of predictions depending on the subject’s characteristics. Therefore, we propose an approach to automatically generate a set of rules that represent the conditions leading to higher or lower model performance compared to the average level.k-means clustering method, Let Then, to generate rules of interest, i.e., to select predictors Problem 1 (3): To define rules that describe subgroups of subjects with the maximum true positive rate (TPR) and true negative rate (TNR) :Problem 2 (4): To define rules that describe subgroups of subjects with the minimum TPR and TNR :In both problems, the third criterion is used to maximize the number of subjects covered by the rule, which aims to find as general a pattern as possible.To solve the multi-objective Problems (3) and (4), we apply the Non-dominated Sorting Genetic Algorithm III (NSGA III), which is a stochastic optimization algorithm operating with a population of solutions whose quality improves during the search . NSGA IIThe number of bits In this study, we used NSGA III implemented in the Platypus package . Table 1Given the stochastic nature of NSGA III, we ran the algorithm 25 times for each problem, i.e., Problem 1 and 2 , and then combined the final populations from all of the runs. In general, any multi-objective evolutionary algorithm (MOGA) might be applied in the proposed approach ; therefoThere are at least The difference between TPR and TNR does not exceed The average model accuracy for subjects covered by the rule is either lower than Lastly, from all generated rules that excessively describe subgroups of different sizes with various levels of TPR and TNR, we selected the final set of rules meeting the following criteria:From a practical point of view, the final set of rules allowed us to reveal the conditions (subjects’ features) that lead to higher or lower model performance, compared to the average level. First, this knowledge is helpful for revising the sample and collecting new data. Second, using these rules, we can introduce four categories of subjects: “easy” cases that are covered only by the rules with First, we trained the LLR model predicting cardiovascular death in 50 runs of five-fold cross-validation and estimated the model performance on the test data: accuracy = 72.527%, TPR = 72.485%, and TNR = 72.552%. Despite splitting the data into the training and test samples randomly, for some subjects the correctness of the model predictions did not vary across the multiple runs: the model outcome was always (or in most of the runs) either right or wrong. To generate rules that would reveal the conditions of “easy” and “difficult” cases, we used a preprocessed set of predictors: we filtered out predictors that were not selected by LLR in at least one run. Thus, in the further analysis, 191 predictors were involved.Next, after 25 independent runs of NSGA III, we ended up with 4355 rules for Problem 1 (3) and 4583 rules for Problem 2 (4). Then, we selected the final set of rules using criteria 1–3 from To visualize the groups of subjects covered by the final set of rules, we introduced an 82-dimentional rule space, where each dimension defined whether a subject was covered by the rule or not. These binary vectors were used by t-SNE (t-Distributed Stochastic Neighbor Embedding) to project subjects into the two-dimensional space . Figure In general, the results can be analyzed from two perspectives. First, we may pay attention to the rules that describe a particular subject and make conclusions at the subject level about the predictors that make it “easy” or “difficult” for the model. Second, we may carry out analysis at the rule level by reviewing the combinations of predictors and their values that commonly lead to “easy” or “difficult” cases. For example, the two following rules describe “difficult” cases:and (BLPRESMED = yes) and (DIUR = yes) and (ISCHAEMIA = no),(AMIHIST = no) What is interesting is that, in both groups, the subjects took medication. However, the first group had neither myocardial infarction in the past nor ischemia in exercise stress test, i.e., rule (5), and the second group had never smoked, i.e., rule (6). This implies that, in these rules, medication works as a factor that confuses the model for both the “cvd” and “no cvd” groups.Lastly, to generalize the analysis of predictors and to extract the most important ones from the whole final set of rules, for each predictor, we calculated the number of unique “easy”, “difficult”, and “ambiguous” subjects covered by the rules that contain this predictor. Thus, these results support our previous conclusion about the role of medication in model performance: Its presence adversely affects both TPR and TNR .Typically, model performance is evaluated using the whole sample, which gives its average estimate. Yet it remains unclear for which samples the model is prone to making right predictions and for which samples its predictions are likely to be wrong. The high-dimensional predictor space commonly makes the analysis of results even more complicated. Therefore, in this study, we proposed an approach that enabled us to generate a set of rules that explain which samples were “easy” (predictions were more accurate) or “difficult” (predictions were less accurate) for the LLR model, trained and tested on the high-dimensional epidemiological data.Since the average level of accuracy achieved by LLR was only 72.5% when predicting cardiovascular death for subjects from the KIHD cohort, the additional post-analysis was aimed at revealing those subjects’ features that lead to high or low model performance. First, the knowledge about “difficult” subjects might be helpful for revising the sample, as they are the first candidates for double-checking. Moreover, collecting new samples, if it takes place, with characteristics corresponding to “difficult” cases might increase the model performance by bringing more information to the poorly modeled areas of the predictor space. Then, applying the set of rules to unseen data provides more confidence in the model predictions if new samples are categorized as “easy” cases. Knowing the weak spots of the model is of the utmost importance for clinical applications, where additional tests should be performed for “difficult” cases to avoid wrong predictions.Moreover, as a well-interpretable tool, rules explicitly report the logic behind decision making while analyzing each subject, which is especially valuable for medicine. At the same time, when extracting the most important predictors from the final set of rules, we generalized the results at the whole sample level and obtained common patterns. Thus, in the KIHD sample, taking no medication , absence of other diseases , and a high standard of living defined by socioeconomic predictors  are common traits of “easy” cases for the LLR model when predicting cardiovascular death. Conversely, “difficult” cases included subjects who had taken medication  .Although in this study, the approach proposed was applied for LLR post-analysis, it is model-independent and might be useful for better understanding of any model behavior, which is especially helpful when “black box” models are trained on high-dimensional data. Such post-analysis is also useful for “white box” models, because variables that are important for making predictions differ from variables that are informative for describing “easy” and “difficult” cases.The main limitation of this study is that all of our findings are applicable to the KIHD cohort, but they cannot be extrapolated to other populations. To make more general conclusions, e.g., about the role of medication in predicting cardiovascular death, we need to perform the same analysis for other populations, and then check if there are rules and variables similar for different populations. Additionally, since this post-analysis was applied to a particular data-driven model with a certain prediction horizon, for other models or different prediction horizons the analysis should be re-run.Despite these limitations, the approach proposed is useful for many epidemiological studies, as the model accuracy has been reported to be far from 100% ,34,35,36