Just over 10 years ago, we conducted a culture study of the Computer Science Department at the flagship University of Illinois at Urbana-Champaign, one of the top five computing departments in the country. The study found that while the department placed an emphasis on research, it did so in a way that, in conjunction with a lack of communication and transparency, devalued teaching and mentoring, and negatively impacted the professional development, education, and sense of belonging of the students. As one part of a multi-phase case study spanning over a decade, this manuscript presents preliminary findings from our latest work at the university. We detail early comparisons between data gathered at the Department of Computer Science at the University of Illinois at Urbana-Champaign in 2005 and our most recent pilot case study, a follow-up research project completed in 2016. Though we have not yet completed the full data collection, we find it worthwhile to reflect on the pilot case study data we have collected thus far. Our data reveals improvements in the perceptions of undergraduate teaching quality and undergraduate peer mentoring networks. However, we also found evidence of continuing feelings of isolation, incidents of bias, policy opacity, and uneven policy implementation that are areas of concern, particularly with respect to historically underrepresented groups. We discuss these preliminary follow-up findings, offer research and methodological reflections, and share next steps for applied research that aims to create positive cultural change in computing. Over the past 50 years, researchers, policy makers, educators, and employers have invested much toward the recruitment and retention of women and people from historically underrepresented minority backgrounds in computing. While some scientific fields, such as the life sciences, have seen great improvement in degree participation by women, computer science and engineering have experienced declining occupational and degree participation by women and people of color, with enrollment and employment trends by these groups in 2013 nearing where they were in the 1960s , the specific changes informed by our study included:Your work is very interesting, but this kind of thing doesn't happen in my research group.” We had interviewed two of his students, one of whom said the exact quote to which he objected. We knew that “kind of thing” happened in his group, and that his own students were lamenting a lack of research mentors. We report only one of our own experiences here, but this pattern of resistance despite empirical evidence is seen in the research on bias. Both men and women have relative reluctance, particularly those who are faculty in STEM departments, to accept evidence of gender biases in their field  were sampled, using an e-mail solicitation sent by Heeren to 18 members of the department. Participants were given a short background of the study, including a message that participation in the study was entirely voluntary. Individuals agreed to participate in the study by scheduling an appointment with the three interviewers: Chambers, Crenshaw and Metcalf. We spent 1 h to 90 min with each participant; we administered a paper version of our pilot survey instrument and conducted semi-structured follow-up interviews based on the participant's survey answers.This paper reports on the pilot study results, focusing on preliminary qualitative findings from the undergraduate and graduate student populations. While we have developed our survey and conducted pilot interviews with two faculty, here we focus on our qualitative student data for the purposes of preliminary comparison between the 2005 study and the current departmental patterns. We will continue to pilot the faculty instruments and will report on these findings in future work.For the undergraduates, the survey instrument covered topics related to their pathway to the department, their courses, the quality of teaching they experienced, their academic advising needs and experiences, their mentoring needs and experiences, sources of support, policy awareness, their sense of fitting in and feeling successful in the department, extracurricular activities, departmental values, future career goals, and demographic information. Survey questions included open-ended response options for demographic questions and at the end of each topic area so that participants could share more freely about their experiences with that topic in the department. For the graduate students, the survey instrument covered the same topics in addition to graduate program requirements, choosing an area of research, and, for doctoral students, the Ph.D. qualification exam.For the analysis of our qualitative data, we conducted inductive thematic content coding . You know? And its more of…sometimes I feel like if we're—especially for the 500 level classes. It's not about the teaching, it's more about the showing.”Within the department, these kinds of teaching faculty are responsible for the CS1 and CS2 introductory programming sequence taken by all undergraduates. Almost all students interviewed in 2016 felt that the department valued teaching, although many noted that it valued teaching only at lower levels, where non-tenure track faculty emphasized teaching as the main part of their job. An underrepresented graduate student of color who works as a teaching assistant for one these courses describes them in this way, “I think the Computer Science department has a split brain. And one half of that brain really values excellent teaching. And the other half of that brain couldn't care less.” An international graduate student was more precise. When describing the teaching quality in the upper division courses, she said, “It depends on the faculty. Because I've taken another 500 level course and that professor was excellent.”Another graduate student, a white man, echoed this sentiment, “Thus, it seems that the department's investment in undergraduate teaching, particularly at the lower-levels, has helped to shift the perception of undergraduate teaching quality and investment. These investments were not matched at the graduate level, where our 2016 participants continue to express high levels of dissatisfaction in the quality and level of investment in teaching. In addition, this investment does not pay adequate attention to equity issues in terms of who is taking on these roles and how the roles are valued financially, as many of the teaching staff positions at the university seem to offer less job security and lower pay; retention of high-quality teaching faculty may be a challenge on campus.In the original study, the quality of teaching was a much-discussed topic among participants. Only 65% of participants felt that the department valued excellent teaching. Among the undergraduates, more than half of the recommendations on how to improve the department were related to teaching. At the time, one participant said,I would like to see T.A.s with more instruction on how to teach a class. The first course a student takes in Computer Science is the most critical, because it is that course that will make a student decide whether to stay in the department.”“Teaching Techniques for Computer Science which discusses techniques for effective teaching specific to computer science.Teaching assistants are assigned to a variety of courses with many different demands; some positions require administrative work, grading or assisting students during office hours. Other courses require leading a laboratory or discussion section. In 2006, teaching assistants for the department participated in a campus-wide 2-day training course. International teaching assistants received an additional 2 days of training. At the time, we pointed out that the University of California at Berkeley offered its own course, CS301, called Teaching Assistant Training,” to prepare its teaching assistants for their service. Student interviews enumerated some of the course's topics, including lesson planning, encouraging student participation, protecting student privacy, cultural sensitivity and Title IX compliance.Today, all Ph.D. students in the department are required to fulfill a teaching assistant role for at least one semester. In response to our recommendation, to better prepare their teaching assistants, the department offers a seminar course, CS591 TA, or “common sense,” topics he did not care about, and some topics of interest unrelated to teaching. He ended his description of his experience with, “Maybe I'm not the perfect candidate for that class. Or maybe I just didn't receive the instruction well.”Among the three graduate students interviewed who took the course, two perceived it as useful. One student, a white man, was not as engaged. He felt it was a mix of “I actually skipped it and I was allowed to skip it because it's offered only in fall and I wasn't a TA in Fall '14 when I joined, I was an audit in Fall '14. But then I chose to be a TA in spring and then that…seminar wasn't offered in spring.”While there is unevenness in terms of how the students are engaging in the course, there are also discrepancies in how the seminar requirement is enforced. Of the five graduate students interviewed, two, both men, reported they did not participate in the seminar before accepting a teaching role in the department. One international student offered, “but it clashed with one of my courses. And then I couldn't …I mean I thought I shouldn't be dropping the course for TA training.”The student went on to explain that his course evaluations offered positive student feedback and so he was not made to take the seminar in Fall 2015. He further offered that he might have taken it in Fall 2015, “tenure-track faculty member” was frustrated by the lack of teaching preparation and opportunity he had found in the department. He offered, “Because I don't think you can just grade your advisor's exams and then say “I'm a teacher now.” An underrepresented student of color and second-year Ph.D. student described “heartbreaking” stories she heard from students during her office hours. She wanted to help, “but I have no idea where to refer them for example, for mental health or counseling. I don't have that information except like if I actually go and search on the internet.” She felt very unprepared to handle emergency or crisis situations presented by her students and described a lack of awareness of the departmental or campus resources to which she could direct her students in these moments.Those students who had taken the course also identified gaps in their preparation for teaching roles. One white third-year Ph.D. student described his career goal as “‘good job, you figured this one out, don't tell these guys because they need to work it out for themselves too.”’ The student went on to say, “And this kid in my group got so mad that I figured out this problem, he literally stood up, threw his chair back and was like, “you're just a freaking girl, you must have cheated on this thing, there's no way…”Undergraduate experiences echoed such gaps in teaching assistant training for crisis situations. A senior undergraduate student, a white woman, described one such experience. Led by a teaching assistant, the students were instructed to work in groups on an in-class assignment. She said she got behind on the problem, and started working on it independent of her group. As the teaching assistant circled the room, she saw the student's work and said, “I just left the class crying.”Nobody did a thing. Met with a roomful of silent bystanders to the violence, the yelling, the insults, the chair-throwing, she said, “Many of the students in our pilot interviews described situations in their courses, online discussion groups, student organizations, and research groups where they witnessed or directly experienced hostile behavior, harassment, and bias and felt unsure about how to handle the situations as teachers, students, or peers.How do you handle that when it happens?” She offered that sometimes it can be useful to call out the behavior, and that she had once seen a person do so effectively. She said, “but that person wasn't from computer science. That was funny to see…they were from physics.”For example when asked about whether she had witnessed inappropriate behavior on campus, an international graduate student spoke generically about seeing such behavior from both students and faculty in the context of coursework and social interactions. We asked her, “More often than not, participants explained that these situations are met with awkward silence. Several of our participants described how the culmination of such experiences, particularly when they have been the target of these words and behaviors, has had a deflating effect. For example, one white undergraduate student explained that she has had to learn to tolerate these behaviors:I don't feel a sense of…of hope that like, oh you know, just push through and it's going to get better, once you get into industry — no. It's just how it is and you have to get used to it or get out of it. It's not really something that's going to be fixed within the next few years.”“She went on to explain that these experiences profoundly changed her formerly exuberant approach to outreach:Obviously if young women have aspirations to be in this field I want them to succeed and I want to help them achieve their goals. But for the young women who are not sure, I'm not going to push them to do something that they're not sure about…you have to be 150 percent sure that this is what you want or you're not going to succeed and then you've just wasted time…I came into this department putting in a lot of effort into…getting women excited about this and we're going to make a change by just showing them how exciting this all is. And to some degree I think that can work. But, on the other hand…[being here is] not a healthy thing.”“Our 2006 study uncovered a student appetite for multiple kinds of mentoring relationships, but a lack of opportunity to find mentors. At the time, 18% of undergraduate and 53% of graduate students reported having a mentor, with many participants wishing for some kind of mentor but feeling unsure as to how to find one.There are multiple ways that a department may offer mentoring relationships to its students. One is through undergraduate advising. However, in 2006, only 2 undergraduate participants, or 3%, said that their advisor also served as a mentor to them. At the graduate level, thesis advisors may also serve a mentoring role, but of the 51 graduate participants who reported having an advisor in 2006, roughly half reported that their advisor was their mentor.Faculty Mentor,” to support undergraduate advising. Yet, in a department as large as Computer Science at Illinois, we have great empathy for how spread-thin faculty can feel. The undergraduate population has doubled in 10 years, with few new positions added due to state budget constraints.In a university context, tenure-track faculty are often counted upon to serve in a mentoring capacity. To that end, the department has instituted a new role, “With such challenges, it becomes even more important to provide multiple and diverse mentoring opportunities. The student population at any department can serve as a rich network of peer mentoring; senior undergraduates can mentor first-year students and Ph.D. students can mentor those in the Master's program. Students of all levels can learn from each other. Our preliminary results provide evidence of how students do mentor each other as well as the challenges that students feel in reaching out.We discuss our preliminary results regarding Faculty Mentors and peer mentoring in turn.Since the original 2006 study, the department has instituted a new role, Faculty Mentor. The departmental website on undergraduate advising offers that,The Computer Science Department has a three-tiered approach to advising: the Office of Undergraduate Programs, faculty mentors, and peer advisors.All students are assigned a faculty mentor, with whom they must meet at least once each academic year, typically before April. The department enforces this requirement with a registration holdWhy do you meet with your Faculty Mentor?” Focused on the required nature of the relationship, all four undergraduates wrote similar responses. One wrote, “Because it's required.” Another wrote, “I seldom do; I only do when required. [The faculty mentor is] more focused on checkboxes than individualized advice + connections.”In the survey for undergraduate students, we asked, “I actually just visited my faculty mentor a few days ago. And it was actually a little more helpful than I thought.” He described that the Faculty Mentor was meeting with multiple students in a group setting. He said, “I was like ‘yea, which courses should I take?’ and then in addition to that, he explained what they were about. And…other students were there and they also gave me some feedback…And a lot of them were seniors and upper classmen so, they were pretty good.”In undergraduate interviews, there were multiple kinds of Faculty Mentor experiences, from helpful to contractual to discouraging. A junior undergraduate who identified as Asian American said, “really helpful.” She recalled a time that she expressed anxiety about her career opportunities and that “My mentor talked about…that I don't need to feel so pressured to do everything technical. She says 'Just do what you want. During the summer, you don't have to do an internship.' She like calmed me down a bit and assured me.”A junior international undergraduate student said her Faculty Mentor was “[Their] Faculty Mentors don't email them until the very end and then like now time's running out. So they just do like a quick 10 min thing and it”s something to get over with. I know a lot of people where it's like that.” Another undergraduate, who identified as white, echoed the “checkbox nature” of his Faculty Mentor relationship. “It was a little bit of advice and a lot of more or less review, to kind of say “you”re doing fine.”She lamented that her friends did not seem to have the same kind of helpfulness in their Faculty Mentors. “not cut out for this department.” In response to an average grade in a course, the Faculty Mentor encouraged her to stop any extra curricular activities. The student said the the meeting was, “Completely unhelpful.” She reflected that if she hadn't had her other chosen mentors, if she had been a less resilient student, she may have dropped out of the program because of the interaction.A senior undergraduate student, who identified as white, said that her Faculty Mentor told her that she was “I never knew if I could or not.”When asked if she looked into getting a different mentor assigned, she reported that she did not. Furthermore, “CSGSO: The Computer Science Graduate Student Organization. Its main offering is a Friday social hour for graduate students.Reflections Projections. The October 2016 conference had 1,800 registrants.ACM: Association for Computing Machinery student chapter. It is open to both undergraduate and graduate students, though its membership is largely undergraduates. A large physical space in the computer science building is reserved for this student club and is often open as a student lounge and homework space. In addition, the ACM students organize an annual tech conference, WCS: Women in Computer Science. This organization works for recruitment and retention of women at all levels of computer science. WCS hosts meetings and social events for both undergraduate and graduate students. Their largest offering is the ChicTech program designed to inspire girls to consider careers in computing. The November 2016 ChicTech retreat saw 75 high schoolers staying overnight on campus to get a feel for the different opportunities in computer science.In 2006, there were a handful of departmental organizations providing academic, research, social support and outreach opportunities for students. The organizations that still exist today are:While not created for the purpose of peer mentoring, we find that these departmental organizations provide students with opportunities to meet other students in the department; such interactions allow students to create social networks that can be leveraged for both psychosocial support and professional development.clicked.”In 2006, though graduate students seemed largely unhappy with the activities organized by CSGSO, undergraduates enjoyed the organizations in which they participated. Ten years later, the graduate students still wished for more interactions. Undergraduates have a greater number of organizations in which they participate and flourish, but some had not yet found a group with which they “Hack Illinois, Women in Computer Science, ACM.” She had even started a new student club specializing in a certain kind of software development. She attributed her club with being, “a major part, if not the only reason I stayed in this field.” With all her extra curricular activities, she found that, “I have friends who are kind of coming to me with career questions and want like, recommendations on those kinds of things.”In the 2016 interviews, a white senior undergraduate listed the student clubs in which she was involved, “People are really enthusiastic and really into the projects and I love that kind of a culture because it drives me.”A multiracial senior undergraduate who had transferred into the department from another on campus really liked how many side projects the ACM students had. He said, “I mean, I have really limited number of interactions with graduate students which is something that should be happening. So CSGSO tries to keep its happy hour…I try my best to be there, but there's only like five or six people there. Generally it's like the pizza comes at 4:45, people come in at 4:40, maybe 4:42 or something. They grab the pizza and they just leave.”Of the five graduate students interviewed in 2016, only one mentioned CSGSO. An international Master's student said, “As far as the department is considered, I think there's more social events for Ph.D.'s students than Master's students. Because my roommate is a Ph.D. student…and he has a lot of social events at the department which are hosted by the department.” He imagined a gathering in which he had the opportunity to meet other kinds of graduate students. “Let's say I'm a second year Master's and I know someone who's doing a Ph.D., who's in their fourth year, I can look up to them as a mentor or something.”This same student lamented his lack of opportunities to interact with other students given that he was in the Master's program. He offered, “a limited number of interactions.” An international graduate student who had spent some time in the Math Department before coming the Computer Science compared how the two department provided “shared spaces to graduate students.” As a smaller department, she said of math, “The first year, the department puts all the TA's in the same place.”Other graduate students echoed this sentiment of “tends to have everybody kind of off doing their own thing. And while that's fine, it means that if you look at our groups papers there's not a lot of cross pollination.” Similarly, a Ph.D. student and underrepresented woman of color said that her research group was “a friendly one” but that, “even within in my group, people keep their work separate from each other. We have like separated work that is the same project, but not really collaborating with the same – on the same topic. It's hard to get that collaboration going.”This sentiment was captured even within research groups. A white third-year Ph.D. student said that his group, “The TA's for CS 225for now. They range from like, civil engineering, industrial engineering, also computer science TA's. But it's more like a group of friends, you know?” An international Master's student said, “That was one reason why I chose to be a TA. Because, I mean, I figured out that I didn't know a lot of many people in the department after a semester. And then when you're a TA you know a lot of students.”The graduate students who did enjoy some kind camaraderie within the department did so in a teaching assistant role. One underrepresented woman of color said that her peer mentor network was, “I had a TA for one of my courses…and she was in the same area as me. She knew my advisor and she helped me connect with him.”When he was seeking a thesis advisor, the same Master's student found mentorship through a teaching assistant, “In the written survey, 3 out of 4 undergraduates answered “No” when asked, “Do you have a mentor?” Yet, in addition to the departmental organizations, we found many students seeking creative sources of career and psychosocial support than in 2006. They found support in their peers, family members, friends, and teaching staff.I've had moments where I feel like some male students will ask to work with me. Not because they think I'll be academically like equally successful, or like I'll be very helpful. But just because they want to like…they want to like say that “oh, yea, I worked with her” and like, because I'm a female student.” She went on to say, “It feels really weird…do you see me like a peer?”One Asian undergraduate felt a sense of unease when it came to doing homework with undergraduate men. She said, “I just feel like I don't really fit in with the people there…I just feel like I don't really connect with them.” Instead, she had cultivated her own small homework and support network, which included her teaching assistants, engineering learning assistant as well as peers, “a group of people who will be supportive in that way and not try to boast and like look down on me, you know for not knowing something.”We asked whether she had looked into any of the departmental organizations. She said, “I do not need a mentor.” He elaborated in the interview, “Not so much that I don't need a mentor, that I think it's more of a student collaboration mentorship. I find a student who has gone through a similar thing and I had talked to them about it. So I have many mentors, kind of one on one, but they're more friends giving friends advice.”In the written survey, a multiracial senior checked the statement, “I will always default to another student before I will default to an advisor…” though he included much of the professional teaching staff and academic office staff in his own mentoring network.We asked if he ever followed up with his Faculty Mentor on any of the topics he wanted to discuss. He replied, “fantastic” sources of mentorship with her discouraging Faculty Mentor interactions.Other sources of mentors were diverse and were often leveraged in times of personal crisis. An white graduate student talked about a family friend, a professional psychologist, that he was able to call in times of struggle. A white undergraduate had a parent and a family friend who both worked on campus; she mentioned them, contrasting these “We are glad that students have support networks, but mentoring should not just be sought as a coping mechanism to deal with the struggles had in the department. Mentoring should also serve as a personal and professional development network that helps all members of the community be engaged in ways that catalyze their successes. Our student participants have expressed wide-ranging negative and harmful mentoring experiences with their tenure-track Faculty Mentors to the degree that they often do not even consider these faculty as part of their mentoring networks and often feel as if their advisors and Faculty Mentors see them as burdens and administrative checkboxes.he really knows what he's doing. And it's really important to have a mentor that they know their field well. And I couldn't really find a …a good mentor that's like they had a passion about like the next 10 years of their field.One international graduate student pointed out her thesis advisor's mentoring qualities, Like I, mean I wouldn't bother my advisor whether I should put this on my CV or not.”Aside from graduate students with thesis advisors, we didn't find substantive evidence of research mentorship. One student, a white man, even hesitated talking to his own advisor about career matters. He said, “In qualitative research, the stories of both the researcher and the research participants are reflected in the themes that emerge Kovach, . As suchTaking inventory of our team, we represent 42 years of experience at the university. From Illinois, we have collectively earned one bachelor's degree, two Master's degrees, and three doctoral degrees. The joys, sleights, and traumas that we experienced on the campus influenced our subsequent professional trajectories. We became senior lecturers, tenured professors, software engineers, and research directors. All the while, we invested energy into the kinds of service that would make the field a better one for underrepresented groups. And, by July 2017, we will all have left the campus.I helped found that group…when I was a sophomore. I was one of the first presidents.” The participant offered up a high-five to celebrate as well as an invitation to rejoin the club's current officers for dinner that evening.With these experiences, during the course of our pilot study, we interviewed participants to gather data, but we also saw an opportunity for connection, collegiality and mentorship. As participants shared their own stories, we found moments of celebration in positive shared experiences. For example, when reviewing the variety of student clubs in the department, one student mentioned the Women in Computer Science as a valuable resource. Our own Chambers said, “Are you wearing an Apple sweater because you're going to go work there?” The participant replied, “I am.” The participant had already signed an offer letter with the company. The audio erupts into laughter and lots of “Congratulations!.”In another moment of celebration, Crenshaw noticed a participant was wearing a particularly unique top. She asked, “I think there's a lot of students who have been programming for years and consider themselves programmers, but find difficulty doing the assignments and then they think, “well I'm a programmer, one, I could do other important things, and two I don't need this.”Not all was celebration. One participant was particularly frustrated with the amount of cheating he was seeing. We commiserated over similar instances and experiences while we were in the department; this allowed us to gather further comments on this culture from the student, including his thoughts on how it began in the department, “played a lot of sports.” Metcalf told him her own story, “When I was here, a friend of mine created a Tuesday night volleyball group…And it wasn't just students within this department, but there were engineering students who would join us too. So, connected departments. And then I started a softball group on Sundays. We would all get together and play softball together.” In a way, sharing her story was a small moment of mentorship for the participant, helping him to see how he might create more opportunities for interaction himself.Another participant who was hungry for more interactions on campus shyly described how he stood out from his peers because he “Yes, we know. We saw it too. That happened to us, too.” This ability becomes particularly important with sensitive topics like gender discrimination. One participant described a situation where she didn't do anything in response to bad behavior from a student. She said that, “every time it happens it's so shocking, that I don't feel like “did that even just happen?”Having our own stories to offer in exchange also helped participants feel less alone. We could say with kindness and integrity, “Because you are shocked at the inappropriateness and you are left speechless about it, right? It's normal for you to feel taken aback and to not really have words in that moment to express a thing.” She also offered, “And that is why it's so important to have people who aren't just silent bystanders but who can be allies in those situations. And who are trained to see them and to call people out on those kinds of behaviors. So it's not just the person who's the target of the remark…”Metcalf weighed in, “I hesitate to say this, but I don't encourage women to enter the field right now. It's not a healthy environment. I…I'm not sure if I'm happy that I did it, myself. There have been multiple moments for all of us when we had to take long breaks from recruiting activities. It is difficult to invite people into a field that can, at times, be so narrow and feel so mean.Some quotes from participants felt eerie, as if words had been taken from our own mouths. At one point, an undergraduate summarized her experiences at Illinois this way, Feeling the Feels.” Chambers wrote a long journal entry during this time, reflecting on how people call her “one of the lucky ones” but that it doesn't particularly feel that way. She expresses anger at the times she's been labeled as “underperforming” without concrete reason and enumerates the heartbreaking stories of sexual assault she's heard from colleagues, and her own moments where she has been “left speechless” by bad behavior. She also reflects on the ways she tries to make her corner of the field a little better. Such reflections resonate with many of the comments shared in our interviewers, and strengthen our feelings of connection to this group.It was in these eerie similarities that we realized the importance of our own self-care in this inquiry. Immediately after the pilot interviews in April 2016, we took a month-long break from our weekly conference calls. We described this period as “This pilot study is the initial step in our follow-up research to understand whether the departmental culture has changed since our original research over 10 years ago, particularly relative to the experiences of women and other underrepresented groups. Our data and discussion above represents only preliminary results from the latest phase in our case study of the University of Illinois' Department of Computer Science. The information gathered in this pilot study has allowed us to update and strengthen our survey and interview instruments.Our fully-revised data-collection instruments are planned for the Spring 2017 semester where we will recruit participants from the larger departmental population, including faculty, for surveys and follow-up interviews on a variety of topics related to the departmental culture, including: workload; sense of belonging; departmental values; policy awareness, understanding, and effectiveness; sources of mentoring, information, and support; collegiality and respect; classroom experiences; social experiences; research experiences; advising experiences; career goals and development; performance evaluation; communication; perceptions about equity; scientific identity, and more. We will gather more data and conduct additional statistical and thematic analyses that will inform further improvements within the department and contribute to the larger body of research on computing culture. In addition, this will allow us to more fully analyze how the experiences of faculty and students at University of Illinois compare and relate to the broader culture of computing. We will be able to explore a set of recommendations to improve retention and outcomes at Illinois and beyond.As in other foundational, in-depth case study research (e.g., Margolis and Fisher, In the meantime, our preliminary findings indicate both improvements and areas of concern. The concrete improvements in undergraduate teaching and the continued success of peer mentoring in the face of increasing enrollment is laudable; in addition, the increased population of women and underrepresented students of color gives hope for the trajectory and future of the department. However, if the department hopes to retain and see success for its growing population of women and underrepresented students of color, several systemic, policy, and cultural issues will need to be addressed.Descriptions of bias, including acts of violence and differential policy application, exacerbated by policy and resource opacity are particularly concerning. The existence of any one of these areas is likely to contribute to attrition. The incidents of bias against women, students of color, and international students in the classroom, research groups, and online departmental forums left unaddressed by peers, instructors, and faculty sadly align with the expansive body of research on bias, harassment, and discrimination in STEM fields (Hill et al., The ongoing disconnect between faculty and students, sense of isolation, reliance on peer mentoring networks to cope with negative experiences in the department, perceived inaccessibility of tenure-track faculty, and marginal and negative mentoring experiences also raise retention concerns. Effective mentoring relationships, especially when structured as mosaics, groups, networks, and multiples, positively influence retention, organizational commitment, educational and career development and progression, skill development, knowledge acquisition, and more. Marginal and negative mentoring relationships, however, contribute to stifled careers, attrition, isolation, negative educational and career experiences, and even harmful long-lasting outcomes Ragins, . The creOver the course of a decade, the department has made some substantial progress toward creating a more inclusive culture. While our initial findings are not entirely positive and have been, at times, painful to reflect upon on a personal level, we remain grateful to a departmental leadership willing to do the difficult work of reflection with us—to learn from its struggles and its successes. While some of what we have found thus far is mirrored in the research elsewhere, this unique 10-year look from a cultural vantage point, rather than an individualistic one, provides the possibility of novel interventions in the systemic root causes of inequity and exclusion. As we continue our data collection, we are hopeful that lessons learned will continue to shape positive change in the department for all of its members. In addition, we hope that this multi-phase case study and our future work to expand it to other computing environments will deepen our collective empirical understanding of how systemic, rather than superficial, change unfolds over time and across departmental and institutional contexts.This study was carried out in accordance with the recommendations of the University of Illinois at Urbana-Champaign Institutional Review Board, IRB No. 16507, with written informed consent from all subjects. All subjects gave written informed consent in accordance with the Declaration of Helsinki. The protocol was approved by the IRB.This work is a product of all four co-authors. All co-authors have contributed in various degrees to the study's survey instruments, research concept, and analytical methods used.The Department of Computer Science at the University of Illinois at Urbana-Champaign supplied partial funding for our study, supplying travel monies, interview space, a $10 remuneration for each participant, transcription services, and an honorarium for HM.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Today, a considerable proportion of the public political discourse on nationwide elections proceeds in Online Social Networks. Through analyzing this content, we can discover the major themes that prevailed during the discussion, investigate the temporal variation of positive and negative sentiment and examine the semantic proximity of these themes. According to existing studies, the results of similar tasks are heavily dependent on the quality and completeness of dictionaries for linguistic preprocessing, entity discovery and sentiment analysis. Additionally, noise reduction is achieved with methods for sarcasm detection and correction. Here we report on the application of these methods on the complete corpus of tweets regarding two local electoral events of worldwide impact: the Greek referendum of 2015 and the subsequent legislative elections. To this end, we compiled novel dictionaries for sentiment and entity detection for the Greek language tailored to these events. We subsequently performed volume analysis, sentiment analysis, sarcasm correction and topic modeling. Results showed that there was a strong anti-austerity sentiment accompanied with a critical view on European and Greek political actions. It is common ground that Online Social Networks (OSNs) have prevailed as the major platform of public expression regarding political matters. Existing studies have performed elaborate analyses in order to investigate the behavior of online users during pre-election periods. The purpose of most of these studies was to generate patterns that distinguish users’ or posts’ favoritism towards one political party or certain ideology. The main predicament in these studies was to generate election predictions that are close or even outperform public opinion polls , to measPerhaps the most seminal review in this area is from Gayo-Avello . This woRegarding noise, a huge proportion of election-related Twitter posts are humorous, ironic or sarcastic and do not portray any party (or ideology) inclination. It is estimated that approximately half of collected tweets belong to this category , 16. FilRegarding demographics, Twitter users belong to a specific social group that is not necessarily representative of the whole electorate. Specifically, studies have indicated that Twitter users belong to a certain age , social This work aims to apply natural language analysis techniques on Twitter data related to two electoral events that happened on 2015, during a politically turbulent period of Greece that was triggered by an effort to negotiate a reconstruction of its national debt. These events were the Greek bailout referendum that took place at 5 July 2015 and the second was the subsequent legislative elections that took place at 20 September of the same year. The purpose of this study is to identify the tweeting patterns, the expressed sentiment and the semantic relations of the most important entities that prevailed during the online discourse that preceded these two events.To accomplish this we have split our analysis in 5 distinct parts. The first is data collection and Entity Identification (EI). EI is the process of extracting the most important notions (entities) that are prevalent in users’ posts. Examples of extracted entities are ‘Prime Minister’ and ‘Debt’. After EI, each entity is represented by a set of words with equal meaning (i.e ‘EU’ and ‘European Union’). Although there is a variety of methods for automatic EI, they are all inferior to various extend to human curators . For thiApplying these methods to Greek tweets entails some additional difficulties. First, the online language, primarily used by the youth, is a mix of Greek grammar with Latin letters. This system, called “greeklish”, stems from early text-based communication systems that had limited support for Greek letters. Moreover, this system has not any standard correspondence between Greek and Latin letters, whereas it often disregards Greek grammar and punctuation. The result is a highly complex language with multiple possible writings even for basic and short words that makes automatic detection a very tedious task , 33. AltOn 25th of January 2015, the new “anti-austerity” government of the SYRIZA party was elected in Greece with a percentage of 36.3%, starting a long negotiation with the Eurogroup about debt reconstruction. Until June 2015, there was no visible progress achieved and SYRIZA decided to throw a referendum on 5th of July 2015 so that the Greek people decide whether to accept or not the current austerity measures proposed by the Eurogroup. Capital controls were enforced in Greece and the result of the referendum was NO (do not accept) with a percentage of 61.3%. Eurogroup did not accept the result of the referendum as a bargaining tool and, under extreme pressure, the government decided to accept the proposed measures. Several disagreeing members of the SYRIZA party threatened to vote down the measures. The prime minister (Alexis Tsipras) decided to expel the disagreeing Members of the Parliament that belonged to the governing party, and announced new legislative elections on 20th of September 2016. SYRIZA won the elections again with a reduced percentage of 35.5% and the party formed mainly from disagreeing members  did not get enough votes to enter the parliament.Our analysis is based on two distinct Twitter datasets. The first one includes all tweets that contain the #dimopsifisma and #greferendum hashtags. These were by far the most prominent hashtags that prevailed throughout the period that preceded the Greek bailout referendum (“dimopsifisma” is the Greek word for referendum). This period was from 25th June 2015 when the referendum was announced, until 5th July 2015 when the referendum took place. Data were collected through Twitter’s API. This dataset contains in total 301,000 tweets out of which 84,481 are neither retweets nor replies. The frequency of referendum tweets is illustrated in To support our analysis and reveal relationships between persons, institutions, events and abstract notions , we performed entity identification  on the eFor sentiment analysis, we used SentiStrength  that is Sentiment analysis is known to be domain-dependent, meaning that applying a classifier to a dataset different from the one on which it was trained often gives poor results . Indeed,Therefore, we enriched SentiStrength for the Greek political domain by creating new general-purpose and political-domain lexicons through manually selecting and annotating words from the Twitter corpora. Human intervention seems likely to be particularly important for narrowly-focused topics for which small misclassifications may result in significant discrepancies if they are for terms that are frequently used with regard to a key aspect of the topic. For the purpose of political domain analysis, we manually created a new SentiStrength-compatible lexicon comprising Greek words with associated positive/negative sentiment strength, aiming to improve the accuracy and effectiveness of political-domain lexical sentiment strength detection. The SentiStrength algorithm for sentiment strength detection across the social web primarily uses direct indications of sentiment. Since our study is domain-dependent and time-dependent , we included indirect affective words too, in order to enhance sentiment detection. These words identify terms that associate with sentiment but do not directly express it, and were derived from the Twitter corpus in question.The new sentiment detection lexicon we compiled, is a merge of the following 3 lexicons in Greek: (i) SentiStrenth’s built-in lexicon that provides general sentiment analysis for the Social Web; (ii) SocialSensor lexicon that is utilized by SocialSensor framework to collect, process, and aggregate big streams of social media data and multimedia to discover trends, events, influencers, and interesting media content in real time ; and  of tweets referring to political issues are of sarcastic or humorous nature and can severely obscure the analysis, particularly in the political domain , 41. To The first step was to construct a sarcasm classification mechanism , 43. IniHaving at our disposal a human-flagged dataset regarding sarcasm, we proceeded to build a classification model. From the total 4,644 flagged tweets, we extracted lexical and semantic features. In order to build the lexical features, we developed a stemmer for the Greek language and we built a stopword collection containing commonly used Greek words. Then, we extracted 1-grams and 2-grams for each tweet. A common set of lexical features is Part-Of-Speech (POS), which we were not able to incorporate since we could not locate an adequate POS dictionary for the Greek language. The semantic features included average sentiments for each word in the tweet and topics. For sentiment, we used the same SentiStrength-compatible dictionary that we constructed for the purposes of this study. We also performed a topic analysis that generated 100 topics related to the context of the collected tweets. The hypothesis here is that some topics are more associated with sarcastic tweets and therefore, by incorporating them as features, we can improve the classification efficiency of our model. The topic analysis method that we used was Latent Dirichlet Allocation (LDA) implemented with the Gensim Python library. For classification, we used a Support Vector Machine (SVM) classifier with a linear kernel and an Euclidean regularization coefficient of 0.1. Subsequently, we randomly divided the flagged dataset into 70% training dataset and 30% test dataset. We trained our model to the training dataset and estimated its performance on the test dataset. The classification results are on For comparison, Charalampakis  performex)) as a sigmoid function to convert this distance to percentages. Finally mapped each “sarcasm value” to one of the following categories: “no_sarcasm” for negative values, “sarcasm_1” for values from 0% to 20% of positive sarcasm values, “sarcasm_2” for values from 20% to 40% of positive sarcasm values, and “sarcasm_3” for values greater than 40% of the “sarcasm value”. In our public repository [Using the trained SVM classifier, we generated “sarcasm values” for all 130,000 original texts in our dataset. These values are in the form of percentages that range from -100% (definitely not sarcastic) to 100% (definitely sarcastic). For each tweet the SVM classifier calculates a confidence score which is the signed distance of that tweet from the optimal hyperplane calculated during training. We then applied the hyperbolic tangent , Hope (used in SYRIZA’s slogans), Merkel (Germany’s Chancellor), memorandums (sets of austerity measures imposed by EU), bankruptcy, drachma (Greece’s currency before Euro) and recovery.voteYES or voteNO entities. The spikes in this plot are indicative of major events during the pre-referendum period. Analysis of the text from these tweets revealed that they were either prompting people to participate in certain demonstrations or they were retweets of the prime minister, urging for “NO” votes. Although the tweets’ volume is not a sufficient indicator of political inclinations of users, it can give insights regarding specific events. In −96) indicates a strong correlation between voting preference and number of tweets. If we limit our analysis in tweets that contain only “YES” or “NO” entities then, in average, “YES” users posted 2.1 tweets with the “YES” entity and “NO” users posted 1.7 tweets with the “NO” entity . This finding is suggestive that “YES” users were engaged in an orchestrated campaign to promote “YES” content. The deliberate use of bots or real conscripted users to promote a particular ideology or party prior to an election event  is a known phenomenon [Another interesting question regarding tweets’ volume is the potential existence of different tweeting pattern between “YES” and “NO” voters. To investigate this we measured the average tweets posted by “YES” users and “NO” users. We define as “YES”, a user that has posted at least one “YES” entity and none “NO” entity. Similarly, we define “NO” users accordingly. In total, our dataset had 1.558 “YES” users and 11.672 “NO” users. In our public repository  we show enomenon . Validatd = log(10 + cmax − c), where c is the number of tweets that contain a specific pair of entities, and cmax is the maximum c (max co-occurrence). We apply the “neato” visualization method of Graphviz software, which emulates spring link attractive forces between nodes [YES and NO entities are central to the discussions with a small in-between distance. Moreover, it is clear that Europe-related entities are closer to the YES point, whereas entities regarding domestic affairs including debt are closer to the NO point.Two entities are co-occurring if there exists at least one tweet that contains both entities. We define distance between entities as: en nodes . In Fig Sarcasm detection revealed some points of interest pertaining to the use of sarcasm in the political domain. Overall, 61.8% of the total referendum tweets and 58.7% of the total election tweets had a positive sarcasm value (>0%). Nevertheless the percentage of tweets with strong sarcasm  was 27.1% and 28.8% for referendum and elections tweets respectively. A plot of the Cumulative Distribution Function of the sarcasm values is included in our code repository . For comIronic posts were prevalent for specific hashtags, which, after looking into the text entities, revealed the level of the citizen aversion to the entities involved in the current situation, namely, the earlier governments and a company in the center of talk about corruption . On the z = 43.225, p<0.001 and for elections, z = 34.839, p<0.001.In both referendum and elections data, it is also worth noting that a negative correlation exists between sarcasm and number of hashtags journalists and the mass media against the positive sentiment towards Alexis Tsipras and freedom. For the elections, although new extreme positives and extreme negatives emerged (such as terrorism and poverty), the same four entities exhibited the highest polarization. Revealing the entities that exhibited the highest polarization of sentiment provided and examining the words that carried that sentiment, one may observe the actual cause of polarization. Upon examining the content of the referendum, the citizens perceived the journalist and mass media input as propaganda, an attempt to steer the citizens to vote for specific pro-austerity parties. Both analyses provided clear indication of the citizen perception of journalists and the mass media contribution to both electoral events. This was further reinforced upon examining the co-occurrence with the remaining two highly polarized entities. The negative sentiment co-occurred in the context of the propaganda while the positive in the context of establishing freedom through voting for the prospective candidate. Co-occurred sentiment polarization provided insight to the connections the citizens perceived and justified towards their elections voting.Although the sarcasm assignment provided a glimpse into the thoughts of the citizens revealing causes and worries related to the outcome of the referendum, there was a different but equally valuable aspect exposed by the sentiment polarity. Looking at the entities that exhibit the highest polarization of sentiment , one can notice how the citizens thought about the forces that actively tried to influence the outcome of the referendum . By retrz = 17.137, p<0.001; and a negative correlation also exists between negative sentiment and sarcasm, Fisher Transformation Test, z = 15.954, p<0.001. Similarly, for the elections: a positive correlation exists between positive sentiment and sarcasm, Fisher Transformation Test, z = 13.508, p<0.001; and a negative correlation exists between negative sentiment and sarcasm, Fisher Transformation Test, z = 19.719, p<0.001.Another feature across both datasets is the correlation between sentiment and sarcasm. With regard to the referendum data: a positive correlation exists between positive sentiment and sarcasm, Fisher Transformation Test, The computation of sarcasm and sentiment levels allows us to plot the temporal sentiment variation for any entity. To eliminate the influence of sarcasm, we applied “sarcasm correction” to the sentiment for tweets with positive sarcasm. Specifically, each tweet sentiment was corrected towards the neutral side proportionally to the percentage of sarcasm that it contained. In p < 0.001). The direction and length of the arrows in the figure portray the vectors of this change. In this figure, we notice a general shift of negative sentiments towards neutrality (the only exception is the “Debt” entity for the “NO” voters). Indeed, elections , did not attract the same negatively charged content. Moreover, after the referendum, “YES” voters expressed more positive comments regarding “ND”  and the prime minister Alexis Tsipras.After the referendum, the government did the highly criticized action to accept Eurogroup’s measures despite the high percentage of the “NO” vote. We expect that this move generated many sentiment shifts for various entities. Therefore, it is interesting to examine how “YES” voters and “NO” voters reacted to this development. To measure this, we split users into two disjoint groups: the “YES” voters and the “NO” voters. We also kept only users that have posted in both referendum and elections datasets. For each user in every group, we measured the average positive and negative sentiment for each entity in both datasets. Finally, for each entity, user group and sentiment, we applied the Mann-Whitney U test between the average sentiment of this group in the referendum dataset and the average sentiment of this group in the elections dataset. In Topic modeling is a powerful method to detect thematic patterns in text corpus. Applications in political discussions can reveal interesting ideological inclinations, tendencies and concept proximities. One of the most frequently used techniques in this area is Latent Dirichlet Allocation (LDA), which has been used in the past to analyze online content, like news items and blog posts, and for spam detection.Topic modeling generates a predetermined number of topics. For each topic it computes a per-entity distribution, or else the probability that an entity belongs to a topic. Additionally, topics can be projected in a 2-dimensional space for better visualization. As a result, two topics lying in distant places after LDA analysis, indicate that they have very different mixture of entity probabilities. In contrast two proximal topics indicate a concordance of entity probabilities.This method has also been used in the past in order to analyze political content in Twitter. During the German federal elections of 2013, LDA was used to quickly identify emerging topics  in TwittThe short length of Twitter posts, the special linguistic elements that they contain, and the variability of the political discussion, make the application of LDA in political-related Twitter content difficult . Here, wGreece might belong by 70% to topic 1 and by 30% to topic 2.For our study, we analyzed the manually-identified entities in the tweets with Gensim Python library, after excluding all tweets that had sarcasm identifier value higher than 5%. For visualization of the generated topics, we used LDAvis  that perFigs VoteYes and VoteNo are unexpectedly close. Entities associated with the VoteYes topic contain stronger negative sentiments, whereas entities associated with the VoteNo topic contain stronger positive sentiments (except the Conservatives entity). Interestingly, the topic that contains the Prime Minister (Tsipras) and his leading party (SYRIZA) lie in the middle of the ‘YES’ and ‘NO’ topics despite the fact he and his party were strong ‘NO’ supporters. We also notice a topic on the left with prevalent negative sentiments that contain entities pervasive to the anti-austerity discourse like ‘Varoufakis’, ‘Troika’ and ‘Eurogroup’.In SYRIZA party at the top, (ii) politically center entities  in the middle, and (iii) the far-left parties at the bottom. One interpretation of this placing is that the conservative parties in Greece were more strongly affiliated than the left and center-left parties.Regarding elections, in We presented a detailed analysis of two Twitter datasets from two politically associated electoral events. Sentiment analysis and sarcasm detection were performed on the data in order to achieve high accuracy. Entity detection combined manual, semi-automatic and scripted processing as well as lexical resources to correctly assign sentiment. This combination was necessary for tackling the traditionally hard-to-analyze political domain by blending entity-level sentiment and data statistics.Our results shed light on the often unnoticed societal and political trends that guide citizen choices and actions, which traditional polls fail to detect. The presented exploratory analysis revealed part of the public sentiment towards main entities along with their semantic proximities. This analysis was applied to two related electoral events, enabling the creation of lexical resources that covered the semantic content of a wide and complex political background. These resources, as an extension of the previously existing resources, were utilized for entity and sentiment detection, and are available both as general-purpose resources and, most importantly, optimized for the political domain. Although there is a plethora of studies in social networks regarding electoral or political events, most of these are concentrating on a specific aspect of the data . Here, we argue that in-depth discovery in similar datasets should include at least 5 types of analysis: volume analysis, entity identification, sentiment detection, sarcasm correction and topic analysis. Results for each type of analysis can improve other types . The analysis from this work aims to reveal quantitative aspects of data analytics that may prove helpful for political analysts and citizens alike. The semantic aspects of the results may be interpreted accordingly by the interested parties, making social interpretations on the qualitative level. Thus, while we are far from creating qualitative conclusions regarding the underlying social dynamics that affect a political discourse, we can certainly be assisted towards that goal by making sense of the vast amounts of social data through this approach.The results also hinted further work. Since sentiment is a descriptive work for all emotions, and not all emotions are the same , an inteFinally, the presented analysis does not include methods to detect bots that are massively employed to spread content in favor of a specific candidate. This phenomenon has been detected in many elections (e.g., U.S presidential elections 2016  and 2013To further improve our methods towards extracting qualitative insights regarding users’ political affiliations and in order to reveal potential malicious efforts to obscure the online discourse, we plan to analyze future electoral events mainly in the European area.


Schizophrenia is a complex, heterogeneous disorder, with highly variable treatment outcomes, and relatively little is known about what is important to patients. The aim of the study was to understand treatment outcomes informal carers perceive to be important to people with schizophrenia.Qualitative interview study with 34 individuals and 8 couples who care for a person with schizophrenia/schizoaffective disorder. Interviews were transcribed verbatim and analysed by a thematic framework based approach.Carers described well-recognised outcomes of importance, alongside more novel outcomes relating to: Safety (of the patient/others); insight ; respite from fear, distress or pain; socially acceptable behaviour; getting out of the house; attainment of life milestones; changes in personality and/or temperament; reduction of vulnerability to stress; and several aspects of physical health.These findings have the potential to inform the development of patient- or carer- focused outcome measures that take into account the full range of domains that carers feel are important for patients.The online version of this article (doi:10.1186/s12888-017-1418-8) contains supplementary material, which is available to authorized users. Improving treatment outcomes and quality of life for people with long-term mental health conditions are key aims of health care policy , 2. SchiWhile people with schizophrenia can give valid and reliable accounts of outcomes –24, sympA qualitative study using in-depth semi-structured interviews was conducted with self-identified ‘carers’ of a family member with a diagnosis of schizophrenia made at least 2 years previously. Ethical approval was obtained from NHS East of Scotland Research Ethics Service (EoSRES) REC 1 by proportionate review (Application Number 13/ES/0143). All participants gave written informed consent.n = 44) cared for someone with schizophrenia, and six cared for someone with schizoaffective disorder.A total of 34 individuals and 8 couples were interviewed (i.e. 50 people in 42 interviews). While qualitative methodology papers tend to avoid prescribing hard guidelines for sample sizes for qualitative studies, 25–30 participants have been deemed an acceptable minimum by Dworkin  and thisMost participants chose to be interviewed at home, but approximately 20% chose to come to the University. At the beginning of the interviews, carers re-confirmed that the patient had received a formal diagnosis of schizophrenia or schizoaffective disorder from a GP or psychiatrist, at least two years prior to the interview. Carers were then asked what they felt were important outcomes of treatment for the patient who they cared for: at present; at a time when the patient was particularly ill or unwell; at a time when they were more stable; and at a time when they were doing particularly well. Prompts were designed to encourage participants to discuss both directly-experienced outcomes, and important/desired but unattained outcomes. In addition, a series of prompts relating to key outcomes were compiled based on the conceptual review of the literature and feedback from a consensus conference, but were not in fact utilised in any of the interviews, as participants spontaneously discussed a broad array of outcomes of importance in response to the preliminary, general questions. After the initial 6 interviews, when it became apparent that participants identified multiple outcomes in response to the primary questions, without need for prompts, the researchers agreed that all future interviews in the study would proceed without prompts. Carers were encouraged to expand upon ideas that they themselves raised in relation to outcomes, rather than directed towards any specific topic. It was felt that this strengthened the data, as it reduced the potential for investigator bias. The topic guide, which was reviewed for tone and content prior to use by two carers and one person with schizophrenia, can be found in online Additional file Interviews were transcribed verbatim by a professional transcriber, and anonymised. Transcripts were analysed in NVivo 8 by JL, using a thematic, framework based approach . This inn = 1–4), ‘some’ for 12–24% (n = 5–10), ‘many’ for 25–50% (n = 11–21), and ‘most’ for >50% (n = 22–42)). It was not necessary for a participant to overtly state that an outcome had been experienced by the person they care for, in order to code their statement as an endorsement of that domain. While ‘endorsement’ of an outcome domain did, in some cases, take this form, any statement that either explicitly or implicitly indicated that a domain was relevant or important to that carer, was also coded within that domain. For example, where a carer identified that the person they cared for experienced ongoing difficulties with engaging in physical activity, or that they wished the person they cared for could have the energy to engage in physical activity, this was interpreted as the carer indicating that being able to engage in physical activity was an important outcome, and hence it was coded within the ‘physical activity’ category.Outcomes of importance in schizophrenia reported by the carers included symptom related outcomes, quality of life, functional outcomes, personal recovery, physical health and lifestyle, and satisfaction with treatment. Table italic font indicates novel/ modified categories which emerged from the current study ; from health risks linked to negative symptoms ; and from potential for deliberate self-harm related to affective symptoms.‘He was absolutely intimidated by his environment… he felt frightened and threatened’ [C25]The importance of reduction of, or relief from fear, distress and emotional  pain was raised in most interviews, often closely related to positive symptoms, but at the level of their physical and emotional consequences.‘He can rationalise…although he hears the voices he has a sense of reality.’ [C40]Insight was also mentioned in most interviews, encompassing both recognition that current/prior psychotic phenomena are not real, and understanding that one has a long-term illness. It was described as a gatekeeper to many other treatment benefits, partly through its impact upon treatment adherence, and was important in helping people deal with residual psychotic phenomena.Side-effects are not described in detail here as they are well reported within existing literature (e.g. ), but th‘[When] he's not taking his medication, he occasionally offends people in the street’ [C25]The concept of ‘social acceptability’ was raised in most interviews, i.e. behaving in a socially appropriate way and avoiding bizarre/unconventional behaviour. Many discussed the importance of treatment in helping patients avoid illegal behaviour (sometimes precipitated by symptoms).‘I think he missed out all his twenties and thirties so maybe catching up in some ways.’ [C03]The domain of ‘life milestones’ was added to encompass many carers’ reports of the importance of reaching key life/developmental milestones, such as attaining qualifications, learning to drive, moving out of the caregiver’s home, or having a family.‘The worst time that we've had was… when he was so unwell he didn’t go out the house for a year’ [C24]Simply ‘getting out’ of the house was mentioned in most interviews, and was consequently added as a sub-category of ‘leisure pursuits’. This encompassed the importance of being well enough to leave the house, which was something many patients needed to achieve before the more ambitious step of engaging in structured leisure activities or even activities of daily living.A novel sub-category of ‘pets’ was added within the ‘role functioning and productivity’ category, because the importance of being able to care for a pet was raised in some interviews.‘He reverted to his old self. You could reason with him, you could have a laugh with him’ [C46]The importance of ‘personality/temperament’ was raised in most interviews, and was often particularly valued by carers themselves. This encompassed emergence of aspects of the patient’s character, such as sense of humour, consideration, and thoughtfulness, and of a generally calmer temperament, more ‘like oneself’.‘Although he seems fairly even I don’t think it would take a huge amount to kick him over the edge.’ [C06]The vast majority of carers also mentioned ‘vulnerability/sensitivity’ to all kinds of stress, in most cases as a residual difficulty that treatment failed to resolve, rather than a positive, attained outcome.‘On such a high dose… of a sedating medication. Motivation is just not there’. [C46]Exercise/physical activity and diet/weight were raised by the majority of carers, who sometimes described how treatment facilitated physical activity and healthy diet (by improving symptoms that create barriers), but also described how side-effects  could act as barriers.‘She was drinking herself to sleep, I think, mostly because she had recurrent nightmares, and day time nightmares’ [C50]Many described the importance of outcomes related to drugs/alcohol/smoking, such as decreased reliance upon substances previously used to self-medicate positive or affective symptoms, or compensate for lack of social/functional activities.Daily routine was mentioned in many interviews, in relation to sleep and waking, eating and self-care, and was described both as a factor that contributed to improving other outcomes, and as an outcome in itself.All the schizophrenia treatment outcomes identified in the literature review and consensus conference preceding the study  were confirmed in these qualitative interviews, along with several novel sub-categories within existing domains and a novel category of physical health and lifestyle, thus giving a deeper understanding of outcomes in this condition. While a large proportion of the sample endorsed most of the themes, it should be noted that frequency information are indicative of the frequency of these domains within our sample, and cannot be extrapolated from to estimate the prevalence of these concerns in carers of persons with schizophrenia.outcome is not expressed in existing outcome measures. This highlights the need to consider physical activity as a potentially relevant outcome domain in its own right. Designing interventions for schizophrenia that include attention to physical health and lifestyle, could help improve outcomes for many patients.While the importance of physical activity for persons with schizophrenia is recognised within the literature , and lowSafety of the patient (and those around them), and reduction of their fear, distress or pain, were considered important by most carers, and it is easy to see why they would value these outcomes, relating to resolution of negative practical and emotional consequences of symptoms. While the importance of these outcomes may be intuitive, they are not explicitly represented in current outcome measures, and this study is novel in highlighting their particular salience. These outcomes could be described as ‘secondary’, in the sense that they could be logically expected to follow on from the more ‘primary’ outcome of amelioration of  symptoms. However, it could also be argued that there are other means of reducing patients’ fear, distress, or pain, aside from by symptom resolution, and thus outcome measures could benefit from assessing the extent to which treatments help to reduce a patient’s experience of these negative states. This could help professionals to gain a fuller understanding of how a given treatment programme is impacting on the individual’s level of fear and distress.when cognitive interventions may be more effective. Exploring the value of educating carers in ways to cope with poor insight in the person for whom they care, could be another important target for future work.Most carers also valued insight which they often reported to be associated with improved communication with the person with schizophrenia, and a return of their personality and/or of a more favourable, ‘normal’ temperament. This is consistent with findings that insight in schizophrenia is associated with social cognition , and lowreaching of milestones would for many be the goal, rather than the resumption of familial, domestic, occupational or educational roles and responsibilities. This highlights the fact that functional outcome measures in schizophrenia may need to take subtle levels of attainment into account, in order to accurately capture small gains.Within functional outcomes, many carers talked of ‘getting out’ (i.e. leaving the house), similar to the existing domain of engaging in leisure pursuits, but at a more preliminary level. Caring for pets, similarly, could be conceptualised as a specific form of role functioning/productivity. Where residual difficulties are considerable and/or recovery is particularly limited, less ‘ambitious’ functional outcomes such as these may be particularly relevant. This is consistent with the observation that traditional social functioning measures may not be relevant to people with severe disabilities related to schizophrenia , and witthe most important outcome; understandably so, considering that these are good outward indicators of wellness and ‘personal recovery’ and directly impact upon the patient-carer relationship. Indeed, temperament has been linked with functional outcomes and psychological health [Within the realm of ‘personal recovery’ many carers highlighted the importance of changes in personality and temperament, and several described the return of the person they used to know as l health . Also rel health . These wIn the sub-category of ‘leading a normal life’, a number of carers spoke of the importance of treatment for helping patients to avoid socially unacceptable/antisocial/illegal behaviours, (often precipitated by positive symptoms), in order to reduce risk of arrest or sectioning, facilitate social interactions and minimise stigma – consistent with findings that socially unacceptable behaviour is strongly associated with stigma in schizophrenia .Consistent with other studies, many carers expressed desire for greater monitoring of physical health . ExercisThis study confirms the key treatment outcome categories found in the current literature, and contributes evidence of additional outcomes that carers feel are important for patients but are not apparently captured in current thinking about, and measurement of, schizophrenia outcomes. However, there are some possible biases in the sample. The majority of carers interviewed were parents of a person with schizophrenia, with a gender bias in the sample, such that around three quarters of participants were female. However, this is in line with the gender balance found in other convenience samples of carers of persons with schizophrenia , and refThe outcomes carers identified as being important for patients may not be identical to the outcomes that patients themselves would identify. However, there is generally good agreement between the two , and as The findings from this study contribute to our understanding of the full range of treatment outcomes that carers feel are important to people with schizophrenia, and could contribute to ensuring research, treatment planning and assessment are aligned with the needs and priorities of patients . The bre


Women’s empowerment is an important goal in achieving sustainable development worldwide. Offering access to microfinance services to women is one way to increase women’s empowerment. However, empirical evidence provides mixed results with respect to its effectiveness. We reviewed previous research on the impact of microfinance services on different aspects of women’s empowerment. We propose a Three-Dimensional Model of Women’s Empowerment to integrate previous findings and to gain a deeper understanding of women’s empowerment in the field of microfinance services. This model proposes that women’s empowerment can take place on three distinct dimensions: (1) the micro-level, referring to an individuals’ personal beliefs as well as actions, where personal empowerment can be observed (2) the meso-level, referring to beliefs as well as actions in relation to relevant others, where relational empowerment can be observed and (3) the macro-level, referring to outcomes in the broader, societal context where societal empowerment can be observed. Importantly, we propose that time and culture are important factors that influence women’s empowerment. We suggest that the time lag between an intervention and its evaluation may influence when empowerment effects on the different dimensions occur and that the type of intervention influences the sequence in which the three dimensions can be observed. We suggest that cultures may differ with respect to which components of empowerment are considered indicators of empowerment and how women’s position in society may influence the development of women’s empowerment. We propose that a Three-Dimensional Model of Women’s Empowerment should guide future programs in designing, implementing, and evaluating their interventions. As such our analysis offers two main practical implications. First, based on the model we suggest that future research should differentiate between the three dimensions of women’s empowerment to increase our understanding of women’s empowerment and to facilitate comparisons of results across studies and cultures. Second, we suggest that program designers should specify how an intervention should stimulate which dimension(s) of women’s empowerment. We hope that this model inspires longitudinal and cross-cultural research to examine the development of women’s empowerment on the personal, relational, and societal dimension. Throughout history and across nations still today, men on average have greater access to power e.g., . The genEmpowerment is a process, from being unpowered to being empowered. Theorizing of empowerment stresses two main perspectives on this process: namely one more individualistic, namely through women’s individual capacities and free exercise of personal choice e.g.,  and one We provide a framework in which we propose that women’s empowerment can be differentiated in three different dimensions, namely personal, relational , and societal  empowerment. We conducted our analysis in the field microfinance services as it offers a large body of empirical studies based on literature from different disciplines, mainly psychology, developmental economics, and sociology, in three steps. First, based on the reviewed literature we define women’s empowerment. Second, we review empirical findings based on the three dimensions of women’s empowerment to illustrate how it has been investigated so far in the context of microfinance services across cultures. Third, we integrate these three dimensions in a Three-Dimensional Model of Women’s Empowerment to improve our understanding of what women’s empowerment entails and how microfinance services may help to increase it.Offering microfinance services  is currently one of the most prominent means to reduce poverty and empower the disadvantaged, including women e.g., . The undresources, agency, and achievements. Resources refer to material, human, and social expectations and allocations. Agency is the ability or sense of ability to define one’s goals, act upon them, and decide on their own strategic life outcomes. Achievements include a variety of outcomes ranging from improved well-being to achieving equal representation of women in politics. In other words, the underlying assumption is that women’s empowerment is the process of having and using resources in an agentic manner to reach certain achievements  which specific components of women’s empowerment are assessed and (2) differentiate those components in personal, relational, and societal empowerment.Importantly, research so far has studied a variety of very different components of women’s empowerment. Indeed, empirical research has investigated women’s empowerment with measures such as agency, autonomy, capacity for action, self-determination, and self-confidence e.g., . However1, the impact of training programs differing in content and length  the micro-level dimension, referring to individuals’ personal beliefs as well as actions where personal empowerment can be observed, (2) the meso-level dimension, referring to beliefs as well as actions in relation to relevant others where relational empowerment can be observed, and finally (3) the macro-level dimension, referring to outcomes in the broader, societal context where societal empowerment can be observed. In the context of women’s empowerment, capturing women’s self-confidence would be located at the micro level, women feeling and acting confident in relation to their partner or social network would be a meso-level outcome, and women’s situation in society would be located at the macro level.not to provide a full literature review, but an overview of different operationalizations of empowerment. We categorize different operationalizations into personal, relational, and societal empowerment to illustrate the importance of differentiating between these three dimensions. More precisely, we selected studies assessing commonly used quantitative and qualitative measures of women’s empowerment on the personal dimension, the relational dimension, and at a broader societal dimension (see Table 1 for an overview of the discussed measures).Importantly, our aim is personal empowerment as they assess different psychological aspects about personal beliefs and actions. We have selected two different commonly used operationalizations, namely control beliefs Further research in this field showed that women reported higher levels of self-esteem e.g., , strongerelational empowerment as they assess different aspects of women’s position in relation to others. Below we will illustrate three different studies, one investigating intra-household decision-making power  female microfinance borrowers was positively e.g., , negativsocietal empowerment. Thus, we suggest that future research should follow female microfinance borrowers over time to investigate how they achieve more opportunities and rights  and use these insights to directly conclude that access to microfinance services strengthens women’s empowerment within her social environment, this could provide a skewed insight and may have undesired policy implications. More specifically, when operationalizing women’s empowerment in terms of women’s personal control beliefs it is possible that women feel personally more in control (‘I know what I am doing’), but not in relation to their partner (‘My partner gets aggressive if I try to have a say in important decision-making’). In fact, previous research suggests that women’s increased autonomy resulting from her participation in microfinance services can destabilize the relationship between the female microfinance borrower and her husband and thereby increase the risk of intimate partner violence , which may then instigate the experience of empowerment on the relational and/or societal dimension. In line with this suggestion, previous psychological research conducted in the context of microfinance services stressed that women should first become aware of the options that they are individually capable of taking – i.e., their personal capacity – before they can actually proceed to influence aspects that are important to them in their daily life . SimilarIn the context of microfinance services, women’s empowerment may thus be seen as a process typically starting with personal empowerment and resulting in empowerment at all three dimensions, with societal empowerment as the final aspect to develop . Such an approach in politics aims to increase women’s presence in legislature and to improve gender-related policy outcomes such as inheritance rights e.g., . This exIn sum, we suggest that time is crucial in predicting empowerment effects. First, the model suggests that the time lag between an intervention and its evaluation may influence when empowerment effects on the different dimensions are likely to be found Second, the model suggests that the three dimensions are related but that the sequence in which they can be observed depends on the implemented type of intervention.In the current article we discussed studies conducted in a variety of different cultural contexts, such as Sri Lanka, Bangladesh, and Tanzania. Obviously, there are important differences between these cultures. Culture can be defined as the dynamic patterns of ideas, practices, institutions, products, and artifacts that are shared by certain groups of people . While iAs highlighted in previous research, gender relations vary both geographically and over time and therefore should always be investigated in specific contexts and pertain to realities of women’s lives rather than being based on a generalized assumption that they are oppressed . Indeed,First, previous research suggests that often-used indicators of women’s empowerment reflect an understanding of women’s empowerment based on culturally specific practices  that may not apply to other cultures e.g., . In lineIndeed, psychological scholars highlight the necessity to draw upon local understandings to resonate with local realities and better serve local communities . Since wSecond, women’s empowerment is considered as a process wherein women challenge existing norms and culture of the society in which they live . AccordiIn sum, the prevalence of gender inequity may obstruct possible structural societal changes resulting from access to microfinance services e.g., . EmpiricTo conclude, we suggest that cultures may differ with respect to which components of empowerment are appropriate indicators of empowerment. Moreover, we suggest that the cultural context should be considered to properly understand the development of women’s empowerment. Accordingly, when developing interventions, cultural norms should be identified and described when presenting impacts, thereby facilitating comparison between studies. To investigate at what time access to an intervention impacts women’s empowerment at each of the three different dimensions across cultures, we encourage future longitudinal and cross-cultural research to examine the development of women’s empowerment on the personal, relational, and societal dimension.In this paper, we aimed to increase our understanding of women’s empowerment and how it should be studied in future research. We can derive four main conclusions based on our work: First, women’s empowerment might best be conceptualized as a multifaceted process, which involves individual as well as collective awareness, beliefs, and behavior embedded in the social structure of specific cultural contexts. Second, based on the research reported above examining the impact of access to microfinance services on the development of women’s empowerment, we concur with conclusions by previous research e.g.,  that incMost of the work discussed in this paper operationalized empowerment based on an understanding of women as individual agents of change. However, including empowerment measures acknowledging the importance of vicarious others in women’s experiences of empowerment – focusing on the beliefs others in one’s network hold about an individual versus own beliefs – may enrich our understanding of women’s empowerment. Thus, concurring with the decolonial feminist perspective e.g.,  we suggeImportantly, we have focused on the measurement of women’s empowerment in the context of microfinance services. As such, the proposed model is most strongly substantiated in this specific context. Nonetheless, we propose that the suggested differentiation between three different dimensions may also apply to different interventions, which aim to strengthen the position of women. Additionally, in accordance with previous work e.g.,  we propoWe derive two main implications from our work. First, we suggest that future research should differentiate between the three dimensions of women’s empowerment specified in the Three-Dimensional Model of Women’s Empowerment, thereby increasing our understanding of women’s empowerment and its development and facilitating comparison of results between studies and cultures. We hope that our model encourages future research to focus more on the development of women’s empowerment over time. As a result, stronger theories may develop regarding how and why certain components on each dimension of empowerment could be impacted by different interventions. Second, but related, we suggest that program designers should specify how an intervention should stimulate which dimension(s) of women’s empowerment. When developing a theory of change , detailiEmpowering women is seen as one of the central issues in the process of sustainable development for many nations worldwide e.g., . Around The Three-Dimensional Model of Women’s Empowerment integrates different literatures studying the impact of offering microfinance services on women’s empowerment. The core premise of the model is to differentiate between three different dimensions of women’s empowerment, namely (1) personal empowerment, referring to individual’s personal beliefs as well as actions, (2) relational empowerment, referring to beliefs as well as actions in relation to relevant others, and (3) societal empowerment, referring to the situation of women in the broader societal context to understand how women’s empowerment may develop. Furthermore, unraveling two important moderators of empowerment, namely time and culture, the model allows a more dynamic understanding of why some women may feel more empowered than others, why some women may express higher levels of personal but not relational empowerment, and why one specific microfinance intervention may show positive impacts on women’s empowerment in one but not another nation. Integrating all three dimensions of women’s empowerment into one research model provides new theoretical insights into how women’s empowerment may develop through access to microfinance services and offers clear practical implications for involved stakeholders in the field.MH, NH, SO, and RL contributed to the discussion and development of the conceptual framework of this article. MH and NH conducted the literature review, analyzed the results, and wrote the manuscript. SO contributed to the rewriting of the manuscript. All authors approved the final version of the manuscript.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.