Pilinahā or the Four Connections Framework. Pilinahā addresses 4 vital connections that people typically seek to feel whole and healthy in their lives: connections to place, community, past and future, and one's better self. This article describes the origins, intent, key concepts, and implementation of this framework. By doing so, the authors hope to add to the growing body of work on community and indigenous well-being, further the dialogue with other indigenous communities, and collectively foster a more meaningful and effective health system for all.This article speaks to the abundance and wisdom of indigenous community members in Kalihi, an urban neighborhood in Honolulu, Hawaii. Its findings result from community members sharing their stories of health, health care, and healing. These stories evolved into a distinct framework for health— The larger context of this article is two discrepancies. First, indigenous people come from a lineage of interconnected well-being that is abundant and sustainable , 2. Yet Among industrialized nations, the United States continues to spend the most on health care and yet has the worst health and health care outcomes, including for infant mortality, mortality amendable to medical care, and access and equity of care . In an elāhui  to its traditional sources of healing and health. It motivates us to widen our gaze from biomedical measures of health and illness to our whole story and to listen—first, foremost, and deeply—to the stories of success, struggle, joy, and pain in our community.As indigenous people, we applaud this effort. Guided by ancestral wisdom, we have understood the significance of SDoH factors for generations, not as a passing trend but as a deep cultural, historical, and spiritual understanding of health as wholeness. However, even with this encouraging move toward SDoH, the US health system continues to frame health care as mostly an individual matter between the patient and the doctor and, in its atomizing market approach, discourages indigenous people from valuing ancestral pathways that have successfully healed and sustained our communities for generations. When seeking care at hospitals and private practices, indigenous people tend not to see their gifts and values reflected back , 8; instPilinahā or the Four Connections Framework. Pilina in Hawaiian means connection or relationship. The Hawaiian language commonly has layered meanings, and here hā references ʻehā, meaning 4, as well as hā, meaning breath of life. Pilinahā represents the 4 connections essential for health and healing. In this article, we share the origins of this framework—what it is, how and why it was created, and what it represents to us. Our hope is to add Pilinahā to the growing body of work on community and indigenous well-being and to encourage others to articulate their own community paradigms of health, wealth, wholeness, and happiness.Findings in this article are a result of Hawaiʻi community members sharing their accounts of health, health care, and healing with two community-based organizations: Kōkua Kalihi Valley Comprehensive Family Services (KKV) and the Islander Institute. The storytelling surfaced common understandings, issues, and values that in time constellated as a distinct framework, which we have come to call Food, nutrition, land, and agriculture were critical components of this process and served as a catalyst, programmatically (as an initiative spearheaded by food- and land-based programs at KKV), socially , and culturally . In keeping with our traditional values, we cannot separate nutrition from this overall framework of a community's health; hence, this article by necessity includes this larger scope. We also hope to illustrate how a food and nutrition program, when effectively linked to culture and community, can begin effecting change at an organizational and even systems level, beyond its conventional programmatic domain.Overall, while listening to our community, we witnessed how the underlying question became not How do we make individuals conform to the system? but, rather, How can we reshape the system to hold a more diverse representation of experiences, whole stories, and healing pathways? One participant shared how “when we place ourselves in a shared story, we become more than just something that needs to be treated.” We hope to uplift these community and indigenous voices to encourage a more meaningful and effective health system for all.waiwai . Roots staff and volunteers operate a café and gardens, annually producing more than 6000 pounds (∼2722 kg) of fresh produce that is distributed to the community. In addition, Roots helps reconnect more than 200 Kalihi residents with their land, culture, food, and community through more intensive programming, such as Ehuola, an experiential and intergenerational learning initiative that helps ‘ohana (family) clarify and progress toward their health goals. Currently, more than 20 families participate, cooking and sharing knowledge and stories of their traditional foods.Returning to Our Roots (Roots): Through community engagement in “growing, preparing and sharing” organic and culturally relevant foods, Roots works to increase food sovereignty, strengthen cultural identities, and nurture Ho‘oulu ‘Āina Nature and Cultural Preserve: In 2004, KKV secured a 20-year lease to steward and sustainably develop 100 acres of state conservation land deep in Kalihi Valley. Dedicated to cultural education and community transformation, this land was named Ho‘oulu ‘Āina, meaning “to grow the land” and “to grow because of the land.” Ho'oulu ‘Āina addresses the health needs of the Kalihi Valley community by strengthening the connection between people and land. The community converges to create a thriving upland resource of forest, food, knowledge, spirituality, and healthy activity. Each year, more than 5000 residents and visitors join in native reforestation, organic gardening, “green job” training, environmental education, and the preservation of cultural and healing practices.KKV is a community-organized and operated Federally Qualified Community Health Center and receives funding through the Health Resources and Services Administration as part of the nation's primary care safety net. We serve the residents of Kalihi, a federally designated Medically Underserved Area in urban Honolulu, Hawaiʻi, and home to more than 50,000 individuals of mostly Asian and Pacific Island descent. This diverse and culturally rich community continues to be the first home for many immigrants and migrants when they move to the state. KKV's more than 200 staff interpret in more than 22 languages and dialects and provide an array of clinical, social, and community-based programs. KKV administers 2 intertwined nutrition-related programs:The conception of the formative work supporting Pilinahā was primarily birthed by the director of the Roots program and supported by KKV's executive director and director of Hoʻoulu ʻĀina.aloha . The two facilitators from Islander Institute who co-led each of the discussions have decades of experience in place-based initiatives, planning, land use, public health, policy, and community organizing. Equally important, they have a long-term working relationship with KKV and have successfully collaborated on other projects in the past.Islander Institute, which was contracted to help facilitate discussions, is a civic enterprise founded to address the most important systemic issues in Hawaiʻi, including education, health, social justice, and land use. Through leadership development, community building, and trusting in the wisdom of island people, Islander Institute works to reinstitute island values such as caring for land, building strong communities, honoring traditions, and living a life of responsibility and manaʻo  to this effort. They agreed that it was important for us—as health providers, patients, and stakeholders of a community—to collectively talk about our understandings of health and healing, in part because everyone agreed that, as a whole, the US health care system did not represent or accommodate indigenous perspectives of health. KKV staff began this endeavor not as a formal research study but, rather, as an internal exploration with the intent of identifying whole measures of health appropriate for our community. Thus, this article is a perspectives piece on our findings rather than a report of research results and was not reviewed by any institution ethics review board. However, all participants were informed of our intentions and were inspired to voluntarily contribute their To explore what a broader framework might look like, organizers focused on bringing people together to elicit community perspectives on the meaning of health. To successfully do this, organizers gave great thought to the sequencing of activities, the invitation list, the types of foods to prepare, and the space needed to hold deep and meaningful conversation on indigenous health. What emerged was a powerful framework that speaks to the traditional practices of our community.1) vocation, including local farmers, producers, educators, academic professors, and leaders of cultural, health, and youth organizations; 2) relationship to KKV, including staff, patients, and stakeholders; 3) relationship to the Kalihi community, including residents, service providers, and advocates; and 4) connection to the Native Hawaiian community, including individuals of Native Hawaiian descent and/or those representing organizations working with Native Hawaiian communities.KKV and Islander Institute held a series of 4 formal gatherings in 2014 (November 7 and 8 and December 10 and 17). The original framework is based on stories collected during these gatherings. Thirty-eight unduplicated adults—25 women and 13 men all currently living in Hawaiʻi—attended. Organizers used purposive sampling to hear the depth and breadth of perspectives of personal and collective health. Purposive, rather than random, sampling is common in qualitative research when rich detailed information is desired. All participants were invited by organizers based on their trusted relationships in the community, decades of experience in their various vocations, and their deep connections to community health and indigenous rights. To help ensure maximum variation in perspectives, organizers used a sampling frame based on individuals’ We did not confine our information gathering to these 4 meetings, in keeping with a more indigenous approach to research. The indigenous search for knowledge traditionally happens over multiple generations rather than during preset projects or funding periods. We continued to pilot test and share the framework during the following 4 years at more than 16 meetings, workshops, and conferences with KKV staff, patients, participants, and stakeholders. Gatherings included group discussions with Native Hawaiian elders at a community health center on a neighboring island and with multiple generations from multiple families at a local community center. These follow-up sessions provided additional stories and reactions that refined and solidified the framework and confirmed that the views shared in the initial 4 gatherings accurately reflected those of our community, patients, and program participants. These follow-up activities are discussed in more detail later.aloha or welcoming circle, in which people held hands and shared their names, the place(s) they called home, and an ancestor they wanted to bring to the circle. Participants bonded through mālama ‘āina (working the land) and sharing several ‘ai pono meals . Participants prepared meals with ingredients that their ancestors enjoyed from land the participants worked together. Our intention was to foster good health through feeding bodies and spirits. Only after the group established trust did we provide a simple discussion prompt: Share a story about the last time you felt healthy. Discussions were held in English, in some cases with participants responding in English and Hawaiian. When responding in Hawaiian, participants would then translate into English for English-only speakers to understand. No audio or visual recordings were made. KKV staff transcribed all group discussions. No quantitative information was collected. Overall, this methodology honored the protocols of our community. The richness of participants’ sharing leads us to believe this type of engagement allowed people to share deeply without feeling studied.Before engaging in the work of discussing health, organizers encouraged camaraderie among participants by holding caring space at Hoʻoulu ʻĀina. Each interaction began with an After the 4 gatherings were completed, Islander Institute conducted the analysis. Because this was not an academic research project, the institute employed no formal qualitative data analysis styles or coding protocols. Instead, our analysts, who had previously fostered trusting relationship in the community through their years of work, both facilitated all discussions and then studied deeply the transcriptions for similarities among participant stories. This process most resembles grounded theory principles, as similarities (themes) were informed directly from the stories themselves. Once analysts identified the 4 connections as links among the different stories, they shared the connections with the core organizers to ensure accuracy and appropriateness. Both Islander Institute and KKV discussed and agreed on all themes during an ongoing series of face-to-face meetings and emails during the following 6 months. Organizers continued to test and discuss the strengths, weaknesses, questions, and growth of the framework—both collaboratively and separately—during a 3½- year period, up until the writing of this article, over email and during many additional formal and informal gatherings.Connection to place—to have a kinship with ‘āinaConnection to community—to love and be loved; to understand and be understoodConnection to past and future—to have kuleana (a purpose in the world)Connection to your better self—to find and know yourselfWhen participants shared their stories of good health, they referred consistently to feeling connected. When sharing stories of poor health, they recalled momentary or chronic loss of connection. Participants spoke about health practices that honor concepts of wholeness and inclusiveness. Health for them is a journey—a series of positive actions that emerged from their struggles and challenges and that the abundance and resilience of community enabled. In their practice of health, they may move forward or backward, but always they desire to move toward greater degrees of connection within themselves and among their community of care, which includes doctors, families, neighbors, ancestors, and places. When we acknowledge the importance of these connections, communities, rather than individuals or symptoms, are the smallest unit of health . Such prThe ideas presented in the following discussion are from participants, including direct quotations.Connection to place is to know, touch, steward, and love the lands, seas, and winds of our ancestors. ‘O ka hā o ka ‘āina, ke ola o ka po‘e (the breath of the land is the life of the people) is a core belief of our land-based programs. Touching, sitting quietly in, or having a conversation in a place that reminds people of home can be spiritually healing in and of itself. One woman explained, “When my father returns home after a long trip the first thing he does to reconnect is not visit with his children, but jump in the ocean.” Other ways of deepening connection to land include mālama ‘āina  and eating traditional foods grown from land that we mālama. When we eat with intention and are “more aware of what we put into our mouth . . . we have a stronger understanding of our responsibility to create a healthy food culture in our family.” Questions that providers, front-line staff, evaluators, and researchers might use to help assess connection to place include the following:What is the story of the place you call home?What places are special to you?Can you access the places you need to be in?Supplemental Appendix A documents more examples of questions and practices for each domain.Connection to community refers to the love, intimacy, trust, security, familiarity, and happiness shared with others. In many indigenous cultures, community encompasses relationships with family, friends, neighbors, and land. However, the notion of community is richer than just a group of people because it also speaks to the abundance held in the communal. Originally, this section was coded as “connection to others”—terminology that was approved by our team and subsequently used in many presentations and workshops with KKV staff, community members, and other stakeholders. During this time, no one raised concerns or opposed this coding. However, when our artist was redrawing and painting the framework image for this article, person after person responded negatively to this term as “othering” and “exclusive.” Our team reconvened to address this new reaction of community members who did not want to describe themselves as separate from one another. We were inspired by the following question: How can we describe that we are parts of a whole without breaking us apart into “others”? To help us understand what to do, we went back to community members and presented them with 2 frameworks that were the same shape and color, but 1 included the term “community” and 1 included “others.” Several people preferred the framework with “others,” explaining, “I like the word others because it makes you think about the things you are connecting to.” However, the majority of respondents explained that they resonated more with the word community, that community “included the idea of others, but was bigger,” and that “everyone knows what ‘community’ means but they won't know what ‘others’ means.”Do you feel good about the people you work with?Who are the people you eat with?Have you told someone you love how much they mean to you?We see the beauty and pitfalls of both terms. Needing to make a decision, we choose to rename this section “community.” All of the foundational stories still speak to this new word. For example, a father shared, “If I cannot talk to my kids, why am I trying to be healthy?” Our relationships to community allow us “to create memories, to remember, to become hungry, and feed each other's hunger.” Practices to deepen this connection include generosity and taking time to talk deeply with others. Questions to help access connection to community include the following:wā ma hope, meaning time past. It refers to the knowledge that our future depends on our history. In Pilinahā, connection to past and future represents the idea that we are all part of a cultural strand. “Our genealogies makes us whole, it's an unbroken chain of stories and practices.” The practices and knowledge of our kūpuna (ancestors) help guide our present actions, which in turn affect future generations. “We teach [this concept] in our indigenous birthing class. That the food the family eats, the words they use, their practices, bathe the sperm and egg inside of them, which hold the sperm and eggs of their great-grandchildren, and all their future generations.” By carrying on traditional practices and having cultural understanding and appreciation, we deepen our connection to the past and future. Questions to access connection to past and future include the following:Do you feel you are making your ancestors proud?Do you use traditional language?When was the last time you listened to an elder?In Hawaiian, the word for future is connection to your better self signifies our ongoing effort to become a better person for our families, community, and lāhui. Any situation in which we find ourselves, even if we are ill, is an opportunity to manifest our better selves. Participants shared stories of friends and family who had cancer or were hospitalized and “in spite of physical ailments, [they were] healthy in spirit, soul, and in many other ways.” The act of reflecting upon and sharing our own stories, happy or traumatic, helps us understand and heal ourselves. Other practices that deepen this connection include finding and sharing our gifts and being aware of what we put into our bodies. Questions to help access connection to your better self include the following:What is the story of your name?When was the last time you ... cried? Laughed hard? Felt proud?Last, āina, or in a fishpond, or gathering limu (seaweed) and doing traditional practices, I actually see them get physically stronger. They get connected to who they are, and to each other. And they feel power—not power over others, but power and control over their own destiny.When I see people working together on the ‘Woven throughout every story was the importance of spirituality. We considered adding spirituality as another connection but ultimately decided that spirituality should not be separated into a distinct domain because it is embedded in everything—in every connection, relationship, and intention. This fact underscores how the 4 domains are not separate but are interconnected. Each connection is necessary; no 1 alone is sufficient for health and happiness. This interconnectedness underscores the need for spaces and practices where people can make multiple connections. One participant shared the following related story:hoaka, a crescent moon representing a full bowl. It is a symbol of abundance. Traditionally, this kind of moon indicates abundance in the land and ocean, and in the framework it refers to the collective abundance of our connections. The 4 woven elements of the framework—community, best self, past–present–future, and place—create a whole, with the open space in the center, intentionally left unnamed, as the interconnecting element of spirituality.Created by Puni Jackson, the painting in The foundational stories and practices of Pilinahā are not new. Pilinahā did not transform KKV's community programs. These programs were already led and designed by indigenous people and steeped in the 4 connections. Roots programming is based on the Native Hawaiian belief that food represents our ancestors and our gods; that as we eat, we are ingesting all the good qualities of our ancestors; and that as we work the land to grow our food, we do it with great respect and intention because in so doing we take care of our ancestors, who in turn will care for us. This approach is highly complementary to other Pacific and Pacific Rim peoples in the valley, including Chuukese, Marshallese, Samoans, Filipinos, and Okinawans, who also connect to ancestors and traditional healing practices through Roots food-based programing and its long-standing emphasis on using food to deepen connections to best self, community, past and future, and place.more intentionally. As a formal lens, Pilinahā reminds us that we perpetuate the practices of our grandmothers not out of habit or sentimentality but, rather, because these deep healing practices keep our communities strong. To illustrate these points, we include the following paraphrased evaluations by two Native Hawaiian community members who completed their dietetic rotations at KKV community and clinical programs, including Roots Café and Hoʻoulu ʻĀina:I learned things at KKV that college didn't teach me. I saw the commitment to listening to the community. I witnessed a holistic approach to patient and community care. I saw how the social determinants of health can be addressed through many venues including integrative medicine, hands on work on the farm, and serving delicious local lunches at the Roots Café. I learned the importance of both the clinical aspects of nutrition as well as the cultural practice of building connections between the health of the land and the health of the people.I participated in the practice of Aloha circles before important meetings to bring staff together. We chanted, prayed, and shared personal feelings and tears with each other. This is something that's not very common anymore, especially at the workplace. Our society today is numbing its feelings; we don't listen to ourselves anymore, because at times the feelings are painful. Roots Café is helping us reconnect with our ancestral roots. This reconnection taught me it was okay to be myself. That everyone's voice is welcomed, which in turn helped me find my way home.What was new and innovative for KKV was how we used Pilinahā as a framework to begin evaluating and reflecting upon all our program activities, objectives, and outcomes Using Pilinahā, we see here how the interns made connections with their coworkers and their community, their better selves, the land, and their genealogy.As we work to understand the ways we already practice the values of Pilinahā, we strive to strengthen this practice and incorporate Pilinahā more widely across the whole organization. Our recent efforts include integrating Pilinahā into program evaluation, personnel assessments, staff development, and staff wellness and healing activities. For example, we have begun conducting an organizational-wide 4-hour curriculum for all of our staff to experience Pilinahā. The director of Ho'oulu Āina leads staff through a series of activities to uncover and share the connections and disconnections in their own lives and the lives of their patients and program participants. It is a way to talk in a safe space about the trauma of disconnection and healing through reconnection. Many KKV staff reported how this curriculum helped them feel “seen” and to “see others” better and to feel “connected,” “loved,” “valued,” and “reinvigorated”—reminding them of the important work they are doing. One participant reflected how “it reminds me about my passion for my work, and why it feeds my soul.”I was concerned for her health prospects and so I passionately told her all the horrible things that could happen to her. She patiently listened to me, nodding her head but not saying much. When I was done, she pulled out photographs of her grandson and explained how proud she was that her grandson was graduating high school and would be the first one in her family to go to college. At the time, I interpreted this abrupt change in topic as the patient's denial of the seriousness of her disease. Now I understand that she was trying to share what was important to her, what she valued, and her source of meaning and health.When asked what he would do differently if he could go back to that day with his current understanding of Pilinahā, he replied:I would listen more intently. I would engage her in a more human way. After listening to her values maybe I would explain my values as her doctor, so we could have an honest mutual understanding of each other. In my career, I have been lucky to have had many loving Samoan grandmas patiently teach me that only after I know my patients’ values can I begin to understand their pathways to healing.Participants reflected on how Pilinahā also helped them better reconnect their patients and program participants to healing services. For example, 1 KKV physician shared a story of how the 4 connections shaped his delivery of health care. As a young doctor, he had a Samoan patient with poorly controlled diabetes:As an ongoing practice, many KKV program leaders hold space for participants to reflect on their experiences throughout their programs. Prompts are open ended and include the following: “What made you participate in the program?” “Please share any insights, transformations, or issues you had during the program”; “What does ‘aipono mean to you?” and “What challenges or barriers do you face and what kind of support would help your practice of ‘aipono?” When possible, we have KKV staff scribe participant reflections. Reading through these documents, with the Pilinahā framework in mind, we clearly hear the 4 connections mentioned in participants’ reflections, without specifically prompting them to talk about connections.lāhui.” Other participants also spoke to learning about the importance of food, “being wiser about what we eat . . . choosing how and what we do. How it affects us now, in the future, and possibly how our kūpuna (ancestors) previously did . . . it makes us re-think whatever it is we're going to do.” Whereas epigenetic studies of native people most often highlight the genetic perpetuation of negative colonial impacts, these ancestral approaches to food and nutrition follow an ancient example of setting genetic intentionality around love, spirituality, and connectivity in a society.For example, 1 woman in our Native Hawaiian birthing group explained that through this program she now understands “feeding my family healthy foods is not only important to my family, but to our I think for me the thing that stands out the most about these past couple years in Ehuola is the community of people that you get to grow with. And the relationships that the kids are building around this particular content area. So that they can have support. As they grow older they know that they're not the only ones trying to live this way, there's a whole community of people who are trying to . . . their friends and with the guidance of their leaders, who they really look up to also.Participants in our Ehuola program explained “how big” the program was in its “holistic” coverage. Ehuola aims to engage children and their families in food-focused learning, maintain connection to land and their communities, and increase community food productions while highlighting Native Hawaiian cultural practices. The importance of connection was a reoccurring theme in participant reflections. One Ehuola parent shared the following:1) forming stronger bonds with one another, 2) remembering what is important to them, and 3) being inspired to share these connections with others. Witnessing these reactions motivates us to share Pilinahā more widely.When we began this endeavor, we had no thought of publishing our findings. Our sole intention was to better understand our community and our work. After creating Pilinahā from the stories of our first 4 meetings, we continued over 3½ years to pilot test and operationalize the framework. To this day, we continue to integrate Pilinahā into our organizational voice, to use it as a lens to evaluate our programs, and to share it with KKV staff and the many hundreds of students, residents, and volunteers who rotate through our programs, to promote self and communal healing. Outside of KKV, Islander Institute and KKV have used Pilinahā as a framework for convening a variety of organizations throughout the state, including insurance companies, foundations, universities, and nonprofits. The events focused principally on how to shape our health system around what matters most to communities. During this testing phase, we saw how the framework resonated widely. We saw how the 4 connections spontaneously surfaced even during single meetings with indigenous and island people. We saw how during facilitated conversations surrounding people's personal connections and disconnections to place, self, community, and past and future, transformations began to take shape, including participants Yet we recognize its limitations. We have no reason to believe Pilinahā applies to every community. We prefer that people engage their own community processes and clarify their own perspectives and frameworks for health, which may or may not resemble Pilinahā. Also, much work remains to be done to refine Pilinahā, including working out how it addresses spirituality, how it translates into more intentional practice in more varied programmatic and clinical settings, and how it may have unforeseen applications.Even with these limitations, we are encouraged by Pilinahā thus far. By planting the seed of valuing connections, we have witnessed multiple instances of personal healing in participants as well as stronger intercommunity relationships that continue to develop and bear fruit. This fruit includes other organizations using Pilinahā in myriad ways. It was integrated into health professional curriculum for students of public health and nursing and for state health employees. It is being employed by a national facilitation organization to bridge conversations between federal, state, and native communities around land access and usage. In addition, a local foundation is currently interested in integrating Pilinahā into its grant-making criteria.mālama āina programs.  We wish for people to see clearly their healing opportunities available through the company of others, through the places and lands they call home, through their histories and traditions, and through their sense of value in whatever context they find themselves. We aspire for health to be an attainable ideal in which people recognize their agency. Pilinahā speaks to the enormity of wisdom and knowledge developed over many generations by cultures and people who lived sustainably for millennia. Our intention is to continue to honor and uplift this insight. From their stories, sparks fly that we hope will re-inspire our systems to hold the dreams and health of all peoples.We hope that by exchanging more stories with others, we can continue to build the tool kit of questions, practices, and metrics for applying Pilinahā. We are especially interested in metrics that focus on those connections we wish to grow, rather than those that overemphasize the development of more health care services. We envision a health system that upholds the gifts of all providers—clinical professionals and traditional healers alike—and all places, be they hospitals or nzz001_Supplemental_AppendixClick here for additional data file.


Evaluation of health technology programmes should be theoretically informed, interdisciplinary, and generate in-depth explanations. The NASSS  framework was developed to study unfolding technology programmes in real time—and in particular to identify and manage their emergent uncertainties and interdependencies. In this paper, we offer a worked example of how NASSS can also inform ex post (i.e. retrospective) evaluation.HealthTracker) had shown promise in a cluster randomised controlled trial (RCT), but its uptake and sustainability in a real-world implementation phase was patchy. To explain this variation, we used NASSS to undertake secondary analysis of the multi-modal TORPEDO dataset  as well as a sample of new, in-depth narrative interviews with TORPEDO researchers.We studied the TORPEDO (Treatment of Cardiovascular Risk in Primary Care using Electronic Decision Support) research programme, a multi-faceted computerised quality improvement intervention for cardiovascular disease prevention in Australian general practice. The technology  was distributed across stakeholders in the system, clinicians’ experiences and concerns, organisational preconditions and challenges, extra-organisational influences (e.g. policy incentives), and how interactions between all these influences unfolded over time.Ex post analysis revealed multiple areas of complexity whose influence and interdependencies helped explain the wide variation in uptake and sustained use of the The NASSS framework can be applied retrospectively to generate a rich, contextualised narrative of technology-supported change efforts and the numerous interacting influences that help explain its successes, failures, and unexpected events. A NASSS-informed ex post analysis can supplement earlier, contemporaneous evaluations to uncover factors that were not apparent or predictable at the time but dynamic and emergent. The fortunes of technology programmes in healthcare are notoriously stormy –4. EffecTG’s group undertook a narrative review of theory-informed frameworks for analysing and evaluating technology programmes in health and social care . PreviouThe NASSS framework is shown in Fig. Domain 1 is the illness or condition. Complexity occurs when the condition is metabolically volatile (e.g. diabetes in pregnancy), inherently unstable (e.g. drug dependency), poorly described or understood (e.g. a newly discovered syndrome), associated with co-morbidities (most commonly in elders), or strongly influenced by socio-cultural factors  .Domain 2 is the technology. Complexity may relate to its material properties , knowledge needed to use it, knowledge it brings into play , supplyDomain 3 is the value proposition—both supply-side  and demand-side . Complexity in this domain relates to difficulties in formulating a credible business plan (e.g. when efficacy or cost-effectiveness studies are unavailable or contested) , 15.Domain 4 is the adopter system: the staff, patients, and carers who will be expected to use the technology (but who may refuse to use it or find they are unable to use it). Complexity may arise, for example, when the roles and practices assumed by the technology threaten professional traditions and codes of conduct .Domain 5 is the organisation(s). Complexity in this domain may relate to the organisation’s general capacity to innovate  , its reaDomain 6 is the wider system, including the policy context, support from regulatory or professional bodies, and public perceptions . This doDomain 7 is embedding and adapting over time. Complexity may arise from the technology’s “brittleness” (inability to adapt to changing context) or from the organisation’s lack of resilience (inability to withstand shocks and setbacks through learning and adaptation) .HealthTracker, into Australian primary care) and the large empirical dataset it generated. We describe how we applied the NASSS framework to the TORPEDO secondary dataset and to a new sample of primary interviews with programme staff. In the “The NASSS framework is a sensitising device that incorporates and combines a range of existing theoretical perspectives on illness and disease, technology adoption, organisational change, and system change . EmpiricIn 2011, in the introduction to “Explaining Michigan” , Dixon-Woods et al. observed: “Understanding how and why programmes work—not simply whether they work—is crucial. Good theory is indispensable to advancing the science of improvement.” (page 167) .The Michigan Keystone Project achieved a dramatic reduction in central venous catheter bloodstream infections in more than 100 participating intensive care units across the state . It was assumed it would work) ”; quotes from the original TORPEDO dataset are labelled with the original coding notation .HealthTracker was designed for use in two kinds of patient: those who already had cardiovascular disease and those  who were potentially at high risk of developing it. Established cardiovascular disease is well characterised, and guidelines for its management are relatively uncontested and widely accepted. The evidence base on managing cardiovascular risk is more complex. It is skewed towards a white European and North American population . Furthermore, since cardiovascular risk is a continuous variable influenced by multiple risk factors such as blood pressure and cholesterol levels, HealthTracker could not offer an unambiguous binary categorisation of patients into “high risk” or “low risk”.In Australia (as elsewhere), cardiovascular disease is strongly patterned by socio-demographic factors: it is commoner in those who are poor, those with low health literacy, and in Aboriginal people. Such individuals are more likely to have comorbidities such as diabetes or mental health conditions (“High risk is a sort of multifarious set of component conditions”—ex post interview 2). They may also have cultural beliefs and practices that affect their ability and willingness to understand the risk communication and comply with preventive treatment.HealthTracker worked well for many patients . But it assumed that patients would be able to understand a visual representation of quantitative risk and make a rational decision to alter their lifestyle based on it. This was not always the case, partly because of numeracy , and partly because of competing priorities in complex lives .The risk communication tool in ... it’s no good saying we’ll change your lifestyle. I’m 66 years old. I have a lifestyle, you know. I’m not an alcoholic. I don’t over-drink. I don’t you know I don’t overeat. I’m just, just a big bloke. Look, I didn’t walk out of there thinking, oh, I’m only going to eat salad and, you know, drink water for the rest of my life.—Interview with patient in TORPEDO study The TORPEDO findings confirmed that cultural habits die hard and familiar folk models of illness and risk may over-ride less familiar epidemiological ones . A 66-yeThe “no symptoms, no problem” mindset helps explain why patients with established cardiovascular disease appeared to engage better than those flagged for primary prevention .HealthTracker had many attractive design features . Many valued the way it structured care . But GPs described technical glitches  that were frustrating and interfered with their use of the technology in real time. Many found themselves regularly on the phone to the helpdesk. A major concern was that “apps that would just chew up memory, make the EMR [electronic medical record] run slowly; people said, ‘I don’t want to have anything to do with this thing, because it’s making my existing work flow worse’” (ex post interview 2).HealthTracker into existing workflows and practices for quantifying risk, advising patients, and prescribing medication. HealthTracker appeared in a side bar on the GP’s screen with pop-up prompts, “and sometimes prompts would go up, [or] wouldn’t; some of them wouldn’t see it because they would [mistakenly] shut it off, [and] … would say, oh, it’s gone, I don’t know where it is.” (ex post interview 1). Because of these technical imperfections, GPs participating in the TORPEDO study soon divided into highly motivated and/or technically adept ones, who persisted with the technology, and the rest, who gave up on it .Despite its visual appeal, “the user design wasn’t very good” (ex post interview 3), and in retrospect, findings suggest that it was not fit for purpose. It was not easy to integrate HealthTracker’s inbuilt algorithms foregrounded “hard” risk data  at the expense of “softer” data  that could have informed a more individualised approach to care  the reverse occurred” (ex post interview 3). This meant that the risk score and management plan did not automatically populate the patient’s record—a feature that contributed to clinicians’ experience of “clunkiness”.The TORPEDO project team sought to facilitate adoption by ensuring from the outset that HealthTracker was developed in a university setting by publicly funded research. There were two implicit potential models for introducing it into Australian primary care. The technology could be sold directly to GP practices or paid for by or through government entities, e.g. Primary Health Networks (PHNs), which are the organisations responsible for planning and commissioning primary care services, one for each of 31 geographically defined locations across Australia; and Medicare, which is the publicly funded universal healthcare system in Australia. The value proposition varied accordingly. The Australian government is prioritising digital health initiatives . In addition, there was hope among the TORPEDO team that HealthTracker would support a shift towards a more prevention-oriented healthcare system. One researcher commented that the Australian primary healthcare system is designed to be “… reactive, not proactive, and what we’re trying to do [by introducing HealthTracker] is to graft on some extra things that make it more proactive” (ex post interview 4).ves see “” sectionHealthTracker were to be scaled up to a larger population, the intervention has potential to prevent major CVD events at under AU$50,000 per event averted. However, at the PHN level, investment decisions for commissioning similar interventions based on cost-effectiveness analyses are scant.The TORPEDO team also anticipated that the value to Primary Health Networks  would be in the form of improved workflow and easier audit and performance management within GP practices. A modelled cost effectiveness analysis showed a small but statistically significant reduction in clinical risk factors within a PHN population based on the TORPEDO trial data, suggesting a small economic benefit from preventing CVD events (paper submitted). The economic evaluation showed that if The heavy burden of preventable and costly cardiovascular disease, particularly in Aboriginal communities, made the value proposition particularly compelling for community leaders. One researcher recalled, when recruiting the Aboriginal Community-Controlled Health Services to the TORPEDO trial, “it was just so pressing how – every one of the [community] board members, either themselves or relatives, knows someone who’s died of heart disease, or stroke, or diabetes, or kidney diseases; it’s just absolutely everywhere” (ex post interview 2).HealthTracker in positive value terms as supporting better (more proactive) care and making it easier and quicker to follow evidence-based guidelines and monitor their own performance. They felt it could potentially save them time “because it got all sorts of information out of the medical record and told you what otherwise you have to go hunting for” (ex post interview 5). Because HealthTracker synthesised several guidelines so as to streamline decision-making in patients with multi-morbidity, it saved considerable time sourcing individual guidelines.Some individual GPs shared this perspective, viewing HealthTracker increased overall consultation length  the majority of patients at high risk have got multiple problems and need much more hands-on working” (ex post interview 4). And some GPs had commented that using HealthTracker would increase their professional status in the eyes of current or potential patients—“that you’re a 21st century doctor and you’re doing the right thing” (ex post interview 4).The value of HealthTracker focused consultation.However, patients’ main priority when choosing a GP was not always quality of care delivered. For example, in ethnically diverse areas of Sydney, “[a large proportion] of the GPs consult in a language other than English, people find the same language, same culture GPs, … that’s what people are looking for, they’re not necessarily looking for them following guidelines” (ex post interview 4). Some patients were driven predominantly by material needs. The Australian copayment system meant that out-of-pocket payments for a GP consultation could be $30–$50, which might place negative financial value on additional medication and GP appointments triggered by a HealthTracker for more than half of eligible patients, even though most expressed positive attitudes to the technology   into” (ex post interview 2). In some, there was a dedicated individual focused on audit and quality improvement  in TORPEDO study, 2282-001). Notably, some of the more confident larger practices sought a high degree of autonomy over how and when HealthTracker was used.There was wide variation in participating practices’ underlying capacity to innovate. Technical infrastructure was sometimes poor, increasing the likelihood of technical crashes . Some larger GP practices and Aboriginal Community-Controlled Health Services (ACCHSs) had “been engaged in quality improvement work very strategically for about 15 years [and] already had an operational structure that they could weave  probably needed a bit of arm twisting to sign up” (ex post interview 2). In extreme cases, the practice was not even able to install the software. More commonly, a “series of cascading negative things [could] then lead to complete abandonment”.Whilst practice size was to some extent a proxy for capacity to innovate, the latter was also influenced by the practice’s governance structure , 40. In In larger organisations, several levels of governance were involved. In ACCHSs, for example, there were three tiers of decision-makers: “[The first tier is] community elected broad members, … the next tier is about senior management support for it, that’s the CEO and their senior level staff, and then the next tier is the providers or clinicians. …. We wouldn’t be able to work with any service without having all three of those processes in place” (ex post interview 2). Whilst strategic-level actors tended to make decisions on the basis of population disease burden and likely long-term benefit, operational-level actors appeared to be more concerned about short-term costs and workload implications and the factors discussed in domain 4.Larger GP practices required greater coordination and aligned governance structures to facilitate the organisational change that was necessary for adoption, and this depended on competing priorities and staff continuity . There was sometimes a mismatch of priorities between the “entrepreneur” GP , who made the decision to sign up for the trial and embraced the technology with enthusiasm, and other staff (fellow GPs and most practice managers) whose engagement was often much lower. As the TORPEDO researchers discovered, “when you sign on a [large] GP practice … usually agreed by the lead GP who may be enthusiastic about intervention, … it really needs all the GPs to be committed and want to use it” (ex post interview 3).HealthTracker, for example, if nurses rather than doctors undertook the risk assessment  could potentially “spend more time explaining to [patients] what it [the HealthTracker data] was all about, talking to them about lifestyle changes, their medication, why they need to be on them, how they could continue taking them and supporting them to do that” (ex post interview 4). Unfortunately, use of HealthTracker could not be easily incorporated into community health workers’ role in some ACCHSs for several reasons including lack of access privileges, low health worker confidence in use of computers, perceived time constraints, low GP confidence in health workers, and governance issues (“they weren’t given the green light by the head of the board”—ex post interview 1).Larger practices had a more diverse and distributed workforce. Potentially, this could reduce the cost of adoption of n the UK ). But laHealthTracker during the TORPEDO trial and subsequent real-world implementation. This was partly for cost reasons  and partly because of concerns that too much external support would limit the external validity of the findings. For these reasons, TORPEDO researchers decided to implement the intervention in a more or less standardised way.Variation in capacity to innovate  raised HealthTracker in participating practices: organisational mission and history , organisational leadership (e.g. ability to energise staff), team environment (e.g. extent to which team members with different skill sets worked in complementary ways), and technical features of the tool (covered in domain 2)  approached it like a guideline, … and missed the point that we weren’t trying to create a new guideline; we were trying to implement existing guidelines” (ex post interview 2).For example, they sought to maximise the chance that professional bodies supported and endorsed its use: “We made a decision very early on that that we would just use [existing] guidelines, whether or not we agreed with the guidelines” (ex post interview 3). This strategy was based on the assumption that if the guidelines emanated from professional societies, most physicians would accept them as reasonable. They had anticipated a potential scale-up platform through the Royal Australian College of General Practitioners (RACGP) and had selected the technology developer because of its existing relationship with RACGP . However, RACGP subsequently discontinued this partnership because of negative feedback from its members, especially in relation to the tool slowing down practice systems. Even though RACGP had a long history of endorsing clinical practice guidelines, they did not endorse HealthTracker on the Medicare Benefits Schedule (MBS), the government-subsidised health services, given Australia’s fee-for-service remuneration model for GPs. But this approach stalled initially: “we put in a submission to the federal government only to be told eventually that from a legislative viewpoint, MBS items can’t be attached to software” (ex post interview 3).By targeting an institutional level higher than professional organisations (i.e. government), the TORPEDO team sought to alter the rules that govern recognition and reimbursement of the use of software in delivering health services more broadly. The team had initially sought to list the use of HealthTracker and similar software, the chances of widespread adoption and scale-up of HealthTracker are probably limited.PHNs have the mandate to facilitate quality improvement programmes as part of their work, with dedicated staff to support that work, though such programmes do not tend to be focused on particular technologies. The TORPEDO researchers hoped to use the results of the trial “to drive the decision-making process a little bit more rationally” (ex post interview 2). This was particularly important at the time, given the absence in Australia of other quality incentives to promote proactive care for people at risk of cardiovascular disease. Without such incentive programmes, or the ability to bill patients or insurers for using The TORPEDO researchers built inter-organisational communication and networking into the study design. It is well established that complex innovation in healthcare is facilitated when different organisations communicate with one another, share experiences, and resources, and progress a shared vision of what they are collectively trying to achieve—perhaps using the quality improvement collaborative model . As Dixothere’s always the challenges of the RCT design, the side of you that you sort of test fixed ingredients or pills, and you don’t change things, adapt things over time. So I think if we had a different kind of design in evaluating this, it would have been more of that kind of cyclical adaptation over time, constantly reiterating, modifying our intervention, potentially taking it into different areas as we started to build a sort of community of practice, and I think all of those things are as important—ex post interview 2The Australian Primary Care Collaborative (APCC) had been established in 2005; it involved over 4000 health professionals from over 2000 services across the country, with a principal goal of improving access and chronic disease care . This inHealthTracker, and the organisations seeking to support its use, had evolved and adapted over time. As noted in the previous section, a desire to keep the intervention fixed to meet the standards of the RCT design existed in tension with the need to make local adaptations to improve its embedding.The TORPEDO study began in 2008, so this analysis allowed us to assess how HealthTracker only sporadically tended to forget the content of the training. Some practices found that it was necessary to retain “someone on the ground who is familiar with the tool inside out and with the IT infrastructure, who can coordinate with the developers” (ex post interview 1). Such support implies a recurrent cost, to be borne by Primary Health Networks or GP practices . Another factor that reduced sustainability of HealthTracker was the limited ability of the software vendor to respond technical difficulties by adapting the technology. It took around 2 years after the TORPEDO implementation study ended for them to release the next generation of the software (which GPs claimed still had “bugs”). This lack of agility had a negative impact on adoption. The TORPEDO team subsequently moved the development of HealthTracker in-house to a technology spinoff of their host research institute.One challenge for practices was maintaining staff skills in the face of high turnover or flagging commitment. GPs who used HealthTracker with other technical systems (see domain 2) was viewed by TORPEDO researchers as problematic in the context of more integrated clinical workflows within primary care and a national policy decision to increase interoperability between primary, secondary, and tertiary care. Some researchers felt that to make the technology more sustainable, it would need to develop the functionality to exchange information between systems rather than simply calculate and visualise risk. They considered that unless HealthTracker becomes fully integrated into the electronic record, it will inevitably have to compete with other third-party add-ons, as “ … one player in a very congested space, competing for that crowded real estate on the GP’s screen” (ex post interview 2). To address this challenge, researchers suggested expanding the number of conditions for which HealthTracker could be used: “if HealthTracker is … for just one condition, you might get a few people to use for a little while, but … if it could be developed for a whole range of interventions that might be sustainable … [for example if it had] multiple uses … like a Swiss Army Knife, … so that it looked the same and did similar things” (ex post interview 4). The counter-argument is that additional functionality would increase both technical and operational complexity and likely generate new problems elsewhere in the system.The limited interoperability of HealthTracker) was made to create item numbers around performing risk assessment and management. A similar submission was recently also made to the Medical Services Advisory Committee, which advises the Australian government on which new medical services should receive public funding. As of April 2019, interim MBS items (to be reviewed over the next 2 years) have been introduced to allow GPs and non-specialist physicians to conduct a heart health check that lasts at least 20 min. This recent development has potential to shift the value proposition (see domain 4) for HealthTracker to make GPs’ use of the technology worthwhile.An opportunity recently emerged to adjust financial incentives. In 2018, the entire Medicare Benefits System programme was undergoing a review (commenced in 2015), and an application for listing . They also recognised that technologies generally do not have universal appeal: “Some people would [be interested], some people might not, it’s the same as almost any other thing, some practices have a spirometer and some don’t” (ex post interview 5). And that if the choice on whether to adopt HealthTracker (or not) was left to individual GPs or GP practices, uptake would likely be slow, because GPs may only realise that the technology was helpful after they had started using it. Purchase by GP practices or Primary Health Networks in such a scenario would depend on price and competing third-party software.Whist TORPEDO researchers were upbeat about the potential for increasing uptake of HealthTracker in the future. First, it is possible that Primary Health Networks will start to provide significant direct support to GP practices to implement quality improvement initiatives, though HealthTracker may or may not be prioritised in this move. Second, with the growth of corporatised GP practice chains, more practices will have key staff such as a practice manager, IT lead, and quality improvement lead. But as the TORPEDO team found in their experience with larger GP practices, buy-in from the CEO of such corporatised practices does not guarantee that front-line clinicians will use the tool.Two changing features of the governance structure of Australian general practice may influence adoption of Another potentially positive development on the horizon is policy support for new digital health initiatives. Whilst Australia has included digital health in strategic documents since around 2005, in 2017, the first National Digital Health Strategy was released. It named several relevant goals to be achieved by 2022: (1) digitally enabled care models to improve accessibility, quality, safety, and efficiency of care; (2) workforce confidently using digital health technologies; and (3) high-quality data with a common understood meaning that can be used with confidence . HoweverHealthTracker, both locally and nationally, will unfold in the future.In sum, whilst there are some positive trends, there remains a high degree of uncertainty about how the fortunes of HealthTracker’s varied and partial uptake. Before listing these, it is worth noting that whilst there were undoubtedly some weaknesses in the original TORPEDO studies, it is striking how many strengths were built into the design and implementation. The technology was developed through extensive co-design; the programme had strong leadership and clear goals; much effort was made to recruit practices working in areas where unmet need was high, and considerable support was provided to practices to set up the technology, train staff in its use, and support a collaborative approach to quality improvement. Despite these strengths, TORPEDO has, to date, had only a limited impact on patient outcomes. Below, we summarise our findings.This ex post evaluation has identified a number of interacting explanations for HealthTracker had some significant software design flaws—e.g. it presupposed a level of technical infrastructure that some organisations did not possess. The value proposition for the technology’s vendor depended on widespread uptake across primary care providers, but because of the prevailing fee-for-service funding model in Australian general practice and lack of specific quality incentives for preventive care, there were costs associated with using HealthTracker for GP practices—for example, it required more time than was funded through Medicare standard consultations. Some GPs resisted using HealthTracker because its guideline-based recommendations conflicted with informally shared assumptions (mindlines) about best practice, and in such circumstances, design features (e.g. red lights) generated psychic costs [Cardiovascular risk is strongly influenced by social determinants and often coexists with comorbidities and entrenched lifestyle patterns; a technology designed to support rational decision-making based on epidemiological risk models may not appeal to many patients. ic costs .HealthTracker into business as usual. The wider institutional environment , whilst not entirely adverse, was not sufficiently aligned and did not provide specific incentives, and inter-organisational networking occurred only to a limited extent. Most of these influences appear set to continue to pose challenges in the future, though recent realignments of financial incentives may positively influence the value proposition for GP practices.Limited capacity to innovate , mismatch of commitment between those signing the organisation up to the study and those who would be responsible for delivering on it, mismatch between implementation strategy (which was standardised) and widely varying capacity and governance structure of GP practices, and underestimating the work of implementation helped explain why some organisations were unable to fully integrate determine particular behaviours or outcomes. Rather, technologies exert their influence  by becoming part of a dynamic network of people and other technologies which generates particular activities in particular contexts. Only when—and to the extent that—the “ensemble” of technologies-plus-people-in-wider-context comes together optimally will target patient groups actually receive better care and expect better outcomes [This study has also shown that the NASSS framework can be applied retrospectively to produce a new theorisation of a historical dataset which extends rather than replaces research and evaluations undertaken at the time. In particular, NASSS was built on the assumption that implementation of technologies in healthcare tends to follow the logic of complex systems , 20. Theoutcomes , 45.HealthTracker was designed by enthusiasts for evidence-based preventive care. Implicit in the software were assumptions—perhaps unintended and also unjustified—about the clinician (assumed to be a GP committed to following guidelines) and the patient . This systems lens also revealed that once a technology is installed in an organisation, there exists a greater or lesser potential to adapt and accommodate it. HealthTracker, for example, might have been better accommodated in Aboriginal Community Controlled Health Centres by creatively extending its use to community health workers who had ongoing relationships with patients and understood their cultural contexts (as opposed to restricting its use to temporary GPs who did not). This phenomenon  is critxibility  and for No previous studies have applied NASSS in an ex post analysis. The findings from this study resonate closely with our own and others’ application of NASSS in the empirical evaluation of health technology projects in the UK –23.Dixon-Woods et al. applied a different theoretical lens to explain the success of the US Keystone Project  and the Dixon-Woods et al.’s emphasis on the social and behavioural aspects of the intended change is captured in domain 2 of the NASSS framework  and also domain 5 . The TORPEDO study had included little in the way of behavioural intervention because the research team were cautious about providing too much support since the resource implications would then make the intervention unscalable. Another finding from Dixon-Wood et al.’s analysis of Keystone was the importance of nurturing a culture of commitment to quality improvement. This was captured in domain 5 of the NASSS framework as part of the work to support change; in TORPEDO, maintaining such a culture was something of an uphill struggle in the absence of specific financial incentives.Finally, Dixon-Woods’ finding that harnessing performance data as a “disciplinary force” and the use of “hard-edged formal accountability” are reflected in domain 6 of the NASSS framework as external (regulatory) influences on the system. In TORPEDO, a major motivator for many GPs was the peer-ranked performance portal described above, but the accountability was not “hard-edged”, since TORPEDO was run as a research study on collegiate lines, not as a policy must-do. The comparison with the Keystone Project highlights the tricky trade-offs that must be made in RCTs of complex interventions between undertaking a theoretically “robust” RCT and taking steps to maximise real-world success.The NASSS framework has proved useful in understanding how and why a technology-enabled quality improvement intervention generated mixed outcomes. Earlier evaluations of the programme, including a randomised controlled trial , processThe limitations of using the NASSS framework as an ex post analytic tool are similar to using any retrospective approach to undertake research. Apart from the narratives of long-standing research staff , the dataset already exists and cannot be extended with new, real-time data. In a large, longitudinal study such as TORPEDO, material that could have enhanced a system-wider analysis might have been inadvertently discarded at the time by researchers operating a more deterministic paradigm.The NASSS framework, originally developed to explain the fortunes of health technology projects in real time, can be applied retrospectively to generate a rich, contextualised narrative of a technology-supported change effort and the numerous interacting influences on its successes, failures, and unexpected events. A NASSS-informed ex post analysis, drawing on the principles of complex systems, can supplement earlier contemporaneous evaluations to uncover emergent interactions and interdependencies that were not fully knowable or predictable at the time.Whilst it is widely recognised that technology implementation in healthcare requires a judicious mix of “top-down” , “bottomWe are currently extending the NASSS framework alongside a complexity assessment tool (CAT) for use as an ex ante tool for planning, managing, and evaluating complex technology projects in health and social care. Further details of the NASSS-CAT tool are available from the corresponding author.


Despite the fact that Butler’s philosophical approach to understanding gender has many resonances with a large body of gender research being conducted by social psychologists, little theorizing and research within experimental social psychology has drawn directly on Butler’s ideas. In this paper, we will discuss how Butler’s ideas can add to experimental social psychologists’ understanding of gender. We describe the Butler’s ideas from Gender Trouble and discuss the ways in which they fit with current conceptualizations of gender in experimental social psychology. We then propose a series of new research questions that arise from this integration of Butler’s work and the social psychological literature. Finally, we suggest a number of concrete ways in which experimental social psychologists can incorporate notions of gender performativity and gender trouble into the ways in which they research gender.A quarter of a century ago, philosopher Judith  This repeated performance of gender is also performative, that is, it creates the idea of gender itself, as well as the illusion of two natural, essential sexes. In other words, rather than being women or men, individuals act as women and men, thereby creating the categories of women and men. Moreover, they face clear negative consequences if they fail to do their gender right.A quarter of a century ago, philosopher Judith We argue that Butler’s philosophical approach to under standing gender has many resonances with, and implications for, a large body of gender research being conducted by social psychologists. Indeed, Butler’s notion of performativity echoes a range of social psychological approaches to gender and gender difference. What we social psychologists might call gender norms and stereotypes e.g., , or gendgender trouble by making genders that fall outside of the binary visible, in order to disrupt binary, heteronormative views of gender within and outside of psychology. We are not the first to point out the relevance of Butler’s work to social psychology. Gender Trouble to social psychologists. While experimental social psychological perspectives on gender have been concerned primarily with the origin and perpetuation of gender stereotypes, Butler’s work is more political in her explicit call to create gender trouble. The political nature of the work is perhaps one reason why experimental social psychologists have been reluctant to build on and integrate Butler’s ideas in their work – but, we would argue, it is indeed one of the reasons they should. Combining these two perspectives seems potentially fruitful, bringing together Butler’s theorizing and her call for social and political change with established experimental social psychological theory and empirically testable hypotheses.Despite past acknowledgments of the importance of Butler’s work by social psychologists, in particular by qualitative psychologist, to our knowledge, little theorizing and research within experimental (and quantitative) social psychology has directly drawn on Butler’s ideas. This is despite the fact that there are identifiable similarities in broad theoretical ideas espoused by many social psychologists with an interest in gender and Butler’s ideas. Thus, we argue that there is great value in (again) promoting the ideas Butler puts forward in In this paper we will first describe Butler’s work in more detail. We will then discuss the extent to which her work fits with different conceptualizations of gender in the social psychological literature, with a focus on experimental social psychology. We will then propose new avenues of research that could potentially grow out of an integration of Butler’s work into social psychology. Finally, we will discuss the different ways in which Butler’s work can inform and challenge the ways in which we, as experimental social psychologists, study and operationalize gender.Gender TroubleIn her book before culture and before interpretation) – and gender as its cultural interpretation. She argues that it is not just gender that is culturally constructed and has prescriptive and proscriptive qualities, but that this also applies to sex as a binary category. Through this, This societal view of gender is also reflected in the works of many feminist writers, who define sex as biological and gender as cultural , and steps are taken to “rectify” this abnormality. For example, the majority of babies born with intersex characteristics undergo surgery and are raised as either male or female , protectTo be clear, Butler does not argue that biological processes do not exist or do not affect differences in hormones or anatomy. Rather, she argues that bodies do not exist outside of cultural interpretation and that this interpretation results in over-simplified, binary views of sex. In other words, biological processes do not themselves result in two “natural,” distinct, and meaningful, categories of people. The two sexes only appear natural, obvious, and important to us because of the gendered world in which we live. More specifically, the repeated performance of two polar, opposite genders makes the existence of two natural, inherent, pre-discursive sexes seem plausible. In other words, Butler views gender as a performance in which we repeatedly engage and which creates the illusion of binary sex. She argues:“Because there is neither an ‘essence’ that gender expresses or externalizes nor an objective ideal to which gender aspires; because gender is not a fact, the various acts of gender create the idea of gender, and without those acts, there would be no gender at all. Gender is, thus, a construction that regularly conceals its genesis. The tacit collective agreement to perform, produce, and sustain discrete and polar genders as cultural fictions is obscured by the credibility of its own production. The authors of gender become entranced by their own fictions whereby the construction compels one’s belief in its necessity and naturalness.” (p. 522)performative. The term performativity, originating in Thus, for Butler, gender is neither essential nor biologically determined, but rather it is created by its own performance and hence it is However, as this binary performance of gender is almost ubiquitous, its performative nature is concealed. The binary performance of gender is further reinforced by the reactions of others to those who fail to adhere to gender norms. Butler argues that “Discrete genders are part of what ‘humanizes’ individuals within contemporary culture; indeed, those who fail to do their gender right are regularly punished” (p. 522). This punishment includes the oppression of women and the stigmatization and marginalization of those who violate the gender binary, either by disrupting the presumed link between sex and gender  or between sex and sexuality  or by challenging the binary system in itself . This stigma is clearly evidenced by the high rate of violence against transgender women, particularly those of color ; surgeriprohibitive , repressing deviating gender performance, as well as generative , creating binary, heteronormative gender performance.These negative reactions and the binary performance of gender, Butler argues, do not exist by chance. Instead, they serve as tools of a system of power structures which is trying to reproduce and sustain itself – namely a patriarchal system of compulsory heterosexuality in which women serve as a means of reproduction to men, as their mothers and wives. These power structures are both is, but rather something one does or performs, Butler argues that gender identity is not based on some inner truth, but instead a by-product of repeated gender performance. Framing gender identity as an inherent part of the self, as many feminist writers did at the time (and indeed still do), she argues, reinforces the gender binary and in turn plays into the hands of the patriarchy and compulsory heterosexuality. Feminists should instead seek to understand how the category of “women” is produced and restrained by the means through which social change is sought .Butler’s work is a call to action to overthrow these structures and end the problematic practices that they engender. However, she criticizes feminist voices who emphasize a shared identity (“women”) to motivate collective action on behalf of the group in order to achieve societal changes. By arguing that gender is not something one cis-gender2 children  engendering new research questions which have not yet been investigated empirically, and (b) challenging our way of studying gender itself.Butler’s work is purely theoretical and thus many of her ideas have not been tested empirically, particularly using an experimental approach. Perhaps the most central question that can be examined by social psychologists is whether creating “gender trouble” by subverting ideas about sex, gender, and sexual desire, can indeed lead to changes in binary views of sex and gender and the proscriptive and prescriptive stereotypes that come with these views. Based on predictions derived from social role theory , we woulOn the other hand, as gender identity is very central to the self-image of many people , challentrans individuals or drag performers. Butler discusses drag as an example of gender trouble in detail, quoting the anthropologist Importantly, this investigation should go beyond examining reactions to women and men who behave in counter-stereotypical ways, such as women in leadership positions or stay-at-home fathers, and include a focus on more radical challenges to the gender binary such as non-binary and Another potential line of research to pursue would be to build on the discursive literature by examining the performative nature of gender from an experimental social psychological perspective, testing how gender is created through speech and behavior. Drawing on some of the findings from qualitative psychological research discussed in the previous section might be helpful in developing predictions and quantitatively testable hypotheses.Finally, if gender trouble is indeed effective in challenging binary, essentialist views of sex and gender, it is worth investigating how disruptive gender performance can be encouraged and used as a means of collective action. The literature on collective action to achieve gender equality has often drawn on (gender) identity-based ideas of mobilization e.g., . As outlany gender can (a) be motivated to engage in collective action to achieve gender equality generally, and (b) be motivated to engage in gender trouble and disrupt binary notions of gender as a form of collective action.More recently, psychologists have argued that it might be more effective to focus on “feminist” (rather than gender) ideologies which acknowledge, rather than ignore, issues of intersectionality see , and to In addition to new research question, Butler’s work also highlights the need for different methodological approaches to gender in experimental social psychology, and indeed there is much that could be learnt from those that work in the discursive tradition. There is also the potential for gender researchers to engage in gender trouble themselves by changing the way in which they treat gender.For the most part, experimental psychologists have tended to examine gender as a predictor or independent variable – examining gender differences in all manner of social, cognitive, and clinical measures e.g., . Indeed,performance of gender, that is, treating gender as an outcome or dependent variable. If experimental social psychologists are to contribute to gender trouble, we should shift our views away from sex and gender as causes for behavior and psychological outcomes . Instead, we should treat gender – whether measured as an identity, in terms of self-stereotyping, as simple self-categorization – as a result of societal and psychological forces. Rather than asking what sex and gender can explain, we need to look at what explains sex and gender.However, to our knowledge, there is very little quantitative or experimental research, that looks at the psychological processes implicated in the is a man or a woman and remains so over the course of one’s life. If, however, we view gender as a performance, then we must also view gender as an act, a behavior, which changes depending on context and audience. Asking participants to tick a box to indicate one’s gender – as many of us often do in our research practices – is an overly simplistic measure and cannot capture the nuances of doing gender. It is neither informative nor, we would argue, terribly interesting. Instead, one could measure gender identity salience and importance or gender performance – for example measuring gender stereotypical behavior or other types of gendered self-stereotyping  or allow open responses. We would also suggest not using the option “other” in addition to “male” and “female” as it can be perceived as stigmatizing. Similarly, if asking about sex rather than gender, at least a third option  should be provided , this overall produces a picture that erases variation and reinforces the idea that there are two opposing genders with clear boundaries. As experimental social psychologists with an interest in gender, we need to do better. Similarly, our theories themselves should allow for a fluid understanding of gender which also takes issues of intersectionality – with sexual orientation, but also with race, class, and other social categories – into account.3. While the use of the gender-neutral singular “they” is often frowned upon and deemed grammatically incorrect (their unicorn in my stable”) and to members of mixed-gender groups .Finally, when we talk about gender, we should do so in a way that makes gender diversity visible rather than way that marginalizes non-binary gender further. For example, replacing binary phrases such as “he or she” with gender-neutral ones such as “they” or ones that highlight non-binary gender such as “he, she, or they” or “he, she, or ze”ncorrect , it has ncorrect . DespiteThe use of new pronouns such as “ze,” specifically developed to refer to people outside of the binary, might be more effortful and equally controversial. However, evidence from Sweden, where the gender-neutral pronoun “hen” has become more widely used since the publication a children’s book using only “hen” instead of “han” (he) and “hon” (her) in 2012, indicates that attitudes toward its use have shifted dramatically from predominantly negative to predominantly positive in a very short amount of time . As gendIn this paper we put forward suggestions for ways in which Judith’s We believe that, as experimental social psychologists, we should be aware that we may inadvertently and performatively reinforce the gender binary in the way in which we do research – in the theories we develop, in the measures that we use, and in the research practices we undertake. By taking on board Butler’s ideas into social psychology, we can broaden our research agenda – raising and answering questions of how social change can be achieved. We can provide a greater understanding of the psychological processes involved in creating gender trouble, and in resisting gender trouble – but above all, we are in a position to create our own gender trouble.The first author of this paper uses they/them/their pronouns, the second author uses she/her/hers pronouns.TM and MR jointly developed the ideas in the paper. TM wrote the paper. MR read the paper and provided feedback on several drafts of the paper.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.


Magnuson et al. : MMLSH, MMLSH describe simulations with the interactive-activation model TRACE . Since words differ in frequency, however, priors are available. The task is then to compute the posterior probability of the words as the product of the likelihood and prior . This is how Shortlist B . Parameters in TRACE can be adjusted to avoid these negative effects, but, as McClelland et al.  showed, (2)MMLSH argue that, on a count of nodes and connections, models with activation feedback are simpler than those without it. TRACE actually performs very badly in such a count because of massive reduplication of nodes over time slices Norris, ; this isAlso on parsimony, MMLSH misinterpret NMC00's: “Information flow from word processing to these earlier stages is not required by the logic of speech recognition and cannot replace the necessary flow of information from sounds to words. Thus it could only be included […] as an additional component” . MMLSH curiously read “not required by logic” as “illogical” . An accurate reading of “not required by logic” is, of course, “not necessary”, and this is the central point about parsimony: additional components should only be added if it is strictly necessary to do so. MMSLH do not address this point.(3)Crucial behavioral evidence is inconsistent with activation feedback (McQueen et al., Theoretical arguments and the available empirical data thus indicate that activation feedback is not necessary in on-line speech recognition. Indeed, activation feedback is unable to improve the already optimal performance of any Bayesian feedforward model.All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.