During the last decade, the public healthcare sector has had to deal with increased competition, a growing influence of patient associations, and a necessity to deliver health services more efficiently and effectively. Despite recognising the patient participant’s role as a critical stakeholder in value creation, there is a limited body of research on the influence and power of patient participants. This article focuses on regional health improvement collaboratives that aim to develop coordinated, multi-stakeholder solutions to their healthcare cost and quality problems. They meet regularly and include health professionals, health insurance providers, and patient participants. In this article, we explore the relationships between these stakeholders and patient participants’ interpersonal dimensions regarding empowerment and valuable collaboration. Data were collected through stakeholder observations during meetings of three regional health improvement collaboratives, as well as through semi-structured interviews with the patient participants involved in these cases. Results show that patient participants can be empowered on a personal level. However, this does not imply that patient participants are empowered within the group dynamics. Interpersonal relationships constitute a crucial hidden aspect of building trust. More dialogue and inquiry are needed to examine how patient engagement is enacted and positioned within healthcare collaboratives. Healthcare costs are rising, and customers are demanding more personalised as well as better care . Market-Studies describe the importance of patient empowerment. However, there is a lack of knowledge about ‘how’ this empowerment can be achieved ,4,5,6. FThis study analyses patient participation in regional health improvement collaboratives (RHIC), focusing on the power/empowerment and the interrelation dynamics of patient participation. RHICs have three defining characteristics: first, an RHIC is a non-profit organisation in a specific geographic region of the country. Second, it is governed by a multi-stakeholder board composed of healthcare providers, health insurance providers or governments, and health consumers. Third, an RHIC helps the stakeholders identify opportunities to improve health and healthcare within the community and facilitates the planning and implementation of strategies for addressing those opportunities. An RHIC provides a neutral, trusted mechanism through which all key healthcare stakeholders in the community—physicians, hospitals, health plans, employers, and patients—develop coordinated, multi-stakeholder solutions to their healthcare cost and quality problems . In the The remainder of the article is as follows: In the past, customers, i.e., patients, were considered passive recipients of complex services . Today, Patient participation can be interpreted as a form of stakeholder involvement. Stakeholder theory draws on four social sciences—sociology, economics, politics, and ethics—to inform how organisations interact with stakeholders . In the In healthcare, the concept of co-production, co-creation, and collaboration  is receiving increasing attention. Co-production of value replaces the “old” ideas of professional dominance and paternalism, conceiving the patient as a co-creator of value rather than a consumer of health services ,17. HoweLegnick-Hall  identifiMutual trust is also a necessary condition in a successful value co-creation process. Trust increases when one party has confidence in a supply chain partner’s reliability and integrity . It is aEquity is the third critical condition to value co-creation. Adams  posited,Power is a central element in supply chain relationships. A frequent concern in management is how to gain and use power with other parties in the supply chain . DjellouGarnishing insights into the mediating role of individual empowerment in patient participation is the crux of this research. Psychological empowerment is a term introduced by Zimmermann  in theirIn their seminal contribution, Cattaneo and Chapman  illustraHowever, the distribution of power influences relationship dynamics, and ultimately, the empowerment of participants. Specifically, power inequities are likely to affect trust development, a vital aspect of relationships . In healAlthough RHICs are seen as a possible mechanism for developing coordinated, multi-stakeholder collaboration to co-create value , the literature indicates that achieving valuable collaborations in a context of power inequities is complex. This tension is what drives our research aim: to better understand the empowerment (or lack thereof) of patient participants in healthcare-related collaboratives and how this is related to the ability of the collaborative to co-create value.Understanding the interaction of interrelational dynamic elements  with individual empowerment  in collaborative contexts lies at the heart of this research , an independent network organisation dedicated to the interests of all (vulnerable) citizens in the province of Limburg. As the specific focus of our study is on power/empowerment and the interrelation dynamics of patient participation, RHICs were selected that have existed for more than two years, are fully functional, and are committed to the Triple Aim Initiative. The Triple Aim Initiative strives to: improve patients’ care experience ; improve populations’ health; and reduce the per capita cost of healthcare (or at least lower the rate of increase). At least one financer, one health professional, a project manager, and a patient participant should be involved in each RHIC. RHICs with other compositions, new RHICs, or RHICs not committed to the triple aim initiative were excluded from this study. Five RHICs met all of these inclusion and exclusion criteria, all of which were approached for our study. One RHIC was discontinued before the study commences, and one did not give permission to conduct observations or interviews. The remaining three RHICs gave permission for observations and interviews.The RHIC represented in Case 1 is a mental health foundation that meets once a year to make procurement decisions for regional care. The RHIC represented in Case 2 aims to improve health in a small region in the North of Limburg. At the time of the observations, they were working on qualitative pharmaceutical care at a lower cost and started a project to substitute more expensive statins (which help lower cholesterol in the blood) with generic medicine. Case 3 is an RHIC funded as a pioneering project. Stakeholder representatives run pioneer sites. These pioneer sites seek the best organisational and management structures to work with and explore new forms of funding (such as shared savings). Case 3 is the longest-running collaborative in the sample . Case 1 is an established collaborative but not as long-running as Case 3 . Case 2 is the newest collaborative in the sample . Data were collected by performing observations of meetings and conducting semi-structured interviews. Participants were asked for consent to be observed, interviewed, or both (video/audio recording of each). The written consent was stored. The observations occurred after desk research and contact with the project managers (telephone or online meetings). We conducted our observations during regular, scheduled meetings of the involved RHICs. These meetings typically lasted between 65 and 100 min and were filmed, followed by detailed verbatim transcription of spoken words and non-verbal signs. Field notes were also taken. The filmed observations were also studied without sound. Subsequently, the patient participants attending the meeting were interviewed. Each patient participant is an experienced expert, between 35 and 65, who participates voluntarily, without payment. The interviews were conducted in person and lasted between 45 and 60 min. The interviews were semi-structured, following a guide based on the literature review. Participant observation is a specific approach to gaining knowledge based on direct contact between the researcher and the social objects of interest. In this kind of observation, it is important to be unbiased and focus on directly observable events . TherefoOperationalised constructs for the observations and interviews were derived from the literature and geared towards exploring the relationships between stakeholders in RHICs, as well as patient participants’ interpersonal dimensions regarding empowerment and collaboration. As discussed above, information symmetry, mutual trust, and equity are crucial to gain power and co-create value in collaborative contexts. Given the power differences among actors in healthcare organisations, mechanisms should be in place to minimise inequity . These mSelf-efficacy, knowledge, and competence are vital elements of individual empowerment. Locke and Latham  identifiA coding and pattern-matching process was used as an analysis technique. Coding helps to interpret the data and to connect the pieces of information . Coding The pattern that emerged was compared with relevant literature. Every effort was made to ensure this study’s validity and reliability . A transIn this section, we describe evidence of individual empowerment (or lack thereof) and the group dynamics occurring in each of the three RHICs based on the observations and interviews. In Case 1, stakeholders sought information from other members, and the perceived self-efficacy of the patient participant was high, as PP1 stated, “I am the best candidate” and “This work has given my recovery a boost because I have to look at my illness with a helicopter view”. PP1 took a class and has a colleague patient participant whom he consults if necessary: “When I have questions about procedures or rules, I often call X”. PP1 would like to see other patient participants take a more active role: “I will try to motivate other PP to participate in this group”.At the beginning of one of the observed meetings, one of the stakeholders asked for advice about a topic irrelevant to this project, asking, “Do you know the * checklist?” . Three other members confirmed. “Is it a useful tool? I have heard negative comments at a symposium” (participation). Another member wanted to know how family counselling is organised in the mentioned organisation. A group participant explained this with help from a third member. Thus, information symmetries and exchanges were present in this group. The insurance company members showed interest in the opinion of the patient participant: “Can you tell us how this is done in this organization? (…) Thank you very much for your explanation” (explanation). Regarding company-related information, the insurance company members only shared information after detailed questions and remained rather superficial when they asked, “Are you satisfied with the use of e-health in your organization?” They added, “We are extremely interested in the patient’s opinion” . Unfortunately, no explanation followed why they are interested .Non-verbal signs informed us about interdependency and transparency related to mutual trust. Interdependency in this RHIC is underpinned by a need for information, which only the patient participant can deliver. Positive non-verbal signs outweighed the negative ones by far for this collective. For example, a favourable sitting position, which is active, open, and turned towards the speaker, was noticed twelve times. Opposing sitting positions such as hanging on the chair or aggressive (arms tight-folded) were found less frequently. Arms fast-folded were seen in the patient participant arguing about an important topic. After approving nods from other stakeholders and smiles, the patient participant leaned back in his chair again. In the interview, the patient participant said that he felt capable of the job, well educated, and prepared but noticed he also noted feelings of insecurity and the need to be reassured, asking, “Did I do well during the meeting?” .Trust is also noticeable, indicating that the patient participant is supported. The participant spoke fast and hesitated at the start of the meeting, and then the health professional helped him/her by summarising and checking with the patient participant whether the summarisation was correct. The other members supported him/her by active listening and positing, with faces turned towards the speaker and many confirming nods. As a result, the patient participant became more confident (observed by speaking at tempo and in a more relaxed sitting position). The patient participant said in the interview, “During the first meetings, I often had to ask for explanations, but I felt confident to do so” (belief in oneself).The long existence of this supply chain, the open atmosphere, and the shared goal to improve health in this organisation added to the individual stakeholders’ unique knowledge and gave credence to value creation in this supply chain. Notable, however, is the remark of one of the participants of the health insurance company. She discussed her long relationship in this network with her company’s board and obtained the answer “Nice, but we do not want you to become too familiar”. This remark makes the equity in this RHIC unclear. To what extent does the health insurance company desire fairness or equity in its relationship with the other stakeholders?In Case 2, information symmetry was observed. All participants understood each other, and when asked for clarification, were provided an explanation. “Stakeholder… gives information about the benefits of the chosen medicine and explains the decision tree for this kind of medication” (explanation). During the lengthy meeting, medical terms needed to be explained only twice, indicating that health literacy was not an issue in this group. PP2 prepared in advance for the meeting, saying “The documents arrive on time and when I have questions, I will get them answered before the meeting”.Nevertheless, patient participation is new in this RHIC. The patient participant describes it as a “fight for my position” (action). Interpersonal relationships differ in this group; a few stakeholders worked together on another project with opposite interests. This tension is noticed during the observations. The noted non-verbal signs confirmed this: six positive and seventy-eight negative non-verbal signs were recorded, such as looking at a mobile phone, turning red, and not looking at the speaker. During the speaking time of one of the stakeholders, another member checked his mobile phone, and the person speaking looked down. Noted are four or five key speakers who talk most of the time, plus interruptions of speakers. Compared with the other observed cases, more negative use of voice aspects were noticed in this RHIC: five negatives and two positives. One of the members emphasised four times that “It is important to make the right choices at the start of the project”.Contrary to Cases 1 and 3, the word “you” was used frequently in this RHIC, referring to other stakeholder groups in this RHIC. For example, “It would be best if you, health professionals, did not only care about the cure; budget control is also one of your tasks”. The use of “we” and “you” indicates tension between the different stakeholder groups. Non-verbal signs observed such as turning red, crossing arms, or taking an upright sitting position confirmed this tension. In the interview, the patient participant said, “This network can improve through better communication”  and additionally said, “More appreciation would be welcome” (sense making). PP2 stated that “It [referring to Patient and Public Involvement] has not been done before, and everything that happens here needs patients’ involvement”. The patient participant believed in the capability to execute the tasks and believed that she was individually empowered: “They embraced my proposal, so yes, I have influence”.Stakeholders are prepared when the meeting starts and are willing to execute tasks. They are motivated to make this RHIC successful, shown by the passionate way of arguing , but there is frustration over only achieving two subprojects. Subproject three had to be rewritten during the three years due to lower priorities set by insurance companies. Two pharmacists said, “It is a pity that the original subproject three had to be changed”, and another stakeholder said, “I, also, am not happy that the original plan has been changed. This project has been tough, but that is inherent in such projects” (other member behaviour). The project leader confirmed this, saying, “Yes, it is a pity, and it is good that we have changed it to another type of medication so that we can bring measurable results” (project manager).Interesting is the remark of the patient participant: “We, as patient participants, belong in this healthcare supply chain, and that is where we are used for, but sometimes we are also used to reduce the tension between members”. She also said “Patient participation is new in this group”, and, “In this project, patient participation started hesitantly. I had to convince the members of my added value. It is my task to seduce the other stakeholders to look with the eyes of a patient” (action). This struggle with her position is also observed; the frequency of the project manager or other stakeholders interrupting her is significantly higher than the frequency that other stakeholders were interrupted. Thus, in this RHIC, the stakeholders struggle with their role and position and the patient participant’s role and position. Furthermore, problematic interpersonal relationships play an inhibitory role in mutual trust building. Equity and mutual trust are imbalanced in this group.In Case 3, information symmetry and equity are balanced. There is a willingness to share information, e.g., “I would like to inform you about a personnel change in my department” (participation). “All three external parties need to be present at the upcoming meeting. Is party three an important one?” Another member answers, “No, with parties 1 and 2, you cover the central area.” “Do we agree to invite parties 1 and 2?” (participation). Everybody nods or says yes. Part of the meeting time is devoted to sharing an example of a not-appropriate functioning healthcare supply chain and comparing it with this project. This group scored the highest amount (180) of coded participation items. PP3 stated, “I gained understanding and knowledge about how things work in healthcare (…) If you are willing to search for information eventually you will find it”.There is group spirit, and the patient participant feels welcome and appreciated. The patient participant noticed that “The commitment of the members and their willingness to see the need to change, despite own interests that conflict with it, makes this network successful” (motivation). Observations confirm this: before the meeting starts, all participants greet each other and exchange informal information. Case 3 is the only observed RHIC where members choose a seat randomly and do not take the seat next to their professional colleagues. PP3 confirms, saying “I have noticed that the members listen to me”.In this group, a professional also participates on behalf of patients. She is the representative of a public organisation stakeholder in this RHIC. She asks for background information and reacts, saying that “It is a pity that only professionals are interviewed and no public or patients” (explanation) and asking “would it be wise to speak with these people individually?” (participation). Patient and professional patient participants actively participate in the meeting and highlight the perspective of the patient but are not explicitly asked to give more information about topics.Despite the recognised importance of healthcare supply chains to create value and the patient participants’ role as critical stakeholders in this context, there is a limited body of research on how the collaboration of stakeholders in RHICs is conducted. This study aimed to fill this gap by analysing individual power/empowerment and interrelational dynamics in RHICs. The most notable result of our study is that when comparing the interview data with the data obtained from the observations, all patient participants reported feeling empowered in interviews, but there was mixed evidence from observed group dynamics observed. Secondly, the data from the observed cases reveal that group dynamics in collaboratives might go through phases of maturity, and interpersonal dynamics can evolve as this occurs. In this section, we discuss the individual empowerment of patient participants in collaboratives, the interrelational dynamics of such collaboratives, and the influences of the maturity/context of collaboratives on these factors.Individual empowerment is vital for patient participants to collaborate and co-create. Mann et al.  stated tDespite the self-reported empowerment of patient participants, we noticed tensions and discrepancies between the self-perceived role and individual empowerment and the observed group dynamics during the RHIC meetings. Patient participants faced the challenge of defending their status as potential change agents within the healthcare system. Ewert and Evers  describeMutual trust and equity differed between stakeholders in the observed cases as reciprocal relationships differed. A strong trust relationship between health professionals and patient participants was found in Case 1. In Case 2, tensions were noticed between health professionals and members of the health insurance company. Additionally, trust between patient participants, health professionals, and insurance company members was also weak. In Case 3, both relationships had a solid trust base. These findings are similar to those of Groenewegen, Hansen, and De Jong , who fouBrett et al.  stated tTrust and power are inextricably linked, and one must be considered in the presence of the other . Theory Tokenism can occur when participants of diverse backgrounds collaborate and do not understand each other . CriticaSufficient time is essential for collaborations to successfully mature. The RHIC in Case 2, existing for a relatively brief time (two years), seemed to struggle with collaboration. Case 3, as a pioneer site, had more time to develop relationships and participatory methods. They decided to continue the partnership after a successful period as a pioneering project. This aligns with Allen et al.  who emphAnother major influence on the success of collaboratives is the context in which they are operating. In Case 2, one subgoal was not achieved due to changed politics , and some stakeholders in the group felt frustrated. As a result, group cohesion was negatively impacted. Watson and Foster-Fischman  stated tIn this study, we found that a patient participant needs to be individually empowered to collaborate with other stakeholders and achieve co-creation. The patient participant must be capable, representable, and qualified to accomplish this individual empowerment. To achieve this, the establishment of a patient participant profile is beneficial. Adequate funding, interaction time, and trust with patient participants can help negotiate roles, balance power, and lead to meaningful patient collaboration. We highlighted trust and equity’s importance in developing meaningful collaboration. We observed that interrelational dynamics and interpersonal relationships are vital for establishing mutual trust, and when there are unsolved issues between stakeholders, mutual trust building is complex. To avoid tokenistic patient participation in RHICs, stakeholders must be willing to build sustainable relationships, have the intention to be genuinely engaged, and be open to mutual and reciprocal learning. Furthermore, we found a discrepancy between perception and observations. Therefore, future research should thoroughly describe the context to convey more clearly what works, why it works, and who created what value.This study provides important insight into our understanding of patient participation in RHICs. However, we acknowledge that there are limitations to this study. First, because the number of cases we observed is limited, the three cases described in this study cannot represent all RHICs, and this study needs to be replicated with a more substantial number of cases.Second, although we observed some challenges and tensions in collaborative processes, the observed cases are relatively positive examples of co-creation. For several reasons, some RHICs did not want to participate in our study. Some of these initiatives had been put on hold or indicated that participants did not feel comfortable being observed. Researching the dark side of patient participation and collaboration and the consequences of failed co-creation would be interesting.Third, although the community plays a role in value co-creation in regional health improvement collaboratives, we did not investigate the role of macro-level actors, such as the policy decision-makers of the regional healthcare system or the local government. Therefore, future research is needed to examine the influence of community and government policy on adopting and implementing co-creation initiatives in RHICs.To summarise, this paper contributes to understanding the role and influence of patient participation in RHICs. Despite the uncertainties, as described in this article, and a fragile evidence base, patient engagement has become widely accepted. Considering these gaps, it would be hard to disagree that patient engagement represents a critical democratic and ethical practice in modern healthcare. Patient participation in RHICs can address the complex challenges of creating responsive and actionable healthcare evidence . Success


End Stage Renal Disease (ESRD) patients undergoing hemodialysis are faced with serious problems in their lives. Hope, as a multifaceted factor, plays a critical role in these patients’ lives. Given the multifaceted process of hope, this study aimed to describe hope and identify the challenges, strategies, and outcomes of hope in Iranian ESRD patients undergoing hemodialysis.This is a qualitative study using content analysis. The participants were selected using purposive sampling. The data were collected using deep, semi-structured interviews with 14 participants; it continued until reaching data saturation. Graneheim and Lundman content analysis approach was used to analyze the data.Five main categories and twenty-two subcategories emerged; the categories consisted of (1) Hope described as a particular event to happen, (2) Opportunities and threats to achieve hope, (3) Negative emotions as barriers to achieve hope, (4) Positive coping strategies to achieve hope, and (5) Growth and excellence as the outcomes of hope.Based on the findings, ESRD patients undergoing hemodialysis described hope as a positive feeling of expectation and desire for a special thing to happen. They faced threats and opportunities to achieve hope, which exposed them to negative emotions as barriers of hope. Thus, they make use of positive coping strategies to achieve hope. Moreover, hope led to growth and excellence. Through awareness of hope, definition and strategies to achieve it, and teaching them, physicians and nurses working in hemodialysis wards can enhance hope in patients.The online version contains supplementary material available at 10.1186/s12882-023-03336-6. End Stage Renal Disease (ESRD) is a chronic and life-threatening disease, the prevalence of which is increasing around the world , 2. ESRDHope is a process and source of compatibility and adaptation with chronic diseases . It is aIn a study, it was indicated that family function predicted hope in hemodialysis patients . As mentThis research was conducted using a qualitative design and content analysis approach. Content analysis focuses on the lived experience, interpretations, and meanings encountered by individuals. Content analysis is usually used in the design of studies the goal of which is to explain a phenomenon. This type of design, unlike the directed method, is often suitable when the existing theories or research literature about the phenomenon under study are limited. In this case, researchers avoid using preconceived categories and instead arrange for the categories to emerge from the data .The study was performed in hemodialysis centers in Nemazi and Shahid Dr. Faghihi hospitals affiliated to Shiraz University of Medical Sciences in Fars province in Southern Iran.The study participants included 14 males and females over 18 years of age selected through purposive sampling. The inclusion criteria for the patients were speaking and understanding the Persian language, experience of dialysis for at least 6 months, and age over 18 years. Patients in the acute phase of ESRD were excluded from the study.The study data were collected through semi-structured interviews. Observation and field notes were utilized as well. The total data collection procedure lasted for two months. The interviews were performed in the Persian language by the first author, who was experienced in qualitative studies. The interviews started with the following open questions: “How do you describe hope in your life?”, “How does hemodialysis and renal failure affect your hope?”, “What factors are facilitators and barriers of hope in your life?”, “What circumstances or conditions help you to increase or decrease your hope?”, “What would be the outcomes of achieving hope?”, and “How does achieving hope affect your life?” Then, probing questions were asked, and the patients’ explanations were continued (see supplementary file). These face-to-face interviews were individually conducted at the patients’ bedside during hemodialysis. They lasted for 30–50 min. The interview was done with 14 hemodialysis patients. Two patients were interviewed twice. Therefore, 16 interviews were used for data analysis. All the patients who had inclusion criteria and were invited to the study accepted to participate in the study. Data collection continued until we reached data saturation. All the conducted interviews with the participants were recorded and immediately transcribed verbatim after the end of the interview sessions. In fact, data analysis was done in parallel with data collection by all three researchers. For this purpose, first, the interview of each participant was analyzed and then the next interview was conducted.The data were analyzed based on the 5 steps of Graneheim and Lundman’s method , includiIn order to confirm the results and determine the study rigor, we used Lincoln and Guba’s (1985) four criteria, namely credibility, dependability, conformability, and transferability . To gainThis study was conducted on 14 hemodialysis patients. There were both male and female patients with the mean age of 32.5 years. Other demographic characteristics of participating patients are summarized in Table The hemodialysis patients described hope as having positive thoughts, expectation and desire for future events; having goal-oriented thoughts; developing strategies to achieve goals; being motivated to expand effort to achieve goals; and having family support in future.“To me, hope means thinking about positive happenings (p 11), thinking about future, ignoring the renal failure and seeing the future clearly…. I describe hope as having good dreams (p 6)”.Having positive thoughts was reported by two participants . “I describe hope as being healthy after hemodialysis, coming back home after hemodialysis, being recipient of kidney transplantation, being a healthy person, and having a good life without any disease complication (p 3)”.Expectation and desire for future events was expressed by participant 3. “To me hope means having goals in the life (p 4)”. “I think about buying a house and getting a job in future in spite of having renal failure. I wait until my children go to the university, and get married (p 14)”. “I think hope is being successful in my family functions, job, social relationship and life in spite of renal failure and undergoing hemodialysis. In other words, I describe hope as being successful in education and university, and renal failure and hemodialysis do not affect my life (p 6)”.The patients described hope as having goal-oriented thoughts, developing strategies to achieve goals, and being motivated to expand efforts to achieve goals. In this regard, participants 4, 6, 5, 14 said “To me, hope means having a family that understand, love, care, and accept me with all my disease limitations (p 11). I express hope as living with my husband and children at the same home (p 13).Having family support in future was described by the hemodialysis patients as the meaning of hope. Perception of opportunities and threats to achieve hope was extracted from the participants’ experiences. On one hand, the patients faced such threats as physical challenges, challenges in social interactions, educational failure, economic challenges, occupational restrictions, and familial conflicts to achieve hope. On the other hand, they pointed to the negative and positive effects of the disease and hemodialysis. Furthermore, spirituality alongside the positive impacts of treatment provided a great opportunity for facing the threats.“When my kidney failed, I turned pale; there was swelling and edema under my eyes down to my cheeks, and I had pain all over my body. From then on, I haven’t been able to sleep or sit; I suffer from shortness of breath; these may affect my hope” … (p 14); “We undergo dialysis 3 times a week. This causes our physical strength to decrease. I became very weak and do not drink water and liquids as much as we want. I feel nauseous and I have no appetite…, I think these impact my hope” (p 1).The hemodialysis patients have physical challenges including skin and nail change, fatigue, weakness, bone pain, difficulty sleeping, muscle cramps, nausea, drowsiness, etc. that affect the patients’ hope. Patients have facial changes such as paleness, swelling under the eyes, spots on the face, changes in the appearance color, and, in other words, a sickly face. In addition, the smell of urea can come out of their mouths. Sometimes, the accumulation of toxic substances in the body causes nausea and anorexia in the patient. In this regard, two participants  expressed Hemodialysis clients are limited in their social activities due to dependence on hemodialysis machines 2 or 3 times a week. Moreover, because of ESRD, they have no physical energy, and are not interested in communicating with others and participating in social activities. Based on the patients’ reports, these challenges in social interactions my influence the patients’ hope. Statements of the participants in this regard are presented below:“We have restrictions on commuting; for example, there must be a dialysis center in the place we intend to visit, so that it is close to us in terms of location…” (p 10); “As much as we want, we don’t have free time to be with friends and relatives. It means that due to our condition and dialysis, we cannot be together andachieve hope; that’s why, our communication has decreased…” (p2).Inability to continue their education and studies and physical and mental problems in the classroom are threats to the lives of ESRD patients undergoing hemodialysis and affect the patients’ hope. They stated:I was accepted in the university. I was not able to study. The disease ruined all my thoughts; I had no motivation and did not have patience for this work. I had no energy…, so how I can be hopeful?” (p 4); “I really could not go to class. You know, I had dialysis in the morning three days a week. I had to go to class in the evening, and I really could not sit in class. Every time I went, I was so sleepy and dull that I can’t tell…” (p13).“The cost of going to and from the dialysis center and the existence of financial problems to follow the treatment processes are some of the economic challenges. Two participants  stated:I sold my house and shop. In these 4 years that I have been ESRD, I spent all my family money on treatment and doctor’s fees …. These led to loss of hope during these years.” (p 8); “I was at work for 12 years when I was transplanted, but when I got dialysis, I said I can’t come, and I am unemployed now….” (p 3).Being limited in work for patients, having to engage in certain tasks, being unable to work, giving up work at the onset of illness, and being impatient to finish work threatened the hemodialysis patients. Two participants’ expressions  are as follows:“We cannot choose any job. Many of us do not go to work after we get dialysis…” (p12); “We are really having trouble finding a job. We are all involved in the hospital and tests, but our conditions do not allow us to find a job…” (p 11).Since my illness was diagnosed, my wife has gradually moved away from me; we don’t have the same interest and love as before… this bothers me…you know, my family is my hope” (p 10); “I have two children. Now that I go to dialysis three days a week, I really can’t see my children…” (p 1).These patients have difficulty choosing a spouse. They face family rejection, family disputes, and inability to fulfill family roles. These may affect the patients’ hope. In this regard, one of the participants said: “Five participants  believed in a supreme power called God as the source of hope. They believed in the purposefulness of creation of living things, were thankful to God, and believed that the disease had come from God, and God would help them. In this regard, two participants  mentioned:“God wanted me to become sick. I know that the disease has come from God. God wanted to tell me to be careful. I think the disease is an examination sent by God…” (p 5); “I know that every problem and pain that happen to me, are from God. I say, God, I will accept whatever you give me. I say, God, you are right. If you have faith in God, you will find faith in yourself, and you will find purpose and hope in your life…” (p 9).Depression, sadness, uncertainty, anger, and hate were the main concerns of the patients that were barriers to achieve hope. These patients were isolated, felt lonely, cried, and talked about tendency towards death. They were also unhappy with their physical and social limitations. Moreover, they were not certain about future and disease complications. Some patients were nervous, lost their temper easily, and showed aggressive behaviors. They said that these negative emotions were the barriers of hope. Besides, some of them hated the treatment process.These patients showed signs and symptoms of depression. They felt tired and energy less. They lost their interest and pleasure in most or all-normal activities. The patients also reported tearfulness and hopelessness. They felt worthless and had frequent thoughts of death.Two participants  stated:“The disease ruined my motivation and my boredom. I got tired of the disease, treatment, and its side effects…” (p 13); “The disease had a bad effect on my mood. I will not live anymore. There is no hope for me to survive. I should get a kidney and be transplanted. . I do not want to undergo dialysis anymore. In the end, I am seeing death…” (p 10).Patients were upset about starting dialysis to treat the disease, not being with their family members during the week, and not being able to fulfill some of their tasks and roles for their children.A participant said in this regard:On days of dialysis, I can’t go to a party with my family. Well, they certainly like us to be together, but I cannot. I get upset and sad…you know, it leads to loss of my hope” (p 12).“Seven participants mentioned their uncertainty about whether they can get a transplant, as well as the complications associated with a transplant. One of them said:“I am undergoing dialysis. I am also a transplant candidate. I see people who were transplanted, and it did not work… Now they are undergoing dialysis. I tell myself that this problem should not happen to me. This uncertainty is frustrating…and leads to loss of my hope” (P1).Patients have become nervous since the onset of the disease, very quickly become frantic due to a simple shock, and show aggressive behavior. In addition, hatred towards dialysis and the equipment of the dialysis department was evident as another negative emotional reaction. One of them said:“I hate dialysis and the equipment of the dialysis department. When I want to go for dialysis, I get very upset that day…” (p 3).Positive coping strategies refer to the behaviors shown by eight participants for solving their concerns in the process of hope. In fact, the participants used positive and solution-oriented strategies, stayed motivated, utilized positive psychological constructs, had supportive exchanges, and were connected to transcendence to cope with negative emotions and achieve hope.The patients with ESRD tried to have purpose in their lives, modify their goals, and make use of positive and solution-oriented strategies to cope with their negative emotions and are hopeful. In this regard, one of the participants stated:“I am counting the days to reach my goal. When a person has a goal in his/her life, he/she becomes hopeful. When there is hope in life, life becomes orderly…” (p 2).The patients with ESRD made genuine attempts to stay motivated by using constructive approaches such as trying, listening to music, grooming themselves, avoiding negative memories, and taking part in recreational activities to overcome negative emotions and achieving hope. In this regard, two participants stated:“When I think about my kidney, to avoid getting depressed, I stand in front of a mirror. I comb my hair, change my clothes, and wear perfume…” (p 7); “When I go to a recreational place with my family or friends, I get a lot of energy… my motivation and hope to continue life increase…” (p 9).To cope with negative emotions and achieve hope, the participants used positive psychological constructs including positive thinking, having positive expectations, cheering themselves up, comforting themselves, and being thankful. Statements of the participants in this regard are presented below:“See! I always try to see the glass half full. I tell myself that the disease is there, and I have to think positively about it…” (p 6); “I compare myself with other diseases such as cancer and thalassemia. Thalassemia patients have many appearance and physical problems. I console myself and say, thank God, there is dialysis. And we hope…” (p 14).Holistic support was found to be a strong strategy for coping with the disease and achieving hope. This included emotional, informational, instrumental, and spiritual support. Two patients’ expressions  are as follows:“In this world, you can only count on your family…Your family gives you hope…Your family always stays by your side…”(p 9); “Whenever I have a question, I ask the hemodialysis center; there is also a counseling room here… I ask a lot of questions, and this helps me to reduce my problems …” (P 11).Connection to transcendence was another strategy used by the patients to achieve hope. This was done through talking about God, praying, having hope in God, and complaining to God. In this regard, three participants  stated:“I asked God to heal me. I easily talk to God…” (p 9); “I am a person who prays and even attend religious ceremonies. Well, this gives me peace. You know that God always cares about His servants… With this disease, my relationship with God has increased…” (p 5); “Now that I have this disease, I am closer to God… There is not a day that I do not wake up and call the name of God… “(p 10).Using the above-mentioned positive coping strategies to improve hope and eliminate negative emotions led to growth and excellence. In other words, hope resulted in the improvement of well-being and finding the meaning in life.Utilization of positive coping strategies led to optimism, happiness, vitality, full energy, tranquility, health, and better social relationships with others, promotion of self-efficacy, and disease acceptance. In other words, using these strategies improved the patients’ well-being.In this regard, two participants  stated:“Hope gives you positive energy. It gives you a feeling of happiness and increases your energy. When you are hopeful, you laugh; you think about good things; you have good dreams in your mind, positive dreams for future, and good feeling; and you feel relaxed…” (p 14); “Hope keeps you physically and mentally healthy. It reduces the disease symptoms and pain. Hope leads to living, earning money, working, everything…I’m sure that I’ll be successful in my life and in my job in future…” (p 7).Utilization of positive coping strategies helped the patients with ESRD find meaning in their lives. In other words, they became purposeful, found order in their lives, and pursued their goals. They were interested in life, recognized themselves, achieved faith in God, understood life, and were able to care for themselves. In this regard, two participants  stated:Maybe, I didn’t think about life very deeply before I got sick…but now that I’m sick and I’m having a hard time, I appreciate life more. I just understand how sweet life can be despite the hardships. Life seems to have a different meaning to me than when I was healthy… “ (p 1); “I know that a person should be comfortable and hopeful for himself… these are necessary to endure problems; I don’t know that maybe this disease will bother us more than what it is now… but I am hopeful about the future and have a happy ending. It makes life sweeter for us…” (p 7).“The results of this study indicate that hope in hemodialysis patients means having positive thoughts, expectation and desire for future events and goal-oriented thoughts; developing strategies to achieve goals; being motivated to expand effort to achieve goals; and having family support in future. In a qualitative study, hope was reported that as “lively, active, and, above all, a personal choice”. Hope was the desired possibility; however, it needs an effort to be achieved .The findings of the present study showed that threats to achieve hope in hemodialysis patients consisted of physical challenges, challenges in social interactions, educational failure, economic challenges, occupational restrictions, and familial conflicts. Although no qualitative study have reported the factors that affect hope in hemodialysis patients, a qualitative study showed that mental well-being and depression were associated to hope in patients who have undergone dialysis . The resDespite these threats, the patients studied revealed opportunities like spirituality, which were effective in the process of hope. Five participants believed in a supreme power called God as the source of hope. In general, Muslim patients consider diseases as the divine destiny that remove their sins. In line with the current study results, other studies demonstrated that spiritual experiences increased hope and had a positive effect on the reduction of stress , 26, 27.The findings of this study showed that the patients who were undergoing hemodialysis encountered negative emotions including depression, sadness, uncertainty, anger, and hate as barriers to achieve hope. A qualitative study on end-of-life patients reported that “limitations imposed by illness, feelings of anguish and helplessness and poor communication with clinicians” were barriers of hope . A quantPositive coping strategies were another finding of the study to achieve hope. Hemodialysis patients used positive solution-oriented strategies, positive psychological constructs, and supportive exchanges; connection to transcendence; and motivation to achieve hope. In line with our study, a qualitative research on end-of-life patients reported that “supportive others, positive thinking and sense of humor, connection with nature, faith in religion and science, and a sense of compassion with others and altruism” were facilitators of hope . The resSupportive exchanges were also the strategies used for overcoming negative emotions and developing hope in hemodialysis patients. In various studies, the role of social support and its effect on improving hope have been proven, which is consistent with the present study , 14, 35.In the current study, growth and excellence were the other prominent strategies related to hope in patients under hemodialysis. In other words, hope resulted in the promotion of the patients’ well-being, which helped them find meaning in their lives. In various studies, in line with the current research, the use of positive coping strategies by patients with ESRD under hemodialysis leads to happiness, vitality, disease acceptance, and well-being , 37. AnoWe used qualitative content analysis to examine the patients’ experiences of hope. Using a design such as phenomenology can help to better understand the patients’ lived experience of this concept. In addition, the lack of similar studies inside the country limited the possibility of comparing the findings with local studies. For this reason, repeating similar studies can help clarify the concept and its effective factors.To the best of our knowledge, this is the first study which aimed to describe hope and identify the challenges, concerns, strategies, and outcomes associated with hope in Iranian ESRD patients undergoing hemodialysis. An in-depth interview with maximum diversity and the use of purposeful sampling was the main strength of this study. The qualitative approach can also help us to describe hope and deeply understand the factors related to hope in patients with ESRD who are under hemodialysis, which can be fundamental for designing nursing interventions.The results of this study indicate that hope in hemodialysis patients means having positive thoughts, expectation and desire for future events, and goal-oriented thoughts; developing strategies to achieve goals; being motivated to expand effort to achieve goals; and having family support in future. The study findings indicated that the patients with ESRD who were undergoing hemodialysis were faced with threats and opportunities to achieve hope. These caused them to develop negative emotions as barriers to achieve hope. In response to these concerns, they made use of positive and solution-oriented strategies, stayed motivated, used positive psychological constructs, had supportive exchanges, and resorted to transcendence to achieve hope. Being hopeful eventually led to growth and excellence. Therefore, there is a need for educational programs to be held with a focus on the use of active and constructive coping styles to increase hope. Thus, healthcare providers should support the patients to achieve coping strategies and in turn create hope and motivation to cope with stress and emphasize active coping in the daily life of these patients; also, the supporting role of the family is emphasized to remain hopeful to effectively deal with the disease and its treatment.Below is the link to the electronic supplementary material.Supplementary Material 1


With the widespread application of computer technology in engineering education, Online Judge (OJ) systems have become an important platform for programming teaching. OJ systems provide a platform for learners to practice programming skills, submit solutions, and receive feedback. They offer a conducive environment for learners to engage in hands-on coding exercises and enhance their programming abilities. This article explores the use of OJ systems as a software tool for enhancing programming education in engineering. It investigates how the difficulty and order of programming problems affect the users' behavior, performance, and cognitive load in OJ environments. The research data were sourced from Project_CodeNet. Using statistical methods, such as Spearman correlation analysis and differential analysis, the study reveals the factors that influence the users' submission situations, answer order, and learning outcomes. The findings provide useful implications for OJ system developers, teachers, and learners in designing, implementing, and using OJ systems for programming education in engineering. The study suggests that problem difficulty and order should be considered and adjusted according to the users' abilities and progress, to provide appropriate challenges and support, balance the cognitive load, and improve the programming skills of the users. OJ are web-based learning platforms that provide engineering students with a variety of programming exercises and tasks related to various programming concepts, help them apply their theoretical knowledge to practical programming problems, and thereby enhance their programming skills and proficiency. Consequently, the OJ platform has become a popular way for individuals to learn programming worldwide. The primary method for students to use OJ is by submitting programming assignments online. When students submit their programming assignments, the OJ system automatically compiles, runs, and evaluates their submitted code against predefined test cases, and provide assessment results The difficulty level of problems on the OJ platform affects students' learning behavior and outcomes substantially. OJ platforms typically offer a wide range of programming problems, each assigned with its own difficulty level, such as easy, medium, and hard. In some cases, problem difficulty is classified more precisely using difficulty levels like 1 or difficulty coefficients like 0.95. For students learning programming through OJ, completing overly simple problems may lack challenge and fail to effectively enhance their programming skills. Conversely, attempting excessively difficult problems can lead to strong feelings of frustration, thereby diminishing the effectiveness of learning. They may even demotivate students and result in learning abandonment. The difficulty level of problems and the submission information related to those problems  serve as crucial means for educators to understand students' learning behavior and outcomes In addition to instructional OJ platforms, competitive OJ platforms such as Codeforces and AtCoder have also gained widespread popularity. For programming learners, participating in programming competitions offers a valuable learning experience and serves as a common way to test programming abilities and improve skill levels. During competition, users' solving problems may exhibit certain order characteristics. They may be influenced by the order of problem arrangement and consciously solve the problems in a structured manner. Alternatively, they may be influenced by the difficulty of problems and consciously avoid overly challenging ones, prioritizing relatively easier problems. In programming competition, there is no direct association between the difficulty level of problems and their sequential appearance. This means that adjacent problems may have a significant difference in difficulty, and more difficult problems may appear earlier than simpler ones. Consequently, this phenomenon gives rise to variations in the arrangement of problem difficulty in competitions. Easy problems and difficult problems present different levels of challenges, potentially leading to varying levels of pressure and test-taking experiences for participants. Therefore, when the order of difficulty varies, users' performance in competition may be influenced.In competitive environments such as competitions and exams, the psychological factors of participants significantly influence their performance in answering problems RQ1. How does the difficulty of a problem affect its submission situation? In particular:• How does the difficulty of a problem affect the number of submissions it receives?• How does the difficulty of a problem affect the number of people who submit it?RQ2. What factors influence the answer order of users? Specifically, in the following aspects:• How does the order of problems influence the answer order of users?• How does the difficulty of problems influence the answer order of users?RQ3. How does the difficulty arrangement order of problems affect users' answer outcomes?RQ4. How does different difficulty arrangement order of problems lead to differences in cognitive load?The rest of the article is organized as follows: section 2This paper aims to explore how the difficulty level and the order of programming problems on OJ platforms affect students' learning behavior, performance, and cognitive load. In this section, we review the related works in the following aspects:In the field of computer-aided instruction and programming education, there have been numerous studies on educational data analysis and learning behavior patterns. Restrepo et al. In the application of educational data analysis, automated assessment and feedback are important research areas, and automated programming assignment assessment is a key focus. Caiza and Del Álamo Ramiro From the comprehensive review of the extant literature, it becomes apparent that while there have been myriad investigations into OJ systems and programming education, certain research lacunae persist. Current studies indicate that assignment difficulty affects students' submission behavior patterns 33.1In this study, four research questions are proposed. In RQ1, to investigate the influence of problem difficulty on submission situation, the data are planned to be grouped based on difficulty levels. All problems will be categorized into four groups for subsequent analysis. During the analysis, descriptive statistics will be performed on the data and, based on its characteristics, appropriate correlational analysis methods will be selected for exploration.In RQ2, to investigate the factors influencing answer sequence of users, 514,128 competition records are planned to be processed according to the definition of orderliness of the sequence  In this study, we used IBM's publicly available Project_CodeNet dataset . In such cases, the influence of problem difficulty on user problem selection becomes more pronounced. On the other hand, on the AIZU platform, users can submit problems at any time, and over time, even challenging problems receive more submissions, weakening the influence of difficulty on user problem selection. Lastly, we retained only the data from official competitions on AtCoder, as after the official competition ends, users can still participate in “Virtual Contests”. Participants in “Virtual Contests” have access to view other users' submitted code, which could potentially affect the accuracy of difficulty assessment. Therefore, in this study, we only kept the submission data from official AtCoder competitions. After preprocessing, the dataset consisted of 4,528,371 submission records. These preprocessing steps help us reduce the noise and heterogeneity in the data, and ensure the validity and reliability of our analysis.3.4Item Response Theory (IRT) is a classical cognitive diagnostic method (1) For each programming problem, calculate the average acceptance rate of all submissions as the probability of the answer being correct for that problem.p. To compute this probability, we use the average acceptance rate of all submissions for a given problem as the probability of the answer being correct for that problem. The Rasch model assumes binary results, i.e., “correct” or “incorrect”. In the AtCoder data collected from Project_CodeNet, the number of test cases used in each submission is not provided, and only the submissions marked as “Accepted” receive a score, while other states such as “Wrong Answer” or “Compile Error” do not receive a score and are penalized. Therefore, all submission states in AtCoder can be categorized as either “Accepted” (solution correct) or “Unaccepted” (solution incorrect), which satisfies the assumption of the Rasch model. Hence, the average acceptance rate of all submissions represents the probability p of the answer being correct for a given problem.In this study, the probability of the answer being correct is denoted as (2) By utilizing the probability of the answer being correct, calculated using the Rasch model in IRT, we can determine the difficulty parameter for each programming problem.diff in this study, is computed by the probability of the answer being correct (p). The calculation of diff can be determined using Equation diff) is restricted to -2 to 2. Any difficulty value exceeding 2 or falling below -2 is truncated to 2 or -2, respectively. A difficulty value closer to -2 indicates a lower difficulty level and higher likelihood of being answered correctly, while a value closer to 2 implies a higher difficulty level and lower probability of correct answers. The difficulty values obtained through Equation The difficulty parameter, denoted as IRT can be used not only to evaluate the difficulty of problems but also to estimate the ability levels of individuals based on their submission data. In this study, the specific steps for assessing the ability levels of participants using the Rasch model are as follows:cr.(1) Calculate the acceptance rate for each user's participation in programming competitions and compute the average acceptance rate as the user's correctness rate, denoted as cr) and apply the Rasch model in IRT to calculate the ability value for each user.(2) Utilize the user's correctness rate (cr) is transformed logarithmically to compute the ability value, referred to as ability in this study. The ability can be calculated using Equation The correctness rate  from being assigned to different groups. If two problems are not in the same group, it guarantees that their difficulty values are not excessively close. The information after grouping is presented in When studying the factors that influence the answer sequence of users, the research focuses on the answer sequence and difficulty sequence of users. Each problem in the competition is assigned an unique index starting from 1. The answer sequence is a sequence of problem indices representing the order in which the user answers the problems in the competition. The difficulty sequence is a sequence of relative difficulty values of the problems answered by the user in the competition . Among these subsequences, the longest length is defined as the maximum ordered length in this study. Thus, the sequence's orderliness can be determined using the following definition:Orderliness of the sequence: Let the length of the sequence be denoted as len_s, and the maximum ordered length of the sequence as len_o. When len_s>3, if len_o>len_s/2, the sequence is considered ordered; otherwise, it is considered unordered. When len_s = 3, if len_o = len_s, the sequence is considered ordered; otherwise, it is considered unordered. This definition applies to both answer sequences and difficulty sequences in calculating their orderliness.• When studying whether the arrangement of problem difficulty affects user answer outcomes, the focus is on the difficulty ordering of problems in a competition and the user's answer outcomes. User answer outcomes refer to the user's performance in the competition, measured in terms of acceptance rate. The acceptance rate is calculated as the ratio of the number of correctly answered problems to the total number of problems in the competition. Regarding the difficulty arrangement of problems in a competition, there are three possible scenarios:(1) Easy problems first, followed by difficult problems. In this case, the difficulty of the problems in the competition increases progressively. In this study, this arrangement is referred to as “easy to difficult”.(2) Difficult problems first, followed by easy problems. In this case, the difficulty of the problems in the competition decreases progressively. This arrangement is referred to as “difficult to easy”.(3) No apparent pattern or characteristic in the difficulty progression throughout the competition. In this case, the difficulty of the problems in the competition does not exhibit a clear trend. This arrangement is referred to as “no characteristic”.Based on this, the trend of difficulty progression from the first problem to the last problem can be calculated to determine the difficulty arrangement of the problems in the competition. Specifically, all the problem difficulties in the competition are treated as a sequential series of numbers. By performing linear regression on this series and calculating the slope of the regression equation, the trend of difficulty progression can be determined. To more accurately distinguish the “no characteristic” scenario from the other two scenarios with clear trends, a threshold of 0.05 is set in this study. Specifically, when the slope value is greater than or equal to 0.05, it indicates an increasing trend in difficulty, suggesting that the earlier problems in the competition have relatively lower difficulty values, while the later problems have relatively higher difficulty values, i.e., “easy to difficult”. When the slope value is less than or equal to -0.05, it indicates that the earlier problems have relatively higher difficulty values, while the later problems have relatively lower difficulty values, i.e., “difficult to easy”. When the absolute value of the slope is less than 0.05, it represents no clear characteristic in the arrangement of problems in the competition, i.e., “no characteristic”.44.14.1.1From The decrease in the number of participants may result from the increasing difficulty that discourages many users from attempting these challenging problems. Additionally, considering the context of the data being sourced from OJ competitions, participants aim to earn more points within a limited timeframe. Therefore, for excessively difficult problems, they may choose to abandon them and focus on relatively simpler problems where they can score points. However, the number of submissions does not decrease with the reduction in participants. Higher difficulty groups contribute the same number of submissions as lower difficulty groups with a larger number of participants. This indicates that more difficult problems have a higher average number of attempts per person. This phenomenon may be due to the fact that more challenging problems require more thinking and attempting to arrive at the correct solution. The decline in the average acceptance rate in These findings provide valuable insights for developers of OJ systems. When designing and developing educational software and online learning platforms, the influence of problem difficulty on users should be taken into consideration to avoid discouraging users by setting the difficulty level too high. Additionally, more challenging problems require more thinking and experimentation. This implies that in non-competitive scenarios, designers should provide users with sufficient time and opportunities to attempt problem-solving, rather than emphasizing rapid task completion or restricting the number of attempts. For excessively difficult problems, students need to have a solid understanding of fundamental concepts, employ effective methods and strategies, and make multiple attempts to tackle the problems. Therefore, in programming education, teachers can offer appropriate guidance and support to students when they encounter challenging tasks, helping them solve problems more effectively and overcome their fear of difficulties. Similarly, OJ platforms can provide suitable assistance and hints to users when they face difficult problems, boosting their confidence.4.1.2In examining the influence of problem difficulty on submission situation, this section employed Spearman correlation Analysis to calculate the correlation coefficients between the difficulty level and the number of submissions, as well as the difficulty level and the number of participants. The analysis results are presented in The Spearman correlation coefficient ranges from -1 to 1, where -1 indicates a perfect negative correlation, 1 indicates a perfect positive correlation, and 0 indicates no correlation. In the analysis results presented in The results of the Spearman correlation analysis confirm the relationship between problem difficulty and submission situation. Specifically, there is a slight negative correlation between problem difficulty and the number of submissions, although this relationship is very weak and can be considered negligible. On the other hand, there is a strong and highly significant negative correlation between problem difficulty and the number of participants. This correlation is statistically meaningful and further supports the findings in section 4.2In this section, the 514,128 competition records were processed according to the definition of orderliness of the sequence in section Due to the non-normal distribution of the data, this study employed the one-sample Wilcoxon signed-rank test to perform a statistical analysis, and the results are presented in These findings indicate that both the order of problems and their respective difficulty levels exert a significant influence on users' answer sequences. Furthermore, the order of problems has a more pronounced impact on users' answer sequences. In the course of addressing problems, the predominant pattern observed involves users attempting to answer problems sequentially. In instances where an insurmountable challenge is encountered, a majority of users opt to bypass such challenging problems temporarily, instead choosing to tackle those problems they can readily solve. This is evident from the higher z-value observed in the test for problem order. The magnitude of the z-value reflects the degree of orderliness in the data, with a larger z-value indicating a stronger degree of order. In the test for problem order, the z-value is 584.982, while in the test for difficulty order, the z-value is 150.191. The higher z-value in the test for problem order suggests a stronger influence of problem order on the user answer sequence. The vast majority of users opt for a sequential approach in answering questions. This is likely due to the fact that systematically completing tasks can aid them in tracking their progress more efficiently, preventing any confusion during reviews. Furthermore, when users encounter challenging questions, they often choose to address the simpler ones first and later circle back to the more demanding questions. Such a strategy might be rooted in considerations of efficiency, aiming to first secure easily attainable scores and subsequently allocate more time and effort to tackle the more challenging problems.4.34.3.1In order to investigate the influence of the arrangement of problem difficulty on user performance, we divided users into three categories based on their ability values using K-means clustering. Categories 1, 2, and 3 represent users with low, medium, and high ability levels respectively. Utilizing K-means clustering for grouping or classifying data is a common practice The F-test results showed significant differences in the means of the fields between different clusters (P<0.05), indicating that the clustering results are statistically meaningful and represent different user groups. This demonstrates the appropriateness of using the K-means method for user classification. The silhouette coefficient is 0.553, and the DBI index is 0.57, indicating a high quality of the classification results and moderate data distinctiveness between clusters, signifying a good classification. The CH index is relatively high, at 181297.7, indicating a good quality of the clustering results. Multiple runs of the K-means algorithm yielded stable classification results. Therefore, the classification of users based on their ability values using K-means clustering is statistically meaningful, with a good-quality classification and reliable stability.4.3.2For the “difficult to easy” and “no characteristic” data, the Cohen's d values for ACR_A, ACR_H, and ACR_M are all less than 0.2, indicating very small effect sizes. The Cohen's d value for ACR_L is 0.433, approaching 0.5, indicating a moderate effect size. However, regardless of whether users' ability levels are distinguished or not, the p-values for the “difficult to easy” and “no characteristic” data are greater than 0.05, indicating non-significant results. In other words, for users at any level, there is no significant difference in acceptance rates between the two difficulty ordering sequences of “difficult to easy” and “no characteristic”.For ACR_A, ACR_H, ACR_M, and ACR_L in both “easy to difficult” and “no characteristic” groups, the Cohen's d values exceed 0.8, indicating that the magnitude of differences is very large in all cases. Moreover, the p-values are less than 0.05, reaching a significant level. This implies a significant difference in acceptance rates between “easy to difficult” and “no characteristic” for users of any level. Furthermore, this difference has a consistent direction: when the difficulty arrangement order is “easy to difficult”, the acceptance rate is significantly higher than the acceptance rate when the ordering sequence has “no characteristic”.Similar to the “easy to difficult” and “no characteristic” data, there is also a significant magnitude of difference between the “easy to difficult” and “difficult to easy” data: the Cohen's d values for all groups exceed 1.3. The p-values are less than 0.05, which indicate the statistical significance results. Therefore, for users at any level, there is a significant difference in acceptance rates between the two difficulty arrangement order of “difficult to easy” and “easy to difficult”: when the difficulty arrangement order is “easy to difficult”, the user acceptance rate is significantly higher than the acceptance rate when the difficulty arrangement order is “difficult to easy”.Furthermore, through comparisons, it can be observed that the difference in acceptance rates between the “easy to difficult” data and the “difficult to easy” data is even larger compared to the differences between the “easy to difficult” data and the “no characteristic” data. These results indicate that when the difficulty arrangement order is “easy to difficult”, the user's acceptance rate is the highest, significantly differing from other difficulty arrangement order with a substantial effect size. When the difficulty ordering sequence is “difficult to easy” or “no characteristic”, the user's acceptance rate is significantly lower compared to the “easy to difficult” scenario. However, there is no significant difference in the acceptance rate between the “difficult to easy” data and the “no characteristic” data.Based on the findings of RQ2, users tend to prefer answering problems in an orderly manner according to the sequence of the problems. The ordering of problem difficulty significantly affects users' answer sequences. Combining these findings can assist teachers in implementing targeted assessments of students' programming skills. Consequently, in the field of computer-assisted instruction and programming education, teachers can dynamically adjust the difficulty and sequence of problems based on students' abilities and progress to accommodate their learning needs and provide appropriate challenges. This personalized learning path can enhance student engagement, learning effectiveness, and self-directed learning abilities. When designing programming exams, teachers can guide students by altering the ordering of problem difficulty, taking into account their tendency to answer in an orderly manner. Teachers may place easier problems at the beginning to stimulate students' enthusiasm and prevent premature abandonment or conduct targeted tests before important competitions or exams to boost students' confidence. Students should be cautious of the potential negative effects of their answering habits. If they encounter challenging problems early on, it is important to adjust their mindset promptly, avoiding misjudging the overall difficulty of the test based on the problem's position and consequently experiencing psychological pressure leading to poor performance.4.44.4.1Cognitive load Cognitive load can affect concentration, clarity of thought, and subsequently influence the effectiveness of answer. Based on the findings of RQ2, users are likely to answer problems in the order presented. Different difficulty arrangement order can result in variations in information processing demands and, consequently, differences in cognitive load. Cognitive load influences cognitive resources and thinking processes, and an increased cognitive load can lead to divided attention, increased difficulty in judgment, and an influence on answer outcomes. Therefore, we can make a reasonable assumption that different difficulty ordering sequences lead to differences in cognitive load, which in turn result in significant variations in answer outcomes.4.4.2Cognitive load is typically assessed through biological indicators , self-reports , or behavioral measures The time interval between users' submissions for different problems reflects the time consumption for thinking about the next problem after enduring cognitive load from the previous one. The longer the consumption, the greater the cognitive load. When users switch problems without being accepted the previous one, it indicates that the previous problem was difficult for the user. The more frequent these switches occur, the less the user is able to concentrate. The level of attentional concentration is a reflection of cognitive load, where decreased concentration implies a higher cognitive load. An increase in submission error rate suggests that individuals face certain obstacles or deficiencies in controlling and supervising cognitive problems. With an increase in cognitive load, individuals need to allocate more attentional resources to complete the cognitive problems, which can affect the control and supervision of problem details and increase the likelihood of submission errors. Therefore, to some extent, a higher error rate can reflect a higher cognitive load experienced by individuals. This study measures users' cognitive load using the three indicators mentioned. Lastly, the study performs min-max normalization on these three indicators and calculates their mean as the evaluation value for cognitive load in this research.Specifically, the statistical methods for the three indicators are as follows:• Time interval: If the current submission is for a different problem than the previous one, calculate the time interval and take the maximum value among all intervals. A larger value indicates a greater cognitive load.• Problem switching number: Calculate the number of times when user switches problems without solving the previous one. A higher value indicates a greater cognitive load.s_num, the number of problems being accepted is ac_num, and the error rate is er. The error rate can be calculated using Equation • Error rate: Let the number of submissions is 4.4.3To investigate whether different difficulty arrangement order of problems leads to differences in cognitive load experienced by users, this study conducted a differential analysis of cognitive load indicators under different difficulty arrangement order. The results of the analysis are presented in The results indicate that there are significant differences (P<0.05) in cognitive load indicators when the difficulty arrangement order of problems varies. When analyzing the data with difficulty arrangement order of “easy to difficult” and “no characteristic”, high-level users show a higher magnitude of cognitive load difference compared to other users. The Cohen's d value for high-level users reaches 0.444, which is close to 0.5, indicating a moderate effect. For other ability levels, the Cohen's d values are around 0.2, indicating smaller effects. This could be attributed to the confidence-boosting effect of the “easy to difficult” difficulty arrangement order, which is more effective for high-level users, significantly reducing their cognitive load.In the “difficult to easy” and “no characteristic” data, the Cohen's d value for the high-level user group is 0.207, indicating a relatively small effect size of cognitive load difference. The Cohen's d value for the median-level user group is 0.415, close to 0.5, indicating a moderate-to-low effect size. Only the low-level user group has a Cohen's d value exceeding 0.5, indicating a moderate effect size. In this group, there are noticeable differences in the magnitude of cognitive load across users at different ability levels.Among all the data group, the largest difference in magnitude is observed between the “easy to difficult” and “difficult to easy” group. For users at any proficiency level, the Cohen's d value exceeds 0.5, approaching 0.8, indicating a large effect size. Specifically, the high-level user group shows the smallest difference with a magnitude of 0.671, while the low-level user group exhibits the largest difference with a magnitude of 0.779. This suggests that regardless of learners' ability levels, the “difficult to easy” difficulty arrangement order has a significantly higher influence on cognitive load compared to the “easy to difficult” difficulty arrangement order, greatly influencing the learning experience and outcomes.As a result, we can observe that when the difficulty order of the problems is “easy to difficult”, users experience the lowest cognitive load, followed by the “no characteristic” order. When the difficulty order is “difficult to easy” the cognitive load is the highest, and the influence is more pronounced for users with lower ability levels. These findings provide insights for computer-assisted instruction, where relevant educational systems can consider these conclusions when designing material difficulty order and learning strategies tailored to learners of different abilities. In programming education, instructors should adopt appropriate task difficulty ordering methods for students with different abilities to reduce cognitive load and improve learning outcomes. In test design, teachers can influence students' cognitive load during the answering process by changing the order of problem difficulty, to some extent, affecting students' performance, boosting confidence, or preventing complacency.4.5When students are in the initial stages of learning, it is desirable for them to experience a sense of achievement promptly. Adjusting the difficulty arrangement order of problems to “easy to difficult” facilitates improved outcomes in problem-solving, thereby enjoying the learning process more. As students utilize OJ platforms for programming practice and skill enhancement, the difficulty level can be incrementally raised. Students can be provided with some challenges by setting some challenging problems. At the examination phase, differentiation in evaluating students' learning outcomes becomes crucial. Altering the problem sequence to “difficult to easy” or “no characteristic” can more effectively discern between students' ability levels.Additionally, teachers can make more specific adjustments based on their objectives. For instance, teachers can adjust the difficulty arrangement order of problems to help students achieve better results, thereby boosting their confidence ahead of significant examinations. They can also adjust the difficulty arrangement order to negatively impact students' performance in routine practice-oriented tests to a certain degree, preventing students from becoming overly proud.5In this study, we explored how the difficulty level and the order of programming problems on OJ platforms affect students' learning behavior, performance, and cognitive load. This study's findings offer pivotal insights for programming education and the conceptualization and enhancement of OJ systems and other digital instructional tools. Statistical analyses were deployed to scrutinize the impact of problem difficulty on submission situations, determinants that sway users' problem-solving sequence, and the nexus between difficult arrangement orders of problems with answer outcomes and cognitive load. The research data were sourced from Project_CodeNet. The outcomes revealed a pronounced influence of problem difficulty on submission situations. In contests, problems of heightened difficulty garnered fewer participants in contrast to simpler problems which drew a larger crowd. Nonetheless, the submission count remained relatively consistent between challenging and simpler problems, signifying more attempts were necessitated for the formidable problems. Such revelations can guide OJ system developers in crafting educational software and online educational platforms, prompting an acknowledgment of the bearing of problem difficulty on users. In both pedagogical and non-competitive contexts, educators and OJ system architects must ensure ample time and avenues for learners to tackle problem-solving, furnishing pertinent assistance during challenging phases.The sequence in which users tackle problems is shaped by both the order in which problems are presented and their difficulty. During competitions, participants frequently address problems sequentially. However, when confronted with particularly challenging problems beyond their grasp, they may deliberately bypass these and prioritize simpler ones. Notably, this investigation determined that the difficult arrangement orders of problems markedly affect users' problem-solving outcomes. When problems are arranged in ascending order of difficulty (“easy to difficult”), the users' acceptance rates are significantly higher compared to the arrangement orders of “no characteristic” or “difficult to easy”. These findings can assist teachers in implementing targeted assessments of students' programming skills, enabling them to adjust the arrangement orders of problem difficulty based on students' abilities and progress, in order to meet their learning needs and provide appropriate challenges.This study comprehensively evaluated behavioral indicators such as “time interval”, “problem switching number”, and “error rate” in programming competitions to analyze and study the cognitive load under different arrangements of problem difficulty. The results indicate that users of different ability levels experience varying degrees of cognitive load under different arrangements of problem difficulty. Significant differences exist in the cognitive load experienced by users under the three different arrangements, with slight variations in the magnitude of the differences. Overall, when problems are arranged in ascending order of difficulty (“easy to difficult”), users experience the lowest cognitive load. When problems are arranged in descending order of difficulty (“difficult to easy”), the cognitive load is highest, especially for users with lower ability levels. The “no characteristic” random arrangement falls between the two. These findings provide insights for computer-assisted teaching, and educational practitioners can refer to these conclusions to develop teaching plans that balance the cognitive load of students in learning and exams by adjusting the arrangement orders of problem difficulty in exercises and assessments, thereby facilitating better programming learning outcomes.The data for this study is derived from competitions, and currently, there is no source of data originating from educational contexts. Therefore, it's not possible to verify if the conclusions of this study are feasible in educational settings. Furthermore, the Project_CodeNet dataset only contains data on programming problems and lacks data on other types of problems. Consequently, the conclusions of this research are specifically for programming problems, and it's uncertain if they apply to other types of questions (such as objective questions or other types of subjective questions).Due to the limitations of the dataset, this study did not consider factors such as users' knowledge background, geographic location, gender, coding proficiency, and the knowledge domains of the problems. In future research, it would be beneficial to explore these aspects. For example, investigating users' submission patterns across different knowledge domains to examine whether there are significant differences in mastery levels among users of varying ability, as well as studying the differences in coding styles among users of different ability levels.Finally, the “Results and discussion” section discusses how the conclusions of this study can better guide teachers in using OJ to assist the overall learning process. However, whether these insights can truly be effective has not been explored through case studies, nor have relevant supporting literature been found. In future work, consideration will be given to conducting several related controlled experiments to verify whether these insights can genuinely take effect.Jinshui Wang: Conceptualization, Formal analysis, Funding acquisition, Methodology, Writing – review & editing. Pengchen Lin: Data curation, Formal analysis, Investigation, Methodology, Software, Writing – original draft, Writing – review & editing. Zhengyi Tang: Conceptualization, Formal analysis, Methodology, Writing – review & editing. Shuguang Chen: Software, Writing – original draft.The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.