The Evidence-informed Policy Network (EVIPNet) is one of the key mechanisms introduced by WHO to reduce the research-to-policy gap. EVIPNet Europe was launched in 2012. We evaluated the performance and achievements of EVIPNet Europe with the overall aims (1) to inform future developments and strategic planning of EVIPNet Europe and (2) to contribute to the evidence base for organisational knowledge translation activities by sharing the lessons learnt.The evaluation covered the WHO Secretariat of EVIPNet Europe and its 21 member countries, from its inception to mid-2018. A mixed methods design was used to assess changes in three domains, including triangulation of quantitative and qualitative methods, based on the EVIPNet Europe Monitoring & Evaluation framework and theory of change. Data were collected between August and October 2018. Data collection comprised documentary review, social media analysis, online country evaluation, key informant interviews and validated tools. Two case studies were also developed.The evaluation showed promising results as well as lessons to guide the future development of EVIPNet in the WHO European Region and other regions of the world. EVIPNet Europe appears to be filling a niche in promoting the capacity of Network member countries for evidence-informed policy-making. There is evidence that EVIPNet Europe’s capacity-building programme of work is improving knowledge and skills at the individual level. There has been an increase in activity and outputs since its establishment and evidence has been used to inform new policies in some member countries. However, the speed at which member countries are developing or publishing products varies greatly and no formalised knowledge translation platforms have yet been created. Financial and human resources are limited and staff turnover is a cause for concern, both at the WHO Secretariat and country team levels.Six years since the launch of EVIPNet Europe, the Network has grown quickly, is clearly valued and has had some successes. However, more work and support are needed if it is to achieve its vision of a Europe in which high-quality, context-sensitive evidence routinely informs health decision-making processes that ultimately serve to strengthen health outcomes across the Region. While more evidence now exists on how to translate the findings from research into healthcare practice, less evidence is available for the policy arena  that we have global events, global capacity-building events where these people can come together, that then depends on headquarters.” “There are certainly sustainability challenges, which are likely to be exacerbated by further growth, with many expressing that, at this point, activities would not be possible or would not continue without WHO funding. There are initial signs of Network member countries having successfully submitted funding proposals and/or seeking their own funding. However, not all member countries seem to have the skills to write business cases and applications for funding. Additional training by the WHO Secretariat could be useful. However, technical training was considered a more important priority, as is securing stable staffing at country level to enable sustainability.Despite this growth, the establishment of KTPs within EVIPNet Europe is still required. Although there are examples of positive developments, these have yet to materialise formally. This lack of institutionalisation of KTPs was also expressed by many respondents during the country online evaluation. Many felt this was the obvious next step and area for future development, both strategically and in terms of support and capacity development for member countries. It was acknowledged that institutionalisation was not an easy or quick process but necessary if this Network was to become sustainable and self-sufficient.So for me personally I think this is something that we have to tackle, because we are spending a lot of effort in the situation analysis … but for me personally not sufficient KTPs, concrete KTPs, have been established.” “EVIPNet Europe has established a structure, including a Secretariat with dedicated staff to coordinate and support the Network. There is also a regional and global steering group and governance mechanisms. However, many stakeholders were not aware of the accountability processes for EVIPNet Europe and these have not been emphasised in publicly available documents. There was widespread understanding from stakeholder interviews that M&E was an important but challenging aspect to providing accountability. However, the documentary review revealed that no country evaluations have been conducted to date. It is expected that the development of the new M&E framework will create a new momentum, including the development of an additional simple data collection/reporting methodology. Progress is captured to some extent in the published annual reports.The logical step-wise approach to and methodology for EIP of the EVIPNet Europe process was appreciated. Interviewees and survey respondents highlighted the same standardised process to be followed, including how and where to look for evidence, considering EIP stakeholders, cost effectiveness, harms and benefits, and producing policy options, having a policy dialogue, etc.… because you cannot disagree with the EVIPNet principles, it’s just so logical that things should be set up in this way.” “Despite the ability to adapt to the country context, stakeholders felt that the sequence in which countries develop KT products could be even further adapted to country-specific needs, such as starting off with rapid response mechanisms to provide opportunities for quicker wins and to demonstrate the benefits of EVIPNet Europe. This would also initiate the creation of a culture for EIP before embarking on concrete KT activities such as an evidence brief for policy and policy dialogue, if appropriate.Challenges reported in the online country evaluation included a lack of available research evidence within countries or lack of awareness of the sources of evidence, the attitudes of policy-makers and lack of a culture of evaluation. The online country evaluation also highlighted the lack of mandate for developing evidence briefs as a challenge; while some countries followed the process, others needed a ministerial order.The documentary analysis highlighted five support instruments and resources developed and published by the WHO Secretariat to date , 53–55. Yes, they were the basics for the work that we have done, so yeah, I mean I could speak for quite some time about each of them, but the general position is yes, each one of them has been very helpful.” “Using these EVIPNet support instruments, one situation analysis had been published  and seveSome stakeholders thought that the development of further instruments might be overwhelming, while others provided examples of additional tools that could be developed, including more support for developing a rapid response service, instruments for appraisal of qualitative evidence, citizen engagement and organisational management (planned for the near future). One area mentioned by several stakeholders as a possible gap was more information or support for institutionalising a KTP within countries, including writing business cases, and how to finance a KTP, ensuring available resources to bring in topic experts while retaining a core of EIP experts.Capacity-development workshops are an important part of the EVIPNet Europe approach. The documentary review revealed that a number of face-to-face and webinar sessions have been conducted by the WHO Secretariat and have grown over time. This includes training as part of multi-country and national workshops. Despite no standardised evaluation question being used, average participant scores for these sessions are high. Interviews also highlighted the value of these capacity-building activities, in both the provision of technical skills and support for practical applications, not just knowledge, which were felt to be missing from other similar programmes. Mostly, it was changes in the skills and knowledge of participants that were reported, although some shifts in attitudes were suggested, particularly from the policy dialogues bringing different people together to understand each other.Capacity-building support and the EVIPNet support instruments are clearly valued, with examples of how these had, or hopefully would, change policy. However, these currently benefit only those who attend sessions and/or develop country KT products. Some mentioned attempts at cascading this knowledge from those who had attended courses to stakeholders within countries but there was still a need identified to widen the pool of EIP experts within countries.Face-to-face communication between countries and with the Secretariat occurred mostly during the development of products and other specific country activities or during multi-country meetings. Most stakeholders suggested that communication was largely one-way from the WHO Secretariat, pushing information out and supporting countries. Others felt that some countries seemed somewhat uncomfortable in approaching the Secretariat for support, highlighting the need to build relationships and trust. Many stakeholders stressed the importance of face-to-face communication in establishing and building these relationships.“… but I think that the multi-country meetings are indispensable in the sense that if you don’t meet the person and see them and talk to them, then, I mean it’s easier to approach someone if you have met them in the past, so in that sense the multi-country meetings are a key moment in that communication.” Documentary analysis suggests that attendance at multi-country meetings has remained similar over time in terms of the numbers of attendees. However, attendance by WHO country office staff has reduced from 92% of member countries represented in 2013 to 16% in 2017. Some members and stakeholders also raised language issues in interviews and as part of the online evaluation. Despite translated products being available and training sessions being held with Russian interpreters, it was felt that this was not the same as having someone who truly understands EIP and the EVIPNet methodology and can explain and convey the details, nuances and passion accordingly, resulting in some elements being potentially lost in translation.Social network communication across the EVIPNet Europe network, as indicated by the use of Yammer, has decreased over time and there has been variation in participation and the type of posts , with limited exposure to other sources, which may limit global reach, e.g. via journal database searches.The documentary analysis highlighted 18 conference presentations related to EVIPNet to date, and seven peer-reviewed publications authored by both the WHO Secretariat and country team members. Initially, these were mostly led by the Secretariat but, increasingly, countries are leading this external promotion, e.g. there are now four peer-reviewed papers with country members as lead authors. However, many of the papers have been published in WHO publications  and study visits to other countries as well as the WHO Regional Office for Europe to learn about EIP.is also a good networking platform and it supports us when I see other WHO [country teams] are struggling with the same difficulties and see how they have solved them, etc.” “[EVIPNet Europe] Greater involvement of EVIPNet Global and the role of headquarters in coordinating and ensuring knowledge exchange and sharing globally was requested.Many of the interviewees felt that individuals, personalities and relationships were important. Individual members have developed strong relationships, with trust and rapport built and fostered over time by the dedicated Secretariat Coordinator, who is key to this. The role of individual personalities and networking skills was seen as vital to the success of the Network.This related not only to the relationship with the Secretariat, with almost all representatives of Network member countries expressing gratitude for the support received, particularly from the Coordinator, but also the internal relationships with the WHO country office and Ministry of Health. Getting the right country team together was highlighted, with a need for team members to be highly motivated individuals. Within countries, developing a relationship between the national champion and WHO country office staff was an important part of the EVIPNet process, facilitated by WHO country offices. The latter are also crucial in coordinating and planning the future of EIP with the national champions.In the country online evaluation, some countries reported coordination between agencies to be a barrier, e.g. interaction between policy-makers and the research community was lacking, coordination between ministries was slow, the Ministry of Health lacked a strong voice and had no interaction with the WHO country office.Different models were adopted for national champions, with some countries maintaining continuity where possible and others frequently nominating new representatives. While stakeholders recognised the pros and cons of these approaches (breadth versus depth), continuous retraining of champions and developing relationships as well as a lack of progression of knowledge and understanding caused by turnover of national champions were felt to be challenging. Furthermore, many stakeholders suggested that there was a need for countries to gain experience, not just in the technical skills-building capacity but also in applying this in practice, facilitated by hands-on technical support and coaching to develop understanding, as has been offered by the Secretariat. This is also made more challenging if staff turnover is high or untimely. Many references were also made to the challenges caused by turnover of staff in all areas, e.g. national champions, WHO country offices, Ministries of Health and the Secretariat. While there were examples of this turnover bringing about positive changes , this was mostly seen to be negatively impacting on Network development.And in some countries they do nominate the same people for every year’s workshop so that they can, you know, increase skills; in other countries it’s more common that every year they want to give a different person the opportunity to participate so then it’s a bit difficult in terms of continuity, at the same time you’re exposing more people to the content.” “Staff turnover was also mentioned as causing delays to the creation of country KT products and institutionalisation of the process, with decision-maker/government changes being the main ones cited. Turnover at the political level also had implications for how members advocate for EVIPNet, given the need for political buy-in. However, with short terms of office, ministers want to see impact quickly, which is challenging, while changing policy and seeing a measurable impact takes longer, and M&E is also not yet well developed in member countries.During the growth of the Network, the Secretariat sought and developed a number of collaborations, highlighted in the documentary review. These include external collaborations with HINARI, SORT IT , EUPHA , and the Cochrane Collaboration, K2P Center, McMaster Health Forum and Wellcome Trust. Internally, collaboration with the Division of Health Emergencies and Communicable Diseases has been established in the WHO Regional Office for Europe, which is expanding.Formal partnerships at the country level in support of the work of national country teams at the time of the evaluation was more limited. However, there are indications of country teams undertaking intercountry collaborative activities, e.g. study visits, informal mentoring from EVIPNet Europe and EVIPNet Global member countries. Cohorts of countries working on the antimicrobial resistance evidence briefs for policy are a new collaborative step demonstrating added value as these have created communities of practice and allow sharing of experiences (Box 1).Human capacity and financial resources were raised as an issue for country teams as well as the Secretariat. When asked about the human, time, skills and financial resources invested, some were not aware of these details. Of those who were, most suggested that resources were limited, and work was done by a small number of staff or consultants hired for a specific activity. Almost all finances to support EVIPNet Europe were from WHO. Country teams are often reliant on limited staff, without dedicated funding or terms of reference.The WHO Secretariat has been creative in identifying alternative sources for human resources and expanded in-kind support, e.g. via half-year short-term fellowships and secondments. The document review highlighted the variation and turnover in WHO Secretariat staff and the variation in financial resources for the Secretariat over time. However, posts are included in the WHO human resource plans, highlighting commitment. Further, the operational plans suggest that the WHO Secretariat has continued to achieve its own EVIPNet Europe objectives, despite these limitations. The Secretariat has also been successful in utilising organisational resources, including core funding for intercountry work and BCA funds (unique to the European Region) for country-level work (Box 2).Box 1 Country cohorts – encourages cross-WHO workingA good example of cross-WHO working mentioned by several stakeholders was the country cohorts. A country cohort is where a group of EVIPNet Europe member countries work together on developing evidence briefs for policy, in this case on antimicrobial resistance-related briefs, supported by an external facilitator, complemented by guidance provided by the WHO Secretariat of EVIPNet Europe and the antimicrobial resistance team from WHO Europe. This has allowed increased visibility and joint funding, increasing the resources available as well as inclusion in country biennial collaborative agreements and, therefore, within priority areas. These cohorts have received promising feedback so far from stakeholders and member countries, including recognition from WHO headquarters and interest in replicating this model in other WHO regions.… this cohort it’s about again instilling them with information, helping them prioritize the information that they really need to do the work and then working towards more of a Community of Practice model where there’s shared agreement on the deliverables each month, and people pre-circulate documents.”“Box 2 Use of BCAs – bringing in other sources of fundingThe Secretariat has also been successful in utilising organisational resources, including core funding for intercountry work, and biennial collaborative agreement (BCA) funds (unique to the European Region) for country-level work. The BCAs express an official commitment as a contractual arrangement between the Member State and WHO to undertake EVIPNet Europe-related knowledge translation activities.BCAs are useful for formalising country-level work with other WHO programme areas. These contractual elements with member countries and internally within WHO were also highlighted as an accountability mechanism by stakeholders and further use of BCAs is encouraged.I think there’s a big benefit in liaising with other programmes and sort of offer a package to a country.” “EVIPNet Europe appears to be filling a niche in promoting the capacity of Network member countries for EIP, with the recent growth of EVIPNet Europe demonstrating the demand for such an initiative in the Region.The Network and its activities are valued and EVIPNet Europe’s tools and approaches are perceived as useful overall. Furthermore, the Network’s capacity-building efforts have improved KT knowledge and skills at the individual level.The EVIPNet Europe approach has catalysed the planning and implementation of country-led KT activities and led to conceptual and instrumental policy influences.However, not all member countries are progressing at the same pace, demonstrating that a flexible and tailored approach and timing are important for success.Whilst finance is needed to ensure progress and sustainability, people and relationships are just as crucial to developing a culture of EIP in member countries and across the EVIPNet Europe network.This formative evaluation showed promising results as well as lessons to guide the future development of EVIPNet in the WHO European Region and other parts of the world. The evaluation showed that a strong Secretariat is essential in setting up a KT network to strengthen the KT capacity of, and collaborative relationships between, Network members. While no country has gone beyond the formation of country teams, many countries have developed situation analyses and are in the process of paving the way towards creating sustainable KTP structures. Commitment and engagement from Network members vary with financial and human resources, including staff turnover, which are issues both at the EVIPNet Secretariat and country team levels.it is perhaps not the information-passing that is the active ingredient in sharing knowledge but the trust, increasingly shared language, and learning about different perspectives, not achievable electronically, that make the difference” [Alongside its capacity-building efforts, the WHO Secretariat put in place numerous and diverse mechanisms early on to develop joint working, exchange of experiences and networking. These included peer support accelerated by the Network’s train-the-trainers programme, mentoring (within and beyond EVIPNet Europe member countries), and the face-to-face multi-country meetings and workshops. These mechanisms and the need for networks to have face-to-face interaction in building relationships on which knowledge that was co-produced can then be shared was highly valued by Network members, and has been recognised by others , 57. LakThe work of EVIPNet is legitimised and reinforced by being grounded in the resolution on health research of the Fifty-eighth Meeting of the World Health Assembly  and is sFurthermore, the work of EVIPNet Europe  is facilitated by the fact that WHO is seen as a credible and neutral broker and is an evidence-guided, normative organisation. Thus, it is in a privileged position to take the lead to support member countries to institutionalise EIP. The institutionalisation of EIP will require multiple and concerted efforts and high-level commitment from all players – at country level from policy-makers, researchers and the civil society and across all levels of the EVIPNet network, including the EVIPNet Europe Secretariat, WHO country offices and country teams/member countries , 60–62. The results of the evaluation reflect that this is a formative evaluation; the financial and human resources available to support the Network are limited; the Network is rather large and has grown quickly; and that capacity-building takes time. Financial information available for EVIPNet Europe suggests that less than US$ 1 million has been spent between 2013 and mid-2018, which suggests that it is offering value for money when compared to other programmes. For example, the Building Capacity to Use Research Evidence (BCURE) programme received £15.7 million (approximately US$ 19.3 million) funding from 2013 to 2017 to build capacity to improve the use of evidence in decision-making in low- and middle-income countries through six linked projects implemented across 12 countries in Africa and Asia . SimilarThis formative evaluation of EVIPNet Europe identified the strengths and good practices that should be further built upon. For example, the country cohort models for evidence briefs for policy  enabled the integration with WHO programme-specific work as well as access to the generous funding that antimicrobial resistance is currently receiving from the donor community. The country cohort models also created communities of practice, an approach that is seen as promising in the KT community –66. The A possible limitation of the EVIPNet approach is the fact that it is not linked to a specific health issue or topic. This may be limiting resource mobilisation and investment in the approach due to the lack of strong interest groups (as would likely exist for particular disease areas). It may also affect and make it more difficult to establish KTPs and embed them in local institutions.As such, new innovative mechanisms have been explored such as establishing strategic partnerships between the Secretariat and specific health issues  to leverage funds for the EVIPNet Europe KT work through the country cohort approach and use of biennial collaborative agreements.Areas where further attention and work are needed were also identified by the evaluation. These include the need for more sharing of lessons learnt and good practices; further expanding the capacity-building work and portfolio of instruments; and exploring the feasibility of a Russian hub/Russian coordinator for EVIPNet Europe to provide more targeted and adapted assistance to the eastern part of the Region. At the WHO Secretariat level, work is required to ensure that the staff posts remain funded. Increased funds may also be needed for further staffing in the Secretariat to support growth of the Network. Furthermore, additional Global EVIPNet capacity is needed to join up and complement regional programmes and to provide capacity-building at the WHO headquarters level.At WHO country office level, EVIPNet Europe should continue to build capacity for EIP and clearly demonstrate and advocate for links to strategic levers, including how EVIPNet Europe can, and has, helped to achieve national health targets. At the country level, it will be important to finance and support the work of EVIPNet, ensuring that, beyond individual champions, a dedicated institutionalised, multidisciplinary EVIPNet Europe country team is created, acting as the EIP focal point to the Ministry of Health with protected time to engage in the planning, implementation and M&E of EIP work on a sustainable long-term basis , 60–62.This is the first evaluation of EVIPNet Europe as a KT network of networks at its different levels . It is also one of the first, if not the first, evaluation of an organisational KT activity that is designed to improve the general climate for linking research to policy and to institutionalise EIP in countries through the creation of KTPs. Previous evaluations of EVIPNet activities have focused on specific activities, such as the evidence briefs for policy and deliberative dialogues , or on tThe use of a peer-reviewed M&E framework, including a theory of change and logic models, is a clear strength of this evaluation. The mix of evaluation methods, including validated tools at the country level –50, alloThe lack of baseline measures and a comparison group were limitations of the evaluation. However, this evaluation can now serve as a baseline for future evaluations, which could also compare changes in the use of research evidence in policy-making between more established or newly developed country teams and new EVIPNet Europe member countries. Another limitation was the possible perceived lack of independence of the evaluator. Although the evaluation was conducted by an external consultant not involved in the development, strategic thinking or delivery of EVIPNet Europe  they could have been viewed as an internal WHO staff member by many of those who were interviewed due to the use of a WHO email address and job title. Furthermore, the response rate for the online evaluation was low (57.5%). While this response rate is higher than similar evaluations of networks , the finAt 6 years since the launch of EVIPNet Europe at the end of 2012, the evaluation showed that the Network is valued and is filling a niche in Europe, as evidenced by the strong and rapid growth of the Network from 13 member countries to 21 in 2018 and more wanting to join. There have been some successes, including examples of KT activities influencing policy. However, no formalised KTPs have yet been created. More work and support are needed if the Network is to achieve its vision of a Europe in which high-quality, context-sensitive evidence routinely informs health decision-making processes that ultimately serve to strengthen health outcomes across the Region.This evaluation of EVIPNet Europe as a KT network of networks that was designed to improve the general climate for linking research to policy is believed to be an important contribution to the literature. It is hoped that this contribution to the evidence base will benefit others involved in a similar process by sharing of the lessons learnt. However, EVIPNet Europe needs to build on this evaluation and regularly undertake follow-up Network evaluations. Evaluations at the country level by country teams are also needed with the support of EVIPNet Europe through capacity-building and making use of the EVIPNet Europe M&E framework. Finally, we would like to encourage other similar initiatives, including other WHO regions, to undertake similar evaluations.Additional file 1. Evaluation tools.Additional file 2. Detailed results for each data collection method.Additional file 3. Case studies.Additional file 4. Findings against the evaluation questions.


Visual contents such as movies and animation evoke various human emotions. We examine an argument that the emotion from the visual contents may vary according to the contrast control of the scenes contained in the contents. We sample three emotions including positive, neutral and negative to prove our argument. We also sample several scenes of these emotions from visual contents and control the contrast of the scenes. We manipulate the contrast of the scenes and measure the change of valence and arousal from human participants who watch the contents using a deep emotion recognition module based on electroencephalography (EEG) signals. As a result, we conclude that the enhancement of contrast induces the increase of valence, while the reduction of contrast induces the decrease. Meanwhile, the contrast control affects arousal on a very minute scale. We watch and appreciate various visual contents such as movies, dramas, animation etc. The scenes from the contents evoke diverse emotional responses from us. The emotions from the scene can be categorized through Russell’s emotion model such as excitement, happy, pleased, peaceful, calm, gloomy, sad, fear and suspense . The stoThe purpose of our research is to examine how the emotional responses from visual contents are affected from the change of the contrast of a scene. Enhancing or reducing the contrast is expected to increase or decrease the emotional responses, respectively. In many applications, positive emotional responses such as happy or peaceful are magnified to delight the audiences further. Negative responses such as sad or fear are required to get reduced to relieve the audiences who feel uncomfortable for the scenes.The assumption of this study that the emotional responses can be affected from the change of the contrast is backgrounded from biological observation. The lower or higher contrast can affect the number of photons arriving at the retina of an eye. Since the difference of the intensities of the photons is reduced, the brain reflecting on the neurons of retina can produce different biosignals, since the different biosignals can be interpreted as the change of valence and arousal. The relationship between the contrast and emotional responses should be examined in a very careful approach.To reveal the relationship between the contrast and emotional response, we collect some scenes from various visual contents of three representative emotions: neutral, positive and negative. Then, we control the contrast of the scenes in both directions, either enhancing or reducing. For each scene manipulated, we employ human participants to appreciate the contents, while their EEG biosignals are captured and processed using a deep learning-based emotion recognition model. We prove that the contrast change of a visual contents affect the arousal or valence estimated from the emotion recognition model. The overview of our framework is illustrated in The estimation of the relations between contrast and valance/arousal by participants’ annotation is a subjective approach. For a reliable estimation of the relation, a series of objective approaches that employ the biosignals estimated from the participants. Many researchers employ diverse biosignals including EEG, electrocardiography (ECG), electromyography (EMG), photoplethysmogram (PPG), respiration rate (RSP) and galvanic skin response (GSR), etc. Since people cannot control their own biosignals, the biosignal-based approaches for estimating valence and arousal gather higher confidence than user survey-based approaches.Furthermore, we carefully experiment the change of emotion recognition for the visual contents and compare the magnitude of emotion change due to the contrast control. From this experiment, we can suggest how to control visual contents to strengthen or weaken the emotion of the contents.The change of emotion is expressed by the value of valence and arousal, which are measured through EEG biosignal. Actually, many researchers have presented a series of models that measure valence and arousal from EEG signal. Among these works, some of them employ classical machine learning schemes including Bayesian , dual trThis paper is organized as follows. In In this section, we describe previous studies around deep learning, emotion recognition approaches and emotional responses from visual contents.Researchers employed handcrafted features to apply them to a variety of machine learning algorithms such as support vector machine (SVM) and principal component analysis (PCA). They showed acceptable accuracy, but the performance of their framework heavily relied on feature selection.Many researchers have employed deep learning-based approach which does not require delicate feature selection process. Jirayucharoensak et al.  proposedKhosrowabadi et al.  presenteThe early frameworks for emotion recognition have a common limitation in selecting features by human experts’ hands. As the progress of convolutional neural network (CNN), many researchers have overcome such a limitation. Salama et al.  presenteMoon et al.  proposedCroce et al.  classifiYang et al.  proposedZhang et al.  presenteRecurrent neural network (RNN) is widely used for processing time series data such as writing. Since EEG signals are time-sequential, many researchers have employed RNN to recognize emotion from the signals. Li et al.  proposedXing et al.  developeSeveral researchers employed conventional feature selection model to improve deep learning-based model. Some researchers often applied deep learning-based model to the other machine learning framework, other researchers used several different deep learning-based models.Yoo et al.  presenteYang et al.  proposedIn early days, Adams and Osgood  presenteRecently, Suk et al.  researchJun et al.  performeRajae-Joordens  insistedWith recent advances in deep learning, various image transfer techniques are introduced. Peng et al.  studied Loving Vincent [The old man and the sea [A man who plants trees [We selected three well-known animations for our study:  Vincent , The old the sea , and A mts trees . These aFrom the selected visual contents, we sampled three different scenes, each of which contain positive, neutral and negative emotion, respectively. For this purpose, we sampled 10 scenes from each of the contents and hired 10 human participants to mark the emotion of the scene for the three categories: positive, neural, and negative. We also asked the participants to suggest the background of their selection. From the votes of the participants, we selected the scenes of highest votes for our experiment. The emotions from the sampled clips for three emotions are illustrated in Loving Vincent, the scene where Armand talks to Marguerite, the daughter of Dr. Gachet, about good memories of Vincent van Gogh was selected as the scene of most positive emotion. The major reason of the selection is that this scene that reminds of a positive memory about Vincent evokes positive emotion.From The old man and the sea, the scene of the fantasy that a boy is playing with fishes in the sea was selected. The major reason of the selection is that playing with fishes in the sea is a fantasy from childhoods.From A man who plants trees, the scene of happy people living in a village filled with a lot of trees was selected. The major reason is that this last scene of the animation shows how the effort of a man can make people happy and a village flourish.From The scenes of positive emotion are illustrated in Most of the participants selected the scenes that happened early in the contents as scenes of neutral emotion. Since the early scenes introduced the actors of the contents and explain their situations, the emotional response from these scenes tended to be neutral.Loving Vincent, the scene where Armand is visiting Auvers was selected. They selected this scene since visiting Auvers is an action that does not evoke any emotion responses.From The old man and the sea, the opening scene where a boy visits the old man and talks about his dream was selected. Similar to the Loving Vincent scene, talking about a dream does not evoke either positive or negative emotions.From A man who plants trees, the scene where the narrator encounters Elzeard Bouffier and visits his house was selected. Similar to the above contents, visiting a house does not evoke emotional responses.From The scenes of neutral emotion are illustrated in Most of the participants selected scenes of conflict as the scenes of negative emotion. The existence of unpleasant objects such as blood accelerated the negative emotion.Loving Vincent, the scene where Louise Chevalier, the maid of Dr. Gachet speaks ill of Vincent van Gogh was selected as a scene of negative emotion. Since most of the participants had good feelings about Vincent, they felt unpleasant and negative about the conversation between Louise and Armand.From The old man and the sea, the scene of bloody struggle between sharks that came to hunt the tuna caught by the old man was selected as negative. The blood spread on the sea added the negative emotion.From A man who plants trees, the scene that describes the process of falling down in a village due to the harsh weather was selected as a scene of negative emotion.From The scenes of negative emotion are illustrated in We applied linear contrast control scheme, which has been widely used in many image processing applications. The reason is that both contrast enhancement and contrast reduction were required. Since we did not want to affect color factor in the contrast control, we converted the color in RGB space into HSV space and applied contrast control for the S (saturation) and V . After the control, we re-converted the color in HSV space into RGB space.A contrast controlled value of saturation, denoted as We implemented our model in a personal computer with Intel Core i7 9600 CPU , 16 GB main memory, and nVidia GeForce TitanX GPU . The model is implemented using Python with Pytorch library. Our multi-column structured emotion recognition model was derived from our previous research . The strHence the model was trained with a DEAP dataset , we provWe followed the data collection strategy in . For a tAs we described before, three scenes were selected from each animation. We assumed that contrast would more or less affects emotional response by the mood of the scene. 30 participants were asked to watch each of the positive, neutral, and negative scenes from an animation, then move to next animation after 30 s of rest. After nine playbacks, another nine playbacks were followed with increased contrast, then last nine with decreased contrast. Each playthrough, consisting of 27 playbacks, was preceded by 3 s of baseline recording. All recordings were started/stopped by human control, allowing less than 1 s of error.To streamline our data to DEAP dataset, similar preprocessing is taken. We used EEGLab with MATLAB to preprocess those recordings alongside the channel location file. The data were downsampled to 128 Hz with a bandpass frequency filter from 4.0–45.0 Hz was applied. The order of the channels was same as that of DEAP. The data were segmented into 27 trials. Our emotion recognition model got preprocessed data and produced estimation upon valence and arousal level for each trial.k is the number of modules of our model. We set k as 5 in this study. In total, the number of training, validation and test samples for our model was 42,240, 9600 and 9600.Our model was trained using the DEAP dataset according to the strategy presented by . For traIn many related literatures, various models have been presented to estimate valence and arousal from EEG biosignal. They employ either classic machine learning techniques ,5,6,7,8 We hired 30 human participants for our experiments and group them into three groups. The distribution of gender and age of the participants for each group are suggested in A participant of a group was guided to watch three movie clips that belong to the same category. The order of three movie clips was shuffled randomly. Each clip was played for 60∼90 s. After watching three movie clips that belonged to one category, the participants had a day break to neutralize their emotional responses. The schedule of our experiment is illustrated in Their emotional responses expressed by their EEG signals were processed through our multi-stage emotion recognition module to extract valence and arousal. We present the valence and arousal extracted from our experiment in t-test. As illustrated in p values for the valence of the pair of neutral and negative and neutral and positive from the three contents were less than 0.01, which denotes that the difference of valence was significant. Meanwhile, p values for the arousals were all greater than 0.01, which denotes that the difference of valence was not significant. From the t-test, we concluded that the movie clips we selected from the three contents showed distinctive valence and similar arousal.In the first analysis, we compared the valence and arousal of the original three emotions suggested in In the second analysis, we compared the change of valence and arousal due to the control of contrast. As illustrated in t-test for the valence and arousal suggested in t-test is suggested in On the contrary, the arousal did not show a specific change from the control of contrast. As illustrated in In this analysis, we compared the enhanced valence from the control of contrast to the valence of original contents. In The same result was observed for the originally positive and negative emotion. In Similarly, in We compared the contents whose valence changed in the greatest scale in The scenes in Since most other works considered mono-color as stimuli, it was hard to directly compare our result with other models. Jun et al.  pointed An unexpected result of our contrast control scheme is that the intensity of the image also varies. Enhancing the contrast may increase the average intensity of the image and reducing may reduce the average intensity. In Even though the average intensity of an image varied, we still argued that the increase or decrease of valence comes from the change of contrast rather than the average intensity in the following two points. The first point is that the magnitude of the change of contrast is greater than the magnitude of the change of average intensity. Since the change of contrast shows greater magnitudes than the change of average intensity, we argued that the change of valence is affected by the change of contrast rather than the change of average intensity.Our second point is from existing literatures. Even though brightness is known to one of the cause of emotional response, the study of Wilms and Oberfeld  indicateOur third point is to employ user-annotated valence-arousal values. Considering user-annotated valence-arousal values with the valence-arousal values estimated from EEG signals through emotion recognition model can improve the confidence on the performance of our approach. However, our experiment on participants collected only their EEG signals. Our next experiment on estimating human emotion using EEG signals will collect user annotated valence-arousal values as well as EEG biosignals and compare them to improve the confidence of our experiment.In this paper, we have proved that the contrast control of visual contents affect the valence of the people who watch the contents. The enhancement of contrast increases valence, while the reduction decreases. Arousal is not affected by the change of contrast. We have proved our argument by extracting EEG biosignals from human participants and by recognizing valence and arousal using a deep emotion recognition model.We have a plan to extend this study to review the relations between other properties of contents such as color with the emotion of visual contents. We also have a plan to apply the result of this study in manipulating visual contents to satisfy users.


More than 18 million Americans are currently suffering from alcohol use disorder (AUD): a compulsive behavior of alcohol use as a result of a chronic, relapsing brain disease. With alcohol-related injuries being one of the leading causes of preventable deaths, there is a dire need to find ways to assist those suffering from alcohol dependence. There still exists a gap in knowledge as to the potential of telemedicine in improving health outcomes for those patients suffering from AUD.The purpose of this systematic review was to evaluate the measures of effectiveness, efficiency, and quality that result from the utilization of telemedicine in the management of alcohol abuse, addiction, and rehabilitation.telemedicine and alcohol abuse, which were then searched in the Cumulative Index to Nursing and Allied Health Literature, Cochrane, and MEDLINE (PubMed) databases. A total of 22 articles were chosen for analysis.This review was conducted utilizing the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. The articles used in this analysis were gathered using keywords inclusive of both The results indicated that telemedicine reduced alcohol consumption. Other common outcomes included reduced depression , increased patient satisfaction , increase in accessibility , increased quality of life , and decreased cost . Interventions included mobile health , electronic health , telephone , and 2-way video . Studies were conducted in 3 regions: the United States , the European Union , and Australia .Telemedicine was found to be an effective tool in reducing alcohol consumption and increasing patients’ accessibility to health care services or health providers. The group of articles for analysis suggested that telemedicine may be effective in reducing health care costs and improving the patient’s quality of life. Although telemedicine shows promise as an effective way to manage alcohol-related disorders, it should be further investigated before implementation. An indiA growing concern in the health care field is the alarming number of preventable deaths that result from alcohol-related injuries. It is estimated that excessive alcohol use is responsible for approximately 88,000 deaths and approximately 2.5 million potential life years lost annually in the United States and in the European Union; approximately 1 in 4 deaths of males aged 15 to 39 years are because of alcohol [Telemedicine, specifically videoconferencing, is identified as a successful tool in reducing the effects of adults suffering from AUD, but videoconferencing is only one small aspect of telemedicine that can offer assistance to this condition . TelemedThe delivery of health care services, where distance is a critical factor, by all health care professionals using information and communication technologies for the exchange of valid information for diagnosis, treatment and prevention of disease and injuries, research and evaluation, and for the continuing education of health care providers, all in the interests of advancing the health of individuals and their communities .The use of telemedicine in treating alcohol dependence helps increase the patient’s access to providers as well as increasing social support outside of the health care setting . Both faA review was conducted on this topic in 2012 . It provThe purpose of this systematic review was to analyze and evaluate current literature in regard to the measures of effectiveness, efficiency, and quality that result from the use of telemedicine in managing alcohol abuse, addiction, and rehabilitation in patients. Effectiveness was measured through outcomes. Efficiency was measured through cost. Quality was measured in terms of safety, timeliness, access, patient satisfaction, or quality of life.telemedicine and alcohol abuse were searched on MEDLINE (PubMed) using the US National Library of Medicine’s Medical Subject Headings (MeSH). A total of 5 subheadings were identified under the search of telemedicine, and 13 subheadings were identified for alcohol abuse. For the purposes of this systematic review, these 18 subheadings were selected as the final set of key terms to be used for searches within the research databases. To ensure we captured at least 10 years of research, searches were conducted on December 31, 2018, and the time period of inclusion was December 1, 2009 to December 31, 2018.The research process was structured following the Kruse Protocol for Writing Systematic Reviews  and the The subheadings were separated utilizing Boolean operators to ensure the search produced articles that are inclusive of both alcohol abuse and telemedicine or terms related to those topics. See A total of 3 research databases were chosen: Cumulative Index to Nursing and Allied Health Literature (CINAHL), PubMed (MEDLINE), and Cochrane. These databases were chosen because of their wide availability, following the Kruse Protocol for writing a systematic review, and because they are recommended by the National Institutes of Health .Articles were eligible for analysis if they were published in the last 10 years and included information on the use of telemedicine for the treatment of AUD. We preferred the articles also included data on outcomes, but that was not an eligibility criterion.Using the key terms from MeSH, the initial search in the CINAHL yielded a total of 42 articles, the PubMed database yielded 150 articles, and Cochrane yielded 12. The search process and criteria are visually represented in The next step of the systematic review process consisted of a screening of abstracts by the authors. The 94 abstracts were divided between 5 authors to be carefully screened according to the objective statement. Each abstract was screened by at least two authors, and a final consensus meeting was held to finalize the selection of articles to be included in the systematic review. During this process, a total of 65 articles were excluded for not being germane, leaving a total of 29 articles.The 29 articles were read in their entirety by at least two reviewers and excluded if articles were not related to the objective statement, were study protocols, or were systematic reviews. Similar to the screening process, articles were divided among the 6 reviewers for analysis commensurate with our objective statement. Workload was divided among 6 authors following the Kruse Protocol : JW, LL,A kappa statistic was calculated to assess the level of agreement between the authors. The calculated kappa score was 0.85 indicating strong agreement on which articles should be included in the group for analysis -16. The During the analysis phase, data items were extracted such as participants, intervention, comparison method, outcomes, as well as measures of effectiveness, efficiency, and quality. Each article was analyzed by at least two reviewers, and independent observations were made. A consensus meeting was held to discuss observations and distill observations to themes. On the basis of the combined set of observations and identified themes, a second read of articles was conducted and another consensus meeting held to ensure an exhaustive data collection. Themes were analyzed for trends. Affinity matrices were created to describe frequencies of data. The frequency of occurrence was recorded for all articles. Multiple articles mentioned multiple themes. Observations of potential bias were recorded.All articles in the analysis discuss telemedicine for the treatment of AUD -38.After performing a punctilious data extraction process, the authors were able to identify several potential outcomes that were used to measure the effectiveness of telemedicine in the management of alcohol use, addiction, and rehabilitation: reduction in alcohol consumption, increase in patient satisfaction, reduction in symptoms of depression, increased accessibility, and decreased cost. The authors selected a total of 7 outcomes as evaluation criteria for article analysis in a total of 35 occurrences (number of times the theme was addressed). Articles can include multiple themes. An affinity matrix of these themes is listed in Out of the outcomes identified, a total of 16 out of 35 (46%) occurrences mentioned that there was a statistically significant reduction in the participants’ alcohol consumption ,32,35,38A total of 4 technological interventions were identified  in 77% (17/22) of the articles analyzed and resulted in no statistical significance in improvement in the remaining 23% (5/22). There were no observations where telemedicine resulted in a decrease in medical outcome or effectiveness.The key intent in this systematic review was to identify measures of effectiveness, efficiency, and quality of telemedicine in managing conditions of AUD or some sort of substance dependence. A total of 7 measures of effectiveness and 4 categories of technological intervention from 8 countries were identified in 22 articles. The intervention of technology was associated with positive medical outcomes in 17 of the 22 articles -33,35,36Subjects, including patients, clients, students, or employees, reported high levels of satisfaction with the technological interventions for several reasons. The technology solutions enabled a variety of positive aspects of care: self-management of subjects’ symptoms or conditions, the technology solutions were self-paced, the technology solutions were both synchronous and asynchronous with providers, and technology solutions were based on subjects’ preferences. Individuals did not have to travel to the provider or counselor, reducing costs, and addressing concerns over a stigma of being treated for AUD . SubjectBeyond the scope of this study and yet an important consideration, providers incurred costs for the technological interventions reviewed in this study, including initial development, capital securement, supportive programming, administration, personnel, and time and resources for ongoing delivery and aftercare. However, organizations’ ability to treat and provide valuable resources to recipients of care outside the brick and mortar far exceeded the cost of the intervention. Organizations were able to expand practice without expanding square footage, and eHealth solutions continued to provide care outside the boundaries of 8 am to 5 pm, a traditional treatment day. An organization with the treatment of alcohol and drug abuse or addiction and depressive symptoms should strongly consider a technological intervention because it would enable the organization to meet this strategic goal without another product line, additional provider staff, or clinic space.The implementation of telemedicine as an alcohol management treatment option has demonstrated great promise in reducing alcohol consumption and positively affecting other factors associated with alcohol use and improved patient outcomes. Alcohol management is the treatment of alcohol abuse through moderation. It addresses the highly complex and personal issue that alcoholism can create . AlcoholThe 22 articles analyzed included participants from several countries. The authors identified a total of 7 characteristics for use in measuring telemedicine’s effectiveness in AUD patients. Most of the articles mentioned a decrease in alcohol consumption (16/22) and an increase in accessibility (3/22). A social stigma was mentioned that is a cause for concern, and it supports the adoption of telemedicine . The feaIt is unfortunate that so few articles qualified for analysis based on our selection criteria. To do this study over again, we would expand our selection criteria to enable a larger number of articles to be included in the analysis. The larger number of articles would strengthen the results and associated conclusions and give us a stronger indication of external validity.Of interesting note, the countries publishing articles on AUD are developed countries. It is unlikely that developing countries do not experience difficulties with AUD. Instead, it is more likely that these countries are not devoting resources to research the topic.Our systematic review illuminated the need for strategic leadership in planning care environments to include innovative, effective, and efficient solutions such as telemedicine. Leaders and experts in the field of health care emphasize the necessity of delivering value-based care according to trends as identified by over 2000 health care leaders in a 2017 survey . Chief eOne of the increasingly frequent realities, the need for cybersecurity, will create a culture of increased trust with consumers and will also serve to address one of the cited barriers to traditional care in this study, threat of social exposure and lack of confidentiality around participation in electronic telehealth around a socially sensitive condition, alcohol misuse, abuse, and hazardous lifestyles ,40. IncrStrategic leaders considering future trends also set a vision of redesigned, decentralized, nontraditional, and remote health care facilities to better meet the needs of consumers, better protect patients and staff, and better prepare for sustainability with greater unpredictability of reimbursement and regulations .Organizational leaders also have the responsibility to incorporate telehealth interventions to not only better reach individuals afflicted by AUDs but also to reduce the societal health burden. To this point, the World Health Organization in its Sustainable Development Goal Agenda of 2030 cites the harmful use of alcohol as a leading risk factor for worldwide health and includes expectations of organizations to assess and implement cost-efficient and multimedia approaches to impact the global issue .The limited number of articles selected in the group for analysis provided was a limitation. With smaller samples, it is possible that the results found within each study cannot be broadly applied to the population. In addition, self-reporting was heavily relied on within these studies and ultimately could have led to self-report bias when patients reported back to the researchers.A strength to be considered is the process used to select the final 22 articles for the review. Utilizing the US National Library of Medicine’s MeSH enabled an exhaustive search of the literature. Adopting the technique from AMSTAR to use multiple, independent reviewers for both screening abstracts and analyzing articles reduces selection bias . FurtherOverall, there are many potential applications for telemedicine in improving patient outcomes and strategy in the delivery of care across diverse environments. Telemedicine has the ability to expand the scope of health care by allowing patients the ability to connect to health care providers without the barriers of proximity, without the negative stigma associated with alcohol or drug addiction or mental health, and without the expense of driving to the clinic for an in-person visit.Telemedicine can allow patients greater access to their provider in times of need. One of the key necessities for successful alcoholism recovery is behavioral therapy and counseling for the patient . The con


Multiple sclerosis (MS) is one of the world’s most common neurologic disorders leading to severe disability in young adults. MS-related fatigue directly impacts on the quality of life and activity levels of people with MS. Self-management strategies are used to support them in the care of their health. Mobile health  solutions can offer tools to help symptom management. Following a user-centered design and evidence-based process, an mHealth solution called More Stamina was created to help persons with MS manage their fatigue.The overall study aims are to explore the feasibility, acceptability, and usability of More Stamina, a mobile app for fatigue self-management for persons with MS.A mixed-methods, multicenter study will be used to assess the feasibility, acceptability, and usability of More Stamina. The study will take place during the third and fourth quarters of 2020 (Q3-Q4 2020) in 3 locations: Argentina, Spain, and Switzerland. A longitudinal cohort study will take place, and think-aloud protocols, open-ended interviews, and short answer questionnaires will be used. Persons with MS will be recruited from the different locations. This study seeks to enroll at least 20 patients that meet the criteria from each site for the longitudinal cohort study .Ethical approval has been granted in Argentina and is pending in Spain and Switzerland. Outcomes will be published in peer-reviewed medical journals and presented at international conferences.Findings from this study will be used to help understand the role that mHealth can play in fatigue management in MS.ClinicalTrials.gov NCT04244214; https://clinicaltrials.gov/ct2/show/NCT04244214PRR1-10.2196/18196 Multiple sclerosis (MS) is one of the world’s most common neurologic disorders leading to severe disability in young adults. More than 2.3 million people live with MS in the world, with higher incidences in Northern Europe and in temperate climates . MS is aMobile health  is the delivery of health care or health care–related services through the use of portable devices . The useHowever, digital health technologies have low adoption rates by patients with chronic diseases, in part explained by an inappropriate fit of these technologies with patients’ daily lives and the high patient burden associated with digital self-monitoring . StudiesIn previous studies, we explored available mHealth apps for persons with MS and found that only a handful exist ,36. ThisUCD is a design philosophy that places the needs and characteristics of end users in the center of software design and development -39. ThroThe evaluation of usability entails a wide array of methodologies that vary in terms of research design, complexity, cost, and duration . DiffereMore Stamina is a gamified task organization tool designed to help persons with MS manage their energy, to minimize the impact of fatigue on their daily life. The tool acts as a to-do list where users can input the tasks they want to accomplish that day in a simple manner . More StThe overall concept is that a person’s energy is represented through a visual metaphor (progress bar) and a symbolic unit that quantifies the amount of estimated effort a given activity might take (Stamina Credits). Users start their day with 100 Stamina Credits and assign them to new activities for that day. As users go through their day and complete the activities, they mark them as done, and the system prompts them to assess whether their effort was underestimated, overestimated, or properly estimated. More Stamina keeps track of these answers as data points and starts analyzing and creating a trend for each activity . Repeated use of More Stamina allows it to learn about the user’s habits, and once sufficient information is gathered on “shopping,” a recommendation feature starts reminding users of their own tendencies.The app logs user activity through the in-built sensors of the device and is able to monitor the number of steps, walking pace, distances, and GPS positioning. Patient-reported outcomes are available and optional in More Stamina. Standardized tools such as the Fatigue Severity Scale  and ChalThe More Stamina app was created following the UCD approach through iterative development, allowing continuous improvement of the app. The progress of the solution through the different phases of the UCD process is reported in our previous studies. The state of the practice of health apps for MS was studied through a systematic app review ,36. The The work intended in this phase will assess the feasibility, acceptability, and usability of the mHealth solution with persons with MS see .The overall study aims are to explore the feasibility, acceptability, and usability of More Stamina, a mobile app for fatigue self-management for persons with MS.The specific objectives are to estimate adherence to the use of More Stamina; estimate the effect of More Stamina adherence on behavior change, measured through changes in the amount of activities and amount of estimated energy per activity; estimate the effect of adherence to the use of More Stamina on the perception of fatigue management; validate the value proposition of More Stamina; identify More Stamina user activity patterns of persons with MS; and identify factors associated with the use of More Stamina.A mixed-methods, multicenter study will be used to assess the feasibility, acceptability, and usability of More Stamina. The study will take place during the third and fourth quarters of 2020 (Q3-Q4 2020) in 3 locations: Argentina, Spain, and Switzerland.A longitudinal cohort pilot study will be conducted using a series of well-established standardized tools for user-based evaluations and surveys (see Standardized Tools). Participants’ background information regarding health information, quality of life, and familiarity with mobile technologies will be collected. Potential cultural differences will also be explored through qualitative approaches. System usage logs will quantify patient engagement with the system .This study is part of a collaborative project between researchers and different institutions across the globe. The work will take place in 3 locations with local teams. Sessions will be conducted by a native language facilitator. Overall study coordination will be carried out by Dr. Guido Giunti from the University of Oulu as part of the More Stamina research project.Hospital Italiano de Buenos Aires (HIBA) is a university general hospital in the Autonomous City of Buenos Aires that includes 2 hospitals  and 22 primary care centers. This organization has 750 beds, conducts approximately 2.5 million outpatient visits per year, and includes a health maintenance organization that delivers prepaid health care to approximately 150,000 members per year.The Argentinian local team will be composed of researchers from HIBA.The Vithas Nisa Sevilla Hospital (VNH) ranks as the number one private center in Andalusia and fourth in all of Spain with a specialized unit aimed at researching and treating MS. This unit is composed of a multidisciplinary team, including neurologists, neuropsychologists, physical therapists, clinical researchers, nurses, and administrative staff.The Universidad de Sevilla is the main house of learning in the Andalusian province of Spain and provides superior education by means of studies, teaching, and research, as well as the generation, development, and diffusion of knowledge to serve citizens and society.The Spanish local team will be composed of researchers from the Universidad de Sevilla and VNH.Kliniken Valens is a center specializing in neurological rehabilitation services located in Valens, Switzerland. Kliniken Valens employs a multidisciplinary staff, including neurologists, physiotherapists, occupational therapists, speech therapists, and sports therapists. In 2019, a total of 550 persons with MS were admitted for neurological rehabilitation, of which 70% suffered from fatigue and this was the primary focus of their stay.The Swiss local team will be composed of researchers from Kliniken Valens and the University of Oulu.Persons with MS will be recruited from patient databases at HIBA, VNH, and Kliniken Valens and invited to participate in the study. Inclusion criteria will require each participant to be >18 years old, have received a confirmed MS diagnosis according to McDonalds criteria  at leastExclusion criteria include refusal to participate, cognitive or physical impairment that prevents the use of mobile phones, and inability to attend the follow-up encounters.This study seeks to enroll at least 20 patients that meet the criteria from each site for the longitudinal cohort study . The sample size is an adequate number because of the methods used that provide extensive, detailed data .The following is a description of the standardized tools that are used in this study. Language-appropriate and culturally validated versions will be used accordingly.The EDSS is a method of quantifying disability in MS and monitoring changes in the level of disability over time. It is widely used in clinical trials and the assessment of persons with MS . The EDSNeuro-QoL  is a measurement system that evaluates and monitors the physical, mental, and social effects experienced by adults and children living with neurological conditions .The Chalder Fatigue Scale is a self-administered questionnaire for measuring the extent and severity of fatigue within both clinical and nonclinical epidemiological populations. Although originally developed to measure the extent of chronic fatigue symptoms within clinical populations, the scale was revised and is now more widely used to measure the severity of “tiredness” rather than just chronic fatigue syndrome .The eHealth Literacy Scale  is an 8-item scale that tends to measure perceived skills at finding, evaluating, and applying electronic health information to health problems . The insThe System Usability Scale provides a “quick and dirty,” reliable tool for measuring a product’s usability . It consThink-aloud is a well-established technique for usability assessment, commonly used to determine users’ thoughts and opinions while they perform a list of specified tasks with a system . A thinkTo guide the usability evaluation, case scenarios were created for what would be considered as normal everyday user interactions with the More Stamina solution. A panel of neurologists, physiotherapists, and designers validated the different scenarios. The specific scenarios patients will have to perform will consist of (1) completing their user profile, (2) creating a new activity for themselves, (3) managing previously recorded activities, (4) planning for a future activity, and (5) responding to one of the integrated surveys .Participants will be instructed to assess the More Stamina app using the think-aloud method, stating out loud what they are doing, what thoughts come to their minds, and how are they interacting with the app. The facilitator will record comments and actions of the participants with screen-capturing software and an audio recorder.Qualitative inquiries are useful to provide insight into complex and multifaceted experiences of individuals when a rich description is the main goal of the study . ParticiWithin More Stamina, Stamina Credits are a numeric continuous variable that ranges from 0 to 100 to represent the estimated effort of an activity. This variable is used to gauge how effective the user is in predicting the effort of each individual activity.User activity patterns over time will be aggregated per week and described. Adherence will be estimated in each follow-up meeting. Adherence with app use will be estimated using the frequency of weekly use throughout the tracking period.Persons with MS from each location will be invited to be part of the study and go through the process of informed consent where they will be briefed about the overall study and their rights as participants . After providing informed consent, participants will be part of a series of 4 workshops. Prospective monitoring will be carried out for 2 months for all participants at all sites, and daily measurements of More Stamina during that period will be considered.Encounter 1 will be used to set up user accounts for participants, introduce the More Stamina solution, and install it on their smartphones. Monitoring will begin after the installation of More Stamina. All participants will complete the following standardized tools: EDSS, NeuroQoL, Chalder Fatigue Scale, eHeals, and System Usability Scale.A subgroup of up to 5 participants per site  will be selected for convenience to represent the main themes of this study: time since diagnosis (prolonged or recent), digital health literacy (expert user or beginner user), and disease burden (high or low). This subgroup will go through the think-aloud protocol by performing the specific scenarios and be interviewed about ease, utility, and perceived benefit. The sessions will be digitally audio recorded and video recorded, capturing on-screen navigation on screens.There will be 3 face-to-face follow up meetings: day 15 (Encounter 2), day 30 (Encounter 3), and at the end of the follow-up at day 60 (Encounter 4). In these meetings, feedback will be obtained on the use of the More Stamina app regarding its functionality, usability problems, daily use, and information concerning disability, quality of life, and adherence. The overall study flow that will be replicated in each site is shown in P value <5% will be considered statistically significant. Statistical analysis will be performed using STATA v15.Descriptive statistics will be used to summarize the participants’ background and characteristics. Categorical variables will be presented as absolute and relative frequencies. Continuous variables will be presented as mean and standard deviation or median with interquartile range depending on distribution. A Quantitative analysis will be performed on the data collected through the mHealth solution. A linear regression model will be used to assess the association between the degree of adherence to the app, changes in the amount of activity, and changes in the amount of estimated stamina per activity. The outcome variables will be changes between baseline and final activities, changes in estimated stamina per activity for specific activities selected by the researcher team based on frequency and relevance, and changes in fatigue self-perception. Changes will be calculated considering the baseline measurement and final measurement after the 3-month tracking period. A multivariate linear regression model adjusted for age, sex, and eHEALS will be used. The need for inclusion of other potential confounders will be evaluated.Participants who are above the median usage of the app will be defined as heavy users. The median is less affected by outliers and skewed data and is usually the preferred measure of central tendency when the distribution is not symmetrical. Potential factors associated with the use of the app will be explored with a univariate logistic regression model.Qualitative analysis will be performed on the data collected through the think-aloud protocol, individual interviews, and focus groups by using thematic analysis to identify emerging themes . Each trCultural contexts influence the way users choose, utilize, and conceptualize products and technology . AccordiQualitative and quantitative data will be explored, paying attention to potential differences that pertain to cultural and geographical contexts such as traditions, habits, and weather conditions. In order to mitigate the risk of spurious results in usability testing due to crosscultural settings, we will use native speakers who are in a trusted position related to the participants.Access to personal information will be restricted to the investigators of the study, health authorities, the Research Ethics Committee, and the monitors and auditors of the study. They will be subject to the duty of secrecy inherent to their profession, when necessary, to verify the data and procedures of the study, but always maintaining the confidentiality of the same according to the current legislation. Participants may exercise their rights of access, rectification, cancellation, and opposition of data according to the European Union General Data Protection Regulations .The information and personal data of the participants will be kept in a completely confidential form with all the rigor of the law. Participants will be asked to not use any names during group discussions. Reports of study results will not include any identifying information. The paper questionnaires used will be digitized. The audio recordings of the group discussions will be typed and kept in secure servers at the University of Oulu. After the transcriptions are finished, the audio recordings will be destroyed. The typed transcripts will remain on password-protected computers, and any hard copy will be kept in a closed filing cabinet. Only members of the research team will be able to listen to the recordings or read the typed versions.The ethical research guidelines of the University of Oulu  and UnivThe participants will be informed about the nature of the research project; the reasons for their participation; risks, benefits, and alternatives associated with the research; and their rights as research subjects before agreeing to participate. Steps will be taken to ensure that data gathered from participants will be kept under strict security, anonymity, and privacy.The study will take place during the third and last quarters of 2020 (Q3-Q4 2020). Outcomes will be published in peer-reviewed medical journals and presented at international conferences.This protocol presents the work intended to take place to assess the feasibility, acceptability, and usability of a fatigue management mHealth solution for persons with MS.Persons with MS have a different attitude than other people towards physical activity  and are By tracking and collecting use and contextual information, the present study will also help to understand underlying factors and causes for MS fatigue.The goal of this pilot study is to further our understanding of the potential issues and challenges that will be used as the foundations for a larger randomized control study.It is possible that the differences between the patient group who participated in the UCD and the target groups of this study are significant enough that there is a preference mismatch. For example, the importance of fatigue management could vary depending on cultural factors or adherence, or interest could even be radically different depending on economic factors. This is expected, and understanding potential differences is part of the aims of the study. Finally, the nature of the study does not allow the assessment of health outcomes due to the lack of randomized exposure and monitoring period. Such explorations will be the subject of future studies.


Many birth cohorts have been carried out. We performed a review of European birth cohorts to see the countries involved, provide a panorama of the current research topics and design, and, more generally, provide input for those creating collaborations and laying out guidelines aimed at unifying cohort methodologies to enable data merging and maximize knowledge acquisition. We searched PubMed and Embase for articles referring to longitudinal, prospective European birth cohorts and searched online cohort inventories. We found references to 111 birth cohorts, 45 of which began enrolment at birth. These cohorts began between 1921 and 2015 and represented 19 countries, with varying sample sizes . As of 5 January 2020, were still recruiting. The main areas addressed were allergic diseases (14 cohorts) and environmental exposure (f12 cohorts) and most cohorts were publicly funded. Given the large costs of running cohorts and the importance of long follow-up periods in identifying the risk factors for disorders thought to have a perinatal/early life etiology, current cohorts must be designed to answer research questions considering several aspects, from genetic ones to psychological, social, and environmental ones. Furthermore, universally recognized methodological aspects are needed to permit the comparison and merging of cohort data. Cohort studies collect data on a group of people in order to identify and quantify the relationship between exposure and outcome. They can be prospective or retrospective. In prospective cohort studies, the population is recruited regardless of exposure or outcome status and is followed for a set period until the diseases or outcomes of interest occur ,2. In reThere are different types of registries, from patient registries based on a disease or exposure, which collect data on patients with that characteristic, to those simply listing patients with specific diseases, e.g., rare diseases, but are not used for evaluating outcomes . A cohorMany birth cohorts have been carried out around the world and many are currently ongoing ,15,16. EBetween January and July 2019, we performed a narrative review of the European birth cohorts taking into consideration multiple sources. The search strategy is described in detail in http://www.birthcohorts.net), created as part of the Children Geno Network (a European FP5 Research Program) in 2005, and improved and redesigned within the European FP7 Program CHICOS project (http://www.chicosproject.eu). We also searched the cohorts listed by two EU funded research projects: The ENRIECO project [We searched PubMed and Embase with the last update on 1 July 2019, limiting the results to the 20th of May 2019, with no restriction on past publication years. We excluded randomized controlled trials and articles focusing on vaccines or on genes or gene expression. Records found were downloaded in the Reference Manager 12 software . The records were reviewed and, for each one, the name of the cohort it involved was noted. When this information was not available in the records’ abstracts, the articles were retrieved when possible. We also searched online birth cohort inventories to see if any additional cohorts could be found. In particular, we consulted the web-based database  worked on different parts of the data extraction process as well as on certain overlapping parts, and all cases of uncertainty, discrepancy, or missing data were resolved through discussion, searches for additional data sources, and consensus. The type of funding received by the cohorts was classified into four types: Public , Foundation, University, and Industry.A total of 8572 articles were found through the internet-based bibliographic literature databases consulted, after exclusion of duplicates as illustrated in the PRISMA flow diagram . Of thesA total of 111 European birth cohorts were identified. Of these, 66 began enrolment in pregnancy (2 of which in pre-pregnancy) and 45 at birth or shortly afterwards. References of articles referring to the 45 cohorts found are listed in The 111 European cohorts represented 27 different countries, including three countries represented only in the four multinational cohorts . The countries most commonly involved, in 16 cohorts each, were Germany and the UK, followed by the Netherlands (15). The number of children recruited in the different cohorts ranged from 107 to 10.8500 (median 1924). The starting year of enrolment in the different cohorts ranged from 1921 to 2016 (median 2002) and the duration of enrolment, excluding 10 with currently ongoing enrolment, and one with missing data, ranged from 1 to 23 years (median 2) (rounded to whole years). Concerning the follow-ups, 62 have ongoing follow-ups, of which 22 are lifelong and the rest of which have a duration of 1 to 31 years. The median could not be calculated because of the general nature of the description of follow-up duration for several cohorts .When only the subset of cohorts that began recruitment at birth was selected, 45 cohorts were present , represeThe starting years of these cohorts ranged from 1921 to 2015 (median 2002). More than half of the cohorts began in 2000 or later and 8 after 2010. The sample size of each cohort varied considerably, from 236 of the 1990 Dutch cohort to more than 21000 children of the TEDS—Twins early development study, with a mean of 4230 (median 2515). The two largest birth cohorts are located in the UK  and in FRANCE .The oldest of the 45 cohorts enrolled participants in 1921 (the 1921 Aberdeen Birth Cohort) and the youngest began enrolment in 2015 (the German KUNO-Kids birth cohort) . As of JConcerning the follow-up, 49% (22/45) of the cohorts are still undergoing follow-up, while the rest are definitively closed. Concerning the more recent cohorts, fifteen of the 26 (58%) cohorts set up from 2000 on, and 6 of the 8 (75%) from 2010 on, are currently ongoing. The duration of the follow-ups ranged from 1 year to life-long .The aims behind the creation of the cohorts are various and cover a broad range of aspects of child health. The most frequently studied individual topics included: Allergic diseases (14 cohorts), environmental exposure (12), and growth , although several cohorts (27) addressed multiple areas and were designed to test a wide range of hypotheses . AllergiWhen divided into three groups based on age of the cohorts to see if, over time, the priorities studied changed, allergic diseases and environmental exposure were more recent priorities. Both were initially studied to a limited extent. Allergic diseases resulted as a priority area among the cohorts for the first time in 1990 and environmental exposure in 1992. More specifically, in the 1921–1995 period, growth (5 cohorts) and allergic diseases, environmental exposure, and nutrition (3 each) were the most commonly addressed areas, between 1997–2004, they were, allergic diseases and environmental exposure (7 cohorts each), and in 2005–2015 they were allergic diseases (4), and growth and general areas with multiple aims (3 each). Only three cohorts addressed the impact of the family context (nurturing care) to a certain extent among their goals, the ELFE , Epifane, and GUS (Growing up in Scotland) cohorts. All three of these cohorts were relatively recent .In the review presented in this article we provide up to date information on birth cohorts in Europe with a focus on those that began data collection at birth. The fact that more than half of the cohorts began in 2000 or later and that many are still ongoing in terms of follow-up of participants suggests that there is a current, active interest in newborns, although with the involvement of only 9 countries, and with different aims. The number of participants included varied largely, although the average was of only just over 4000. With larger sample sizes, aided by the use of standard measures in the pooling of cohorts, and the joining of data from large epidemiological studies from other countries, it is possible to understand the epidemiology of diseases . A long More than a third of the cohorts were established in northern Europe, where this kind of study has a long-lasting tradition . Health Unlike the work carried out by Larsen and colleagues , we limiThe main areas addressed by the cohorts were allergic diseases and environmental exposure, both of which have become priority study areas more recently. The numerous cohorts addressing environmental exposure reflect increasing attention to the negative effects of pollution on health. Growth was studied more by the older cohorts, while obesity is a new research area, although all of the areas currently remain topics of interest for research, expansion of knowledge, and appropriateness of interventions. Many cohorts were designed to test a wide range of hypotheses, such as the Spatz cohort . This apOther cohorts were more focused on specific topics, such as respiratory diseases  . The expThe research results show that few cohorts have followed in detail child development as well as neurodevelopment. In general, child health is a product of biological factors and diverse sets of environmental influences, including intrauterine and social ones ,14,15,34The cohorts were supported mostly by public funds. Setting up and running cohorts, especially over long periods of time, is very important, but is also extremely costly. More economic support would be useful for setting up cohorts in all countries, and for making it possible to collect enough information, and in a suitable format, to make the cohorts comparable enough to merge their data with that of other cohorts. The industry had a limited presence in the cohorts described in this review.The limited funds available for running cohorts inevitably influence the type of data collection employed. While most of the cohorts collected data via predefined questionnaires and face to face interviews, which are less costly, patient visits involving clinical assessment were carried out in just over half the cohorts. The Nordic countries often draw patient data from different inter-related registries, facilitating collection of also clinical data, and reducing costs ,38. The The lack of commonly acknowledged guidelines on the use of common measures for data collection, along with the various data sources used by cohorts, lead to the extreme difficulty in merging or comparing data from different cohorts, a process that would permit more far-reaching, significant conclusions from the research. This is a well-recognized issue and different groups are working to address it .Few cohorts also focused on family context (nurturing care) and its impact among their research areas. The family context is a fundamental issue  and shouFew cohorts involved the general pediatrician  as the person delegated to collect data, highlighting the fact that primary care is a neglected resource for research . With thA lot of work must be done, however, to determine why some babies go on to develop disease, while others remain healthy. It is in this context that the pediatrician, or the general practitioner taking care of the newborn, play a fundamental role in describing the importance to the families of specific actions to be taken to guarantee a child’s health . In suchOur aim was to describe the birth cohorts’ research topics and design, to understand their interest in the impact of the family context (nurturing care) and the general pediatricians’ role in child care and data acquisition, and to provide input for future cohort studies and for those working towards universally acknowledged guidelines for unifying cohort methodologies in order to enable data merging and the consequent maximum acquisition of knowledge. The results of this study show that a limited number of countries participates in multinational birth cohort studies and that adequate, universally recognized methodological aspects , and common health priorities, are needed in order to permit the comparison and merging of cohort data. Such an expanded amount of comparable data would permit researchers to draw more solid conclusions and stakeholders to implement the knowledge in initiatives aimed at improving people’s health.www.birthcohorts.net.To our knowledge, this is the first inventory of birth cohorts, both at the European and worldwide level, starting recruitment after birth. Considering pregnancy and birth cohorts together, inventories have been produced in Canada  and AsiaPotential limits of this study exist. It is possible that we did not identify all the European birth cohorts, but we attempted to use the most rigorous and extensive search strategy for identifying the cohorts, so we expect that a potential percentage gap would be small. This review is of a descriptive nature; we did not contact the principal investigators of the cohorts, but searched for information only via web, and this may have limited the completeness of data or led to partial data, since data found in one publication may be different from those in other publications referring to the same cohort. Furthermore, classifying the cohorts’ aims into individual scientific areas was difficult given the overlap between areas , but the distinction was useful in order to provide a general description of the cohorts and to show their differences. For example, the four remaining cohorts labelled as addressing general areas with multiple aims were not classifiable because their aims were so widespread.The strengths of this study are that it reports on a large number of active initiatives whose role is to look ahead, starting from birth, to monitor the development of European newborns. The findings of these cohort studies can be useful for stakeholders in allocating resources towards appropriate endeavors in order to work towards improving the health of citizens from birth.The continuing follow-up of existing cohorts is the most efficient way to detect areas of improvement and windows of collaboration. Longitudinal data investments need to be directed at capturing the circumstances of tomorrow’s children and adults, i.e., current cohorts must be able to answer upcoming research questions considering several aspects: genetic and biological, psychological/social environments, medical care and medications, and lifestyle and environmental parameters. In this regard, new cohorts are periodically being set up to address the more pressing issues, such as child health and pollution . We also