Females  and participants who had ever delayed acceptance or refuse the vaccine  had higher odds of COVID-19 vaccine hesitancy. Participants with a postgraduate degree  presented the highest willingness to be vaccinated against COVID-19. More than two-thirds  of the participants did not mind a booster dose required following primary vaccination. A total of 63.2% (95% CI: 60.0%–66.0%) of the participants only accepted a nearly 90% effective or above vaccine at preventing COVID-19. At the same, 86.4% (95% CI: 84.4%–88.4%) of the participants reported only accepting a vaccine with minor side effects. Conclusions: The moderate levels of COVID-19 vaccine acceptance found in the early phase of the pandemic demonstrate that it is important to improve the implementation of effective management for vaccine promotion and the acceptability of the vaccine to slow or delay transmission.Background: This study aimed to survey the attitudes toward COVID-19 vaccines and their acceptability among the Japanese public as soon as the United States Food and Drug Administration (FDA) authorized vaccines and their rollouts started around the world. Methods: An anonymous cross-sectional survey was conducted in Japan between 4 January and 5 March 2021. A questionnaire was administered to evaluate attitudes toward COVID-19 vaccines according to demographic characteristics, vaccine characteristics, and vaccine production. Results: A total of 1037 completed responses were received. More than half (63.5%) of the participants responded positively (extremely likely/likely) toward COVID-19 vaccines. The highest acceptance to be vaccinated was discovered among the youngest age group. As expected, participants who had never delayed acceptance or refused the vaccine in their history of vaccination had a significantly higher willingness to be vaccinated against COVID-19 ( The enormous burdens of morbidity and mortality, including economic, social, and mental impacts due to coronavirus disease 2019 (COVID-19), have affected people worldwide since the World Health Organization (WHO) declared a global pandemic on March 11, 2020 . VaccineVaccine hesitancy, as has been confirmed by multiple studies, is a threat to preventing persistent community transmission and controlling COVID-19, and, in fact, WHO named it one of the top ten threats to global health in 2019 ,8,9. COVIn the early phase of the COVID-19 pandemic, a heterogeneous willingness to accept vaccines was found; participants from Asian nations and middle-income countries had relatively high acceptance . A multiThis study synthesizes the findings on the acceptance of and attitudes toward COVID-19 vaccine characteristics and also explores the factors influencing COVID-19 vaccine choice from the data collected in Japan. These findings provide insights into the challenges surrounding the introduction of new vaccines and will be useful for future infectious disease pandemics. Furthermore, government agencies can use this knowledge to devise strategies to improve the vaccination rate in the future.We conducted a cross-sectional web-based survey using an online questionnaire between 4 January and 5 March 2021. A total of 1037 individuals from Japan participated in this survey. It was necessary for individuals to be 18 years or older, to be Japanese residents who had not yet received a COVID-19 vaccination, and to provide informed consent online.The questionnaire was developed in English. A native language option for the questions was available for surveys carried out in Japan. The items of the questions were content validated by content experts. Translations into target languages were carried out using standard forward–backward translation by native speakers. The translated questionnaires were also validated by new independent bilingual native speakers . To ensuThe general information of the questionnaire: (1) age, sex, and highest education attainment; (2) if they had ever delayed acceptance of or refused a vaccine despite the availability of a vaccine service. The response options were “yes” or “no”.The attitudes towards COVID-19 vaccines were examined by asking participants to state their opinions on 6 statements, including number of doses, effectiveness threshold, side effects, duration of protection, technology used in COVID-19 vaccine production, and producing country.The acceptability of COVID-19 vaccines was assessed with the response options “extremely likely”, “likely”, “unlikely”, and “extremely unlikely”.p-value < 0.05 was considered statistically significant.R software (version 4.1.3) was used to analyze the data. The distributions of socio-demographics across vaccine willingness were examined using a χ2 test or Fisher’s exact test, if appropriate. The effect directions of demographic characteristics on vaccine characteristics influencing vaccination attitudes were first evaluated with the Spearman coefficient and visualized with a heatmap. Then, a univariate logistic regression followed by a multivariate logistic regression were applied to explore the effect sizes of the demographic characteristics on attitudes towards COVID-19 vaccines or vaccine characteristics influencing vaccination attitudes. A result with a In total, 1037 participants across Japan were admitted to this study. As shown in The proportion of responses for COVID-19 vaccine acceptance is shown in p = 0.007). As expected, participants who had never delayed acceptance of or refused a vaccine had a significantly higher willingness to be vaccinated against COVID-19 (p < 0.001). The distribution of the willingness to be vaccinated against COVID-19 among the participants with the highest educational attainment was not statistically significant (p = 0.053), while a higher willingness to be vaccinated against COVID-19 was found in participants with high educational attainment.The willingness to be vaccinated against COVID-19 by the demographics of the participants across Japan is shown in The factors influencing COVID-19 vaccine willingness with a logistic regression model are shown in Participants who had ever delayed acceptance of or refused a vaccine  had higher odds of COVID-19 vaccine hesitancy after adjusting for age, gender, highest educational attainment, and ever delayed acceptance of or refused a vaccine. Participants who had a postgraduate degree  presented the highest willingness to be vaccinated against COVID-19 compared with participants in secondary school and with lower degree. Compared with the youngest age group, a gradual increase in COVID-19 vaccine hesitancy with age was found; however, the point estimation revealed that the oldest age group presented a positive willingness to be vaccinated against COVID-19.We further explore the associations between influencing factors and the willingness to be vaccinated against COVID-19 among participants who had not ever delayed acceptance of or refused a vaccine. The results on attitudes toward vaccine characteristics influencing vaccination attitudes are shown in Next, the correlation between demographic characteristics and vaccine characteristics influencing attitudes towards vaccines was visualized with a heatmap . Among aA multivariate logistic regression was also performed to examine the relationship between demographic characteristics and vaccine characteristics influencing vaccination attitudes. As shown in This investigation aimed to assess attitudes around COVID-19 acceptance among the general public in Japan during the early phase of the pandemic, and the findings could offer insight into the strategies required to effectively address issues surrounding vaccine hesitancy in pandemics of infectious diseases.Compared with a previous global study of COVID-19 vaccination among people in Japan , our stuAfter adjusting for age, gender, highest educational attainment, and ever delayed acceptance of or refused of a vaccine, females and participants who had ever delayed acceptance of or refused a vaccine experienced higher levels of COVID-19 vaccine hesitancy. In this study, higher levels of COVID-19 vaccine hesitancy were seen in these groups due to pre-existing vaccine hesitancy, which is based on a lower trust in healthcare professionals, the health system, and the government . In partVaccine choice factors revealed that most participants would only accept a vaccine with minor side effects, indicating that the safety of the newly developed COVID-19 vaccines was extremely important. It is imperative for the public to know that the COVID-19 vaccines have been rigorously tested, including through large clinical trials, despite being rolled out for emergency use, and that early trial data indicated that severe adverse effects were extremely rare ,26 and tStudy limitations must be considered when interpreting the results. Firstly, an online survey may lead to sampling bias, so the results may not be generalizable to the larger community, as indicated by the absence of a representative from some of the prefectures. Secondly, this study was a cross-sectional study, although, we were able to identify associations between attitudes towards COVID-19 vaccine characteristics and acceptability; however, we could not infer cause and effect. Lastly, as the responses were self-reported, there may have been self-reporting bias and a tendency to report responses that are socially desirable. Although the current study has limitations, it is the first survey of its kind to be conducted in Japan, providing valuable first-hand information about the attitudes and acceptance of COVID-19 vaccines in Japan.The current study showed that Japan’s public had a moderate acceptance of COVID-19 vaccines when the vaccines were newly introduced. High vaccination coverage rates indirectly protect the overall community by slowing transmission. Clearly, the different types of COVID-19 vaccines with diverse characteristics currently available may increase uncertainty and difficulty in making a decision, resulting in people delaying or refusing vaccination. The present study underlines the importance of herd immunity during early epidemic outbreaks, targeted at the hesitancy segment of the population, to moderate the perceived risks of COVID-19 vaccination and generate long-term benefits from the immunity group. However, this strategy usually needs strong and sustained support from the government.


Patients with no formal education had significantly higher hesitancy  than participants with graduation and above education. More than half of the study’s participants did not trust information shared on social media about COVID-19 vaccines and cited newspapers/news channels as their main source of information. The study provides important insights into COVID-19 vaccine acceptance and the impact of vaccination campaigns. Many unvaccinated COVID-19 patients in hospitals highlight the need for an effective vaccination drive to protect people from acquiring infection and subsequent hospitalization.Vaccine hesitancy is widespread in many parts of the globe, particularly in low–middle-income countries. Therefore, we surveyed a sample of hospitalized COVID-19 patients to assess COVID-19 vaccine acceptance and vaccine hesitancy in a low–middle-income country. A cross-sectional sample of 385 confirmed reverse transcriptase–polymerase chain reaction COVID-19 patients treated at secondary and tertiary care hospitals in Punjab, Pakistan, were analyzed to assess COVID-19 vaccine uptake and vaccine hesitancy. The construct validity and reliability of the 11-item vaccine hesitancy questionnaire were also examined. In addition, multivariate logistic regression was used. The majority of the COVID-19 patients admitted to hospitals were not vaccinated (84%). Of those who were willing to receive vaccination, the majority (55%) considered vaccines an effective way to protect people from COVID-19. However, those who were not willing to receive their COVID-19 vaccine had significantly higher hesitancy than those willing to receive their COVID-19 vaccine. In addition, older hospitalized COVID-19 patients aged 60 years or above (20–29 years: OR 0.10; 95% CI 0.01–0.72,  The discovery and development of various vaccines against coronavirus disease 2019 (COVID-19) have transformed the response of all governments toward the COVID-19 pandemic. Nationwide lockdowns are increasingly becoming a thing of the past, and many countries are contemplating unrestricted movements for vaccinated individuals. Despite this, the world is witnessing intelligent, calculated, and, most importantly, falsified misinformation campaigns against the available vaccines and mass vaccination programs against COVID-19. The movements against vaccines are notoriously famous for spreading misinformation about the rigorously investigated vaccines.Pakistan is a culturally diverse country wherein health beliefs are often intermingled with cultural and political views. The continual existence of poliomyelitis and tuberculosis is a testament to significant vaccination hesitancy or disbelief against two otherwise widely eradicated infectious diseases. One-third of Pakistani children have not received routine childhood immunization, with the immunization rates as low as 37% in some of the rural areas of Pakistan .Pakistan has been fortunate to have a wide range of COVID-19 vaccines available for public use. To date, the BBIBP-CorV vaccine (Sinopharm), CoronaVac vaccine (Sinovac), Ad5-nCoV vaccine (CanSino), mRNA-1273 vaccine (Moderna), ChAdOx1-S vaccine (AstraZeneca), BNT162b2 vaccine (Pfizer-BioNTech), Gam-COVID-Vac vaccine (Sputnik V), and Pakistani-made Pakvac vaccines have been approved for use in the COVID-19 national immunization campaign .In order to avoid high morbidity, mortality, and economic catastrophe and the unusual burden on the healthcare delivery system caused by the COVID-19 pandemic, the majority of the country’s population should acquire immunity through a comprehensive vaccination campaign . A succeThe low vaccination rate is not surprising considering the widespread confusion and misinformation campaigns. As most of the available COVID-19 vaccines are relatively new, campaigns against COVID-19 vaccines can be easily operated by campaigners who are against the vaccines. Countries with low literacy rates and poor socioeconomic status are particularly vulnerable to campaigns against vaccines due to poor health literacy and the lack of trust toward the established professional and governmental bodies’ among their citizens. There have been many public concerns about the safety of COVID-19 vaccines since the beginning of this pandemic, contributing to vaccine hesitancy . VaccineThis was a cross-sectional study which utilized an online interviewer-administered tool among patients admitted to COVID-19 wards (between 1 August 2021 and 30 September 2021) with a positive real-time reverse transcriptase–polymerase chain reaction (RT-PCR) test in four district headquarters hospitals and three tertiary care hospitals in Punjab, Pakistan. Patients who were at least 18 years of age, of any gender, admitted to the COVID-19 wards with a positive RT-PCR test, and who provided consent were invited to participate in the present study. The survey was administered by healthcare professionals providing care to the hospitalized COVID-19 patients.The questions included in the study instrument were developed based on the input received from healthcare providers involved in COVID-19 vaccination and the factors potentially associated with vaccine hesitancy.The study instrument comprised three sections. The first section of the study instrument included questions about demographic information . The second section of the study instrument was designed to collect information about the vaccination status of the participants . Finally, the last part of our instrument had a new questionnaire measuring vaccine hesitancy among hospitalized COVID-19 patients. Vaccine hesitancy is defined as a delay in accepting or refusing safe vaccines despite the availability of vaccine services ,17. The The English version of the study instrument was translated into Urdu with forward–backward translation . TranslaThe sample size was estimated using the total number of COVID-19-positive people when this survey was conducted in the Punjab province of Pakistan. The sample size of 383 was estimated, considering a 5% margin of error, a 95% confidence interval, and a 50% response distribution. We invited all consecutive patients with a positive RT-PCR result admitted to the COVID-19 wards in four district headquarters’ hospitals and three tertiary care hospitals. Due to the COVID-19-related restrictions in the participating hospitals, an online, web-based version of the study tool, with an Urdu translation, was developed and used by healthcare professionals involved in data collection. These healthcare professionals (involved in the research project) accessed the online form from mobile phones or other devices. They asked questions to the patients (interviewer-administered) after explaining the study objectives and receiving verbal consent from the patients.® version 27) was used, with a p = 0.05 indicating a significance level.Data are presented as frequencies, percentages, mean/median, and standard deviation. Since a new vaccine hesitancy questionnaire has been developed, the construct validity was examined using the exploratory factor analysis (EFA) method. EFA method was used with Kaiser’s alpha factoring and rotated using varimax orthogonal rotation, which improves the solution compared to unrotated ones and allows factors to be independent of one another. The sample adequacy was measured using the Kaiser rule (Kaiser–Meyer–Olkin test (KMO)) and an eigenvalue of more than one ,21. In ap > 0.05).A total of 385 COVID-19 patients admitted to COVID-19 wards in secondary and tertiary care hospitals participated in this study. Only a small proportion of COVID-19 patients admitted to hospitals were vaccinated  and either received one or two doses of COVID-19 vaccines. On average, vaccinated patients were slightly older than unvaccinated patients (47.2 years versus 50.4 years) . Among tInterestingly, the majority of the family members or friends of the study participants had received their COVID-19 vaccines . More than three-fourths of the family members or friends of vaccinated and non-vaccinated COVID-19 patients had received their COVID-19 vaccine doses . In addip < 0.001)  = 6069.91, < 0.001) . The stu< 0.001) . The EFA< 0.001) . The ovep = 0.001). Males had significantly lower odds of receiving the COVID-19 vaccine  and had higher hesitancy  than females. Regarding geographical location, COVID-19 patients from urban areas  or sub-urban areas  had significantly higher odds of receiving the COVID-19 vaccine than patients from rural areas. Compared to patients with an education attainment of graduation and above, patients with no formal education had significantly higher hesitancy .The present study explored the vaccination status and vaccine hesitancy among hospitalized COVID-19 patients and examined the factors associated with vaccine uptake and hesitancy in Pakistan. The current study revealed a low COVID-19 vaccine uptake (16%) among COVID-19 patients admitted to COVID-19 wards in four districts and three tertiary care hospitals in Punjab, Pakistan. Similarly, a previous study from LMICs indicated that the acceptance of COVID-19 vaccines in Pakistan is reported to be the lowest among the LMICs ,26. MoreThe low uptake level of COVID-19 vaccines in our cohort of patients could be due to multiple factors, and we believe that some of these factors are common in the global community. Firstly, uncertainties exist regarding the safety of the COVID-19 vaccines, especially vaccines that utilized mRNA technology, since there have been no prior experience or successes. Secondly, there could be doubts over the efficacy of the COVID-19 vaccines since there have been reports of infections in those who have been fully vaccinated. Thirdly, numerous campaigns launched by those against vaccines undermine people’s trust in COVID-19 vaccines. People against vaccination often utilize social media to disseminate fabricated, false, and sometimes misleading news to achieve their agenda. Nevertheless, the present study identified important factors associated with vaccine hesitancy, such as being younger than 55 years of age and originating from rural areas. Therefore, to achieve enough immunization coverage against COVID-19, populations from the above groups can be specifically targeted to improve their acceptance of the COVID-19 vaccine. The factor that could increase vaccine hesitancy, especially among the rural population, is a lack of education or access to quality education. In our study, patients with no formal education showed higher vaccine hesitancy compared with patients with secondary or above education. The low literacy rate in Pakistan coupled with easy access to unauthentic information is a major concern that should be addressed to improve future vaccine uptake in the region.Interestingly, in our study, we noted that males were significantly less likely to accept the vaccine. Such a finding is in contrast with the findings of several other studies ,28,29,30One encouraging finding was that more than 60% of the unvaccinated study population in our study are willing to receive their vaccines to protect themselves against COVID-19 infection. Another study published in Ethiopia demonstrated that a similar proportion of the non-vaccinated study population was willing to receive the vaccine . Our finDue to the highly contagious nature of COVID-19 and poor health infrastructure in Pakistan, this study reinforces the importance of the promotion of COVID-19 vaccines at all levels as early as possible to reduce the risk of infection, which could consume already limited healthcare resources. This is especially urgent given the concerns that the new virus variants are emerging. In addition, numerous strategies can effectively accelerate vaccination administration among hesitant population groups . For exaThere are a number of limitations that could affect the generalizability of this study’s findings. A few limitations are inherent in our study as we used a convenient sampling technique to recruit study participants that may be associated with some degree of selection bias. Moreover, the hospitalized COVID-19 patients from only seven hospitals in the province of Punjab were approached by the investigators in this study to extract necessary information from them and to record their responses; therefore, we should be careful when considering generalizing our results. Although we have adjusted for confounders, other factors could affect the findings . Moreover, we did not attempt to compare trust levels among sample subgroups. In this study, the information about hesitancy and potential confounders was based on the self-report instrument. Therefore, there is possibility for information bias and the misclassification of the vaccination status. Despite the limitations, we are confident that the findings of our study provide baseline and robust information about vaccine hesitancy among hospitalized COVID-19 patients and highlight the importance of tackling this problem to prevent spreading the infection and exhausting the health-related resources.In conclusion, the present study provides important insights into vaccination status and the impact of a vaccination campaign in Pakistan. Less than a quarter of hospitalized COVID-19 patients have received COVID-19 vaccine doses. Interestingly, vaccinated patients were more hesitant to receive COVID-19 vaccines than non-vaccinated patients. This hesitancy could be due to their negative experience of acquiring COVID-19 infection and subsequent hospitalization after receiving their COVID-19 vaccine doses. This could also discourage other non-vaccinated people from accepting the COVID-19 vaccine. Our findings may guide the efforts to strengthen the COVID-19 vaccination campaign or any other vaccination campaigns in the future.


The health, social, and economic costs of sexually transmitted infections (STIs) represent a major public health concern. Young people are considered one of the groups most at risk for acquiring and transmitting STIs. Correct and consistent condom use has been shown to be the most effective method for reducing STIs; however, condoms are often not used properly. Evidence shows that brief behavior change interventions that focus on skills, communication, and motivation to acquire safe sex practices should be adopted into routine care to reduce STIs. Funding for sexual health services in England has declined dramatically, so novel ways of reducing clinic attendance are being sought. The home-based intervention strategy (HIS-UK) to promote condom use among young men has shown promise in feasibility and pilot studies by demonstrating high acceptability of the intervention in participant and health professional feedback, including aiding men to find condoms they like and feel more confident when using condoms.The aim of this study is to determine the effectiveness and cost-effectiveness of HIS-UK when compared to usual condom distribution care among young men.The 3 trial arms consisting of “e-HIS” , “ProHIS” (HIS-UK delivered face-to-face), and control condition  will be compared against the following 3 primary outcomes: the extent to which correct and consistent condom use is increased; improvement of condom use experiences (pleasure as well as fit and feel); and decrease in chlamydia test positivity. Eligibility criteria include men aged 16-25 years at risk of STIs through reporting of condom use errors  or condomless penile-vaginal or penile-anal intercourse with casual or new sexual partners during the previous 3 months. Prospective participants will be recruited through targeted advertisements and an opportunistic direct approach at selected sexual health and genitourinary medicine services and university-associated health centers and general practitioner practices. Community and educational establishments will be used to further advertise the study and signpost men to recruitment sites. Participants will be randomly allocated to 1 of 3 trial arms. A repeated measures design will assess the parallel arms with baseline and 12 monthly follow-up questionnaires after intervention and 3 chlamydia screening points .Recruitment commenced in March 2020. Due to the COVID-19 pandemic, the study was halted and has since reopened for recruitment in Summer 2021. A 30-month recruitment period is planned.If effective and cost-effective, HIS-UK can be scaled up into routine NHS usual care to reduce both STI transmission in young people and pressure on NHS resources. This intervention may further encourage sexual health services to adopt digital technologies, allowing for them to become more widely available to young people while decreasing health inequalities and fear of stigmatization.ISRCTN Registry ISRCTN11400820; https://www.isrctn.com/ISRCTN11400820 Sexually transmitted infections (STIs) are a major public health concern. Individuals affected by STIs can face poor physical and psychological outcomes, and STI testing and treatment are a costly burden on limited health care service budgets ,2. The UMale condoms, when used correctly and consistently (used from the start to the finish of every sex act), are highly effective in preventing STI transmission ,8. WhileBehavior change intervention programs to promote condom use typically aim to improve knowledge and skills, reduce communication barriers, and improve partner negotiation. Research shows, however, that negative condom attitudes form a significant barrier to usage, often shaped through experience of decreased sensation and sexual pleasure, as well as erection difficulties during condom application and use . Many inA review of the evidence on safe-sex advice underlined the need for novel brief behavior change interventions focusing on skills acquisition, motivation , and communication competencies to ensure the adoption of safer sexual behaviors . FurtherThe HIS-UK intervention is novel in that it aims to increase condom use by enhancing the fit and feel of condoms and, thereby, increasing outcome expectancies related to enjoyment of sex while using condoms . HIS-UK HIS-UK has been designed as an extension to the usual condom distribution care model currently practiced by health promotion professionals, which typically comprises the delivery of safe sex messages, a condom application demonstration, and the supply of generic condoms.Two development and feasibility studies were conducted to inform the adapted design of the HIS-UK intervention, research methodology, and data collection tools -24. The The aim of this trial is to assess the effectiveness and cost-effectiveness of HIS-UK delivered by the 2 intervention delivery models (ProHIS and e-HIS) to reduce chlamydia test positivity among men aged 16-25 years by enhancing condom use experiences and improving correct and consistent condom use, as compared to usual NHS condom distribution care. Intervention implementation, usability, acceptability, as well as mechanisms for impact will be assessed using a mixed-methods approach guided by our logic model and the intervention evaluation framework proposed by Saunders et al [The HIS-UK study is a randomized, controlled, superiority trial with 3 parallel arms , with an internal pilot. A repeated measures trial design is being used with baseline measurement (T0) and monthly follow-up questionnaires (T1-12) after randomization and 3 STI screening points for chlamydia . Cost and outcome data are being collected to compare the resource use and cost-effectiveness of the 2 HIS-UK delivery models compared to usual condom distribution care. The protocol for this trial was registered with the ISRCTN Clinical Trials Registry in October 2019.The trial is multicentered, delivered from NHS Trust sites across England. Participants are recruited through opportunistic direct approach and patient identification by trained site staff at integrated sexual health and genitourinary medicine services located within the participating Trust sites. Targeted advertising in sixth form colleges and youth advisory services and via social media platforms  is also being used.Eligible participants must have a penis, be aged 16-25 years, resident in England, and at risk of STIs through reporting either condom use errors  or condomless penile-vaginal or penile-anal intercourse with casual (nonregular) or new sexual partners during the previous 3 months. Those with a recognized latex allergy, restricted internet access, or limited proficiency in spoken English are excluded from the trial.Lifeguide, an interactive web-based intervention software platform and secure validated data management system designed to collect participant information and deliver digital interventions to support health behavior change [Lifeguide platform can be accessed via any internet-enabled device, and recruitment sites are provided with handheld tablet computers for recruitment purposes.The trial is delivered using r change ,26. The Lifeguide algorithm to eliminate direct exposure of the allocation process by any members of the research team or recruitment site staff. The algorithm uses randomly permuted blocks of varying length to preserve concealment and maintain balance, with stratification by participant ethnicity and sexual risk. Site staff are informed of the randomization outcome  by an on-screen notification.Following eligibility screening and the taking of informed online consent, participants are registered for the trial and complete a baseline questionnaire (T0). At baseline, participants are asked about their sexual behavior, condom attitudes, condom usage experiences, STI screening, demographics, quality of life (using standard measures), and any recent NHS and public sector resource use . On submission of T0, participants are randomized to a trial arm (1:1:1) by an in-built Lifeguide using a series of intelligent “agents” . The purpose of the agents is to manage and monitor the learning of individuals by observing and recording e-learning behavior  to receive tailored prompts (through automated texts or emails) that guide and assist effective learning  and to ensure participants are exposed to a variety of learning elements.Participants randomized to the ProHIS arm receive a HIS-UK education and training consultation delivered by a health professional (approximately 10 minutes), and participants allocated to receive e-HIS are provided with log-in access to the e-HIS education and training website for them to access at their own leisure. The webpages of e-HIS are delivered and managed by HIS-UK participants (both ProHIS and e-HIS) are also provided with a condom kit containing 24 condoms , 12 lubricant sachets (at least 3 different types), a condom-measuring guide, and an instruction guide to home-based self-practice .Participants randomized to the control arm receive information, advice, and supplies per usual condom distribution care .Individual trial participants are not blinded in the trial. Site staff are required to deliver all intervention arms and, as such, are also not blinded. To avoid bias and potential contamination between trial arms, in a 7% random selection of cases, the participant-staff interaction will be audio taped (with participant consent) and assessed for intervention fidelity. In addition, in all cases where ProHIS and usual care are delivered, participants are asked to complete a postintervention fidelity checklist and recall the topics covered during their intervention consultations.Lifeguide platform.At the baseline visit, participants are further required to undertake screening for chlamydia (the most commonly occurring STI among young men), as per usual clinical practice; a single urine sample is requested from men who report only sexual contact with women, and a triple test  from men who report sex with men. Sites are responsible for the collection and laboratory analysis of samples, and screening results are subsequently shared by site staff with the research team via the Lifeguide registration, and T0 baseline questionnaire before being contacted by site staff to complete the final recruitment tasks ; (2) participants randomized to the control arm or to e-HIS are not required to attend in person to a recruitment site to complete the final recruitment tasks; these instead can be performed during a telephone consultation. For participants randomized to ProHIS, the recruitment tasks can be performed either using the web-based video-consultation software or in clinic. ProHIS cannot be delivered via a telephone consultation due to the requirement of a visual condom demonstration; (3) all HIS-UK condom kits can be mailed out to participants instead of being collected in person; (4) postal STI screening kits can be used to collect and return samples for chlamydia screening; and (5) the number of NHS Trust sites recruiting to the trial has been increased, with no upper limit proposed.Due to the uncertain future of the COVID-19 pandemic and the need to reduce direct-contact exposure between site staff and participants, the following amendments to the original protocol were proposed and approved by the UK Health Research Authority in February 2021: (1) potential participants can self-refer to the study by completing a web-based expression of interest application via the study website . EligiblFollowing education and training, HIS-UK participants commence a 2-week condom and lubricant self-practice period using the contents of the supplied kits and guided home-based exercises . As young men try out each condom and lubricant, they are asked to think about what they like and dislike about the products , and to focus on pleasurable sensations when using them to build positive associations between condom use and sexual enjoyment. After experimentation, participants are asked to complete a web-based rating and reflection form for each condom and lubricant used.Automated emails or SMS text message reminders are sent to participants to complete their ratings. Those who complete at least three ratings over the 2 weeks can order 12 condoms of their choosing and 6 sachets of lubricant to receive by post.All participants receive monthly automated text message or email notifications to complete a web-based questionnaire (T1-T12), as per T0.HIS-UK participants who successfully complete a follow-up questionnaire, and who report sexual activity within the previous month, are offered the opportunity to order further supplies of condoms and lubricants of their choosing (12 condoms and 6 lubricants).At T6 and T12, all trial participants are requested to provide samples for chlamydia testing and are offered the choice of using a postal screening kit or attending at their trial registration clinic. As per usual clinical practice, only participants reporting sexual activity in the previous 6 months are screened. Furthermore, if a participant has reported chlamydia screening within 4 weeks prior to the T6 and T12 testing period, the self-reported results of these tests are used, and no further samples are taken to minimize screening burden.All participants receive electronic voucher payments totaling £50 (US $60) to compensate them for their time; £10 (US $12) after active participation for 3 months, £15 (US $18) after 6 months, and £25 (US $30) after 12 months.Chlamydia test positivity is measured at baseline (T0), 6 months (T6), and 12 months (T12) through biomarker testing and treatment, and at T1-T5 and T7-T11 through questionnaire self-report. The primary health end point is test positivity at 6 months. To examine longevity of intervention effect, test positivity is assessed again at 12 months after randomization.Condom Barriers Scale [Condom Use Errors and Problems Survey [Correct Condom Use Self-Efficacy Scale [UCLA Multidimensional Condom Attitude Scale [Condom Fit and Feel Scale [The following secondary process indicators  are measured via web-based self-completion questionnaires using validated scales at baseline (T0) and at monthly intervals to 12 months (T1-T12)—the rs Scale , with its Survey ,29 to ascy Scale  to measude Scale , in whicel Scale , in whicAdditionally, participants are asked at baseline and monthly intervals about their sexual partners, frequency of intercourse, STI screening and test positivity, condom use motivation, use of contraception and condoms, and any episodes of condomless anal or vaginal intercourse.Alongside the clinical outcomes collected in the trial and in line with guidance by the National Institute for Health and Care Excellence, quality-adjusted life years will be used as an outcome measure to assess the cost-effectiveness of HIS-UK as compared to usual condom distribution care . BecauseResource use data will be collected prospectively to estimate the costs associated with the 2 HIS-UK intervention arms, compared to usual care. Within the trial, the resource use and costs associated with delivering ProHIS and e-HIS and any follow-up care will be captured via trial reporting mechanisms. This will include the costs of condom kits, consultation costs, digital delivery costs, and other resource use associated with intervention delivery. The baseline and monthly questionnaires (T0-T12) further capture the wider NHS and public sector resource use by participants, including the use of medication, general practitioner and sexual health service visits, and other public sector resource usage. The monthly questionnaires also collect data on personal costs experienced by participants connected with their involvement in the trial .The clinical effectiveness of HIS-UK delivered by ProHIS and e-HIS will be analyzed with an overall Type I error rate of 5% (2.5% per comparison), comparing test positivity in each of the intervention arms with the control arm . Data published by the National Chlamydia Screening Programme suggest a test positivity of 11.9% in 2017 and 12.2% in 2018 among young men aged 15-24 years in England tested in specialist and nonspecialist (including community-based) services . Our triTo have 85% power to obtain the projected difference in the outcome at T6, the study requires 476 participants in each of the arms (G*Power 3.1.9.2). To minimize risk to the trial and to reflect 36% attrition at follow-up (observed during feasibility testing) , a totalIt is estimated that it will take 30 months to recruit the target sample of 2231 young men, based on a recruitment rate of 15 per month, per NHS Trust site, using a phased recruitment strategy.The first 135 young men recruited and followed up for 6 months will form our internal pilot to assess trial implementation, participant responsiveness, intervention fidelity, and the acceptability of randomization and chlamydia screening for trial continuation. During the assessment, the following questions will be answered: “Can young men be recruited at a reasonable rate and to the numbers anticipated?” “Are young men willing to be randomized within the trial?” “Is chlamydia screening at T0 and T6 sufficiently acceptable and feasible to implement?” “Do young men remain in the study in sufficient numbers at 6-month follow-up?” “Are the intervention and study design sufficiently acceptable?” “Are site staff able to deliver the intervention with reasonable fidelity?”To assess intervention implementation and engagement, participant access to and usage of e-HIS is recorded along with fidelity of ProHIS intervention delivery and completion of condom ratings. In-depth qualitative interviews with all site staff involved in the recruitment of internal pilot participants will also be conducted to explore acceptability of the research design and ease of trial delivery. Furthermore, at 6 months after randomization, internal pilot participants allocated to the ProHIS and e-HIS trial arms will be invited to participate in interviews to explore study acceptability, issues of contamination and protocol adherence, and intervention benefits. We expect that 20 interviews will be sufficient to reach theoretical saturation; however, if necessary, additional interviews will be undertaken with participants from subsequent recruitment phases.Analysis and presentation of data will be in accordance with the revised CONSORT 2010 statement . The staThe primary analysis will be undertaken using generalized linear modelling to compare the effectiveness of HIS-UK against usual care in reducing chlamydia test positivity at T6. The analysis will be repeated at T12 to examine longevity of intervention effect. Analyses will be extended to include the investigation of possible intervention moderators and mediators, the exploration of process measures , and the identification of which young men most benefit from ProHIS and e-HIS . Similar comparative analyses using the secondary process indicators collected at T0-T12 will be undertaken using generalized linear mixed modelling to allow for the analysis of repeated measurements over time and comparison between the study arms. To meet our intention-to-treat principles analysis, withdrawals and protocol violators will be analyzed in their arms as randomized.To assess the costs and benefits of HIS-UK (delivered via ProHIS and e-HIS) compared with usual care, both a within-trial analysis and a model-based economic analysis will be undertaken. The main within-trial economic analysis will assess cost-effectiveness based on incremental cost per quality adjusted life year gained at 6 months, with a secondary analysis of cost per case of chlamydia avoided at 6 months, reflecting the primary outcome of the trial; this analysis will then be repeated to measure cost-effectiveness over a 12-month period. Initially, the base case analysis will be framed in terms of a cost-consequence analysis for the 3 trial arms, and data will be reported in a disaggregated manner on the incremental cost and important consequences assessed in the trial.If the trial shows that ProHIS or e-HIS are effective in reducing chlamydia positivity and other condom use health behavior outcomes, compared with usual condom distribution care, there are likely to be important cost implications for the health care sector, the wider public sector, and for society as a whole. If deemed necessary, a decision-analytic model will be used to extrapolate costs and outcomes beyond the end of the trial and synthesize data on costs and outcomes from a range of sources . The eviThe results will be presented using cost-effectiveness acceptability curves to show the uncertainty surrounding the cost-effectiveness of the ProHIS and e-HIS interventions, for a range of thresholds for cost-effectiveness . Both deEthics approval for the randomized controlled trial has been obtained from the University of Southampton Research Integrity and Governance Committee, and the Health Research Authority South Central Oxford B Committee (REC reference: 19/SC/0486).Funding was secured in February 2019, and recruitment was commenced in March 2020; however, due to the COVID-19 pandemic, recruitment was halted in April 2020. As a result, the study was adapted to reduce clinical contact between recruitment site staff and participants (see “Post–COVID-19 Amendments”), and subsequently reopened to recruitment in July 2021, with a planned participant recruitment period of 30 months and a 12-month participant follow-up.The hypothesized main findings of this trial are that the HIS-UK intervention (delivered by either ProHIS or e-HIS) will reduce chlamydia test positivity among young men (16-25 years) by enhancing condom use experiences and improving correct and consistent condom use, as compared to usual condom distribution care. In addition, it is anticipated that the HIS-UK intervention, as compared to usual condom distribution care, will be cost-effective.The strengths of this protocol include the use of a randomized controlled trial design, the targeted large sample, and the use of chlamydia biomarker testing. A limitation of the design is the restriction of participants to men aged 16-25 years. Future directions include adapting and evaluating the intervention among women and men of a broader age range.The results from this study will be presented at scientific conferences, published in peer-reviewed journals, and shared on social media. We will also share our findings with key stakeholders, including young men, clinicians, and commissioners of sexual health services.This is the first randomized controlled trial evaluation of the HIS-UK intervention with young men. If the intervention is effective and cost-effective, this could have a positive impact on NHS services by reducing the incidence of STI rates and relieving pressure on staff time, financial costs, and other resources in the treatment of STIs. The intervention may also encourage sexual health services to adopt additional digital technologies and ultimately improve access to such services for young people, decreasing health inequalities engendered by fear of stigmatization.


Objective: To evaluate the continuity and completeness of electronic health record (EHR) data, and the concordance of select clinical outcomes and baseline comorbidities between EHR and linked claims data, from three healthcare delivery systems in Taiwan.Methods: We identified oral hypoglycemic agent (OHA) users from the Integrated Medical Database of National Taiwan University Hospital (NTUH-iMD), which was linked to the National Health Insurance Research Database (NHIRD), from June 2011 to December 2016. A secondary evaluation involved two additional EHR databases. We created consecutive 90-day periods before and after the first recorded OHA prescription and defined patients as having continuous EHR data if there was at least one encounter or prescription in a 90-day interval. EHR data completeness was measured by dividing the number of encounters in the NTUH-iMD by the number of encounters in the NHIRD. We assessed the concordance between EHR and claims data on three clinical outcomes . We used individual comorbidities that comprised the Charlson comorbidity index to examine the concordance of select baseline comorbidities between EHRs and claims.Results: We identified 39,268 OHA users in the NTUH-iMD. Thirty-one percent  of these users contributed to the analysis that examined data continuity during the 6-month baseline and 24-month follow-up period; 31%  of the 12,296 users had continuous data during this 30-month period and EHR data completeness was 52%. The concordance of major cardiovascular events, nephropathy-related events, and heart failure admission was moderate, with the NTU-iMD capturing 49–55% of the outcome events recorded in the NHIRD. The concordance of comorbidities was considerably different between the NTUH-iMD and NHIRD, with an absolute standardized difference >0.1 for most comorbidities examined. Across the three EHR databases studied, 29–55% of the OHA users had continuous records during the 6-month baseline and 24-month follow-up period.Conclusion: EHR data continuity and data completeness may be suboptimal. A thorough evaluation of data continuity and completeness is recommended before conducting clinical and translational research using EHR data in Taiwan. Electronic health records (EHRs) are rich in clinical information and may serve as a good data source for research. However, as patients may seek care at different delivery systems for different conditions, the EHRs from a single medical center or delivery system may only contain a fraction of patients’ health data. Although EHR data reflect clinical decisions and practice patterns within the real-world setting, ensuring their completeness is critical for generating valid evidence . There iTo our knowledge, there had been no comprehensive evaluation of EHR data completeness in Taiwan. In Taiwan, the National Health Insurance (NHI) provides universal coverage, and patients are free to choose their healthcare providers and delivery systems for both primary and specialty care . TherefoThis study evaluated EHR data completeness and continuity in patients prescribed an oral hypoglycemic agent (OHA) in three healthcare delivery systems in Taiwan. As in prior studies , we measThe study included three EHR databases from three delivery systems: the Integrated Medical Database of National Taiwan University Hospital (NTUH-iMD), the Chang Gung Research Database (CGRD) from Chang Gung Memorial Hospitals (CGMHs), and the EHR database of National Cheng Kung University Hospital (NCKUH). The NTUH-iMD was used in the primary analysis, which evaluated EHR data continuity, data completeness, and EHR-claims data concordance. A secondary analysis was performed using all three EHR databases to assess EHR data continuity across different healthcare delivery systems. Data between June 2011 and December 2016 of these EHR databases were used for this evaluation.We chose these three delivery systems because they reflect different levels of hospital care and geographical diversity. The NTUH  and NCKUH  are two medical centers located in the Northern and Southern metropolitan areas of Taiwan respectively. The CGMHs  consist of two medical centers, two regional hospitals, and three district hospitals located in different cities in Taiwan. The three EHR systems cover 17% of the reimbursed healthcare services in Taiwan’s NHI program, which insures more than 99% of the 23 million population and includes all reimbursement-related data since 1997 .The three EHR databases contain data on diagnosis, procedure, prescription and dispensing, and date of encounter. Diagnosis was recorded by International Classification of Diseases (ICD) Clinical Modification codes. The Ninth Revision of the ICD Clinical Modification codes (ICD-9-CM) were used before 2016, and the Tenth Revision of the ICD Clinical Modification codes (ICD-10-CM) were implemented in the beginning of 2016. Each hospital has its own coding systems for procedures and prescription medications, but all these codes can be mapped to standard codes for NHI reimbursement. We converted available EHR data in the three systems to a standard, common data model based on the Sentinel Common Data Model to facilitate multi-site data quality evaluation . Code maIn addition to the EHR databases, the nationwide claims database, National Health Insurance Research Database (NHIRD), was used to assess the completeness of NTUH-iMD. We considered the NHIRD as the reference for the evaluation of EHR data completeness because the NHI program in Taiwan provides universal coverage and comprehensive health services. Like other claims databases, the NHIRD does not capture out-of-pocket health services. However, OHAs and most care are fully covered by the NHI program, so misclassification due to out-of-pocket health services should be minimal in the present study. The NTUH-iMD was linked to the NHIRD by national identity numbers unique to each patient through a third party. NHIRD contains information on diagnosis, procedures, and prescription medication reimbursed by the NHI; up to three outpatient or five inpatient diagnosis codes per encounter are available in the NHIRD. We used the de-identified linked data from June 2011 to December 2016 to evaluate the completeness and concordance of the EHR data, using the linked claims data as the reference. The individual study protocols were approved by the institutional review boards of NTUH (201906066RINB), NCKUH (A-ER 108-097), and Chang Gung Medical Foundation (201900108B1).The study cohort was individuals aged ≥20 years who were prescribed at least one OHA see . HealthcWe first assessed data continuity in the NTUH-iMD. Data continuity was examined in successive 90-day intervals before and after the index date, defined as the date of the first OHA prescription recorded in a given OHA user identification period . We defiWe examined two pre-index  periods: 6 and 12 months, and three post-index  periods: 12, 24, and 60 months. To obtain sufficient data to assess data continuity for different lengths of pre-index and post-index periods, the OHA user identification period, the data continuity assessment period, and the number of eligible OHA users varied by analysis. We evaluated EHR data completeness by comparing the encounter records in the NTUH-iMD to the encounter records in the linked NHIRD among OHA users with at least 6 months of baseline and 24 months of follow-up data. Data completeness was measured by the mean proportion of encounters captured (MPEC), calculated by dividing the number of encounters in the NTUH-iMD by the number of encounters in the linked NHIRD :MPEC= or stroke. Nephropathy-related events were defined by a primary or secondary diagnosis of these events in any outpatient or inpatient records. The heart failure outcome was defined by hospitalization with a primary diagnosis of heart failure. All outcome events were assessed during the 24-month follow-up period. The diagnoses codes used to identify the outcomes are presented in We selected these outcomes to represent a mix of events with different characteristics. We hypothesized that patients with MI or stroke, both of which are acute conditions, were more likely to have an emergency department or inpatient care in hospitals different than the ones they visit regularly for routine care. Since MI and stroke are potentially life-threatening events, patients are more likely to receive care from the nearest hospital when an event occurs, rather than being admitted to their routine care hospitals. Therefore, a low event capture rate was expected for cardiovascular events in the EHRs. Diabetes-associated nephropathy often develops gradually and is more likely to be diagnosed in outpatient visits; therefore, EHR data completeness was expected in patients with regular visits to the delivery system. Unlike acute MI and stroke, heart failure-related admission can be planned in Taiwan. We hypothesized that planned admissions were more likely to be captured by the EHRs studied.Two indicators were calculated to examine the outcome concordance: EHR-claims outcome agreement and EHR outcome representativeness. We defined outcome agreement between the EHRs and the linked claims data as the number of patients whose outcome status were coded in the same way in both the EHRs and the linked claims database divided by the total number of patients who appeared in both databases. A patient defined as having the outcome in both the EHRs and the claims database, or not having the outcome in either database, would suggest high agreement. We defined outcome representativeness in the EHR database as the number of patients classified as having the outcome in both the EHRs and the claims database divided by the number of patients classified as having the outcome in the claims database. These two measurements reflected minimum outcome concordance as we did not require the event to appear on the same date between the EHRs and the linked claims database. The definitions of these two indicators are shown in We compared the baseline comorbidity profile during the 6-month baseline period between the NTUH-iMD and the linked NHIRD in OHA users with continuous EHR data for at least 6 months at baseline and 24 months following the index date. We examined the individual comorbidities that comprised the Charlson comorbidity index . BaselinTo further understand whether EHR data continuity varied among healthcare delivery systems, a secondary analysis was conducted to compare the data continuity of three EHR databases from the NTUH, NCKUH, and CGMHs. To maintain consistent sample selection criteria across the health delivery systems, we first applied the sample selection criteria of the NCKUH dataset  on the NTUH-iMD and CGRD data and then selected OHA users for comparisons across the healthcare delivery systems.We identified 39,268 OHA users in the NTUH-iMD between June 2011 and December 2016. As expected, the sample size decreased as we lengthened the baseline period due to the lower number of patients who could potentially be captured by the EHR system . We retaSimilarly, the sample size decreased as we lengthened the follow-up period but the attrition was more modest: sample retention was 91%, 82%, and 51% in the analysis that examined 12-, 24-, and 60-month follow-up, respectively ; the corExamining data continuity during the baseline and follow-up period simultaneously reduced the sample size significantly. For example, only 26% of the original sample of 39,268 OHA users contributed to the analysis that examined data continuity during the 12-month baseline and 24-month follow-up period; data continuity was 25% within this patient subset.In addition, when considering baseline and follow-up data continuity together, we found that requiring longer baseline data continuity slightly improved the continuity of follow-up data but reduced the sample size considerably . The pron = 8,222), and 33% (IQR 21–49%) for OHA users with at least 24 months of continuous follow-up data . For patients with continuous data for the entire 30 months , the median MPEC was 41% (IQR 20–67%) for the 6-month baseline period, 48% (IQR 27–78%) for the 24-month follow-up period, and 52% (IQR 29–80%) for the entire 30-month period. The median MPEC increased to 50% in the 6-month baseline period (IQR 41–89%), 66% in the 24-month follow-up period (IQR 46–99%), and 71% (IQR 48–99%) in the entire 30-month period for OHA users when only hospital-level claims were counted.In the NTUH-iMD dataset that linked to the NHIRD claims data, the median MPEC was 42%  for OHA users with at least 6 months of continuous baseline data , about 3% of them had a T2DM diagnosis in the NTUH-iMD during the 6-month baseline period, but approximately 68% of them had a T2DM diagnosis in the linked NHIRD.The outcome agreement and outcome representativeness of the NTUH-iMD compared to the linked NHIRD for select clinical outcomes are summarized in There were 38,134, 155,738, and 23,719 OHA users with at least one T2DM diagnosis identified in the NTUH-iMD, CGRD, and NCKUH EHR database, respectively, between June 2011 and December 2016 –6. SimilSample retention patterns were similar between the NTUH-iMD and the CGRD during follow-up. Sample retention was 83% in the analysis that examined 24-month follow-up continuity in both the NTUH-iMD and the CGRD; data continuity was 70% in these two databases. In the NCKUH EHR database, the proportion was 80% for sample retention and 73% for data continuity.When considering baseline and follow-up data together, only 31% of the original sample contributed to the analysis that examined data continuity during the 6-month baseline and 24-month follow-up period in the NTUH-iMD and the CGRD, while 77% of the sample remained in the NCKUH EHR database. Data continuity for the entire 6-month baseline and 24-month follow-up period was approximately 30% in the NTUH-iMD and CGRD and 55% in the NCKUH EHR database. Extending the baseline from 6 to 12 months with a 24-month follow-up decreased the sample size without improving data continuity.To the best of our knowledge, this is the first study in Taiwan that provides a comprehensive evaluation of EHR data from multiple large healthcare delivery systems. We adopted the perspective of an epidemiologist or real-world data scientist who would like to utilize the data for research and clinical investigation. In this study, less than half of the OHA users in the NTUH-iMD had continuous records in the 6-month period before their first recorded OHA prescription, whereas 78% and 70% had continuous 12 and 24 months of follow-up data, respectively. The median MPEC was 52% for patients with at least 6-month baseline and 24-month follow-up data in the NTUH-iMD. The NTUH-iMD only captured half of the cardiovascular events, nephropathy-related events, and heart failure admissions recorded in the linked NHIRD database. The concordance of important baseline comorbidities was also low. Therefore, the EHRs may not fully capture potentially important clinical outcomes or covariates, which may limit their applications in certain research studies. The fact that only 3% of OHA users had a diagnosis of T2DM in the NTUH-iMD (vs. 68% in the linked claims database) within 6 months before their first observed OHA prescription record also highlights the challenge of identifying complete baseline or medical history in a single EHR database.We observed low EHR data completeness in this study. Although medical records were collected and processed for NHI reimbursements, EHR and claims data serve different purposes for healthcare organization administration. EHR databases often accommodate more transactions and changes than claims as they reflect the complexity and dynamics of clinical practice. The diagnoses in the EHRs can be re-coded to meet the reimbursement payment schemes. On the other hand, only three outpatient or five inpatient diagnosis codes are available in the NHI claims data; diagnosis codes that were placed at a lower position in the EHRs might not be recorded in the claims.We faced considerable sample loss when creating a patient cohort with continuous care using the EHR databases studied. Requiring 6-month baseline and 24-month follow-up continuous data left only 10% of the initial sample size in the NTUH-iMD and the CGRD and 42% of the initial sample size in the NCKUH EHR. This suggests that an EHR-based study of diabetes might suffer from issues such as left-censoring, loss to follow-up, or out-of-system care. Ideally, patients should only visit one healthcare delivery system for severe or complicated diseases under the shared-care and dual-referral system in Taiwan. In reality, however, patients can easily choose different healthcare providers and visit a specialist without referral under the NHI. The NTUH and CGMH are medical centers located in the metropolitan areas with a high density of medical institutions, while the NCKUH is located in a city with a lower density of medical institutions. With low geographic or financial burden, patients with multiple diseases are likely to go to different healthcare delivery systems for different conditions, which may be reflected in our finding of low data continuity recorded in the NTUH-iMD and CGRD. Given the high sample attrition and considerable variation in sample loss across healthcare delivery systems, it is important for studies to assess the continuity of patient care when conducting EHR-based research for long-term outcomes.A trade-off of high EHR data continuity and completeness is potential selection bias and may affect the generalizability of study results . As EHR This study had some limitations. First, the differences in clinical practices regarding disease coding and treatment may lead to data variation across the three EHR systems. Second, this study only focused on OHA users, so the results may not be generalizable to other patient populations. Third, due to the data governance policies, we could only link one EHR database, the NTUH-iMD, to the claims data. Further evaluations of EHR data completeness in different delivery systems are necessary. Third, although claims databases capture medically attended events that occur across multiple delivery systems, only three outpatient or five inpatient diagnosis codes are available in the NHIRD, limiting its ability to capture all diagnoses. Finally, the EHR data assessed in this study were from 2011 to 2016, which covered the initial period of coding transition from ICD-9-CM to ICD-10-CM. Further studies may evaluate EHR data completeness with more updated data.In conclusion, EHR data continuity for patients receiving OHAs was relatively low and varied across healthcare delivery systems. The degree of out-of-system care may affect the appropriate length of study follow-up, as well as data completeness of outcomes and baseline comorbidities. The study results suggest that careful evaluation of data completeness is necessary to ensure the validity of real-world evidence generated by EHR data.


The coronavirus disease 2019 (COVID-19) pandemic has impacted healthcare systems across the world. Chest radiography (CXR) can be used as a complementary method for diagnosing/following COVID-19 patients. However, experience level and workload of technicians and radiologists may affect the decision process. Recent studies suggest that deep learning can be used to assess CXRs, providing an important second opinion for radiologists and technicians in the decision process, and super-human performance in detection of COVID-19 has been reported in multiple studies. In this study, the clinical applicability of deep learning systems for COVID-19 screening was assessed by testing the performance of deep learning systems for the detection of COVID-19. Specifically, four datasets were used: (1) a collection of multiple public datasets (284.793 CXRs); (2) BIMCV dataset (16.631 CXRs); (3) COVIDGR (852 CXRs) and 4) a private dataset (6.361 CXRs). All datasets were collected retrospectively and consist of only frontal CXR views. A ResNet-18 was trained on each of the datasets for the detection of COVID-19. It is shown that a high dataset bias was present, leading to high performance in intradataset train-test scenarios (area under the curve 0.55–0.84 on the collection of public datasets). Significantly lower performances were obtained in interdataset train-test scenarios however (area under the curve > 0.98). A subset of the data was then assessed by radiologists for comparison to the automatic systems. Finetuning with radiologist annotations significantly increased performance across datasets (area under the curve 0.61–0.88) and improved the attention on clinical findings in positive COVID-19 CXRs. Nevertheless, tests on CXRs from different hospital services indicate that the screening performance of CXR and automatic systems is limited (area under the curve < 0.6 on emergency service CXRs). However, COVID-19 manifestations can be accurately detected when present, motivating the use of these tools for evaluating disease progression on mild to severe COVID-19 patients. Reverse transcription polymerase chain reaction (RT-PCR) is the reference standard method in COVID-19 diagnosis but can present significant turnaround time and remains subject to potential shortage. Lateral flow tests (LFT), which allow much faster turnaround time, suffer however from limited and highly variable sensitivities (38.32–99.19%)2. Furthermore, the development of new variants of SARS-CoV-2 can have an impact on the sensitivity of both RT-PCR and LFTs3.First identified in late 2019, the coronavirus disease 2019 (COVID-19) is caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). While the majority of cases causes only mild symptoms, COVID-19 can cause difficulty breathing, pneumonia, acute respiratory distress syndrome (ARDS) and ultimately death. The SARS-CoV-2 can be easily transmitted, which makes the identification of infected individuals of the utmost importance in the containment of the pandemic4. However, most COVID-19 patients do not develop pneumonia and present a normal CXR, which significantly lowers the screening value of CXR5. Nevertheless, it has been proposed that people with severe respiratory symptoms could be quickly screened with CXR to distinguish between COVID-19 and other pathologies6. Considering that COVID-19 manifestations on CXR can often be subtle, significant experience is needed for an accurate reading of these images. Nevertheless, a high proportion of CXRs are read by technical staff, rather than by experienced radiologists7, and the pandemic has likely worsened this issue due to staff shortages and an even higher numbers of radiological exams. Automated image analysis for the detection of CXR radiological features could thus play a significant role in providing a 2nd opinion to support clinical decisions in the triage of COVID-19 patients.Given the involvement of the respiratory airways in COVID-19, chest radiography (CXR) was initially proposed as an alternative screening method. While there is no single radiological feature that is indicative of COVID-19, changes include ground glass, coarse horizontal linear opacities and consolidation, most often with bilateral involvement8. Figure 9. We focus particularly on early works corresponding to initial response to the worldwide pandemic outbreak, period in which the inclusion of these type of automatic tools would have allowed to significantly reduce the workload of specialists and where high performance claims were made. Please refer for example to9 for a descriptive summary of these methods. As shown in Fig. The scientific community has responded quickly to the challenge and numerous studies have been published in literature on automated COVID-19 diagnosis in CXR10. The authors compare the performance of their method with a ResNet50 and a VGG19, showing that their solution performs better while requiring less computational complexity. In particular, an overall accuracy of 93.3% and a COVID-19 positive predictive value of 98.9% are reported. Despite Wang et al.’s success using a custom architecture, other authors have reported high performances using pre-existing architectures. For instance, Apostolopoulos et al. tested multiple CNN architectures in both 2-class (COVID-19 and not COVID-19) and 3-class  scenarios, obtaining a 98.75% accuracy using a 2-class VGG1911 architecture12. Ozturk et al. proposed a variation of the DarkNet-1913 architecture, DarkCOVIDNet, and, similar to Apostolopoulos et al., tested both 2-class and 3-class scenarios, obtaining an accuracy of 98.08% in binary classification14.Most methods rely on pre-trained deep networks, namely ResNet or DenseNet-based that have as main goal distinguishing COVID-19 patients from non-COVID-19 patients. Since COVID-19 shares radiological manifestations with pneumonia, a large number of methods approach this problem as a 3 class task. Other authors prefer to design dedicated networks for the task. A notable example is COVID-Net, one of the first solutions to be proposed for automatic COVID-19 screening15, 508 CXRs were interpreted by 4 radiologists and it is shown that sensitivity and specificity change as the disease progresses, from 55% to 79% sensitivity and from 83% to 70% specificity at 16 and COVIDx10, which were compiled by combining pre-pandemic public datasets  with more recent COVID-19 positive CXRs, mostly extracted from academic articles and online publications. This could cause automatic methods to learn non-radiological features such as image quality, acquisition settings and equipment, laterality or other markers, hospital system of origin, etc.. Because the positive and negative classes belong to different sources, these spurious differences highly correlate with the classes, easing the learning of shortcuts rather than actual radiological features. This was highlighted by DeGrave et al.17, where it was shown that a system trained on a dataset from different sources with an internal test area under the curve (AUC) of 0.992 falls to 0.76 on an external dataset.Even though these results are extremely promising, the performances reported are significantly different from what has been reported for visual reading by radiologists. In Stephanie et al.Critically comparing the intra- and inter-dataset performance in both public and in-house datasets of a deep learning model trained following similar methods to the ones proposed during the worldwide pandemic outbreak. In particular, we show that performance claims on the literature are overconfident due to dataset bias;Building on the last topic, we show that using annotations from medical experts can significantly mitigate dataset bias, allowing the model to obtain a similar COVID-19 screening performance based solely on radiological manifestations of the disease to radiologists;To help the development and validation of future algorithms, we make publicly available the radiologists’ annotations on the public datasets.Figure The goals of this study are thus twofold: (i) to develop an automatic COVID-19 detection method in CXR to serve as a 2nd opinion to support clinical decisions in the triage of COVID-19 patients; (ii) to assess the clinical applicability of deep learning systems for COVID-19 screening using CXR images. In particular, we contribute to the development of more robust COVID-19 automatic detection methods by:Normal, Pathological (not COVID-19) and COVID-19. Ground truth labels for all CXRs were obtained from the ground truth available on each dataset. The COVID-19 label corresponds to a positive SARS-CoV-2 RT-PCR result and not necessarily to the presence of radiological features of COVID-19. Table Three public datasets and one private datasets were used in this study. All data was collected retrospectively. Note that for the public datasets, the exact criteria for inclusion in the dataset and referral for CXR are unknown. For all datasets, the same processing and criteria were applied to ensure uniformity. Only frontal CXRs—postero-anterior (PA) and antero-posterior (AP)—were included and CXRs were divided into 3 classes: 14, combining pre-pandemic public datasets with recent COVID-19 positive CXRs, mostly extracted from academic articles and online publications. Normal and Pathological CXRs were obtained from the CheXpert18, ChestXRay-819 and Radiological Society of North America Pneumonia Detection Challenge (RSNA-PDC)20 datasets. COVID-19 positive cases and a residual amount of Normal and Pathological cases were extracted from online repositories of CXRs, namely the COVID-19 IDC16 and COVIDx10 datasets. Further CXRs were obtained by manual extraction of images published online on Twitter and the Sociedad Española de Radiologia Médica (SERAM) website. Finally, COVID-19 positive CXR were obtained from the COVID DATA SAVE LIVES dataset, made available by the HM Hospitales.The Mixed dataset is a combination of multiple public CXR datasets, similar to those used in most early publications on automatic COVID-19 detection in CXRGiven that there is significant overlap in some of the datasets included in the Mixed dataset, repeated images were excluded.Normal and Pathologic (non-COVID-19) cases originate from different hospitals and from a different timespan than COVID-19 positive cases. Normal and Pathological CXRs were obtained from the BIMCV-COVID19-PADCHEST, a subset of the larger BIMCV-PadChest21 public dataset which is composed of CXRs from the Hospital Universitario San Juan De Alicante, Alicante, Spain from 2009 to 2017. COVID-19 positive CXRs were obtained from the BIMCV-COVID-19+22 public dataset which is composed of CXRs from 11 public hospitals in the Valencian Region, Spain from 6th February 2020 to 1st April 2020.The Banco digital de Imagen Medica de la Comunidad Valenciana (BIMCV) dataset is composed of CXRs from public hospitals in the Valencian Region, Spain and is made available by the BIMCV. While this dataset is composed uniquely of images from BIMCV, 23. All CXRs were acquired with the same equipment and consist entirely of PA views. All patients underwent a RT-PCR test within 24 hours of the CXR. Manual selection of CXRs was performed to balance the positive and negative RT-PCR test results. Note that because no information except for RT-PCR results is available, RT-PCR negative patients cannot be labelled exclusively as either Normal or Pathological.The COVIDGR dataset contains CXRs from the San Cecilio University Hospital at Granada, Spain and is made available by the Andalusian Research Institute in Data Science and Computational Intelligence (DaSCI)COVID-19 whereas all other patients were labelled as Normal/Pathological. Note that, similarly to COVIDGR, RT-PCR negative patients cannot be labelled exclusively as either Normal or Pathological. Given that the dataset encompasses all patients who were suspected for SARS-CoV-2 infection and no manual selection of CXRs was performed, this dataset is the most representative of a clinical setting.The CHVNGE dataset contains CXRs collected retrospectively at the Centro Hospitalar de Vila Nova de Gaia e Espinho (CHVNGE) in Vila Nova de Gaia, Portugal between the 21st of March and the 22nd of July of 2020. All data was acquired under approval of the CHVNGE Ethical Committee and followed all relevant guidelines and regulations. Informed consent was waived by the CHVNGE Ethical Committee given that all data was anonymised prior to any analysis. All CXRs of patients who underwent an RT-PCR test within the study time frame were extracted. Both PA and AP views were included and no other exclusion criteria were applied. The information regarding the CXR equipment used for acquisition was preserved as different CXR equipments were used in each hospital service. CXRs obtained in the emergency department (Samsung DX) can thus be analysed separately from those who were in inpatient services  and in intensive care units (FUJI CR). CXRs from patients with a positive RT-PCR test were labelled as Normal, Not indicative of COVID-19 , Indicative of COVID-19 and Undetermined. The Indicative of COVID-19 class was defined as CXRs where the patient presented findings indicative of COVID-19, namely bilateral pulmonary opacities of low/medium density. The Undetermined class was defined as CXRs where the patient presented findings that could be indicative of COVID-19 but which could also be indicative of another condition, namely unilateral lung opacities, diffuse bilateral opacities of ARDS pattern or diffuse reticular opacities. The Not indicative of COVID-19  class was defined as CXRs where the patient presented findings indicative of any other pathology except for COVID-19. CXRs where the patient presented medical devices were classified as Normal if the underlying pathology was not visible. Additionally, CXRs without sufficient quality for visual assessment by the radiologists due to bad image quality, patient positioning or any other factors could be labelled as Compromised for exclusion.In order to evaluate the performance of radiologists in the detection of COVID-19 radiological features in CXR, manual annotation of a subset of CXR images from each dataset was performed by two radiologists using an in-house software. The software presented CXRs from a randomly selected subset and allowed for window center/width adjustment, zooming and panning. Radiologists were asked to label CXRs into one of 4 classes: Manual labelling of CXRs was performed in two stages. First, both radiologists independently classified each CXR. CXRs where the two radiologists disagreed were then selected for the second stage where the two radiologists assessed the CXRs together to achieve consensus. At no point were radiologists given access to the ground truth label, RT-PCR results or any other information besides the CXR image.24 architecture for the detection of written labels in CXRs. For this purpose, 317 CXRs were randomly selected from the Mixed dataset and bounding boxes were manually drawn around all written labels. The network was then trained on the 317 CXRs (733 bounding boxes). Previous to CXR annotation by the radiologists, all CXRs were visually inspected and any missing or incorrect bounding boxes were corrected manually. All annotations on the public datasets are available at 10.25747/342B-GF87.To ensure that written information present in the CXR image  did not bias the annotation, all written labels were blacked out during before annotation. This was done in a semi-automatic way using a YOLOv325 was used in all experiments. While larger networks such as DenseNet121 have shown excellent performance in CXR disease classification18, the smaller ResNet18 was chosen to try to reduce overfitting to the limited data in the COVID-19 class and the inherent bias of the existing datasets.For automatic COVID-19 detection in CXR, an 18-layer deep residual neural network architecture (ResNet18)25, except that the input is a 1-channel gray image (512 × 512 pixels) and the number of output nodes is the number of classes being considered.The ResNet18 architecture used was identical to the architecture proposed in He et al.26:C is the number of classes, c is the ground truth class and c. To balance the effect of different classes and avoid bias, class weights N is the total number of images in the training set and c. Model optimization was performed using Adam27 with a learning rate of 0.0001 and a batch size of 24. These values were chosen empirically based on previous experiments and hardware capacity. Given the large number of images in the Mixed dataset, an epoch was defined as 1200 batches (approximately one tenth of the dataset) and all models were trained for a maximum of 100 epochs with a patience of 10 epochs as an early stopping criterion.The loss function used during training was the weighted binary cross-entropyi.e. when a model trained on dataset A is tested on dataset B, the full dataset B is used for testing.To estimate the performance of the predictive model in practice and avoid a possible bias due to random division of the data, a 5-fold cross-validation scheme was performed during training and testing. Folds were constructed through random CXR selection so that the class and dataset distributions in each fold are similar to the full dataset and so that all CXRs from each patient are placed in the same fold. In interdataset settings, A total of 2,442 CXRs were selected for annotation by the two radiologists. Of these, 1256 belong to the Mixed dataset, 289 belong to BIMCV, 300 belong to COVIDGR and 597 belong to CHVNGE  were randomly selected for repeated annotation to determine intraobserver variability. The repeated images were mixed in with new images in a 1:10 ratio during annotation. Radiologists were unaware that repeated images were being introduced to avoid bias.During annotation, a total of 77 CXRs (25 28 (the weights in the first layer were taken from the red channel on the pretrained model). The model was first trained for binary Normal vs Not Normal classification and only then trained with three output nodes corresponding to the Normal, Pathological and COVID-19 classes where the Pathological class included all classes except the Normal and COVID-19 classes. This two-step training strategy aims at leading the model to learn CXR-related features prior to learning COVID-19-related features, increasing feature relevance while reducing overfitting. This model was trained using the Mixed dataset to allow a direct comparison to previous studies on COVID-19 detection in CXR and will be referred to as The network was first initialized with weights from an Imagenet pretrained modelIn order to estimate a best case scenario in terms of performance for each dataset, 17. In order to decrease this effect and improve the learning of radiological features, a pseudolabelling strategy29 was implemented. This was performed by obtaining the predictions of the model trained on the Mixed dataset on all images of the same dataset. All images with predicted probability lower than 0.9 were then removed to include only high confidence predictions and the predicted class was then used as ground truth label for retraining the model. This model is referred as Given the characteristics of the Mixed dataset and the high correlation between dataset sources and classes, it is expected that COVID-19 during training can introduce high levels of noise and promote the learning of shortcuts for classification, which do not represent COVID-19 manifestations. Using radiologist annotations during training avoids this issue as COVID-19 positive CXRs without manifestations will be presented to the model as Normal CXRs during training, enforcing the learning of features that represent COVID-19 manifestations. For this purpose, Undetermined, the principle of precaution was applied: given the definition of this class as CXRs where the patient presented findings that could be indicative of COVID-19 but which could also be indicative of another condition, these CXR were considered to belong to the COVID-19 class. CXRs marked as Compromised were discarded. This model is hereinafter referred to as Given that not all COVID-19 positive cases present manifestations on CXR, considering these cases as 30:TP is the number of true-positive detections, FN the number of false-negatives, FP the number of false-positives and FN is the number of false-negatives. COVID-19 or not COVID-19). Given the definition of the Undetermined class, agreement was computed considering as positives either only the Indicative of COVID-19 class (C) or both the Indicative of COVID-19 and Undetermined classes (C+U). Where applicable, the statistical significance of the difference between the radiologist annotations and the ground truth labels was tested using an adaptation of the McNemar test  and recall whereas the intra- and interobserver variability were evaluated in terms of accuracy (Acc.) and Cohen’s kappa 33. The ECE is a summary of the Reliability diagram, which plots the expected accuracy as function of the predicted class probability. Briefly, all N test samples are binned probability-wise in B groups for which the accuracy b. Results are evaluated considering 10 bins.Model performance was evaluated with receiver operating characteristic (ROC) curve, specifically in terms of AUC. Confidence intervals were computed taking into account the achieved average performance for all folds. To further validate the models, Grad-CAM++34. When comparing AUCs across different test sets, the permutation test for continuous unpaired comparison of AUCs proposed in Venkatraman et al.35 was used. Note that when comparing a model to radiologist annotations, only the subset of CXRs annotated by radiologists was taken into account. Statistical significances computed across folds were fused according to Fisher’s combined probability test36 to obtain a single p-value and when performing multiple comparisons, statistical significance was considered after applying the Bonferroni correction37.The statistical significance of the differences in the performance of the models (and the radiologists) was performed according to the DeLong test, which allows for paired comparison of AUCsIndicative of COVID-19 (C) gives a higher average precision but with low recall, whereas including as positives CXRs marked as Undetermined (C + U) significantly increases recall, but at the expense of precision. Comparing across datasets, it can be seen that radiologists achieve the highest precision and recall on the COVIDGR and Mixed datasets, whereas the lowest performance is obtained for the CHVNGE dataset. Table Indicative of COVID-19, whereas other combinations could not be performed at power Figure Figure On the test set, it can be seen that intradataset train-test scenarios obtained the best results on all datasets, i.e. when training includes CXRs from the dataset used on testing human performance. On the other hand, the studied finetuning approaches helped to mitigate the overfit behaviour. Indeed, results suggest that revisiting cases where COVID-19 radiological manifestations are more evident helps the model converge to a feature representation that better encodes the radiological manifestations of the pathology. In fact, both The AUC differences found between inter- and intradataset train-test scenarios Table  corroborts Table , withoutThe hypothesis that finetuning with good quality labels improves the system’s reliability is further supported by the models’ calibration performance Table . InteresFinally, comparing the activation maps of Indicative of COVID-19 and Undetermined) is higher in images from intensive care units and inpatient services in comparison to the emergency department (Fig. The performance of both 9. These results have been replicated in this study on intradataset train-test scenarios (e.g. As discussed in the Introduction section, several methods for automatic COVID-19 diagnosis in CXR have been proposed in literature. Particularly in the beginning of the pandemic extremely high performances have also been reported (see Fig. In spite of the promising results obtained in this study, there are limitations that must be taken into account in the interpretation of results and future directions. As highlighted in this manuscript, the training of deep learning systems relies heavily on the available data and while this study includes data from several sources to achieve a good representation of multiple environments, the dataset which intends to represent clinical reality is limited. For one, it represents the reality of a single hospital system in Portugal, which may limit the reproducibility of these results in other hospitals. Although we believe that the performance differences reported on Table This study assessed the performance of a deep learning system for COVID-19 screening using CXR and compared it with expert radiologists. The detection of COVID-19 in CXR images is non-trivial due to the wide range of radiological manifestations associated with the infection. Consequently, radiologists tend to confound COVID-19 patients with other pathologies. Similarly to other recent studies, it was found that the performance reported for deep learning approaches is overconfident. Indeed, this study shows that the screening performance is not robust to changes in data origin. However, the results shown suggest that finetuning the model with labels provided by human experts allows the network to improve the quality and meaningfulness of the extracted features, improving explainability and reducing data bias.Results also suggest that the applicability of these systems for initial patient triage, when radiological manifestations of COVID-19 are minimal, is limited. However, when radiological manifestations of COVID-19 are present, these can be accurately detected and pinpointed by these tools. Although the achieved results are promising, there is still need to understand how well these findings translate to other time points/variants of COVID-19 and different clinical realities. Based on this study, future directions in this field should also focus on the use of deep learning systems for tracking the evolution of mild to severe COVID-19 infections, providing a robust 2nd opinion and thus contributing to mitigate the consequences of the pandemic.CheXpert:https://stanfordmlgroup.github.io/competitions/chexpert/ChestXRay-8:https://nihcc.app.box.com/v/ChestXray-NIHCCCOVID-19 IDC:https://github.com/ieee8023/covid-chestxray-datasetCOVIDx:https://github.com/lindawangg/COVID-NetRSNA-PDC:https://www.kaggle.com/c/rsna-pneumonia-detection-challengeSAVE LIVES:https://www.hmhospitales.com/coronavirus/covid-data-save-livesSERAM:https://seram.es/images/site/TUTORIAL_CSI_RX_TORAX_COVID-19_vs_4.0.pdfTwitter:https://twitter.com/ChestImagingBIMCV PADCHEST:https://bimcv.cipf.es/bimcv-projects/padchest/BIMCV COVID-19-PADCHES:https://bimcv.cipf.es/bimcv-projects/bimcv-covid19/COVIDGR:https://dasci.es/transferencia/open-data/covidgr-2/Manual annotations by the radiologists on the public datasets are publicly available at 10.25747/342B-GF87. The CHVNGE dataset was acquired under approval of the CHVNGE Ethical Committee as detailed in Section“Datasets”. However, the CHVNGE Ethical Committee determined that the data cannot be used beyond the purpose of the current study, and thus cannot be shared publicly with other institutions.The public datasets used in this study are available in the following repositories: