The initial-fit provided by the hearing aid manufacturer’s software is generally a display of measurement done in the ear simulators. The need for verification of hearing aid output and gain in the real ear using probe-microphone measurement to match the prescriptive target is highlighted. The objective of the study was to evaluate the difference in real-ear aided response (REAR), real-ear insertion gain (REIG), aided thresholds, articulation index (AI) and word recognition score (WRS) in quiet, with hearing aid programmed to NAL-NL1 first-fit and NAL-NL1 optimized-fit using the probe-microphone technique.In a repeated measure experimental design, 11 participants with a mean age of 41.09 (SD=±9.95) years having moderate and moderately-severe sensorineural hearing loss were tested monaurally in two aided conditions, with a 16-channel hearing aid programmed for manufacturer’s NAL-NL1 first-fit and optimized-fit to NAL-NL1 using probe-microphone verification. The REAR, REIG, aided threshold, articulation index and word recognition scores in quiet were obtained for both aided conditions.The REAR, REIG, aided threshold, AI and WRS in quiet were significantly better with the NAL-NL1 optimized-fit compared to manufacturer’s NAL-NL1 first-fit.The optimized-fit yields better audibility and improved word recognition in quiet. This supports best practice guidelines of many professional organizations regarding the use of probe-microphone measurement as the “Gold standard” for verification of hearing aid fitting, thereby providing better satisfaction and quality of life to hearing aid users. The probe-microphone measurement is accurate and reliable to verify the gain/output of the hearing matched to the given prescriptive formula like the National Acoustics Laboratory Nonlinear fitting procedures, NAL-NL1 and NAL-NL2 or the Desired Sensation Level input/output formula. Though, the probe-microphone technology is available for more than three decades, a small percentage of audiologists use this to verify the hearing aid fittings.The Independent Hearing Aid Fitting Forum (IHAFF), International Society of Audiology (ISA), American Academy of Audiology (AAA), and American Speech and Hearing Association (ASHA) consider the use of probe-microphone verification as one of the best practices for adult hearing aid fittings. It was found that the measured 2cc coupler output of the manufacturer’s initial-fit was significantly lower than the NAL-NL1 targets by as much as 10-15 dB,9. The real rear measurements of the initial-fit and the verified-fit from various studies were also consistent with the fact that the gain and output provided for the high frequencies are significantly lower with the initial-fit program-13. Hearing aid with initial-fit led to reduced speech recognition and subjective satisfaction. When fitted with programmed-fit to NAL-NL2 targets a significant improvement was observed in speech intelligibility index, speech recognition in quiet and noise, and self-perceived benefit-17.Many hearing aid practitioners prescribe hearing aids based on the manufacturer’s first-fit hearing aid fitting. When the first-fit is alone used, the accuracy of the target match can be low as it is carried out only in ear simulators/coupler and there is no verification of the gain of the hearing aid in real ear suggested that the NAL-NL2 first-fit would have used instead of manufacturer’s proprietary first-fit, the difference in audibility would have been present but not as poor as manufacturer’s proprietary first-fit. It probes further to investigate the difference in performance with first-fit of a given generic prescriptive targets and optimized-fit to those targets. The comparison of objective measures like aided threshold, speech intelligibility index (SII) or articulation index (AI) and word recognition in quiet between hearing aids fitted with first-fit and optimized-fit to NAL-NL1 target is sparse in literature compared to the studies on coupler measures. It is also important to note that when compared to NAL-NL2, the NAL-NL1 prescribes high gain with lower compression ratios for adults to maximize the speech intelligibility. It is important to explore the difference in aided performance with NAL-NL1 first-fit and optimized-fit to NAL-NL1 as it is one of the commonly used prescriptive targets with the adult population. Therefore, the current study aimed to investigate the aided performance with manufacturer NAL-NL1 first-fit and optimized-fit to NAL-NL1 target.Those studies compared the benefit with manufacturer’s proprietary first-fit and programed-fit to a given generic prescriptive target. Valente et al.To compare the real-ear aided response (REAR), the real ear insertion gain (REIG), aided thresholds, articulation index (AI) and the word recognition score (WRS) in quiet obtained with hearing aid set at the manufacturer’s NAL-NL1 first-fit and the optimized-fit to NAL-NL1 target using probe-microphone measurement.The objectives of the study were addressed through a cross-sectional within group repeated-measure experimental design.The study was carried out in 11 native Kannada speaking adults, in the age range from 23 to 55 years (mean age = 41.09±9.95). This sample size was found to be adequate in G*Power analysis using F-family of tests set to 0.80 power at an error rate of 5% for detecting the mean difference in aided threshold and word recognition between hearing aid fittings with a large effect size as demonstrated in this study.The inclusion criteria include (a) bilateral sensorineural hearing loss with either flat or sloping audiogram configuration, (b) pure-tone average (PTA) between 40 and 70 dB HL, (c) greater than 60% speech identification score in quiet, (d) naïve hearing aid user, (e) ‘A’ type tympanogram with reflex present at least in one frequency, (f) post-lingual onset of hearing loss. Those having any history or presence of middle ear infection, neurological (including auditory neuropathy and/or retrocochlear pathology) or psychological complaints were excluded. The experiments were carried out in the right ear if the hearing loss was symmetrical; and only in the better ear of the participants in case the hearing loss was asymmetrical. The demographic details of the participants were given in . Otoscopy was performed to ensure that there was no contraindication for hearing aid testing. Pure-tone audiometry was carried out using modified Hughson-Westlake procedure to establish the air-conduction and bone-conduction thresholds for octave frequencies from 250 Hz to 8000 Hz and 250 Hz to 4000 Hz respectively. Speech audiometry was carried out to obtain Speech Recognition Threshold (SRT) using the Kannada paired-word list and Word Recognition Score (WRS) at 40 dBSL (re: SRT) using phonemically balanced Kannada word identification test for adults. Immittance audiometry was carried out using 226 Hz probe-tone at 85 dB SPL to rule out the middle ear pathology.All procedures carried out in this study adhered to the institutional ethical guidelines for bio-behavioural research on human subjects. The study protocol was also approved by the ethical committee of the institute (ECC-Res.art/03/2020). Informed consent was obtained from the participants prior to the experiments. All testing was carried out in a single- or double- sound treated room situation having ambient noise levels within the permissible limits A 16-channel non-linear digital behind the ear (BTE) hearing aid with a fitting range of mild to severe hearing loss was used for aided testing. The hearing aid was coupled to the test ear with appropriately sized ear-tip. A personal computer with NOAH and the hearing aid specific software installed was used for hearing aid programming. HiPro was used as the interface between the hearing aid and programming computer. The participant’s details and audiogram were entered in the NOAH software. Initially, the hearing aid was programmed for first-fit with the NAL-NL1 prescriptive formula. The acclimatization level was set as ‘2’ as the participants were naïve users of hearing aids. The compression parameters were set according to the NAL-NL1 prescription. The other features like noise reduction, directional microphone was switched off to avoid the influence of these on speech identification testing. Then, the probe-microphone measures, like REAR and REIG, for this setting were carried out. The protocol for probe-microphone measurement is given in Each participant was instructed to sit comfortably with their head held straight forward during the measurements. The participant’s audiogram details were entered in hearing aid test system. The positioning of the probe-tube was made using visually assisted positioning method. A marking was made on the probe-tube at 30 mm from the probe-tube tip. The probe-tube was inserted into the ear canal with the marking on the probe-tube located at the intertragal notch. Thus, it was ensured that the tip of the tube was placed 5-6 mm farther to the tympanic membrane. Then, the real-ear measurement system was leveled to ensure that the input to the hearing aid was controlled across frequency. Followed by leveling, the real-ear unaided response (REUR) was obtained at 60 dB SPL. The hearing aid and ear-tip were inserted into the ear canal without disturbing the probe tube. The REAR and resulting REIG were measured at 60 dB SPL. These probe-microphone measurements were made with the manufacturer NAL-NL1 first-fit. This program was saved as ‘Program 1’ in the hearing aid which was the NAL-NL1 first-fit program.A second program ‘Program 2’ which was a duplicate of the ‘Program 1’ was created in the same hearing aid. The ‘Program 2’ was further programmed to match with the NAL-NL1 targets during verification. The frequency-gain adjustments were carried out until REIG closely matched to NAL-NL1 target at 60 dB SPL input level. The REAR and REIG data were recorded for the ‘Program 2’ which was the optimized-fit to NAL-NL1 target. The Program 1 (Aided 1 condition) and Program 2 (Aided 2 condition) were used to collect data on aided thresholds, AI, and aided WRS. The aided 1 and aided 2 conditions were randomized across participants to avoid the order effect.0 Azimuth. The unaided and the aided thresholds for the initial-fit (Aided 1) and optimized-fit (Aided 2) for octave frequencies from 250 Hz to 4000 Hz were obtained using warble tones. The aided thresholds were then plotted on AI audiogram given by Humes [23] which represents a speech frequency importance function weighted AI. The AI ranges from 0 to 1 or 0.99. The audiogram has a total of 33 dots covering the speech dynamic range and each dot in the audiogram contributes 0.03 times to the AI. The total number of dots below the aided threshold curve was counted and multiplied by 0.03 to get the AI for both aided conditions.The calibrated clinical audiometer was used to assess the unaided and aided performance in sound-field where the loudspeaker was placed 1 meter from the participant at 0. It consisted of four lists of 25 words each. The word lists were presented at 45 dB HL. It was made sure that the word lists were randomly presented across participants and test conditions. Each correct response was given a score of one and incorrect response was scored zero. The maximum word recognition score was 25.The unaided and aided WRS for the two aided conditions were obtained using phonemically balanced Kannada word identification test for adultsp2) and Cohen’s d for ANOVA and paired t-test respectively.The obtained data were analyzed using Statistical Package for the Social Sciences Version 20 software . Descriptive statistics was carried out to obtain the mean and standard deviation for REAR, REIG, aided thresholds, AI and WRS. The Kolmogorov-Smirnov Z-test of normality revealed that the data were in normal distribution (p>0.05), thus parametric inferential statistics was carried out. The effect size was reported using partial eta squared  equaling 0.954 and 0.951 respectively. Pair-wise comparison with the Bonferroni adjustment also indicated that the mean difference in the REAR and REIG between the two aided conditions was significant for all frequencies considered (p < 0.005).The result of repeated measures ANOVA revealed that this difference between the first-fit and optimized-fit was statistically significant for both REAR  and REIG , with an effect size of partial eta squared .The repeated measures ANOVA was carried out with aided threshold dependent variable and the test condition (first-fit and optimized-fit) as independent variable. The results revealed a significant difference in aided threshold between the two test conditions .-10. The difference in the individual ear canal acoustic characteristics would attribute to the reduction in gain measured with manufacturer first-fit setting. This significant reduction in gain especially in high frequency region with the first-fit would lead to a disadvantage in the audibility and speech recognition.The primary objective of the study was to compare the REAR, REIG, aided threshold, articulation index and word recognition score in two aided conditions, i.e., with the manufacturer NAL-NL1 first-fit and optimized-fit to NAL-NL1. There was a substantial difference in the measured REAR and the REIG between the manufacturer NAL-NL1 first-fit and optimized-fit to NAL-NL1 target which was more pronounced for the high frequencies. The findings of previous studies have also showed that the gain/output provided by the manufacturer’s first-fit will be significantly lesser than the generic prescriptive targets when measured both in 2 cc coupler and probe-microphone approach-16. In the present study, the high frequency advantage in the audibility with the optimized-fit was reflected in the advantage seen in the word recognition in quiet.The current findings showed that the audibility for soft sounds was improved with optimized-fit by 10 dB HL as revealed by the aided thresholds. There was more advantage in the high frequency with optimized-fit than the first-fit. Previous research suggests that a first-fit reduces the audibility of high-frequency signals, which could negatively impact speech recognition depicted that the derived speech intelligibility index (SII) from manufacturer’s default fitting was significantly lower than that of the programmed-fit to NAL-NL2 targets. They reported that the average SII obtained for moderate level sound with default fitting ranged from 0.46 to 0.56 compared to 0.65 for the NAL-NL2 fitting. Similar line of findings was observed with the AI in this study suggesting that the manufacturer setting will compromise the speech audibility and speech recognition.Sanders et al. This was because of the improvement in insertion gain at high frequencies as obtained in probe-microphone measurement for the programmed-fit. The gain for the high frequency was increased to 10 to 12 dB from the NAL-NL1 first-fit to match with the NAL-NL1 targets. It is well documented that the high frequency information is important for perception of majority of the consonants. The high frequency advantage with the programmed-fit resulted in improvement in word recognition score.The results on the word recognition revealed a mean programmed-fit advantage of 15% for a 60 dB SPL input level. This agrees with the findings reported by Valente et al., it is very rarely used as a verification tool in hearing aid fitting because of various limitations pertaining to it. A major drawback of aided threshold is the inability to verifying the non-linear hearing aids. The non-linear hearing aids vary the gain depending on the input level. Also, to obtain the true aided thresholds, the sound field audiometer must be calibrated, the patients must be seated with minimal head movements, and the involvement of the non-test ear must be eliminated. In the present study it was ensured that all these factors were controlled. The audiometer was calibrated, participants were instructed to maintain the position of the head throughout the testing, and the better ear was the test ear whenever there was an asymmetrical hearing loss. Though caution is to be exercised while interpreting the aided thresholds, it is one of the commonly used verification method for implantable hearing devices such as bone anchored hearing devices, middle ear implants and cochlear implants, where probe-microphone measures cannot be used as verification option.Though the current study tried to highlight the necessity of routine use of probe-microphone measurement in hearing aid fitting, the limitations in this research need to be discussed. Aided thresholds were used as one of the verification measures. Though aided threshold can draw the information on the aided audibility for soft level sound. However, the aided thresholds can be utilized for calculating the articulation index (AI) or speech intelligibility index (SII) and thereby predict the speech recognition-27. This is based on a same underline theory that speech recognition is related to the audible speech sounds. In this study, the AI was used utilizing the count-the-dot audiogram which is considered as one of the easiest clinically feasible method to estimate AI. AI was calculated from hearing threshold of the listener and the long-term average speech spectrum reaching the listener’s ear.The recognition of speech cannot be directly inferred from the aided thresholds. The SII calculation incorporated updated technical information on spread of masking, standard speech spectrum level, relative importance of various frequencies to speech intelligibility, speech level distortion, and hearing loss desensitization. Thus, make the SII to predict speech recognition appropriate for many conditions of use where the traditional AI does not apply. These procedures are automatized and incorporated in commercially available probe microphone equipment. The display provides SII for different speech level like soft (55 dB SPL), average (65 dB SPL), and loud levels (75 dB SPL) which enables the audiologist to counsel the patient on the effect of hearing loss on unaided and aided speech recognition different input levels.Modifications were made into the calculation of AI to make it more effective for predicting the speech recognition and it was renamed as SII in ANSI S3.5‐1997 standards,14, it can be assumed that other hearing aids models would also exhibit poor aided performance with manufacturer’s first-fit program. It is also important to evaluate perceptual effect of first-fit and optimized-fit with the other available generic formulae in different population, especially in pediatric population.This study was carried out using a single hearing aid model of a manufacturer. It is important to gather more evidence with other hearing aid makes and their default first-fit settings. Based on other reports showing the disadvantage of high frequency gain with manufacturer’s defaults with a variety of hearing aid models found that the measured REAR values were significantly different from the manufacturer predicted initial-fit REAR values in 41 participants with two audiogram configurations using two different input levels (50 and 90 dB SPL). This difference would also influence the aided response and subjective satisfaction in subjects with different degree and configuration of hearing loss which need to be investigated further. In addition, the long-term outcome or acclimatization and real-world benefit with given fittings need to be investigated.The participants recruited for the study were with moderate and moderately severe SNHL. It will be interesting if future research of same kind would target on different degree and configuration of hearing loss. Aarts and Caffee,30.As a health care professional, it is the responsibility of the audiologists to make their clients satisfied with the hearing aid fitting and to encourage them to use the hearing aid regularly. It will further prevent the auditory deprivation, cognitive decline and social isolation associated with hearing loss. A standard hearing aid fitting protocol including the probe-microphone measurement can improve the patient’s self-perceived benefit of their hearing aids, contentment with the hearing care professional and result in better patient loyalty. It was reported that inclusion of probe-microphone measurement in clinical protocol positively influences the user’s trust and improve the acceptance of hearing aid and its useTo conclude, the result of the current study also revealed improvement in aided threshold, articulation index and word recognition when the verification of gain and output was carried out through probe microphone measurement. The probe-microphone measurement is considered as the “Gold Standard” for verification of hearing aid fitting in the best practice guidelines for hearing aid fitting. Many audiologists are not using the probe-microphone technique regularly to verify the hearing aid gain and output when dispensing it. It is depicted that lack of verification will result in underamplification in the higher frequencies, that in turn cause inaudibility for soft and average level sounds and this in turn would compromise the speech recognition. A large number of hearing aid users are now using unverified fit that may be one of the reasons for rejection of amplification and dissatisfaction in adult population. Thus, it can be inferred that the clinical use of probe-microphone measurement is warranted to yield better audibility and speech recognition and in turn better satisfaction and quality of life.


Slower than planned recruitment is a major factor contributing to the delay or failure of randomised controlled trials to report on time. There is a limited evidence base regarding the optimisation of recruitment strategies. Here we performed an observational review of our experience in recruitment for two large randomised controlled trials for people with secondary progressive multiple sclerosis. We aimed to explicitly determine those factors which can facilitate trial recruitment in progressive neurodegenerative disease.Recruitment data from the sequential MS-SMART [NCT01910259] and MS-STAT2 [NCT03387670] UK randomised controlled trials was reviewed from the largest recruiting site, University College London (UCL). The trial population was similar which allowed comparison over the two recruitment periods of 2015–2016 and 2018–2021. This included sources of referral, progress through stages of recruitment, reasons for participant ineligibility and the impact of publicity events upon recruitment.ineligibility. In both studies, however, around 60% of those directly telephoned to discuss the study were not eligible, with difficulties related to travel to trial visits, or excluded medication, being the most common issues. Eighty-four percent of those deemed potentially eligible following telephone calls were enrolled in the MS-STAT2 study, compared to only 55% for MS-SMART.In MS-SMART, 18% of patients contacted were enrolled, compared to 27% for MS-STAT2. Online registration of interest portals provided the greatest number of referrals , with publicity in national media outlets producing a demonstrable increase in the number of potential participants. The introduction of an online self-screening questionnaire for MS-STAT2 resulted in 67% of potential participants (3080 of 4605) automatically determining their own Through a detailed review of recruiting participants at the largest centre into two large randomised controlled trials with similar entry criteria, we have identified a number of approaches that may improve recruitment efficiency. We highlight here the importance of mandatory online self-screening questionnaires, a coordinated publicity campaign, and simple interventions such as eligibility checklists and appointment reminders. Recruitment approaches should be further assessed through a studies within a trial (SWAT) design.NCT01910259; registered July 2013 and MS-STAT2: NCT03387670; registered Jan 2018MS-SMART: The online version contains supplementary material available at 10.1186/s13063-022-06588-z. Randomised controlled trials (RCTs) are the gold-standard for assessing the efficacy and safety of medical treatments , 2. SpeeMS-SMART NCT01910259) was a multi-centre phase 2b RCT simultaneously assessing the efficacy of 3 potentially neuroprotective treatments in slowing the rate of brain atrophy compared to placebo in participants with SPMS [10259 wasMS-STAT2 (NCT03387670) is an ongoing multi-centre phase 3 RCT assessing the efficacy of high-dose simvastatin compared to placebo in slowing the rate of disability progression in SPMS. It involves 31 trial sites across the UK, recruiting 964 participants. Recruitment began in May 2018 and finished in September 2021. UCL was again the largest trial site, recruiting 33% of the trial cohort.In both MS-SMART and MS-STAT2 trials, detailed recruitment data was systemically recorded at the main UCL site. Our analysis of recruitment data is therefore restricted to the UCL site alone, which contributed around a third of the total cohorts.The trial populations recruited at the UCL site were similar in MS-SMART and MS-STAT2, as shown in Table pre-screening to discuss trial participation and review eligibility criteria. This telephone pre-screening was performed solely by clinical research fellows with experience in caring for people with MS and research trials. It was performed ad hoc in MS-SMART, and this experience prompted the development of formalised pre-screening eligibility checklists, which were used throughout MS-STAT2 , local databases, neurology consultant or other clinician referrals, patient self-referrals and patient outreach meetings at local MS centres. Secondly, appropriate referrals were then manually contacted for telephone TAT2 see . FinallyThe MS-SMART online portal was created on a web-based portal linked to a database maintained by the Edinburgh Clinical Trials Unit. Potential participants were directed to it via publicity generated by the UK Multiple Sclerosis Society, including national publicity campaigns. It collected name, contact details and a location to ascertain the most appropriate MS-SMART site to refer the potential participant to.The MS-STAT2 online portal was created on the Research Electronic Data Capture (REDCap) platform using UK-MS Register infrastructure . In addiDates of publicity events for the MS-STAT2 trial were prospectively collected during recruitment, including UK Multiple Sclerosis Society Website updates and local and national media coverage. Additional publicity events were retrospectively identified through online searches.For the lead UCL site only, the main sources of recruitment and progress of participants through the stages of recruitment were compared between MS-SMART and MS-STAT2. Percentages obtained from differences sources and percentages passing each stage of recruitment were calculated from research databases, identifying the major cause of ineligibility at each stage. In the rare event of participants being referred twice from different sources, only the first referral source was included.The impact of various publicity events on trial recruitment is assessed for MS-STAT2 only, by qualitatively comparing UK-wide activity on the online portal to the timing of various publicity events. A 7-day rolling average of completed online portal entries was calculated and then inspected for temporal associations with recent publicity events.Sources of referral for MS-SMART Fig.  and MS-SData on the number of participants randomised by each recruitment source were known for MS-STAT2 only. Whilst the online portal generated the greatest number of referrals (51%), a smaller percentage of such patients reached randomisation compared to consultant referrals (28%): 22% were randomised from the online portal compared to 35% from consultant referrals.Across all UK trial sites, the introduction of a self-screening questionnaire into the online portal for MS-STAT2 facilitated the early identification of many potential participants who were unlikely to be eligible for the trial. From a total of 4605 online portal entries, 3080 (67%) were automatically informed that they were unlikely to be eligible. The main criteria that excluded such candidates were the age limit (25–65 inclusive) and questions related to their ability to travel to trial sites.The progress of potential participants through each stage of screening in the two trials is compared in Table The temporal relationship between UK-wide MS-STAT2 online portal entry completions (shown as a 7-day average) and various publicity events carried out during the recruitment period are shown in Fig. As expected, the largest number of responses were generated at the opening of the online portal, which was coordinated with publicity via the UK Multiple Sclerosis Society. Subsequently, spikes in online portal completions appeared in association with national BBC publicity (October 2019 and December 2020) , 15.It is harder to infer causality for smaller spikes in online portal completions. In December 2018, increasing interest was observed following coverage of MS-STAT2 in the Association of British Neurologists newsletter and website. This is sent by email to all neurologists in the UK, and it is plausible that this may have generated interest if neurologists subsequently mentioned it to their patients in the following weeks. In March 2021, an increase in interest coincided with the launch of the OCTOPUS trial (EudraCT: 2021-003034-37) . The UK Multiple Sclerosis Society and Guardian newspaper carried the story, and both referenced to the ongoing MS-STAT2 trial and contained the online portal link. Various smaller newspaper stories may have generated some additional interest. Overall, local events, such as the university website/newsletter coverage or visits to local MS centres, did not appear to generate a perceptible increase in online portal completions. The onset of the COVID-19 pandemic and the first UK national lockdown between March and June 2020 coincided with both reduced publicity events and reduced online portal completions.Figure We have reviewed here the operational aspects of clinical trial recruitment in progressive MS from two large academically led, non-commercial UK studies. This includes phase 2b and phase 3 experience at the major recruiting trial centre, with over 2000 intial contacts, and ultimately nearly 500 participants randomised. Based upon this experience, we summarise in Table national media outlets. Continuing support from the UK Multiple Sclerosis Society, who provided regular updates on the trial via their website, in addition to coordinating larger media events, was essential for this. Weaker signals were obtained for smaller publicity events, but involvement of the Association of British Neurologists, aiming to increase neurologists’ awareness of the trial, is likely to have been additionally helpful. This may be particularly beneficial given the observation that consultant referrals to the MS-STAT2 trial were more likely to be randomised than potential participants obtained via other sources . We speculate that this difference is likely to relate to the consultants being skilled in determining potential trial eligibility, and with their consultants’ support, patients may be more enthusiastic about participating.Our experience of trial publicity in MS-STAT2, as shown in Fig. do not appear to have a quantifiable impact upon online portal completions. Certainly, the efforts involved in creating such events appear to be out of proportion to the recruitment outcomes that we are able to detect here. We acknowledge, however, that we cannot quantify the good will and any subsequent word of mouth publicity that may be derived from such events. Continuing such events as remote teleconferences, with the added efficiency that this can bring, may be a very useful option for the future.Smaller publicity events, such as visits to MS therapy centres, local presentations and updates to departmental websites, Both studies relied heavily on the online portals to generate and manage referrals, representing 76% of referrals to UCL in MS-SMART, and 51% of referrals to UCL in MS-STAT2. Combined with appropriate trial publicity, an online portal therefore appears to be an important aspect for recruitment in multi-centre trials in progressive MS.Incorporating a self-screening questionnaire into an online portal is recognised as an effective strategy to facilitate quicker and more efficient identification of potentially eligible participants . This isDespite the introduction of this important self-screening questionnaire for the MS-STAT2 trial, however, only 40% of potential participants were deemed eligible after the manual pre-screening telephone call, similar to the results found in MS-SMART (39%). With each pre-screening discussion typically taking up to 30 min, across the two trials, we estimate over 500 hs of specialist clinician time was spend discussing the studies with ultimately ineligible candidates.eligible, the most common reasons for subsequent ineligibility at manual telephone pre-screening were as follows: transport/distance to visits (n=126), medication (n=98) and no longer interested in the study (n=25). Issues related to the specifics of their multiple sclerosis  were a relatively less frequent reason for ineligibility at manual pre-screening after the self-screening questionnaire.In assessing why the rates of ineligibility remained similar between MS-SMART and MS-STAT2, despite the introduction of the self-screening questionnaire in MS-STAT2, it should firstly be noted that 49% of referrals for MS-STAT2 came from sources other than the online portal, and hence, these potential participants did not complete the self-screening questionnaire. Secondly, for those MS-STAT2 candidates that did complete the questionnaire and were deemed potentially Once deemed eligible at pre-screening, 55% of participants reached randomisation in MS-SMART, compared to 84% in MS-STAT2. Whilst we cannot exclude a role of additional unidentified factors in contributing to this, we suggest that two simple interventions are likely to have contributed. Firstly, the introduction of detailed pre-screening checklists for MS-STAT2 may have contributed to the higher rate of eligibility at face to face screening. Secondly, the use of appointment reminders for MS-STAT2 is likely to have contributed to the improved attendance at screening visits .For both trials, the most common reason for either pre-screening or screening ineligibility related to issues with transport to trial visits , followed by contraindicated medication  or statins (MS-STAT2)) .Overall, we conclude from our experience of assessing eligibility that firstly, all potential participants, regardless of their referral source, should be directed to a mandatory trial online portal, including a self-screening questionnaire. This should improve efficiency both for trial staff and participants by ensuring that all candidates have the opportunity to automatically determine their own likely eligibility. The resources required to establish such online portals should clearly be considered as part of trial funding applications.Additionally, emphasis should be placed on identifying and overcoming logistical barriers to trial participation, as these are the most common limitations to recruitment. When designing the online portal questionnaires, clear details on the locations of participating trial sites should be included, and such sites should be geographically well distributed to improve inclusion. The number of appointments requiring face to face attendance should be detailed, together with increasing use of remote visits where possible (see below). With this information, potential participants should then be asked to indicate, as part of self-screening, whether they feel able to meet the burden of visits required by the trial, and whether they have considered the logistics of regular travel to the trial sites. If candidates feel unable to meet such requirements of a trial, they should be provided with a mechanism to report the reason for this, allowing trial sites to troubleshoot the issues and explore opportunities to overcome them, aiming to maximise inclusion.Further improvements in identifying medical exclusions, particularly related to the common finding of contra-indicated medications, may additionally be achieved by allowing potential participants access to update their submitted details .There is increasing interest in the remote assessment of participants in clinical trials, the importance of which was additionally highlighted due to the impact of the COVID-19 pandemic on clinical trials , 18. As Within the MS literature, a previous analysis from an exercise-based RCT (ExIMS) found that the majority (60%) of participants were recruited from MS hospital clinics . This apBeyond MS, a Cochrane review of randomised studies investigating trial recruitment found limited evidence to support recruitment approaches . Open asMore recently, programmes such as PROMETHEUS  have facilitated the generation of higher quality evidence regarding recruitment processes . These hNon-randomised studies reporting on recruitment strategies provide a more limited evidence base . Our expstudies within a trial (SWATs) [As highlighted above, the main limitation of this study is the observational design. We therefore cannot demonstrate causality for the inferences we have made through comparison of our experience of trial recruitment in MS-SMART and MS-STAT2. Other, unidentified factors may also have contributed to differences in recruitment between the two studies. Building upon our observations, future research should therefore focus on improving the evidence base for recruitment strategies through prospective, randomised  (SWATs) . These s (SWATs) .We have described our experience of recruiting participants with SPMS into two large RCTs in order to identify areas where the efficiency of recruitment could be improved. We have outlined in Table Additional file 1. Prescreen Checklist. Investigator groups.


A bone-anchored prosthesis (BAP) eliminates the need for a conventional socket by attaching a prosthesis directly to the user’s skeleton. Currently, limited research addresses changes in gait mechanics post BAP implantation.Examine changes in frontal plane movement patterns after BAP implantation.Participants were individuals with unilateral transfemoral amputation (TFA) enrolled in the US Food and Drug Administration (FDA) Early Feasibility Study examining the Percutaneous Osseointegrated Prosthesis (POP). The participants completed overground gait assessments using their conventional socket and at 6-weeks, 12-weeks, 6-months, and 12-months following POP implantation. Statistical parameter mapping techniques were used in examining changes in frontal plane kinematics over the 12-months and differences with reference values for individuals without limb loss.Statistically significant deviations were found pre-implantation compared to reference values for hip and trunk angles during prosthetic limb stance phase, and for pelvis and trunk relative to the pelvis angles during prosthetic limb swing. At 6-weeks post-implantation, only the trunk angle demonstrated a statistically significant reduction in the percent of gait cycle with deviations relative to reference values. At 12-months post-implantation, results revealed frontal plane movements were no longer statistically different across the gait cycle for the trunk angle compared to reference values, and less of the gait cycle was statistically different compared to reference values for all other frontal plane patterns analyzed. No statistically significant within-participant differences were found for frontal plane movement patterns between pre-implantation and 6-weeks or 12-months post-implantation.Deviations from reference values displayed prior to device implantation were reduced or eliminated 12-months post-implantation in all frontal plane patterns analyzed, while within-participant changes over the 12-month period did not reach statistical significance. Overall, the results suggest the transition to a BAP aided in normalizing gait patterns in a sample of relatively high functioning individuals with TFA. Osseointegration for individuals with transfemoral amputation (TFA) has received significant attention in recent years. This surgical procedure consists of implanting a bone-anchored prosthesis (BAP) in the residual femur that extends through the skin surface to link the bone directly to other components of the prosthetic limb , 2. ConvResearch involving BAPs in individuals with TFA often focused on quantifying adverse events with the implanted device , force loaded on the implant during activity, and the impact of a BAP on outcomes such as hip range of motion, sitting comfort, and quality of life –12. OverFewer studies report the effect of a BAP on gait biomechanics. Frossard et al. and Pinard et al. found the cadence, gait cycle duration, and sound side step lengths in those with a BAP were equivalent or improved compared to those with conventional socket and suspension systems , 15. ElsThe purpose of this study was to compare frontal plane gait kinematics of the trunk, pelvis, and hip pre-, 6-weeks post-, and 1-year post-implantation with a BAP in individuals with TFA. We hypothesized that a transition from a conventional socket and suspension system to a BAP would facilitate frontal plane gait kinematics more closely resembling individuals without an amputation.Study participants were individuals with traumatic unilateral TFA enrolled in a United States Food and Drug Administration (FDA) Early Feasibility Study (EFS) for a new BAP device named the Percutaneous Osseointegrated Prosthesis (POP) , 20. ThiTesting was completed on an overground walkway. Each participant was instructed to walk at a comfortable self-selected walking speed (SSWS). Full-body kinematic data were collected for walking at 200 Hz using a six-degrees of freedom marker set  and a 10th order zero-lag Butterworth filter with cutoffs  determined by visual inspection. A 15-segment full-body model was created and the frontal plane kinematics of the hip joint relative to the pelvis segment (hip adduction angle), pelvis segment relative to the lab coordinate system (pelvis-lab angle), trunk segment relative to the lab (trunk-lab angle), and trunk segment relative to the pelvis segment (trunk-pelvis angle) were extracted from the walking trials. Force data were used to identify initial contact and toe-off gait events with gait cycles defined as consecutive initial contacts of the same limb. Kinematic data were temporally normalized to the gait cycle and averaged. Step width was calculated as the distance along the mediolateral plane between the proximal ends of each foot segment for consecutive heel strikes. Gait speed was calculated by dividing the stride length by the stride time.Data from the 71 markers and 2 force plates were lowpass-filtered in Visual 3D motion analysis software  using a 42 test of the residuals. If the normality assumption was violated, then a non-parametric permutation SPM test was used instead [The time points of interest were the pre-implantation test, the 6-week post-implantation test, and the 12-month post-implantation test. Group data from other time points are presented for descriptive purposes but were not included in statistical testing due to the small sample size. A one-way ANOVA was used to assess the effect of testing period on step width and a paired t-test was used to evaluate changes in walking speed between pre-implantation and the 12-month post-implantation visit. Both tests were conducted using JMP v16 . Frontal plane kinematics were statistically analyzed with one dimensional Statistical Parametric Mapping (SPM) techniques. This approach allows for analysis of the entire gait cycle curve by calculating a test statistic at each gait cycle percentage and utilizing random field theory to assess significance . This me instead . Testing instead . ResultsTen Caucasian males with traumatic unilateral TFA were enrolled in the FDA EFS for the POP device. Eight individuals completed all testing sessions. Two participants experienced complications with the implant and withdrew prior to the 12-month testing session. 6-month post-implantation visit data for one of the withdrawn participants were used in the statistical analyses in place of 12-month data. Additionally, one participant used assistive devices during the initial and 6-week post-implantation data collections but no assistive device at the other collections. This participant’s data were excluded due to the potential effect of the assistive devices on gait mechanics. As a result, statistical analyses were conducted with pre-implantation and 12-month post-implantation data from 7 participants as well as pre-implantation and 6-month post-implantation data from 1 participant. Full demographic information for the 8 participants can be found in The group demonstrated a mean step width of 0.21 (SD: ±0.06) m at the pre-implantation tests. The step width decreased to 0.19 (SD: ±0.04) m at the 6-week implantation visit and to 0.18 (SD: ±0.05) m by the 12-month post-implantation tests. The effect of time period on step width did not reach statistical significance (p = 0.53).Descriptively, participants demonstrated prosthetic side hip abduction throughout the stance phase with a transition to a minimally adducted position during swing at the pre-implantation tests . Large dPre-implantation, a contralateral hike of the pelvis was developed early in stance on the prosthetic limb and was retained throughout the remainder of the stance phase . This waTrunk movement pre-implantation was characterized by a large ipsilateral trunk lean during stance on the prosthetic limb . This paPre-implantation, participants demonstrated an ipsilateral lean of the trunk relative to the pelvis during stance on the prosthetic limb, but a more orthogonal position during swing . This trPrior research suggests sagittal plane movement patterns improved after BAP implantation in those with TFA compared to when the individual used a conventional socket and suspension system , 16. WhiBefore BAP implantation, the individuals in the current study demonstrated differences in frontal plane movement patterns from reference values in line with those reported in prior work, including widened step width, increased hip abduction, and increased ipsilateral trunk lean during single limb support on the prosthetic limb , 18. ThiChanges at the 6-week exam were of interest as eliminated or significantly diminished frontal plane deviations by the first post-implantation test could have suggested the improvements were more likely related to the implant than other physical rehabilitation related factors. However, the 6-week post-implantation data showed frontal plane movements patterns did not dramatically improve in such a short-term. Analysis of the frontal plane gait parameters at the first post-implantation visit Figs –5 suggesAn increase in functional strength of the prosthesis side hip muscles may be one potential reason for finding the BAP device promoted normalization of frontal plane movement patterns. Those with TFA commonly present with weakness and atrophy in the hip abductor muscles , 32, andDespite the reductions in frontal plane deviations from reference values, not all were eliminated at the 12-month post-implantation visit. The remaining differences may have persisted because the individual’s prosthetic limb did not fully compensate for the functional loss of their biological knee, foot and ankle, regardless of a direct connection provided by the BAP. For example, our results showed step width narrowed slightly over time but remained well above norms . ProstheThough a more normalized frontal plane movement pattern was demonstrated at the 12-month post-implantation test relative to the reference data, there was no statistically significant change in the frontal plane measures between the pre- and 12-month post-implantation tests. An explanation for this finding may be the functional level of the participants. Though these individuals elected to transition away from a conventional suspension system due to socket-related issues, they were relatively high-functioning prior to the BAP implantation as evidenced by both their high K-level and SSWS . As a grThere are several other points to consider when interpreting and generalizing the study’s results. First, the sample size within this preliminary study was limited by constrained design. This FDA EFS only allowed for 10 participants of which 8 individuals had suitable data for analysis, including one individual whose 6-month post-implantation data was utilized in place of 12-month post-implantation data. The small sample size limits the ability to examine differences between other time points within this group. The participants were also prescribed different prosthetic knee or ankle componentry, and provided alignment changes based on their individual clinical needs. Moreover, the prosthetic device changed for some individuals during the 12-months . It is unknown what affect these changes may have had on an individual’s gait patterns. Additionally, all participants in this study underwent amputation due to trauma, and as such, may not represent the response of those who may have lost limbs due to vascular dysfunction or diabetes. However, as implantation with a BAP is not widely performed yet in those with limb loss due to vascular conditions, the individuals in this study are more representative of those who actually undergo BAP implantation. Moreover, other factors that could have affected gait, such as pain levels, were not assessed in this study and the potential effects of these factors on the current results is unknown. This data also only encompasses the first year after BAP implantation. While we hypothesize that further normalization in gait patterns may occur later than one-year post-implantation, the data to assess this hypothesis was not collected as part of the study. Continued follow-up is planned to assess the long-term response of the participants to the BAP implant. Finally, for comparisons to normative values, we used a reference group of young, healthy individuals without limb amputation walking on a treadmill. While some research has previously found kinematic differences between treadmill and overground walking, the differences in frontal plane motion of the hip, trunk, and pelvis previously found are much smaller than the deviations described here –45. TherThis study is the first to describe frontal plane gait kinematics after a BAP implant in those with TFA. While comparisons between frontal plane movement patterns pre-implantation and 12-month post-implantation were not statistically different, deviations from reference values that occurred during both prosthetic side stance and swing were gradually reduced or eliminated over the first year after implantation with the POP implant system. These results add to the literature suggesting that transition to a BAP may aid in normalization of gait parameters in those with TFA that fail to tolerate traditional socket and suspension systems.S1 File(DOCX)Click here for additional data file.S2 File(DOCX)Click here for additional data file.S3 File(DOC)Click here for additional data file.S1 Dataset(XLSX)Click here for additional data file.


The performance of a neural network depends on the availability of datasets, and most deep learning techniques lack accuracy and generalization when they are trained using limited datasets. Using synthesized training data is one of the effective ways to overcome the above limitation. Besides, the previous corroded bolt detection method has focused on classifying only two classes, clean and fully rusted bolts, and its performance for detecting partially rusted bolts is still questionable. This study presents a deep learning method to identify corroded bolts in steel structures using a mask region-based convolutional neural network (Mask-RCNN) trained on synthesized data. The Resnet50 integrated with a feature pyramid network is used as the backbone for feature extraction in the Mask-RCNN-based corroded bolt detector. A four-step data synthesis procedure is proposed to autonomously generate the training datasets of corroded bolts with different severities. Afterwards, the proposed detector is trained by the synthesized datasets, and its robustness is demonstrated by detecting corroded bolts in a lab-scale steel structure under varying capturing distances and perspectives. The results show that the proposed method has detected corroded bolts well and identified their corrosion levels with the most desired overall accuracy rate = 96.3% for a 1.0 m capturing distance and 97.5% for a 15° perspective angle. Bolted joints serve to connect load-bearing members in various structural systems. During their life cycle, bolted joints are frequently exposed to different environmental conditions of moisture and air pollutants and therefore are susceptible to corrosion ,2. Thus,In the past few decades, many structural health monitoring (SHM) methods have been developed to gradually replace time-costly traditional visual inspections ,4,5,6,7.Due to those promising features, the machine learning and vision-based approaches have been applied for SHM of steel bolted joints ,20,21,22The performance of a neural network depends on the availability of datasets, and most deep learning techniques lack accuracy and generalization when they are trained using limited datasets. With the aim of advancing the perception of deep learning models, many studies have developed computer graphics to synthesize realistic virtual models that support creating the training graphic datasets. Several deep learning models trained on synthesized datasets have recently demonstrated an outstanding performance when tested on realistic datasets ,30,31. RThe objective of this study is to develop a deep learning model for identifying early-corroded bolts in steel joints and to present a data synthesis technique to autonomously generate training datasets. Firstly, a Mask-RCNN-based corroded bolt detector is proposed. The Resnet50 integrated with the feature pyramid network is used as the backbone for feature extraction in the detector. Next, a four-step procedure is proposed to autonomously generate three datasets: clean bolt (Bolt), partially corroded bolt (PCBolt), and fully corroded bolt (FCBolt), along with their corresponding masks. Thirdly, the generated datasets are used to train the Mask-RCNN-based corroded bolt detectors. Finally, the robustness of the method is demonstrated by detecting corroded bolts in a lab-scale steel structure under varying capturing distances and perspectives.The Mask-RCNN model is built based on the Faster RCNN model for instant object segmentation with three primary changes: (1) a Resnet50 backbone is integrated with the feature pyramid networks (FPN) to generate better-scaled features; (2) RoiAlign layers are used instead of the RoiMaxPooling layers to provide more sub-pixel levels than RoiMaxPooling layers; and (3) a mask branch for pixel-level object segmentation is attached to generate pixel masks. According to the previous studies ,33 a MasAs illustrated in Detection results from each category are marked by the different colors. For example, clean bolts (ClBolt) are assigned by a white color at the bounding boxes and masks, whereas the partially corroded bolts (PCBolt) and fully corroded bolts (FCBolt) are marked by yellow and red colors, respectively.The architecture of the Resnet50-FPN is depicted in Different from other networks, the Resnet50 network , a very As fundamental components of an object recognition system , the FPNAs the second module, the region proposal network (RPN) layers see  receive The third module is the RoiAlign layers. They receive feature boxes and scale these features to a uniform size. By using an interpolation of feature calculation, the RoiAlign layers (the 24th layer) are used in Mask-RCNN instead of the RoiMaxPooling layers because they accurately provide sub-pixel levels better than RoiMaxPooling layers .The feature prediction module of Mask-RCNN see  is constThe classification layer contains fully connected (FC) layers, and the final FC layer outputs four-class types, ClBolt, FCBolt, PCBolt, and BG. The classification loss function of the classification layer is defined in Equation (1), in which oU index .(1)LclaThe box regression layers aim to generate bounding boxes covering the objects. This branch similarly operates as RPN layers: it refines the location and size of the bounding box. Equation (2) shows the bounding-box loss function , the thi test image, and the Average precision , a four-step synthesized dataset generation process is proposed in In the first step, 577 raw images, including 568 bolted connection images see b and ninIn the second step, the FG bolt images are cropped out with respect to the six edges of the bolt surfaces and then grouped into three groups, namely, FG-1 with 566 ClBolt images, FG-2 with 332 FCBolt images and FG-3 with 865 PCBolt images. The size of the FG images ranges from 32 × 42 × 3 to 173 × 187 × 3 pixel resolutions. Afterwards, the BG images are divided into sub-background (SBG) patches. Each BG image produces 46 SBG patches with the same size of 512 × 512 × 3. As seen in In the third step, the domain randomization (DR) approach  was usedIn the final step, three synthetic datasets are autonomously created, including D1 (ClBolt), D2 (FCBolt), and D3 (PCBolt). Each dataset contains 1875 synthetic images of 512 × 512 × 3 pixels with 1875 corresponding annotated masks. All processing and image data generation processes use the python programing language (version 3.8). The training process was conducted by a desktop computer system  using python programing language (version 3.8) and the supporting libraries tensorflow , karas , and ope is used . To inve−4 to 10−5. After the first 43 epochs, those loss lines are quite stable until the end of the learning process. Thus, the learning rate of 10−5 should be considered an optimal value for training the Mask-RCNN corroded bolt detector.As observed in The corroded bolt detector was experimentally evaluated for identifying the early-corroded bolts in a lab-scale steel girder. The structure was constructed from two steel H-girder segments connected by bolted flanges, as shown in The camera of an iPhone X  was used to shoot the image of the bolted connection with a 3042 × 4028 pixel resolution. During the experiment, the light intensity at the bolted joint was controlled at around 154~157 lux  are shown in As shown in In this study, the Mask-RCNN model was developed for the early identification of partially corroded bolts in steel structures. In the model, the Resnet50 integrated with the feature pyramid network was used as the backbone for feature extraction. To overcome the issue of limited training data, a four-step data synthesis method was newly proposed for the Mask-RCNN corroded bolt detector. To evaluate the accuracy of the proposed method, the experiment was conducted on a lab-scale steel structure for which corroded bolts were detected under varying capturing distances and perspectives. (1)Clean bolt, partially and fully corroded bolts along with their corresponding masks were autonomously created by the proposed data synthesis method. The Mask-RCNN-based detector was successfully trained using the generated datasets.(2)The trained detector was accurate for corroded bolts in the tested structure. The corroded bolts and their corrosion levels were detected with the most desired accuracy of 96.3% for the 1.0-m capturing distance and 97.5% for the 15° perspective angle.(3)The number of pixels for the test image of the bolt connection should not be less than 263K to ensure the accuracy of the bolt identification results.From the experimental evaluation, the following concluding remarks are drawn:In comparison to a previous study , it was The proposed data synthesis method is autonomous and fast. Thus, it is promising for computer-vision-based damage detection in practice using standard and high-quality datasets at a low computational cost. In particular, the method could be used to simulate damage  on the structure surfaces that have rarely been caught during their operation period. Furthermore, the generated synthetic data could be used as training data sets for multi-targets, such as semantic segmentation, object detection, etc., with only minor changes in the use of programing language.In some cases, the detector could not distinguish the partially and fully corroded bolts. This might be due to the presence of bolt washers, as those were missed from the generated training datasets. Therefore, research needs remain (1) to improve the accuracy of the developed corroded bolt detector, (2) to compare the performance of the detector on different data types with and without using the pre-trained model, (3) to investigate the accuracy of the model by changing parameters such as the sizes of the training images, sizes of the corroded bolt foregrounds, the flashlight of cameras, and the lights of the environment, and (4) to estimate the corrosion areas of detected bolts.


Fibromyalgia is characterized by stiffness and widespread pain which cause a negative impact on health-related quality of life and activities of daily living. In this regard, these activities are usually presented as dual-task situations (execution of two tasks simultaneously). Some physical fitness tests have been used under dual-task condition in order to evaluate physical fitness under a more ecological approach. It is hypothesized that physical fitness test could provide more information than physical fitness test under single-task condition. Therefore, the aim of the present study was to explore the relationship between 10m walking test performance under both single and dual- task conditions and the fear of falling and the intensity of pain in women with fibromyalgia.A total of 38 women (55.65 [9.28] years-old) participated in the study. Participants performed the 10-m walking test where they have to walk on a 10 meters straight-line as fast as they can. Moreover, the fear of falling and the intensity of pain were measured by a visual analogue scale (0-100).Furthermore, the performance of 10-m walking test correlated positively with the fear of falling in both under single  and dual-task conditions . However, the performance of 10m walking test only positively correlated with pain under dual-task condition . In single task condition did not significantly correlate .The performance of 10m walking test under single and dual-task conditions are correlated with the fear of falling. However, the pain intensity only significant correlated with 10m walking test performance under dual-task condition. This could indicate that pain intensity has more influence than the fear of falling on decreasing the performance of a daily living activity such as walking. However, further studies under dual-task paradigm are needed to confirm this hypothesis.


The identification of SARS-CoV-2 variants across the globe and their implications on the outspread of the pandemic, infection potential and resistance to vaccination, requires modification of the current diagnostic methods to map out viral mutations rapidly and reliably. Here, we demonstrate that integrating DNA barcoding technology, sample pooling and Next Generation Sequencing (NGS) provide an applicable solution for large-population viral screening combined with specific variant analysis. Our solution allows high throughput testing by barcoding each sample, followed by pooling of test samples using a multi-step procedure. First, patient-specific barcodes are added to the primers used in a one-step RT-PCR reaction, amplifying three different viral genes and one human housekeeping gene . Then, samples are pooled, purified and finally, the generated sequences are read using an Illumina NGS system to identify the positive samples with a sensitivity of 82.5% and a specificity of 97.3%. Using this solution, we were able to identify six known and one unknown SARS-CoV-2 variants in a screen of 960 samples out of which 258 (27%) were positive for the virus. Thus, our diagnostic solution integrates the benefits of large population and epidemiological screening together with sensitive and specific identification of positive samples including variant analysis at a single nucleotide resolution. The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is an RNA virus that causes the coronavirus disease 2019 (COVID-19). The virus was first identified in Wuhan (China) in December 2019, spread rapidly and was declared a pandemic in March 2020 by the World Health Organization. Today, more than two years later, COVID 19 has already resulted in over 400 million confirmed infection cases and over 5.7 million deaths across the world.COVID-19 symptoms are varied, ranging from none to severe according to the Centers for Disease Control and Prevention (CDC). Many patients develop moderate pneumonia and in the most critical cases respiratory failure and multiorgan disfunction occur. The world average mortality is 2%.Prevention of viral transmission is essential as COVID-19 spreads through the respiratory route. Transmission prevention is achieved through social distancing, masks and hand washing in addition to early identification of potential infected individuals by different diagnostic methods.SARS-CoV-2, a positive-sense single-stranded RNA virus is a highly pathogenic member of the coronavirus family . The seqS-, E-, N- and M- genes encode for structural proteins , ORF region encodes for non-structural proteins as RNA-dependent RNA polymerase (RdRp) or papain-like protease .Several methods have been developed in order to assess the infection status of individuals. Most of them rely on the detection of viral RNA.The gold-standard method for diagnosis of COVID-19 is real-time reverse transcription polymerase chain reaction (one step rRT-PCR) , 4, whicWhile, real time PCR combined with robotic systems, can be used as a high throughput method, the number of amplicons that can be detected are limited, and fluorescent probes and special reagents are needed in order to increase the sensitivity of the method. Moreover, as the global spread of the virus continues despite the use of preventative measures and positive effects of vaccination , SARS-CoNext-generation diagnostic solutions, which are based on the current PCR test coupled with next generation sequencing (NGS) offer major advantages, such as significantly larger scale testing , reduced cost per test, reduced amount of reagents per test and potential readout of viral sequence variation.In NGS-based methods, individual samples can be uniquely labeled with molecular barcodes and multiple fragments can be amplified in parallel and pooled together for high throughput detection and processing of thousands of specimens together. A few different protocols have been developed , 8 and oHere we propose a solution that allows high throughput testing by barcoding and pooling test samples, up to hundreds at a time, in one well  were run as a pilot with 17 barcodes and a multiplex of two viral genes (N1 and E) and one human internal control (RNaseP). Following multiplexing, amplicons were pooled. A single library was prepared and ran on a MiSeq (Illumina) instrument.Following demultiplexing, number of reads for each barcode was obtained. RNase P reads were used for internal control and were required to be positive in order to consider the result as valid.The viral genes were used for the diagnosis and detection of SARS-CoV-2 specifically in each specimen. For each barcode, the associated viral reads were counted and the results are presented in Following this experiment, the stability of 96 barcodes was tested using positive SARS-CoV-2 standard as an identical template for each barcode. Furthermore, an additional viral region (N2) was amplified and added to the multiplex. The stability of 96 barcodes is presented in Moreover, the N2 viral fragment added additional information regarding the viral identification and can provide an improvement in diagnosis of the specimens.In order to improve specificity and sensitivity, additional viral regions for barcoded amplification were examined. Each new set of primers was first tested on positive and negative SARS-CoV-2 standards. The count of reads is presented in In order to choose the most effective combination of genes targeted, the following multiplexes were tested on 96 clinical RNA specimens for optimization . MultiplOur results demonstrate that when genes N1 and N2 are multiplexed together, a non-specific DNA fragment is generated due to the physical proximity of the two amplicons (unshown data). This 944bp DNA fragment was formed as the elongation product of the N1 forward primer and N2 reverse primer. As this fragment is added to the library preparation and run together with all the amplicons, it can lead to a competition in the NGS run and to a lower number of reads that are obtained from the fragments of—interest.In order to scale up our technology, 960 clinical RNA specimens were tested using multiplex 3. Amplicons were pooled (each plate separately), libraries were prepared for each plate with a different index and all ten libraries were sequenced together using the Next-Seq instrument (illumina). Following demultiplexing, obtained sequences were analyzed and positive samples were identified using the unique and specific barcodes . The numUsing a Gradient Boosting machine learning classification model, results were obtained with an accuracy of 93.3%, precision of 91.7%, sensitivity of 82.5% and specificity of 97.3%  in a valhttps://blast.ncbi.nlm.nih.gov/) we have identified six known and one newly found missense mutation. Among the 258 SARS-CoV-2 PCR-positive samples, 30 carried the same N-gene missense mutation and 6 carried an additional synonymous substitution in ORF1a , SARS-CoV-2 barcoded pooling provides a platform for detection of single nucleotide mutations  in pre-din ORF1a . Eight PSARS-CoV-2 (severe acute respiratory syndrome coronavirus 2), a beta coronavirus, is the novel coronavirus that caused the COVID-19 outbreak originating in Wuhan, China in 2019. This virus has become a worldwide pandemic requiring immense and rapid viral detection methods throughout the world. The classic method for detection of infection involves sampling nasopharyngeal swabs from patients in order to detect viral infection. The swab is placed in transfer buffer and then lysis buffer is added to lyse the potentially infected cells. RNA is extracted from the solution then subjected to a one-step RT-PCR reaction. The RNA is first converted to cDNA by reverse transcription which is immediately followed by Multiplex quantitative Real time PCR (qPCR) with primers and a probe specific to the virus to detect viral presence.Generally, an internal control is added in order to eliminate the possibility of false negative results and includes detection of a human gene as a proxy for host RNA content. However, some assays only amplify an exogenic internal control gene (added during the extraction step) which provides evidence for the validity of the qPCR reaction yet, is not reliable enough to exclude false negatives.Here we describe a method we developed for combined screening of an infectious agent and simultaneously determine the presence of a genomic variant in a population, by using a DNA-barcoded sample from each subject in a population. The method includes performing one-step reverse transcription and amplification of RNA extracted from the samples with barcoded primers, pooling and sequencing the obtained amplicons, and identifying the infected individual out of all tested samples in the pool via the subject-specific barcode. The different primers target three viral genes as well as one human endogenous gene. The detection of an endogenic human internal control assay that is specific to each specimen, even though they are sequenced as a pool, is of extreme importance and represents one of the major advantages of this technique.As the virus has spread globally, it has naturally acquired thousands of mutations. Most of these are deleterious or neutral, however, some confer selective advantage and have given rise to many viral variants that became the dominant viral strain in many countries. Some of these variants as they have been shown to increase infection rate, cause higher mortality and most importantly may evade the immune response obtained by vaccination . It is tWe provide a method that can be used to identify and screen specific loci on the viral genome which are of concern at the population level. For SARS-CoV-2, one of the loci can be the entire ‘Spike’ sequence, while other areas can potentially be identified as more variants and lineages evolve. We suggest that once new SNVs are identified in this screen, specific samples can be identified based on their unique molecular barcode followed by full genome sequencing to provide a comprehensive sequence-analysis for lineage tracking.Our method utilizes amplified cDNA produced from the infectious agent and at least one SNV in a sequencing read, to determine via the subject-specific barcode whether the subject is infected and simultaneously enables the identification of a genomic variant. The method presented here can potentially be modified using primers designed upstream of the variable S gene to enable the detection of the known variants  whose sequences are mutated in this specific locus (Spike gene). Our aim was to integrate sequence-analysis which can provide information at a population level by searching for SNVs on multiple short sequences from hundreds of test samples and by positively identifying SARS-CoV-2 variants at an individual level. Finally, sensitivity and specificity can be improved by applying machine learning algorithms. This integrative approach can be used to identify SARS-CoV-2 as a stand-alone method or, to be used as an additional information layer which screens the RT-PCR positive samples to identify the viral variants.In order to generate a SARS-CoV-2 whole genome read with high sensitivity and specificity (98% and 97%), at least 1,000 copies of the viral genome per milliliter are required [To overcome this limitation, additional NGS-based mass screening approaches have provided proof that such methodologies can be used to identify new variants, detect genetic epidemiology and follow viral lineages . HoweverBased on our findings, we propose coupling individualized sample identification and pooling with machine learning approaches to optimize current NGS resources to quickly identify viral variants in the population. This approach combines the benefits of high throughput screenings by focusing on specific areas-of-interest in order to increase the number of individual screening tests and employs machine learning algorithms to increase diagnostic accuracy.Population mass-screening diagnostic methods are required to be cost-effective. Based on a calculation of reagents and labor, we have generated a cost estimate to analyze each sample using the suggested method for SARS-CoV-2 variant detection. Given the large-capacity and pooling nature, which stand at the basis of this diagnostic method, it is estimated that an individual test would roughly cost 1–2 USD . This estimate provides organizations which currently run large PCR-based SARS-CoV-2 screening assays, a viable cost-effective option to add an important variant-analysis layer to their data.In conclusion, in comparison to the gold standard real time PCR method, that identifies amplified cDNA produced from the infectious agent and is only able to provide a binary answer (SARS-CoV-2 positive or negative), the method presented here, enables specific diagnosis of SARS-CoV-2 variants within a population of thousands. As this solution can either provide a stand-alone identification method for SARS-CoV-2 or be integrated with the current RT-PCR viral test, we propose adopting this technique to large population and epidemiological screening practice.Oro- and nasopharyngeal swabs were collected from patients by trained health workers with personal protective equipment and directly placed into e-swab tubes (COPAN) for conservation and transport. The study number 0196-20-EMC was granted Ethics approval by the head of the Emek Medical Center Helsinki committee Dr Lee Goldstein M.D. and signed by the Deputy Director of Emek Medical Center Dr Eldar Berkovits M.D. M.H.A.Following sampling and storage of the oro- and nasopharyngeal swabs, nucleic acids were isolated and purified from swab specimens using STARMag 96 X 4 Universal Cartridge Kit (Seegene) on Star or Starlet instruments (Hamilton) following the manufacturer’s instructions.The targets analyzed were amplified using previously reported sets of known primers, while unique barcodes were added to the forward primer to enable pooling of the products before sequencing. These primers sequences are summarized in The complete table of primers including barcodes appears in the supplementary data of this publication.The purified nucleic acids were reverse transcribed into cDNA and amplified using Qscript XLT one-step RT-PCR (Quantabio) and specific primers for each viral and human amplicons. Each forward primer included a 10-nucleotide barcode at the 5’-end, which was unique for each sample tested and enabled pooling of the PCR products prior to sequencing. The total volume of reaction mix (25 μl) contained the following: 12.5 μl of 2X concentrated ToughMix; 0.4 μM reverse primer, 0.4 μM barcoded forward primer; 1 μl of 25X qScript XLT and 5 μl of SARS-CoV-2 positive and negative standard (BioRad). The one step RT-PCR was conducted in a C1000 touch thermocycler (BioRad) as per the following: cDNA Synthesis , Initial denaturation , PCR cycling -25 cycles .The purified nucleic acids were reverse transcribed into cDNA and amplified using Qscript XLT one-step RT-PCR (Quantabio) and specific primers for 3 viral amplicons  and one human . Each forward primer included a 10-nucleotide barcode at the 5’-end, which is unique for each individual tested (the barcode generation is described in the next paragraph). The total volume of reaction mix (25 μl) contained the following: 12.5 μl of 2X concentrated ToughMix; 0.4 μM each reverse and barcoded forward primer; 1 μl of 25X qScript XLT and 8 μl of the RNA sample. The one step RT-PCR was conducted in a C1000 touch thermocycler (BioRad) as per the following cDNA Synthesis , Initial denaturation , PCR cycling -25 for pool of 96 specimens /35 cycles for pool of 960 specimens .Sequences of unique ten-base barcodes were created using DNABarcodes R package . BarcodeAll the 96 separate reactions of one plate were pooled together. Pooled PCR products were purified using MiniElute Spin Columns (Qiagen) according to manufacturer’s instructions.® Ultra™ II DNA Library Prep Kit for Illumina® (New England BioLabs) according to the manufacturer’s instructions (fragmentation and size-selection were not required). Final sequencing libraries were sequenced on the Illumina Miseq sequencer (Illumina) using 2x150 bp paired-end reads (Micro kit).Purified PCR amplicons were used as input DNA to generate Illumina-compatible sequencing libraries using NEBNextA pool of 960 clinical samples was tested using the technology. Ten 96 plates with 96 different specimens (about 25% positive) were amplified using the barcoding-multiplex PCR. Wells from each plate were pooled together and a library was prepared using unique index. All ten libraries were sequenced together using the Next-Seq instrument (illumina).http://hannonlab.cshl.edu/fastx_toolkit/). One mismatch [http://sourceforge.net/projects/bbmap/). Finally, the set of reads were quantified using Salmon version 0.14.2 [First, a raw FASTQ file was demultiplexed by DNA-barcodes using FASTX barcode splitter, i.e., splitting into numerous files based on DNA-barcode matching, one FASTQ per barcode Click here for additional data file.S2 Data(CSV)Click here for additional data file.S3 Data(CSV)Click here for additional data file.


Healthcare provider profiling involves the comparison of outcomes between patients cared for by different healthcare providers. An important component of provider profiling is risk-adjustment so that providers that care for sicker patients are not unfairly penalized. One method for provider profiling entails using random effects logistic regression models to compute provider-specific predicted-to-expected ratios. These ratios compare the predicted number of deaths at a given provider given the case-mix of its patients with the expected number of deaths had those patients been treated at an average provider. Despite the utility of this metric in provider profiling, methods have not been described to estimate confidence intervals for these ratios. The objective of the current study was to evaluate the performance of four bootstrap procedures for estimating 95% confidence intervals for predicted-to-expected ratios.We used Monte Carlo simulations to evaluate four bootstrap procedures: the naïve bootstrap, a within cluster-bootstrap, the parametric multilevel bootstrap, and a novel cluster-specific parametric bootstrap. The parameters of the data-generating process were informed by empirical analyses of patients hospitalized with acute myocardial infarction. Three factors were varied in the simulations: the number of subjects per cluster, the intraclass correlation coefficient for the binary outcome, and the prevalence of the outcome. We examined coverage rates of both normal-theory bootstrap confidence intervals and bootstrap percentile intervals.In general, all four bootstrap procedures resulted in inaccurate estimates of the standard error of cluster-specific predicted-to-expected ratios. Similarly, all four bootstrap procedures resulted in 95% confidence intervals whose empirical coverage rates were different from the advertised rate. In many scenarios the empirical coverage rates were substantially lower than the advertised rate.Existing bootstrap procedures should not be used to compute confidence intervals for predicted-to-expected ratios when conducting provider profiling. Provider profiling involves the comparison of outcomes between healthcare providers . ExampleHistorically, the most common approach to risk-adjustment was to compute provider-specific observed-to-expected ratios, comparing the observed mortality at each provider to the mortality that would be expected given the case-mix of its patients. An observed-to-expected ratio can be computed by using a conventional logistic regression to regress the binary outcome . Using the fitted model, the predicted probability of the outcome, conditional on their baseline covariates, is determined for each patient. These probabilities are summed up within each provider to generate the expected number of deaths at each provider given the case-mix of its patients. Then the observed number of deaths is divided by the expected number of deaths, to produce the provider’s observed-to-expected ratio . Providers whose ratio is greater than one have observed mortality that exceeds the mortality that would be expected given the case-mix of its patients. Providers whose ratio is less than one have observed mortality that is less than the mortality that would be expected given the case-mix of its patients. Hosmer and Lemeshow provided a closed-form expression for the standard error of the observed-to-expected ratio, allowing for estimation of confidence intervals around the ratio . ProvideKrumholz and colleagues suggested a modification of the observed-to-expected ratio . Rather ith patient at the jth provider  and where where The objective of this study was to evaluate the performance of different bootstrap estimators for provider-specific predicted-to-expected ratios. We consider the conventional bootstrap procedure for non-clustered data, a bootstrap procedure for multilevel data, and a recently-proposed parametric bootstrap procedure for estimating confidence intervals for predicted cluster-specific random effects . The papIn this section we provide a brief review of bootstrap procedures for clustered (or multilevel) data and a brief commentary on why some methods are not appropriate for making inferences about cluster-specific predicted-to-expected ratios.The conventional bootstrap draws a random sample with replacement from the original sample, such that the random sample has the same size as the original sample . While tThree different bootstrap procedures for use with linear mixed models have been described by van der Leeden and colleagues, by Goldstein, and by Carpenter and colleagues: the parametric bootstrap, the residuals bootstrap, and the non-parametric bootstrap –16. We wThe parametric bootstrap estimates the random effects logistic regression model in . In partThe residuals bootstrap is very similar to the parametric bootstrap described above. It differs from the parametric bootstrap in that, rather than simulating cluster-specific effects from the estimated distribution The non-parametric bootstrap, also referred to as the cases bootstrap, takes a bootstrap sample of the clusters. Once a cluster has been selected, all that cluster’s subjects are included in the bootstrap sample. Note that an average bootstrap will contain 63.2% of the clusters and omit 36.8% of the clusters. Importantly, those clusters that are contained multiple times in a given bootstrap sample are given different cluster identifiers so that they are treated as distinct clusters.As described elsewhere, these three bootstrap procedures allow one to make inferences about model parameters , however, they cannot be used to make inferences about the predicted cluster-specific random effects nor on quantities derived from them . With boj = 1,…,J. For each cluster, one then simulates a cluster-specific random effect: Austin and Leckie described a novel cluster-specific parametric bootstrap procedure for making inferences about cluster-specific random effects . After eWe conducted a series of Monte Carlo simulations to examine the performance of different bootstrap procedures for estimating confidence intervals for hospital-specific predicted-to-expected ratios generated using random effects logistic regression models. The design of the simulations was informed by empirical analyses of patients hospitalized with AMI.th and 75th percentiles: 16 and 148, respectively).We conducted a series of empirical analyses to determine the values of parameters that would be used in the data-generating processes in the subsequent Monte Carlo simulations. We used data from the Ontario Myocardial Infarction Database (OMID) which contains data on patients hospitalized with AMI in Ontario, Canada between 1992 and 2016 . For theWe considered two binary outcome variables: death within 30 days of hospital admission and death within one year of hospital admission. Outcomes were determined through linkage with the provincial death registry. Of the 19,559 patients, 1479 (7.6%) died within 30 days of hospital admission, while 2951 (15.1%) died within one year of hospital admission.We considered 11 variables for predicting mortality: age, sex, congestive heart failure, cerebrovascular disease, pulmonary edema, diabetes with complications, malignancies, chronic renal failure, acute renal failure, cardiogenic shock, and cardiac dysrhythmias. These 11 variables comprise the Ontario AMI mortality prediction model, which was derived in Ontario and was subsequently validated in Manitoba and California .We used conventional logistic regression to regress each of the two binary outcomes (death within 30 days and within one year) on the 11 variables in the Ontario AMI mortality prediction model. For each of the two fitted models, we determined the linear predictor for each subject. Thus, each subject had two linear predictors: one for each of the two outcomes. Each of the two linear predictors was standardized to have mean zero and unit variance across the sample. Each binary outcome was then regressed on the standardized linear predictor using a random effects logistic regression model that incorporated hospital-specific random effects. We computed the residual intraclass correlation coefficient (ICC), which is equivalent to the variance partition coefficient (VPC), using the latent variable approach .The mean intercept and the fixed slope for the random effects logistic regression model for 30-day mortality were -3.06 and 1.17, respectively, while the mean intercept and the fixed slope for the 1-year mortality model were -2.26 and 1.39, respectively. The residual ICC was 0.01 for both models, indicating that 1% of the variation in mortality unexplained by the standardized linear predictor was due to between-hospital differences. These quantities will be used in our subsequent data-generating processes.We allowed three factors to vary in our simulations: N , ICC (the intraclass correlation coefficient denoting the within-cluster homogeneity in the binary outcome), and the intercept and slope of the logistic regression model . N took on two values: 50 or 100 patients per hospital. ICC took on three values: 0.01, 0.02, and 0.05. The intercept and slope took on three combinations: , , and . The first intercept and slope were from the empirical analysis of 30-day mortality above. The second intercept and slope were from the empirical analysis of 1-year mortality above. The intercept in the third pair was set to zero so that the prevalence of the outcome would be approximately 0.5, allowing us to examine the performance of the bootstrap procedures in a scenario in which the prevalence of the outcome was high . We used a full factorial design and thus considered 18 (2 × 3 × 3) scenarios.We simulated data for subjects hospitalized at 50 hospitals . Our objective was to examine coverage of estimated confidence intervals for hospital-specific predicted-to-expected ratios. Thus, it is important that these hospital-specific ratios be treated as fixed parameters that are fixed across simulation replicates. Thus, within each of the 18 different scenarios we generated 50 hospital-specific random effects from a normal distribution: 2τ2+π2/3 . These 5ith patient at the jth hospital. Since the mean intercept and the fixed slope  we fit a random effects logistic regression model in which the binary outcome was regressed on the continuous baseline covariate. The model incorporated cluster-specific random effects. The predicted-to-expected ratio was computed for each of the 50 clusters, resulting in 50 cluster-specific predicted-to-expected ratios; (ii) we drew 1000 bootstrap samples from the simulated sample for the given simulation replicate; (iii) in each bootstrap sample we fit a random effects logistic regression model ) and computed the predicted-to-expected ratio for each of the 50 clusters (we thus had 1000 predicted-to-expected ratios for each of the 50 clusters); (iv) we constructed 95% confidence intervals for each hospital’s predicted-to-expected ratio. This was done using normal-theory bootstrap methods and percentile-based bootstrap methods. For the normal-theory bootstrap method, for each hospital, we computed the standard deviation of the estimated predicted-to-expected ratios across the 1000 bootstrap replicates. This quantity serves as an estimate of the standard error of the estimated predicted-to-expected ratio. A 95% confidence interval for each hospital’s predicted-to-expected ratio was then computed as the estimated predicted-to-expected ratio from the original simulated sample ± 1.96 × the bootstrap estimate of the standard error of the predicted-to-expected ratio. For the percentile-based bootstrap method, the end points of the 95% confidence interval were the 2.5We then conducted the following analyses across the 200 simulation replicates. First, for each of the 50 clusters we determined the ratio of the mean bootstrap estimate of the standard error of the predicted-to-expected ratio across the 200 simulation replicates to the standard deviation of the estimated predicted-to-expected ratio across the 200 simulation replicates. If this ratio is equal to one, then the bootstrap estimate of the standard error of the predicted-to-expected ratio is correctly approximating the standard deviation of the sampling distribution of the predicted-to-expected ratio. Thus, we obtained 50 such ratios, one for each of the 50 clusters. Second, for each of the two types of bootstrap confidence intervals , we determined the proportion of estimated 95% confidence intervals that contained the true value of the predicted-to-expected ratio for that cluster. If the estimated confidence intervals had the correct coverage rates, we would expect that 95% of the constructed confidence intervals contain the true value of the predicted-to-expected ratio for that hospital.We examined four different bootstrap procedures. First, we used the standard bootstrap in which subjects were sampled with replacement and the multilevel structure of the sample was not accounted for. We will refer to this as the naïve bootstrap. Second, we used a within-cluster bootstrap, in which a bootstrap sample of subjects is selected from within each cluster. Third, we used the parametric bootstrap procedure described above (this procedure was included despite our hypothesis that it would not perform well). Fourth, we used the bootstrap procedure for making inferences about cluster-specific random effects that was described above.The simulations were conducted using the R statistical programming language (version 3.6.3). Random effects logistic regression models were fit using the glmer function in the lme4 package (version 4_1.1–21).We report our results separately for each of the four bootstrap procedures.th percentile, median, 75th percentile, and maximum quantity  across the 50 clusters. On Fig. Results for the naïve bootstrap are reported in Fig. th percentile of the ratio of the mean estimated standard error to the empirical standard error was 0.31. Across the 50 clusters, the median ratio of the mean estimated standard error to the empirical standard error was 0.34. Across the 50 clusters, the 75th percentile of the ratio of the mean estimated standard error to the empirical standard error was 0.36. Finally, across the 50 clusters, the largest ratio of the mean estimated standard error to the empirical standard error was 0.42. These five quantities are represented by five different plotting symbols along the horizontal line. Note that all five quantities are to the left of the vertical line denoting a ratio of one. Thus, for none of the 50 clusters was the mean estimated standard error an accurate estimate of the empirical standard error.We provide a guide to interpreting Fig. In examining Fig. In examining Figs. These analyses demonstrate that the use of the naïve bootstrap results in inaccurate estimates of standard error and confidence intervals with lower than advertised coverage rates.Results are reported in Figs. 1Results for the parametric bootstrap are reported in Figs. Results are reported in Figs. We provide a case study illustrating the application of the four bootstrap procedures to a sample of 19,559 patients hospitalized with a diagnosis of AMI at 157 hospitals.ith patient at the jth hospital, and jth hospital.We used the OMID dataset that was described above. The outcome was death within 30 days of hospital admission. We used the 11 variables in the Ontario AMI Mortality Prediction Model (described above) for risk-adjustment. We regressed the binary outcome on these 11 variables using a random effects logistic regression model that incorporated hospital-specific random effects. The fitted model was The predicted-to-expected ratio was computed for each hospital. Each of the four bootstrap procedures was used to compute 95% confidence intervals around each hospital’s predicted-to-expected ratio. For each bootstrap procedure we constructed two confidence intervals: one using normal-theory methods and one using bootstrap percentile intervals.For comparative purposes we also fit the random effects model within a Bayesian framework using Markov Chain Monte Carlo (MCMC) methods . DiffuseCaterpillar plots illustrating each hospital’s predicted-to-expected ratio and its estimated 95% confidence interval are reported in Fig. Figure Figure We examined the performance of four bootstrap procedures for estimating confidence intervals for provider-specific predicted-to-expected ratios. We found that all four bootstrap procedures had suboptimal performance.The primary limitation of the current study was its reliance on Monte Carlo simulations. Such simulations were necessary since we were examining the performance of resampling-based procedures, for which analytic derivations are not feasible. Due to our use of simulations, we could only examine a limited number of scenarios due to the time-intensive nature of these simulations. Despite considering a limited number of scenarios, the performance of the different bootstrap procedures was consistently poor across these scenarios, indicating that, in general, these bootstrap procedures should not be used for estimating confidence intervals for predicted-to-expected ratios. A second limitation was that the simulations only used 200 iterations per scenario. The rationale for this decision was the computational intensity of simulations of bootstrapping of random effects models. For example, with 200 iterations per scenario, the simulations for the four bootstrap procedures required approximately 23, 27, 29, and 30 days of CPU-time for the 18 scenarios . Increasing the number of simulation replicates to 1000 would have been prohibitive in terms of computation time. With the use of 200 simulation replicates, empirical coverage rates that are less than 0.92 or greater than 0.98 are significantly different from the advertised rate of 0.95 using a standard normal-theory test.In the current study we focused on frequentist estimation of the random effects model used for computing the predicted-to-expected ratios. An alternative approach, as illustrated in the case study, would be to use Bayesian methods to estimate the posterior distribution of the model parameters and the resultant predicted-to-expected ratios. Different authors have suggested that Bayesian methods be used for provider profiling , 22, whiDirection for future research includes developing closed-form expressions for the standard error of the predicted-to-expected ratios or of developing bootstrap procedures that are appropriate for use with these measures of provider performance.Four bootstrap procedures were observed to result in inaccurate estimates of the standard errors of healthcare providers’ predicted-to-expected ratios and in confidence intervals that did not have the advertised coverage rates. We recommend that Bayesian methods be used for analyses involving predicted-to-expected ratios.