This paper aims to develop an automated web application to generate validated daily effective reproduction numbers (Rt) which can be used to examine the effects of super-spreading events due to mass gatherings and the effectiveness of the various Movement Control Order (MCO) stringency levels on the outbreak progression of COVID-19 in Malaysia. The effective reproduction number, Rt, was estimated by adopting and modifying an Rt estimation algorithm using a validated distribution mean of 3.96 and standard deviation of 4.75 with a seven-day sliding window. The Rt values generated were validated using thea moving window SEIR model with a negative binomial likelihood fitted using methods from the Bayesian inferential framework. A Pearson’s correlation between the Rt values estimated by the algorithm and the SEIR model was r = 0.70, p < 0.001 and r = 0.81, p < 0.001 during the validation period The Rt increased to reach the highest values at 3.40  and 1.72  due to the Sri Petaling and Sabah electoral process during the second and third waves of COVID-19 respectively. The MCOs was able to reduce the Rt values by 63.2 to 77.1% and 37.0 to 47.0% during the second and third waves of COVID-19, respectively. Mass gathering events were one of the important drivers of the COVID-19 outbreak in Malaysia. However, COVID-19 transmission can be fuelled by noncompliance to Standard Operating Procedure, population mobility, ventilation and environmental factors. As of 8 October 2021, more than 237 million cases have been reported with over 4.8 million deaths worldwide2. As the COVID-19 pandemic progresses, several countries globally have reported super spreading events that have attributed to the exponential rise in COVID-19 cases4. These events are likely to cause the observed sudden surge of cases due to increase disease transmission at mass gatherings and overcrowding events at particular localities5. In addition, infected individuals with higher disease transmissibility would further increase the risk of larger outbreaks6. Mass gatherings including political events, conferences, and sporting events are known to be super spreading events that have resulted in large COVID-19 outbreaks in several countries8.The novel Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) which was first discovered in Wuhan, China, late in December 2019 was declared a pandemic by the World Health Organization (WHO) in 20209. A much larger second wave started on 27 February 2020, following a 4-day mass gathering at Sri Petaling, Kuala Lumpur10. This mass gathering resulted in a massive transmission of COVID-19 infection in Malaysia which subsequently caused the largest COVID-19 cluster during the second wave with 3375 confirmed cases with 34 deaths reported12. Currently, Malaysia is facing the third COVID-19 wave which was triggered by the Sabah state electoral process on 26 September 2020 resulting in several mass gatherings and population movements13.In Malaysia, the first wave of COVID-19 that lasted for a month started as a result of imported cases on 25 January 2020 and reported a total of 22 cases14. The MCO aimed to flatten the epidemic curve and reduce disease transmissibility by enforcing movement restrictions and the prohibition of gatherings15. The strict MCO that was initially implemented was gradually relaxed accordingly with the reduction of reported cases. To strike a balance between lives and livelihood, adjustments to the stringency of the MCO are carried out through continuous assessments of the available disease transmission indicators such as incidence rate, growth rate, doubling time, and reproduction number15.In response to the COVID-19 pandemic, the Malaysian government instituted various Non-Pharmaceutical Interventions (NPIs), one of them being the Movement Control Order (MCO) which was first implemented during the second wave on 18 March 202017. Therefore, reproduction numbers are effective and useful indicators for disease surveillance and control purposes15.While indicators such as incidence rate, growth rate and doubling time may be useful for outbreak monitoring, they are unable to estimate disease transmissibility and cannot be used in forecast models. Studies have shown that estimating disease transmissibility during an epidemic, which is denoted by the reproduction number (R0), is important as it provides an aggregated measure of the transmissibility of an emerging infection with the ability to describe the progression of an epidemic, and provides a threshold of which an outbreak will either halt or continue to grow17. Values of the Rt identify changes in disease transmission over different points in time as a result of disease control interventions and population immunity17. Therefore, Rt values are key to characterizing the evolution of an outbreak, assessing the effectiveness of disease control measures, be used as an early warning outbreak indicator as it can provide a threshold for which an outbreak may increase or decrease exponentially, and be used as a parameter in forecast models18.The basic reproduction number (R0) is the average number of secondary cases of infections caused by a single infected individual over an infectious period which is static over time,, On the other hand, the time-varying reproduction number (Rt) is defined as the average number of secondary cases per primary case at a particular time point19. As cases of COVID-19 in Malaysia are increasing, there is a need to develop a process to estimate accurate and validated daily Rt values for Malaysia at the national and state levels daily. Hence, this paper aims to develop an automated web application to generate validated daily Rt values which can be used to examine the effects of super-spreading events due to mass gatherings and the effectiveness of the various MCO stringency levels on the outbreak progression of COVID-19 in Malaysia.COVID-19 is a novel disease that is characterized by sudden increases in transmissibility as a result of super-spreading events. Therefore, it is important to monitor the outbreak continuously  to provide sensitive information on the progression of the outbreak. Similarly, the MCO measures implemented to control this outbreak require constant adjustments to strike a balance between life and livelihoods. Rt estimates can provide a sensitive indicator on the current disease transmissibility and progression of the pandemic which is important for adjusting the stringency levels of the MCO14. As for disease spreading events, the duration of the event was measured as the period of the event and 14 days after to take into account the maximum incubation period of the disease for its effects to be observed, and also to factor in the effects of delay in case reporting. A 14-day incubation was selected as previous works in the literature have estimated the incubation period of COVID-19 to be anywhere between 2 and 14 days20.We measured the effects of different events on COVID-19 disease transmissibility by measuring the change and progression of the daily estimated Rt values for Malaysia and the states of Selangor and Sabah. These events include disease spreading events or disease propagation events such as large gatherings and movements of populations, and disease control events which are the MCO stringency levels during the first, second and third wave of COVID-19 outbreak in Malaysia. These MCO levels were categorized into enhanced Movement Control Order (MCO) phases 1 to 3, conditional Movement Control Order (CMCO), and recovery Movement Control Order (RMCO) which was enforced from 18 March 2020 to 29 March 202121. In addition, the introduction of vaccination during the fourth wave would further have an effect on the disease dynamics and confound the effect of the super spreading events and movement control orders21.The study period from 25 January 2020 to 29 March 2021 was selected as this study essentially focusses on the effects of super-spreading events and epidemic control measures  during the first, second and third waves of the COVID-19 outbreak in Malaysia. However, the period after 29 March 2021 was not included in this study as it was categorized as the fourth wave where there were several changes to the dynamics of the COVID-19 outbreak in Malaysia which would not allow for accurate measurement of effects of movement control measures and mass gathering events. During the fourth wave the categorization of the movement restrictions was changed into National Recovery Plan category 1 to 4 which is not comparable to the movement control order levels during the first three wavesEpiEstim EXCEL spreadsheet with macro function for the estimation of the Rt values18. R statistical software version 4.0.4 with packages such as incidence, xts, shiny and jsonlite were used for the development of the automated system for the generation of real-time daily Rt values22. The validation phase involved analyzing the data in R statistical software version 4.0.422 which involved performing Pearson’s correlation and Bayesian regression analysis for estimating the model fit.Data analysis was performed using the 18. The Rt was calculated by adopting and modifying the Rt algorithm developed by Anne Cori et al.18. This method produces statistically robust R values which incorporates uncertainty in the distribution of the serial interval using the earlier-mentioned EpiEstim EXCEL spreadsheet and calculates the Rt values using locally validated parameters18. The Rt values were estimated using daily case numbers sourced from the Crisis Preparedness Response Centre (CPRC), Ministry of Health Malaysia (MOH)13.Daily Rt values were estimated from 1 February 2020 to 8 November 2020 to quantify the temporal changes in the transmission intensity of the COVID-19 outbreak in Malaysia23. The generated Rt values for the first, second and third COVID-19 outbreak waves, and mass gathering events in Sri Petaling and Sabah, and the MCO phases were described in terms of the median, highest and lowest Rt values, and median Rt trends. Changes in the Rt point estimates and its 95% confidence intervals for each wave and corresponding events throughout its duration were quantified in terms of percentage increase or decrease by fitting a linear equation to the Rt trends of each wave using the EXCEL trend line function24. The daily percentage of increase or decrease is calculated based on the gradient of the linear equation whereby a negative gradient indicates a reducing trend and a positive gradient indicates an increasing trend25. The Rt values estimated during the first and second COVID-19 waves in Malaysia were generated using all reported cases. However, for the third wave, we estimated the Rt values using only community cases and excluded detention case clusters to specifically measure only the effects of mass gathering events and not detention clusters in the community13.The parameters and input data used in this algorithm to estimate the daily Rt values are (i) prior distribution mean, (ii) standard deviation, (iii) serial intervals, and (iv) daily COVID-19 case numbers. These parameter values were obtained from literature and validated for use in the Malaysian context23. In addition, the estimation of Rt by this algorithm requires a selection of a range of days (for the sliding window) where we attempted to use a 7- and 14-day width for the sliding window. The length of the window would affect the sensitivity of the estimates  in the estimates). Subsequently, a seven-day sliding window was selected as it provided the best balance between sensitivity and noise18.Parameter validation was carried out based on identifying the most appropriate parameter values by matching the generated Rt values using this algorithm to the R0 values obtained using the Susceptible-Exposed-Infected-Recovered (SEIR) models at specific time intervals, and subsequently, fitting the generated Rt trends to the COVID-19 cases in Malaysia. The final Rt algorithm was parametrized using a prior distribution mean of 3.96 and a standard deviation of 4.7515. Parameter estimates and uncertainty quantification were obtained using Bayesian methods of inference. R programming software was used to develop the codes together with observed data comprising of 55 data points of COVID-19 cases from 1 September 2020 to 25 October 2020. During the validation, a 14-day period was selected for the moving window so that the date in focus would be the central date of that window. Next, 95% and 99% equal tail credible intervals were obtained based on Bayesian posterior distributions and associated Monte Carlo computations26. The estimated Rt values were checked to see if they lied within each of their credible intervals, and a Pearson’s correlation analysis was performed27. A high percentage of coverage by the credible intervals and a significant correlation indicate that the Rt values obtained by the Rt algorithm were adequate measures for capturing the underlying disease dynamics. The output of Rt values estimated using the algorithm and outputs from the the SEIR model were also correlated using Pearson’s correlation during the second  and third  COVID-19 waves in Malaysia. Computing Pearson’s correlation required us to check various assumptions of linear regression under which interpretation of the correlation remains valid. These are explained in detail in the next paragraph.Similarly, the Rt values generated by the algorithm in this study were validated based on corresponding R0 values found using a moving window SEIR model28. Outliers were tested using box-plots where scores either less than or greater the lower and upper fences, respectively, were identified as outliers. The lower fence was calculated as the first quartile minus 1.5 times the interquartile range whereas the upper fence was calculated as the third quartile plus 1.5 times the interquartile range28. All scores were within the ranges of the fences, indicating no presents of outliers.Prior to conducting the computations related to the Pearson’s correlation, the assumptions of linear regression which include normality, homoskedasticity and linearity were tested. Normality of data was tested using the Kolmogorov–Smirnov (K–S) and Shapiro–Wilk tests where in p-values have shown evidence in favor of distributive normality29. The Rt values from the proposed algorithm and the R0 values from the moving window SEIR model were taken to be the response and explanatory variables, respectively. Spearman’s rank correlation coefficient was calculated for the resulting residuals and the corresponding predicted values of Rt. The value of the Spearman’s correlation had a corresponding p-value which was non-significant indicating that the residuals formed a uniform band around zero in the residual scatter plot, which also indicates the adequacy of the linearity and homoscedasticity assumptions29. Linearity was additionally corroborated using scatter plots between the Rt estimates generated from the algorithm and SEIR model where we found no evidence of ce of coning in the scatter plot, also indicating linearity29. In summary, all assumptions needed for obtaining the Pearson’s correlation coefficient were satisfied. Appendix 1 shows the results of normality, identification of outliers, homoscedasticity and linearity for the data during the validation period in the second and third waves.Homoscedasticity was tested using the Spearman’s rank correlation coefficient from a linear regression analysisincidence, xts, shiny and jsonlite packages. This application extracts the parameters and case data from relevant sources such as the National Security Council Malaysia (MKN) and MOH COVID-19 portals13. Automated parsing and structuring of extracted data were done in shiny server-side backend processing. Sourced data from MKN/MOH portals were transformed into a time series format using the incidence and xts packages of R, and the reactive function provided by R’s shiny application allows for reactive computation of outcomes based on user input or default parameters. Data syncing and back-end computations were done to ensure the stability and ease of use of this application.To provide real-time daily Rt updates, the algorithm was automated by developing a web-based application using the R programming software with reactive function from shiny enables cycle updates every second through the enveloped function which refreshes the required data and computes the Rt values resulting in real-time Rt updates. This process begins by using the jsonlite package to reformat the data sourced from MKN/MOH portal into an R readable format that is compatible with the algorithm. Preprocessed data were then fitted into the EpiEstim package for calculation of the real-time Rt values.This automated process updates the latest Rt values and minimizes inconsistencies due to human error. The In addition, the application allows for real-time changes in user input parameters by the continuous updating of backend data computations and reactive functions. Features developed for this application include the automated rendering of outputs, optional sliding windows size input and changes in parameter values which are used for the testing and optimization processes. An interactive interface with visual elements was also incorporated into the application to allow for smooth page rendering. These automated and reactivity functions were developed in this application to provide the most optimal user experience.To ensure the application produces continuous daily Rt outputs and to prevent downtime, several measures were taken. Firstly, downtime was prevented by introducing redundancies through hosting the application on a cloud-based server and a backup local server. Secondly, access to COVID-19 case data and parameter values were sourced from multiple open access data sources in the event of data inaccessibility at the source portals. Thirdly, the computational processes of the application can be overridden manually as a final failsafe mechanism in the event of automation failure.The automated web application system generates the daily Rt flow using a back-end and a front-end process as shown in Fig. The study was registered with National Medical Research Register (NMRR-21-300-58289). No ethics approval was required.The validation of the Rt values generated by the algorithm was performed by comparing them with the corresponding R0 values estimated by the moving window SEIR model fitted to observed COVID-19 cases. To account for uncertainty, the validation—was performed using the posterior distribution of Rt obtained from the Bayesian inference methodology. An illustration of this methodology is provided for the date of 6 October 2020 where the Rt value of 1.78 was found by the algorithm. The resulting posterior distribution of Rt had the highest posterior frequency around the same value of 1.78 see Fig. . For thiThus, to validate accurately the Rt values generated by the algorithm, these Rt estimates were corroborated by R0 values produced by the moving window SEIR model as above. A Pearson’s correlation between the Rt values estimated by the algorithm and the moving window SEIR model was r = 0.70, p < 0.001 and r = 0.81, p < 0.001 during the validation period in the second wave  and third wave  respectively as shown in Fig. The daily Rt values were estimated for all three COVID-19 waves from 1 February 2020 to 29 March 2021 as shown in Table For the first wave, a total of 26 Rt values were estimated with the highest and lowest values were estimated at Rt = 1.64  and Rt = 0.32  on 8 February 2020 and 23 February 2020 respectively. The range was between Rt = 0.32 to 1.64 with a median Rt = 0.95. The median daily Rt reduction of 2.80%  was observed from 25 January 2020 to 26 February 2020 (first wave).As for the second wave, a total of 206 Rt values were estimated with the highest and lowest values were estimated at 3.40  and 0.28  on 29 February 2020 and 11 June 2020 respectively. The range was between Rt = 0.28 to 3.40 with a median Rt = 1.00. The median daily Rt reduction of 0.43%  was observed from 27 February 2020 to 19 September 2020 (second wave).During the second wave, there was a total of six events observed, a mass gathering in Sri Petaling (disease propagation event) and another five MCO phases (disease control events) shown in Table As for the mass gathering event in Sri Petaling Gathering, a total of 18 Rt values were estimated with the highest and lowest values were estimated at Rt = 3.40  and Rt = 1.37  on 29 February 2020 and 12 March 2020 respectively. The range was between Rt = 1.37 to 3.40 with a median Rt = 1.95. The median daily Rt increase of 2.82%  was observed during this event.During the MCO phase 1, a total of 14 Rt values were estimated with the highest and lowest values were estimated at Rt = 1.84  and Rt = 1.06  on 18 March 2020 and 30 March 2020 respectively. The range was between Rt = 1.06 to 1.84 with a median Rt = 1.25. The median daily Rt reduction of 5.27%  was observed during this event.As for MCO phase 2, a total of 14 Rt values were estimated with the highest and lowest values were estimated at Rt = 1.10  and Rt = 0.89  on 3 April 2020 and 10 April 2020, respectively. The range was between Rt = 0.89 to 1.10 with a median Rt = 1.02. The median daily Rt reduction of 1.00%  was observed during this event.For the MCO phase 3, a total of 14 Rt values were estimated with the highest and lowest values were estimated at Rt = 0.94  and Rt = 0.64  on 16 April 2020 and 21 April 2020 respectively. The range was between Rt = 0.64 to 0.94 with a median Rt = 0.78. The median daily Rt reduction of 1.10%  was observed during this event.While for the CMCO phase, a total of 42 Rt values were estimated with the highest and lowest values were estimated at Rt = 1.69  and Rt = 0.55  on 26 May 2020 and 2 June 2020 respectively. The range was between Rt = 0.55 to 1.69 with a median Rt = 0.94. The median daily Rt increase of 0.43%  was observed during this event.Finally, during the RMCO phase, a total of 83 Rt values were estimated with the highest and lowest values were estimated at Rt = 1.37  and Rt = 0.28  on 22 July 2020 and 11 June 2020 respectively. The range was between Rt = 0.28 to 1.37 with a median Rt = 0.93. The median daily Rt increase of 0.48%  was observed during this event.The highest estimated Rt during the second wave of COVID-19 was 3.40, this was for the Sri Petaling event. Following this event, the Rt started to drop for each phase of the MCO to reach a median value of Rt = 0.93 for the RMCO. The reduction from the highest estimated Rt during the Sri Petaling event (Rt = 3.40) as a result of the various phases of MCO for the second wave of COVID-19 ranged from 63.2% to 77.1%. Figure The highest recorded Rt in Selangor State following the Sri Petaling Mass Gathering event was Rt = 3.80 (95% CI 3.01–4.68) on 19 March 2020 as shown in Fig. During the third wave, a total of 191 Rt values were estimated with the highest and lowest values were estimated at 1.72  and 0.81  on 25 September 2020 and 3 March 2021 respectively. The range was between Rt = 0.81–1.72 with a median Rt = 1.06. The median daily Rt reduction of 0.24%  was observed from 20 September 2020 to 29 March 2021.As for the third wave (as of 29 March 2021), there was a total of three events observed of which one was a propagation event which was a mass gathering following electoral processes in Sabah and the others were MCO and MCO with conditional MCO (CMCO) as shown in Table While for the mass gathering following the electoral process in Sabah, a total of 29 Rt values were estimated with the highest and lowest values were estimated at Rt = 1.72  and Rt = 0.96  on 25 September 2020 and 16 September 2020 respectively. The range was between Rt = 0.96 to 1.72 with a median Rt = 1.42. The median daily Rt increase of 1.40%  was observed in this event.For the MCO, a total of 114 Rt values were estimated with the highest and lowest values were estimated at Rt = 1.34  and Rt = 0.92  on 14 October 2020 and 1 December 2020 respectively. The range was between Rt = 0.92 to 1.34 with a median Rt = 1.09. The median daily Rt reduction of 0.04%  was observed in this event.While for the MCO with CMCO phase, a total of 53 Rt values were estimated with the highest and lowest values were estimated at Rt = 1.00  and Rt = 0.81  on 25 March 2021 and 3 March 2020 respectively. The range was between Rt = 0.81 to 1.00 with a median Rt = 0.91. The median daily Rt increase of 0.01%  was observed during this event.The highest estimated Rt for the third wave of COVID-19 was Rt = 1.72, this was due to the mass gathering following the electoral process. Following this event, the Rt started to drop to reach a median value of Rt = 0.91 for the MCO with CMCO phase. The reduction from the highest estimated Rt during the Sabah electoral process (Rt = 1.72) as a result of the various phases of MCO for the third wave of COVID-19 ranged from 37.0 to 47.0%. Figure 30. This is because of the novel nature of this disease, the absence of baseline data as a reference and the explosive nature of this pandemic. Besides this, the indicators used to monitor the progression of the COVID-19 pandemic such as case incidence, doubling time and growth rate are unable to provide accurate and sensitive monitoring of the pandemic as these indicators were unable to determine the disease transmissibility17. However, literature suggests that the most effective way to measure disease transmissibility is by determining its R value. This is because R values are able to provide sensitive indicators that reflects the disease transmissibility and provide time-dependent variations in the transmission potential and progression of the pandemic17.Challenges faced by health systems in monitoring COVID-19 have resulted in failures to contain the spread of the pandemic31. More specifically these methods are Exponential growth rate, Maximum Likelihood, Markov Chain Monte Carlo, Next-Generation Matrix Approach, Epidemic curve model, and compartmental model methods32. However, estimating the Rt using these methods are complex, time consuming, and may cause challenges to provide real time daily Rt estimates for multiple geographical areas. In addition, developing these model requires advance statistical capabilities, and furthermore, it may not be possible to be carried out at the ground level where the outbreak is occurring as it requires complex computations for each Rt value generated31.There are several methods to estimate the Rt values that are based on fitting mechanistic or deterministic transmission models to epidemiological data18. The Rt algorithm used in this paper was simple to implement and is more flexible than using the above mention methods or fitting mechanistic or deterministic transmission model methods. In addition, this algorithm is able to produce similarly accurate Rt values quickly and efficiently, compared to those complex methods as above. Furthermore, we validated this algorithm for its accuracy, as to date there has not been any Rt values generated for the COVID-19 pandemic in Malaysia that has been statistically validated using Bayesian inferential methods. In this study the Rt values generated was validated using Bayesian methods as it was the most appropriate validation approach, allowing for model validation under conditions of parameter uncertainty and with limited testing data33. The results of the validation reported highly correlated values that suggest that the Rt values generated by the algorithm were accurate. Validation using the above method ensures that the Rt values generated using the algorithm in this study are valid, reliable, context-specific and represents the COVID-19 disease trends in Malaysia. With this validated algorithm, it can be concluded that the generated Rt values are sensitive indicators that are essential for monitoring and controlling the COVID-19 pandemic in Malaysia.In this paper we are able to generate real time daily COVID-19 Rt values for Malaysia at national and state levels by adopting and modifying the Rt estimation algorithm developed by Anne Cori et al.MySejahtera mobile application and the MOH COVID-19 website would increase public awareness and compliance to the COVID-19 Standard Operating Procedures (SOPs). The Rt estimation algorithm is an important intervention developed in this study which assists in the control of the COVID-19 pandemic in Malaysia.Since the Rt estimation algorithm in this study generated valid daily Rt values that are essential in monitoring and assisting in the control of the COVID-19 pandemic in Malaysia, we developed an automated web application that uses the validated algorithm to estimate daily real-time Rt values for Malaysia. In the absence of a real-time COVID-19 transmissibility monitoring system in Malaysia, this automated application assists the surveillance system by providing daily Rt values in real time to the health authorities and the public. Providing regular COVID-19 Rt updates allows for evidence-based adjustment of control measures, enables the assessments of the effectiveness of control measures and assists in the accurate forecasting of COVID-19 case trends in Malaysia. In addition to this, continuous access of Rt values to the public via the 35. In India the religious mass gatherings had resulted in a large increase in COVID-19 cases observed in March 20207. While in the United States, electoral process had led to a 45% rise in daily cases in November 202034. In addition, mass confinement of individual as observed in Diamond Princess cruise ship in Japan resulted in a high R value of 11 which further suggests that mass gatherings especially in confined environments are key drivers for the explosive transmission of COVID-1935. This can be explained by the increased risk of disease transmissions due to the large aggregation of individuals in a particular area in a given time, which allows for propagation of droplets and airborne transmission as observed during electoral process seen in Sabah5. In addition, this method of transmission increases especially in confined spaces over prolonged periods as observed during the Sri Petaling mass gathering. Furthermore, a lack of enforcement and compliance to Standard Operating Procedures (SOP) such as the use of face masks, hand/cough hygiene, physical distancing during these events and environmental factors such as temperature, relative humidity, pollution, solar radiation may further increase the explosive nature of the outbreak36.Being able to generate daily validated COVID-19 Rt values, we examined the outbreak propagation and control events in Malaysia by measuring the changes and progression of the daily estimated Rt values as a result of these events. We found that in Malaysia, mass gathering events were one of the main drivers for the COVID-19 pandemic. For example, the Sri Petaling mass gathering in February 2020 and the electoral process in Sabah in September 2020 were catalysts for the second and third COVID-19 waves in Malaysia respectively. Similar findings of mass gatherings events resulting in increased transmissibility of COVID-19 have been reported in India, United States, JapanThe above suggestions can be explained by the findings in this study where we found that the National daily estimated Rt increased to 1.72 during the Sabah electoral process which was a mass gathering event. However, the Rt during the Sri Petaling event was much higher at 3.40 as it was a mass gathering event conducted in a confined space. In addition, due to this, the daily case numbers started to increase 1 to 4 weeks following the onset of these events.37. Compared to the Sri Petaling event, the Rt during the Sabah electoral process was lower, as this mass gathering event did not occur in a confined area and consisted of multiple propagated small events occurring in open areas out over a longer period. Furthermore, there were already established ongoing COVID-19 control measures and SOP compliance compared to the Sri Petaling event which resulted in this difference.In addition, the high Rt observed during the Sri Petaling mass gathering could also be attributed due to the event occur early on in the pandemic, whereby there was a lack of awareness among the public regarding COVID-19 resulting in poor compliance to SOPs To further confirm the effects of these events on the transmissibility of the COVID-19 pandemic in Malaysia, we found that the rise in Rt values were higher in the states where the events occurred as compared to the National Rt during the similar period. Where in the Rt for Selangor in the second wave and Sabah in the third wave was 3.80 and 3.46 respectively which was significantly higher than the National Rt of 3.40 and 1.72 during the second and thirds waves respectively. This finding shows that the increase in Rt was attributed to the mass gathering events as evident by the increase of Rt at the states where these events occurred. This supports our suggestion that these mass gathering events were among the factors responsible for the second and third COVID-19 waves in Malaysia.14. Instituting the MCO has been shown to be an effective measure to control the transmission of COVID-19 as these restrictions limit physical contact by avoiding unnecessary movements of population15. In this study we found that during the second wave of COVID-19 pandemic in Malaysia the MCO phase was able to reduce the Rt value by 63.2% to 77.1%. Similarly, a reduction of 37.0% to 47.0% was observed during the MCO and CMCO phase in the third wave. Similar findings have been observed in Canada, China, Mexico and Niger whereby during the lockdown periods, the R values were reduced by 73% to 82% compared to pre-lockdown levels38.In response to the surge of cases as a result of these mass gathering events, the Government of Malaysia instituted Non Pharmaceutical Interventions such as the MCO39. In order to strike a balance between life and livelihood, the MCO allows for adjustments to its intensity by a step up and stepping down approach for example the MCO, CMCO and RMCO40. In this paper, we found that higher MCO stringency levels resulted in a larger reduction in daily Rt values, wherein we found the MCO phases resulted in a daily Rt reduction ranging from 1.10 to 5.27%. Compared to MCO, the CMCO and RMCO resulted in an increase in the daily Rt at 0.43% and 0.48% respectively. While the there was an increase in daily Rt during the CMCO and RMCO phases with median Rt of 0.94 and 0.93 respectively compared to the Rt of 3.4 during the Sri Petaling event, our results suggest that by adjusting MCO stringency level it is possible to balance the need to control the pandemic while ensuring the continuation of social and economic activities. Similarly a study in China found an inverse relation between the severity of movement restriction and Rt values, wherein a 20 to 60% mobility restriction resulted in Rt reduction of 25 to 50% respectively41.While movement restrictions are effective mechanism to control the COVID-19 pandemic, however its implementation has an effect on social and economic situation of the nationThis study has several strengths, first, we developed a web-based automated system that operates on Rt algorithms that was validated using computationally advantageous approaches (i.e. SEIR models and the existing Bayesian framework) to produce continuous, real-time Rt values which are context-specific for Malaysia at a national and state level. To date, the validation approach used has not been done in Malaysia as of this paper. Second, the Rt algorithm used in this study is an easier to implement, more flexible method that accounts for the uncertainty of the serial interval distribution which is sensitive to the size of sliding windows. Third, to ensure the Rt values generated truly reflect the community transmissibility of COVID-19 in Malaysia, we excluded detention cases from the daily case incidence during the third wave, as during this period there were clusters involving detention centres reported daily.The limitations of this paper include firstly, data limitations due to surveillance effects such under reporting and late reporting due to delay in testing. In addition, inaccuracies of testing protocol resulting in false positive or negative results, however this is minimal as confirmation of cases was test using PCR test. Second, limitation of methodology of estimating Rt, where in the Rt algorithm in this study does not include all the factors  potentially driving the spread of COVID-19’. Future research should aim to use the current models with appropriate modification/adjustment, to account for the effects of vaccinations  and introduction of other variants.42. Furthermore, we provided evidence that the MCOs were able to reduce the daily Rt values, especially MCOs with higher stringency levels would correspondingly result in decreasing Rt values. Finally, we demonstrated the applicability and utilization of the Rt estimation algorithm in this paper in assisting both the health authorities and the public to monitor and control the COVID-19 pandemic in Malaysia.The findings of our study confirmed that the Rt estimation algorithm was able to generate validated daily Rt values at the National and State level for Malaysia. These values were able to provide accurate and sensitive information on the disease transmissibility, transmission potential, disease progression and forecast in a timely fashion for the control of the COVID-19 pandemic in Malaysia. This paper has shown the effects of mass gathering and movement control measures on the transmissibility of the COVID-19 outbreak in Malaysia. In addition, we provide strong evidence that mass gathering events were one of the important drivers and contributed to towards the spread of COVID-19 in Malaysia, especially mass gathering in confined areas which increases the disease transmission significantly compared to open areas. However, the spread of disease can also be fueled by noncompliance to SOPs , poor ventilation and environmental factors Supplementary Information.


Breast cancer is the second most commonly diagnosed type of cancer among women as of 2021. Grading of histopathological images is used to guide breast cancer treatment decisions and a critical component of this is a mitotic score, which is related to tumor aggressiveness. Manual mitosis counting is an extremely tedious manual task, but automated approaches can be used to overcome inefficiency and subjectivity. In this paper, we propose an automatic mitosis and nuclear segmentation method for a diverse set of H&E breast cancer pathology images. The method is based on a conditional generative adversarial network to segment both mitoses and nuclei at the same time. Architecture optimizations are investigated, including hyper parameters and the addition of a focal loss. The accuracy of the proposed method is investigated using images from multiple centers and scanners, including TUPAC16, ICPR14 and ICPR12 datasets. In TUPAC16, we use 618 carefully annotated images of size 256×256 scanned at 40×. TUPAC16 is used to train the model, and segmentation performance is measured on the test set for both nuclei and mitoses. Results on 200 held-out testing images from the TUPAC16 dataset were mean DSC = 0.784 and 0.721 for nuclear and mitosis, respectively. On 202 ICPR12 images, mitosis segmentation accuracy had a mean DSC = 0.782, indicating the model generalizes well to unseen datasets. For datasets that had mitosis centroid annotations, which included 200 TUPAC16, 202 ICPR12 and 524 ICPR14, a mean F1-score of 0.854 was found indicating high mitosis detection accuracy. Segmentation of Mitoses and Nuclei using Conditional GANs,,,,,,,Mitosis is the process of cell duplication, in which one cell divides into two genetically identical daughter cells.In this work, we consider three open source datasets for mitosis detection and segmentation: TUPAC16, ICPR12, and ICPR14. From each dataset, random patches of size 256×256 were extracted from the HPF from each dataset such that patches had at least one mitosis that was not on the boundary. TUPAC16 has a total of 656 HPF for 73 patients, scanned at 3 different centers. Annotations for both nuclei and mitoses were generated for TUPAC16 since it contains more variability (three centers with wide stain variations). Annotations were generated using Pathcore SedeenG1) and global (G2) enhancer which aims to maintain and transfer global information from G1 to G2. G1 and G2 generators are jointly trained to generate masks from input images and the inputs are 128×128 and 256×256 in size, respectively. The last convolution layer in the architecture is followed by Tanh. The discriminator in D1, D2, D3) which use images of different resolutions to predict image realism based on multiple resolutions. For each scale, the discriminator adopts the PatchGAND1, D2, and D3 discriminators are trained simultaneously to distinguish the generated segmentation masks from ground truth images.The proposed cGAN architecture called MiNuGAN for dual mitosis and nuclear segmentation is adapted from the original pix2pixHD cGAN architectureLGAN and the feature matching (FM) loss LFM as inIn this work, a focal lossλ balances the influence of the feature matching loss LFM with the adversarial loss LGAN to prevent overfitting to specific distributions. The feature matching loss considers dissimilarities between feature representations in local and global levels. The adversarial loss tries to maximize the probability of real images and minimize the probability of fake inputs to the discriminator. In this work, LGAN is computed by considering a focal loss parameter γ that reduces or increases the relative loss for well-classified or poorly classified pixels, as inwhere The output of the model is a mutliclass image with pixel-level labels for each of the predicted mitosis and nuclear objects. To measure segmentation performance, pixel-level predictions are compared to ground truth delineations on a per-pixel basis. For mitosis detection, the mitosis segmentation masks are used to automatically determine the centroids of each mitosis object.We evaluate segmentation performance of the cGAN model using the dice similarity coefficient (DSC) which compares automated segmentations to manual delineations for the mitosis and nuclear classes separately. To measure mitosis detection performance on datasets with ground-truth mitosis centroids, the F1-score is utilized. Precision and sensitivity are determined by the number of correctly predicted mitosis centroids within a 25 pixel-size window centered on the ground truth centroid. This corresponds to a distance of 7 μM and was chosen since it is the average of the mitosis competitions.All models are developed using TUPAC16 data with pixel-level annotations of nuclei and mitoses . Average segmentation performance is reported for all 418 patches for nuclei and mitoses separately. Model fine-tuning is done using the detailed annotation dataset and the final dual nuclear and mitosis segmentation model termed "MiNuGAN" is carried forward for further analysis. To measure generalization performance of the proposed system, two held-out test sets with pixel-level annotations were used. There were 200 patches from TUPAC16 that had mitosis and nuclear annotations, and 202 patches from ICPR12 with mitosis delineations. The segmentation results of MiNuGAN are compared to baseline models, such as U-Net and U-Net with a focal loss, which all were trained using the same 5-fold cross-validation datasets. In addition to measuring segmentation performance, additional experiments were conducted to measure mitosis detection accuracy through the F1-scores. These are reported all held out datasets for mitosis centroid annotations, which includes: ICPR12 (202 patches), ICPR14 (524 patches), and for the held-out TUPAC16 set (200 patches). All models are trained on Nvidia 2080TI GPU with 11 GB memory. The training images consist of original images along with RGB label maps . All images are of size 256×256.γ = 0; 0:1; 1; 2; 5 to increase the importance of mitosis when classifying pixels. We varied the number of discriminators, residual blocks, λ, and the focal loss and report some of the top-performing configurations for mitosis and nuclear segmentation. In total, this resulted in 10 configurations as outlined in γ = 2. This configuration also results in the lowest coefficient of variation, indicating that this model segments mitoses with the highest consistency. Visual results for the top-performing method  and the same architecture without focal loss are shown Performance of the proposed mitosis and nuclear segmentation model is reported in this section. In total, 418 patches  from TUPAC16 are used for training and model fine-tuning without any preprocessing such as stain normalization. The dataset is split in a 5-fold stratified fashion  and average performance on the validation data over 5-folds is reported. Each model is trained for 100 epochs with a learning rate of 0.5. The baseline pix2pixHD model has three multiscale discriminators, nine residual blocks in the generator and uses λ = 5. In addition to these parameters, we also investigated a focal loss with γ = 2) with focal loss and call this configuration MiNuGAN.When considering the DSC for the nuclear and mitosis class separately, it is apparent that lower performance is generally obtained for the cells in mitosis. This could be a result of the complex nature and variable appearance of mitotic cells and could also be due to a relative class imbalance. However, as shown in Using the finalized model MiNuGAN just described, the mitosis segmentation testing performance on the held-out TUPAC16 and ICPR12 is computed. In total, there were 402 patches, and the results are summarized in Second, the ICPR12 dataset is of relatively high contrast, which may permit for better mitosis segmentation. Third, the discriminator function allows the classifier to learn realistic mitosis patterns. The high segmentation performance across both datasets is a good indication that these features are beneficial for generalization to new data. To compare the performance of the MiNuGAN to baseline systems, we compared segmentation results over the validation set with U-Net and U-Net with a focal loss. These systems were trained for 100 epochs using Adam optimizer, 0.001 learning rate, batch size of 16, 4 layers and with/without focal loss. The same training and validation datasets were used in a 5-fold cross-validation manner. Using the baseline cGAN system with focal loss (MiNuGAN), the detection performance of the proposed method is evaluated on images from multiple datasets and scanners . As the model predicts both nuclei and mitoses on a per-pixel basis, only mitosis labels are retained for detection. Morphological processing is applied to the mitosis masks  and the centroid of every object is determined. The ground-truth centroid is compared to the predicted centroids through the F1. See This work proposes a cGAN-based model called MiNuGAN that segments both mitoses and nuclei at the same time from H&E stained breast cancer images. It is based on the baseline pix2pixHD model, except a focal loss has been added to bring attention to the imbalanced mitosis class. Testing results show that segmentation performance is high for both mitoses and nuclei in TUPAC16 with mean DSC = 0.721 and 0.784 for mitoses and nuclei, respectively, over 200 image patches and mean DSC = 0.782 for 202 ICPR12 images. Results show the proposed method generates more realistic and accurate segmentation maps compared to baseline U-Net models, even U-Net with focal loss, indicating the discriminator forces continuity in the segmentation mask that is beneficial for segmentation. Using the mitosis segmentations, mitosis centroids were detected and compared to datasets with centroid annotations. The F1-scores were computed over three multicenter H&E datasets  and an average F1-score of 0.854 was achieved. Ablation studies show the effectiveness of the focal loss to increase the performance of mitosis segmentation and detection. In the future, stain normalization will be considered, as well as augmentation to see if performance across scanners and centers can be improved. In this work, we focused mainly on mitosis segmentation and detection in breast cancer H&E images. In the future, it is possible to evaluate MiNuGAN on different tissue types and datasets.


Controlling transmission in restaurants is an important component of public health and social measures for coronavirus disease. We examined the effects of restaurant measures in Hong Kong. Our findings indicate that shortening operating hours did not have an effect on time-varying effective reproduction number when capacity was already reduced. As of April 14, 2021, a total of 11,608 cases and 207 deaths from coronavirus disease (COVID-19) had been reported in Hong Kong   to assess the effect of the ban on dine-in services after 6:00 pm on Rt, accounting for the effect from other PHSMs  decrease in Rt (Table).We estimated that the ban on dine-in services after 6:00 t increased to 3.1 on November 16, 2020, and then decreased to ≈1.0 after PHSMs began (>3 other PHSMs was associated with a 40% (95% CI 28%–47%) decrease in Rt. Another model that excluded basic civil service arrangement in other PHSMs showed that a ban on dine-in service beginning at 6:00 pm did not have an effect (Table). We performed sensitivity analysis to remove the effect of superspreading in wave 3 by changing the start date to July 1, 2020; we found the ban on dine-in service from 6:00 pm did not have an effect (In wave 4, Rn effect  Table 2.pm might not have had an effect in both waves when capacity was already reduced. A complete closure of restaurants in Hong Kong would have considerable social impact because dining out is very common. We hypothesize that encouraging restaurants to extend dine-in hours, but with capacity restrictions to reduce crowding, could be a reasonable approach to reduce transmission. Our analysis suggested that the PHSMs were critical for suppressing the third and fourth waves of COVID-19 in Hong Kong. However, we found that a ban on dine-in hours after 6:00 pm might have an effect if it began earlier than other PHSMs or in regions with high incidences. In addition, changes in Rt are a consequence of individual behavioral changes such as avoiding crowded areas; increasing incidence and implementation of multiple PHSMs could raise the public’s perception of risk. Determining the effectiveness of alternative PHSMs would provide evidence-based guidance on control strategies.A limitation of our analysis is that we cannot distinguish the effect of some PHSMs because they began simultaneously. We cannot rule out that a ban on dine-in service after 6:00 Additional information about public health and social measures to reduce transmission of coronavirus disease in 2 waves of the epidemic, Hong Kong, 2020–2021


Supplemental Digital Content is available in the text. Among patients listed for kidney transplantation, the Karnofsky Performance Status (KPS) Scale has been used as a proxy for frailty and proposed as a predictor of long-term posttransplant outcomes. The KPS is required by the Organ Procurement and Transplantation Network for all transplants; however, the interrater reliability of KPS reporting in kidney transplant candidates has not been well investigated, and there is concern regarding limitations of using KPS that may influence transplant eligibility.We performed an observational study using existing Scientific Registry of Transplant Recipients data from 2006 to 2020 to examine the variability, reliability, and trends in the KPS among patients on the kidney transplant waitlist.Our analysis included 8197 kidney transplant candidates with >1 KPS in a 3-mo period. We observed 2–7 scores per patient with an average score of 78.9 . We found substantial variability in KPS reporting, in which 27% of the patients had scores that varied widely with 20–80 points in difference. Interrater reliability in the 10-point scale was poor (30%). When using a condensed 4-category scale , 38% of patients experienced at least a 1-category shift in their score.The lack of reliability in KPS reporting raises concerns when applying the KPS as a proxy for frailty and a metric to be considered when evaluating candidacy for kidney transplantation. Hence, investigators have used proxy metrics to evaluate the implications of frailty in studies of transplant candidates. The Karnofsky Performance Status (KPS) is a subjective measure of functional impairment. In prospective kidney transplant candidates, KPS is collected by a healthcare provider at the time of listing and repeated at the time of transplant as well as in follow-up visits. These data are required by the Organ Procurement and Transplantation Network (OPTN) for all transplants and until 2011, was used to risk-adjust center-specific outcomes.4Frailty is a pathobiological process characterized by loss of physiologic reserve and increased vulnerability to stressors.5 The scale ranges from normal functioning (100%) to dead (0%) with increments of 10% in between. Each incremental increase describes the level to which the patient is able to perform his or her daily activities independently . Despite its limitations, the KPS continues to be used as a proxy for frailty and may influence kidney transplant eligibility. In turn, this may translate into poorer access to transplantation in some disadvantaged populations.The reliability of reporting KPS in kidney transplant candidates has not been well studied. Several studies have evaluated the interrater reliability in patients with cancer.To address our concerns regarding reliability and validity of the KPS as a proxy for frailty, we performed an observational study using existing SRTR data to examine the degree of variation in reporting of KPS. Patient listings at multiple sites allows for a natural experiment in which multiple scores are collected by different raters in a relatively short period of time. Using data available from 2006 to 2020, we evaluated the variability, reliability, and trends in the KPS among patients on the kidney transplant waitlist.This study was performed in accordance with the Declaration of Helsinki and was approved by the Stanford Institutional Review Board (protocol no. 40876). The clinical and research activities being reported are consistent with the Principles of the Declaration of Istanbul as outlined in the “Declaration of Istanbul on Organ Trafficking and Transplant Tourism.” Because of the retrospective and observational nature of the research, the need for written informed consent was waived.This study used data from the SRTR. The SRTR data system includes data on all donors, waitlisted candidates, and transplant recipients in the United States, submitted by the members of the OPTN. The Health Resources and Services Administration, US Department of Health and Human Services, provides oversight to the activities of the OPTN and SRTR contractors. We selected adult (age 18+) patients on the kidney transplant waitlist from January 1, 2006, to June 2, 2020. We opted to exclude listings before 2006 where there was a mix of coding the KPS as 3 categories instead of the 10-point scale used today. We selected pretransplant scores only and excluded multiorgan  candidates. Ideally, we wanted measurements as close in time as possible to reduce the chance for functional decline without losing generalizability across the US sample. In reality, multiple listings require time for referral and travel. To address this, we restricted the analysis to patients with at least 2 scores within 3 mo.The KPS was categorized into 4 groups: normal activity (80%–100%), capable of self-care (70%), requires assistance (50%–60%), and disabled (10%–40%). The additional category at KPS = 70% was partitioned out to address some of the skewness in the data . We included the following covariates: age, sex, race, ethnicity, Quételet (body mass) index (BMI), education, employment status, insurance type, year of listing, diabetes, peripheral vascular disease, chronic obstructive pulmonary disease, cerebrovascular disease, coronary artery disease, blood type, dialysis vintage, and previous transplant. We included patient ID and center ID to adjust for subject-to-subject and center-to-center differences. These variables are all recorded in the transplant candidate’s file within the SRTR database and updated at each time of listing. We specified the KPS and covariates at the time of the first listing as baseline measurements.11 Instead we estimated the intraclass correlation coefficient (ICC) from random effects models with patient and center random effects12 . The ICC is an estimate of the intrarater agreement in which <50% is considered poor reliability, 50%–75% is moderate reliability, 76%–90% is good reliability, and 91%–100% is excellent reliability.13 We conducted all analyses with SAS for Windows version 9.4,14 R Studio v1.3,15 and Winbugs v14.17We described covariate data by baseline KPS category as frequencies and percentages. We performed a descriptive analysis on the range of scores for each patient by comparing the maximum and minimum scores in the 3-mo period. Because not all patients were evaluated at all the centers and there were variable numbers of measurements per patient, usual kappa tests of interrater agreement could not be performed.There were 677 019 adult patients listed for a kidney or pancreas transplant in the SRTR KI/PA candidacy file with a KPS score 1972–2020) and a valid center ID. From these, we excluded patients who were listed for a pancreas transplant along with patients simultaneously listed in the other transplant candidacy lists . We excluded all listings before 2006 and KPS scores that did not fall in 0%–100% range (recorded as 2010–2100 in the SRTR files). After these exclusions, 377 109 (55.7%) remained. Finally, we restricted the cohort to 8197 (2.2%) patients with >1 KPS in a 3-mo period  of patients had a first KPS in the normal range, whereas only 1% (n = 86) had a KPS in the disabled range. Lower KPS was associated with racial  and ethnic diversity . We observed a dramatic shift in the KPS by year of observation, where lower KPS tended to be more common in recent years (2016–2020) than historically (47% versus 8% disabled). Patients with low KPS were more likely to be on dialysis and have longer dialysis vintage than patients with a high KPS  . Thirty-eight percent of patients were repeatedly scored the same , 36% had minimal difference (10 points), whereas the remaining 27% of patients had scores that varied widely with 20–80 points in difference . Therefore, we only reported the nonlinear random effects model results in Table SAS software for analyzing nonlinear random effects models is currently limited to a single random effect. To check the fit of the linear random effects model to the KPS data, we compared results against a nonlinear random effects model assuming an ordinal response. Data were sparce at the low end of the score; thus, we collapsed scores in the 10–30 range to make an 8-category score  that could be directly compared between models. ICC estimates were similar for the 8-category model, suggesting slight bias in the estimate of the ICC. When the data were reduced to a 4-category score, the linear random effects model underestimated the ICC by a difference of 13% . The ICC for the 3-category score was 54% (moderate agreement) compared with 43% (poor agreement) for the 4-category score.We compared results between the 3-category score used historicallyhttp://links.lww.com/TXD/A334). Increasing the time window by 3 mo reduced the reliability of the KPS by 5%.To test for selection bias in our cohort, we expanded the 3-mo criteria in the cohort definition to include patients with >1 visit in 6 mo and in 1 y  for the 3-category score represents a significant gain in improvement over the ICC estimate of 43% (poor agreement) for the 4-category score and the estimate of 30% (poor agreement) for the 10-level score; however, it is not surprising to see an improvement with fewer categories, as fewer choices should improve intrarater agreement. By either scale , these deviations are purported to represent sizeable differences in functional (“performance”) status. The lack of reliability in KPS reporting raises concerns when applying the KPS as a proxy for frailty and a metric to be considered when evaluating candidacy for kidney transplantation.We sought to estimate the reliability and variability of the KPS in a national sample using the SRTR data. We took advantage of the process of multiple transplant program listing to obtain independent assessments of the KPS. We found substantial variability in KPS reporting and a poor interobserver reliability of 30% in the 10-point scale. More than 1 in 4 (27%) patients experienced more than a 10-point shift in KPS using the 100-point (10-category) scale, while 38% experienced at least a 1-category shift when using a condensed 4-category scale. The ICC for the 4-category score could also be considered an estimate of the consistency between measures because it allows for more error than the 10-point scale.To abrogate the risk of capturing a real decline in performance status, we needed to limit our analysis to patients who were evaluated at 2 or more programs within a 3-mo period, ergo, the relatively small fraction (2%) of the overall kidney transplant population in our analytic cohort. This limits the generalizability, as multiple listings may not be feasible for most patients. A crude comparison of baseline KPS showed patients with multiple listings at different centers tended to be healthier (mean score = 82 versus 79). They may also be better resourced because traveling to multiple centers can be costly. It is possible that patients with fewer KPS also have lower scores, and these would be more reliable. However, when we relaxed the 3-mo criterion included in the cohort definition, we found that the reliability in the scores declined. This trend indicates that our estimates may overestimate the reliability of the KPS in patients with end-stage kidney disease. Retrospective studies are limited in their capacity to control for unknown factors, and it is possible that these factors might affect the precision and hence the reliability of the KPS. Our results are also limited to transplant candidates and may not reflect the reliability of the KPS in patients not on the waitlist.21 Although the majority of patients are in the normal performance category, we observed a significant shift from 7% of patients in 2006–2010 in the lower 2 KPS categories to 10% of patients in 2016–2020. Such a shift in the demographics of candidates toward an increasingly frail population supports an increasing need for a reliable estimate of frailty when considering allocation of a scarce resource and the immense benefit that can accrue to successful kidney transplant recipients.In the last decade, there has been an increased proportion of transplant candidates with lower KPS compared with previous years, supporting a growing trend of kidney transplantation in more frail candidates. Moreover, changes in the kidney allocation system in December 2014, which increased the availability of transplantable kidneys to many candidates who previously had more limited access may have contributed to this trend.22 performed a meta-analysis comparing the reliability of reporting of the KPS, the palliative performance scale and the Eastern Cooperative Oncology Group performance status. They found that the KPS demonstrated the best reliability compared with other measures. Yet, with the exception of only 2 studies, the analysis was performed among patients with advanced or terminal cancer and only 1 of these 2 studies investigated KPS in patients with end-stage kidney disease. The advanced cancer population differs enormously from the kidney transplant candidate population. Patients with advanced cancer often experience substantial degrees of debility and disability. Although patients with advanced chronic kidney disease, including those receiving dialysis, commonly experience functional impairment, generally healthier patients are typically referred for kidney transplant evaluation. In transplant clinics, the KPS is often performed as quick assessments during clinical visits by varying providers  with no standardized approach among transplant centers. A recent letter to the editor also suggested that the subjective nature of the KPS makes it prone to bias in patients undergoing liver transplantation.23 Bias in the evaluation of kidney transplant candidates is no exception.Most studies evaluated the reliability of the KPS in patients with cancer. Chow et al24 Hutchinson’s is the only study of which we are aware of that previously considered the interobserver reproducibility of the KPS in patients receiving hemodialysis. Their experiment was performed at 1 center, whereas our study evaluated the use of KPS in practice on a national scale. They found that the proportion of patients with scores in agreement was 71% for the 3-category and 29% for the 10-point score. Comparatively, we found 60% agreement for the 4-category scale and 30% agreement for the 10-point score. Some functional decline over time is expected in candidates on the waitlist, and this may vary depending on the patients’ baseline functional capacity and its trajectory.19 Although it is possible that functional decline may have influenced our analysis, the variability we observed between KPS scores is not likely attributable to major change in a patient’s physical function within a span of just 3 mo. Aside from the relatively short-time frame, one would not expect that a patient who might have experienced an acute illness or a sharp decline in health or functional capacity would undergo evaluation and testing and an alternative, often distant transplant center. We did not observe a significant linear trend in the scores over the 3-mo span; however, time was retained in the model to adjust for potential differences in elapsed time between measurements. Given the consistency between our estimates and those reported in previous studies, we believe our values to be an accurate representation of the variation and reliability (or lack thereof) in KPS score reporting nationally.Our reliability estimate of 43% in the 4-category score is comparable to the Kappa estimate of 46% in 1979 by Hutchinson et al for a 3-category score for patients receiving hemodialysis.25 One study in patients with cancer was able to achieve 97%–100% agreement by providing staff with training and over an hour to several days of interaction to assess the patient.18 Still, there is evidence that KPS performance is worse in patients with CKD compared with patients with other comorbidities.24 The studies that report higher reliability also typically involve patients with scores in the bottom half (10%–50%) of the scale.23 Some studies show self-assessment to be a convenient measure and reasonable measure among patients who are on the waitlist for kidney transplantation. Examples include the self-reported 10-item Short Form-36 physical functioning questionnaire (SF-36 PF) or a patient self-reported KPS score.27 However, self-reported measures are also subjective and can be influenced by factors unrelated to physical fitness.Some reasons for the poor agreement in the KPS may be due to the subjectivity of the measure, lack of sufficient training among staff, and lack of standardization in implementation across transplant centers.28 Xu et al developed the Liver Frailty Index specific to patients waitlisted for a liver transplant, which includes grip strength, chair stands, and balance. They found that the KPS was not predictive of mortality compared with their more objective measure.29 The Fried physical frailty phonotype is based on a combination of subjective (exhaustion) and objective measures . A score based on the Fried physical frailty phenotype was associated with higher waitlist mortality and lower kidney transplant rate.30 Given the importance of frailty in predicting transplant outcomes, transplant centers should adopt less subjective and more reliable measures that can be used for candidate assessment, counseling, and therapeutic measures.KPS is the only proxy metric of frailty required by the OPTN; however, several centers have considered more objective methods such as the 6-min walk test, the sit-to-stand test and grip strength.We conclude that the KPS is not a reliable proxy for frailty in kidney transplant candidates because of its high interobserver variability. More reliable and valid measures are needed to optimally risk stratify the transplant candidate population and improve efficiency in kidney allocation. We propose a move toward use of more objective measures when assessing frailty in kidney transplant candidates.


Self-reporting digital apps provide a way of remotely monitoring and managing patients with chronic conditions in the community. Leveraging the data collected by these apps in prognostic models could provide increased personalization of care and reduce the burden of care for people who live with chronic conditions. This study evaluated the predictive ability of prognostic models for the prediction of acute exacerbation events in people with chronic obstructive pulmonary disease by using data self-reported to a digital health app.The aim of this study was to evaluate if data self-reported to a digital health app can be used to predict acute exacerbation events in the near future.This is a retrospective study evaluating the use of symptom and chronic obstructive pulmonary disease assessment test data self-reported to a digital health app (myCOPD) in predicting acute exacerbation events. We include data from 2374 patients who made 68,139 self-reports. We evaluated the degree to which the different variables self-reported to the app are predictive of exacerbation events and developed both heuristic and machine learning models to predict whether the patient will report an exacerbation event within 3 days of self-reporting to the app. The model’s predictive ability was evaluated based on self-reports from an independent set of patients.Users self-reported symptoms, and standard chronic obstructive pulmonary disease assessment tests displayed correlation with future exacerbation events. Both a baseline model  and a machine learning model  showed moderate ability in predicting exacerbation events, occurring within 3 days of a given self-report. Although the baseline model obtained a fixed sensitivity and specificity of 0.551 (95% CI 0.508-0.596) and 0.759 (95% CI 0.752-0.767) respectively, the sensitivity and specificity of the machine learning model can be tuned by dichotomizing the continuous predictions it provides with different thresholds.Data self-reported to health care apps designed to remotely monitor patients with chronic obstructive pulmonary disease can be used to predict acute exacerbation events with moderate performance. This could increase personalization of care by allowing preemptive action to be taken to mitigate the risk of future exacerbation events. Chronic obstructive pulmonary disease (COPD) is a collection of progressive lung diseases, characterized by breathing difficulties and an irreversible reduction of lung function. It is one of the most prevalent chronic conditions in the world . All users of the myCOPD app are clinically diagnosed with COPD, with app usage limited to patients “prescribed” the app by clinicians as part of agreed care plans. A single report features a self-assessed symptom score , which iThis work received ethics approval from the University of Southampton’s Faculty of Engineering and Physical Science Research Ethics Committee (ERGO/FEPS/52137) and was reviewed by the University of Southampton Data Protection Impact Assessment panel (DPIA 0045), with the decision to support the research.We use a symptom-based definition for exacerbation events where an event is marked to have occurred if a patient self-reports a score of 3 or 4 corresponding to a moderate or severe deterioration, respectively, from a patient’s normal symptoms. A score of 3 indicates that a patient is more breathless than normal, coughing up sputum or with change in sputum color, and has needed to self-medicate using steroids or antibiotics. A score of 4 indicates that a patient’s breathing is much worse than normal despite treatment, has chest pain or a high fever, and has needed to seek emergency care or was admitted to the hospital  13]..13].In total, 5170 users were included in the extract who reported a total number of 94,882 reports in the study period. User registration was incremental  throughout the period of the study, and self-reports were not necessarily submitted every day. To create our study cohort, we followed a selection process outlined in From our cohort of 2374 users, 1672 users were between the ages of 60 years and 79 years inclusive . Only 65For each daily self-report, we created a binary variable that indicated whether a report is followed by an exacerbation event in the following 3 days. A 3-day window was chosen empirically based on clinical guidance to be close enough to the future exacerbation event for any signal to be present in the data but sufficiently far from the event such that a range of preemptive actions could be available to patients. For the training of the prognostic models, we selected only reports in which the patient did not report an exacerbation event on the same day  within 3 days, and users reporting a symptom score of 2 are predicted to be at heightened risk (7.2% risk) of exacerbation within 3 days. Percentages in brackets correspond to the mean 3-day exacerbation rate for all reports in the training set with symptom scores of 1 or 2, respectively. The heuristic model is equivalent to a decision tree with a depth of 1. Supervised machine learning models make use of patient demographics, lifestyle information, self-reported information, and aggregate features that summarize a patient’s (recent) self-reporting history. A full schema of variables used by our models is presented in Table S2 of P<.001) than the mean CAT score (19.5) when a user reports a symptom score of 2. Such a correlation is to be expected; research has shown increased CAT scores correlate with an increased exacerbation frequency [requency , which iIn P<.001) more likely to experience an exacerbation event in the next 3 days compared to those reporting normal symptoms, with a relative risk of 4.16 (95% CI 3.8-4.5). On the holdout test set, the baseline model obtained an area under the receiver operating characteristic (AUROC) of 0.655 (95% CI 0.676-0.689). The logistic regression model obtained an AUROC of 0.697 (95% CI 0.689-0.711) and the random forest model 0.727 (95% CI 0.720-0.735) on the holdout test (P<.001) performance of the random forest model suggests either interactions between variables are important in discriminating between reports associated with exacerbation within 3 days or nonlinear relations are present.out test . The sigIn In The etiology of COPD exacerbations is now well understood, with bacterial and viral infections, exposure to extreme weather, and air pollution being the key drivers set against the background of poor disease control. Effective treatments are available for COPD, with early recognition of symptoms of deterioration and prompt intervention having been shown to be associated with better outcomes . CurrentEarly prediction of COPD exacerbation events has the potential to change clinical practice and transform management of COPD. With the preemptive warning of a future exacerbation, the new app codeveloped with patients with COPD will alert patients of the risk and provide information on appropriate therapy options. More effective treatment of exacerbations offers the possibility of faster recovery, less relapses, and overall, therefore, better health-related quality of life, improved disease control, and potentially fewer exacerbations. Our study has shown that symptom information self-reported to the myCOPD app displayed correlation to the start of the future exacerbation events E, and weVarious approaches to continuous remote monitoring of patients with COPD in communities have emerged in recent years that use sensor technologies to measure physiological parameters  and physical activity . The conMachine learning models are well positioned to address these requirements because they can dynamically learn complex nonlinear relations between variables, which are inaccessible to handcrafted models, and despite the complexity of the models, they can be easily integrated into digital health apps such as myCOPD. This greatly facilitates the potential uptake of the model since they can be directly implemented in an active digital ecosystem for patient care that fosters data acquisition and can position predictions within new models of patient-to-clinician interaction with predictions updated every time a user provides new data. For machine learning models, it is important to match their configuration to the escalation policy. If models are used as a binary alert system, this is achieved by analyzing the  sensitivity-specificity trade-off, considered in The predictive performance of the machine learning models for COPD exacerbation can be improved by adding COPD-related variables. Health and lifestyle activity factors (including comorbidity and socioeconomic status) are believed to impact COPD exacerbation frequency . These cReporting compliance is a concern for safety, effectiveness, and acceptance of models. There is a need to ensure the burden of technology is reduced to a minimum, that incentives and benefits of reporting are aligned, and where appropriate, automatic observations and measurements are used to capture data . Our self-reports were collected in a prospective fashion by using momentary assessments of a patient’s COPD symptoms. Reporting compliance and individual variation in reporting behaviors could still be detrimental to our results, with exacerbation frequencies or severity being misreported . Our decTo conclude, our results suggest that data self-reported to a digital health app, designed for the management of people with COPD, can be used to identify users at risk of exacerbation within 3 days with moderate discriminative ability . Further research utilizing additional linked data  are expected to increase the accuracy of these models.