The serial interval of an infectious disease, commonly interpreted as the timebetween the onset of symptoms in sequentially infected individuals within achain of transmission, is a key epidemiological quantity involved in estimatingthe reproduction number. The serial interval is closely related to other keyquantities, including the incubation period, the generation interval , and time delays between infection and theobservations associated with monitoring an outbreak such as confirmed cases,hospital admissions, and deaths. Estimates of these quantities are often basedon small data sets from early contact tracing and are subject to considerableuncertainty, which is especially true for early coronavirus disease 2019 data.In this paper, we estimate these key quantities in the context of coronavirusdisease 2019 for the UK, including a meta-analysis of early estimates of theserial interval. We estimate distributions for the serial interval with a meanof 5.9  and SD 4.1  days , the generation interval with a mean of 4.9  andSD 2.0  days (fitted gamma distribution), and the incubationperiod with a mean 5.2  and SD 5.5  days. We quantify the impact of the uncertaintysurrounding the serial interval, generation interval, incubation period, andtime delays, on the subsequent estimation of the reproduction number, whenpragmatic and more formal approaches are taken. These estimates place empiricalbounds on the estimates of most relevant model parameters and are expected tocontribute to modeling coronavirus disease 2019 transmission. The calculation of tR can be madepragmatically using simplifying assumptions, or in a more formal manner, for whichother key parameters are also required, in particular the delay between infectionand diagnosis. Given that these parameters are all associated with uncertainty weinvestigate how this uncertainty may affect our estimates of the reproductionnumber, and we qualitatively compare the pragmatic approach compared to the moreformal approach.The purpose of this paper is to determine the best estimates for the parameters weneed to calculate the effective reproduction number  has caused a global pandemic of disease. Thespeed with which the virus spreads is dependent on biological determinants thatenable viral replication within individuals and onward transmission to others. Theminimal set of parameters required for understanding the dynamics of any novelinfectious disease pathogen include the potential for transmission, the duration ofinfectiousness (often captured in models as a recovery rate), and the generationalinterval: the time between two subsequent cases in a chain of infection.Estimating the generation interval is particularly challenging due to thefundamentally hidden nature of transmission events. In  the generation interval is defined as the time between the infection of aninfector and infectee, and in practice is not easy to observe, as infection goesthrough some latent period, during which it is undetectable, and a pre-symptomatic phase during which an individual may be infectious, butpossibly detectable through screening, before the disease manifests with clinicalsymptoms. The latent period and pre-symptomatic phase are together usually referredto as the incubation period (incubationT). From theonset of symptoms, the diagnosis will be confirmed by some canonical test after sometime delay (onset→testT), later the patient may requireadmission to hospital (onset→admissionT), or may die(onset→deathT). These subsequent events clearlymay not happen in that order, and even diagnosis may occur after death. The timebetween these key events  for two sequentially infected people in a chain oftransmission are described as serial intervals, although in general usage, and in the rest of this paper, the term “serialinterval” is taken to mean the interval between onset of symptoms(onsetSI). In caseSI), “admission interval”(admissionSI), and “death interval”(deathSI) to differentiate the other intervals.The generation interval is by definition a non-negative quantity, but all the othermeasures may be negative if the variation of the delay from the event of infectionfrom person to person exceeds the period between infections. This is more likely fordeath interval than for serial interval, and for diseases with long pre-symptomaticperiods, for example, HIV. The generation interval is defined for all transmission pairs, but the otherintervals may or may not be, if for example one of the infector, or infectee isasymptomatic, or does not go to hospital, or does not die, in which case the relatedintervals are not defined for this pair.As described by Svensson,tR) is a key measureof the state of the epidemic. In its simplest form, for a given population, theeffective reproduction number is the number of secondary infections that areexpected to arise from one primary infection, at any given instant, and depends onthe biological properties of the virus, the immune status, and the behavior of thatpopulation. Estimation of the effective reproduction number can either be doneforward in time, where we estimate the number of infections that arose from casesdetected on a given day (case or cohort reproduction number), or backward in time where we estimate the number of cases that caused theinfections observed on a given day (instantaneous reproduction number).The effective reproduction number , hospital admissions, and deaths. As depicted in Direct measurement of the serial interval distribution is further complicated by thefact that symptom onset is often not observed due to the scale of the outbreak andthat infection may be asymptomatic.tR arepredicated on infections a “formal” approach to estimation is to infer theunobserved incidence of infection from the observations we have usingbackpropagation or de-convolution,9 using the generation intervalas the infectivity profile. However, this requires knowledge of the temporalrelationship between unobserved infections and observed cases, admissions ordeaths.Since renewal equation-based estimates of  is to simply calculate tR using a serial intervalas the infectivity profile, and un-adjusted case numbers with a simple correctionfor the time delay between infection and cases by shifting ourtR estimates backward in time. As the renewalequation methods assume that the time between two infections cannot be negative, theserial interval distribution must be truncated at zero in this pragmaticapproach.A pragmatic alternativetR , infection and case identification (test), infection and admission, andbetween infection and death. The formal method also needs an understanding of theshape of the distribution of these delays. In the rest of this paper, we estimatethese key quantities and investigate the impact that uncertainty in these quantitieshas on an estimation of Rt . and with existing meta-analyses. From these papers, serial interval mean and standard deviation estimateswere extracted, along with information about assumed statistical distributions,and the sample size of the study. A random-effect meta-analysis was conducted on the subset of papers that reported confidence intervals. Theassumption of normal distribution underpinning the meta-analysis1618 is reasonable for themean of the serial interval, given the central limit theorem, however, we cannotextend this to the standard deviation of the serial interval distribution andhence cannot assess the overall shape of the serial interval distribution. Toaddress this and combine parameterized distribution estimates from multiplestudies into a single distribution we undertook a re-sampling exercise, asdescribed below, the goal of which was to reconstruct a data set of serialintervals that accurately reflect the distributions reported in each of thestudies above, so that they could be combined and analyzed together.Firstly, we conducted a literature review for studies that describe serialinterval estimates using PubMed and the search terms “(SARS-CoV-2 or COVID-19)and ‘Serial interval’” and reviewed the abstracts of relevant original researchpapers. These were compared to papers reported on the MIDAS Online Portal forCOVID-19 Modeling Research,20 This gives us a set of samples that are consistent withthe shape and uncertainty of the distributions reported by the source studiesand which represent their sample size in a comparable way. Other studiesreported empirical distributions and for these, we obtained original serialinterval data were available and made 100 random bootstrap sub-samples withreplacement, to a relative size determined by the original sample size of thestudy.Most studies reported a central estimate of a probability distribution for theserial interval, specified in terms of the mean and standard deviation of agiven serial interval distribution. They also reported uncertainty on these meanand standard deviations as confidence or credible intervals. For these studies,we randomly selected one hundred probability distributions consistent with thecentral estimates and confidence intervals reported in each paper . From this family of probabilitydistributions, we drew random samples based on the original samplesize.The 100 empirical and 100 probability distribution-based serial interval sampleswere combined into 100 groups, with each group containing serial intervalsrepresentative of each source article with numbers proportional to the size ofthe original study. This allows us to calculate 100 empirical probabilitydistribution estimates from which we can derive summary statistics withconfidence intervals .x, with interval-censored ranges spanning from thevalue R,22 Thisadditionally gives us 100 parametric probability distribution estimates for eachdistribution from which we can derive confidence intervals for the parametersand 100 empirical probability distributions estimates of the combination of allthe source studies.As much of the data is captured at the resolution of a single day, serialintervals appear as an integer number of days in the data. The discrete natureof the data was estimated using continuous distributions by replacing integervalues, 24 which provides a limitednumber of linked cases of proven transmission, mostly within households, andinterval-censored symptom onset dates. To this data, we fitted normal, gamma,and Weibull distributions using the same methodology as above.As a comparison, we also used data collected under the “First Few Hundred”(FF100) case protocols by Public Health EnglandIn both cases, when fitting gamma and Weibull distributions we truncated theinterval-censored data at zero, to prevent negative values, and for the gammadistributions we required that the shape parameter had a lower bound of 1, whichenforces that the distribution density is zero at time zero. This however makesgoodness of fit statistics not comparable between the normal distribution fitand the two other distributions. and Sanche et al. for China in the early phase of the epidemic. The FF100 data containsinterval-censored exposure data coupled to symptom onset; using this we deriveda UK-specific estimate to assess if it was consistent. Furthermore, the OpenCOVID-19 Data Working Group28 provides a largeinternational data set that includes some travel history and symptom onsetdata.The incubation period has been previously estimated by Lauer et al.26 where travel principally originated from Wuhan, thetravel cases in the Open COVID-19 data set include travelers between a far widerset of destinations, and in a later stage of the epidemic, which we believereduces selection bias. As we are performing this analysis retrospectively we also benefit fromfewer issues due to the right censoring of the data.Compared to data on which earlier estimates of the incubation period weremadeFor both data sets, we use random samples with replacement and fit 100 of each ofgamma, Weibull and logn-ormal probability distributions to the interval betweenputative exposure and symptom onset, accounting for censoring were present, toestimate the incubation period distribution and uncertainty associated with itsparameters. We remove from the data records where the time from exposure tosymptom onset is negative or when it is longer than 21 days .For subsequent phases of the analysis, we retain the 100 parameterizeddistributions of the distribution with the best overall fit to the data from theOpen COVID-19 Data Working Group data set. From these, we can generate aparameterized bootstrap sample representative of the Open COVID-19 Data WorkingGroup data set but without censoring. and the standard deviation must be smaller than the mean . The minimization function we employed was the absolute differencein inter-quartile ranges of simulated and observed distributions. This processwas repeated for 100 different simulated samples that were compared to the 100different empirical re-sampled serial interval estimates from the previous stageof our analysis to get confidence intervals on our estimates of the generationinterval distribution. This approach does not use the parameterized serialinterval estimate and only the samples direct from the combination of data andliterature. For the incubation period, we use parametric re-sampling of multiplefits to the original interval-censored incubation period data to generatebootstrap samples. This approach minimizes any assumptions we make about thedistribution of these other quantities when estimating the generationinterval.The generation interval is the fundamental variable for modeling transmission.Under the assumption that the generation interval follows a gamma distributionwe can infer its parameters using the bootstrap samples from the re-sampledserial interval data set, and the bootstrap samples from the incubation perioddata set from earlier stages, and the constraint that the generation interval isa non-negative quantity. We combined random samples from a parameterized gammadistribution for the generation interval with two of the bootstrap samples fromthe incubation period to satisfy the following relationship, thus simulating theserial interval:tR. To investigate this we used the forwardequation approach implemented in the R library EpiEstim.57 We estimate values oftR for 4 time points in the first wave ofthe COVID-19 pandemic in England representative of the ascending phase, thepeak, the early descending phase, and the late descending phase. We used dataretrieved from the Public Health England API31 representing cases withpositive test results, hospital admissions, and deaths within 28 days of apositive test, in the first wave of the outbreak in England. For this analysiswe assume the infectivity profile can be represented using a parameterized gammadistribution, and estimate tR for a wide range ofcombinations of mean and standard deviation, using a fixed calculation window of7 days, at each of our 4 time points. The resulting relationship betweentR, mean infectivity profile, and standarddeviation of infectivity profile were compared visually to qualitatively examinehow the serial interval estimates influence the estimates oftR.With various estimates of serial interval and generation interval, we wished tounderstand the qualitative impact this variation might have on our estimates ofonset→testT,onset→admissionT, andonset→deathT) using the CHESS data set. The CHS data set is hospital-based, and was initially limited tointensive care admissions, but a subset of hospitals have reported alladmissions, and this is what we focused on . Withinthe CHESS data set there are a set of patients who have symptom onset datesrecorded, dates that a specimen was taken that subsequently was tested positive,hospital admission date, and date of death, if the patient died. We restrictedcases to those in which a positive test was found no more than 14 days beforeand 28 days after symptom onset on the grounds of biological implausibility.Because we conducted this analysis at the end of the first wave of COVID-19 inthe UK when patient numbers in hospitals had fallen to a low level there wasminimal right censoring present in the data and this was not accounted for.Estimation of the time interval between the onset of symptoms and theobservations of the positive test result, hospital admission, and death wasperformed  as a proxy for infection events, and used a truncatedempirical serial interval distribution derived from our re-sampling procedure asa proxy for infectivity profile. In line with the pragmatic approach, simple tothe dates of these tR estimates was made to alignthe estimate to date of infection rather than the date of observation.Firstly, following the “pragmatic” course outlined in the introduction, we based To do this, we applied a non-parametric back-projection algorithm fromthe surveillance R package, based on work by Becker et al. and Yip et al., to infer three putative infection time series from observed cases,admissions or deaths. The inferred time series were then used to estimatetR through EpiEstim using the parametricgamma-distributed estimate of the generation interval from above. In applyingthe de-convolution we discovered it requires a full-time series beginning withzero cases for sensible results and this required we impute the early part ofthe hospital admission time series, which we did by assuming an early constantexponential growth phase.Secondly, the more formal approach was employed: using the time delaydistributions from the previous stage, we used de-convolution to infer a set oftime series of infections from the same observational data and used our estimateof the generation interval as a proxy for the infectivity profile. This secondapproach has been recommended by Gostic et al.tR. Theresulting tR time series were comparedqualitatively.In both cases, we used the renewal equation method with a 14-day sliding windowto estimate a continuous time series of 3548 The mean and standarddeviation of parameterized distributions were extracted and are presented in noted that the serial interval was not infrequently negative.Our PubMed search retrieved 62 search hits of which 14 were original researcharticles containing estimates of serial intervals.The random-effects meta-analysis on the subset of studies that reported modeleddistributions, resulted in an overall estimate of the mean of the serialinterval of 4.83 (95% CI 3.93–5.70) . This estimate is limited in its value by the fact that it only covers thesubset of the studies in the table above, and does not represent thedistributional nature of the serial interval.In  the FF100 data shows evidence of negative serial intervals. The mean ofthe serial interval from FF100 data was 3.50 (gamma) or 3.52 (Weibull) when datawas truncated to exclude negative serial intervals and 2.81 (norm) with notruncation. This is on the lower end of the values reported in theliterature.In tR however the use of negative serialintervals is not supported as a proxy for the infectivity profile. In thepragmatic approach to estimating tR, we truncate theempirical distribution at zero to support this. Once truncated the resultingestimate, therefore, has a mode of 3.86 days, shorter than the normaldistribution parameterization and is a truncated empirical distribution, with amean plus 95% confidence interval of 5.88 days , and a standarddeviation of 4.12 days , which is more in line with the gammadistribution parameterization.As noted in the methods, the re-sampling process allows us to estimate the serialinterval as an empirical distribution. Within EpiEstim, our chosen framework forestimating 26 The best fit to the OpenCOVID-19 data is obtained with a log-normal distribution as shown in  The mean and standard deviation of these distributions are notindependent, and this is explored further in Supplemental figure 3.The generation interval is then inferred from the incubation period and empiricalserial interval distribution prior to truncation. Our best estimate for this isa gamma distribution, with a mean plus 95% confidence interval of 4.87 days, and a standard deviation of 1.98 days , as shown intR, of using these estimates as a proxy forthe infectivity profile. This uses data at 4 time points on an epidemic curvefrom the first wave of the COVID-19 outbreak in England, as shown in With our three estimates of the serial interval and one generation interval andobserved COVID-19 case counts, we investigate the impact on the estimates oftR underthe range of different assumptions about the mean and standard deviation of theinfectivity profile, modeled as a gamma distribution. In the top left, bottomleft and bottom right panels the effect of increasing the mean of theinfectivity profile is to push the resulting estimate oftR away from the critical value of 1 at whichthe epidemic is growing. In the top right panel, at the peak, the mean of theinfectivity profile has a less clear-cut effect. The impact of changes tostandard deviation is likewise varied. In the top left, bottom left and bottomright panels during ascending and descending phases there is relatively littleimpact of changing the standard deviation of the infectivity profile on theestimates of tR, and any small changes that do occurdepend on the shape of the preceding epidemic curve. At the peak, however, inthe top right panel, the wider the standard deviation the more historicalinformation influences the estimation of tR and thisacts to delay the estimated transition from positive to negative growth. Theoverall result of this is that estimates of the infectivity profile with a highstandard deviation will predict tR crossing 1 laterthan estimates based on an infectivity profile with a low standard deviation,but the point of crossing 1 is relatively insensitive to the value of the meanof the infectivity profile.At each of these 4 time points, tR we can see from the colored crosses intR but at other times the impact is much smaller.When we consider using the various estimates of the serial interval or generationinterval, as a proxy for the infectivity profile, on the resulting estimates ofonset→caseT), admission(onset→admissionT), or death(onset→deathT). Symptom onset to test (caseidentification) can be a negative quantity if a swab is taken during diseasescreening and the patient is pre-symptomatic. In this data the time point thatdefines the time of test is the date when the specimen is taken, which willsubsequently be tested positive for SARS-CoV-2, so does not include sampleprocessing delays. However, in this hospital-based data source of admittedpatients, the onset data were collected retrospectively. We also note peaks at 1day, 1 week, 2 weeks, and so on which suggests approximation on data entry, andthere may well be biases in the data collection.There are 9902 patients in the subset of the CHESS database we examined of which82.3% had a symptom onset date. In It is more obvious from the clinical course of COVID-19 that admission shouldoccur after disease onset. In the data, a large number of cases are reported tohave symptom onset on the day of admission. This is potentially a reportingartifact as in the absence of certain knowledge about the onset, it is possiblethat the day of admission may have been captured instead, and we again see thepeaks at 1 week, 2 weeks, and so on, suggesting approximations in dataentry.The time between the onset of symptoms and death can also be assumed as apositive quantity given this is based on an in-hospital cohort. Thisdistribution shows a large tail, and some patients in that tail were noted to beadmitted many months before the appearance of COVID-19. These patients likelyrepresent hospital-acquired cases in chronically unwell patients. The extremeoutlying values  were removed as they prevented sensibleestimation of the rest of the distribution.By combining the incubation period distribution in panel B in  when using EpiEstim. The results of the back-propagation are shown intR are shown in panel B based on de-convolvedtime series plus generation interval, versus raw observation counts, re-sampledserial interval estimates, with time adjustment on the resultingtR estimates to aligntR estimate date to putative date of infection.We have not calculated confidence intervals for the estimates oftR. The estimates oftR differ from their mean  by between 3.33% and 6.51% .With the estimates of a delay from infection to observation, we are able to use anon-parametric back-propagation as described in the methods to estimate a timeseries of infections as recommended by Gostic et al.Our estimate of the serial interval from the FF100 UK data was found to be short,compared to international estimates. The FF100 data was collected in the early stageof the epidemic and is based mainly on household contacts of international travelerswhich may impart bias. As the participants in the study were put into self-isolationupon discovery observations the contact period tends to be shortened, leading toshorter serial interval estimates, and because of this, we do not believe theestimate from the FF100 study to be specific to the UK but rather that the data setis not broadly representative of the UK population.tR. instead of which we relyon the set of empirical distributions resulting from our sampling analysis. Even so,the negative values of the serial interval within these are theoreticallyproblematic for their use as a proxy for the infectivity profile of SARS-CoV-2 intransmission modeling, which requires the infectee is infected after the infector.As a way around this, we truncate the re-sampled serial interval at zero and usethis as the infectivity profile, but as seen here this truncation increases the meanof the resulting serial interval distribution from 4.87 to 5.88 days.In contrast, our literature search allowed us to estimate the serial interval from amuch larger pooled sample drawn from multiple studies. Random effects andre-sampling meta-analyses produced comparable results (random effects: mean 4.83(95% CI 3.93–5.70) days and re-sampling: mean 5.9  and SD 4.1  days), but our re-sampling study more clearly showed the potential forthe serial interval to be negative, due to the relatively long and variableincubation period of SARS-CoV-2. This is seen in the unsatisfactory fits ofparameterized probability distributions in  Derivation of the generation interval from the serial interval is possibleusing knowledge of the incubation period. We again looked at the FF100 data forestimates of the incubation period and again found that the UK data suggests a valuethat is shorter than international estimates, for the same reasons as mentionedabove.5052 Wecross-referenced this with a second estimate derived from the Open COVID-19 DataWorking Group data set,28 which is based on a large international data set of people whotested positive for SAR-CoV-2 after traveling from areas with outbreaks. Theresulting estimates of the mean incubation period of 5.5 days  are much closer to previous estimates, and we expect to be lessinfluenced by right censoring. Again, we regard the short incubation periodcalculated from the FF100 data set to be a feature of the data set rather than aUK-specific finding. In fact, the estimate based on the Open COVID-19 Data may initself be an under-estimate as the majority of travel histories only include thereturn date of the visit in question and not the start date.An alternative to this is to use the generation interval as a proxy for theinfectivity profile, but there are limited estimates of this quantity available inthe literature. Although the confidence intervals for the mean and standard deviation of thegeneration interval are comparable, the variation in the shape and rate parametersof the underlying gamma distribution are quite large. Care should be taken whengenerating bootstrap samples from the generation interval when it is specified as anuncertain gamma distribution parameterized with mean and standard deviation, as itis possible that some combinations of parameters produce unrealistic distributions,particularly when the standard deviation is small and the mean is large, whichcould, in theory, result in posterior estimates for tRbeing largely determined by the reciprocal of just a small number ofobservations.With incubation period and serial interval estimates, we derived an estimate forgeneration interval, assuming a gamma distribution. This was comparable to previousestimates in that although it is based on a different serial interval, and hence hasa different mean, the standard deviation of our estimate is in accordance with thatestimated by Ganyani et al.tR using incident casesin England and the forward equation approach. We found the bias between the smallestand largest of our estimates to be as high as 20–25% of the central estimate oftR when tR was high,but somewhat smaller when tR values were 1 or lower.Distributions with lower values of the mean tended to result in estimates oftR that were closer to one. This suggests thatbiases introduced by the use of different serial interval distributions should notinfluence the answer to the key question, “is tR greateror less than 1?” which defines whether the epidemic is expanding or contracting insize. It is also to be expected that the nature of change oftR over time is not affected by this bias, so anincreasing value of tR will be increasing regardless ofthe infectivity profile that is used to estimate it.Using estimates of the serial interval distribution and generation intervaldistribution as a proxy for infectivity profile, we investigated the resultingvariation in the estimation of tR is based on knowledge of the incidence ofinfections. This is not a quantity that is readily observed in the SARS-CoV-2epidemic, and pragmatically use of observations including symptom onset, caseidentification, admission, and death are expected to be used as proxy measures forinfection.7However, as pointed out elsewhere the variable time between infection and such observations causes the signalto be both delayed and blurred. By combining our estimates of the incubation periodwith data from hospital admissions in the CHESS data set we were able to makeestimates of the distribution of the delay from infection to observation for the UK.There are several caveats to this part of our analysis that must be kept in mind.The CHESS data set relies on a retrospective report of onset of symptoms, and thisfield is not recorded for all patients. We select only those patients who havereported an onset date and it may be the case that these patients represent asubgroup of patients whose symptom onset is significantly different from the averagepatient. The data collection around these dates as noted above to show patternssuggestive of rounding or approximation and this could also introduce some bias. Thedelay distributions are also unlikely to remain fixed during the outbreak, as wewould hope to see the time from infection to case identification shorten during theepidemic, and the time from infection to death lengthen as treatment improves, andas the cohort of susceptible individuals changes. Our confidence in thesedistributions is therefore somewhat low, although we note acceptable agreementbetween our estimates produced using a mixture of international and UK data, andpreviously published estimates from different countries, most notablyChina5255 which cite an onset to anadmission of 2.7–5.9 days52 and onset to death delay of 16.1–17.8 days.52 Given ourestimate of the incubation period is log-normally distributed, it is unsurprisingthat the combination of incubation period and delay from onset to observation isalso best described by log-normal distributions , given the current state of knowledge.tR estimation, when using the forward equationmethod. Whilst these introduce significant variation in the worst-case scenario, wefind that even large differences in infectivity profile can have only small impactson estimates of tR when tRis close to 1. Larger values for the mean of the infectivity profile appear toresult in tR estimates that are further away from 1.This is a relatively reassuring finding in that the answer to the key question “isthe epidemic under control?” is insensitive to the mean of the infectivityprofile.As there is a wide range of candidate values for these quantities, we have assessedthe bias that the variation in choice of parameters introduces totR byback-propagation inference of infection rate and estimates of the generationinterval, produces tR estimates that are more variablewhen compared to the more pragmatic direct use of case counts as a proxy forinfection. Both methods agree on when the epidemic crossed thetR threshold of 1. However there is considerableuncertainty in the quantities needed to perform the back-propagation, and we are notable to ensure that all this uncertainty could be faithfully quantified in ourresulting tR estimates. We did not set out to assesswhether one method is better than another, and this would be a natural extension tothis work, however, we note that new methods for combining back-propagation with anestimation of tR are under active development59 and may verywell address such further questions.Using more formal methods for estimating Click here for additional data file.Supplemental material, sj-pdf-1-smm-10.1177_09622802211065159 for Meta-analysisof the severe acute respiratory syndrome coronavirus 2 serial intervals and theimpact of parameter uncertainty on the coronavirus disease 2019 reproductionnumber by Robert Challen, Ellen Brooks-Pollock, Krasimira Tsaneva-Atanasova andLeon Danon in Statistical Methods in Medical ResearchClick here for additional data file.Supplemental material, sj-pdf-2-smm-10.1177_09622802211065159 for Meta-analysisof the severe acute respiratory syndrome coronavirus 2 serial intervals and theimpact of parameter uncertainty on the coronavirus disease 2019 reproductionnumber by Robert Challen, Ellen Brooks-Pollock, Krasimira Tsaneva-Atanasova andLeon Danon in Statistical Methods in Medical Research


The trends of extreme precipitation events during the Indian summer monsoon measured by two different indicators have been analyzed for the period of 1901–2020, covering the entire India in 9 regions segregated by a clustering analysis based on rainfall characteristics using the Indian Meteorological Department high-resolution gridded data. In seven regions with sufficiently high confidence in the precipitation data, 12 out of the 14 calculated trends are found to be statistically significantly increasing. The important climatological parameters correlated to such increasing trends have also been identified by performing for the first time a multivariate analysis using a nonlinear machine learning regression with 17 input variables. It is found that man-made long-term shifting of land-use and land-cover patterns, and most significantly the urbanization, play a crucial role in the prediction of the long-term trends of extreme precipitation events, particularly of the intensity of extremes. While in certain regions, thermodynamical, circulation, and convective instability parameters are also found to be key predicting factors, mostly of the frequency of the precipitation extremes. The findings of these correlations to the monsoonal precipitation extremes provides a foundation for further causal relation analyses using advanced models. These climatological parameters have high interannual and interdecadal variabilities3 and a part of these could be explained by natural climate variability. Nevertheless, there is a high probability that man-made global or regional climate changes could have also affected these quantities with an extent yet to be examined8. While the overall ISM rainfall is believed to have decreased during the twentieth century9 then reversed since the turn of this century10, it has been indicated that the extreme precipitation events might have been rising in some parts of India12, with hypothesized causes ranging from urbanization16, increase in dew point temperature18, to climate variability20. These hypotheses, as indicated in a recent review by Singh et al. (2019)21, were largely proposed based on comparing the trends of extremes with that of a single isolated explanatory variable and thus tended to disagree with each other. Understanding the variation alongside the causes of these extreme events is essential not only for predicting future climate change, but also for making effective mitigation strategies. Here, by applying advanced data science methods in analysing more than a century long surface rain gauge data as well as best available data for other meteorological and climatological variables, or land-use and land-cover (LULC) changes, it has been demonstrated that extreme precipitation events have been increasing in most regions of India, and that such an increase appears to be closely correlated with the long-term changes of certain climatological factors caused by anthropogenic forcing.The amount and distribution of precipitation during the Indian summer monsoon (ISM) have a substantial impact on the region’s agricultural systems and thus the livelihood of more than a billion people23. For this reason, a trend analysis should be ideally performed to an optimal number of regions covering the entire India, each with similar climatic rainfall characteristics to derive suitable regional thresholds for extremes and thus consistent trends, as well as to identify potentially unique driving factors. Following this principle, the trends of two threshold-based rainfall extreme indicators, i.e., the frequency and the intensity (see Methods), have been derived using the Indian Meteorological Department (IMD) high-resolution rainfall gridded dataset, arguably the best available dataset for the purpose. To define relevant study regions, we applied Ward’s minimum variance clustering method to the daily rainfall dataset and found it to be optimal to segregate India into nine different climatologically homogeneous regions . As shown in the following discussion, derived trends of extreme events in our study are similar to those in some previous analyses25 over certain regions that largely overlap with those in the latter works. Nevertheless, due to the fact that several areas with unique precipitation characters were aggregated into the same region in the latter works, the derived trends of regional extreme events could thus entitle to some issues. In contrast, due to the quantitative segregation method used in our study, our trend analysis allows us to assess the precipitation characteristics in certain previously unstudied, yet climatologically interesting regions, such as our coastal region 1, a critical zone of monsoon onset rain belt.Precipitation extremes are low probability events of relatively small spatial scale occurring unevenly across Indiasity see , have beons Fig. b proves 26. Indeed, there was a sharp increase in the number of included stations for regions 4 and 8 during the 1970’s, potentially affecting our effort of defining extreme events there , the confidence in the precipitation data was not sufficiently high to perform the trend analysis, owing to the number of included recording stations in the IMD dataset being either too low or too fluctuating during the considered period. It has been suggested that the varying numbers of stations included in the IMD dataset from year to year could cause certain issues in deriving long-term rainfall trendsHowever, in the seven remaining regions, the number of recording stations, although not exactly constant throughout the century, has not experienced a sharp increase, and this provides us with more confidence when performing the trend analysis using the data of these regions. Furthermore, to ensure the validity of our results, we have performed an additional regional trend analysis only on the grid cells of the seven remaining regions where the number of recording stations is stable throughout the century , and a statistically insignificant increasing trend in region 6 in the east coast.To summarize, among the total 14 trends derived here using two different indicators in seven regions, we have identified 12 statistically significant (95% confidence level) increasing trends and one significant decreasing trend. The remaining trend (frequency in region 6) showed increasing signs but did not pass the Mann–Kendall trend test. These results indicates that the extreme precipitation events during the ISM seasons have increased evidently across majority of India since 1901.16. Indeed, India experienced an intense urbanization between the 1950s and 1980s. The other LULCs such as agricultural lands (particularly the croplands), grasslands and forests were also modified substantially during this period27. Such an anthropogenic factor is believed to have an important impact on climate and rainfall28, implying that the LULC changes could potentially amplify the rising trends of precipitation extremes in the second part of the twentieth century. To verify this difference in the trends, we have applied a running slope difference (RSD) t-test, a statistical method designed to detect the trend turning in a time series29. We find that among the 12 statistically significant increasing trends, 10 of them display a statistically significant breaking point between 1940 and 1970, suggesting that the anthropogenic forcing resulted from man-made LULC changes might have had impact on the trends. The breaking points in the time series are even more sharp when looking at the trends in intensity.In most derived rising trends particularly of the intensity of extremes, there is a sharper increase in the second part of the twentieth century compared to the first part. Certain previous studies also found a similar difference between the pre-1950 and post-1950 trends and attributed it to urbanization30, the potential predictors behind the above-discussed evolution of ISM precipitation extremes since 1901 have been further analyzed with a synergy of best available data of LULC changes and certain climate variabilities. Firstly, 120 year-long multivariate input data have been used to fit a random forest regression model against observations of extreme event indicators. Furthermore, using the successfully trained random forest regression model and a feature importance functionality, we were able to determine the most responsible variables, or effective predicting features, to the rise of extreme events. A major purpose of this practice is to identify the correlations between long-term (century) evolutions of certain climatological (rather than episodic) factors or features and the observed climate trends of the ISM precipitation extremes, thus providing leads for additional attribution analyses should additional data were made available and most importantly, for using advanced models to further examine the causal relations between certain effective predicting features and the observed extreme event trends.Using an ensemble multi-variate and nonlinear machine learning technique, i.e., random forestNote that random forest is a nonlinear regression algorithm, meaning that unlike classical linear regression, it can find complex nonlinear correlations between the input features and the output result. Hence, it is better suited for complex systems like the climate system with different feedbacks. For example, an input parameter such as sea surface temperature might have had an important positive influence on the extreme events for a certain period of time, while at some other period, another parameter  could become more influential. Random forest regression model can capture this type of behavior, and its feature importance tool also gives us a good insight on the variables that had the most impact on the trends throughout the whole time period.31. These classical feature importance methods can still correctly rank the driving features of the trends of extreme events32 , but they might likely lead to overestimating the importance of some alternative features that are correlated to the true predictors33. Therefore, we chose to calculate the conditional feature importance 33, which can better reflect the true impact of each predictor variable on the different trends, even in the presence of highly correlated features. This allows us to have a high confidence in the derived variables’ contribution through the feature importance. In addition, we also performed feature selection (Methods) in order to keep only the features showing high contributions to the prediction using random forest model.The feature importance has been calculated using the conditional permutation method in order to determine the contribution percentages of each given feature in predicting the accurate values of the different extreme events indicators (Methods). Traditionally, the feature importance is calculated using either the Gini impurity decrease or the classical permutation importance. However, the presence of correlated input features, which is often the case with meteorological data, has been shown to possibly impact the ability of both methods to identify strong predictors2 = 0.39), meaning that we did not manage to find accurate predictors for this particular trend. Consequently, we do not show the feature importance results for this trend.Note that this feature importance only reflects the contribution of each feature to the prediction of the testing dataset (Methods). For example, a conditional permutation importance of 50% for the urban fraction does not necessarily mean that 50% of the increase in extreme events was actually due to the increase in urban fraction, but rather represents the importance of the urban fraction in the random forest model prediction of extremes. Nevertheless, by including a considerable number of features arguably for the first time, our analysis still represents a step forward from previous works using only single or a few factors. In fact, the regression scores of 13 out of 14 regressions are very good, with accuracy ranging from 0.68 to 0.96 Fig. , suggestWe have included seventeen different input features in the multivariate analysis using a random forest model Table , and mer35. However, it was argued that dew point temperature could be a better predictor of extreme events than temperature in the tropics, especially in India18, hence, we decided to include both features. It has also been suggested that the increase in SST over the Arabian Sea may play a role in the enhancement of extreme events by increasing moisture availability24, while other studies highlighted the more prominent role of the temperature gradient between the ocean and the land on monsoon rainfall10. Another key mechanism that has been suggested as a potential cause for the rise in extremes in several previous studies is the monsoon circulation. While some leaned towards the strength36, others highlighted the direction and variability of the monsoonal flow19. As most of the moisture over India during the summer monsoon season is advected from the Arabian Sea, in order to evaluate the impact of the monsoon circulation strength on the trends of extremes, we chose to use the zonal and meridional components of the surface wind (U and V respectively), as well as the combined wind speed as input features, all averaged over the Arabian Sea. In addition, some studies have also linked atmospheric instability to precipitation enhancement. For instance, one of our previous studies37 argued that the surface MSE is a useful parameter for quantifying the degree of strength of the monsoon convection, while another study36 stated that an enhancement in convective instability, measured by calculating the MSE at different pressure levels, could also lead to enhanced precipitations. Furthermore, a recent effort19 also identified the increase of convective available potential energy or CAPE over the Bay of Bengal as a potential contributor to the observed intensification of severe storms. The link between monsoon depressions forming over the Bay of Bengal and extreme precipitation events, established on a daily timescale, receives certain acceptance among the tropical meteorology community. Hence, we also included a feature representing the number of depressions per monsoon season, even though it was previously demonstrated that extremes did not show any correlation to the number of depressions at a decadal timescale12. Lastly, we would reiterate that we use the long-term climatological evolutions instead of episodic variations of features in the analysis. This better serves the purpose to establish their correlations with observed climatological trend of the monsoonal precipitation extreme events.Note that in category (2), (3), and (5), the quantities of included physical parameters can be affected by, besides anthropogenic impacts , global climate processes and feedback involving natural climate system features. The variables in each category were selected based on current knowledge and certain hypotheses of monsoon climatology, as detailed hereafter. Specifically, temperature and humidity are commonly regarded as being correlated to extreme precipitation trends, because the warming of the atmosphere could lead to enhanced moisture availability and hence more intense rainfall12. Region 6 is also the only region where the frequency of extremes seems to be impacted by ENSO. Furthermore, the LULC changes are also important predicting factors for the trends of frequency, appearing (at least one of its four components) among the important features in five out of six regions. The above results thus suggest that in causing the frequency increase of extreme events, both local anthropogenic factors and climate variabilities could have played active roles.For the trends in frequency of extreme events, a very wide variety of factors appear to have played important roles in influencing the observed trends Fig. , as eachWe find that for all regions, the most important features related to the intensity trends of extremes are almost exclusively the LULC changes Fig. . In ever28. For instance, LULC changes can affect the radiative budget, either by modifying the surface albedo or through effects on the surface latent and sensible heat fluxes, hence atmospheric water vapor or cloud properties. The LULCs would further affect directly or indirectly the local water cycle, by perturbing the evapotranspiration fluxes, or by inducing rainfall modification. In our analysis, we find that the most significantly correlated LULC feature to the long-term trends of extremes are the changes in urban fraction. It is hypothesized that the change in urban areas could modify the rainfall through various mechanisms38. Some of them involve land surface heterogeneity feedbacks: convection could increase via thermal perturbations induced by the urban heat island, or the increase in surface roughness39. Others are linked to urban anthropogenic emissions, in particular the aerosol indirect effect: urban aerosols can act as cloud condensation nuclei and thus modify cloud microphysics, radiation, and precipitation41. Previous studies did link the rise in extreme events in some parts of India with urbanization by comparing the pre-urbanization and post-urbanization trends16, or by analyzing rainfall at different urban locations42. By exploiting a multivariate analysis using the longest available data, our unique result not only confirms what has been suspected, but also underlines the importance of the other LULC changes. These results are, however, spatially heterogeneous. Some regions, like the Northeastern region 2, seem to display important correlations between the trends of extremes and LULC changes, particularly changes in urban fraction. Note that out of the seven studied regions, region 2 has experienced the most intensive urbanization, where the urban fraction has been leveraged from 0.3% on average in 1901 to 5.5% on average in 2015, or a spectacular + 1670% increase. According to our study, this tremendous increase may have contributed to the rise in the frequency and intensity of extreme events.Our machine learning study confirms what has been hypothesized in the earlier trend analysis, that there indeed is a strong marker of the LULC changes in every calculated trend particularly of the intensity of extreme events. LULC changes can impact the regional and global climate through several processes, though their extent and underlying mechanisms may not be well defined to this day12. In region 1, the other coastal region adjacent to the Arabian Sea, the land–ocean temperature gradient appears in the important factor list in addition to the meridional component of the monsoon wind, suggesting that the extreme events in this region are more subject to processes related to the land-sea breeze effect. This finding is also explained by the fact that region 1 is located on the windward side of the mountain range of Western Ghats and receives directly the south-westerly monsoonal winds that bring moist air from the Arabian sea. However, what is surprising is that the wind and land–ocean temperature gradient, commonly believed to be a good indicator of the monsoon strength, do not appear as important predictors for precipitation extremes in the remaining regions, at the considered time scales.Nevertheless, in certain regions, the markers of urbanization and other LULC changes are not that obvious. For example, in region 9, the region adjacent to the Arabian Sea, there is not any LULC feature appearing in the selected important features of the frequency trend. Instead, SST variations seem to have a dominant role in modulating both frequency and intensity indicators in this region. This could be explained by the fact that a warmer SST would consequently increase the quantity of water vapor over the Arabian Sea through evaporation. Given that the summer monsoonal winds are essentially south-westerly, this additional water vapor would in turn be advected to the Indian subcontinent by the monsoon circulation and enhance moisture supply, thus favoring the occurrence of extreme precipitation events43, since a warmer atmosphere can contain a higher quantity of water vapor thus provoke more frequent and more intense rainfall. However, in our analysis, the surface air temperature only appears as an important predictor in region 7 and with very little contribution, whereas the dew point temperature is found to have modulated the trends in frequency in regions 1, 7 and 9, confirming a previous hypothesis stating that it is a better predictor of extremes in tropical regions18. The reason behind this finding could be that the monsoon precipitation events can actually induce a cooling of the atmosphere due to the high quantity of liquid water evaporation, hence there is no positive correlation between extreme precipitation events and surface air temperature. The positive correlation becomes apparent, however, when looking at the dew point temperature, a direct measure of the absolute humidity of the atmosphere.Moreover, it is understood from the Clausius–Clapeyron equation that the atmospheric temperature could be a main driver of the extreme events globally45, the relationship between ENSO and extreme precipitation events in India is still not well understood. Future analyses on the correlation between ENSO and extreme precipitation events in this specific region would be required to assess the real impact of this large-scale climatic feature.It is also interesting to note that ENSO seems to be an important predictor for the frequency trend in region 6. While the effects of ENSO on the average monsoon rainfall have been studied and some links have been proposed19 and to the intensification of extreme rainfall46. Our findings show that it may also have an impact on the long-term trends of frequency of extremes, meaning that a stronger average CAPE during a monsoon season would indicate a higher number of extreme events, in this coastal region.Finally, the convective instability parameters appear as important predictors of extremes in both west-central (region 9) and east-central coastal region (region 6), especially for the trends of frequency. Particularly, the long-term variation of CAPE calculated over the Bay of Bengal seems to have some impact on the frequency of extreme events in region 6, which is located right next to the Bay of Bengal. By favoring occurrence of strong convective activity, increase in CAPE have been linked to wet spellsThe trends of extreme precipitation events defined using two different indicators during the Indian summer monsoon season in the past 120 years have been analyzed. Instead of focusing on a few selected locations, this analysis uniquely covers the entire India, consisting of 9 regions segregated using a clustering method based on precipitation characters. It is found that the majority of India has experienced a statistically significant increase of monsoon precipitation extremes throughout the analyzed 120-year period. Furthermore, the effective predicting factors behind such an increase have also been analyzed using a nonlinear and multivariate machine learning regression, the random forest, based on the best available data of 17 input features describing anthropogenic activities, climate dynamical and physical processes, and variabilities. The results reveal that the man-made land use land cover changes appear to be the most critical features in predicting the observed climatological trends of monsoonal precipitation extremes, implying implicitly that these features might have played an important role in causing the discovered rise of monsoonal precipitation extremes particularly of their intensity. Whereas several climate variability factors including dew point temperature as well as SST, and main monsoonal wind strength, all over the Arabian Sea are also critical to predict the trends of extremes in several regions, especially for their frequency. Nevertheless, certain indicators commonly believed to be drivers for mean monsoonal rainfall strength such as land–ocean thermal contrast, ENSO variation or convective instability are found to be less correlated to the trends of precipitation extremes than expected.https://scikit-learn.org/) applied to a NxL size data matrix.The regions with similar climatological precipitation characters and thus suitable for performing trend analysis were defined by applying Ward’s minimum variance clustering method. Ward’s method is an agglomerative hierarchical clustering method, where each data point is initially considered as a single cluster, then grouped together by calculating the Euclidean distance between them. Here in our case, a data point corresponds to the daily rainfall time series of a single grid cell for the period of 1901–2020, taking only the monsoon days (from June to September included), i.e., the total number of samples is equal to L = 14,640 for each data point. The number of data points N is thus equal to the number of grid cells in the rainfall dataset, i.e., N = 4,954. At the start, each data point is treated as one cluster, so the initial number of clusters is N. Then, a cluster is formed by joining the two closest data points, resulting in N-1 clusters. This step is repeated until one big cluster is formed. The optimal number of clusters is then determined by plotting the dendogram, a figure that represents graphically the distance between data points as well as the distance between clusters, and then choosing the number of clusters that maximizes the inter-cluster distance. We plotted the dendogram using the function of Scikit-learn Python library 30, a non-linear supervised ensemble machine learning algorithm that uses multiple decision trees to fit targeted output  with selected input data or features .Step 2: The dataset is split using the different features. The resulting variance for each branch is calculated and subtracted from the variance before the split to obtain the variance reduction.Step 3: The feature with the largest variance reduction is chosen for the decision node.Step 4: The dataset is divided based on the values of the selected feature. This process is recursively repeated until all data is processed based on chosen thresholds.This construction process can be summarized with these steps:The prediction of a new sample is simply calculated using the path created by the decision tree and averaging the values of the samples in the final node .33 to determine the importance ranking of each one of the 17 input features, while minimizing the impact of the multi-collinearity. To perform the feature selection, we calculated the average importance of the features, and removed the features that showed an importance inferior to the mean importance. This finally yielded between 3 to 6 important selected features in the different regions, which are displayed in Figs. We first used the cforest function of the R Party package library to perform random forest regressions of observed extreme trends for each region using various input features. Up on the success of regression, we then applied the conditional permutation feature importance functionalityWe have selected seventeen features in random forest regression and feature important analysis Table . For locSome input features are calculated within each region, such that the input trends are region dependent. This is the case for the LULC changes, surface temperature, dew point temperature, and relative humidity. For these features, their annual values were derived from monsoon seasonal and regional means. Other features  have been calculated over the Arabian Sea, as the moist monsoon winds that provokes precipitation are essentially south-westerly, while the CAPE has been derived over the Bay of Bengal, following considerations from previous studies. All these parameters have been calculated at the ocean surface. The land–ocean temperature gradient has been calculated by taking the surface air temperature difference between the Arabian Sea and the Indian subcontinent. Finally, we also tested the influence of large-scale climate variabilities including the El-Nino Southern Oscillation (ENSO) and the Indian Ocean Dipole (IOD), by using the Extended Multivariate ENSO Index (MEI) v2 and the Dipole Mode Index (DMI) averaged over the summer monsoon seasons as input features. Before fitting the model, each feature is normalized to a range of −1, 1]. The list of features is detailed in Table , 1. The 2 of the prediction, defined as:test and ypred being respectively the value of the testing data and the value predicted by the model.To evaluate the accuracy of the random forest model, we used the common train/test split method, which consists in fitting the model with a random subset of the data, and then testing the accuracy with the remaining testing data. This process is repeated fifty times to ensure that the random subsets cover the whole range of our initial dataset, the final score being the averaged value of these fifty scores. Here, 70% of the input data is chosen randomly to train the model, and the regression score corresponds to the coefficient of determination RThe best possible score is 1 and corresponds to a model that predicts exactly the right value. The score can also be negative if it fails to deliver any information on the data.Supplementary Figures.


Falling is a major cause of personal injury and accidental death worldwide, in particular for the elderly. For aged care, a falling alarm system is highly demanded so that medical aid can be obtained immediately when the fall accidents happen. Previous studies on fall detection lacked practical considerations to deal with real-world situations, including the camera’s mounting angle, lighting differences between day and night, and the privacy protection for users. In our experiments, IR-depth images and thermal images were used as the input source for fall detection; as a result, detailed facial information is not captured by the system for privacy reasons, and it is invariant to the lighting conditions. Due to the different occurrence rates between fall accidents and other normal activities, supervised learning approaches may suffer from the problem of data imbalance in the training phase. Accordingly, in this study, anomaly detection is performed using unsupervised learning approaches so that the models were trained only with the normal cases while the fall accident was defined as an anomaly event. The proposed system takes sequential frames as the inputs to predict future frames based on a GAN structure, and it provides (1) multi-subject detection, (2) real-time fall detection triggered by motion, (3) a solution to the situation that subjects were occluded after falling, and (4) a denoising scheme for depth images. The experimental results show that the proposed system achieves the state-of-the-art performance and copes with the real-world cases successfully. According to the reports from the World Health Organization (WHO), approximately 28–35% of the elderly fall each year, and it increases to 32–42% for those over 70 years of age . GloballThe studies about fall detection systems are mainly divided into two categories: fall detection based on wearable devices and situational awareness devices . AccelerIn previous studies, RGB cameras were frequently used as the input sensors for fall detection. Since RGB cameras are highly affected by the lighting conditions, the detection accuracy of the system is highly affected by the scene where the system is operated. In addition, fall accidents happen during the night very often when the elderly gets up from bed to use the restroom. Fall detection using an RGB camera fails to function without illumination. In addition, the elderly may frequently fall when using the restroom or taking a bath due to the slippery floor and using RGB images can lead to privacy violations. Accordingly, to tackle the problems mentioned above, the depth cameras have become a better choice of sensors for fall detection. The depth cameras obtain the depth information and achieve lighting independence through infrared structured light emission. As opposed to the RGB images, users’ detailed facial information is not recorded in the depth images and thus ensures users’ privacy. However, even though the depth images can be adopted to overcome the limitations of RGB images, it can be observed that the depth images are noisy and provide weak local gradient information of objects compared with RGB images , making For fall detection systems, compared with the normal activities of daily living (ADLs), the occurrence rate of fall accidents is relatively low. From the perspective of machine learning, data imbalance between positive samples  and negative samples (video clips with ADLs) is not conducive to the approaches of supervised learning classification in fall detection. In addition, the fall detection systems are expected to work in the real-world scenarios that the subjects are occluded after falling since the subjects are usually surrounded by beds, chairs, tables, and other furniture in the nursing home, as shown in (1)A sequence-to-sequence denoised GAN structure is proposed for fall detection, which achieves state-of-the-art performance.(2)Our approach can perform multi-person fall detection.(3)A denoising scheme is designed for depth image inputs.(4)The proposed system can perform real-time fall detection in the real-world situation that subjects were occluded by the furniture after falling.T consecutive frames. Finally, fully connected layers took the outputs from LSTM to classify if the subject was falling or not. Even though the depth images were used as the inputs to achieve lighting independence, the detection performance of their method is still greatly affected by the noises in the depth images. In addition, the problem of uneven collection of positive and negative samples still exists with their approach.With the recent advances in convolutional neural networks, several deep learning-based frameworks have been proposed for fall detection. Núñez-Marcos et al.  presenteTsai et al.  proposedNogas et al.  proposedN consecutive frames are fed into the generator with the corresponding N consecutive frames as the ground truth, of which half of the frames come from the inputs, and another half of the frames are obtained from the following frames after input frames to learn the spatiotemporal features and perform future frame predictions in the inference phase.In this study, IR-depth images and thermal images are used as the input source for fall detection to reduce the influence of lighting conditions and avoid privacy violations. In addition, only the frame sequences with normal ADLs are collected in the training dataset for frame reconstruction, so that the frame sequences with fall accidents will lead to bad reconstruction by the trained model and trigger the anomaly alarm. To achieve that, a model with generative adversarial network (GAN) structure that takes frame sequences to capture the properties of spatiotemporal changes in motion and produces the corresponding reconstruction for future frame predictions is proposed. The proposed fall detection system takes IR-depth images and thermal images as inputs for the purpose of achieving lighting-invariant and privacy protection. For performance evaluation, two public datasets were used in the experiments: Thermal Fall Dataset  and UR FThermal Fall Dataset:Thermal Fall Dataset  consistsUR Fall Detection Dataset:The UR Fall Detection Dataset  containsIn the proposed framework, the continuous motion features are extracted, and then used for future frame reconstruction by the generator. Meanwhile, the discriminator network is used to distinguish the reconstructed frames from the authentic frames. Moreover, to further reduce the influence of the noises from depth images, the reconstructed frames are sent back to the generator network to obtain the encoded features for comparison with the corresponding authentic frames, checking if any anomaly occurs in the testing phase. The design details are elaborated as follows.The generator network is based on an encoder–decoder structure  that takN by N patches for judgments. (N = 70 in our experiments). After each block is judged, the ensemble scheme is applied to average the prediction results for classification. For the generated images, the high-frequency parts were frequently checked by the discriminator. As a result, it further reduces the influence of the noises in each small patch from the depth images.For the discriminator network, we followed the structure and settings of PatchGAN ,22,23. SOther than the hole-filling algorithm and PatchGAN architecture, the encoded feature comparison is also designed for denoising when using depth images as the inputs. Since only the main components from the dataset are encoded as features when training the generator model, the jittering noises in the depth images are not the stable components to be encoded and would be ignored. As is shown in As a result, if the input sequence contains the continuous frames with the motions of normal activities, the trained generator can reconstruct the corresponding future frames more precisely because the motions of normal activities have been learned from the training dataset and are predictable. Subsequently, the encoded feature vector The proposed framework based on the generative adversarial network (GAN) structure comprises the generator network, the discriminator network, and the FlowNet. In training, the hybrid loss function that takes L2 loss, gradient loss, optical-flow loss, and adversarial training loss into consideration is designed for network optimization, as is shown in p represents the pixel in the image P, and N is the total number of pixels in the image. In Equation (2), py stands for the pixel value of p from the target image, and pred(p) is the predicted pixel value of p from the reconstructed image.For most of the image reconstruction tasks, However, compared with the frames in the input sequence, many high-frequency details are missing in the reconstructed frames. To tackle this problem, the loss function that considers the image gradient differences is needed in the optimization to enhance the detailed contours in the reconstructed results.i and j stand for the spatial position of a pixel in the depth image. The variable Y is its corresponding ground truth. Herein, only the gradient intensity differences between two adjacent pixels are considered in the gradient loss. The adoption of gradient loss is expected to retain more high-frequency details so that the information of subjects in the depth image can be well captured.To enhance the high-frequency details in the reconstructed frames, the gradient differences between the reconstructed results and the corresponding ground truth are taken into consideration in training. The mathematical expression of the gradient loss is asI is the corresponding ground truth.The L2 loss and gradient loss mainly focus on the spatial characteristics of the differences in pixel intensity between the predicted frames and their corresponding ground truth. To capture the temporal features of frame-to-frame motion changes for future frame prediction, the FlowNet  is embedk-frame sequence  for an iteration due to the limitation of memory capacity from GPU, and Adam [All experiments were run on a and Adam  was chosIn this section, the public datasets, the thermal dataset  and the For network optimization, our goal is to search for the optimal loss function combination to boost the performance. In the beginning, L2 loss was applied to minimize the differences between the predicted frame and the corresponding authentic frame for optimization. However, many high-frequency details in the reconstructed images were missing when using the L2 loss only. Thus, the gradient loss was added to further enhance the reconstruction results. The reconstructed frames with a standing subject by “L2 loss” and “L2 + Gradient loss” are demonstrated in In addition, the unpredictable noises after hole filling lead to the jittering effect among frames, affecting the prediction results with respect to the inputs/outputs of frame sequences. In In our proposed framework, the reconstruction error increases when an anomalous fall event occurs since the model cannot correctly reconstruct the abnormal frames and triggers the alarm for fall detection. However, frame-by-frame analysis for determining whether the reconstruction error becomes higher may cause false alarms easily . ConsequCσ and Cµ are the approaches with their proposed anomaly scoring scheme in [Cσ and Cµ in [In cheme in  that inccheme in . Our prond Cµ in  that neend Cµ in  can stris were computed to validate if the two classes  are well-separated in the systems. The larger score gap indicates the greater difference between two classes, i.e., “fall” and “non-fall”, that can be well-distinguished by the systems. As a result, the system with a larger score gap can achieve better accuracy in detection, leading to a lower false alarm rate. The score gap Δs is computed, followed by Equation (7).In addition, the score gaps ΔN1 and N0 are the total number of frames defined as “fall/1” and “non-fall/0”, respectively. The comparison of performance with score gap Δs and AUC between DeepFall-DSTCAE-C3D [In Equation (7), TCAE-C3D  and our TCAE-C3D  on both The better detection performance achieved by our proposed framework should contribute to our denoising schemes of applying the PatchGAN-like structure and the encoded feature comparison. Without a proper denoising scheme to tackle the noise problem in the depth images, the anomaly detection system cannot distinguish whether the poor reconstruction is caused by the abnormal motion from the foreground subject or just the unpredictable noise from the background, arousing higher false-positive rates in detection. The experimental results show better and more robust performance on both the UR Fall Detection dataset and the thermal dataset. Some examples of the fall detection results are shown in The performance comparison among the previously proposed methods that conducted their experiments on the UR Fall Detection dataset is presented in Except for the above two public datasets used for performance comparison with the previously proposed frameworks, our designed fall detection system was further assessed to determine whether it can work on the real-world situations. For the practical fall detection systems, the cameras are usually mounted on the ceiling or walls with a certain angle to the subjects. Therefore, here we had our sensor of Time-of-Flight (ToF) mounted at 2.2 m height with a certain angle toward to the subjects and the maxima measuring distance of 4.5 m to simulate the real-world situations. The model trained on the UR Fall Dataset was used for inference since the sensor of ToF also generates the depth images as the inputs. In addition, in the inference phase, the model and the input sequence take about 3.7 GB in the GPU memory, and the execution frames-per-second is 12, tested under NVIDIA RTX 2080 Ti. Accordingly, to achieve real-time detection for the real-world applications, the input frame sequences are collected by sampling every 3rd frame, so the proposed fall detection system will check whether the subjects are falling every second.In this study, a GAN-like network that inspects the reconstruction quality of future frame sequences based on the subject’s motion was proposed for fall detection. The proposed framework can cope with the extreme scenario when the subjects were blocked or unseen after falling. In addition, to ensure users’ privacy, the depth images or the thermal images are used as the inputs, and several denoising schemes are proposed to prevent the noises in the input images from interfering with the prediction results. Accordingly, the unsupervised learning approach is adopted that considers fall accidents as an anomalous event to tackle the problem of an imbalanced dataset from data acquisition. Except for the advantages mentioned above, the experimental results on two public datasets show that our proposed framework for fall detection achieves the state-of-the-art performance. For future work, a scheme to derive the adaptive threshold that performs device-invariant features for the system can be developed. In that case, the framework can work for various sensor devices without having separate threshold settings for each device.


UAVs are widely used for aerial reconnaissance with imaging sensors. For this, a high detection performance (accuracy of object detection) is desired in order to increase mission success. However, different environmental conditions (negatively) affect sensory data acquisition and automated object detection. For this reason, we present an innovative concept that maps the influence of selected environmental conditions on detection performance utilizing sensor performance models. These models are used in sensor-model-based trajectory optimization to generate optimized reference flight trajectories with aligned sensor control for a fixed-wing UAV in order to increase detection performance. These reference trajectories are calculated using nonlinear model predictive control as well as dynamic programming, both in combination with a newly developed sensor performance model, which is described in this work. To the best of our knowledge, this is the first sensor performance model to be used in unmanned aerial reconnaissance that maps the detection performance for a perception chain with a deep learning-based object detector with respect to selected environmental states. The reference trajectory determines the spatial and temporal positioning of the UAV and its imaging sensor with respect to the reconnaissance object on the ground. The trajectory optimization aims to influence sensor data acquisition by adjusting the sensor position, as part of the environmental states, in such a way that the subsequent automated object detection yields enhanced detection performance. Different constraints derived from perceptual, platform-specific, environmental, and mission-relevant requirements are incorporated into the optimization process. We evaluate the capabilities of the sensor performance model and our approach to sensor-model-based trajectory optimization by a series of simulated aerial reconnaissance tasks for ground vehicle detection. Compared to a variety of benchmark trajectories, our approach achieves an increase in detection performance of 4.48% on average for trajectory optimization with nonlinear model predictive control. With dynamic programming, we achieve even higher performance values that are equal to or close to the theoretical maximum detection performance values. Unmanned aerial vehicles (UAVs) with imaging sensors in the visual or infrared spectrum are increasingly used in various fields in civil, commercial and military applications. Examples include surveillance and reconnaissance missions ,2, envirnonlinear model predictive control (NMPC) and dynamic programming & optimal control (DP&OC), which were proposed in our previous works  see  and proc0,1] see .(31)cmiz that incorporates the best combination of benefit and effort. Furthermore, this optimal path is the set point for the NMPC optimization and yields the optimal control input This results in a time step-dependent combined minimum cost n, an optimal control strategy From the processing of all time steps y π∗(x0)  results Dynamic programming and optimal control is an optimization method that can be used for generating optimal sensor-model-based UAV flight trajectories for discrete-time and discrete-value systems. This approach was first described in [Dubins paths, which is described in the next section. This allows the explicit limitation of the admissible curve radius to meet g-load constraints, which could not be realized with our previous approach.ribed in  where thDubins path planning was first outlined by Dubins [y Dubins  and descDubins paths are used in this work to model the state transitions for the discrete optimization method of dynamic programming. This is motivated by two major advantages of Dubins paths: the paths are curvature constrained, taking the flight envelope limit for the allowable acceleration into account. Furthermore, the principle of the Dubins path results in the shortest (flight) path between two configurations, which ultimately minimizes the flight duration.For modeling the trajectory of a fixed-wing UAV using Dubins paths, a forward velocity pp curve . The Dubpp curve ,61,62.v of the UAV must be set constant.The velocity The maximum permissible roll angle In the following, the basic principles of Dubins path planning are presented before they are combined into a global optimal trajectory of concatenated path segments using dynamic programming. For this, the following two criteria must be satisfied ( p. 880):L is a left-hand curve of maximum curvature, R is a right-hand curve of maximum curvature, and S is a straight line segment. Equations (The Dubins path is generated from the set of motion primitives quations  to 34)  {L,S,R},g, and the constant flight path velocity v. As a result of the discrete control inputs With L and R, Equation ctrThe state transition cost represents the length of the Dubins path calculated in Equation  from theEquation : (43)ctrv. On the other hand, Here, smin in  is the m smax in  is defin smax in  shows thstate-dependent costs in  and the state transition costs (index tr) are exemplarily plotted for two time steps in an acyclic graph.In optimal controlFrom the backward calculation and the principle of the optimality follows that each state in the time steps termined .global optimal trajectory can be found by comparing the total cost global minimum total costThe A major disadvantage of DP&OC is that the discretization of the state space increases the computational cost quadratically to the number of system states onality”  for thisFor a more complete description of the dynamic programming algorithms, we refer to the work of Bellman  and Bertnonlinear model predictive control as presented in In the final step, the trajectory, which is composed of concatenated Dubins path segments, is smoothed. This is to achieve a continuous roll angle transition along the entire trajectory, as required in Benchmark trajectories will be used as a baseline to validate the sensor-model-based trajectory optimization. For this purpose, the detection performance resulting from the benchmark trajectory and the optimized trajectory from The benchmark trajectories used in the following are based on common loitering patterns used in aviation. It is assumed that these trajectories are either generated automatically by a flight management system on board the UAV or are determined by a UAV operator. In both cases, the planning is carried out without the knowledge or consideration of the sensor performance models from Circle pattern.Racetrack pattern.Figure-8 pattern.The following three loitering patterns are used as benchmark trajectories in this work:All three patterns have in common that they are made up of an easy-to-model geometry and consist of a closed set of lines. Thus, each pattern can be passed through an unlimited number of times. The shape of the benchmark trajectories in our work, which are depicted in The implementation of the NMPC functionality was carried out in C++, whereas the other parts of the program, such as the program control, the coverage path planning, the calculation of the perception maps and the evaluation were implemented in Python. The communication between the C++ process and the Python modules was realized using ROS 2 . In controute reconnaissance scenario .For trajectory optimization with DP&OC, two simulations are performed for the route reconnaissance scenario and two for the area reconnaissance scenario. In each case, one simulation is carried out with the CC-SPM sensor performance model and the other one utilizing Yolo-SPM. The result in each case is a global optimal reference trajectory whose expected detection performance is compared with the theoretically maximum average detection performance.For the NMPC-optimized reference trajectory and the benchmark trajectory, which is only used for the NMPC optimization as a comparison, the sensor footprint velocities are constant compare  and equaIn contrast, the length of the DP&OC-optimized reference trajectory is longer since it is based on a cartesian grid with equidistant spacing. As the UAV velocity is constant and consistent with the other trajectories, the flight time increases. Additionally, because the UAV trajectory is also matched to the sensor movement, the sensor footprint velocity must be dynamically slowed down, however, this will not be discussed in detail in this paper.The following is assumed for the execution of the simulation: The route or area to be reconnoitered is defined and known a priori. Coverage path planning has already been carried out and is identical for the reference trajectory as well as for the benchmark trajectory. It is assumed that the benchmark trajectories are set by a UAV operator or a flight management system on board the UAV, without the knowledge or consideration of the corresponding sensor performance model. However, the expected detection performance for both trajectories is determined based on the same performance model. This is necessary to be able to compare the detection performance results with each other.With the route reconnaissance scenario, the course of the route was designed in such a way that the topographic conditions included both rural  and urban regions . In addition, the routing should contain several changes of direction to show the ability of the reference trajectory for adaptation. The route has a length of about 2.2 km. The arrow marks the direction in which the reconnaissance task is conducted. The illustration of the different perception maps is intended to emphasize that the regions with high detection performance can vary significantly from map to map. For example, in the top left perception map, the area of high detection performance is far from the center of the map and thus far from the center of the sensor footprint. In contrast, in the lower left perception map, the area of high performance is concentrated near its center. In the upper right map, the area of high performance is even more localized. To achieve a high overall detection performance for the reconnaissance task, the trajectory optimization has to calculate a reference trajectory that ideally passes only through these areas of high performance, while taking into account additional constraints such as roll angle and roll rate limitations of the UAV.The perception map resulting from the Yolo-SPM model is consistent across the route and is displayed in atmospheric states for the CC-SPM sensor performance model. It is considered that these are determined by mission planning  and the local weather conditions in the reconnaissance area. Due to the localized extent of the reconnaissance area, these are assumed to be constant during the actual reconnaissance operation.topographic states are depicted in The The variation of the topographic states along the sensor footprint path results in a highly dynamic change of the perception maps as depicted in atmospheric states for the Yolo-SPM performance model are listed in The To give the reader an idea of the trajectory optimization results, In maximum average detection performance is determined from the maximum values of each perception map, which is graphically represented by the upper bound (black line) in average detection performance is calculated from the average of each of the six simulations with the CC-SPM or the Yolo-SPM performance model. With the NMPC-optimized reference trajectory utilizing the CC-SPM model, an average increase in detection performance of 4.46% is achieved. Additionally, the NMPC-optimized reference trajectory with the Yolo-SPM, an enhancement of 4.90% is obtained. The length of the flight trajectory is about 2.6 km and approximately identical for both NMPC and benchmark trajectories.The simulation results for the route reconnaissance scenario with NMPC-optimized reference trajectory and benchmark trajectory are summarized in By utilizing DPOC optimization, global optimal reference trajectories are generated, thus eliminating the need for a direct comparison with a benchmark trajectory. It follows that only two simulations were performed for route reconnaissance: one using the CC-SPM performance model  and (b), and the Circle pattern in (c) and (d) in maximum average detection performance and the average detection performance is carried out similarly to the description in If we omit the separation into route and area reconnaissance, the average increase in detection performance by the NMPC-optimized reference trajectory compared to the benchmark trajectory is 4.09% with the CC-SPM performance model and 4.88% with the Yolo-SPM model. If all 24 simulation results are considered equally without differentiating between route and area reconnaissance or sensor performance models, the NMPC-optimized trajectory yields an increase in detection performance of 4.48% compared to the benchmark trajectory.The results of the simulation with DP&OC are summarized in In the case of area reconnaissance with the CC-SPM performance model, the length of the flight trajectory is approximately 6.3 km. In comparison, the optimized trajectory for the Yolo-SPM is about 5.3 km long. Both trajectories are depicted in The reference trajectories that are generated using sensor-model-based trajectory optimization are calculated in advance of the actual reconnaissance process and act as setpoint inputs. Therefore, the computation of these optimized trajectories is carried out prior to the execution of the flight and are therefore not subject to any real-time requirements. The optimization of the computation time was therefore not the focus of this work. Nevertheless, we would like to briefly mention the computational effort: The computations were performed on a desktop PC with a six-core processor running at 3.3 GHz. As mentioned in By using our approach for sensor-model-based trajectory optimization, we were able to show that an increase in detection performance of approximately 4.5% on average was achieved with trajectory optimization using nonlinear model predictive control. With dynamic programming optimized reference trajectories, we even obtained detection performances that are equal or close to the theoretical maximum detection performance values.Using the reference trajectories obtained by DP&OC optimization, a level of detection performance can be achieved that exceeds the average detection performance of both the benchmark and the NMPC-optimized trajectory. For the reference trajectories optimized for the Yolo-SPM model, even the theoretical maximum detection performance for route and area reconnaissance is achieved. However, this high detection performance comes at the cost of a significantly longer flight trajectory, which also increases the reconnaissance duration. For route reconnaissance, the flight duration increases by a factor of 1.3 to 2.1, depending on the sensor performance model used. For area reconnaissance, the flight duration increases by a factor of 1.5 to 1.8. Therefore, for mission planning, it must be weighed whether the gain in detection performance justifies the increase in reconnaissance duration.The comparison of the computation time is intended to indicate that the generation of a global optimal reference trajectory with DP&OC optimization is associated with a computational effort about 40 times higher than for NMPC-optimized trajectories. It is therefore highly dependent on the application case, which optimization method shall or can be applied. We assume that the computation time for the DP&OC optimization could significantly be reduced by a high parallelization of the dynamic programming task. Alternatively, the resolution of the discretization can be reduced in order to vastly decrease the number of computations.In this paper, we utilized sensor-model-based trajectory optimization to enhance detection performance in unmanned aerial reconnaissance. For this, we presented a newly developed sensor performance model, which maps relevant environmental states (including elevation angle and ground sample distance) to the expected detection performance for a perception chain with a YOLOv3 object detector. By utilizing the sensor performance model and optimization methods NMPC and DP&OC, we computed optimized reference trajectories for the UAV that are coordinated with the spatio-temporal positioning of the sensor footprint on the ground. By conducting several experiments in a simulation environment, with these reference trajectories, we achieved an increase in detection performance compared to the detection performance resulting from various benchmark trajectories. Furthermore, it could be shown for the DP&OC optimization method that the state transitions based on Dubins paths resulted in valid trajectories with consideration of curvature constraints.In summary, we have verified that our approach of sensor-model-based trajectory optimization is capable of enhancing the resulting detection performance. Additionally, different requirements concerning perception , sensor platform , environment  and multiple mission aspects  are considered for the optimization.