More and more sport associations introduce innovative devices to support referees and umpires respectively, affecting a strong need for the evaluation of these devices. This study evaluates the use of the new vanishing spray for free kicks in the German Bundesliga. In more detail, the aim of the study is to investigate if the spray reduces violations of the required minimum distance and consequently the respective punishments, if it reduces errors concerning the distance set by the referee and if it leads to a higher success rate of free kicks.Therefore, 1833 free kicks of the 2013/2014 and 2014/2015 season of the German Bundesliga were screened using a self-designed observational system. For the statistical analysis two parallel samples were built of 299 free kicks each.2 = 4.58; p < .05). However, none of these violations were punished appropriately. Concerning the success of free kicks, no significant impact was found neither for shots nor for crosses. In addition, no influence on the distance set by the referee could be identified.The results showed no decrease of free kicks with distance violations but a significantly lower extent of these violations  effects. Due to the lack of punishment, the authors raise concerns about the current application of the minimum distance rule. An increasing amount of innovative devices is introduced in several sports to support the umpires and referees respectively. In general these devices serve as another tool to ensure the legitimate outcome of sport competitions  effects of the innovation, which can be positive as well as negative. This is quiet surprising, as investigating the merit or worth of interventions, should be a main goal due to most of the common definitions of evaluation Scriven . The evaNo evaluation research was run for the use of the vanishing spray in soccer yet, so the aim of this study is to overcome this lack by investigating five hypotheses. In addition to those, the using patterns of the device were described. The spray was introduced to help the referee to enforce the minimum distance rule and stemming discussions about the spot of the ball or wall respectively. Thus, the first hypothesis to be tested is that the spray leads to fewer violations (H 1.1)—or respectively a lower extent of violations (H 1.2)—of the required minimum distance by players of the defending team. According to the official laws of the game of the Deutscher Fußball Bund , these violations should be punished with a yellow card and a repetition of the free kick. Taken this into account, it was checked if the spray affects fewer warnings and less free kicks that have to be retaken, to provide further information concerning the patterns of rule violations. Without reference of these targets, but as the innovation created more awareness of the required distance, a third hypothesis, that the spray reduces estimation errors of the set distance between the spot of the free kick and the wall, was investigated (H 2). In addition, we assumed that no violations of the required distance benefit the kicking team, resulting in a higher success rate for free kicks taken as crosses (H 3.1) as well as for those that were taken as shots (H 3.2).The data set consisted of all free kicks of the 2013/2014 and 2014/2015 season that were taken as shots or crosses with a distance less than 35.0 m of the goal line. By signing a contract of employment as a professional soccer player in the German Bundesliga, each players sings a statement of consent to being monitored during matches. The provided data included a match ID, the teams involved, event time, two dimensional coordinates of the spot of the free kick and whether it was taken as a shot or cross. Using a self-deigned observational system two specific trained experts collected data for further variables that are shown in Table The distance between the spot of the ball and the wall was obtained with a custom made analysis software. Using homography, this software enables the user to determine points on the field by transforming video coordinates into real world coordinates. Therefore this software requires not just positional data of a match, but also the respective specific tracking video. These data and videos respectively were only available to the authors for a subsample for which the distance set by the referee could be obtained. The respective variable |DistanceError| shows the absolute difference to the regulatory 9.15 m.The examination of the inter-rater-reliability showed excellent scores for most of the variables. Cohen’s Kappa reached a value of 1.00 for the use of the spray, .91 for the punishment of violations and 1.00 for the result of shots. The results of crosses and the identification of rule violations felt behind with .79 and .80 respectively but were still acceptable. The correlation coefficient (.98) as well as the relative observed agreement (92.6 %) for the numbers of players in the wall also showed a good agreement between the observers.Spray/NNoSpray). 81 pairs of free kicks of these two groups represented the subsample for the investigation of the set distance by the referee.To investigate the influence of the vanishing spray, sprayed free kicks after the introduction were compared to those before, based on the idea of evaluating interventions by comparing respective variables on different time points of a program Cronbach . Thus, tt test was calculated for the set distance between the ball and the wall. On the other hand, Chi square tests for the violations, the punishment of these violations, retakes and the success of free kicks. All statistical analysis were performed with SPSS , except the respective effect sizes Cohen’s d and Cramér’s V that were calculated manually. The magnitudes of the effect sizes were evaluated based on the limits: .10 , .30 (medium) and .50 (large) for Cramér’s V .The introduction of the vanishing spray showed no significant influence on violations of the required minimum distance by players of the defending team, as the share of free kicks with violations remains on a similar level . However, the share of massive violations decreases significantly by 6 %, representing a trivial effect size but an odds ratio of .60 (95 % CI .36–.99). Despite these violations of the required minimum distance, none of the free kicks of the sample was retaken and no yellow cards were awarded for this reason. Six verbal cautions for such violations were recorded for the treatment group, showing a significant increase with small effect size compared to the zero verbal cautions of the control group . The average absolute difference from 9.15 m, the prescribed distance by the laws of the game, showed a non-significant mean difference of −5.12 cm (CI −23.2 to 12.9). Overall, 50.6 % of the estimations are within a |DistanceError| of just .5 ms, 38.6 % showed an absolute error of .5 to 1.5 m and 10.8 % were off by more than 1.5 m.The comparison of the set distance between the ball and the wall by the referee showed no influence of the vanishing spray in any direction. The average distance was 9.24 m for each group  and more shots in the category “Missed” . The number of shots that were blocked by players standing in the wall decreased slightly by .3 % . The success of free kicks that were taken as crosses was just .65 as high after the introduction of the vanishing spray 95 % CI .29–1.46). As stated in Table 9–1.46. AThe aim of this study was the evaluation of the new vanishing spray in soccer, which is used for 42.5 % of the respective free kicks, especially for those in promising positions. Therefore, five hypotheses were tested of which just one, the lower extent of rule violations, could be supported. The four other hypotheses concerning the amount of violations, the distance set by the referee and the success of either crosses or shots were not supported by the results. After separately discussing the findings regarding each hypothesis, the authors point out the significance of these findings and null-findings respectively for the evaluation research process itself as well as the understanding of the underlying phenomena.A reduction in violations of the minimum distance, could not be proofed by the share of free kicks with violations. One out of four free kicks is still affected by such an incident. Nevertheless, the number of massive violation showed a significant decrease. Even though the respective effect size is just .09, the risk ratio shows a 1.56 times higher risk for massive violations for the control group. Thus, the spray had a positive effect on the extent of violations, and the main goal of the introduction was at least reached in curbing such incidences. In addition it is worthy to point out a limitation of this study. For sprayed free kicks, the observers could use the marked line as a reference, which (obviously) wasn’t possible for other free kicks. Consequently, for the latter category, the observers were instructed to note only violations if they can see the defenders moving towards the ball. Thus, if the wall reduced the distance while weren’t covered by the broadcast, this couldn’t be identified by the observers. This step was necessary to reach a good inter-rater agreement, but likely affected an underestimation of violations for free kicks without spray due to methodological reasons.Despite this frequent incidence, the results also showed that violations of the distance rule were punished neither before nor after the introduction of the vanishing spray. Just a few verbal cautions were awarded after the introduction, but these verbal cautions not fulfilling the required extent of punishment. Two reasons could lead to this lack of punishment for rule violations. The first reason simply is that the referee can’t identify the violation. Since a free kick is a situation that has a comparatively clear structure in which the referees can focus more specific, the authors assume that the referees should be able to identify violations of the required minimum distance. In addition it should be even more possible for the referees with a marked line on the field, which now is available after introduction of the vanishing spray. Thus it is possible that some kind of unwritten rules come into play. According to D’Agostino , unwrittFor positioning the wall the referees have to estimate the distance to the ball without any external aids. In more than 50 % of the free kicks the referees were within a distance of half a meter, but also a decent amount of errors of more than 1.5 meter occurred. We think the mean absolute error of .70 m before the introduction was still acceptable, especially considering that the mean distance to the ball is just 9 cm to high. The vanishing spray can’t support this process directly but led to more awareness of the distance in general. However, this didn’t affect the quality of estimation by the referees as there was no significant decrease in the absolute error.Round about a third of the shots in this study were on goal, slightly more than one out of four is blocked by the wall and the remaining round about 40 % missed the target or were blocked by another player outside the wall. The results are pretty similar to previous findings of Carling, Williams and Reilly . In an iThe overall rate of successful free kicks taken as crosses was short to 23 %. To compare the values to those of previous studies, it is necessary to consider that we used a different definition of “successful”. Casal et al.  for instThe use of vanishing spray should benefit the offensive team. Despite a slight increase of the goal rate, this could not be proved with the results of this study. No significant differences were found for the success of free kicks, neither for free kicks taken as shots nor for crosses. The rate of successful crosses even decreased by 7.5 % which, however, wasn’t a significant difference. Comparing the free kicks with massive violations of the distance rule indicate that this violations don’t affect the outcome of the free kick, but there wasn’t a large enough sample to run reliable analysis. Another reason could be that the curbing of rule violations is too small to create a significant benefit for the kicking team, especially considering the variations in the distance set by the referee.Summed up, the introduction of the vanishing spray basically fulfilled its main goal, by reducing the extent of violations of the minimum distance rule, but didn’t lead to further positive effects. Especially for the purpose of evaluation research, the respective null findings are as valuable as other findings to estimate the worth of an intervention. Rather spoken, this worth is estimated by the respective stakeholders, which must thoroughly consider all results of an evaluation for the decision making process (e.g. Stufflebeam In addition, evaluation researches create new knowledge about the underlying phenomena (Stufflebeam and Shinkfield The findings of this study point out the importance of evaluating innovative devices that support the referees in game sports. Based on the objective targets of the respective association the authors showed that the vanishing spray fulfilled its main goal by decreasing the extent of violations at least in some extent. But in addition to that, these evaluations not just generate feedback concerning the new device but also on the underlying phenomena. In this study, the results also indicate a lack of application for the distance rule. Despite a frequent incidence of violations of the minimum distance, none of the investigated free kicks was retaken neither a yellow card was awarded due to this reason. The authors suggest two ways to overcome this discrepancy. On the one hand, the associations could try to increase the awareness of the appropriate punishment of violations of the distance rule. On the other hand, the rule itself could be adapted by changing the extent for the punishment.


An important objective in the development of targeted therapies is to identify the populations where the treatment under consideration has positive benefit risk balance. We consider pivotal clinical trials, where the efficacy of a treatment is tested in an overall population and/or in a pre-specified subpopulation. Based on a decision theoretic framework we derive optimized trial designs by maximizing utility functions. Features to be optimized include the sample size and the population in which the trial is performed (the full population or the targeted subgroup only) as well as the underlying multiple test procedure. The approach accounts for prior knowledge of the efficacy of the drug in the considered populations using a two dimensional prior distribution. The considered utility functions account for the costs of the clinical trial as well as the expected benefit when demonstrating efficacy in the different subpopulations. We model utility functions from a sponsor’s as well as from a public health perspective, reflecting actual civil interests. Examples of optimized trial designs obtained by numerical optimization are presented for both perspectives. In the development of targeted therapies the investigation of potentially predictive biomarkers is critical. If efficacy is limited to an identifiable subgroup of patients, developing a therapy for an unselected patient population is ethically problematic and will also require unnecessarily large sample sizes because of a diluted treatment effect. On the other hand, erroneously restricting a drug development program to a subpopulation is also unethical, as it excludes patients from an effective treatment. Furthermore, it will entail a financial loss for the sponsor because of unnecessary costs of biomarker development and screening and the lower prevalence of the future patient population.Several one and two stage clinical trial designs have been proposed in which the treatment effect is tested in an overall population as well as in a subgroup of biomarker positive patients –6  and biomarker negative (S′) patients and there is prior evidence suggesting that the treatment effect may be more pronounced (or only present) in the biomarker positive group. Let λS and λS′, satisfying λS + λS′ = 1, be the prevalences of biomarker positive and biomarker negative patients in the full population. For this situation we consider three design options for a pivotal clinical trial: (i) The classical design that does not account for the biomarker status and tests for a treatment effect in the full population only, (ii) the stratified design that also recruits patients from the full population but where the biomarker status of each patient is determined and the treatment effect is tested in the full population and the subpopulation, and, (iii) the enrichment design, where patients are screened for the biomarker status and only biomarker positive patients are included in the trial.Consider a setting where a single potentially predictive binary biomarker has been identified in advance, separating the full population The choice of trial design will in general not only be based on power arguments, but on the overall expected utility of different designs, accounting for the potential rewards and costs. Rewards can be quantified by the sales revenue, from a sponsor’s view, or by a measure of the overall health benefit, from a public health view. The costs of the trial are determined by fixed and per patient costs as well as investments in biomarker development and the determination of the biomarker status for the patients in the trial. Based on a decision theoretic framework, we first optimize each of the three trial designs by choosing optimal sample sizes (and an optimized multiple testing procedure for the stratified design). Then, the optimal design can be selected among the three optimised designs based on their expected utilities. The optimal design choice depends on the type of utility function used , the reward and cost parameters, the prior distribution on the effect sizes and the prevalence of the biomarker positive subgroup.Δ =  denote the treatment effects for the primary efficacy endpoint in the subgroup and its complement, respectively. Furthermore, let π(Δ) denote a prior distribution on Δ. We focus on priors that satisfy π(Δ) = 0 for δS < δS′. This accounts for settings where there is some evidence that the effect size in the biomarker positive treatment group may be larger than in the biomarker negative group but not the other way around.Let S or F is the demonstration of a significant treatment effect in the respective population by a suitable multiple testing procedure controlling the familywise error rate (FWER) at level α in the strong sense. Consider the two null hypothesesδF = λSδS + λS′δS′, and let, for some trial design d, ψd =  denote a multiple testing procedure such that ψi,d = 1 (0) if there is a statistically significant (no significant) treatment effect in population i = S, F.For simplicity, we assume that the basis of marketing authorization is a single pivotal trial. We further assume that a necessary condition for regulators to approve a drug for the populations We consider three types of trial designs, the classical, the stratified and the enrichment design. Let Classical designCn refers to a classical parallel group design with per group sample size n recruiting patients from the full population and testing HF only. HF is tested by a non-stratified test Stratified DesignHF and HS are tested with a weighted multiple testing procedure with parameter αS. As multiple testing procedure we apply the closed Spiessens-Debois’ test [HS ∩ HF with the closed testing principle so as to obtain a test for the rejection of either HS or HF (or both). Let pS and pF denote unadjusted p-values for testing HS and HF, respectively. Here we assume that HF is tested with a test stratified for the biomarker . For αS, αF ≥ 0, the closed Spiessens-Debois’ test then rejects HS if pS ≤ α and either pS < αS or pF < αF. Similarly, it rejects HF if pF ≤ α and either pS < αS or pF < αF. To control the familywise error rate at level α in the strong sense, the significance levels αS and αF need to satisfyαF is determined by αS ≤ α is given. Note that the corresponding function αF(αS) depends on the subgroup prevalence λS.is’ test , 13. ThiF is not only determined by the treatment effect in F, but that regulators additionally require some evidence that there is a treatment effect in both S and S′, so that the rejection of HF is not completely dominated by a treatment effect in a single subgroup only. Thus, we assume that the regulators’ decision rule corresponds to a hypothesis test where HF is only rejected, if, in addition, the p-values pS and pS′ of tests for efficacy in the two subgroups fall below corresponding thresholds τS and τS′. The resulting modified Spiessens-Debois’ test , which is defined as the reward  minus the trial costs. We model the sponsor’s reward as a function of (i) the outcome of the regulatory approval process, (ii) the price the sponsor can achieve, and (iii) the size of the population the drug is licensed for.i = S, S′ that specify the reward obtained in the respective population. The reward function may depend on the observed data, the design of the pivotal trial d and the prevalence of the subgroup. We model the reward as the product of the price of the drug for the treatment of a single patient times the market size. Given an overall market size N, the market sizes of the two subgroups are λSN and λS′N, respectively. Furthermore, we assume that the payers are willing to pay more if a larger treatment effect was observed. If the drug is authorized for neither subgroup, both reward functions are set to zero. If the drug is authorized for the subgroup S only, the reward for the complement S′ is set to zero. If the drug is authorized for the full population, we assume that the same price is charged in both subgroups.To model (i) and (ii) we define reward functions S and S′ areμi denotes a minimal clinically relevant effect size for population i = S, F and (⋅)+ denotes the positive part. δS and δF obtained from the trial data. The constants ri for i = S, F are the marginal prices (the change in price if the observed effect size increases by one unit) and N denotes the total market size, which for the sponsor is defined as the number of future patients within the patent life of the therapy in the unselected, full population. Note that, given that efficacy is shown in the full population, a common treatment effect estimate We assume that  the price increases linearly with the observed effect size. Then the reward functions for the subgroups S and S′ are given byWith the public health utility function we model the utility of trial designs under the assumption that the drug is developed by public health authorities. Therefore, the utility of a trial is given by the total health benefit to the society (adjusted by the production cost of the drug) minus the cost of running the trial. We assume that the benefit of the drug is measured on a monetary scale representing the expected, accumulated (over the whole treated population) treatment effect. Costs are assumed to be the same as under the sponsor view. The reward functions for the subgroups ri for i = S, F are the marginal benefits (the change in benefit if the effect size increases by one unit), and N denotes the size of the future (unselected) patient population. Note that the benefit depends on the actual effect sizes δi and not on the corresponding trial estimates The first term in the utility function π denotes a prior distribution on the effect sizes Δ. For a given utility function Uv) and with probability λS′ distributed as N. If the biomarker is either prognostic  or predictive (such that δS ≠ δS′) the overall treatment effect estimate θi,S − θi,S′|, i = T,C. For simplicity, in the numerical examples we assume that the biomarker is predictive only but not prognostic . For the stratified design, we assume that the subgroup estimates, λSn (resp. λS′n) observations per group from the subgroups S and S′. However, if patients are not selected for the trial based on biomarker status, the number of subjects from each subgroup is binomially distributed, though, for large n, the random sample sizes have only little impact and the approximation becomes accurate. Therefore, in the numerical investigations, we introduced a minimal sample size of nmin = 50 patients per treatment arm. For the contour plot (in Subsection 4.1.3) optimization was performed by evaluating the objective function for a grid of candidate sample sizes . For the optimizations in the other plots, a further optimization step was applied by optimizing the objective functions with the R Version 3.2.4 procedure optim [To be able to compute the expected utilities by numerical integration and not to have to rely on simulations, we approximated the sampling distributions for both the classical and the stratified designs by normal distributions . For the classical design, each subject recruited to the trial belongs to re optim  using grα = 0.025 and the consistency thresholds in the multiple test for the stratified design to τS, τS′ = 0.3. We consider discrete prior distributions πδS,i,δS′,i on a grid ,i = 1, …, l of effect sizes and specify two priors corresponding to scenarios where there is either only weak or strong prior evidence that the biomarker is predictive. The prior distributions used in the examples are defined in δ. In the examples below we set δ = 0.3 with the exception of Subsection 4.1.3 where optimal designs for other choices of δ are explored.The one-sided significance level is set to The reward and cost parameters in the sponsor and the public health utility function are specified via the following three cases:Case 1 Corresponds to a large market, where the biomarker costs are negligible, i.e. NrS = NrF = 10,000 Million US Dollars (MUSD) per unit of efficacy and cscreening = cbiomarker = 0.Case 2 Corresponds to a small market, where the biomarker costs are still negligible, i.e. NrF = NrS = 1000 MUSD per unit of efficacy.Case 3 We add biomarker and screening costs, cscreening = 5000 USD per patient and cbiomarker = 10 MUSD. The reward parameters NrS and NrF are the same as in Case 2.cper-patient = 0.05 MUSD and csetup = 1 MUSD. Note that the setup costs are assumed to be the same for the enrichment, classical and stratified design and therefore have no impact on the order of their expected utilities. However, they do have an impact on the sign of the utility, and thus whether any trial design is superior to no trial at all. In the reward functions Eqs μS = μF = 0.1, which is a third of the effect size δ = 0.3 used in the prior distributions in Section 4.1.1 and 4.1.2.For all three cases we choose We discuss the optimal designs for the weak and the strong biomarker prior and the three cases specifying the cost and reward parameters.Large market, no biomarker costs (Case 1) The optimized utilities and corresponding optimal classical, stratified and enriched designs are shown in Optimal utility. For the sponsor utility function, the stratified design has the largest expected utility, except for low prevalences where the classical design is optimal. The latter is on first sight surprising, because in Case 1 we assume no biomarker costs. However, in the stratified design , to show efficacy in the full population, we require that pS and pS′ do not exceed τS = τS′ = 0.3 (in addition to rejection of HF in the multiple testing procedure). Thus, for low prevalences the sample size of the stratified design needs to be substantially increased to reach a sufficient power to show efficacy in F and therefore its expected utility is lower.λS the expected utility of the classical design is almost identical to that of the stratified design. This holds because the power to reject HS in the optimized stratified design approaches the power to reject HF in the classical design and the rewards obtained for authorization in populations S and F are similar. Why is the stratified design for the sponsor view still optimal in this case? This results from the fact that the size of the reward in the sponsor view depends on the observed rather than the true treatment effect: for trial outcomes where HF can be rejected in the classical design but, due to the variability of estimates, S may become larger than the reward in F. However, while the classical design leads to rejection of HF in such cases, the stratified design rejects HS and not HF because of the consistency threshold.For the public health utility function we observe a similar pattern. However, for large Optimal sample size. Overall, the optimized sample sizes for the public health utility function are larger than for the sponsor utility function. They are lowest for the enrichment design, and—for smaller prevalences—largest for the stratified design. For the latter, the sample size increases sharply for low prevalences. This is due to the fact that a sufficient sample size in the subgroup is required to achieve adequate power for the rejection of both HS and HF (for the latter due to the consistency threshold τS). Furthermore, the relationship of the optimal sample size and the prevalence is qualitatively different for the three designs. For both utility functions the optimal sample size is increasing in the prevalence for the enrichment design (because the gain when demonstrating efficacy in S increases), decreasing for the stratified design  and non-monotone for the classical design .Significance levels. In the intersection hypothesis test of the optimal multiple testing procedure in the stratified design αS is larger than αF for almost all prevalences. To make up for the lower sample sizes in the subgroup, the optimal design uses a larger αS than αF. For increasing prevalences, the correlation of the test statistics used to test HS and HF increases such that less multiplicity correction is required and both αS and αF increase.Power. We define the power corresponding to a specific trial design as the overall probability (averaged over the prior) of regulatory approval in any population. This is a slight generalization of the traditional concept of power, which in the current context may be defined as the probability of regulatory approval conditional on a specific pair of subgroup effects. The power obtained by averaging over a prior has also been referred to as assurance [ssurance . The curHF is largest for intermediate values of λS and much lower than for the classical design if λS is large (even though the optimized sample sizes are similar in this case). This is due to the application of the consistency thresholds which are a more difficult to meet if one of the subgroups S or S′ is small.Note that for the stratified design, the probability to obtain marketing authorization in Small market, no biomarker costs (Case 2) Case 2 differs from Case 1 only in that the rewards NrF and NrS are reduced by a factor 10. Because of the lower rewards the optimized expected utilities are smaller compared to Case 1  are substantially smaller than in Case 1. Overall, the expected utilities follow a similar pattern as in Case 1 but the range of prevalences where the classical design has a higher expected utility is larger than in Case 1 for both the sponsor and the public health utility functions. The assumption of a smaller market qualitatively changes the optimized sample size of the stratified designs as a function of the prevalence. For low prevalences the optimized sample size is much lower than in Case 1: because the reward is lower, it does not pay off to invest in a large overall sample size to meet the threshold Small market with biomarker costs (Case 3) Note that the addition of biomarker costs has no impact on the expected utility of the optimal classical design (as it does not require the biomarker). However, the expected utilities of the enrichment and the stratified design become smaller compared to Case 2 because of the additional costs. Therefore, the classical design now dominates the stratified design for a broader range of  values of λS and the stratified design becomes optimal only for larger values of λS , this is compensated by the lower costs because no biomarker is required. In the sponsor view in contrast, the difference between the expected rewards of the stratified and the classical design is larger because it is determined by observed treatment effects . Therefore, the stratified design dominates also for large values of λS. Moreover, the biomarker costs lead to a reduction in sample size compared to Case 2.S is identical in both.First, note that the expected utility and optimal sample size of the enrichment design is the same for the weak and the strong biomarker prior because the prior distribution on the treatment effect in Large market, no biomarker costs (Case 1) For the sponsor’s utility function the stratified design is still optimal, with the exception of very low prevalences  While, as in Case 1, the stratified design is optimal for the sponsor view for all but very low prevalences, the difference between the expected utilities of the stratified and enrichment design is small.In contrast, for the public health utility function the enrichment design achieves the highest expected utility see . FurtherHF for low and on HS for large prevalences. This holds for both the public health and the sponsor utility function.Note that the optimal test in the stratified design gives most weight on Small market, biomarker costs (Case 3) The pattern is very similar to Case 2, however, the range of λS values where the classical (for the sponsor utility) or no trial  are optimal becomes larger . For larger effect sizes the classical design is optimal for very low and very large prevalences and the stratified design otherwise.For the public health view in contrast, for very low effect sizes, the optimal decision is to perform no trial at all. Under the weak biomarker prior, the enrichment design is optimal under the public health view only in the scenarios without biomarker costs, for small δ, the public health utility is optimized by the enrichment design . For larger δ the stratified design is optimal, again with the exception of very low prevalences. In addition, in the scenario with biomarker costs the classical design becomes optimal for large prevalences.Under the strong biomarker prior and intermediate The current study suggests decision-theoretic models for optimizing confirmatory biomarker trials, both from a sponsor and a public health perspective. Furthermore, it explores the potential discrepancies between the two perspectives.rF, rS in both utility functions). This finding is likely due to the fact that sponsor benefit is based on the estimate of the benefit in the trial, whereas the public health benefit depends on the actual benefit. Thus the public health perspective implies a higher standard for the evidence. This finding provides a quantitative basis for the qualitative observation that health authorities tend to require a higher standard of evidence than desired by some sponsors. Mechanism design theory could potentially be applied to try to create mechanisms which align the incentives more completely.The optimized designs depend sensitively on the particular configuration of parameter values. Besides the priors on the effect sizes, the assumptions on the market size and costs have a substantial impact on the optimized designs. Therefore, formulating simple rules of thumb for trial designs is hardly feasible. However, a few general observations can be made. The optimized sample sizes for the public health utility function are consistently larger than for the sponsor utility  gives a positive NPV in the sponsor view .In the public health view the enrichment designs are optimal for a range of scenarios. Especially, if there is sufficient prior medical understanding that the biomarker negative subpopulation is unlikely to be positively affected by the drug, it can be a waste of resources to conduct the trial in this population. Ethical considerations reinforce this, as it can be argued that genuine informed consent  implies When applying the presented framework to practical design decisions, the different model components should be scrutinized. In the numerical example we have assumed for simplicity that the biomarker is not prognostic but in practice this will often not be the case. If the biomarker is also prognostic, the variability of the effect size estimates will be increased with a consequent decrease in the expected utility of the classical design.N denotes the number of patients, determined by the patent life, for which full payment will be received upon regulatory approval. On the other hand, for the public health authority, N denotes the total number of future patients. In an extended model, N could be fixed to always be the total number of future patients and a factor could be added next to N for the sponsor. This factor would then represent the fraction of patients corresponding to the patent life, and could be made to depend on the choice of trial design in various ways. For example, in the enrichment design we accounted only for the screening costs arising from the determination of the biomarker status of patients. However, if the restriction of the trial population leads to slower recruitment and consequently a later authorization of the drug, the result will be a reduction of the market size and the remaining patent life. This, in turn, may reduce the potential reward in different ways for the two perspectives. Another simplification made in our framework is the assumption of a zero discount rate for the sponsor. In practice, a commercial sponsor would use a non-zero rate to discount future revenues, which would lead to a further reduction of its expected utility as compared to a public health authority.As regards the market size for the sponsor, In the considered model we assumed that the subgroup prevalences in the trial are the same as in the total patient population. However, unless the recruitment is stratified by subgroup, the actual prevalence in the trial will be stochastic. Furthermore, the propensity to participate in the trial may vary between subpopulations. While our results are generally robust to random variations in the prevalence, varying propensities for trial participation may lead to a biased estimation of the effect size in the full population in the classical design . The question of generalizability of trial results to general patient populations is however not specific to the development of targeted therapies but a more general issue.μS and μF in the reward functions  parameters , 30. AnoS′. Therefore, our results could be seen to suggest that the investigation of rare diseases is not recommended in either perspective. Consequently, the question arises if research in rare diseases should receive special priority and be subsidised by society such that drug development occurs even though the expected utility to society is negative, or in some case weakly positive but less positive than other alternative expenditures. However, this argument raises ethical questions because the purely utilitarian viewpoint that underlies the decision theoretic framework does not account for other ethical principles as fairness and justice. Similar issues arise for small subgroups of common diseases, an increasing issue in cancer given the fact it being subdivided into many small molecular subclasses. In the case of cancer, increased benefit due to matching between molecular subgroups and targeted therapies may mollify this issue, but this remains to be seen in individual cases, so that the ethical and public policy dilemma may still be present.Finally, we note that the case of a very low prevalence, small market size and no biomarker costs mimics the situation of a rare disease, except there is no complementary subgroup Δ for the enrichment, the classical and the stratified design. The overall expected utilities are then obtained by integrating over the prior distribution.We derive the expected utilities for a given effect size Δ is given byϕ denotes the density and Φ denotes the cumulative distribution function of the standard normal distribution, For the enrichment design, In the classical design, S and S′ differ, it follows that the observations in the experimental treatment and control group follow a mixture distribution of two normal distributions. Therefore, the variance of Δ for the classical design is given byδF = λSδS + λS′δS′ and If the mean response in Δ is given byAF denote the region in AS be the region where ZS, ZS′ the z-statistics computed from the observations in S and S′ respectively. ThenAF and AS depend on the specific values of the parameters and the design variables . However, the regions may in all cases be described by means of a finite number of straight lines, implying that the expected values above can be computed using standard software for numerical quadrature in two dimensions. But since the integrands are linear in zS′ and ZS′ follows a normal distribution, one-dimensional integration may be carried out analytically in the zS′-direction before applying a numerical method. This leads to faster numerical evaluations, which is useful when investigating how the optimal solution changes over the parameter space.The utility of the stratified design is given byΔ may be written asFor the public health view the expected utility given S1 File(PDF)Click here for additional data file.


All children were examined by the same physician, and data were collected prospectively. We included in this analysis only children less than 3 years of age at the examination date, as the natural history of IH is to spontaneously involute during the first few years of life.A total of 244 patients with CMN under the age of 3 years were seen in this time period, with a mean and median age of 0·78 years and 0·53 years, respectively. Of these, 142 were females, giving the same male : female ratio of 1 : 1·4 as has previously been reported for our CMN cohort.This systematic study of the prevalence of IH in a cohort of patients with CMN has found no increase above that of the normal population, and a sex ratio in line with what we would expect for IH alone. This study does not support a connection at a genetic level between CMN and IH, either at germline predisposition or at somatic mutation level.


When reaching towards an object while standing, one’s hand responds very quickly to visual perturbations such as the target being displaced or the background moving. Such responses require postural adjustments. When the background moves, its motion might be attributed to self-motion in a stable world, and thereby induce compensatory postural adjustments that affect the hand. The changes in posture associated with a given hand movement response may, therefore, be different for the two types of perturbations. To see whether they are, we asked standing participants to move their hand in the sagittal direction away from their body to targets displayed on a horizontal screen in front of them. The target displacements and background motion were in the lateral direction. We found hand movement responses that were in line with earlier reports, with a latency that was slightly shorter for target displacements than for background motion, and that was independent of target displacement size or background motion speed. The trunk responded to both perturbations with a modest lateral sway. The two main findings were that the upper trunk responded even before the hand did so and that the head responded to background motion but hardly responded to target displacements. These findings suggest that postural adjustments associated with adjusting the hand movement precede the actual adjustments to the movement of the hand, while at the same time, participants try to keep their head stable on the basis of visual information. When we reach out for objects in daily life, we often do not only adjust the angles of the joints of our arm. Sometimes, adjustments to the joints in the leg and trunk contribute directly to reaching the object, such as when stretching out and standing on one’s toes to reach something on a high shelf. If there is no direct contribution to bringing the hand to the object, we refer to adjustments as postural adjustments. Postural adjustments are frequently required for the control of balance . This is, for instance, so when shifting one’s hips backward while leaning forward to reach a floating toy in a tub. In general, balance is challenged by moving the arm, while the efficiency and accuracy of the goal-directed arm movement are challenged by the postural requirements of maintaining balance  participated in the experiment. They had normal or corrected-to-normal vision. None of the participants had any disease that can affect motor or sensory function. The study was approved by the Research Ethics Committee of the Faculty of Behavioural and Movement Sciences, Vrije Universiteit Amsterdam. Written informed consent was obtained from each participant.A screen  was positioned horizontally on a table  with two cameras was used. One camera was behind the participant and one to his or her right. The sampling rate was 200 Hz. The posture was recorded with customised cluster markers: three markers attached rigidly to each other in a triangular configuration. Cluster markers were attached to the forehead, 3rd thoracic vertebra (referred to as ‘upper trunk’), 1st sacral vertebra (referred to as ‘lower trunk’) and the wrist (ulnar side). A single marker was attached to the nail of the index finger of the right hand. The latter was used to control the experiment on the basis of the movement of the finger.Each session started with a calibration procedure for determining the position of the index finger when in contact with given positions on the screen in Optotrak coordinates. After the calibration, participants started individual trials by moving their right index finger to the starting point. The starting point of each arm movement was a dot (radius: 1 cm) that was shown on the checkerboard-like background (square length: 7 cm). Once the index finger had been resting on the starting point for a random time between 0.6 and 1.2 s, the starting point disappeared and the target dot (radius: 1.5 cm) appeared. The participant was instructed to tap on the target as accurately and quickly as possible with the tip of the right index finger.While the participant moved towards the target, either target displacement or background motion could occur, triggered by the finger having moved 5 mm from the starting point on the screen. Due to delays in measuring the movement of the finger and rendering images on the screen, the actual perturbation was 60 ms after the finger crossed this threshold. If the target was hit , a sound indicated success. Otherwise, the target drifted away from where the finger touched the screen.The experiment consisted of nine conditions: four target jump conditions, four background motion conditions, and a no-perturbation condition. In the target jump conditions, the target jumped either 1 or 4 cm, leftwards or rightwards, across the stationary background. In the background motion conditions, the background moved either leftwards or rightwards at 2 or 6 cm/s, ‘below’ the stationary target. We chose perturbation sizes that would be likely to make the magnitude of the response of the hand comparable for target jump and background motion conditions. This was based on a pilot study, in which we varied the target jump amplitude and background velocity. There were 300 trials in total: 30 trials each in four target jump conditions and four background motion conditions, and 60 trials in the no-perturbation condition. All trials were presented in a completely random order. The participants practiced for about 20 trials (random conditions) before the start of the experiment. During the experiment, they could rest at any time they liked by not moving to the starting point.The 3D kinematic data of all markers were filtered using a second order low-pass Butterworth filter with a cut-off frequency of 30 Hz. We determined this cut-off frequency by determining the minimum variance in the distances between the three markers on a cluster  and the finger touching the screen again. We excluded trials (5%) for which the duration or the delay in presenting the perturbation was not within ± 3SD of the mean, or for which the moment of the perturbation could not be determined properly . Movements to the right and away from the body were considered positive.As the perturbations were always in the lateral direction, we only analysed responses in the lateral direction. The lateral velocity of the finger was calculated from the measured position data using the central difference algorithm. We defined time zero as the moment at which the perturbations actually happened, which was about 60 ms after the finger had been raised 5 mm from the screen. Responses were determined by comparing movements after rightward and leftward perturbations (differences between the movements in the direction of the perturbation were considered positive). Since this gives the sum of the responses in both directions, we divided the difference between the responses to rightward and leftward perturbations by two. The result is equivalent to a response to a rightward perturbation.We used the extrapolation method to determine the latency of responses to the perturbations: the latency was the time at which a line through the points at which the response reached 25% and 75% of the peak response intersected a baseline value Fig. . The basThe cluster markers not only allow us to determine the lateral motion, but also rotations. Although we do not have predictions for the rotations, the azimuthal rotation might be informative. Despite optimal filtering at 30 Hz, the filtered signal turned out to be quite noisy. Fast Fourier transformed data revealed a peak in the spectrum of rotations that was absent in the translations (possibly due to cluster vibration). To present interpretable data on rotational movement, we had to filter the orientations at 10 Hz. Since this additional filtering smooths the responses considerably, we did not analyse the orientation data quantitatively.Descriptive data are shown as means or means ± standard deviations (SD) across participants. The movement times were calculated for each trial, and then averaged across the four target (or background) conditions for each participant.The mean movement times were 382 ± 44 ms for unperturbed trials, 400 ± 44 ms for trials with target jumps and 396 ± 43 ms for trials with background motion. Participants hit the target in 95.0 ± 3.9% of the 300 trials. This means that participants were able to successfully follow the target jumps and compensate for their initial responses to background motion  did not affect the hand response latencies. The latencies of the lateral responses to the target jumps were shorter than those to the background motion for the finger (109 vs. 123 ms), as well as the wrist . Both the perturbations and the responses were in the medio-lateral direction. The upper trunk moved in the same direction as the perturbation, before the hand did so. The postural adjustments to the two types of perturbation were similar for all parts of the body that we measured, except the head. One limitation of the study is that we only use trunk and head displacements as a measure of anticipatory postural adjustments, although some muscle activations may control posture without leading to observable displacements. This limitation would have made the interpretation of a null-finding problematic, but does not hamper the interpretation of the clear anticipatory postural adjustments that we observed.yes, arm responses to target perturbations are preceded by anticipatory postural adjustments.We interpret the very early upper trunk response as an anticipatory postural adjustment. The reason is that it occurred about 33 ms before the response became apparent in the hand, and thus did not contribute directly to moving the hand. Both when the target jumped and when the background moved, the first response was in the upper trunk . Following this reasoning, the head motion induced by moving the background is likely to be a reaction to the motion of the background being attributed to self-motion (see introduction). Hence, there are different postural responses to target jumps and background motion, but there are also common components to the postural responses, which answers our second question.longer latency (Kadota and Gomi We found no effects of target jump size on hand response latencies, which is in line with earlier research (Brenner and Smeets The difference in latency suggests that the responses to the two types of perturbations rely on different pathways. The background-induced responses are likely to involve cortical pathways. Global motion, such as the background motion in our experiment, activates many visual areas (Palmisano et al. In conclusion, when visual perturbations occur during a hand movement while standing, there are postural adjustments that precede the manual adjustments. These postural adjustments are very similar, but not identical, when the target of the action is displaced and when the background moves. The main difference is to be found in the movement of the head.


Objectives: The stockpiling of neuraminidase inhibitor (NAI) antivirals as a defence against pandemic influenza is a significant public health policy decision that must be made despite a lack of conclusive evidence from randomised controlled trials regarding the effectiveness of NAIs on important clinical end points such as mortality. The objective of this study was to determine whether NAIs should be stockpiled for treatment of pandemic influenza on the basis of current evidence.Methods: A decision model for stockpiling was designed. Data on previous pandemic influenza epidemiology was combined with data on the effectiveness of NAIs in reducing mortality obtained from a recent individual participant meta-analysis using observational data. Evidence synthesis techniques and a bias modelling method for observational data were used to incorporate the evidence into the model. The stockpiling decision was modelled for adults (≥16 years old) and the United Kingdom was used as an example. The main outcome was the expected net benefits of stockpiling in monetary terms. Health benefits were estimated from deaths averted through stockpiling.Results: After adjusting for biases in the estimated effectiveness of NAIs, the expected net benefit of stockpiling in the baseline analysis was £444 million, assuming a willingness to pay of £20,000/QALY . The decision would therefore be to stockpile NAIs. There was a greater probability that the stockpile would not be utilised than utilised. However, the rare but catastrophic losses from a severe pandemic justified the decision to stockpile.Conclusions: Taking into account the available epidemiological data and evidence of effectiveness of NAIs in reducing mortality, including potential biases, a decision maker should stockpile anti-influenza medication in keeping with the postulated decision rule. The epidemiology of only a small number of influenza pandemics has been well studied and evidence for the effectiveness of remedial influenza treatments in a pandemic scenario is scant. Yet, governments around the world still have to decide whether or not to stockpile anti-influenza medication like neuraminidase inhibitor (NAI) antivirals, such as oseltamivir (Tamiflu®) and zanamivir (Relenza®), as a defence against pandemic influenza.Like many other potentially catastrophic events for which governments need to prepare, influenza pandemics are rare. Although the risk is considered to be 3–4% per annum3. However, taking a default position of not stockpiling, or making the decision on the basis of intuition alone, is not justifiable given the rare but potentially catastrophic losses associated with pandemic influenza and the large cost of stockpiling.The stockpiling of NAIs has been a controversial issue. Firstly, stockpiling may be seen to be a waste of large amounts of public money if the pandemic fails to materialise or if it is mild. In the United Kingdom, the previous Chief Medical Officer was criticised for spending £560 million on medicine that went largely unused in the 2009–10 pandemic5. A caveat of these studies, which were required for licensure of drug in healthy adults, is that they were not powered to determine low frequency but critical end points such as mortality in a largely healthy adult population. A further meta-analysis of observational data from pandemic influenza did find evidence of a reduction in the risk of mortality when NAIs were given to patients hospitalised with influenza6. Some authors have criticised it for being subject to a large degree of bias and rejected it as a suitable form of evidence with which to formulate policy decisions8, though others argue that this evidence strongly supports the use of NAI treatment for influenza in hospitalised patients9.Secondly, there has been a lack of conclusive evidence on the effectiveness of NAIs. Recent meta-analyses of randomised controlled trials (RCT) of seasonal influenza cases demonstrated reductions in rates of hospitalization, lower respiratory complications, and a decreased time to symptom alleviation but were unable to confirm or refute an effect of NAIs on more important clinical end points such as mortality6. The treatment has a plausible rationale and it worksin vitro and in animal models for this zoonosis10. An arguably extreme position is to assume that these observations contain no information regarding effectiveness in preventing the rarer, but more severe outcomes, such as death. People who take to heart Bradford Hill’s list of factors that should affect the interpretation of data  × QALY × λFirstly, the number of pandemic influenza deaths was calculated by multiplying the number of adults in the UK (Pop) by: the probability of there being a pandemic within the stockpile shelf-life (Prop), the clinical attack rate (CAR), and the case fatality ratio (CFR). We further multiply by the probability a pandemic influenza death occurred in hospital , and the probability one of these patients receives NAIs (Treated). The number of deaths averted by NAI treatment in this population of NAI-treated adults was given by the relative risk reduction in mortality associated with NAI treatment (1 – θ). Finally, the value of these deaths averted was calculated by multiplying by the quality adjusted life years (QALY) associated with each pandemic influenza mortality (QALY), and the societal willingness to pay per QALY (λ). This model is further explicated in6 is less certain in children and to determine if the decision to stockpile could be justified on the basis of any benefit among adults alone.We considered reductions in mortality among symptomatic adults resulting from stockpiling, but did not take into account possible additional effects on complications such as pneumonia or that community use might reduce complications, hospitalisation, or mortality. Only adults were considered on the grounds that NAI effectivenessDecision modelling is founded in the Bayesian paradigm, which was used to evaluate the stockpiling decision for a future pandemic with unknown epidemiological variables and unknown effectiveness of NAI. A sub-model was specified for each epidemiological variable in the decision model. Data from previous pandemics were assumed to be observations from an underlying common distribution, the parameters of which were estimated using these data as described in the following section. The decision was then evaluated over posterior predictive distributions for the epidemiological parameters. We used a bias corrected effectiveness estimate for the effectiveness of NAIs as described below. The model was estimated using Markov Chain Monte Carlo (MCMC) with 10,000 iterations using R 3.2.3 and Stan 2.11.0. This method obviates the need to conduct separate probabilistic sensitivity analyses since the posterior distribution of the net benefits represents the uncertainty about future influenza pandemics and NAI effectiveness. The expected net benefits represent the gains or losses from stockpiling, on average, given the different distributions for the different parameters. Convergence of the MCMC chains was assessed by visual inspection of autocorrelation, running mean, and trace plots in R.The data and statistical code are provided with the paper.1. The shelf-life of oseltamivir, the principle drug comprising the vast majority of the NAI stockpile, is ten years23.The data used to estimate the parameters in the model were obtained from documents compiled to assess pandemic influenza and thus represent the decision maker’s prior knowledgeThe clinical attack rate and case fatality ratios from previous pandemics were assumed to be observations from beta distributions. Improper non-informative priors with a lower limit of zero were assigned to the parameters of these distributions, which were then updated with the data from the previous pandemics. We excluded the observation of a clinical attack rate of 60% in the 1889–92 Asiatic flu pandemic as the UK government’s worst case scenario is a clinical attack rate of 50%. The probability that a pandemic occurs in the shelf life of the stockpile was similarly estimated from the data with each decade between 1900 and 2010 as a binary observation equal to one if a pandemic occurred in that decade and zero otherwise. These binary observations were assumed to be observations from a Bernoulli distribution.4. We based our effectiveness estimate on a recently published pooled meta-analysis of observational, patient-level data from hospitalised pandemic influenza virus patients6. We converted the odds ratios (OR) for mortality associated with NAIs (irrespective of time from onset) provided in the paper into relative risks (RR):RR =OR/(1 –p + (p ×OR)) wherep is the baseline (approximately 10%)24. The study was based on hospitalised patients, in order to apply the observed relative risk from hospitalised patients to the general population considered here, we made two conservative assumptions. First, we assumed that there would be no difference in the patients that would be hospitalised and those that would remain in the community in a no stockpile and stockpile scenarios. This is conservative because community treatment will be given earlier, on average, in the course of the disease if it can be administered in the community and there is evidence that the earlier the treatment is given, the better6. Secondly, we assume that only deaths occurring in hospital in the non-stockpile scenario would be averted under the counterfactual stockpile scenario. A study of mortality in the A/H1N1 2009 pandemic in England, found that 92% of deaths (125 of 136 cases studied) occurred in hospital25. Assuming that none of these 8% of deaths taking place in a non-stockpile scenario would be averted under the counterfactual is as conservative as it can be. The logic of our approach is laid out inNo RCT evidence for the effectiveness of NAIs in reducing the risk of mortality in pandemic influenza was available. Too few deaths were observed in RCTs of seasonal influenza8, although others dispute the strength of these criticisms9. We used a method previously published elsewhere to model bias26. Five reviewers  who were not associated with the observational data study independently completed a bias questionnaire and provided their beliefs about both additive and proportional bias present in the study across a range of domains. The reviewers were selected on the basis of their experience with observational data research and its associated biases, with expertise in health care and public health research. The median values for the mean and standard error of the bias across reviewers were used to ‘correct’ the observational evidence26. The method for bias modelling used here was originally intended for individual studies so that they could be adjusted prior to an evidence synthesis26. This method has been applied here since the study in question is an individual patient pooled meta-analysis, analysed using a similar method to that any single study would use, except that the data originate from multiple locations and are of varying quality. The reviewers considered this an additional source of uncertainty when evaluating the quality and potential for bias.In addition to these conservative assumptions regarding the application of in-hospital relative risk reductions to a community population, we also took into account the observational nature of the hospital based evidence itself. A number of authors have raised this issue in connection with the study used here28, no data were available from the 1889–92 pandemic. To estimate QALYs lost due to an influenza death, the remaining life expectancy was calculated by differencing the average age at death from the UK life expectancy at birth (i.e. 81.5 years)24. These years were weighted by the average QALY weight for a person aged over 45 of 0.825, and then discounted at the rate of 3.5% per annum as recommended by the National Institute for Health and Care Excellence (NICE)29.The distribution for the average age associated with an influenza death in previous pandemics was assumed to be drawn from a scaled Beta distribution with an upper limit of 81.5, which is the UK life expectancy at birth. The parameters of this distribution were then estimated from data; the average ages of influenza deaths from prior pandemics were 27 (1918), 65 (1957), 62 (1968), and 45 (2009)25. We further considered a number of scenarios for the distribution of NAIs and the proportion of symptomatic pandemic influenza cases that would receive the drug. Our base case was 100%, however we also considered the decisions that would be made in the range of 0% to 100% in a deterministic sensitivity analysis – the value of the deaths averted was multiplied by a number between zero and one. The cost of stockpiling was assumed fixed at £560 million  and was based on the figures quoted in the above mentioned Select Committee hearings3. We considered the adult population of the UK, which was 50.5 million in 201530. The willingness to pay per QALY was selected as £20,000/QALY  for the base case analysis, the lower end of the range  specified by NICE as being cost-effective29. We examined the decision that would be made under a range of willingness to pay per QALY values of £5,000/QALY  to £30,000/QALY .We also estimated the probability a pandemic influenza death occurred in hospital using data on 2009 pandemic influenza deathsThe observed relative risk was 0.83  and the bias corrected relative risk was estimated as 0.89 . The principle sources of bias identified by the reviewers were selection bias, due to a lack of randomisation, the possibility that studies with a positive finding may have been more likely to volunteer their data for the meta-analysis, and attrition bias. Not all reviewers were in agreement about the overall effects of bias, but the median response was that there was an overestimation of treatment benefit.Raw data of ‘stockpiling neuraminidase inhibitors for pandemic influenza usage’Click here for additional data file.Copyright: © 2017 Watson SI et al.2017Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).6, and in the United Kingdom specifically this proportion was 75%31, suggesting that 56% is achievable, and that therefore, stockpiling is supported by the available evidence.This study has found that the available evidence suggests that stockpiling NAIs for pandemic influenza is rational under a range of assumptions. Many of these assumptions are conservative, such as no reduction in adverse clinical outcomes other than mortality, no benefit in patients who would not have been hospitalised had there been no stockpile, and no effect in children. However, this decision required at least 56% of the influenza patients who would have died without a stockpile to receive NAIs if the threshold willingness to pay was £20,000/QALY. In the 2009 pandemic, 64% of hospitalised patients received NAIs32, these patients would require far fewer doses than the 1.16 million courses (at a minimum) of NAIs dispensed in the 2009 pandemic33. For a population of 50.5 million adults with a CAR of 25%, a hospitalisation probability of 0.5% would lead to only approximately 60,000 admissions. The evidence also suggests that more timely treatment of NAIs (within two days of symptom onset) is more effective than treatment at any point6, which would suggest that the effectiveness of NAIs could be more favourable than modelled under the stockpiling policy. In all cases the decision would remain to stockpile NAIs.This paper is predicated on the purchase of a stockpile large enough to treat a large proportion of the population (80% in the UK) in the community and in hospital with NAIs. This may well be the correct strategy if new evidence emerges that community-based treatment reduces either complications, hospitalisations or mortality. Further research will be required; indeed, the Bayesian decision analysis used here can be extended to consider how much to stockpile rather than simply whether to stockpile. However, if the evidence base were to remain limited to mortality reductions in hospitalised patients, or if the societal willingness to pay per QALY was low, as it may be in many resource poor settings, a ‘hospital-treatment only’ policy might be considered. This would reduce the cost of the stockpile significantly. For example, in the 2009 pandemic only 0.5% of symptomatic cases were hospitalised19. However, our study does not take observational evidence at face value, but ‘downgrades’ it, thereby yielding a reduced estimate of effectiveness and wider credible limits. We have calculated the distribution of possible deaths from pandemic influenza using a relatively simple mathematical model and then ‘averaged’ over the distributions of the variables rather than examining cost-effectiveness on a scenario-by-scenario basis. This approach is intuitively simple and is aimed to provide correct inferences using a simple logical framework for the synthesis of the commonly available evidence in order to assist decision makers with a complex decision. The model allows the logical basis of the decision to be ‘reverse engineered’, allowing the decision to be critiqued within the framework established by the model. We note though that many people are highly sceptical about the benefits of NAIs. This study is not intended to be proscriptive. AsOur conclusions are in line with the decision that would be made on the basis of cost-effectiveness evidence from previous studies34 It is not surprising, given the considerable uncertainties surrounding the meta-analysis cited here, that the differences between the reported effects and our bias corrected effect resembles the differences in empirical studies comparing observational studies and RCTs36.Obtaining an estimate for the bias in any particular study, or consolidated group of studies, is clearly an uncertain undertaking. There is an evidence base on bias arising from meta-regressions or other analyses comparing the results of imperfect studies to those of a ‘gold standard’. A recent Cochrane review comparing treatment effects reported in observational studies as compared to RCTs found that, “on average, there is little evidence for significant effect estimate differences between observational studies and RCTs...”4. The authors also reported a possible increase in the risk of psychiatric adverse events. However, this only reached statistical significance in exploratory analyses including a supra-licence dose and off-treatment periods. A more recent meta-analysis based on individual-level patient data of clinical trials focusing on licensed dose only found no such effects, but the number of events was small5. Neuraminidase inhibitors may also have protective effects against some adverse events such as cardiac events, and may reduce the risk of influenza-associated pneumonia and hospitalisation5. The benefit of treatment is unlikely to be grossly over-estimated and is likely to be under-estimated given our conservative assumptions. We have also not considered potential effects on children or from reductions in complications, hospitalisations or mortality that might be associated with community-based treatment, or any benefit arising from changing disease dynamics and reduced transmission; nor have we considered wider societal effects, such as productivity gains, reduced community transmission, and the value placed on a stockpile for a potentially risk averse population, all of which may increase the benefits of stockpiling.We acknowledge weaknesses in our study. The only outcome considered in the analyses was mortality. Adverse events caused by NAIs may also generate increased costs and hence reduced benefit. For example, a review of clinical trial evidence of NAIs found an increased risk of nausea and vomiting associated with treatmentWe note that our analysis is focussed on the United Kingdom but that it may be of use to other countries. The model for the benefits of NAIs can be simply applied to new contexts. However, the determination of the costs of the stockpile remains difficult. The costs depend on the treatment strategy planned for a given country and any price negotiations between the manufacturer and the government. A useful tool in this context is the ‘headroom’ method that asks instead what the maximum amount a decision maker should be willing to pay for an intervention, given a willingness to pay per unit benefit. This is a useful direction for future research.37. Nevertheless, the data are admittedly scant, and it is expected that this is a neutral assumption. Of course, if they are positively correlated then our conclusions become more conservative.We have assumed independence between the clinical attack rate and case fatality ratio, as well as other variables, however there is some evidence to suggest that they could be correlatedOur model examines the decision in the abstract and does not concern itself with externalities such as the possibility that availability of the drug will affect attitudes and hinder the effort to contain the spread of the disease, or that resistance to antivirals may develop. Nor have we considered the sensitivity of clinical diagnosis of influenza in identifying true positives or the costs and logistics of establishing a distribution process for the NAIs. The propensity to consult is also an important factor that may have affect the proportion of true positives, which in turn may have a bearing on the use of a stockpile if used on a “first come, first served” basis. Further research is required to optimize distribution and behaviour during a pandemic to ensure the cost-effectiveness of the stockpiling.Taking into account the existing evidence on pandemic influenza and the effectiveness of NAIs the decision should be to stockpile, provided a utilitarian decision-making framework is used of minimising expected losses and hence maximising expected benefits.The data referenced by this article are under copyright with the following copyright statement: Copyright: © 2017 Watson SI et al.Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).1.The data used to estimate the parameters in the model were obtained from documents compiled to assess pandemic influenza and thus represent the decision maker’s prior knowledge10.5256/f1000research.9414.d13265344.F1000Research: Dataset 1. Raw data of ‘stockpiling neuraminidase inhibitors for pandemic influenza usage’, This revised version (2) of the paper addresses all the criticisms I raised in my report.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. This article reports on a probabilistic cost effectiveness of stockpiling neuraminidase inhibitor antiviral drugs to mitigate against pandemic influenza deaths.  The study is methodologically sound. The decision-theoretic approach which selects the optimal course of action based on the utility of each outcome and the probability of each outcome conditional on each decision is appropriate. In this case the actions are whether or not to stockpile NAIs for use in a pandemic, and the outcome is the cost of the antiviral stockpile and the expected number of deaths during the shelf-life period of the antivirals. The model of the expected number of pandemic deaths identifies is structurally sound and uses various appropriate data sources to quantify uncertainties present in all the parameters. The conclusion that stockpiling NAIs is cost-effective for a sufficiently high willingness-to-pay cost per QALY follows from the model and the data used. However, I think that more attention needs to be drawn to dependency of this result to the crucial antiviral effectiveness parameter. The methodology where all parameters are treated probabilistically in a uniform way is excellent; however additional one- or two- way sensitivity analyses are still valuable for providing insight into the effect of the most important parameters. The authors have done this for the proportion of hospitalised cases receiving antivirals; it seems appropriate to also do this for NAI effectiveness, given the ongoing debate on the subject. I have divided further comments into three sections: major essential revisions, commentary with discretionary additions, and minor technical clarifications needed. Essential RevisionAs stated above, the reader would be well served with an additional figure similar to Figure 3, but plotting QALY threshold against mean NAI effectiveness. In my shallow experimentation with the author's model, it looks like at 20,000 GBP / QALY, NAIs cease being cost effective at around 0.94 effectiveness . ------------------ Commentary ---------- The suggestions made below I think might improve the article however I think the authors can best judge whether the additional effort and added complexity would be worthwhile or would be too diverting.The methodology of using expert opinion to mitigate potential bias in the studies estimating NAI effectiveness is a practical measure that is probably worthwhile. Some additional detail on the process would be appreciated. For example: how were assessors selected? How much time did the reviewers take in their bias estimates?In the discussion it could be noted that in a future pandemic with a large CAR or CFR, the proportion of severe cases receiving hospital care and the level of care are likely to be lower, simple due to hospitals being overwhelmed. The estimates of proportion of deaths occurring in hospital are from the 2009 pandemic which was very mild.If NAIs have any effect in preventing further transmission, e.g if they shorten the period of viral shedding, then mass administration of antivirals may reduce the overall attack rate and consequent mortality even if NAIs are not effective for mortality reduction of severe cases. As the study's model does not capture this, this is another way in which the study is conservative. I commend the authors for including the model code in the Appendix, I managed to run this code with relatively little effort. Minor Technical RevisionsThe CAR and CFR parameters used in the model are for a pandemic without NAI usage. Given that NAIs were used in the 2009 pandemic, should the CAR and CFR estimates for 2009 be included along side those of previous pandemics? If the 2009 CAR and CFR estimates are for example based on global data where NAI usage might be negligible that would be OK; but if they are based primarily on UK or US data they should possibly be excluded. In the Appendix page 4 there is a citation [20] that isn't given in a reference list.In the last sentence of the 3rd paragraph, the RR derived based on the OR and 10% mortality is stated as 0.89. This is the same as the bias-corrected RR given in the next paragraph. Is this intentional? Or should it be the RR value based on the OR and 10% mortality (but without bias correction), in which case it should be 0.825 (from the formula).The R / BUGS code in the Appendix worked almost without alteration. I found that I had to: Install BUGS (OpenBGUS). Hoist the npv function to the top. Remove the codaPkg=TRUE setting to obtain a result object.  If F1000 allows additional appendix files this could be supplied as an additional plain ASCII file, to avoid scraping the text from the PDF and correcting resulting formatting.I can't find the support for the n_hosp data value of 136. The tot_hosp value of 125 appears in the Donaldson BMJ paper. That paper gives 138 for the total number of confirmed deaths due to pandemic influenza. -------------------------I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. We thank the review for their comments and detail our responses below, point by point. The referee's text is in Italics.Essential Revision ------------------As stated above, the reader would be well served with an additional figure similar to Figure 3, but plotting QALY threshold against mean NAI effectiveness. In my shallow experimentation with the author's model, it looks like at 20,000 GBP / QALY, NAIs cease being cost effective at around 0.94 effectiveness . We have replaced figure 3 to incorporate these considerations and additional commentary.Commentary ---------- The suggestions made below I think might improve the article however I think the authors can best judge whether the additional effort and added complexity would be worthwhile or would be too diverting. I commend the authors for including the model code in the Appendix, I managed to run this code with relatively little effort.The methodology of using expert opinion to mitigate potential bias in the studies estimating NAI effectiveness is a practical measure that is probably worthwhile. Some additional detail on the process would be appreciated. For example: how were assessors selected? How much time did the reviewers take in their bias estimates? We have added additional description  in the Methods section although we also refer the referee to the cited articleIn the discussion it could be noted that in a future pandemic with a large CAR or CFR, the proportion of severe cases receiving hospital care and the level of care are likely to be lower, simple due to hospitals being overwhelmed. The estimates of proportion of deaths occurring in hospital are from the 2009 pandemic which was very mild.If NAIs have any effect in preventing further transmission, e.g if they shorten the period of viral shedding, then mass administration of antivirals may reduce the overall attack rate and consequent mortality even if NAIs are not effective for mortality reduction of severe cases. As the study's model does not capture this, this is another way in which the study is conservative. We will add these points to the discussion  Minor Technical Revisions -------------------------The CAR and CFR parameters used in the model are for a pandemic without NAI usage. Given that NAIs were used in the 2009 pandemic, should the CAR and CFR estimates for 2009 be included along side those of previous pandemics? If the 2009 CAR and CFR estimates are for example based on global data where NAI usage might be negligible that would be OK; but if they are based primarily on UK or US data they should possibly be excluded.  We would argue that the 2009 observed CAR and CFR are relevant data points to infer the distribution of possible CAR and CFR values. It is possible that mass NAI distribution may alter the parameters of these distributions, however, without further information it is not possible to model this. Excluding the 2009 pandemic would bias our estimates, and given the small amount of data, this data point provides a relatively large amount of information. We therefore opt to use all available data.In the Appendix page 4 there is a citation [20] that isn't given in a reference list. This has been amended.  In the last sentence of the 3rd paragraph, the RR derived based on the OR and 10% mortality is stated as 0.89. This is the same as the bias-corrected RR given in the next paragraph. Is this intentional? Or should it be the RR value based on the OR and 10% mortality (but without bias correction), in which case it should be 0.825 (from the formula). We believe the referee may be in error, as it is 0.83 in the third paragraph of Appendix B. However, we have resubmitted the revised appendix to ensure the correct version is available.  The R / BUGS code in the Appendix worked almost without alteration. I found that I had to: Install BUGS (OpenBGUS). Hoist the npv function to the top. Remove the codaPkg=TRUE setting to obtain a result object.  If F1000 allows additional appendix files this could be supplied as an additional plain ASCII file, to avoid scraping the text from the PDF and correcting resulting formatting. We have provided a file for use in the program Stan to run the program as well. The models were initially run in WinBUGS before ‘upgrading’ to Stan. We have noted this in the Appendix but opt to provide both pieces of code for people using both programs.  I can't find the support for the n_hosp data value of 136. The tot_hosp value of 125 appears in the Donaldson BMJ paper. That paper gives 138 for the total number of confirmed deaths due to pandemic influenza. This typo has been amended.   This is a well designed, carefully executed and documented study, that provides important insights into the cost-effectiveness of national stockpiles of neuraminidase inhibitors to be used during influenza pandemics.  The analysis is relying on a number of key assumptions, such as the effectiveness of NAI antivirals against mortality due to influenza, the probability of a pandemic occurring during the shelf life of the stockpile and the proportion of pandemic influenza deaths occurring in hospital. Many of these assumptions are based on a limited or controversial evidence base, however the authors acknowledge and address most of these limitations. The assumption that most pandemic deaths occur in hospitals, is based on the observation during the 2009 pandemic in the UK, however in many countries, already during severe influenza A(H3N2) epidemics, and during many previous pandemics, the majority of deaths are likely to occur in the community, outside of hospitals. It is confusing that the authors compare the costs of a population wide (80%) stockpile with the estimated benefits on hospital mortality only. Although this is discussed in the second paragraph of discussion, it would be helpful to see an analysis or results taking also into account outpatient and community mortality. It is likely that such an analysis would be useful for other countries than UK. Please discuss briefly the limitations of this approach and these assumptions, when replicating the study in other settings . In Box 1. the two columns are not aligned when viewing as a pop-up on MS Internet Explorer. In Figure 1. the references to UK, and the national pandemic flu service are not helpful and distract from the more general main message of this figure.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. We thank the review for their comments and detail our responses below, point by point. The referee's text is in Italics.This is a well-designed, carefully executed and documented study, that provides important insights into the cost-effectiveness of national stockpiles of neuraminidase inhibitors to be used during influenza pandemics.  The analysis is relying on a number of key assumptions, such as the effectiveness of NAI antivirals against mortality due to influenza, the probability of a pandemic occurring during the shelf life of the stockpile and the proportion of pandemic influenza deaths occurring in hospital. Many of these assumptions are based on a limited or controversial evidence base, however the authors acknowledge and address most of these limitations. The assumption that most pandemic deaths occur in hospitals, is based on the observation during the 2009 pandemic in the UK, however in many countries, already during severe influenza A(H3N2) epidemics, and during many previous pandemics, the majority of deaths are likely to occur in the community, outside of hospitals. It is confusing that the authors compare the costs of a population wide (80%) stockpile with the estimated benefits on hospital mortality only. Although this is discussed in the second paragraph of discussion, it would be helpful to see an analysis or results taking also into account outpatient and community mortality.https://doi.org/10.1093/cid/cix127), however we opt to remain conservative in our analyses. We did not consider non-hospital mortality as there were no data on the effectiveness of NAIs outside of the hospital setting, where there may be differences in compliance and other factors, when the study was conducted. We note that the way we have set up the analysis is to try to be as conservative as possible: the highest stated costs with a justifiable patient pool. On this basis we note that if a decision to stockpile is supported under our assumptions then it will certainly be supported if there is any benefit outside of the hospital. Recently published analyses outside of the hospital setting suggest a potential benefit . We have amended the discussion to reflect this. In Box 1. the two columns are not aligned when viewing as a pop-up on MS Internet Explorer.This is an issue for the journal. In Figure 1. the references to UK, and the national pandemic flu service are not helpful and distract from the more general main message of this figure.We have removed that box from the figure.