One key contextual feature in Verbal Autopsy (VA) is the time between death and survey administration, or recall period. This study quantified the effect of recall period on VA performance by using a paired dataset in which two VAs were administered for a single decedent.This study used information from the Population Health Metrics Research Consortium (PHMRC) Study, which collected VAs for “gold standard” cases where cause of death (COD) was supported by clinical criteria. This study repeated VA interviews within 3–52 months of death in PHMRC study sites in Andhra Pradesh, India, and Bohol and Manila, Philippines. The final dataset included 2113 deaths interviewed twice and with recall periods ranging from 0 to 52 months. COD was assigned by the Tariff method and its accuracy determined by comparison with the gold standard COD.The probability of a correct diagnosis of COD decreased by 0.55 % per month in the period after death. Site of data collection and survey module also affected the probability of Tariff Method correctly assigning a COD. The probability of a correct diagnosis in VAs collected 3–11 months after death will, on average, be 95.9 % of that in VAs collected within 3 months of death.These findings suggest that collecting VAs within 3 months of death may improve the quality of the information collected, taking the need for a period of mourning into account. This study substantiates the WHO recommendation that it is reasonable to collect VAs up to 1 year after death providing it is accepted that probability of a correct diagnosis is likely to decline month by month during this period.The online version of this article (doi:10.1186/s12963-016-0105-1) contains supplementary material, which is available to authorized users. Cause of death (COD) data are essential for informed planning in the health sector. Mortality statistics are a key input into tracking changes in the burden of disease in a population over time. This information can then be used to set, or reset, priorities for health interventions, monitor the efficacy of public health programs, and inform the allocation and distribution of limited resources within the health sector. Unfortunately, reliable cause of death information is least available in low-resource settings where arguably it is most crucial to public health research and decision-making . Such crA VA is a questionnaire administered to the caregivers or family members of a person who has died (the decedent) to elicit specific signs and symptoms that occurred in the period before death. VAs have been used in demographic surveillance sites and in studies of the epidemiology of disease for over 40 years –7. ModerAs a survey methodology, there is reason to believe that recall bias may affect the validity of VAs –12. CurrThe standard instruction for collecting VAs is to allow for a period of mourning after death and then to administer the VA as soon as is practicable. Recall bias is not a major issue in well-functioning CRVS systems with continuous registration of deaths or in Demographic Surveillance Sites with short intervals between survey rounds. It is, however, an issue in the collection of VAs in cross-sectional surveys, such as national censuses or Demographic and Health Surveys, where there are long intervals between survey rounds.In this study we aimed to quantify the effect of the recall period on VA diagnostic performance using a dataset in which the true cause of death was known with a high degree of certainty  and applThe general methodology of the PHMRC study has been described in detail elsewhere and is summarized here for convenience . Gold StThis study was based on VAs of deaths in the PHMRC GS validation dataset that occurred during 2007 and 2008 in Bohol, Manila, and Andhra Pradesh. For each death a gold standard cause of death had been identified and a verbal autopsy interview been collected during the PHMRC study between 6 days and 5 months after the death. In this paper, we will refer to these as first-round VAs. We revisited a subset of these and collected a second VA from the same families (second-round VAs). When revisiting households the same respondent was interviewed again; however, if this was not a possible a different family member was interviewed for the second VA.This study had a convenience sample aiming to explore the question of time between death and VA interview and accuracy. The symptom recall study revisited families who had provided verbal autopsies for the Population Health Metrics Research Consortium gold standard verbal autopsy validation study (PHMRC study). Data collection as an extension of the PHMRC study occurred 3–20 months after death in Bohol, Manila, and Andhra Pradesh for second-round VAs. The samples were based on all cases that had been studied over a calendar year in the PHMRC study which could still be identified in the community.Further data collection funded by the Australian National Health and Medical Research Council (NHMRC) occurred in Bohol, where additional second-round VAs were collected 18–52 months after death. For clarity, we describe the two separate data collection periods for second-round VAs as Bohol (1) and Bohol (2) from the PHMRC and NHMRC grants, respectively. The Bohol (2) cases were selected from the calendar year 2007–2008 in an attempt to test the validity of longer periods of recall.The methods of this study were approved by the Internal Review Boards of the University of Washington, Seattle, WA, USA; School of Public Health, University of Queensland, Australia; University of Sydney, Australia; National Institute of Public Health, Mexico; Research Institute for Tropical Medicine, Alabang, Metro Manila, Philippines; Muhimbili University, Tanzania; Public Health Laboratory Ivo de Carneri, Tanzania; and CSM Medical University, India. All data were collected with informed consent from participants.COD was assigned using Tariff 2.0, the VA analytic method recommended for use in routine mortality surveillance systems based on the comparative performance of existing methods . The TarWe created a series of variables as the basis for logistic regression. We generated a binary indicator for “correct assignment”; this was coded as zero if the prediction  from applying the Tariff method to the VA differed from the gold standard COD, and one when it was identical. A continuous variable labelled “time” was defined as the number of months between death and administration of the verbal autopsy. Binary indicators were also created for the different sites/populations , Bohol (2), and Manila) and the different modules .The aim of the analysis was to quantify the effect of recall period on the probability of correctly assigning the COD using Tariff 2.0. The effect of recall period on deriving the correct cause assignment (assessed against the gold standard) was measured using a logistic regression framework as specified in Equation 1, controlling for confounding due to site of data collection and survey module . The predicted probabilities of correct assignment were then calculated from these models. The data were also clustered on the individual to control for the paired  observations. All analyses were done using Stata 13.1 .Each module has a different set of causes associated with it , and Bohol (2). Differences between the sites captured by this covariate are likely to include variations in data collection procedures (despite using a standardized protocol), cultural differences, and cause composition of deaths at different sites (Regression 1).Regression 1:To see if our results were robust, we conducted a sensitivity analysis which treated recall period as a series of categorical variables. Regression 2 uses the recall period of 0–2 months as a reference as compared to a recall period of 3–11 months and recall period ≥ 12 months.Regression 2:In a third regression model, treating also the recall period as categorical variables, we used the recall period of 0–2 months as a reference as compared to a recall period of ≥3 months:Regression 3:In a secondary analysis we controlled for different characteristics of the respondent including the type of their relationship with the decedent, education, and sex. This was done within the same logistic regression framework, controlling for collection site and module and using a sandwich estimator to relax the assumption of independence of VAs from the same decedent. This analysis was stratified by module because the type of relationship between respondent and decedent varied with the age of the decedent. As a sensitivity analysis we also performed the three regressions using only adult decedents.A total of 4226 verbal autopsies were collected for 2113 decedents in the three separate sites. Data collected from Bohol at the two different time periods are tabulated separately. Table Tables Simple  concordance and kappa by site and module for the 4226 VAs are summarized in Table Table The fundamental objective of this research was to generate an empirically based recommendation for the optimal recall period, or time, between death and VA interview, based on an analysis of repeat interviews of families for which a gold standard cause of death was known.The first regression equation indicates that the probability of a correct diagnosis declines by 0.55 % per month. This means that the probability of a correct diagnosis in VAs collected in the third month after death would be 98.3 % of that in the first month and in the fourth month after death, 97.8 %. We used the second and third regression equations as a sensitivity analysis. We assumed best practice to be collection of VAs in the first 3 months after death and examined decline. The equations indicate, first, that the probability of a correct diagnosis in VAs collected between the fourth and twelfth months after death would be 95.9 % that of one collected the first 3 months after death, but with fairly wide limits of confidence that include the null value. This is consistent with a monthly decline of 0.55 % probability of correct diagnosis. The sensitivity analysis also indicates that there was little or no decline in accuracy for recall periods ≥ 12 months. However the evidence for lack of decline during this period is less firm because it conflicts with the evidence from the first regression equation of a monthly decline in accuracy of 0.55 %. Far fewer VAs were collected at 12 months or more than in the 12 months after death, affecting the precision of our estimates. On the other hand, this lack of decline is consistent with findings from the study which showed no difference in sensitivity and specificity of VA for recall periods up to 3 years .Clearly, recall period is only one factor among many affecting the accuracy of COD assignment from VAs. We note the variation between sites and between modules. In considering variation by site we need to consider variation in factors affecting the interview itself: language, norms and biomedical concepts, and type of respondents and interviewers . We shouIt is important to keep in mind that deaths included in this study took place in health facilities. It should also be noted that participating in a VA interview after a death of a relative/friend raises different emotions for respondents, and may refresh memories, trigger the search for further information, or cause the reinterpretation of the events leading to the death. Therefore, it is difficult to assume the conditions for answering a first and second VA interview about the same death are the same, especially if the informant is the same. This process can introduce variations in responses that, although difficult to quantify, should be taken into account.We conclude that accuracy of VAs declines at a rate of 0.55 % per month in the first year after death. Findings of this study suggest that collecting information in the first 3 months after the death –taking into account a period of mourning – may improve the quality of the information. However, the probability of a correct diagnosis in VAs collected between four and 12 months after death will, on average, be 95.9 % of those collected in the first 3 months.So far, decisions about the best recall time for VA collection have been based on empirical or practical considerations, without a systematic and evidence-based approach to make such decisions. The results of this study have, therefore, important practical implications for the collection of VAs, both in research settings or reinforcing civil and vital registration systems. This study substantiates the WHO recommendation that it is reasonable to collect VAs up to 1 year after death providing it is accepted that accuracy is likely to decline month by month during this period. We have some evidence to suggest that the rate of decline decreases after the first year, but this is not strong enough to change the WHO recommendation.Verbal autopsy is increasingly being considered for routine application in civil registration and vital statistics systems, yet many aspects related to best implementation practices remain unclear. Key among these is the optimal period in which to administer a verbal autopsy after death, and how longer waiting periods might affect the accuracy of diagnosis. By providing empirical evidence on this important issue, we hope to better inform decisions about verbal autopsy use in countries and promote its wider application to generate policy-relevant information on leading causes of death.


Within traumatology a common indication for acute surgery of fractured clavicles is bone shortening over 2 cm. This indication is among but a few indications; which are recommended to be treated operatively within the very first weeks after a fracture. Theoretically clavicle fractures could become less shortened over time due to decreasing muscle pull. If this reduced shortening does indeed happen, some patients with initial bone shortening over 2 cm could perhaps be treated conservatively? However, it is unknown what happens to the length of the clavicle within the first weeks after a fracture. The aim of this study was to investigate if the length of the fresh fractured clavicles changes within the first three weeks.Rested length measurements using navigation ultrasound were done on 59 patients with a fractured clavicle. Measurements were performed within the first three weeks after a lateral or mid-clavicular fracture. The inclusion period was from March 2014 to February 2016. Median age was 40 years and age range was 18–81 years. The data was analyzed using mixed linear models.p = 0.24).The clavicle length showed no change within the first three weeks after fracture (Fractured clavicles retain their length for the first three weeks. Clavicular fractures comprise approximately 2–4% of all fractures . AlthougThe authors decided to investigate if this clinical problem could be solved by measuring length changes of the clavicle within the first weeks are a fracture. The solution using existing conventional methods was limited due to the risk of bias from radiographic magnification . It was clinicaltrials.gov with the identifier NCT02089282. All participants gave their written and oral consent upon entering the study.The local ethical committee of Region Zealand, Denmark (reference number REG-67-2013) as well as the National Danish Data Registry approved the study. The study was reported to The study was based on the STROBE statement  and was The following definitions were used: 1) A medial fracture was located in the medial third, a midshaft fracture in the mid-third and a lateral fracture in the lateral third . 2) A diPatients were found by screening all clavicle radiographs during the inclusion period. If a eligible clavicle fracture was found the patient was contacted. Oral and written consent was gathered from the patients before they entered the study. At each session, the patients’ injured and uninjured clavicle length was measured using NavUS. Clavicular length was measured twice in both the injured and uninjured side. Patients were included within 1–9 days after sustaining a clavicular fracture with one or two follow-up sessions 20–26 days after fracture.The NavUS system on the General Electric Loqic E9 ultrasound scanner was used. The program for musculoskeletal measurement  was used along with the ML6–15 probe. The NavUS system itself consists of a bracket for the ultrasound probe with two attached sensors and a stationary transmitter. The transmitter generates a magnetic field which the sensors utilize to determine their exact spatial position. The magnet was placed opposite the side in question e.g. left to right, right to left. A distance of approximately 15 cm was kept from the shoulder. Calibration of the NavUS was done after each measurement with probe and sensors kept still on the sternum and with sensors facing away from the magnet.The following procedures were followed.Positioning: Patients were sitting in a wooden chair  with arms along their side. The participants were instructed to sit immobile during measurements.Measurement: The rest measurements started with marking the sternoclavicular joint. The scanner was set in a perpendicular position to the sternoclavicular joint. Once this point was marked, the probe was moved the acromioclavicular joint while the participant remained still. A second point was marked on the acromioclavicular joint with the probe in a 30° sagittal/horizontal angle. The machine reported the distance between the two points as the resting clavicle length.Statistical analysis was performed with STATA software . A power calculation had been made before the study with a power of 80% and an alpha of 0.05. The needed patients were 63 patients. The local ethical committee recommended that a further seven patients should be included for a total of 70 patients. Standard descriptive statistics were done for fracture type, age and gender. Mixed linear model regression analysis was used after the recommendation from an out of study statistician as multiple measurements were done at various time points after the fracture. Furthermore the use of mixed linear models allowed to adjust for gender and left/right length differences on the individual level. The length measurement performed on injured clavicles was used as the continuous response variable. Time was used as a continuous variable. Explanatory variables (covariates) were type of fracture, displaced fracture y/n, side and gender. These were chosen on their apparent connection to the pre-fracture length of the clavicle. During the model building phase the interaction terms of displacement versus days was added into the model as it was found to be statistically significant for the fit of the model.Fifty-nine patients who had a fracture during the period March 2014 to February 2016 were included. This number included a total of 42 male patients and 17 female patients with the median age of 40 and age range of 18–81 years. The inclusion flow-chart is shown in Fig. p = 0.24). This was true for displaced fractures as well as for the interaction terms between solely displaced fractures and time which also showed a coefficient of 0 mm . Specification of data is shown in Table The coefficient describing the length changes over the first three weeks after the fracture found a change in length of 0.1 mm which statistically insignificant (SE.: 0.1 mm; CI: −0.4 mm to 0.1 mm; It was found that the length of fractured clavicles did not change within the first three weeks. The ultrasound modality, NavUS, has been proven to have the ability to achieve accuracy comparable to 3D rendered CT-scans for clavicular length measurements . The resClavicular bone shortening is one of very few indications for referral to subacute surgery as the majority of patients today are operated months later in their recovery mainly due to non-union of the fracture . The facIn continuation, as it is not the bone itself which returns to its normal length; the results indicate that there may be other factors that are the cause for the shoulder to apparently return to its normal appearance. These factors could be decreasing muscle pull or reorientation of the scapula. For the time being no biomechanical study exists which describes what happens to the shoulder girdle within the first weeks and could be an interesting field for future research.Limitations of this study include use of a technology that has not, apart from a validation study, been used in this setting before. The authors are confident of the method since it has a reliability that rivals 3D rendered CT-scans and a validity above conventional radiographs . The useA final limitation was that the authors only recruited approximately 30% of the total fractures of which the majority was male and the median age was 40 years Fig. . This inThe strengths of this study are that this is the first study of its kind to follow the length variations closely within the first weeks post-fracture. It is also the first study to implement ultrasound technology that provides the best possible accuracy other than high resolution CT. This study is unique since an equivalent study using CT would have meant a significant radiation dose to the participants.Clavicular length of fresh fractures is stable throughout the first three weeks after fracture when measured in neutral position. If radiographically measured midclavicular bone shortening is followed as an operative indication, there is no need to wait for the shortening of the fracture to change. However, as the relevance of clavicular bone shortening as an operative indication continues to be discussed scientifically and clinically our results may prove to be of use only if an operative paradigm is followed.


Three (3) different methods  were applied to five (5) internal datasets and one (1) external, publically available dataset where covariate shift existed. In all cases, k-NN’s performance was inferior to either logistic regression or covariate shift. Surprisingly, there was no obvious advantage for using covariate shift to reweight the training data in the examined datasets. Intuitively this is justified, as one should not expect to learn a classifier on one distribution of examples and apply it to accurately predict labels of examples drawn from a different distribution. Covariate shift is a machine learning technique that can be utilized in supervised learning when the training and prediction distributions are known to differ, but the concept being learned remains stationary. While standard machine learning classifiers are trained and then used to predict on arbitrary compounds, covariate shifted classifiers must be trained specifically for each prediction dataset. This is because covariate shifted classifiers weight the training distribution to be more similar to the prediction distribution. A recent book provides an excellent overview of the current state of the art in covariate shift methods2.A common prerequisite in supervised learning algorithms is that the training and prediction data arise from the same distribution and are independently and identically distributed . Initially the computational tool is trained using series A but must now predict on series B. The concept “binding to hERG” is fixed, however the area of interest has transitioned from chemical series A to chemical series B. The feature vectors describing these two sets are likely related but potentially different; and as such, their covariates have shifted. Put more mathematically, the probability of observing a feature vector from the prediction set is different from the probability of observing a feature vector from the training set. That is, the training and prediction sets are3 that the appropriate importance weighting factor w(x) for each instance “x” in the training set is:Covariate shift methods typically reweight instances in the training data so that the distribution of training instances is more closely aligned with the distribution of instances in the prediction set. This is accomplished by providing more weighting during model building to an instance in the training set that are similar to an instance in the prediction set. It has been showntp (x) is the probability of seeing instance x in the training set andpp (x) is the probability of seeing x in the prediction set. It is important to note that only the feature vector values (not their labels) are used in reweighting. The importance weighting scheme is intuitively understandable. If the probability of seeing a particular instance from the training set in the prediction is very small, then this instance should carry little weight during the training process and consequently have little effect on the decision function.where4. The KLIEP method is based on the Kullback-Leibler divergence theorem and attempts to find weights to minimize the divergence from ptrain(x) to ppredict(x). Briefly, the importance is modeled as a linear function:For our experiments, we use a logistic regression classifier where each training instance is weighed by its importance w(x). For the calculation of w(x) we use the Kullback-Leibler Importance Estimation Procedure (KLIEP) method developed by Sugiyamaiα are the weights to be learned andiφ the basis functions. The importance weight fromThepp(x) to its estimatep^p(x) can then be expressed as:The KL divergence fromw^(x) and adding constraints to ensure proper normalization, a final objective function to be maximized can be derived as  that allows calculating weights for a training instance x. These weights can then be incorporated when training a classifier to obtain a covariate shifted version of the classifier.To demonstrate the use of covariate shift methods, we repeated a simple toy experiment as detailed inThe red training points are drawn from two (2) two-dimensional Gaussian distributions representing a class 1 and a class 2. The green prediction points are drawn from a slightly rotated version of the training distributions. The red line plots the classifier obtained when training on only the training points; the green line plots the classifier trained on both the training and prediction points . The blue line plots the classifier trained on the training data that was weighted by the importance factor as estimated by the KLIEP method. Note how the blue line is shifted towards the optimal classifier, demonstrating the effect of the KLIEP algorithm and covariate shift.50 data derived from the ChEMBL databaseThe beta secretase ICUnits are in nM.Click here for additional data file.Copyright: © 2016 McGaughey G et al.2016Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).5. In principle, covariate shift is applicable to any classifier that allows weighting of input instances (e.g. support vector machines and random forest). For this study we wanted to isolate the effects of covariate shift and therefore selected a classifier without adjustable parameters and used logistic regression (LR). Logistic regression is a classification technique analogous to linear regression and is applicable when the dependent variable is categorical6. We combined logistic regression with KLIEP and applied it to five different in-house ADME  assays and one external dataset (beta secretase). The cutoff values for determining the binary categories for the compounds in each dataset are listed inUsing the Python programming language we implemented the KLIEP method for determining weights for use in covariate shift7 path fingerprints are used as feature vectors. We experimented with different fingerprints provided by OpenEye  and found that they had no significant effect on the outcome.For each dataset the molecules were sorted by compound registration date. The first 75% of the data comprised the master training set while the remainder formed the master prediction set. Temporal ordering of the data represents the evolving coverage of chemical space by drug discovery projects and consequently captures the natural “shifting” of the covariates. Classifier performance statistics are generated by performing twenty different runs, each on a random 80% of the master files. Performance statistics for each classification task are then obtained by averaging the results of the twenty individual folds. In all cases, OpenEyeTo ensure the data was amenable to covariate shift we generated classifiers separating “training” from “prediction” data.9 such that others could use their own fingerprints and independently support or refute our claims. We chose the beta secretase IC50 data as it is a well established biochemical screen, highly accurate and contains > 7000 data points crossing multiple orders of magnitude, which are publically available. Using OpenEye path fingerprints and K-Means clustering we clustered the dataset into two clusters, A and B. Under cross-validation, a logistic regression classifier was able to separate the two clusters with a high level of accuracy (90%) indicating that the clustered dataset would be appropriate for application of the covariate shift algorithm. Ten random subsets of molecules from cluster A were used to train a logistic regression classifier using covariate shift which was then used to predict on molecules from cluster B. The performance of the shifted classifier was compared to an unshifted classifier trained and tested on the same clustered datasets and random splits. The process was repeated by training on molecules from cluster B and predicting on molecules from cluster A. Analogous to the internal datasets, as measured by overall classifier accuracy, there was no statistical advantage for application of covariate shift .We extended the study to include an external dataset provided by ChEMBLA possible explanation for the failure of the covariate shift method to provide a boost in predictive performance could be that the calculated importance weights are all similar. This would cause each training example to exert the same influence on the decision function and thus the importance weighting would have no effect. This was not the case.We have applied the KLIEP method to five (5) internal data sets and one (1) external data set where covariate shift was evident. Although KL+LR was an advantage over k-NN, there is no statistical advantage of reweighting the training dataset. We are surprised with this outcome and are currently exploring other datasets where application of covariate shift may improve the predictions.The data referenced by this article are under copyright with the following copyright statement: Copyright: © 2016 McGaughey G et al.Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).F1000Research: Dataset 1. The beta secretase IC50 data derived from the ChEMBL database,10.5256/f1000research.8317.d11788210 The only issue I am still having trouble with is equation 1. w(x)=Pp(x)/Pt(x) where the P’s represent probabilities for molecule x. Given that the training and test sets are distinct points (molecules) in a chemical descriptor space and the training and test sets do not overlap, how are the probabilities for molecule x calculated from the other molecules?I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The revised paper has been significantly improved and addresses a number of issues raised in the original reviews. However, a few issues remain that should be addressed. 1. The description of Figure 1 is inconsistent with the figure legend. It seems that the red and blue labels in the legend of Figure 1 need to be swapped to make the figure consistent with the description in the text: The red and green curves look like pdfs whose integrals are very similar and close to 1 while the blue curve has a much larger area inconsistent with a pdf. 2. For the ChEMBL data set the inactivity/activity cutoffs used should be mentioned.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Having read the revised paper it seems better, but I’m still somewhat puzzled about a few things.I am getting the feeling that the K-NN method is meant as a baseline control method since by definition K-NN looks at only the training set compounds close to the test set compounds, so there is an implicit selection of training set compounds,  and this should have a similar effect as covariant shift.  This is not explicitly said in the paper.The authors do not  try sophisticated but more “standard” classification methods like random forest or SVM, and don’t say why not.Both myself and the other reviewer seem confused by Figure 1.  The red line is supposed to be the importance weight.  However, it implies that the highest weights are given in a region of descriptor space far away from both training and test sets.  I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. I'm responding directly to the questions you posed, in that order. 1.  Agreed that we could have added such a sentence to the paper. 2.  We didn't examine more standard classification methods as we were specifically studying whether there was a benefit in using covariate shift. We were not interested in exploring RF or SVM as that was out of scope for this paper. 3. The three curves in Figure 1 were all drawn as per a Gaussian distribution and we thought it would look odd to have the red line go straight to zero when, for example, x=1.  The purpose of the importance weight being close to 0 (on the y axis) when x=2, for instance, is because there's much overlap between the training and the prediction set.  When, for instance, there isn't much overlap , the importance weight goes up.  I can see why one might be confused when x=1;  that was merely to show that when there's minimal overlap between prediction and training, that the importance weight is quite high. Concerning Figure 1: According to equation (1) w(x) = p_p(x)/p_t(x) there is a certain symmetry between the importance weight w(x) and the training pdf p_t(x). This means that in Figure 1 the red curve can either show the importance weight and the blue curve the training pdf or vice versa without compromising the accuracy of the figure.  What speaks against interpreting the blue curve as a pdf is that it clearly has a much larger area than the green curve representing the prediction pdf, which should be 1 in both cases. From visual inspection the red curve has an area much closer to 1 than the blue curve and it should thus be interpreted as training pdf while the blue curve represents the importance weight. Even though the figure is only used for illustration purposes it could be improved by either relabelling or by rescaling the red and blue curves accordingly. The figure is really meant to be an illustration of the importance weight - and not mathematically accurate with respect to pdfs. In actuality, the green and blue curve look like pdfs but they really are not. The green and blue curves are as such not normalized using the same scales - so the area under each curve does not sum to 1. We could make the blue curve a shifted version of the green curve then recompute the importance weight. What we were hoping to illustrate is that if an example from the training (blue) was pulled at x = 1 it would be very important for training because it is rare and the testing (green) set has non-zero support at x = 1. Additionally, another perhaps confusing aspect about the figure is that the y-axis represents two values on different scales: 1) the importance weight and 2) the probability (relative) of seeing a training/test example. We will redraw the figure and upload a new version. The study investigates the influence of accounting for covariate shift in classification performance using logistic regression models. Overall, this short paper is very well and clearly written, however the method section should be expanded (see below). Although no increase in performance could be established by accounting for covariate shift, it provides an excellent basis for further investigations.Suggestions/Corrections:I assume all models were trained as binary classifiers. This is potentially confusing as the chosen ADME properties in the experimental data could also have been modelled using regression models. This should be stated clearly and explained how labels (good/bad) are assigned to the training instances for the different ADME properties (and how labels are assigned to the ChEMBL data given the potencies).Which basis functions (kernels?) were used in equation (2)?What distance measure was used for k-NN ?In Figure 3 (and 4), given the imbalance in data size between training and test set, consider reporting the balanced accuracy. E.g. a trivial classifier classifying each compound as "training" compound would have an accuracy of 75% based on the imbalance of the data set, which needs to be taken account when interpreting Figure 3.The authors provide a data set for download although they do not explicitly report the results for that data set. The results should be reported.The method section should be expanded:In the formula for KL on page 3 the two vertical bars should have the same size.In Figure 1, the labels for the red and blue line are mixed up.Typos:I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. This is potentially an important negative result in QSAR, however I think some revision is necessary because some aspects are unclear.The title “Understanding covariate shift…” is a little weak. One could say “Failure of covariant shift to improve model performance…”It needs to be explicitly pointed out in the introduction that in most QSAR one builds a model then is able to predict arbitrary compounds. On the other hand, to use covariant shift, one must know which molecules one is predicting before one can generate the model. One can regard “lazy learning” as an extreme version of covariant shift:  neighbors of the test set molecules are given weights of 1.0 and all other molecules are given weights of 0.I need a little more explanation in words of how the weighting is done for training set compounds. Since we are using substructure descriptors here, I am finding it hard to visualize. For example, are we just using distance to the nearest test set example, or are we looking at overlap of the training set descriptors with the distribution of test set descriptors?Practically no explanation is given as to what QSAR methods are being used. I know what K-NN is and I presume LR is linear regression. Why weren’t popular methods like random forest, SVM, or PLS tried? enclosed figure.The color key in Figure 1 does not seem to match what is in the text. In any case, perhaps a better way of looking at would be theI have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.


Binding reaction kinetics of analyte-ligand at the level of a sensitive membrane into a microchannel of a biosensor has been limited by the formation of the boundary diffusion layer. Therefore, the response time increases and affects the overall performance of a biosensor. In the present work, we develop an approach to engineer fluid streams into a complex configuration in order to improve the binding efficiency. We investigate numerically the flow deformations around a parallelepiped with square cross-section inside the microfluidic channel and exploit these deformations to simulate the analyte transport to the sensitive membrane and enhance both association and dissociation processes. The effect of several parameters on the binding reaction is provided such as: the obstacle location from the inlet of the microchannel, the average flow velocity, and the inlet analyte concentration. The optimal position of the obstacle is determined. An appropriate choice of the inlet flow velocity and inlet analyte concentration may reduce significantly the response time. Their development has been conditioned by the technological ability of adapting microfabrication techniques originally designed for electronic fluidic applications. These microfluidic devices provide many of the features that make bioassays advantageous, such as short analysis time and the ability to operate with small samples and high sensitivity4. Recently, there has been an increasing interest to integrate advanced biosensors into lab-on-a-chip systems by introducing microfluidics6. Jang et al.7 presented in their work a simple method to fabricate microfluidic devices for potential applications as biosensors in micro-total-analysis-system (µ-TAS). Lee et al.8 have developed a chip based microfluidic device that has a multi-channel configuration to detect microarray immunoassay samples based on a surface plasmon resonance (SPR) detection system.Microfluidics has gained tremendous attention over the last decade as an attractive research field9. Integrated microfluidic system for rapid screening of CRP aptamers utilizing the systematic evolution of ligands by exponential enrichment has been developed by Huang et al.10.In clinical applications, quantitative analysis of proteins plays an important role in the diagnosis of diseases. For example, C-reactive protein (CRP) is a biomarker of inflammation whose concentration increases several hundred times in response to acute injury, infection, or tissue damage11, the quartz crystal microbalance (QCM) sensor13, the impedance based sensor14 and the immunoassays are the most frequently used devices for the quantitative monitoring of biomolecules. Although the processes of detection are different, they all implicate the same kinetics of specific binding of analytes and immobilized ligands. More specifically, the system mixes a small concentration of a biological analyte, such as CRP, with the fluid in a microchannel. The binding efficiency on the reaction surface is usually large enough to bind practically all analyte molecules appearing there. Thus, the reaction is said to be transport limited and it usually causes the formation of a diffusion boundary layer16. The development of the diffusion boundary layer increases the response time and limits the overall performance of the biosensor18. In practice, a detection cycle can take many hours, which is a problem that must be solved.The SPR sensoret al.19 developed a three-dimensional microfluidic confinement device for efficient sample delivery to biosensor surfaces: application to immunoassays on planar optical waveguides. We expand the study of the flow confinement effect on the binding reaction in order to enhance efficient mass transport20. This technique has been studied in great details, for flow confinement, a sample flow is joined with a perpendicular makeup flow of water or sample medium. The makeup flow confines the sample into a thin layer above the sensing area. Therefore, the velocity flow increases to improve the binding rate. Sigurdson et al.21 suggested a technique to enhance microfluidic immuno- sensors by using AC electrokinetically-driven microscale fluid motion to enhance antigen motion towards immobilized ligands. The binding rate can be increased in the first few minutes by a factor of seven using a 6 Vrms applied potential. However, these observations are related only to the association phase of the binding reaction. Huang et al.22 performed a two dimensional full time scale finite element simulation on the binding reaction kinetics of two proteins (CRP) and immunoglobulin G (IgG). The electrothermal force results from the application of a non uniform ac electric field to the flow in the microchannel of the biosensor. It can generated a vortex field to stir the flow and reduce the thickness of the diffusion boundary layer. Consequently, the reaction rate is increased and the association and dissociation processes are accelerated. There is, however, a lack of thermal studies on the effect of the temperature rise on the chemical reaction. Zimmermann et al.23 modeled and optimized the reaction kinetics in the capillary-flow-driven microchannel immunoassays using finite difference method. Hu et al.24 designed a new poly(dimethylsiloxane) (PDMS) microfluidic chips for heterogeneous immunoassay using electrokinetic control. Hu el al.25 also presented in another work, a theoretical model that includes only the convection and diffusion of the bulk analyte flow and binding reactions at the solid surface. The used model shows that electrokinetically driven immunoassays have better reaction kinetics than pressure-driven ones, resulting from the plug-like velocity profile. Hart et al.26 studied the enhancement of heterogeneous immunoassays using AC electroosmosis (ACEO) through the use of finite element method and fluorescent immunoassays. They also indicated in their work that the heterogeneous immunoassay is enhanced by the use of combined ACEO and dielectrophoresis (DEP). Munir et al.27 developed a novel and simple strategy based on tagging analyte with MNPs towards the sensing zone by using magnetic field force. Amini et al.28 developed a hierarchical approach to engineer fluid streams into a board of complex configurations. They investigated the flow deformations around a set of cylinders within a microfluidic channel. In addition, utilizing this platform to guide liquid in microchannels is useful in a variety of applications. This technique presents a passive mechanism to stir the flow. It mixes the fluid quite effectively and does not increase temperature. It is more suitable for biomedical applications than active mechanism employing the electrothermal force. In our previous study, we have studied the combination of the effect of the flow confinement and the effect of the electrothermal force on the heterogeneous immunoassays as a promising approach to increase mass transport at the level of the sensitive surface of a biosensor29.Currently, there are ongoing scientific research efforts to develop several mechanical and physical strategies for replenishing the target antigen to the sensor surface and to enhance the mass transport in microfluidic networks. Hofmann In this study, we develop a novel approach to engineer fluid streams in order to enhance the binding efficiency of immunoassay for a microfluidic biosensor. We will also investigate the flow deformations around a cylinder with square cross-section within a microfluidic channel and using fluid transformations to reduce the diffusion boundary layer. We will perform two dimensional simulations and study the effect of changing the geometrical locations of the obstacle on the biosensor performance. In addition, we examine some important factors such as the average inlet flow velocity, and the initial analyte concentration that control the immunoreaction in the microfluidic biosensor. The sample of CRP pairs in a biosensor immunoassay will be used to predict the response time of the biosensor as a function of the previous parameters.P is the pressure,  the Cartesian components of the velocity vector. The fluid properties such as the dynamic viscosity and the density are assumed to be constant. The dynamic viscosity is μ = 10−3Pa.s, and the fluid density is ρ = 1000 kg/m3.In the microfluidic channel, the flow velocity field is described by the Navier-Stokes equations. The fluid is assumed to be Newtonian and incompressible. In the 2D Cartesian coordinates and for a steady flow, the conservation and Navier-Stokes equations can be written as:0] varies in the range 6.4 × 10−6 mol/m3 to 22 × 10−6 mol/m3. However, the initial surface concentration of the ligand is constant [B0] = 1.8 × 10−8 mol/m2. The molecule weight of human CRP is 115 kDa. It consists of five identical noncovalently bonded monomer subunits linked in the form of a cyclic pentamer30. The CRP is known as a marker for infections and inflammatory processes in human blood serum. CRP is usually present in human serum with a concentration <1 µg/ml. Nevertheless, CRP concentration level can increase up to tens or even hundreds times when the inflammation occurs.A biological analyte is introduced with the fluid in a microchannel containing a sensitive surface. The analyte may be C-reactive protein (CRP) and its concentration is very small so that the density and viscosity of the fluid are not changed. In this study, the initial analyte surface is involved in binding reaction and the variations of [A] are due to the transport phenomena.The analyte is then carried to the sensitive surface due to diffusion and convection. We denote by [A] the concentration of the analyte inside the flow and [A]−11 m2/s) and G denotes the reaction rate. Here G equals to zero because no reactions take place in the fluid bulk.The spatial and temporal variations of the analyte inside the microfluidic channel are described by the convection-diffusion equation:The analyte portion diffused towards the sensitive membrane reacts with the antibody ligands immobilized on the reaction surface. The chemical reaction process is as follows:32. In this work, we will use the first order Langmuir adsorption model33. This model is often used to study the dynamics of molecular reactions in biochips and SPR biosensors34.The chemical reaction can be observed as being an adsorption process since it takes place at the binding surface. Many models have been used to describe the adsorption phenomena. Some of them are empirical models whereas some others are based on statistical physicsAsurf], the concentration of free antibodies [B] and the association rate constant kon. Similarly, the dissociation reaction rate depends on the concentration of bonded ligands [AB] and on the dissociation rate constant koff. However, the concentration of free antibodies is equal to the difference between the total concentration of free antibodies [B0] and the concentration of bonded ligands [B] = [B0] − [AB].The complex concentration [AB] produced during the reaction increases as a function of time until saturation. The binding rate depends on the analyte concentration on the surface  = [A0];The inlet condition which consists of a uniform concentration: , are all zero. The initial surface concentration [B0] is assumed as 1.8 × 10−8 mol/m2.The initial conditions for both the concentrations of the analyte in the bulk,  is approximated using quadratic interpolation in each element37.The system of the transport equations coupled with the first Langmuir adsorption model is solved using the finite element method (FEM) with the Galerkin methodTo ensure that the convergence has been reached and the computed results are independent of the mesh size, Fig. For the calculation, we solve firstly, the Navier-Stokes equations to obtain the velocity field. Then, we solve the analyte transport equation coupled with the complex concentration equation. These two equations are time-dependant. The total concentration accumulated at the capture area can be found by integrating local concentration over the whole reaction surface.38. In the experiment, the hybridization kinetics of the DNA strands is controlled by the fluorescence microscopy, and the flow is turned on for 50 min, then turned off for 310 min and again turned on for the rest. The total interval of the experiment is 1400 min. When the flow is turned off, only the diffusion mechanism takes place. However, when the flow is turned on, there is a competition between the diffusion and convection mechanisms. Figure The validation of the numerical model in this work for the mass transport inside the microchannel is performed on the basis of the experimental data for 29 basis pairs of DNA strands from the literature38. The numerical result presents a good agreement with the experimental data. This proves that the model used to describe the kinetic reaction is valid. It is worthy to notice that other works validate the first order Langmuir model for other complexes39. We deduce that this model can describe suitably the mechanisms inherent to the association and dissociation phases of the binding reaction. This proves that the surface concentration of the complex AB is computed accurately. The characteristic of the binding reaction is in quantitative agreement with the theory works. The results obtained with the present model have also the same behavior as the experimental and numerical results presented by Han and Park40.Our numerical results are compared with the experimental data available in the literature42. For this purpose, we investigated numerically the flow deformations around a parallelepiped with square cross-section inside the microfluidic channel on the binding reaction.Rapid detection time of biomolecules is a major challenge for microfluidic-based biosensor. Especially, detection time for heterogeneous immunoassays is long because the analyte takes a long time to transport by convection and diffusion to the sensing zone to bind with the surface-bound receptors. The limitation of mass transport presents a critical barrier which restricts the performances of the devices. It should be overcome for many biosensor applications0] = 6.4 10−6 mol/m3 and vave = 100 µm/s. The concentration increases steeply over a short time and then it reaches a saturation value. The time (t = 1000 s) of interrupting the injection of the analyte is set manually. We defined the equilibrium binding time TR as the response time of biosensor the time for which the average concentration equals 95% of its saturation value. For a channel without obstacle, we obtained TR = 440 s whereas TR = 242 s when the obstacle is included. We observed also when the obstacle is involved, the evolution of the complex concentration declines rapidly during the dissociation phase.Figure R and improve the biosensor response. The effect of the obstacle may be explained by the deformation of the streamlines and the modification of the fluid topology as shown in Fig. et al.28. Indeed, the obstacle restrains the cross-section of the channel and leads to an increase of the velocity near the sensing area of the biosensor. Our numerical results present a good agreement with experimental results performed by Hofmann et al.19 and theoretical results obtained by Hu et al.25. Indeed, they found that the equilibrium binding time could be reduced by increasing the flow rate.The numerical result proves the usefulness of the introduction of an obstacle to reduce TTo determine the optimal position of the obstacle inside the microchannel, Fig. R corresponds to d = 100 µm. The enhancement of the binding reaction may be explained by the deformation of the streamlines and the modification of the fluid velocity field which leads to increase the velocity in the vicinity of the sensitive membrane. This deformation is more advantageous when the obstacle is located in front of the center of the sensitive membrane.The optimal position giving the smallest T33. Indeed, Fig. R. Figure Raising flow velocity is an effective means to reduce the thickness of the diffusion boundary layerR, the inlet velocity in the case of using the obstacle reduced by a factor about 10 compared with the case that the obstacle is not included. We conclude that the insertion of an obstacle inside a microchannel is an effective means to reduce the response time instead of increasing the inlet velocity.To apprehend more quantitatively the effect of the presence of the obstacle on the binding reaction, Fig. vave. For each case, the analyte concentration at the inlet is constant (=6.4 nM). This figure shows simultaneously the effect of two parameters on the equilibrium binding time. It is then possible to deduce the best way to reduce TR. It is clear that the reduction of the response time is proportional to an increase of the flow velocity. It is noted also that the optimal position corresponds to d = 100 µm.Figure 25. Indeed, Fig. R with minimized consumption of the analyte concentration by using the obstacle inside the microchannel. Therefore, to obtain a given value of the response time it is more efficient to involve the obstacle than to increase the inlet analyte concentration.The enhancement of the mass transport comes at the increase of the inlet analyte concentrationR versus the obstacle position d and the inlet analyte concentration. The inlet flow velocity is fixed at 100 µm/s. The smallest equilibrium binding time is reached when the obstacle position is placed near the reaction surface, i.e. at the position  µm which is in fact the optimal position regardless of the value of the inlet analyte concentration.Figure By an appropriate choice of the obstacle position, inlet flow velocity and the inlet analyte concentration, the response time can be significantly reduced as revealed in Figs This paper presents a numerical simulation of the effect of the flow deformations around an obstacle on the binding reaction inside a microchannel of a biosensor based on heterogeneous immunoassays. 2D numerical simulation is performed using the finite element method to study the effects of obstacle position, average inlet flow velocity, and initial analyte concentration on the response time. The results show that the obstacle yields better binding efficiency of a biosensor when it’s located in front of the center of the sensitive membrane. An appropriate choice of the inlet flow velocity and inlet analyte concentration may reduce significantly the response time.In the future, we intend to expand the investigation in three dimensional simulations and we intend to examine in more details the bio-chemical models related to the adsorption.


Background: The reproducibility policy at the journal Biostatistics rewards articles with badges for data and code sharing.  This study investigates the effect of badges at increasing reproducible research.Methods:  The setting of this observational study is the Biostatistics and Statistics in Medicine  online research archives.  The data consisted of 240 randomly sampled articles from 2006 to 2013 (30 articles per year) per journal.  Data analyses included: plotting probability of data and code sharing by article submission date, and Bayesian logistic regression modelling.Results:  The probability of data sharing was higher at Biostatistics than the control journal but the probability of code sharing was comparable for both journals.  The probability of data sharing increased by 3.9 times  after badges were introduced at Biostatistics.  On an absolute scale, this difference was only a 7.6% increase in data sharing .  Badges did not have an impact on code sharing at the journal .  64% of articles atBiostatistics that provide data/code had broken links, and atStatistics in Medicine, 40%; assuming these links worked only slightly changed the effect of badges on data  and on code .Conclusions:  The effect of badges at Biostatistics was a 7.6% increase in the data sharing rate, 5 times less than the effect of badges at Psychological Science.  Though badges at Biostatistics did not impact code sharing, and had a moderate effect on data sharing, badges are an interesting step that journals are taking to incentivise and promote reproducible research. In 2009, Roger Peng mentioned in an editorial inBiostatistics that the minimum standard that could bridge the gap between replicability and nothing is “reproducible research”1. ‘Reproducibility’ is the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator2. Reproducibility was defined by Peng in terms of sharing the data and computer code used to analyse the data and he described it as the “cornerstone of the scientific method”1. In a perspective piece in 2011, Peng likened reproducibility to a spectrum, at one end being the gold standard of full replication, and at the other, publication only3. Given the expectation that data will be accessible, researchers who refuse to share the evidentiary basis behind their conclusions, or the materials needed to reproduce published experiments, fail to maintain the standards of science4. Although in some instances highly-sensitive data cannot be shared for legal or privacy reasons.Historically, the replication of a scientific experiment has been the measure of its validity, however, not all experiments can be replicated in their totality5. Ioannidiset al. discuss changing the reward criteria to include ‘reproducible’ and ‘sharing’ using the PQRST criteria – productive, high-quality, reproducible, shareable, and translatable6. A systematic review of incentives that motivated researchers to share their data in the health and medical research community, uncovered only one evidence-based incentive that increased data sharing at the journalPsychological Science from 1.5% pre-incentive (2012) to 39.4% post-incentive (2015)8. This incentive was an open data badge developed by the Center of Open Science (COS) and introduced at the journal in January 20148.Scientific journals are critical to changing the culture of research. Many journals are introducing data sharing policies, but studies have shown that policies alone are not effective in promoting a culture of sharing and that scientists potentially need to be rewarded for good behaviourBiostatistics introduced badges, or what they called kitemarks (named after the UK kitemark system of establishing product safety), on 1 July 2009 as part of their policy to reward reproducible research1. The policy was introduced by Roger Peng, the then Associate Editor for reproducibility (AER)1. Sharing was not enforced, rather authors were encouraged to consider the reproducibility of their research1. From here on, kitemarks will be referred to as badges, using common terminology.Badges for reproducible research were not an innovative creation of COS however. The journal1. The policy rewarded articles with data available with the letterD on the front page of the published article PDF, articles with code available with aC, and articles with data and code available and which were tested for reproducibility by the AER anR for reproducibility1. It is important to note that data refers to raw data and not simulated data, which are commonly used in statistics.The reproducibility policy at the journal instructed authors to indicate in their submission if they intend to submit supplementary materials that include data, code, or bothBiostatistics provided an ideal opportunity to replicate the findings of the Kidwellet al. badge study by examining sharing rates at another journal that offered a reward or incentive for reproducible research8. We note that Kidwellet al. examined data and material sharing only, as badges were not offered for code.The policy change atNature in 2016 indicates that the scientific community is in the midst of a reproducibility crisis9. The current culture in science provides strong incentives for innovation and relatively weak incentives for certainty and reproducibility10. Within the current ‘post-truth’ era there is much public scrutiny and suspicion around the validity of science. Such a debate, compounded by the reproducibility crisis, signals a time for a cultural shift in the scientific research process11. The sharing of data, as well as the computer code used to analyse the data, should, where possible, be integral components of the research process, however data sharing rates have been as low as 0%12. Of course, not all data can be shared due to legal and ethical constraints, but these are neither the only, nor main reasons behind low sharing rates13. Scientists are still exploring the barriers towards sharing and a key concern is that researchers are not incentivised to share3.A survey conducted byBiostatistics.Our aim is to investigate the effect of badges at increasing reproducible research, specifically, data and code sharing, atStatistics in Medicine as the control journal as it did not have a badges or any type of reproducible research reward scheme during those years, but is in the same field of research with similar goals of publishing papers on statistical methods development in health and medicine. Additional control journals would have increased the representativeness of our study and increased the statistical power. However, no other similar journals from the field of biostatistics satisfied the inclusion criteria, as they all introduced a reproducibility policy before or between 2006 to 2013. Therefore, the study setting is theBiostatistics andStatistics in Medicine research archive. All the information required was publicly available online, as such participant consent was not required and an ethics exemption (exemption number: 1700001051) was granted by the Office of Research Ethics and Integrity at the Queensland University of Technology.This is an observational study with two journals, intervention and control, using a pre-post study design, with 30 randomly selected papers per year from 2006 to 2013 for each journal. We choseet al. study8. This uses a two-sided 5% significance level. We felt this sample was unrealistically small, hence we instead based our sample size on the practical considerations of reading papers and examining their data and code sharing choices, given the time constraints of the first author’s (ARF) PhD. Thirty papers per year from 2006 to 2013 for two journals is a total sample of 480 papers, which is practically possible, and provides good coverage over the time of the policy change atBiostatistics.A sample of only 19 papers per journal would have given us a 90% power to detect a difference in data sharing of 37.9%, based on the effect of badges from the KidwellFor each year and journal, a random number generator was used to select the research articles (in Microsoft Excel 2016). Articles were included if they:Generated and analysed original data (article had data and code to share), orConducted secondary analyses on a pre-existing dataset from another study (article had data and code to share), orGenerated simulated data (article did not have data to share but had code to share)Articles were excluded if:They were meta-analyses, meta-regressions, or systematic reviews, as these papers usually contain the data within the paperThey were case series, opinion pieces or some other publication type without data or codeBiostatistics articles).If an article was excluded then we sampled another article from the same year and journal to maintain the sample size. ARF read the research papers and extracted the details of the articles included in the study. Each article was screened using these search terms: “data”, “code”, “package”, “available”, “https”, “www”, “figshare”, and “github”. For the included articles, the following variables were documented: submission date, data sharing statement, data availability, hyperlink to dataset, code sharing statement, code availability, hyperlink to code, and badge allocation (forThe second author (AGB) independently assessed data and code sharing for 20 randomly selected articles. There were minor discrepancies between the authors, which were resolved by discussion.5, each research article was categorised for data and code sharing as:Using definitions from our previous workData sharingavailable: articles that had a functioning link to a publicly available dataset deposited at a third-party site or attached as supplementary material to the electronic version of the articlepotentially available: articles that indicated that the dataset was potentially available upon request from the authorsnot available: articles that did not indicate the availability of the dataset analysed in the article or where the link to the data was no longer workingnone to share: articles that used simulated data and so did not have a raw dataset to shareCode sharingavailable: articles that had a functioning link to publicly available code deposited at a third-party site, or attached as supplementary material to the electronic version of the article or available within the article itselfpotentially available: articles that indicated that the code was potentially available upon request from the authorsnot available: articles that did not indicate the availability of the code used to analyse the data (raw or simulated) or where the link to the code was no longer workingBiostatistics and using the article’s submission date as this is when authors are thinking about the journal requirements and perhaps becoming aware of the badge. Since the policy change was on 1 July 2009, papers submitted toBiostatistics after that date were in the intervention period. We included a six month gap before the policy change as an interim phase because papers submitted during this time (1 January 2009 to 1 July 2009) could experience the badge policy upon re-submission, so papers submitted in this period were categorized into the interim period. Any papers submitted toBiostatistics before 1 January 2009 were in the control period and all papers submitted toStatistics in Medicine were controls.We defined the intervention period based on the policy change date atThe first analysis examined data and code availability and probability of sharing over time using submission date. As a sensitivity analysis, we used the articles’ publication dates extracted from PubMed in place of submission date. We conducted this sensitivity analysis to examine whether the policy was associated with a change based on the very latest date that authors could make changes to their papers.We plotted the binary data and code sharing over time and included a smooth curve to estimate the mean sharing rate over time in each journal. The smooth curves were made using a LOESS smooth with a span of 0.9, and we also plotted the 95% confidence intervals. Papers where there was no data to share  were excluded from these plots.14. Due to possible convergence issues with a standard logistic regression model using a log-link to estimate prevalence ratios, we ran a Bayesian logistic regression model usingWinBUGS (version 1.3.4). Using a Bayesian model has the added advantage of giving 95% credible intervals and Bayesian p-values that are far easier to interpret than frequentist confidence intervals and p-values. The Bayesian p-values used here estimate the probability that sharing increased after the policy change atBiostatistics. As well as showing the change in data and code sharing probability, on the relative scale, of the prevalence ratio, we also show the absolute increase in sharing probability after the policy change together with 95% credible intervals.To test for a difference in the probability of making data and code available after the introduction of badges, we used logistic regression and presented the results as prevalence ratios rather than odds ratios, as prevalence ratios are generally easier to understandBiostatistics andStatistics in Medicine from the same year. We did this to adjust for other changes over time, for example a potential increase over time in data and code depositories such as GitHub, Figshare, and Dryad, and a potential decrease in data and code availability for papers published many years ago because of broken links15.In a sensitivity analysis we used a strong control for time by including year as a random effect, assuming that each year has its own data sharing rate. This essentially matches papers fromBiostatistics indicated that when the publisher (Oxford) switched to a new publishing platform in January 2017, some of the supplemental material was lost in the transfer . As such, we conducted a sensitivity analysis assuming these broken links worked before Oxford changed publishing platforms.The current editors ofstatistical software R (version 3.2.3).The data analysis was made using theBiostatistics had broken links and atStatistics in Medicine, 21 out of 53 (40%) articles that provided links to data and code had broken links. We examine the impact of these broken links in sensitivity analyses.We often encountered issues with broken hyperlinks at both journals. Forty-nine out of 76 (64%) articles that provided links to data and code atBiostatistics had 8 articles with no data to share, bringing the sample with possible data available to 232; 20 of which had data available, 3 had data potentially available and 209 had no data available.Statistics in Medicine had 31 articles with no data to share, bringing the sample with possible data available to 209; 2 of which had data available, 4 had data potentially available and 203 had no data available.Flow charts show the frequency of data and code availability for each journal .BiostatBiostatistics, and 1 January 2009, six months prior to the policy change (interim period). It is clear that data availability and probability of sharing were greater over time inBiostatistics than in the control journal,Statistics in Medicine, but the probability of sharing data atBiostatistics was still low, at well below 0.25. Interestingly an increase in data sharing atBiostatistics took place before badges were introduced at the journal. The results of the sensitivity analysis using publication date are shown inBiostatistics before badges were introduced. The results of the sensitivity analysis assuming the broken links were working using submission date as the time variable are shown inThe data available and probability of sharing by submission date together with a smooth mean and 95% confidence intervals are inCode availability over time.Statistics in Medicine had 24 articles with code available, 27 potentially available, and 189 with no code available, whileBiostatistics had 14 articles with code available, 22 potentially available, and 204 with no code available.The frequency of code availability for each journal is inBiostatistics andStatistics in Medicine are mostly on top of each other in this graph, except for a drop-off in sharing atBiostatistics in later years. This indicates no great difference in code sharing at these journals.Biostatistics andStatistics in Medicine are again mostly on top of each other, showing an increase in code sharing over time at both journals, but around mid-2011 the two curves diverged, withStatistics in Medicine showing an increase in code sharing andBiostatistics a drop. The results of the sensitivity analysis assuming the broken links were working using submission date as the time variable are shown inThe code availability and probability of sharing by submission date together with a smooth curve and 95% confidence intervals are inThe logistic regression model estimated that the probability of data sharing increased by 5.7  times that of the control period in the interim period of 1 January 2009 to 1 July 2009. This Bayesian p-value gives an estimated 94.7% probability that the mean rate of sharing increased. After the interim period, the probability of data sharing increased by an estimated 3.9  times after badges were introduced. On an absolute scale, this difference was only a 7.6% increase in data sharing (95% CI: 2 to 15%). After controlling for time, badges increased the probability of data sharing at the journal by an estimated 4.9 times . This is comparable to the prevalence ratio of 3.9 when time was not added as a random effect, which shows that controlling for time only slightly increased the effect badges had on the probability of data sharing. After assuming the now broken links were working at the time of publication, the logistic regression model that controlled for time gave a slightly different estimate of the mean effect of badges from the previous 7.6% to 6.7% .During the interim period, badges did not have an effect on code sharing . After the interim period there was an estimated 0.61% increase  in sharing. After adjusting for time, this absolute difference reduced to –1.4% . This suggests that badges did not have an impact on the probability of sharing code. After assuming the now broken links were working at the time of publication, the logistic regression model that controlled for time estimate a slightly changed mean effect of badges from the previous 0.61% to –2% .et al. badge study8 cannot accurately deduce the effectiveness of badges because of the biases of the non-randomised study design. The Kidwellet al. 2016 badge study received criticism from Hilda Bastian on its study design, analyses, and claims16. One of the criticisms was that the badges scheme was not the only intervention offered at the journal, there were four other co-interventions offered in 2014, and so any effect could not be attributed to badges alone16. Bastian reasonably argued that to isolate the impact of badges, groups that had the same conditions except badges were needed16. Our study is also exposed to similar limitations with regard to confounding as other changes may have occurred that we were not aware of. However, we can derive some insight into the effect badges had on data and code sharing from the results of both observational studies.The results of this observational study and those of the related KidwellBiostatistics, the probability of data sharing increased 3.9 times. This prevalence ratio might seem like a large increase but on an absolute scale it is only a 7.6% increase in the rate of data sharing, which is much lower than the 37.9% effect of badges atPsychological Science8. When the now broken links were assumed to indicate sharing, the badge effect reduced slightly to 6.7%. The large difference between the effect of badges atBiostatistics andPsychological Science could be related to differences in the culture of sharing between the two fields, and the timeframes of the studies: 2006 to 2013 for our study, versus 2012 to 2015 for Kidwellet al. Our study analysed incentives for data and code sharing at an earlier time when the reproducibility crisis was not yet a testified reality, hence researchers may have been more primed to change behaviour in the Kidwellet al. study. Also, since statisticians typically re-analyse existing datasets, it might be harder for them to share the data as they might not have the rights. This is contrary to research in psychological science where original data is normally collected and analysed, making sharing a potentially simpler task.After the introduction of badges atBiostatistics  in the article. The policy atBiostatistics included referencing “…software that is widely available from central repositories ”. It is true that providing a link to a third-party repository where software packages are deposited, such as vignettes, typically contain some general code, but it often takes specialized skills to work out the code at these repositories, and they might not always explain the analyses covered in the actual published article. This is in line with what Stoddenet al. recommended in their piece on reproducibility inScience, “Data and code underlying discoveries must be discoverable from the related publication, accessible, and reuseable”18.Badges did not appear to have an effect on code sharing as the prevalence ratio was 1.1 When the now broken links were assumed to indicate code sharing, the badge effect on code changed slightly from 0.61% to –2%. This is an unexpected outcome as code is of great importance in the field of biostatistics. A possible explanation behind the lack of badge effect on code sharing could be our definition of code sharing, which might seem traditional compared with the reproducibility policy atBadges have been promoted as a simple solution because they are low cost. However, while collecting data for our study, we noticed that articles did not always appear to be allocated with badges correctly, implying that assigning badges is not always clear cut and journal staff may need to spend more time on verification. An alternative approach is that peer-reviewers check for data and code availability and assign badges as part of the standard peer review process. It could be that peer-reviewers prefer to have access to the data and code in order to review the article anyway, so this model might work, but it still requires additional time and effort on their part and as they receive little recognition for their work, plus it might be unfair to expect all peer-reviewers to check for data and code sharing.19. Data and code sharing is an indispensable part of the movement towards science that is open; where scientific truth is not a questionable commodity, but is easily accessible, replicable, and verifiable20. The cultural shift towards reproducible science is complex and it calls for a twofold change in the attitudes of individual researchers toward reproducibility, and the leadership provided by the systems and services that support scientific research. As such, journals, universities, government bodies, and funders are key players in promoting this culture. Transparency and reproducibility are elements central to strengthening the scientific method, and data and code provide the key to scientific truth12. As Peng argued inScience, the culture of reproducibility will not drastically change overnight, but simply bringing the notion of reproducibility to the fore and making it routine will make a difference3. Badges are already being used by journals includingBiostatistics,Psychological Science,British Medical Journal Open Science, andAssociation for Computing Machinery to encourage researchers to share the evidence behind their work21. Based on this observational study and a previous study, it appears that badges do help to increase data sharing, but a randomised trial is needed to better estimate their true effect, as well as studies of the additional time needed to implement and maintain them.Efforts are underway by the global meta-research community to strengthen the reliability of the scientific methodhttps://doi.org/10.6084/m9.figshare.5687548.v222Anonymised data and the R code used in the analyses are publicly available at:An ethics exemption was granted by the Office of Research Ethics and Integrity at the Queensland University of Technology for this study (exemption number: 1700001051). No consent was needed as all data collected and analysed in this study were publicly available. The authors did a great job and addressed the comments to the first version of this manuscript in good way. Thus I fully approve of this version!I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The revision addresses the issues raised in the first round of review. The rationale for inclusion is now clearer, and the sensitivity analyses are helpful. I am happy to approve this article again.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Biostatistics - which introduced badges in 2009 - with a control journal, Statistics in Medicine. The main finding is that there seems to be a small effect on data sharing associated with the introduction, while badges appear to have no effect on the sharing of code. This study evaluates the effects that badging has on availability of both data and source code. Badges that highlight and reward articles that share data/code have been described as a simple way to increase sharing behavior and thus reproducibility. This makes this study highly relevant and interesting to the field of meta-research. To measure whether badging has this desired effect the authors compare the availability of data/code in the journal Statistics in Medicine is used as a control journal it is unclear how much of these differences are cultural between the audiences of the two journals . A larger set of control-journals would potentially minimize this 'culture'-effect . While the statistical methods are appropriate and sound, the study has some limitations given by the data set on how the authors coped with missing data. First of all, and this is probably the main limitation, there is only a small set of publications in the authors' set of 480 that have data/code associated with them. Only around 8% of all publications have code attached, a number that drops to less than 5% for data. Given this overall low rate of sharing - and the large time span covered - there is a lot of fluctuation in the observed sharing behavior for both code and data  between the two journals. Given that only 1). In line with this the authors find that the sharing of data in Biostatistics already increased prior to the introduction of badges, even when going for conservative measures. Additionally there is an observed increase in data sharing in Statistics in Medicine in later years, albeit from a lower baseline. I think this general trend should be included in the discussion.There is a general trend that "Open Science" and data sharing are becoming more popular Biostatistics and highlight the issue that OUP had with losing (not only) supplementary data. The authors treat these articles as having data/code not publicly available. This not only leads to a marked decrease overall decrease of articles with data/code available but can furthermore be a potential source of bias. For OUPs Molecular Biology and Evolution it was seen that these losses on behalf of OUP not only affected supplementary materials but even individual years worth of full text articles (c.f. http://rossmounce.co.uk/2017/04/02/open-letter-to-oxford-university-press/). If OUP also lost the supplementary materials of certain publication date ranges forBiostatistics this will heavily bias the analysis presented here. The authors additionally find that the links for 64% of articles that provide data/code are broken in  In addition to this there are further effects that complicate a clear inference of the effect of badges: Given these limitations I would be cautious to assume whether badges have a positive effect or not on data availability at this point.2) I recommend taking the original data/code availability statements at face value and not treat broken links as publications without data/code. I think it is defendable to assume that the data/code was available at the time of publication when the main text suggests so. Doing this the potential effect of badges should become more pronounced as it is not hidden by the 'link-rot' that OUP is responsible for.   To improve the resolution by adding more data and to decrease the biases mentioned in Overall, I think this is a valuable contribution to the field of meta-research and our understanding of incentives around sharing and reproducibility. I hope that the minor improvements suggested above can be useful to make this data set even more useful.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Biostatistics were the following:We looked for additional control journals to minimise the ‘culture’-effect described above.  The journals that met the eligibility criteria of being in the field of applied statistics and/or biostatistics and which had a similar relative high prestige in the field as perStatistical Methods in Medical ResearchBiometric Journal BiometricsRoyal Statistical Society – Series C Applied Statistics   However, none of these journals were suitable as controls as they all had reproducibility policy changes during the time frame of our study (2006 to 2013) or had a reproducibility policy in place before that timeframe.  We have added an explanation of this in our methods.     The general trend towards more open science is now discussed in the Discussion. We used sensitivity analyses that adjusted for the effects of time (including trends), and the effects of badges were relatively similar to the unadjusted results. We have conducted sensitivity analyses for data and code sharing to account for the broken links at the journal. These results have been added and discussed in the article. Biostatistics) to one that did not (Statistics in Medicine). The main finding was that badges were associated with an increase in open data publication, although with a considerably smaller effect than that observed in the one previous investigation of this question. The finding is important and may influence publishing policies.This study is an observational investigation of the effect of badges/kitemarks on sharing of data and code. The authors compared one journal that introduced badges in 2009  as a reference period is arbitrary, and it is not possible for a reader to assess whether six months would capture most papers undergoing review and revision at the time the policy was introduced. If papers took longer, that could contribute to the increasing rates of data sharing observed in the run-up to the introduction of badges, seen in figure 3. Thus, the choice of reference period is likely to yield a conservative estimate of the effect of the policy. Papers were coded by one person only, and that is a minor weakness. My experience with coding articles for open data and similar outcomes leads me to think that the main reason to have two coders in this context is to reduce the risk of error, rather than to handle interrater differences in assessments. It is not likely, in my opinion, that errors in the coding have effects on a magnitude approaching that seen for the intervention. I have cursorily reviewed the data and analysis code also. The data variable names are self-explanatory. The code, written in Markdown, is sufficiently documented with inline headings. However, the code requires some modification before it can be run independently, and this is not documented. For example, on line 18, the code attempts to load the data, by calling a file that is not provided. The data must instead be loaded by opening the provided R workspace file. I suggest that the data be made available in a csv or txt file, which is safer for accessibility in the long term and across softwares. The code could then call this file. The ancillary finding of high rates of broken hyperlinks to data at both journals is interesting, as is the explanation that supplementary data were lost by one of the journals during a platform migration. I have several times advanced the argument that this risk is one motivation for publishing data and code using dedicated repositories, but I have not previously seen an empirical observation such as this one. I suggest that this finding should be mentioned in the abstract. One minor detail: In figure 4, the legend does not explain the difference between panels a and b, as in figure 3. In summary, this article makes an important contribution towards our understanding of effects of badges on publication of open data and open code. I am happy to approve this article.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. We have updated the Rmarkdown code and added an R data set so anyone can independently run the analysis. Although ARF coded all the data, AGB verified 20 observations and the two authors debated a handful of cases where it was not clear. In general the sharing of data or not was relatively clear-cut and hence we believe that one coder should be sufficient. We agree that the six month date window prior to the introduction of badges is arbitrary, and this is why we also plotted the sharing rates over time so that readers could see when any changes in sharing took place, and why we tried the alternative time scale of submission date. We have now mentioned broken links in in the abstract. We have corrected this: "One minor detail: In figure 4, the legend does not explain the difference between panels a and b, as in figure 3".