DNA microarrays are a powerful technology that can provide a wealth of gene expression data for disease studies, drug development, and a wide scope of other investigations. Because of the large volume and inherent variability of DNA microarray data, many new statistical methods have been developed for evaluating the significance of the observed differences in gene expression. However, until now little attention has been given to the characterization of dispersion of DNA microarray data.® with 22 different types and we demonstrate that the Gaussian  frequency distribution is characteristic for the variability of gene expression values. However, typically 5 to 15% of the samples deviate from normality. Furthermore, it is shown that the frequency distributions of the difference of expression in subsets of ordered, consecutive pairs of genes (consecutive samples) in pair-wise comparisons of replicate experiments are also normal. We describe a consecutive sampling method, which is employed to calculate the characteristic function approximating standard deviation and show that the standard deviation derived from the consecutive samples is equivalent to the standard deviation obtained from individual genes. Finally, we determine the boundaries of probability intervals and demonstrate that the coefficients defining the intervals are independent of sample characteristics, variability of data, laboratory conditions and type of chips. These coefficients are very closely correlated with Student's t-distribution.Here we examine the expression data obtained from 682 Affymetrix GeneChipsKα coefficients defining these intervals are invariant; these coefficients offer a convenient universal measure of dispersion of data. The fact that the Kα distributions are so close to t-distribution and independent of conditions and type of arrays suggests that the quantitative data provided by Affymetrix technology give "true" representation of physical processes, involved in measurement of RNA abundance.In this study we ascertained that the non-systematic variations possess Gaussian distribution, determined the probability intervals and demonstrated that the This article was reviewed by Yoav Gilad (nominated by Doron Lancet), Sach Mukherjee (nominated by Sandrine Dudoit) and Amir Niknejad and Shmuel Friedland . Reviewed by Yoav Gilad (nominated by Doron Lancet), Sach Mukherjee (nominated by Sandrine Dudoit) and Amir Niknejad and Shmuel Friedland . For the full reviews, please go to the Reviewers' comments section.®, we first have to derive a representative value of gene expression from the signals of individual probes . The Affymetrix MAS 5 and GCOS use Tukey's biweight algorithm and yield an absolute expression value for each probe set . The method of low-level analysis, developed by Li and Wong  is de is de®, So far, very little attention has been given to the general properties of the dispersion of gene expression levels. With respect to applicability of various statistical methods it is useful to know how the standard deviation behaves across the expression range and whether this behavior is consistent from one assay to another and among the different types of arrays. Verification of normality of the frequency distribution of random fluctuations is particularly relevant. All parametric methods are based on concordance of the observed frequency distribution with the normal (Gaussian) distribution. Most physical and chemical systems, where random variations result mainly from collective interactions of large ensembles of particles, exhibit frequency distributions close to the Gaussian. The underlying mechanisms of microarray data variability are certainly of the same nature as the collective phenomena in physical systems but the ensemble of the processes involved is so complex that one would expect some compound distribution, far from the simple form expressed by the Gaussian prototype.Kα. It was found that the values of Kα coefficients obtained using various arrays are, at least in the first approximation, invariant. Finally, we compare the probability of coefficients Kα with the corresponding values of inverse t-distribution.The object of the present study is to examine the frequency distributions, general properties of the standard deviations and coefficients of the probability intervals. It was found that the general characteristics of dispersion are useful for quality control, reduction of a system dimension and other purposes. Firstly an overview of the frequency distributions is given for both replicate arrays (five or more replicates) and consecutive sampling of the expression difference in the ordered pairs of genes in two-array comparisons. Subsequently, we describe the consecutive sampling analysis and evaluation of the linear characteristic function, approximating the standard deviation of the data variability across the arrays. The standard deviation function is then employed to define the probability intervals encompassing specific percentages of the observed values. The boundaries of these intervals are defined by probability coefficients In the present investigation we analyzed 682 Affymetrix microarrays of 22 different types. Our main objective was to study the microarray data derived from particular biological investigations, generated in many different microarray core laboratories, rather than the sets of arrays produced in the context of technology development or testing methods of analysis. Only a few "testing" sets were included. We evaluated the CEL files using MAS 4  and employed the "Average Difference" as expression signal value. Because MAS 5 and GCOS distort the frequency distributions in the near-zero region by ignoring the negative values, MAS 5 and GCOS outputs are not suitable. Prior to the analysis, we verified the linearity and quality of the data, in particular, the absence of clusters with significantly different expressions. All data on each array were normalized to 100% of the array mean; all Affymetrix control genes were excluded.®. Typically, for good-quality data, between 85 and 95 percent of samples passed the test. Moreover, a limited number of tests using the data obtained from fiberoptic bead-based oligonucleotide microarrays by Illumina led to the same conclusion  uses for ANOVA the threshold of 0.01), nonetheless the noted deviations call for caution when using parametric methods, in particular considering that every analysis involves multiple testing. Our conclusion differs from that of Gilles and Kipling , Churchill, BioTechniques (2004), 37:173 [c], Kerr, Churchill. Genet Res. (2001), 77:123 [d], Kerr, Churchill (2001), PNAS, 98:8961 [e], Kerr et al. (200), J. Comp. Biol., 7:519 [f], but we did not find confirmation of normality in dispersion of single-color microarrays. In case of the spotted two-color microarrays the authors detected non-normal distribution of residues [d-f]First we would like to thank Dr. Gilad for his review and for bringing up an interesting issue of the MM signals. Regarding the question of normal distribution, we looked over Dr. Churchill's papers dealing with microarray technology, including Cui et al. Biostatistics (2005) [a], .Regarding the question of usefulness of MM signals: Our main objective is to show that the dispersion across all types of arrays, experimental conditions and organisms exhibits some common basic properties and we did not intend to make a case in favor of the Affymetrix approach. Nonetheless, we feel that the fact that such common description can be found supports reasoning of the Affymetrix.The discussion is still on and various arguments have been brought up both for and against using the MM signals. We fully agree with the reviewer that a major part of the signal of MM probes is due to the "specific" RNA of interest. However, this is to be expected, since among all RNA molecules attaching to the MM probe, the particular RNA of interest is most likely closest to its structure. It is exactly the ability to distinguish between the perfect match and "almost-perfect-match" that makes the measured signal reliable. If a substantial quantity of the specific RNA is present and the signals of both PM and MM are equal or MM exceeds PM, it suggests either saturation or low distinguishing power. Under such circumstance the MM signal provides useful information, indicating that the particular PM signal might not reflect the true RNA concentration. In case of saturation, taking the PM signal only would correctly indicate that the specific RNA is present, however, the relationship abundance-signal would be strongly nonlinear. As we mention in the Discussion, the results of various studies aiming at validation of different approaches are inconclusive. Evidently, more research is needed to establish the optimal technology and corresponding statistical procedures. It is likely that no single methodology could be found universally optimal and different circumstances would call for different approaches.Sach Mukherjee, Department of Statistics, University of California, Berkeley, CA, 94720-3860, USA The authors present an empirical study of the distributional characteristics of data from Affymetrix gene expression microarrays. One of the questions posed at the outset concerns the relationship between the mean and variance of microarray data  and is subsequently answered in the following way: "...the standard deviation is linearly proportional to the mean expression level" . However, this latter finding seems widely recognized already, and has been discussed in some detail in the literature . Yet none of these papers are cited in the article.Author Response: We agree that the fact that the standard deviation in the high region is proportional to the signal and at the low end it does not converge to zero has been generally accepted, but we are not aware of the study that systematically verified the linear relationship. Rocke and coworkers derived a similar model from theoretical considerations and corresponding references were included.The authors also criticize the use of log-transformation  (again without referring to the literature on the topic) but then seem to use just such a transformation as a pre-processing step before regression . Yet under a data model with both multiplicative and additive noise, data are only log-normally distributed at high expression levels . Furthermore, log-transformation may inflate the variance of observations with low expression levels. Indeed, the authors find that "...larger percentages of failures [in passing a K-S test of Normality] occur in the near-zero region." . Might not this effect simply be due to the log-transformation?Author Response: It appears that our procedure was not clearly described and we revised the text accordingly. We actually use the logarithmic transform only in the regression procedure to balance the residuals, i.e. to prevent the residuals of the high-expression genes to outweigh the low-expression range. Thus instead of regressingSD(Ymean) → a1 + a2Ymeanwe fitSD(Ymean)) → log(a1 + a2Ymean)log data and not log-transformed data. This is the only occasion when we use the log-transform, in all other procedures we employ non-transformed normalized data.This reviewer found the approach taken to "consecutive sampling" in studying the mean-variance relationship in paired arrays somewhat ad hoc. For example, the 250 probe sets having highest expression level are excluded from the analysis. What effect does this exclusion have on the analysis? Is it appropriate to leave out data (arguably some of the most interesting data) from an empirical study of this kind? This issue is not really discussed. The authors also state that "we usually require the mean discrepancy of the ten consecutive samples below 1%" . Does this mean the data are ignored if the discrepancy is higher than this threshold?Author Response: The consecutive samples provide reliable representation of the standard deviation only if the within sample differences of means are small, say below 1%. At the maximum of the expression range the density of the points is small and, consequently, differences in the mean values are large. To obtain dependable coefficients of the standard deviation function these data have to be excluded from the regression procedure. Subsequently, we assume that validity of the standard deviation characteristic function can be extrapolated to the maximum expression value. Indeed, in the differential expression analysis all data to the maximum expression are included. The text was revised to avoid misunderstanding.Regarding a discrepancy in identity (4): The consecutive sampling program automatically keeps track on assumptions and signals detected problems. Identity (4) is a convenient check of the variability of means and, generally, of the reliability of characteristic function; it is typically fulfilled within 0.1%. If the difference between the right-hand and left-hand sides of the equation exceeds 1% the program raises a flag, indicating problematic data. In such case the researcher conducting analysis examines the data and determines the reason for discrepancy; if no corrective measure can be found, the sample is excluded from the analysis.The presentation of mathematical details is not always very clear in the paper. Equations (2) and (3) would benefit from either a derivation or a reference. Equally, some of the phrasing is somewhat difficult to interpret, e.g. "...we can calculate an estimator of the standard deviation of gene expressions variability of a population of replicate arrays from two-array comparisons" .Author Response: Derivation of equations (2) and (3) is described in The sentence in question was reformulated.Kα coefficients with t-distribution, §5 onwards] puzzling inasmuch as it did not relate to, or strengthen, any of the main arguments of the paper. The case presented was also far from convincing: given that there are so many existing methods for detecting differential expression, it is surely reasonable to expect any new method to be accompanied by strong empirical evidence and/or theoretical arguments in its favor.Finally, this reviewer found the introduction of a new methodology for finding differentially expressed genes [Results, Probability intervals and correlation of the Author Response: Actually, the application of the probability intervals to the differential expression analysis had not been included in the earlier versions of the manuscript. However, during the internal reviews we frequently encountered a question "how can the dispersion analysis and probability intervals help biologist to analyze data and to detect significant differences in gene expression" . To answer this question we included a brief description of the consecutive sampling and coincidence analysis, which we use as a standard procedure, usually in combination with the RMA and/or other approaches. To provide better description we revised the text and included an additional reference.In conclusion, the basic idea behind the paper, of characterizing microarray data distributions using a large set of real-life experimental data, is a very good one, but the paper is not well tied to the literature and suffers at times from a somewhat ad hoc approach.References:Rocke DM, Durbin B. Approximate variance-stabilizing transformations for gene-expression microarray data. Bioinformatics. 2003 May 22; 19 (8):966-72.Durbin BP, Hardin JS, Hawkins DM, Rocke DM. A variance-stabilizing transformation for gene-expression microarray data. Bioinformatics. 2002; 18 Suppl. 1:S105-10.Huber W, von Heydebreck A, Sultmann H, Poustka A, Vingron M. Variance stabilization applied to microarray data calibration and to the quantification of differential expression. Bioinformatics. 2002; 18 Suppl. 1:S96-104.Amir Niknejad and Shmuel Friedland, Department of Mathematics, Statistics and Computer Science University of Illinois at Chicago 851 S. Morgan Street Chicago, IL 60614 USA The paper addresses issues related to analysing DNA Microarrays data focusing on differences of gene expression. The paper is an extension of previous paper of J.P. Novak (reference# 8) by employing various parametric and nonparametric statistics tools and extensive use of statistical packages for very large data sets. The premise of the paper is that the standard deviation of samples of difference of gene expression in DNA microarrays is a linear function of their mean. The paper is a very good work in the area of quality control of Data in DNA Microarrays and certainly a contribution to the field. There are several points that the authors should clarify:1. The authors mentioned that "majority of microarray samples (85%–95%) conform to a Gaussian distribution". What is the reason for the rest of 5%–15% of microarrays sample which do not conform with normality? Is it a biological reason or just manufacturing technology problem?Author response: We thank Dr. Niknejad and Dr. Friedland for the very helpful review. In response to the question above: According to our extensive experience with the Affymetrix arrays and limited experience with the Illumina fiberoptic bead-based oligonucleotide microarrays, the manufacturing technology is an unlikely reason. The outliers, the most frequent cause of non-normal distribution, are probably caused by random fluctuations in the experimental procedures, such as hybridization or labeling. A discontinuity in the frequency distribution  or its derivative is difficult to explain. 2. The authors mentioned that "filtering 15% of genes would enhance the performance for clustering methods". The question is how is this filtering being done, and what is its effect on the data set as a whole and the biological ramification of it.Author response: Generally, all clustering methods are sensitive to noise, however, the problem is more difficult in unsupervised clustering, where members of presumed classes are unknown. Hierarchical clustering and principal component analysis appear to be among the most sensitive, while self organized maps are more robust. Approach to the problem and optimal percentage of the probe sets filtered out depends on a given set of data. Actually, we did not specify percentage in the text – fifteen percent is relatively low and should refer to the set used for analysis, usually reduced by eliminating probes with overall near-zero values. For the clustering procedures we raise the required threshold and try to identify "informative" probe sets, i.e. probe sets likely to be characteristic for presumed classes. Very small groups are virtually impossible to discover in noisy data, so we assume some minimum number k of samples in any particular group – say k ~ 5. Then we select only the probe sets, with k or more expression at least r-fold larger or s-fold smaller than the total median; typically 2 < r, s < 5. It is important to repeat the clustering procedure for several sets of parameters to ensure that the identified classes are independent of filtering constanst.We are also concerned with mostly focusing on Affymetrix technology for coming up for means of quality control receipt. It will be a good idea to see how their model fares for other brands of microarrays.Author response: Our experience is limited to the Affymetrix and Illumina microarrays. However, in our opinion, it is likely that dispersion characteristics of all single-color arrays are similar.It would be helpful if the authors mention how their findings can help molecular biologists to make inferences about gene expression data of various microarray data sets and their biological implications.Author response: Beside filtering of the randomly variable probe sets in noisy data the most practical application is combination of the consecutive sampling analysis and coincidence testing applied to evaluation of the observed differences between experiment and control arrays. The unique advantage of this approach is that it can be applied to assays with small number of replicates (two or more). It is a nonparametric method, theoretically equivalent to repeated random selection, and it is easy to estimate the probability of false positives. Moreover, it is based on pair-wise comparisons and it enables automatic detection of problematic arrays. We extended the discussion of this application in the section Results and added two references.Derivation of equalities (2) and (3).Click here for file


Whereas ultra-short stay  following breast cancer surgery was introduced in the US and Canada in the 1990s, it is not yet common practice in Europe. This paper describes the design of the MaDO study, which involves the implementation of ultra short stay admission for patients after breast cancer surgery, and evaluates whether the targets of the implementation strategy are reached. The ultra short stay programme and the applied implementation strategy will be evaluated from the economic perspective.The MaDO study is a pre-post-controlled multi-centre study, that is performed in four hospitals in the Netherlands. It includes a pre and post measuring period of six months each with six months of implementation in between in at least 40 patients per hospital per measurement period.Primary outcome measure is the percentage of patients treated in ultra short stay. Secondary endpoints are the percentage of patients treated according to protocol, degree of involvement of home care nursing, quality of care from the patient's perspective, cost-effectiveness of the ultra short stay programme and cost-effectiveness of the implementation strategy. Quality of care will be measured by the QUOTE-breast cancer instrument, cost-effectiveness of the ultra short stay programme will be measured by means of the EuroQol (administered at four time-points) and a cost book for patients. Cost-effectiveness analysis will be performed from a societal perspective. Cost-effectiveness of the implementation strategy will be measured by determination of the costs of implementation activities.This study will reveal barriers and facilitators for implementation of the ultra short stay programme. Moreover, the results of the study will provide information about the cost-effectiveness of the ultra short stay programme and the implementation strategy.Current Controlled Trials ISRCTN77253391. Opinions on more optimal working methods ('best practice') for breast cancer treatment have been changing continuously-6. IncreBreast cancer causes a prominent burden on the health care budget. Of the total costs of breast cancer treatment 35–50% is spentA decrease in admission time should not affect quality of care in a negative way. Therefore, it is important to introduce a structured programme such as has been developed at University Hospital Maastricht (uhM) that includes current quality of care criteria,5,19. MoAt the uhM, an ultra short stay programme for breast cancer surgery was developed and evaluated, incorporating the aspects mentioned above. The programme consists of a structured care organisation including education and counselling, dedicated anaesthesia, active participation of the patient in her own treatment plan and in the decision to go home, and home care nursing. NABON guideline quality indicators are also incorporated in the programme . AlthougTraditionally, implementation strategies have been chosen for pragmatic reasons guided by personal acquaintance with a particular strategy. In this1) how is current care organised?2) how does it deviate from optimal care according to the innovative care programme?3) what are barriers and facilitators for implementation of the ultra short stay programme?The uhM experience accumulated valuable insight and knowledge about implementation of an ultra short stay programme for breast cancer. If such a programme is implemented on a wider scale, in other hospitals, these experiences may contribute to the design of implementation strategies that are tailored to local needs.The implementation strategy will be developed on the basis of the results of the diagnostic problem analyses, and tailored to the needs of the stakeholder groups. The insights generated from this study may be translated into generally applicable insights on implementation of the innovative care programme on a larger scale while again defining patient benefits and demonstrating cost containment through the type of care provided.The aim of this study is to implement a programme incorporating breast cancer surgery in an ambulatory/24 hour stay setting, .To evaluate:1) whether the percentage of patients treated in ultra short stay increases without an increase in the number of complications.2) whether the quality of care as perceived by patients between the baseline measurement and measurement after implementation does not deteriorate.3) the cost-effectiveness of the ultra short stay programme versus care as usual (programme cost-effectiveness) in the baseline period.4) the cost-effectiveness of implementation of the ultra short stay programme versus 'doing nothing' (policy cost-effectiveness).5) the actual performance of the professionals in the test hospitals according to performance indicators.6) the perceived barriers and facilitators for the implementation of the programme incorporating surgery in ambulatory/24 hour stay setting in the participating hospitals so others preparing to change to ultra short stay care can profit from this information.The MaDO study is a multi-centre pre-post uncontrolled prospective study and involves breast cancer patients who are operated on with curative intent. Both pre and post- measurements last six months with an implementation period of six months in between.The study is performed in four hospitals which have been selected on three criteria. First, all participating hospitals had to be recognised as early adopters (an active group with high status within the target group) within tAll consecutive patients from the four participating hospitals, aged over 17 years, diagnosed with breast cancer, and scheduled to undergo surgery are eligible for participation in the study.Concerning surgical techniques, there is no contra-indication for ultra-short stay. Patients whose physiology impedes participation, as assessed by the breast surgeon and breast nurse, are excluded from the study. Patients who cannot rely on sufficient informal care in the home situation during the first night following surgery and patients with complaints that necessitate postoperative monitoring  are scheduled for at least one overnight stay. This decision is made in consultation with the anaesthesiologist. The breast nurse scores the reasons for patients who do not participate.Eligible patients are informed about the study by the breast nurse during a consultation separate from the consultation in which they are informed about the diagnosis breast cancer. Patients are given an informed consent form by the breast nurse when they are informed about the study, and they are asked to return the signed informed consent within a week. The informed consent regards consent to being asked to complete questionnaires with the aim to evaluate the cost-effectiveness of (the implementation of) the innovative care programme.In the uhM the percentage of patients discharged within 24 hours has increased from 13% to 84%. Our assumption is that the clinically significant difference of ultra short stay admissions between the pre and post measurement is at least 30% in all participating hospitals. To achieve this percentage, 40 patients are sufficient for both the pre and the post measurement in each of the hospitals to demonstrate statistical relevance (p < 0.05) with a power of 0.90.The intervention in this study concerns the comprehensive care programme for breast cancer surgery in ultra short stay (= ambulatory/24 hours stay setting). The key figures in the programme are the multidisciplinary team, the breast nurse, and the patient.Traditionally, the surgeon has been the specialist who analyses the type of breast pathology presented. Preventive procedures, early diagnosis of non-palpable lesions, breast conserving therapy, and lymph node sparing therapy have raised possibilities to reduce the burden of breast (cancer) surgery. The role of the breast nurse was introduced to improve patient counselling. Through these developments, diagnosis and treatment of breast pathology have gained a more multidisciplinary character. A decrease in the burden of surgery may limit the need for hospital-based supportive care. Similarly, this leads to a demand for strict coordination of the different steps and disciplines involved. Moreover, responsibilities for aspects of care need to be reallocated to other persons: from hospital- based supportive caregiver towards informal caregiver, from clinician to nurse specialist, from in house nursing staff to outpatient nursing staff, from hospital-based nursing staff to home care nursing staff etc.While the clinician's activities are limited to solving medical problems, the breast nurse performs all coordinating tasks to create a programme that runs smoothly.The patient is the key figure in the process. In a one-hour consultation, following the preoperative diagnosis, the patient is informed by the breast nurse about all aspects of surgery . A checklist is used for this consultation and all oral information is supported by written information. Parts of this information (e.g. information on home care nursing and what to do in case of problems) are repeated on the morning of surgery, and at the moment of discharge.The first step in the implementation process concerns the assessment of the guideline recommendations by consensus among experts, who define the complete set of recommendations (n = 29) as well as a set of key recommendations Table .A so-called diagnostic analysis assesses the usual care, and explores factors that impede or facilitate the bridging of the gap between usual care and the programme to be implemented. Information for the diagnostics analysis will be retrieved from interviews during outreach visits, meetings, and telephone and e-mail conferences. The results of the assessment of usual care in the participating hospitals are compared to the standards of the innovative care programme. Information on perceived barriers and facilitators for implementation of the ultra short stay programme will be collected during outreach visits to the hospitals. These visits will be conducted in the recruitment and preparation phase. They will be aimed at preparation of the actual implementation and listing of the barriers and facilitators that were perceived by the participants for implementation of the care programme. This set of barriers and facilitators will be described on different levels: study guideline, care provider, patient, colleague, organisation, and financial resources and reimbursement.Subsequently, a hospital-specific strategy is applied to implement a hospital-tailored version of the comprehensive care programme. This strategy is aimed at overcoming the barriers. Multi-faceted implementation strategies are used, providing insight, inducing change and acceptance, and feedback to maintain changes. These strategies are based on several components: 1) The promise that each early adopter makes to the project leader (MvM) to appoint a breast nurse fully available for coordination for the programme; 2) High-frequency outreach visits and study group meetings representing all disciplines involved in breast cancer care, provide the forum in which all steps for implementation are prepared, and issues are discussed and resolved. This local working party is also responsible for the communication of progress of implementation and of the findings within the hospital, to patients, and to other involved parties.; 3) The care process is measured through indicators mentioned in the Case Record Forms that are scored within the hospital, followed by feedback on performance by the researchers.- Information on the achievement of the targets of the implementation strategy is collected through Case Record Forms.- Quality of care is measured from the patient's perspective through the QUOTE-breast cancer, a questionnaire that was developed by and for breast cancer patients and concerns those aspects that breast cancer patients perceive as most important when assessing quality of care,25. The - Information on the performance of professionals is gained through Case Record Forms. The performance indicators have been developed by means of a consensus procedure in the uhM project group.- Information on societal costs are retrieved from hospital information systems and through a prospective cost book that is filled out by patients during six weeks from the moment of discharge. The day before surgery, patients are asked to fill out a retrospective cost book to assess the patients' health-care related costs for a period of two weeks prior to surgery.- Patients fill out EuroQol questionnaires at baseline (one day before surgery) and at 24 hours, one week and six weeks after surgery.When patients sign informed consent and receive the questionnaires, a stamped envelope is given to them which can be used to return the set of questionnaires.Programme cost-effectiveness  consists of the two components costs and effectiveness, and is expressed as incremental costs per Quality Adjusted Life Year (QALY) of the ultra short stay programme versus care as usual. Incremental costs are expressed as the difference in costs between the ultra short stay programme and care as usual. Costs are calculated by multiplying the volumes with the appropriate cost prices. Information on  costs are retrieved from hospital information systems and through a prospective cost book that is filled out by patients during six weeks from the moment of discharge. Effectiveness of the programme is determined through the measurement of safety,  and generic health-related Quality of Life. Generic health-related Quality of Life is recorded through the EuroQol-5D (EQ-5D), which i- Policy cost-effectiveness of implementation of the ultra short stay programme versus 'doing nothing' is expressed as incremental costs per Quality Adjusted Life Year (QALY). Costs of the implementation strategy is measured through cost forms that participants in the study groups fill out, through scoring by a researcher (MdK) of all other costs  in a data sheet. Effectiveness of the implementation strategy is measured through Case Record Forms containing treatment information. More details on programme cost-effectiveness and policy cost-effectiveness can be found in the section 'Economic evaluation'.The study lasts three years. The first year of the study will be spent on the analysis of the breast cancer care processes in the different hospitals and the development of the measurement instruments. The following six months are used for baseline measurements, and preparation of the implementation of the new programme by the members of the local study groups. Recruitment of patients will be performed by breast nurses and advanced practice nurses working at the breast units. Then, six months are used for implementation and running of the actual programme, followed by six months of measurements 'after implementation'.® for Windows (SPSS INC 1989–2003). P values < 0.05 will be defined as statistically significant. The pre and post comparison of the interval scaled outcome variables will be analysed with a multivariate linear regression model. Apart from the independent variable hospital, other independent variables included in this model are type of surgery, age, informal care at home, involvement of home care nursing and starting time of surgery. Furthermore, correlation of data within a hospital are accounted for by clustered data analysis.Intention to treat will be used for data analyses, and analyses will be performed using the SPSS package, version 12.0.1 In the economic evaluation, programme cost-effectiveness will be combined with cost-effectiveness of the implementation strategy, to determine policy cost-effectiveness.The time horizon of six weeks seems appropriate as patients may receive additional treatment after surgery . Therefore, we expect that only short-term quality-of-life effects and costs can be attributed to the ultra short stay programme. Cost analysis includes health care costs and costs outside health care such as out-of-pocket costs or productivity costs due to reduction in paid work and/or domestic activities. Health care costs refer to hospital and other health care facilities costs. Hospital costs consist of personnel, material, capacity costs, and overhead, associated with the diagnosis, surgical treatment, hospital stay and follow-up. Other healthcare costs include costs of visits to the general practitioner, medication, home care etc. Health care costs are estimated by multiplying the resource utilization with the cost price per unit. Most of these costs are estimated by using existing resource registration systems and available cost prices through the financial departments of the participating centres. If true cost prices are not available in the participating centres, 1) existing cost prices of the uhM or 2) official directive prices are used. HealthcIn each centre, the costs of the implementation process are calculated based on time and materials invested during the different phases of implementation. The diagnostic analysis may be looked upon as intervention. Its costs should be included in the costs of implementation. To determine cost-effectiveness of implementation, the costs of implementation will be weighed against the proportion of patients that is treated according to the ultra short stay programme.The cost-effectiveness of implementation, i.e. the extra cost per patient treated according to the ultra short stay programme, will be combined with the cost-effectiveness of the breast cancer care programme to determine policy cost-effectiveness, using Mason's model,30. In tBreast cancer surgery on an inpatient basis is a burden on the health care budget. Although previous studies have shown that ambulatory surgery is feasible for breast cancer patients, this has not been widely recognised in Europe. This is, to our knowledge, the first study that systematically evaluates the impact of an implementation strategy on patient and professional outcomes, patient experiences, costs, barriers and facilitators for change, for ultra short stay after breast cancer surgery.A randomised controlled trial is usually preferred over any other design. However, for several reasons a randomised controlled trial was not regarded as feasible. The first reason concerns the expected lack of willingness on the part of hospitals to be allocated to the control condition as they would get nothing in return for their efforts had we chosen a design with a concurrent control group. The number of participating hospitals in the study is too small to allow a controlled study design. A controlled study would have led to an unacceptable rise in study costs and the study period was too short to construct a waiting list control group. However, we think that the numbers of patients from the four participating hospitals provide enough power to draw conclusions on the estimation of change.Random allocation at patient level was not considered for legal and organisational reasons; patients always have the right to go home whenever they want, making randomisation for early or late discharge rather meaningless. Moreover, it is unethical to have one patient discharged the day of surgery while her neighbour is allowed to stay for another day or two. The extent of organisational changes to implement short stay within the existing processes of care are of such a scale that it is hardly feasible nor acceptable to allow different admission periods at the same time within one hospital.We are aware that the decision for a pre-post design instead of a randomised controlled trial cannot rule out the influence of developments and ongoing changes in health care, including reductions in lengths of hospital stays. This aspect will, therefore, be taken into account when conclusions are drawn based on the results of the study.In line with the most recent insights in implementation science a diagnostic analysis will be performed with the intention to tailor the implementation strategies to the needs of each hospital. However, this analysis will encompass a lot of time, energy, and costs, while there is no evidence as yet that such an analysis will assure a positive effect of implementation.Since we wittingly selected merely early adopters, the external validity may be limited with regard to the smoothness of implementation. However, we think that although not every hospital can be labelled an 'early adopter', every hospital willing to participate in an implementation study must be eager to make the study a success, regardless of the degree of novelty at that time.Although diagnostic analysis explores the willingness to implement as well as the opportunities for implementation, the number of perceived barriers and facilitators will be underestimated, or at least not overestimated. A thorough exploration of actual barriers and facilitators must, therefore, be performed during the actual implementation process.There is a risk of contamination of the baseline measurement by socially desirable behaviour because the outreach visits for preparation of the study have taken place before and during the baseline measurement period. However, fixed moments to actually start with the innovative programme, are scheduled within each hospital and agreed upon. Feedback is given by the researchers if they suspect that hospitals may start the programme earlier than planned. Therefore, we think this influence is limited.A cost-effectiveness analysis is performed on the ultra short stay programme itself, and on the implementation strategy used. The goal of this study is to implement the ultra short stay programme for breast cancer surgery aiming at improved patient information, improved organization of care, a high degree of quality of care, and a reduction in mean hospital stay which results in a reduction of health care costs.MaDO- Breast cancer surgery in day careQALY- Quality Adjusted Life YearQUOTE- breast cancer – questionnaire on quality of breast cancer care , developed through qualitative and quantitative methods with breast cancer patientsUhM- University Hospital MaastrichtThe author(s) declare that they have no competing interests.MdK is responsible for data collection and drafting the manuscript. MdK works under direct supervision of MvM and TvdW. CF, TvdW, CD, FK, CvdV, JR, AB, FvdE and MvM all participated in discussing the design of the study and developing the research protocol. They were co-applicants on the successful funding proposal. All authors read and corrected draft versions of the manuscript.The pre-publication history for this paper can be accessed here:


The definition of a distance measure plays a key role in the evaluation of different clustering solutions of gene expression profiles. In this empirical study we compare different clustering solutions when using the Mutual Information (MI) measure versus the use of the well known Euclidean distance and Pearson correlation coefficient.Relying on several public gene expression datasets, we evaluate the homogeneity and separation scores of different clustering solutions. It was found that the use of the MI measure yields a more significant differentiation among erroneous clustering solutions. The proposed measure was also used to analyze the performance of several known clustering algorithms. A comparative study of these algorithms reveals that their "best solutions" are ranked almost oppositely when using different distance measures, despite the found correspondence between these measures when analysing the averaged scores of groups of solutions.In view of the results, further attention should be paid to the selection of a proper distance measure for analyzing the clustering of gene expression data. In recent years, DNA microarray technology has become a vital scientific tool for global analysis of genes and their networks. The new technology allows simultaneous profiling of the expression levels of thousands of genes in a single experiment. At the same time, the successful implementation of microarray technology has required new methods for analyzing such large scale datasets. Clustering is a central analysis method of gene-expressions that has been implemented extensively in various works and applications -5. The pMany clustering algorithms depend heavily on 'similarity' or 'distance' measures  that quantify the degree of association between expression profiles. The definition of the distance measure is a key factor for a successful identification of the relationships between genes and networks . DiffereClustering co-expressed genes usually requires the definition of 'distance' or 'similarity' between measured datasets, the most common choices being Pearson correlation or Euclidean distance... it is widely recognized that the choice of the distance may be as crucial as the choice of the clustering algorithm itself . However, as pointed out by Brazma and Vilo (2000), the appropriateness of similarity measures has not been systematically explored and these measures are used on an ad-hoc basis." Many publications use traditional clustering algorithms  that have roots in conventional data-extensive research fields, such as signal or image processing. In these fields, the similarity measures rely on the unique characteristics of the specific data structure. For example, in signal processing it is commonly assumed that identical codeword vectors are distorted by white noise components during transmission. Under such an assumption, it is reasonable to use a vector-quantizer encoder which is based on the Euclidean distance  aKNNimpute and SVDimpute, were found to be more robust to missing value estimation [The preparation of the dataset was performed in a similar manner to Gat-Viks et al. (2003) . We usedtimation .The compared algorithms were tested with respect to the yeast cell-cycle dataset to obtain their best clustering solutions with 5, 6 and 7 clusters. The comparison between the algorithms was performed between solutions with the same number of clusters. Clustering solutions were graded by the MI-based homogeneity and separation scores (see Methods section). This grading method is conventionally used to determine the quality of a clustering solution when the true solution is unknown [efficiency frontier on the MI-based homogeneity-separation plane. This manner of evaluation of clustering solutions is accepted when the "true" clustering is unknown  we found the following results  scores with respect to all the solutions obtained by the other algorithms. Two of these solutions are the lower-left solutions in the sIB efficiency frontiers shown in Figure When considering sIB algorithm obtains the worst results relative to all the other algorithms. For 7-clusters solutions the K-means algorithm obtains the best solution. For 5-clusters solutions there is no clear dominate solution. However the Click solutions, which have the highest Homogeneity scores, obtain Separation scores that are very competitive to the solutions of the SOM and the K-means. These results emphasize once again that despite the conformity between the compared distance measures, as informed in the literature [Note that when scoring the different algorithms with respect to the Euclidean or the Pearson measures, the (relative) ranking can be totally different. For example, Figure terature ,17, furtThis paper presents two related experiments. In the first experiment, which is based on known clustering solutions, we show the statistical superiority of the average MI-based measure independently of the selected clustering algorithm. In the second experiment, we show that the use of different distance measures can yield very different results when evaluating the solutions of known clustering algorithms. This is particularly true when looking at the score of the "best clustering solution" rather than the averaged score of a group of solutions, as we did in Experiment 1. This important fact is often overlooked in the literature. The essential question "which distance measure to use?" remains without a definitive answer, yet, in view of the first experiment, we propose to further investigate it by integrating the MI-based distance measure into known clustering algorithms.The statistical superiority of the MI-based score in the first experiment can be attributed to several appealing properties of this measure, as elaborated in the Methods section. The first property is the use of MI as a generalized measure of correlation between variables, a property which seems valuable for gene expression data -17. The sIB algorithm outperforms the other clustering algorithms with respect to the MI-based homogeneity and separation scores. The first reason is the nature of the stochastic transition matrix (STM) which is used by the sIB as an input. Although the STM is generated from a conventional pairwise similarity matrix, once it is formed it represents the (dis)similarity relations between all the patterns in the dataset. Based on the STM, the sIB associates an expression profile to a cluster by using the joint probability distribution of all the expression patterns in the data. This is a possible explanation as to why the sIB generates solutions with higher MI-based homogeneity score than those obtained by the other algorithms that rely only on the pairwise distances, or only on the patterns in the particular associated cluster. Moreover, the construction of a joint probability distribution that reflects the associations between all the patterns in the dataset provides an additional normalization procedure that naturalizes external effects, such as measuring and experimental errors.We can indicate two possible reasons why the sIB is evidently the use of MI in its objective function. The sIB aims at minimizing the loss of information on the gene-expression profiles during the clustering process. The algorithm objective is to find a clustering solution that  maximizes the mutual information between clusters and a relevance variable which is associated with the gene patterns [sIB objective function is considerably different than the use of MI when computing the homogeneity and separation scores. The former is based on the joint distribution of all gene patterns, while the latter is based on the joint distribution between each gene and its associated cluster. Moreover, our way to derive the joint distribution of patterns and clusters  is very different than the exponential transformation used by the sIB to obtain the joint distribution of patterns. Nevertheless, it seems that although these definitions and calculations are different, they are sufficiently associated such that the sIB produces a clustering solution with high MI-based scores.The second possible reason for the superiority of the patterns . Note, hIn this work we analyze the performance of the Mutual Information (MI) distance measure for clustering of gene-expression data. In comparison to the Pearson correlation and the Euclidean distance, the MI measure is known to have some advantages: it is a generalized measure of statistical dependence in the data, and it is reasonably immune against missing data and outliers. In this work we show that the average MI measure also yields a higher power of test among different clustering solutions, thus, this measure is potentially more robust for differentiating erroneous clustering solutions. A comparative study of known clustering algorithms reveals that their best solutions are ranked almost oppositely when using different distance measures. In view of these results, further attention should be paid to the selection of a proper distance measure for the evaluation of clustering of gene expression data. One future direction is the integration of the MI-measure in known clustering algorithms. Another potential research is to implement Daub et al. (2004)  methods Euclidean distance and the Pearson correlation that are commonly used measures in gene expression clustering [mutual information (MI) measure. These measures quantify a pairwise distance between expression profiles over n conditions that are represented by the two vectors x =  and y = .This section discusses three similarity measures and their properties: the ustering ,37-39 anThe Euclidean distance between two expression profiles is given byIt measures similarity according to positive linear correlation between expression profiles, which may identify similar or identical regulation . The meaNumerous biological researches  i-14 i10-1where The Pearson correlation reflects the degree of linear relationship between two patterns. It ranges between -1 to +1, reflecting respectively a perfect negative (positive) linear relationship between the patterns. A zero correlation value implies that there is no linear relationship between the two patterns, yet it gives no indication regarding nonlinear relationships that might exist between the patterns.The correlation coefficient is invariant under any scalar transformation of the data. Accordingly, two expression profiles that have "identical" shapes with different magnitudes will obtain a correlation value of 1. The ability to measure (dis)similarities according to positive and negative correlations can help to identify control processes that antagonistically regulate downstream pathways . NonetheGene expression measurements, like other empirical measurements, suffer from noise effects. Variations in the measurements might come from many sources: intrachip defects, variation within a single lot of chips, variation within an experiment, and biological variation for a particular gene . Both throw average or singular value decomposition [Both the Pearson correlation and the Euclidean distance require complete gene expression profiles as input. However, gene-expression microarray experiments often generate datasets with missing expression values. Therefore, another source of uncertainty when implementing these measures is the need to use methods for estimating missing data, such as position ,36.information (MI) provides a general measure for dependencies in the data, in particular, positive, negative and nonlinear correlations. It is a well known measure in information theory [X, Y with respective ranges xi ∈ Ax, yj ∈ Aj and probability distributions functions P(X = xi) ≡ pi, P(Y = yj) ≡ pj, the Mutual information between two expression patterns, represented by random variables X and Y, is given byThe Mutual n theory  that hasn theory ,17,44,46X and Y are statistically independent, meaning that X contains no information about Y and vice versa. A zero MI indicates that the patterns do not follow any kind of dependence, an indication which is impossible to obtain from the Pearson correlation or the Euclidean distance [The MI is always non-negative. It equals zero if and only if distance . This prdistance .Another important feature of the MI is its robustness with respect to missing expression values. In fact the MI can be estimated from datasets of different sizes. This is advantageous in analyzing expression datasets that contain a certain amount (up to 25%) of missing values .The MI between a pair of expression patterns is upper bounded by their marginal entropies. Accordingly, the MI measure exhibits a low value if the marginal entropies are low, even if the patterns are completely correlated. Therefore, there is a need to normalize the MI measure, giving a high score for highly correlated sequences, independent of their marginal entropies. There are several ways to carry out such normalization. Michaels et al. (1998)  normalizO(nlogn), where n is the number of expression values. Such sorting is not required when calculating the Pearson coefficient or the Euclidean distance measure. The number of bins should be moderate enough to allow good estimates of the probability function. If this number is too small or too large, then all bins will contain approximately the same number of expression values. In such a case the joint distributions of all pairs of expression patterns will be similar and will lead to the same MI value. There is no optimal solution to choose the number of bins, since it depends on data normalization and on the particular biological application [As noted above, the use of the discrete form of the MI measure requires the discretization of the continuous expression values. The most straightforward and commonly used discretization technique is to use a histogram-based procedure ,18,44. Wlication  and Law lication  and use Ml = ⌊1 + log2n⌋ and Mu = The homogeneity and the separation functions are often used to determine the quality of a clustering solution when the true solution is unknown . When coN elements (genes or samples represented by expression patterns or profiles) X ≡ {X1, X2, ..., XN} divided into k clusters. Denote by Xi and C(X i) the expression pattern of element i and the expression pattern of its cluster respectively, then the homogeneity is given byConsider a set of D(·) represents a given similarity measure, i.e., the Euclidean distance, the Pearson correlation or the Mutual Information. The solution separation score is evaluated by the weighted average similarity between cluster expression patterns: denote the expression patterns of clusters t1,...,tk, by Ct1,...,tkC,  then, the average separation is given bywhere Ni, Nj are the number of elements in cluster ti, tj respectively. The homogeneity and the separation are typically conflicting functions – usually the better is the homogeneity of a solution, the worse is its separation, and vice versa.where K-means was implemented by Matlab procedures. The SOM algorithm was implemented by GeneCluster 2.0 [Click was implemented by Expander [sIB was implemented by IBA_1.0 [The underlying concepts and the parameters of each of the clustering algorithms are given in an additional file ster 2.0 ,49. The Expander ,51. The  IBA_1.0 . Further IBA_1.0 -55.IP participated in the design of the study, produced the necessary data and software, carried out the analyses and drafted the manuscript. IBG conceived the study, designed and supervised the work, helped to analyze the results and drafted the manuscript. OM helped in the research coordination and review. The authors read and approved the final manuscript.Mutual Information (MI) measure, the Euclidean distance and the Pearson correlation coefficient. The robustness comparison is performed by using four public gene expression datasets. Each dataset contains two types of samples with a clear biological distinction, leading to a 'true' bi-clustering solution. Section 2: Details of the underlying concepts and parameters of the four clustering algorithms that were used in experiment 2. Additionally, the full results of the algorithms comparison are presented.Used software and parameters for comparison of clustering algorithms. The file contains the following 2 sections. Section 1: A comparison study of the Click here for file


Lately, there has been a great interest in the application of information extraction methods to the biomedical domain, in particular, to the extraction of relationships of genes, proteins, and RNA from scientific publications. The development and evaluation of such methods requires annotated domain corpora.We present BioInfer (Bio Information Extraction Resource), a new public resource providing an annotated corpus of biomedical English. We describe an annotation scheme capturing named entities and their relationships along with a dependency analysis of sentence syntax. We further present ontologies defining the types of entities and relationships annotated in the corpus. Currently, the corpus contains 1100 sentences from abstracts of biomedical research articles annotated for relationships, named entities, as well as syntactic dependencies. Supporting software is provided with the corpus. The corpus is unique in the domain in combining these annotation types for a single set of sentences, and in the level of detail of the relationship annotation..We introduce a corpus targeted at protein, gene, and RNA relationships which serves as a resource for the development of information extraction systems and their components such as parsers and domain analyzers. The corpus will be maintained and further developed with a current version being available at  Recent advances in biomedical research methods have greatly accelerated the rate at which new information is published. As a result, there has been an increased interest in applying Natural Language Processing (NLP) methods to the domain of biomedical publications accessible in literature databases such as PubMed -4. The aInformation extraction systems automatically identify entities and their relationships from free text, producing a structured representation of the relevant information stated in the input text. Such systems can, for example, support researchers in literature searches and serve as the basis for the inference of semantic relationships, such as candidate pathways, stated across several publications.An annotated corpus is a collection of texts that have been enhanced with markup specifying linguistic and domain information such as syntactic structure, named entity identification, and entity relationships. In this paper, we introduce BioInfer (Bio Information Extraction Resource), a manually annotated corpus resource for IE method evaluation and development in the biomedical domain, accompanied by supporting software. The corpus consists of 1100 sentences and represents 15 man-months of annotation efforts. We further present an annotation scheme that combines key annotation types with a detailed definition of entity relationships as well as present two new biomedical ontologies that define bioentity and relationship types.The availability of annotated corpora is an important prerequisite for NLP research as they provide gold-standard data for method evaluation and development. The impact of the MUC, TREC and SENSEVAL conferences -7, for eMost important among the steps typically applied by IE systems to extract information from text are named entity recognition, parsing and domain analysis, where named entity recognition identifies the entities whose relationships are to be found, parsing recovers the syntactic structure of the text, and domain analysis extracts the relationships among the named entities using the information from the other processing steps. Several biomedical corpora have been developed to facilitate the development of the separate IE system components, providing, for example, bioentity, entity relationship, syntax, abbreviation, and coreference annotation – we discuss these corpora and relate them to BioInfer in the Results and Discussion section. By contrast, the BioInfer corpus provides all these key types of annotation together, for a single set of sentences. By providing gold-standard input for the major stages in IE, the corpus allows the parallel development of all IE system components. The shared data facilitates the study of the interplay of the components, for example allowing the source and propagation of errors to be analyzed.Additionally, the corpus addresses a number of issues in the prevailing relationship annotation approach, in which relationships are expressed as pairs of entities. The BioInfer annotation allows relationships with a complex structure, such as relationships between relationships or relationships of more than two entities. Moreover, the BioInfer annotation scheme defines entity and relationship types that are organized into two interdependent hierarchical ontologies. The entity type ontology incorporates the established Genia ontology of physical types . For theIn previous studies, we have used an early subset of the corpus syntactic annotation to evaluate the performance of the Link Grammar and Connexor Machinese Syntax dependency parsers in the biomedical domain ,11. The The entity relationship annotation brings insight into the various relationship types that can hold among entities in the text. It further reveals the often complicated structure of these relationships and the entities themselves. The primary use of the annotation is to develop and test an IE system that targets the relationships stated in the text. Due to the fact that both the dependency and the relationship annotation cover the same set of sentences, the interplay between the syntax and the relationships can be studied as well. Further, as both the entity and relationship types are organized in hierarchical ontologies, the annotation facilitates easy step-by-step development whereby only certain types of relationships and certain types of entities are addressed at once.The paper is organized as follows. We first briefly introduce the corpus annotation and state the main results. We then describe the main features of the relationship and entity annotations, the ontologies used for defining the types of relationships and entities, and the binding of these annotations to text of the corpus sentences. We then present key features of the dependency annotation and finally present corpus statistics and supporting software, and discuss the relationship of the BioInfer corpus to other biomedical corpora. In the Methods section, we discuss in further detail the three main annotation types, presenting, for example, the annotation for anaphora and abbreviation definitions.entity annotation, entity relationship annotation, and dependency annotation. We now briefly introduce these annotation types; more detailed descriptions are given in the Corpus annotation section.The corpus annotation is divided into three key types, termed named entities of the gene, protein, and RNA types. When relevant to relationships, other physical entities as well as abstract process and property entities pertaining to the named entities are also identified in the annotation. For example, in the sentenceThe foundation of the entity annotation is the identification of Deletion of SIR4 enhanced METI5 silencing.SIR4 and METI5, but also the processes deletion of SIR4 and METI5 silencing that pertain to the genes. Together with entity typing, this extended annotation allows a detailed resolution of the relationships stated in the sentence.the annotated entities are not only the genes The entity relationship annotation describes relationships holding between the entities as stated in the individual sentences. The relationships are annotated through logic formulas where the predicates define relationship types and predicate arguments identify the entities that are related. For example, the relationships of the preceding example sentence are annotated asSTIMULATEenhanced is expressed using the predicate STIMULATE. Throughout the text, we follow the convention that predicate names are capitalized. The relationship annotation scheme is also used to annotate abbreviation definitions, and, where necessary to extract relationships, also coreference through, for example, pronouns.Note that the relationship stated with the word Finally, the dependency annotation describes the syntax of each sentence in the corpus. Each sentence is given a full dependency syntax annotation based on the Link Grammar dependency syntax formalism. An example of the dependency annotation is given in Figure Figure We now briefly summarize our contribution. The primary outcome of the research presented in this paper is the corpus itself, a public resource that facilitates the development of IE systems in the biomedical domain, available along with supporting software in a format that is straightforward to process . The corAnother outcome of the research are the ontologies, in particular the relationship type ontology, which defines a large number of possible relationship types and organizes them in a hierarchical manner. The ontologies can be used to focus an IE system only to certain types of relationships and entities, possibly considering different levels of generality. The ontologies can also be used to support various automated reinterpretations of the annotation based on rules attached to the various entity and relationship types in the ontology. Moreover, the binding of entities and relationships stated in the corpus text to specific types in the two ontologies defines mappings from natural language statements to controlled vocabularies, providing data for the analysis of how each concept is expressed in practice.The process of designing the annotation scheme and building the corpus has further contributed to a better understanding of the domain of entity relationships. An analysis of the corpus shows that 10% of the relationships are of a complex nature with more than two related entities and relationships affecting other relationships rather than only entities. Further, 14% of the entities involved in a relationship are processes or properties as opposed to physical entities. These results illustrate the information loss in the prevailing annotation approach where only pairwise relationships are used to relate only physical entities.Moreover, we find that there is no one-to-one correspondence between relationship types and the words that state the relationship in the text. Many expressions are ambiguous, that is, they can be used to state different relationship types, depending on the context. This emphasizes that context-based disambiguation is necessary to resolve relationship types in the text.The BioInfer corpus consists of individual sentences, each of which is separately assigned the entity, relationship, and dependency annotations. For example, in the sentenceInteraction of Munc-18-2 with syntaxin 3 controls the association of apical SNAREs in epithelial cells.Munc-18-2, syntaxin 3, SNAREs) and a process related to a protein (association of SNAREs). Further, there are two relationships: the interact relationship between the Munc-18-2 and syntaxin 3 proteins and the control relationship between the previous interact relationship and the association of SNAREs process. The dependency syntax specifies, for example, a subject-predicate dependency between the words interaction and controls and a object-predicate dependency between the words association and controls. There is an interdependence between the various annotation types in the sense that, for example, the dependency annotation respects named entities and, conversely, certain rules for annotating the entities and relationships make use of the syntax. For the purpose of this presentation, we will, however, treat the annotation types separately.we find four entities: three proteins  in the sentenceThe relationships belong to a variety of mDia binds profilin to promote actin polymerization.are of distinctly different types. Further, a relationship can be stated on different levels of specificity. In the sentenceAip1p interacts with cofilin to disassemble actin filaments.interact and disassemble. Clearly, disassemble is a more specific statement than interact.there are two relationships: In order to capture the various relationship types as well as their different levels of specificity, we introduce the relationship type ontology that defines relationship types and organizes them in a hierarchical manner, from the most general to the most specific relationship types. The ontology specifies over 60 relationship types at 5 levels of generality. Each annotated relationship is categorized within this ontology at the most specific level applicable.Apart from belonging to a large variety of types, the relationships are also structurally complex. The prevailing approach in previously published biomedical corpora has been to only annotate pairwise relationships between physical entities. However, it becomes clear that this representation in many cases fails to accurately capture the stated information. Consider, for example, the following sentence.Four yeast spliceosomal proteins  interact to promote U2 snRNP binding to pre-mRNA.U2 snRNP. However, any such decomposition leads to significant imprecisions. The sentence does not specify which pairwise interactions take place among the four spliceosomal proteins, nor does it assert a pairwise relationship between any of the spliceosomal proteins and, for example, U2 snRNP. Instead of using only pairwise relationships, the information in the sentence can be better captured through structurally more complex relationships: an interact relationship among all the four spliceosomal proteins and a promote relationship between the interact relationship and the pairwise bind relationship. In the BioInfer corpus, the relationships are annotated using logic formulas. Each relationship is expressed through a predicate whose arguments are instantiated with entities or other predicates. The predicate name, arity, and the semantic roles of its arguments are defined in the relationship type ontology together with a description of the relationships intended to be annotated using this predicate. The relationship annotation of a sentence is then a set of formulas that state all the relevant relationships in the sentence. Let us consider the previous example sentence. The formulaUsing only pairwise relationships, the sentence would most likely be decomposed into all pairwise relationships among the spliceosomal proteins and all pairwise relationships between the spliceosomal proteins and INTERACTexpresses the relationship among the spliceosomal proteins. In the usual interpretation, it does not explicitly assert a pairwise relationship between any two of the four proteins. The formulaSTIMULATE, BIND)describes the relationships stated in the sentence, accounting for their complex nature. It is this formula that is used in the corpus to annotate the example sentence.INTERACT can be decomposed in the obvious manner and predicates whose arguments are other predicates can be either omitted or recursively decomposed. It depends on the predicate whether better approximation can be achieved by omission or by recursive decomposition. Several examples of the relationship annotation are provided in Appendix I.The formula-based annotation system is more powerful than the simple pairwise relationship annotation. However, practical considerations and the current state-of-the-art in IE systems often call for pairwise relationships, even at the cost of introducing inaccuracies such as those discussed above. It is thus necessary to employ a scheme where the annotation can be decomposed in terms of pairwise relationships. The BioInfer annotation scheme allows such decomposition. Multiple-argument predicates such as actin is a named entity, actin expression pertains to a named entity, but a 50 kDa protein as an entity is not considered named and thus not annotated in the current version of the corpus. The typical use in literature and databases such as Swiss-Prot is used as a guide in deciding for borderline cases whether an entity is considered named or not named.The BioInfer corpus is focused on the development of IE systems for extracting relationships between genes, proteins, and RNAs. This focus influences the entity annotation as currently only entities that are relevant to this focus have been annotated. As a typical IE system extracts relationships between named entities, we require that an entity is named or pertaining to a named entity, more specifically a named gene, protein, or RNA, in order to be relevant to the corpus focus. For example, extended named entities, defined as terms denoting other physical entities, processes and properties pertaining to named entities, are also annotated when relevant to relationships. Consider the following example:In addition to named entities, Deletion of SIR4 enhanced mURA3 and METI5 silencing.SIR4, mURA3, MET15). Further, the entity deletion of SIR4 is a process extended named entity pertaining to SIR4, similarly for mURA3 silencing and METI5 silencing. In the sentenceThe sentence contains three named entities .the annotation identifies three entities: the named entities While it is common to restrict IE systems to only take into account named bioentities as objects participating in relationships, consideration of extended named entities allows a better resolution of the actual statements in the sentence. Consider the following example:Abundance of actin is affected by the differential calreticulin expression.calreticulin affects actin, a more detailed resolution of the entities is desirable. The affect relationship in this sentence is between two property/process entities: abundance of actin and calreticulin expression. Abundance of actin is a property pertaining to actin and calreticulin expression is a process pertaining to calreticulin. Both actin and calreticulin are named entities. As a further illustration, consider the sentenceAlthough it is not incorrect to state that Characterization of gelsolin truncates that inhibit actin depolymerization by severing activity of gelsolin and cofilin.gelsolin truncates, actin depolymerization, activity of gelsolin and activity of cofilin rather than the proteins gelsolin, actin, and cofilin themselves.where the related entities are abundance of actin is a protein amount property pertaining to the protein actin leads to two crucial entity annotation concepts: entity nesting and entity typing. Entity nesting allows annotation of entities within entities with the outer entity typically being a semantic modification, extension or specification of the inner entity. For instance, the outer entity abundance of actin is a specification of the inner entity actin in the sense that it identifies a single property of actin. Each entity is given a type from the hierarchical ontology of entity types developed for this purpose. The ontology captures all relevant entity types, both physical entity types such as gene and protein as well as abstract entity types such as process and property. Similarly to the relationship type ontology, the entity types are organized in a hierarchical manner and in the annotation each entity is given the most specific applicable type from the ontology.The need to resolve that, for instance, gene, protein, and RNA types are annotated in each sentence. In addition, those named entities that participate in a relationship are nested inside the broadest entity relevant to the particular relationship. Thus, up to special cases described later, every entity that participates in an annotated relationship has a named gene, protein or RNA at its innermost level of nesting.The introduction of entity nesting and typing allows us now to state a general principle regarding the entity annotation. Only named entities of the This is demonstrated in the example sentenceA from such sites suggests that profilinB is involved in reorganization of actin cytoskeletonDispersal of profilin.profilinA, profilinB, actin, actin cytoskeleton, and reorganization of actin cytoskeletonwould be annotated. Since the process dispersal of profilinA does not participate in any annotated relationship, it is not annotated. The protein actin, on the other hand, is nested in the entity actin cytoskeleton, which is in turn nested in the entity reorganization of actin cytoskeleton, the broadest entity relevant to the stated relationship PARTICIPATE.where the entities molecular function ontology of Gene Ontology (GO)  core bundles] and the unresolved bracketing [actin filament core bundles] is not critically important because actin is the only protein in the phrase and the inner structure of the phrase can be ignored. On the contrary, the correct resolution of the coordination [[[myosin heavy chain] and [actin]] isoforms] is important because the coordinates are protein names and hence of importance to the IE system. While the inner structure of the coordinated phrases is still not necessarily fully resolved, the two protein names are separated by the coordination. Further, while major wide-coverage dependency parsers such as Connexor Machinese Syntax, MiniPar, and Link Parser do not resolve the attachment of pre-modifiers in noun phrases, they do resolve coordination. The annotation in BioInfer thus coincides with the intended coverage of these parsers.In a corpus that is used to develop tools for extracting bioentity relationships, it is not critical to resolve the internal structure for every noun phrase, a general principle followed by both BioInfer and, for example, the Genia treebank. In the BioInfer corpus, pre-modifier attachment in noun phrases without internal coordination is not resolved, as these phrases typically only contain at most one named entity. In noun phrases that contain internal coordination, however, each of the coordinates can represent a different named entity and resolving the coordination and thus separating the named entities from each other is vital to the IE task. The distinction is illustrated in Figure During the manual dependency annotation, the dependency types were not specified. Due to the large number of dependency types defined in LG  the effort to manually determine the type of each dependency is prohibitive. However, dependency types are important for many uses of the corpus. Thus, we have developed a reliable heuristic method for determining dependency types automatically given the dependency structure. This method has been used to assign dependency types to the whole corpus. In addition, quality control of the dependency types has been performed by manual examination.The method to determine dependency types has four main steps: reducing the complexity of the sentences, parsing the simplified sentences, assigning dependency types based on the parses, and extending the types to the full sentences using a set of rules. In the following, these steps are described in more detail.NP macro-dependencies have an important role in the simplification of the sentences. As the attachment of pre-modifiers in noun phrases covered by NP macro-dependencies can be determined with simple rules without parsing, any NP macro-dependency present in the sentences can be truncated so that all the pre-modifiers spanned by the macro-dependency are removed. This procedure reduced the number of words by more than 20% and removed all problems related to the parsing of these noun phrases. The sentences were further simplified by capitalizing common uncapitalized proper names and removing citations.We parsed the simplified sentences with a modified version of the LG parser that overcomes some of the problems of the unmodified LG parser for biomedical text . The modFor each sentence, we determine by comparison against the manually created dependency structure the subset of parses with the maximal number of recovered dependencies. The intuition underlying this selection heuristic is that parses having correct dependency structure also have the correct dependency types. From the set of maximally correct parses, we then determine for each dependency the majority type, that is, the type which most commonly appears for the dependency in the parses.We transfer the majority types from the simplified to the full sentences and resolve the types of the dependencies associated with the truncated noun phrases using straightforward rules.We found that the quality of dependency types is significantly higher for sentences that receive fully correct parses in the parsing step. We thus mark sentences depending on whether or not they were fully correctly parsed, allowing users of the corpus to choose whether to use all dependency types or only the most reliable ones. To evaluate the quality of the automatically produced dependency types, we manually examined 30 randomly chosen sentences with fully correct parses and 30 randomly chosen sentences for which fully correct parses were not obtained.For the fully correctly parsed sentences, the heuristic method assigned the correct dependency type to 575 out of 586 dependencies (98%). For the sentences without fully correct parse, 828 out of 962 dependencies (86%) were assigned their correct type. Out of the 1100 sentences, 741 (67%) received a fully correct parse.The sentences that form the corpus text were selected using the following procedure. Pairs of proteins that are known to interact were extracted from the Database of Interacting Proteins DIP ,46. ThesThe BioInfer corpus is provided in an XML format. XML is a standard text-based format for structured documents, and XML parsers are freely available. Due to the relatively complex structure of some of the annotation types in the BioInfer corpus, we provide supporting software to further ease the use of the corpus.The supporting software is based on an extendable framework for parsing the corpus and representing it as data structures that can be accessed through a fully documented API (Application Programming Interface) providing methods for accessing different aspects of the annotation. In addition, we provide programs built on this API that visualize the annotations and extract them in a simplified, human-readable form (see Appendix I). The user interface of Bioinfer Visualizer is shown in Figure It is common practice to use automatically parsed text as the starting point for syntax annotation. However, since general English parsers could not provide sufficiently accurate parses due to their generally poor performance on biomedical text, we chose to create the dependency annotation manually without a baseline parse. Fully manual annotation also helps to avoid bias toward systematic errors introduced by the parser.The dependency annotation was created by six annotators who worked in rotating pairs to reduce variation and avoid systematic errors. The set of sentences was divided into batches of nine sentences and each batch was assigned to one pair for annotation. For each batch the dependency annotation was first created by both members of the pair independently. The annotations were then compared and in case of differences, the matter was discussed until an agreement was reached. Two of the annotators were biology experts and the other four annotators had a possibility to consult an expert. Further, all annotators acquired thorough knowledge of the LG dependency formalism.The entity and relationship annotations were created in parallel, partially based on a previous unpublished entity and relationship annotation of the corpus. This annotation was created by a biology expert, difficult cases and annotation rules being discussed with two IE researchers. The pair annotation strategy used in producing the dependency annotation was not applied for the entity and relationship annotation. The annotation was produced in several passes through the data as the rules and annotation types were refined, taking into account experience gained in the previous annotation. The last full pass was carried out shortly before publication.We estimate that the annotation of the BioInfer corpus consumed 2500 man-hours, that is, 15 man-months, not including time spent on previous, unpublished annotation of the data, tool building, development of supporting software, and the design of the annotation scheme.In this Appendix, we list several annotated sentences from the corpus, in order to demonstrate the relationship annotation scheme on non-trivial, real-world cases. For each example, we list all annotated relationships along with other information when necessary.Examples(a) Biochemical analyses revealed a strong induction of VEGF-receptor-2 (flk-1/KDR) tyrosine-autophosphorylation by VEGF which was maximal after 5 minutes and was followed by receptor down-regulation.EQUALEQUALinduction of by:INITIATEPHOSPHORYLATE.Autophosphorylation is represented with a one-argument predicate (b) In investigating the mechanism by which pRb induces senescence, we have found that pRb causes a post-transcriptional accumulation of the cyclin-dependent kinase inhibitor <ap27a>  that is accompanied by an increase in <cp27c>(<dKIP1d>) specifically bound to cyclin E and a concomitant decrease in cyclin E-associated kinase activity.EQUALEQUALMEMBERcauses:CAUSEbound to:BINDaccompanied by:COOCCURaccompanied by:COOCCURconcomitant:COOCCUR(c) Analysis of the formation of the calf spleen complex in the presence of varying concentrations of divalent cations gave evidence for the presence of a high-affinity divalent-cation-binding site on the spleen actin  which appears to regulate the interaction with profilin.regulate:CONTROL)regulate:CONTROL)profilin with the spleen actins is annotated directly as there are no coreferents for the proteins.The stated interaction of (d) Neuropilin 1 (NP-1) is a receptor for vascular endothelial growth factor (VEGF) 165 (VEGF165) and acts as a coreceptor that enhances VEGF165 function through tyrosine kinase VEGF receptor 2 (VEGFR-2).EQUALEQUALEQUALreceptor for:BINDMEMBERthrough:MEDIATE)through:RELATE(e) SNF11, a new component of the yeast SNF-SWI complex that interacts with a conserved region of SNF2.component of:CONTAINcomplex: CONTAINcomplex: CONTAINinteracts with:INTERACTSNF11 and component are syntactically bound through a dependency that unambiguously identifies the coreference  Deletion of SIR4 enhanced mURA3 and MET15 silencing, but deletion of SIR1 or SIR3 did not affect silencing, indicating that the mechanism of silencing differs from that at telomeres and silent mating loci.enhanced:STIMULATEenhanced:STIMULATECOREFERCOREFERnot:NOT)not:NOT)silencing acts as a coreferent for two silencing processes.Here, in contrast with the previous example, the coreference cannot be trivially recovered from the dependency and therefore it is annotated. Moreover, (g) Discrete segments (70–150 amino acids) of PRT1 and TIF35 were found to be responsible for <abtheirab> binding to TIF34.COREFERCOREFERresponsible for:CAUSE)responsible for:CAUSE)TIF34. In order to capture these relationships accurately, there are two separate entities annotated that mark the pronoun their as a coreferent for the proteins.There are two distinct proteins and a segment on each of the proteins, where each segment is responsible for the binding of its respective protein to (h)c<abGP Although talin has been suggested to act as a linkage protein mediating the attachment of <ab><aIIba>-<bIIIab>c> f<deGPde><dIIbd>-<eIIIae>f> to this cytoskeletal protein has not been demonstratedto actin filaments, direct binding of <.COREFERmediating:MEDIATE)binding of to:BINDCONTAINCONTAINCONTAINCONTAINtalin and this is annotated and therefore, under the textual-replacement interpretation of coreference, the binding is with talin protein.The coreference between (i) A 2.2-kb truncated NRP1 cDNA was cloned that encodes a 644-aa soluble NRP1 (sNRP1) isoform containing just the a/CUB and b/coagulation factor homology extracellular <ab domains of NRP1 ab>.EQUALencodes:ENCODEcontaining:SUBSTRUCTUREcontaining:SUBSTRUCTUREhomology:SIMILARhomology:SIMILARtruncated NRP1 cDNA rather than its wild-type form NRP1 cDNA. This is also reflected in the entity types, where NRP1 cDNA is given the type gene, while truncated NRP1 cDNA is given the type mutant.The sentence states a relationship of a mutant gene (j) CD26 is a T cell activation antigen known to bind adenosine deaminase and have dipeptidyl peptidase IV activity.bind:BINDhave activity:FNSIMILARCD26 having dipeptidyl peptidase IV activity is interpreted as a functional similarity between the two proteins. Like other names that can refer to both a type of function and a distinct protein, dipeptidyl peptidase IV is considered a protein in BioInfer.The statement of (k) Data is presented to suggest that the G1 cyclin D1 and the cyclin-dependent kinase inhibitor p27KIP1 may be involved in subversion of the G1/S traverse by signaling pathways activated by HER-2 function.MEMBERinvolved in:PARTICIPATEinvolved in:PARTICIPATEsubversion of the G1/S traverse by signaling pathways activated by HER-2 function gives rise to the entity nesting <subversion of <traverse by <pathways activated by < <HER-2> function > > > >The text Project name: BioInfer corpus and supporting softwareProject homepage: Operating systems: Platform independentProgramming languages: PythonOther requirements: Tkinter library for PythonLicence: LGPLSP and FG are the primary authors of this work and their contribution is equal. SP and FG have conceived the corpus annotation scheme in its present form, with a significant contribution by JH to the biomedical aspects of the annotation. The text of the paper was written by SP and FG. JH was the principal annotator for the entity and relationship annotation. The dependency annotation was produced jointly by JH, JBj, SP, and FG. The entity type annotation was produced by JH and JBj. The work was supervised by JBo, JJ, and TS since its beginning in late 2001, and they have contributed to both the content and the form of the corpus and the paper. All authors read and approved the final manuscript.