In the majority of China, the Centre for Disease Control (CDC) at the county level provides both clinical and public health care for TB cases, with hospitals and other health facilities referring suspected TB cases to the CDC. In recent years, an integrated model has emerged, where the CDC remains the basic management unit for TB control, while a general hospital is designated to provide clinical care for TB patients. This study aims to explore the factors that influence the integration of TB services in general hospitals and generate knowledge to aid the scale-up of integration of TB services in China.This study adopted a qualitative approach using interviews from sites in East and West China. Analysis was conducted using a thematic framework approach.The more prosperous site in East China was more coordinated and thus had a better method of resource allocation and more patient-orientated service, compared with the poorer site in the West. The development of public health organizations appeared to influence how effectively integration occurred. An understanding from staff that hospitals had better capacity to treat TB patients than CDCs was a strong rationale for integration. However, the economic and political interests might act as a barrier to effective integration. Both sites shared the same challenges of attracting and retaining a skilled workforce for the TB services. The role of the health bureau was more directive in the Western site, while a more participatory and collaborative approach was adopted in the Eastern site.The process of integration identifies similarities and differences between sites in more affluent East China and poorer West China. Integration of TB services in the hospitals needs to address the challenges of stakeholder motivations and resource allocation. Effective inter-organizational collaboration could help to improve the efficiency and quality of TB service. Key words: TB control, service delivery, integration, hospitals, China. Tuberculosis (TB) control is of great public health importance, particularly in many low-and middle-income countries. The World Health Organization (WHO) TB strategy, brand named "DOTS", is based on five essential elements: political commitment; case detection using sputum microscopy; standardized short course chemotherapy; regular drug supply; and monitoring and evaluation . WHO andIntegration of TB services into primary health care has been common since the 1970s , howeverChina has one of the highest TB burdens in the world, with more than 1.3 million new cases each year . The CenBefore 2003, TB control performance was poor, with detection of smear-positive TB at around 30%. The inaChina's TB control has benefited from a series of measures to strengthen the public-health system following the outbreak of severe acute respiratory syndrome (SARS) in 2003 . Since 2In the majority of China, the county CDC provides both clinical  and public health functions  for TB cases. General hospitals and other health facilities such as community-based health services are required to refer suspected TB cases with cough of two weeks or more or haemoptysis to the CDC. Only complicated and severe TB cases are treated in the general hospitals. The CDC traces all referred cases not visiting the CDC within three days. Only TB patients registered and treated in the CDC are entitled to free treatment under the government's TB programme. This model is known as "CDC model".This model to TB services has come across a number of challenges. Thirty-four percent of patients with a TB diagnosis first presented at the general hospital system after the onset of their symptoms. However, only 25% of these cases were reported and 13% referred to the CDC . As a reSince early 2000s an integrated model of TB service delivery was implemented in some eastern provinces such as Shanghai, Zhejiang and Jiangsu, on their own initiatives and some of the western provinces with support from the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM). In this model, the CDC continues to provide public health functions such as referral management, reporting, staff training and case tracing. A general hospital was "designated" to undertake diagnosis, prescription and follow-up management, with other hospitals and health facilities referring those suspected of having TB to the designated hospital. A comparison of the TB functions between the two models is summarized in Table To date, a few studies in China have reported the experience of integrating TB services in the township hospitals  of Guangxi ,21, ShanThis study adopted a qualitative approach, particularly valuable when seeking to explore processes of implementation .In order to better understand the drivers and barriers of policy implementation, we chose sites from different economic development regions for this study, i.e. from the East and West provinces. Although GFATM supported districts might exist, we purposefully selected districts which initiated the integrated model on their own initiative, as a bottom-up approach was thought to be more replicable and sustainable. At the time of this study, very few districts in China had implemented the integrated model. Thus research sites were pragmatically selected, considering the availability of the sites and their willingness to participate. Finally, two districts were selected from the database of the China National TB Programme. One was Changning, a district of Shanghai (SC), and another was Nanning, the capital city of Guangxi(GN). Geographically, they are located in eastern and south western China respectively. SC had much higher GDP per capita and TB expenditure per person than GN. Both sites had similar cure rates of new smear positive TB cases Table . A prefePurposive sampling techniques were employed to select participants based on their disciplines and roles. These individuals represented a wide spectrum of health professionals working within TB control. 21 individual interviews were conducted, 11 at SC and 10 at GN, in the autumn of 2008 Table . The intThe interviews were conducted by a core research team, mainly consisting of TB researchers from the University of Leeds based in China  and Shandong University (QS). One member was responsible for ensuring consistency throughout the data collection process. Interviewees were invited for interview through the CDC and appointments made via telephone or Email, outlining the purpose of the research and questions to be asked. Informed consent was obtained prior to the interviews. Each interview lasted approximately 40-50 minutes, and was audio recorded with permission of the interviewees. The recording was transcribed within 48 hours by trained postgraduate students from Shandong University and checked by at least one core research team member. This ensured that researchers could return the unclear points of the transcripts to the interviewees for clarification in a timely manner. Ethical approval was granted by the Ethical Committee of School of Public Health of Shandong University.A thematic approach was used, which allows both application of an existing framework and inclusion of further emerging themes from the data . All theIn qualitative research, participant and researcher bias may affect the validity and reliability of findings. Efforts were made to overcome the sources of biases which could derive from questions design, sampling, data collection and analysis . WorkingSeven categories of factors which influenced TB service integration were: historical context, clinical capacity, motivation for integration, resource allocation, incentive and staffing, management coordination, and technical exchange.In both sites, the integration process was found to be contingent on the restructuring process of the public health organizations Table . This reThe capacity of CDC staff to treat TB patients remained challenging, reflecting the practical need for such integration. For example, the doctors in GN's designated hospital criticized the CDC doctors' lower clinical capacity, and poorer knowledge on complications such as liver damage and respiratory failure. The CDC staff seemed to agree with such criticism and suggested that integration could make the best of the public health strength of the CDC and the clinical strength of the hospital."It was difficult for us to treat TB patients, with limited technology, equipment and staff. We are especially not confident in treating the complications. After the clinical function was transferred to the hospital, we could focus more on management... We are public health doctors so we should do more on epidemiology and health education." The initial integration process faced some barriers. It was apparent that there was a lack of motivation of staff to move towards the integrated model within the CDCs. Interviews suggested that political will and economic interests would be potential barriers to integration. One member of CDC staff in SC expressed they favored the CDC model, which allowed them to better control the affiliated TB clinic. The CDC staff from GN reported that they did not want integration initially, complaining this would mean "sacrificing" their service."We did not agree to take this approach initially. Although treatment of the smear-positive patients was free, we could still charge from the smear-negative patients. We (CDC) were just set up at that time, with very constrained funding. We could have supplemented the funding through TB treatment. We made great sacrifice through the integration initiative. At that time, we were located in the same yard with the hospital. Both hospital staff and we (CDC staff) waited outside the doors to 'compete' for TB patients (laugh...) But we gradually accept this reality with the increasing funding from the government." Motivation for integration appeared to be mixed within the two hospitals. GN's hospital began to treat TB in 1996. Some recalled they did not want to take over the full TB service as the funding was rather limited to support their activities. On the other hand, some seemed to be happy with full integration as they thought treatment of TB could improve their reputation and increase the hospital revenues. In SC, a smooth transition process was observed in the hospital. By 2000, other districts of Shanghai had set up the TB clinics in the general hospital which may have motivated the local hospital to take over the TB service. In addition, as a designated hospital for other infectious diseases such as hepatitis, they had already established a good relationship with the CDC. Lastly, the health bureau agreed to allocate the related TB control budget, previously allocated to the TB clinics of the CDC, to the TB clinics within the designated hospital, with staff costs met by the district government. These factors are likely to have contributed to the smooth transition in SC.In GN, the hospital received limited funding from the health bureau from 1997 to 2003. Since 2004, it received project funding from the Ministry of Health (MoH), matched by local government. However, some felt that this funding was too limited to cover costs of service operation and were concerned about the approach to budget allocation which was often reallocated from the CDC."We have to pay extra money to buy a mini-car to conduct patient supervision activities and pay the associated costs from our general budget. We also need to subsidize the associated staff and electricity costs to conduct the X-ray service for TB diagnosis. The more we do, the more we have to pay from our own pocket... I believe there will be further support from the government when the Project comes to a close but the key issue is how to allocate the TB control budget to the hospital in the most effective way." This difficulty with funding may have contributed to the hospital attempting to make additional money from treating TB patients, such as prescription of unnecessary drugs and hospitalization. Although staff suggested that only serious TB cases were admitted, the estimated annual hospitalization rate of the TB patients was up to 20%. CDC staff were concerned that this would increase the financial burden of TB patients and may lead to multi-drug resistant TB due to the improper treatment. However, CDC staff were just sympathetic to the hospital's financial dilemma:"Hospitals are not fully funded by the government; it is understandable to have a higher hospitalization rate but they just admit too many. But what can we do? We could only open an eye and shut an eye as long as the patients are willing to afford. See, hospital directors always mention money." In SC, service operation has benefited from a more sustainable resource allocation system. The health bureau directly allocated the operational budget to the designated hospital and funded the TB clinical staff. The hospital mandated that severe TB patients should be referred to the specialist lung hospital or other infectious disease hospitals for treatment. Since funding was not a major concern, there was little sign of unnecessary hospitalization of TB patients. Rather, interviews revealed a more friendly and patient-orientated service provision in SC:"Our doctors normally would not admit the general TB patients and referral of severe TB patients is their responsibility. If we admit the general TB patients, other patients will complain. Especially the migrant patients, [who] do not have medical insurance so they can't be reimbursed for the inpatient cost".Interviews in both sites identified a difficulty in attracting and maintaining skilled TB staff. In both sites, both the doctors and nurses rotated from the infectious disease or respiratory departments on a regular basis. There was serious concern over staffing turn-around and its effect on quality of service:"We are anxious about the quality of service since no doctors are willing to work in the TB clinics for a longer term. The performance is satisfactory now, but there is crisis in the long run." The perceived reasons for an unstable TB team in the hospital included a concern about payment and the potential transmission of TB. For example, TB doctors in both sites thought they did not receive fair pay from the hospital regardless of their qualifications and experiences and the potential transmission risks from TB. They attributed the unfair pay to the fact that the performance-based payment system was not allowed in the TB clinics:"I feel unfair as a senior physician since there was no major difference for the payment between staff working in the TB department." "TB clinical staff received a 'bonus' from the respiratory department. The amount was much less than that for staff working in the wards, albeit they had the same qualifications and experience." (SC CDC staff)Other reported reasons for an unstable TB team included poor infrastructure, poor job prospects and lack of "preferential" policy  for the staff working in the communicable disease control of the hospital."It is not possible to talk about the preferential policy. See, our clinical building was the shabbiest in the hospital, without a waiting room. Most of the patients wait under the tree; other departments at least have waiting rooms." "The infectious disease allowance was too humble to attract medical staff to work for the TB clinic. It was the shrinking job market that brought young graduates to work here, neither the payment nor promising future of the TB work." In both sites, an integration hospital leadership group was established including leaders and experts from the health bureau, CDC and hospitals. This group met regularly to discusses the strategic issues, service coordination and emerging problems of TB control.In SC, some CDC staff thought such official meetings could not address problems effectively. They seemed to less rely on formal meetings and the health bureau to solve problems, but more on personal relationships:"Only for the bigger issues, we would invite the health bureau to coordinate. Once we ruin the personal relationship with the hospital, nothing can be done."In GN, staff also agreed personal relationships were important, but they emphasized a lot on the official coordination from the health bureau which was deemed effective to solve problems. As reported earlier, there was a lasting competitive relationship between the hospital and the CDC due the economic reasons. The health bureau might have thus worked as an effective mediator between the two parties."The health bureau organizes 4-5 special meetings for the CDC and designated hospital annually. The coordination from health bureau is effective. For example, the hospital tried to minimize the distribution of the case management allowance (to community doctors) and it was the health bureau that solved the problem. Our coordination with hospital is generally getting better and better." In GN, TB staff in the hospital attended the general TB training sessions organized by the CDC. The hospital also organized five to six training workshops for the TB-related staff per year. In SC, however, more collaborative and participatory approaches were observed. At the district level, the CDC organized a multi-disciplinary group of TB-related health professionals to discuss TB diagnosis and the quality of x-ray examinations on a weekly basis, including doctors from the respiratory and radiological departments of the designated hospital, community health centers and other general hospitals. This approach was likely to have filtered down from higher level CDC working culture, where the CDC at the municipal level organized multi-disciplinary consultation meetings on the monthly basis. These meetings allowed TB doctors from all the designated hospitals to exchange their experiences of TB diagnoses and treatment. The hospital staff in SC also felt that they received sufficient technical support from the district and municipal CDC during the initial period of integration:"The public health doctor of the district CDC came to guide our sputum test once per month in the first six months. Experts of the municipal CDC also came to check the quality of smears and sputum culture (three times)." The similarities and differences of the two sites in the process of integration are summarized in Table This study aimed to investigate the factors that influenced the process of integration of TB service from health care professionals' perspectives. The development of public health organizations and the enactment of the Law for Licensing Medical Practitioners appeared to influence how effectively integration occurred. In addition, an understanding by staff that hospitals had better capacity to treat TB patients than CDCs was a strong rationale for integration. However, the economic and political interests might act as a barrier to effective integration of TB control. The more prosperous site (SC) in Eastern China was more coordinated and thus had a better method of resource allocation and more patient-orientated service, compared with the poorer site (GN) in the West. Both sites shared the same challenges of attracting and retaining a skilled workforce for the TB services. The role of the health bureau was more directive in the poorer West, while a more participatory and collaborative approach was adopted in richer East China.Our results adds to existing research on factors that influence integration of TB into hospitals -11. ThesThe TB programme has long been working in parallel with the general health services. Integration not only involves the shift from a specialized organization to a horizontal one, but also a real transfer of responsibilities, rights and duties . In ordeResources are vital to ensure successful service integration ,31. The Different practices of hospitalizing TB patients in SC and GN could also reflect the conflicts between the clinical and public health perspectives of TB care delivery. Clinicians in the hospital should be educated to improve the public health awareness, and specifically implement an outpatient-based systematic public health TB approach. Both sites presented the challenges of retaining qualified TB personnel. To ensure that TB services are sustainable and maintain a high quality of care, fair pay should be provided to attract and stabilize a motivated and skilled workforce.Successful integration relies on effective inter-organisational collaboration, which could contribute to the efficiency and the quality of TB services by combining resources and expertise from different organizations . Recent Experiences from the two sites may not be generalizable within China, although they provide common but distinctive discussions around the drivers and barriers to service integration. This study is limited on the discussion of the process of integration, though a concurrent study is being conducted to compare the process and effect of implementing the integrated model with the CDC model in China. This study only focused on the collaboration between the CDC and designated hospital and neglected other important aspects of integration, such as the internal management of the designated hospital, the collaboration between the designated hospital and other general health facilities including primary health care organizations. However, we believe that the relationship between the CDC and designated hospital is the best entry point to understand the issues of integration. This study cannot escape the subjectivity and recall bias inherent qualitative studies. However, efforts have been made to improve the reliability and validity of the study findings.The process of integration identifies similarities and differences between sites in more affluent East China and poorer West China. Integration of TB services in the hospitals needs to address the challenges of stakeholder motivations and resource allocation. Effective inter-organizational collaboration could help to improve the efficiency and quality of TB control.The authors declare that they have no competing interests.GZ, XW, JW, QS and JY were involved in conception and design of this project, while GZ, XW, QS and JY were involved with the implementation of the project and analysis and interpretation of the data. GZ and XW have drafted the manuscripts while JW and QS have provided critical comments. All authors read and approved the final manuscript.The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6963/12/21/prepub


One of the key functions of health insurance is to provide financial protection against high costs of health care, yet evidence of such protection from developing countries has been inconsistent. The current study uses the case of Ghana to contribute to the evidence pool about insurance's financial protection effects. It evaluates the impact of the country's National Health Insurance Scheme on households' out-of-pocket spending and catastrophic health expenditure.We use data from a household survey conducted in two rural districts, Nkoranza and Offinso, in 2007, two years after the initiation of the Ghana National Health Insurance Scheme. To address the skewness of health expenditure data, the absolute amount of out-of-pocket spending is estimated using a two-part model. We also conduct a probit estimate of the likelihood of catastrophic health expenditures, defined at different thresholds relative to household income and non-food consumption expenditure. The analysis controls for chronic and self-assessed health conditions, which typically drive adverse selection in insurance.At the time of the survey, insurance coverage was 35 percent. Although the benefit package of insurance is generous, insured people still incurred out-of-pocket payment for care from informal sources and for uncovered drugs and tests at health facilities. Nevertheless, they paid significantly less than the uninsured. Insurance has been shown to have a protective effect against the financial burden of health care, reducing significantly the likelihood of incurring catastrophic payment. The effect is particularly remarkable among the poorest quintile of the sample.Findings from this study confirm the positive financial protection effect of health insurance in Ghana. The effect is stronger among the poor group than among general population. The results are encouraging for many low income countries who are considering a similar policy to expand social health insurance. Ghana's experience also shows that instituting insurance by itself is not adequate to remove fully the out-of-pocket payment for health. Further works are needed to address the supply side's incentives and quality of care, so that the insured can enjoy the full benefits of insurance. The World Bank  posited The current study analyzes the case of Ghana National Health Insurance Scheme (NHIS) and contributes to evidence on the financial protection provided by health coverage. Using data from two rural districts of Ghana, we evaluate the potential protective impact of NHIS on the financial burden of health care as measured in the total OOP amount spent on health and the probability that an individual encounters catastrophic health payment. While several studies have looked at the implementation aspect of NHIS , its equ"Within the next five years, every resident of Ghana shall belong to a health insurance scheme that adequately covers him or her against the need to pay out of pocket at the point of service use in order to obtain access to a defined package of acceptable quality of health service."The next section provides an overview of Ghana's health financing and the NHIS. This is followed by a description of the methods used to address the study's research questions. We next present our results and discuss the implications of the study's findings. The final section concludes.With a gross domestic product per capita of US$ 306 in 2007, Ghana is a low income country in the sub-Saharan Africa region. The country compares positively with peers in sub-Saharan Africa in a number of important health indicators, such as maternal and child mortality, fertility rate, and HIV/AIDS prevalence among adults ages 15-49 . HoweverSimilar to many countries coming out of the colonial regime, after independence in 1957, the Ghanaian government adopted a tax-based health financing system in which the government paid for services in the public sector. By the early 1970s, general tax revenue in Ghana, with its stagnating economy, could not support a tax-based health financing system. Ghana started to introduce nominal user fees in the public sector, which in 1985 were raised significantly with the aim to recover at least 15% of recurrent expenditure ,13. To oThe establishment of the NHIS in 2003 is a major step by the Ghanaian government to address the perverse problems of the "cash and carry" system. NHIS was fueled partly by the relative success of the numerous mutual health organizations (MHOs), which existed in 67 out of 138 districts in Ghana with very diverse management structures and benefit packages . HoweverFunding for NHIS comes mainly from a National Health Insurance sales tax of 2.5 percent, a transfer of 2.5 percent of formal sector contributions to the Social Security and Pension Scheme Fund, and from member contributions. In the beginning years of the scheme, 70 - 75 percent of total insurance revenue came from tax revenue, 20-25 percent from formal sector contribution, and only about 5 percent from contribution of members in the informal sector . PremiumEarly experience in the implementation of NHIS reveals a number of problems. Despite the explicit policy to exempt premium for the indigents, poor people were much less likely to be insured than the better off ,9. ThereTo date the impact of NHIS on health service utilization and OOP expenditure has not been rigorously evaluated at the national level. Several studies conducted in a small number of districts suggested that NHIS has encouraged utilization of curative health services ,20. NoneOne of the key values of health insurance is to help people deal with the unpredictability of illness and medical spending . It is wLooking at the absolute amount of OOP payment changed as the result of having health insurance is not adequate to judge its effect on the financial burden of health care. Low OOP payment may just derive from the fact that insurance does not trigger any increase in utilization; hence no additional payment is needed. On the other hand, high OOP payment may not necessarily be bad if it buys substantial improvements in quality and/or quantity of services and if the households can afford to pay . Thus, tAlong this line of argument, Wagstaff and van Doorslaer  defined Following the practice of the existing literature, we adopt two measures of the financial protection effects of insurance in this study. They include the absolute amount of OOP expenditure and the likelihood of high or catastrophic expenditure.i with the outcome of interest Y can be estimated as followed:A generic model for estimating the impact of health insurance for an individual where the F function in health service research typically takes the form of linear, probit, logit, or poisson depending on the distribution of the outcome variable, HI denotes insurance status, X is a vector of observable characteristics, Z is a vector of characteristics unobservable to the researchers, and ε is the stochastic term, usually assumed to be normally distributed. Note that included in X and Z could be values shared by all household members.Because Z is unobservable, it is included in the error term. The estimate of the health insurance (HI) effect on Y will be biased if Z contains at least one element that is systematically correlated with both insurance status and the outcome of interest (HI is endogenous). For example, if Y is a measure of health service utilization, typical candidates for such an element of Z would be the individual's risk aversion or poor health condition, which are likely to be positively correlated with both insurance and Y. In such a case, failure to control for this interfering effect of the omitted variables will lead to an overestimation of the insurance effect. Unfortunately, while tools for dealing with this omitted variable bias are widely known , in reality these tools often are not readily available.In this study, we obtained information on the individual chronic health condition and whether the study respondents assessed their health status as "bad." While ultimately there will be other confounding factors that we have not controlled for, poor health status is a commonly cited reason for adverse selection in the insurance literature . Thus beThe data for this study come from a household survey in two districts of Ghana conducted in September-October 2007. The survey was funded by the United States Agency for International Development and conducted by the Health Systems 20/20 Project, led by Abt Associates Inc., in collaboration with the Research and Development Division of the Ghana Health Services. The two districts are Nkoranza and Offinso, both predominantly agricultural and relatively poor. Nkoranza is one of the districts in the country where a MHO operated before the NHIS; Offinso had no prior experience with community-based financing.Table The household survey applied a two-stage cluster sampling approach. In the first stage, municipalities in the urban and rural areas (towns and villages) were selected based on population size and geographical dispersion. In the second stage, households were randomly selected from each municipality. Insured people were oversampled to increase power and sampling weights were used to adjust for this fact. A total of 2,500 households were surveyed, giving a final sample of 5,879 individuals in Offinso and 5,738 in Nkoranza.The household survey collected detailed information on the insurance status of all household members. It recorded any incidence of illness and injury over the two weeks preceding the survey, antenatal care, delivery, and hospitalization over the preceding 12 months, as well as the associated OOP payment for such events. From this information, we computed a variable for annual OOP health expenditure . Note that using two-week expenditure on illness and injury to extrapolate to a 12-month figure could be problematic due to seasonality of illness and related health care seeking. However, because the field survey only last for more than a month, any potential seasonality problem should be the same for NHIS members and non-members. The derived OOP expenditure variable covers all sources, informal and formal care, from both the government and private sectors. Note that this is expenditure associated with specific health problems only; it does not include preventive care. The survey also collected information on ownership of land, assets , and living conditions . From this information, a wealth quintile variable was constructed using the principle component method.In this study, outcomes of interest include annual OOP health expenditure on curative care and measures of catastrophic health payment. For the latter, a practical difficulty is that the survey did not collect information on household income or consumption expenditure to be used for comparison. We therefore use an indirect measure, comparing the health expenditure of individuals in each wealth quintile in the sample with the average income and non-food consumption expenditure of the corresponding quintile obtained from Ghana's latest Living Standard Survey 2005-2006) . In doin . In doiThe key independent variable of interest is whether an individual had NHIS at the time of survey. In addition to education, occupation, and gender of the household head, the covariates include individual self-reported chronic health conditions and self-assessed health status, gender, ethnic Akan, household size, assets and living conditions, and whether at least one member of the household joined a local solidarity scheme. These covariates are included because they can potentially confound the insurance - OOP expenditure relationship. For example, richer households may be more likely to afford health insurance while paying more for health services at the same time. Therefore, not controlling for wealth status may lead to overstating the amount that insured people pay for health services. The last variable, membership in a local solidarity scheme, serves as an indicator of social capital, which could potentially be correlated with insurance status and health service outcomes.In this paper, binary outcomes will be estimated with a probit model. While both probit and logit are typically used for binary outcome, the former is preferred because it produces estimates for marginal effect, which shows the difference in the probability of having catastrophic payment between the insured and the uninsured. For OOP expenditure, a linear model is not suitable because data are highly skewed with a large number of zeros. Taking log of expenditure data is also not preferred because the large number of observations with zero expenditure will be dropped out. Following the common practice in health service research, we employ a two-part model, which adds flexibility by modeling separate processes to explain the probability of positive expenditure and the average OOP amount . In the Table As shown in Table Some interesting observations emerge from Table Table Table The socioeconomic gradient observation is corroborated in Figure Table Both models 1 and 2 show that the insurance effect is in the expected direction, i.e. insurance has been shown to reduce OOP among its members. The marginal effects of insurance are smaller  when negative health status indicators are not controlled for (model 2 compared to model 1), suggesting that the derived estimates could be on the lower bound if adverse selection due to poor health status has not been fully controlled for. If there remain omitted variables that drive adverse selection in the same way as health status , the true insurance effect will be larger than that observed in models 1 and 2.It is noteworthy that in model 1, insurance effect is only marginally significant (at 10 percent level) and rather small . To examine its magnitude relative to household capacity to pay, we compare that amount with the average figure on household non-food consumption expenditure from Brong Ahafo region, where Nkoranza is located. We take the household non-food consumption expenditure per capita for Brong Ahafo reported in the Ghana Living Standard Survey 2005-2006 (Tables 9.2 and 9.5)  and adjuTable A question of interest is whether the effect of insurance works differently for the poor versus non-poor. We examine this by performing separate estimations for the poorest 20 percent versus the rest of the sample. The results, presented in Table This study finds evidence that the NHIS in Nkoranza and Offinso has helped reducing the incidence of catastrophic OOP expenditure among its members. Depending on the indicators, the probabilities of catastrophic expenditure decreased by 0.5 to 1 percentage point among NHIS members, a reduction of 36 percent to 67 percent of the sample means. In terms of the absolute amount of OOP expenditure, insurance's marginal effect is small and only significant at 10 percent. These seemingly contradictory patterns can be explained by the higher volume of service consumed by insured patients, as shown in the existing literature ,20, and The protective effect of insurance against catastrophic expenditure is particularly strong among the poor, who are typically more vulnerable to health shocks than the rest of the population. This finding supports the Ghanaian government's decision to exempt premium for the indigenous. In fact, given that insurance coverage is substantially lower among poor people, further efforts are needed to enroll this vulnerable population. For example, the criteria for being considered as "indigent," and hence exempted from premiums, are currently too strict . They exA closer examination of the results of this study reveals a more complicated picture. Despites NHIS's generous benefit packages, its members still incurred an OOP payment that equals 72 percent of that incurred by nonmembers. A nontrivial portion of this amount is spent on items that should be covered by insurance, such as drugs, tests, antenatal care, and hospitalization. These findings raise a number of questions on the implementation aspects of insurance which are highly related to the early experience with NHIS reported in the existing literature . For exaAs noted earlier, because OOP payment is the net effect of the change in unit price and volume of services utilized, the presence of OOP payment among the insured is not necessarily bad. It has been shown that NHIS has brought about a significant increase in health care demand among its members ,9. InsurThe current analysis ultimately has some limitations. We have not been able to address fully the selectivity of insurance. Although we can control for chronic conditions and self-assessed bad health status, there remain potentially omitted variables that bias these results. As such, the documented association may not represent the true magnitude of the NHIS effects on the outcomes of interest. Most likely, we believe that bias, if exists, will make the results conservative and the true magnitude of the effects may be higher than what is documented here. The second drawback results from the lack of household income and expenditure data in our sample, which renders the estimates imprecise. The low percentage of the studied population having had any positive health expenditure and the skewness of expenditure data have always been a challenge in health service research. Finally, our results only pertain to two out of 138 districts of Ghana, and hence cannot be generalized to the country as a whole.Future studies can build on this research in many respects. For example, to assess more systematically the equity aspect of the Ghana NHIS, benefit incidence analysis would be suitable to document how much people with different wealth status benefit from the NHIS. With better data, one could explore the effects of differentiating the threshold z for catastrophic health expenditure across income levels. It would be useful to quantify the degree of adverse selection as it will affect the financial viability of the insurance fund, especially when coverage is still low. Moral hazard from members could eventually become a problem with the NHIS due to the generous benefit package and lack of requirements for coinsurance. Finally, some in-depth case studies will be useful to shed light on the operational aspects of NHIS and on quality of care for its members.Limitations notwithstanding, this study makes a potentially valuable contribution to the evaluation of the Ghana NHIS in particular and impact evaluation of health insurance in general. It shows that with strong commitment to social health insurance , a poor country can achieve financial protection against health care cost for its population, in particular the most vulnerable group. Another lesson learnt from Ghana is that instituting insurance by itself is not adequate to remove fully the out-of-pocket payment for health care. Insured patients are still required to pay for items that should be covered by insurance and for informal care. Without proper regulation and incentives for the supply side to improve quality and availability of services, insurance cannot be an attractive product. This in turn will be a hindering factor in coverage expansion and ultimately will affect the prospect of achieving universal coverage.Ghana's experience is applicable to many developing countries, especially the countries in sub-Saharan Africa. The context for social health insurance is rather similar. Most countries in the sub-Saharan Africa in the 1980s and 1990s underwent health financing reform where user fees in the public facilities were instituted and different forms of the MHOs were piloted. Social health insurance is being considered as a way to go for alleviating financial burden and improve access to health services. The results on protection effect against catastrophic OOP expenditure are encouraging for many low income countries, which are on the way to develop and expand their social health insurance.Proof of the sign of bias in linear modelsSuppose there is only one variable in Z, called z, that is systematically associated with HI status and Y, then the true relationship between Y, HI, and z can by modeled as:Not being able to observe z, we could wrongly model the relationship as:Because z is systematically related to HI, the relationship between them can be specified as:Substitute (3) in (1), we have:1+ γ1δ3), while the true underlying effect is β1. γ1δ3 is the bias caused by omitting z. If z is something that is positively correlated with both insurance and health expenditure , both γ1 and δ3 will be positive and the bias will be positive. Note that the bias is positive as well if z is negatively correlated to both insurance and health expenditure . Because the expected insurance effect on Y in this study is negative (a reduction in OOP), a positive bias will result in an estimation that is smaller than the true effect in absolute value.Thus, without controlling for z, the estimated effect of HI is (βThe authors declare that they have no competing interests.HTHN developed the analytical approach, performed the statistical analysis, and drafted the manuscript. YR and HW contributed to the drafting of the manuscript. All authors have read and approved the final manuscript.


Verbal autopsy methods are critically important for evaluating the leading causes of death in populations without adequate vital registration systems. With a myriad of analytical and data collection approaches, it is essential to create a high quality validation dataset from different populations to evaluate comparative method performance and make recommendations for future verbal autopsy implementation. This study was undertaken to compile a set of strictly defined gold standard deaths for which verbal autopsies were collected to validate the accuracy of different methods of verbal autopsy cause of death assignment.Data collection was implemented in six sites in four countries: Andhra Pradesh, India; Bohol, Philippines; Dar es Salaam, Tanzania; Mexico City, Mexico; Pemba Island, Tanzania; and Uttar Pradesh, India. The Population Health Metrics Research Consortium (PHMRC) developed stringent diagnostic criteria including laboratory, pathology, and medical imaging findings to identify gold standard deaths in health facilities as well as an enhanced verbal autopsy instrument based on World Health Organization (WHO) standards. A cause list was constructed based on the WHO Global Burden of Disease estimates of the leading causes of death, potential to identify unique signs and symptoms, and the likely existence of sufficient medical technology to ascertain gold standard cases. Blinded verbal autopsies were collected on all gold standard deaths.Over 12,000 verbal autopsies on deaths with gold standard diagnoses were collected . Difficulties in finding sufficient cases to meet gold standard criteria as well as problems with misclassification for certain causes meant that the target list of causes for analysis was reduced to 34 for adults, 21 for children, and 10 for neonates, excluding stillbirths. To ensure strict independence for the validation of methods and assessment of comparative performance, 500 test-train datasets were created from the universe of cases, covering a range of cause-specific compositions.This unique, robust validation dataset will allow scholars to evaluate the performance of different verbal autopsy analytic methods as well as instrument design. This dataset can be used to inform the implementation of verbal autopsies to more reliably ascertain cause of death in national health information systems. Verbal autopsy (VA) is a critically important tool to measure causes of death in populations without complete medical certification of causes of death. A variety of methods have been proposed for VA cause assignment ,2, rangiSeveral validation studies of VA cause assignment methods have been published ,12,20-31The Population Health Metrics Research Consortium (PHMRC) gold standard verbal autopsy validation study was initiated in 2005 to address these research limitations and to ensure that comparative assessments of VA performance were based on clinically reliable diagnoses. We designed the study as a multisite collaboration that aims to address some of the key limitations of previous validation studies and stimulate the development of new methods or refinements of existing methods. The primary goal was to collect a dataset that would help provide more definitive answers as to which VA approaches are more valid and to capture data in a standardized way. In this paper, we describe the design of the study, the criteria used to establish a gold standard (GS) cause of death, the implementation of fieldwork, and the creation of standardized datasets for developing and testing new methods.Gold standard VA data collection was implemented in six sites in four countries: Andhra Pradesh, India; Bohol, Philippines; Dar es Salaam, Tanzania; Mexico City, Mexico; Pemba Island, Tanzania; and Uttar Pradesh, India. Table Research at the Andhra Pradesh, India, site was implemented and coordinated through the George Institute for Global Health, India, and was centered in the main capital city, Hyderabad, as well as the neighboring areas of Ranga Reddy, Medak, and Nalgonda. Hyderabad is 100% urban with a population of roughly 3,830,000 inhabitants. The neighboring area Ranga Reddy has a similar population size  and is roughly half urban and half rural. The Medak and Nalgonda areas are similar to each other, both roughly 14% urban, comprised of 3,248,000 people in Nalgonda and 2,670,000 in Medak.The Bohol Island site was led by the Research Institute for Tropical Medicine in Manila. Bohol is a tropical island province located in the Central Visayas of the Philippines, with 46 municipalities and Tagbilaran City. Verbal autopsies were collected over the entire island, as well as a small proportion from Manila. According to the 2007 census, 1,230,000 people live in Bohol. Manila is urban, while Bohol is divided into roughly 46% urban and 54% rural.The research site in Dar es Salaam, Tanzania, was managed by collaborators at the Muhimbili University of Health and Allied Sciences. Verbal autopsies were collected from all over the city of Dar es Salaam, which has a population of roughly 2,487,000 people according to the 2002 census, with 94% of people living in urban areas and 6% living in rural areas.The Mexican study was coordinated by the National Institute of Public Health in the Federal District and the state of Morelos. According to the 2010 Census, 8.85 million inhabitants live in the Federal District and 1.8 million live in Morelos. Sixteen percent of the population of the state lives in rural areas .Pemba Island, Tanzania, is the smaller of the two islands of the Zanzibar archipelago. The research there was coordinated through the Public Health Laboratory Ivo de Carneri as part of a collaboration between the Ministry of Health and Social Welfare and Johns Hopkins University. Verbal autopsies were collected from all areas of the island. This island has a population of roughly 400,000 inhabitants. The island is 99% rural and 1% semi-urban.Finally, the Uttar Pradesh site in India was led by collaborators at the CSM Medical University  in Lucknow. Verbal autopsies were collected from a wide range of districts in the state of Uttar Pradesh: Ambedkar Nagar, Bahraich, Barabanki, Basti, Faizabad, Gonda, Hardoi, Lakhimpur, Lucknow, Rae Bareli, Sitapur, Sultanpur, and Unnao. Table The instrument development was based on the WHO standardized verbal autopsy instrument , which iThe PHMRC instrument is comprised of a general information module, an adult module, and a child and neonatal module. Skip patterns were integrated into the general information module to collect the age of the deceased and then direct interviewers to the correct module to administer. In administering the WHO instrument, the interviewer must first determine the age of the deceased and select the correct instrument to deliver, which results in the potential for more interviewer error and a less fluid interview. The general information module, which is administered in all verbal autopsies, collects items such as education of the decedent, household characteristics, and a household roster. The adult module collects a history of chronic conditions, symptoms of the deceased, women's health questions if the decedent is female, alcohol and tobacco use, and injury information; it also transcribes any available medical record and death certificate information. The child and neonatal module first asks background questions on information such as whether the mother is still alive, where the deceased was born, the size of the decedent at birth, and the delivery date. The questionnaire then ascertains whether the decedent was a stillbirth and, if so, collects symptom questions, such as signs of injury. If not, the questionnaire collects more general information such as the age of the baby or child when they became ill and the age at death. If the decedent is under 28 days (inclusive of stillbirths), a maternal history is collected. In addition, if the decedent is under 28 days and was born live, a full set of neonatal symptom questions are collected. If the decedent is between 28 days to 11 years, infant and child symptom questions are asked. All available health records and death certificates are transcribed for both neonatal and child deaths. Finally, for all ages, the open narrative section was moved to the end of the interview, after the structured questions. This was done to ensure that in future work, we could remove the open-ended items without concern that the results collected in this study were a function of the open-ended items coming prior to structured content.In addition to the structural changes, there are important differences between the PHMRC instrument and the WHO instrument. First, the WHO adult module is administered on ages 15 and above, while the PHMRC adult module begins at age 12. This expansion of the ages included in the adult module ensures that conditions clinically present, such as maternal mortality in 12 to 14 year olds, are captured through this instrument. Second, a substantial portion of the questions were reworded to ensure clarity. Medical terminology was converted to easily understandable descriptions to target a lay population. For example, "Did s/he have abdominal distension?" was reworded to "Did [NAME] have a more than usual protruding belly?" Information was also added for precision, or removed to ensure only the most diagnostically relevant information was collected. Similarly, we added or dropped entire questions to capture the most essential information, while reducing the duration of the interview as much as possible. One common question type dropped from the instrument was the duration of certain symptoms. For example, the PHMRC instrument asks whether adults had developed a lump in the neck, armpit, breast, or groin but dropped the follow up question "For how long did s/he have the lumps?" as the presence of the symptom alone was the most important information. Another common question type dropped from the WHO instrument was about treatment that had been received by the decedent, as they were less important in informing the cause of death. Finally, the PHMRC instrument did not include questions about chronic conditions in children, such as cancer, tuberculosis, and diabetes. Additional file A key challenge for the study was to identify the cause list for each of the three age groups for which we would seek to collect a sample of gold standard deaths. Our selection of the target cause list was based on consideration of the WHO estimates of the leading causes of death in the developing world in each age group, those causes for which verbal autopsy might be able to function adequately because unique signs and symptoms could potentially be collected in an interview, and the potential to find, in the six sites, deaths with sufficient laboratory, medical imaging, and pathological detail in order that a gold standard cause of death assignment could be made. The cause lists were also designed so that they were mutually exclusive and collectively exhaustive. The target cause list for adults, children, and neonates included 53, 27, and 13 GS causes, respectively, plus stillbirths . The medical information from all qualifying cases selected by the clinicians was extracted and sent to the George Institute Hyderabad office for enrollment in the verbal autopsy study.In Bohol, the majority of deaths were reviewed at the Bohol Regional Hospital. This facility is the referral hospital for Bohol Province with the highest available standards of clinical investigation and hence diagnosis. Three nurses monitored all deaths in the hospital. They ensured that all reports of investigations (imaging and laboratory) were located and attached to the charts. In addition, to augment the number of deaths collected, 467 deaths were recruited from two hospitals in Manila: the Veterans Memorial Medical Center and the Rizal Medical Center. In all locations, the nurses summarized the case notes, including reports of investigations, onto the medical data extraction forms. MDEFs were first reviewed by two study physicians who assigned cause of death and decided by diagnosis and GS level which VAs should not be collected. Deaths were reviewed as soon as possible after the death.At the Dar es Salaam site, five health facilities were used as recruitment sites. These were Mwananyamala Hospital, Temeke Hospital, Muhimbili National Hospital, Ocean Road Cancer Institute, and Hindu Mandal Hospital. Mwananyamala and Temeke are both district hospitals, each of which records roughly 1,500 deaths per year. Ocean Road Cancer Institute is the only cancer treatment facility in Tanzania and was an important source for causes such as cervical cancer, esophageal cancer, breast cancer, leukemia, prostate cancer, and lymphomas. Muhimbili National Hospital is a referral and teaching hospital with a higher mortality rate than the other enrolled facilities. Hindu Mandal Hospital is a private hospital in the heart of Dar es Salaam. It has a well-established HIV/AIDS clinic and commonly receives noncommunicable disease cases. At each location, a nurse affiliated with the study reviewed medical records to identify qualifying cases. The cases identified by the nurses were reviewed by physicians, who filled out the MDEFs with the gold standard levels for the cases that were eligible for enrollment. The nurses spoke with family members of the deceased if present at the hospital to enroll them in the study, collect their consent, and obtain mapping information and directions for a verbal autopsy interview.In Mexico, after obtaining authorization to work in each medical unit, a group of six trained physicians reviewed the medical records of cases (and when available the reports from autopsies) that could be included in the study, filled an extraction form for each case, and classified them as levels 1, 2, or 3 according to the gold standard criteria proposed by the PHMRC. Only cases classified as levels 1 and 2 were considered eligible for the study. The original design considered the inclusion of only one to three large hospitals in Mexico City, but due to the difficulty of completing the quota of gold standard cases, hospitals from the health service network of the Federal District government and from the Ministry of Health of the state of Morelos were included. The data were collected from 36 public hospitals: 33 from the Federal District and three from Morelos.In Pemba, there are four major government hospitals on the island, though most facilities do not have a certified medical doctor present and are managed by medical assistants and nurses. Surveillance systems were put in place in all four hospitals to identify deaths and to classify them into GS categories. The hospital supervisor recorded complete identification information upon admission of each patient, and the attending physician medical assistant confirmed the admission diagnosis. Hospital supervisors ensured that the signs and symptoms experienced by the patient were recorded and that a mortality form with the cause(s) of death was filled out by the attending physician in the event of a death. All forms were sent back to the field headquarters for data entry. A computer algorithm was run to identify cases meeting GS criteria, and all GS cases were recorded in a database. A computer listing was prepared with identifier information to schedule the VA interviews.In Uttar Pradesh, the gold standard deaths were enrolled at CSMMU, Lucknow, which is a tertiary care government facility with patient inflow from all over Uttar Pradesh and bordering states, including districts in the neighboring country of Nepal. The catchment area spreads over a radius of more than 500 km, of which about 85% cases come from 13 districts surrounding Lucknow. There was 24-hour surveillance at facilities and all patients were enrolled with an address. When a death occurred, the project medical officer reviewed the patient case sheet in consultation with the resident doctor in order to assess the GS levels against standard criteria.Once enrolled, the VA interviewers at each site attended a training session led by consortium members using standardized materials and an interviewer's manual. The training manuals provided information on the study background, the roles and responsibilities of the VA interviewer, background on how VA cases were selected, instructions for administering the questionnaire, and information on every question in the instrument. The manual provided guidance on how to handle an array of questions or concerns, tips for building rapport with the respondents, and probing as needed to collect reliable information.Following the training, VA assignments were given to interviewers blinded to the medical information or cause of death of the decedent along with directions or map queues to the households. In some sites the families were contacted in advance to schedule an appointment, though this decision was left to the sites' discretion. All interviews were collected after a culturally appropriate grieving period had passed. The minimum grievance period was six days in Bohol and the maximum was six months in Mexico . The maximum amount of time post-death that an interview was collected was eight months in the Mexico site.The rate of interview refusals varied by site from 1.8% to 9.5%. For those that consented to a verbal autopsy, the instrument was administered on paper in the field, and returned to the field headquarters for double data entry. Interviews lasted an average of 45 minutes across all of the sites.To ensure the highest quality data was collected, quality control checks were performed both at the individual site level, as well as at the Institute for Health Metrics and Evaluation (IHME), where all data were transmitted through a secured password-protected site for analysis.In all sites, supervisors were trained in the protocols for monitoring quality control at the site level. Supervisors were instructed to observe VA interviewers in the field during the early stage of data collection to ensure they were conducted properly and to provide guidance. Supervisors additionally checked every VA form collected throughout the study to ensure that it was filled out consistently and correctly. If issues were identified by the supervisor, a reinterview was conducted as needed. The field interviewers had periodic meetings with their supervisors to discuss performance, progress, and challenges. Supervisors at most sites additionally reinterviewed a portion of the verbal autopsies to spot check the quality of the information collected.At IHME, we systematically evaluated all datasets electronically for numerous types of quality issues by a comprehensive set of codes. First, we reviewed the dataset for missing values and for incorrect skip patterns that result in specific questions having been filled in or left blank erroneously. The dataset was also evaluated to determine if any of the observed values fell outside of expected ranges. For example, if the response for a neonatal symptom duration was greater than 28 days , this value was flagged. Next, if the dataset was submitted in multiple sections, we examined the final comprehensive database for any technical issues that may have occurred in merging the individual files. Finally, we merged the dataset with the gold standard medical record information, which was separately transmitted to IHME by the site coordinator. We examined the observations for consistency between the two sources of information, such as the sex of the decedent as reported in the medical record and as reported by the verbal autopsy respondent. Any issues determined through this stringent checking process were compiled into a report and sent to the site to review. Site coordinators were asked to speak with the interview staff and rectify any correctable issues such as data entry mistakes.In addition to the full dataset as it was collected, we have also created a series of dichotomous variables from each of the polytomous  and continuous (duration) variables. Some analytical methods can only use dichotomized variables, so this effort to create the dichotomous variables increases the information available to these types of empirical methods. For each continuous duration item, depending on the item, we identified a short or long cutoff. For example, a duration of 8.8 days marks long duration of a fever. If a VA reports a fever of 10 days, it is considered to have the symptom of "having a long fever." We determine the cutoff as being two median absolute deviations above the median of the mean durations across causes (MAD estimator). The MAD estimator can be used as a robust measure of the standard deviation and is especially useful in cases where extremely long durations may be reported, which would bias measures such as the standard deviation. Additional file There has long been concern that the performance of a VA instrument and the associated analytical method for assigning cause could be different for deaths where the decedent died in a hospital or had made extensive use of health services prior to death, compared to deaths with no health care experience (HCE). As an attempt to examine how VA may work in communities with limited or no access to health care services, Murray et al.  studied Given this empirical finding, we believe it is useful to test how excluding household recall of health care experience likely provides a more realistic assessment of how VA performs in communities without access to health services. As such, we have created two versions of the datasets developed above, one version with all variables and one version excluding recall of health care and medical records. Specifically, the without HCE dataset excludes the following information. First, a series of questions asked if the deceased had any specified conditions, which would likely indicate a health care provider had diagnosed the individual. Each of the following conditions was asked: "Did decedent have ?" Second, if any medical records were available, the interviewer was asked to provide a transcription of the last note on the medical record. Third, if a death certificate was available, the interviewer was asked to record the immediate cause of death, first underlying cause, second underlying cause, third underlying cause, and contributing causes from the death certificate. Finally, at the end of the questionnaire, an open-ended section was provided to collect any comments from the interviewer, as well as to ask the respondent "to summarize, or tell us in your own words, any additional information about the illness and/or death of your loved one?" Excluding this entire section excludes both open narrative recall of HCE but also, in the case of PCVA, excludes any other information on timing and sequencing of signs and symptoms that might be conveyed in this section.The structured instrument includes various open text items. First, some questions in the instrument ask the respondent to choose from a list of specified response options. For example, "Where was the rash located?" has the following response options: face, trunk, extremities, everywhere, or "other (specify: ____)." If the response is not one of the listed options, the respondent is asked to fill in the location of the rash as the "other" response. The questions that include an "other" free text response option are as follows: "Where was the rash located?"; "Where was the pain located?"; "Which were the limbs or body parts paralyzed?"; "What kind of tobacco did [NAME] use?"; "Did [NAME] suffer from an injury or accident such as a ____?"; "Where was the deceased born?"; "What were the abnormalities?" in reference to any abnormalities at time of delivery; "Where did the deceased die?"; "What was the color of the liquor when the water broke?" in reference to labor; "Where did the delivery occur?"; and "Who delivered the baby?" In the questions that collect information about a health facility or midwife, free text responses collected the name and address of the place or person. In addition to these free text items, if any medical record or death certificates were available, the interviewer was asked to transcribe the information from the records as free text. Finally, at the end of each interview, the open narrative question "Summarize, or tell us in your own words, any additional information about the illness and/or death of your loved one?"(as described above) was collected in addition to any notes from the interviewer.Open text could in theory be highly informative, especially household recall of HCE and an interviewer's direct recording of death records or hospital records kept by the household. These observations are likely to be available in populations with some access to health care services. To make this information available to automated methods, we processed open text in the following steps. First, all free text was compiled into a database and a dictionary was created to map all similar words to the same stem word. For example, the terms AMI, myocardial infarction syndrome, acute myocardial infarction, ISHD, MI, coronary heart disease, CHD, IHD, MCI, and MYIN would all be mapped by the dictionary into the same variable . Next, a program called README  extractsFor empirical VA methods that must be developed using the pattern of responses observed in a dataset, validation needs to be undertaken on a set of deaths that were not included in the development of the method. This is the concept of a training dataset distinct from a test dataset. Further, as recommended in Murray et al.  it is imIn order to have an efficient cause list for the analysis, we have reduced it in two steps as illustrated in Table Table It is interesting to note the cause distribution by quality of the gold standards. Table The distribution of cases  across the six sites is shown in Additional file PHMRC was able to obtain completed VA interviews for more than 12,000 deaths with GS assignment of true cause of death. Because of the poor quality of medical record-keeping and limitations of diagnostic technology in many hospitals, to identify more than 12,000 GS deaths required reviewing and screening a much larger number of records. While it was difficult in many sites to obtain sufficient documentation for some causes of death overall across all six sites, we were able to find enough deaths for 46 adult causes, 22 child causes, and 12 neonate causes, excluding stillbirths, from the original cause list. The implementation of the project revealed just how poor the quality of medical records and diagnosis is in some institutions. This finding reaffirms our original hypothesis that convergent validity between verbal autopsy and poorly assigned hospital cause of death is not a measure of criterion validity.An important potential limitation of the study is the extent to which the cause of death based on fulfilling the clinical, laboratory, medical imaging, and tissue pathology criteria in this study are the true cause of death. Studies in high-resource settings  suggest Some readers may object to the use of "gold standard" in describing our dataset. We believe, however, that we have implemented the best possible approach to assigning causes of death. In nearly all settings, postmortem rates are low and subject to severe selection bias toward diagnostically challenging and nonrepresentative deaths for a cause. For both implementation and selection bias reasons, we do not foresee VA validation studies being undertaken using large samples of deaths with postmortem autopsies. Clearly defined clinical, laboratory, imaging, and tissue pathology criteria as used in this study are the best that can be implemented. As such, we believe the use of the term gold standard for this dataset is appropriate.A particularly vexing issue in VA validation studies is that by their nature they are conducted on deaths that have occurred in hospital. What would be the performance of VA for deaths in the community? There are potentially three distinct aspects to this question. First, the cause-composition of deaths in the hospital and the community will be different. Fortunately, because we create multiple test datasets with widely varying cause compositions, this issue will not influence the results from VA validation studies as long as the methods recommended by Murray et al.  are follIdeally, all countries would have in place functioning vital registration systems that capture all deaths and include a medically certified cause of death according to the procedures and rules of the International Classification of Diseases in force at the time. While progress toward this goal is being made, it is painfully slow, and without greater government commitment, will not be a reality for most developing countries for decades to come ,46. To mOne lesson learned from the complexity of converting free text into dichotomous variables is that future VA instruments may want to incorporate a series of checklist questions based on the free text variables that improve VA performance. Rather than free text, items could be included such as "Did anyone tell you or do you have any documentation mentioning acute myocardial infarction, MI, ischemic heart disease, or coronary heart disease?" These checklist items would be completed by the interviewer after questioning the respondent and examining the medical records and other documentation available. In this way, the task of reading free text and translating it through a dictionary would be simplified and focused only where it is likely to change the results.We have described the development and usefulness of the largest, perhaps only dataset with gold standard cause of death assignment and matching verbal autopsies for more than 12,000 deaths in four countries. We expect that this will facilitate further development of verbal autopsy and perhaps other cause of death measurement approaches in countries with poor vital registration and certification practices. The utility of this dataset will undoubtedly improve if additional cases, in different populations, and for different diseases than those reported here, are added in future studies, provided the same protocols and standards are applied. In this way, confidence in the utility of verbal autopsy methods will increase and result in their wider application in countries to reduce ignorance about the comparative importance of leading causes of death.CSMF: cause-specific mortality fractions; GS: gold standard; HCE: health care experience; MAD: median absolute deviation; MDEF: medical data extraction form; PCVA: physician-certified verbal autopsy; PHMRC: Population Health Metrics Research Consortium; VA: verbal autopsy; WHO: World Health OrganizationThe authors declare that they have no competing interests.CJLM, ADL, and RB conceptualized and organized the study. CJLM, ADL, RL, and SLO drafted the manuscript. DP, RJ, and BN directed the data collection at the Andhra Pradesh site; IR, ML, DS, VT, and HR directed the data collection at the Bohol site; ZP, ED, WF, SM, and MS directed the data collection at the Dar es Salaam site; BH, SG, RL, DRV, and MR directed the data collection at the Mexico site; SS, SMA, UD, and AD directed the data collection at the Pemba site; and VK, RA, VD, AK, and RP directed the data collection at the Uttar Pradesh site. AB, LD, ADF, HK, RL, and IR conceptualized analytic strategies and development of methods. SLO organized and managed collaboration, data collection, and analytics. All authors have read and approved the final manuscript.Differences between the standardized WHO instrument and the PHMRC instrument.Click here for fileGeneral module of the full verbal autopsy instrument used in the field in the PHMRC study.Click here for fileAdult module of the full verbal autopsy instrument used in the field in the PHMRC study.Click here for fileChild and neonate module of the full verbal autopsy instrument used in the field in the PHMRC study.Click here for fileTarget cause list reduced to analysis cause list.Click here for fileGold standard (GS) definitions in the PHMRC study for adults.Click here for fileGold standard (GS) definitions in the PHMRC study for children/neonates.Click here for fileMedical data extraction form (MDEF) used to extract gold standard data for the PHMRC study.Click here for fileDuration cutoffs used when making variables dichotomous.Click here for fileConversion of polytomous symptoms into dichotomous symptoms.Click here for fileLow-content polytomous items including response frequencies.Click here for fileREADME dictionary developed to convert free text responses into usable key terms.Click here for fileFinal analysis cause list and numbers of deaths by site.Click here for file


Excessive loads of nutrients transported by tributary rivers have been linked to hypoxia in the Gulf of Mexico. Management efforts to reduce the hypoxic zone in the Gulf of Mexico and improve the water quality of rivers and streams could benefit from targeting nutrient reductions toward watersheds with the highest nutrient yields delivered to sensitive downstream waters. One challenge is that most conventional watershed modeling approaches  used in these management decisions do not consider uncertainties in the predictions of nutrient yields and their downstream delivery. The increasing use of parameter estimation procedures to statistically estimate model coefficients, however, allows uncertainties in these predictions to be reliably estimated. Here, we use a robust bootstrapping procedure applied to the results of a previous application of the hybrid statistical/mechanistic watershed model SPARROW  to develop a statistically reliable method for identifying “high priority” areas for management, based on a probabilistic ranking of delivered nutrient yields from watersheds throughout a basin. The method is designed to be used by managers to prioritize watersheds where additional stream monitoring and evaluations of nutrient-reduction strategies could be undertaken. Our ranking procedure incorporates information on the confidence intervals of model predictions and the corresponding watershed rankings of the delivered nutrient yields. From this quantified uncertainty, we estimate the probability that individual watersheds are among a collection of watersheds that have the highest delivered nutrient yields. We illustrate the application of the procedure to 818 eight-digit Hydrologic Unit Code watersheds in the Mississippi/Atchafalaya River basin by identifying 150 watersheds having the highest delivered nutrient yields to the Gulf of Mexico. Highest delivered yields were from watersheds in the Central Mississippi, Ohio, and Lower Mississippi River basins. With 90% confidence, only a few watersheds can be reliably placed into the highest 150 category; however, many more watersheds can be removed from consideration as not belonging to the highest 150 category. Results from this ranking procedure provide robust information on watershed nutrient yields that can benefit management efforts to reduce nutrient loadings to downstream coastal waters, such as the Gulf of Mexico, or to local receiving streams and reservoirs. In addition to local effects, excessive transport of nutrients has been linked to eutrophication of downstream lakes, bays, and estuaries, and also linked to hypoxia in the Gulf of Mexico . Under rTo achieve the nutrient standards under consideration for rivers, streams, lakes, and reservoirs, and to meet the goal for a reduced size of the hypoxic zone in the Gulf of Mexico, nutrient loading from the upstream watersheds will need to be reduced. Although reducing nutrient loadings from all watersheds throughout the MARB would provide a comprehensive way to achieve these goals, it would not be the most efficient strategy because not all watersheds contribute equal quantities of nutrients to local streams and downstream coastal waters, including the Gulf of Mexico  from watersheds vary by up to several orders of magnitude among streams across the United States (U.S.) (2) throughout the MARB to the Gulf of Mexico, and used this information to compare the relative contributions from each state.A third strategy is to identify and rank the areas contributing the largest proportion of the total load and concentrate management action in these areas . TargetiWatersheds can be ranked on the basis of their respective loads, yields (by compensating for watershed size), incremental yields , or delivered incremental yields (by compensating for the loss of nutrients with transport from each watershed to the downstream receiving body)  delineated from 1-km digital elevation models , commercial fertilizer use on seven major crops, and animal manure that is recovered from confined animals on nearby farms and applied to crops as fertilizer. Nonrecovered manure pertains to nutrients derived from unconfined animals in pastures or losses from feedlots.The SPARROW model was used to simulate the mean annual flux of TN and TP from each reach of the ∼25,000 reaches in the MARB as a function of ten nutrient sources (eight for TP), six climatic and landscape factors that influence land-to-water delivery (five for TP), and two factors describing nutrient removal in streams and reservoirs . Although the model cannot be used to evaluate year-to-year changes, it also is not biased by the short-term meteorological variability that may occur throughout the study area in a given year. The specifics about the source and transport process specification, calibration, accuracy, and precision of the final models are presented in Model parameters for the sources, land-to-water delivery factors, and instream-decay terms were statistically estimated using weighted nonlinear least squares (WNLLS) regression, based on a calibration to the long-term mean annual loads (1975-1995) (2) are used as the units for evaluating uncertainties in the rankings of SPARROW predictions of the nutrient incremental yields that are delivered to the Gulf of Mexico. The incremental nutrient yield (and delivered yield) was computed for each HUC8 watershed by summing the loads (or delivered loads) from each reach within the HUC8 watershed, and then dividing by the total area of the HUC8 watershed. The HUC8-level aggregation of the ∼25,000 reach-level SPARROW predictions in the MARB provides the most reliable spatial scale for evaluating the model predictions, one that is generally consistent with the spatial scale of the monitoring stations used to estimate the model parameters.The 818 HUC8 watersheds in the MARB ; however, changes from 1992 to 1997 are used for animal manure nutrients and from 1990 to 2000 are used for population as described in The model predictions of the incremental yields and delivered incremental yields that are used in the HUC8 ranking procedures reflect the simulated nutrient yields in circa 2002. The simulated yields account for changes from 1992 to 2002 in agricultural sources  applied to each of the squared residuals at monitored reaches, resulting in 200 realizations of the estimated coefficients, yields, delivered yields, and residuals. The distribution of the estimated delivered yields for each reach from the 200 iterations was used to estimate the standard errors in the yields from SPARROW. The estimated confidence interval for delivered yields required explicit consideration of the distribution of residuals in the model, rather than just the summary statistical properties of the residuals. The bootstrap method for incorporating the distribution of the model residuals is based on the empirical distribution of the combined bootstrap-iteration estimate of the modeled component of delivered flux and a randomly selected weighted error from the original 425 monitored values obtained in the original calibration of the model. The 90% confidence interval for the delivered incremental yield from each HUC8 was then estimated using a ratio formulation of the hybrid bootstrap confidence limit . Delivered incremental TN yields from HUC8 watersheds range from 0 to 5,540 kg/km2/year  and deli  to 818 (lowest yield). All incremental yields and delivered incremental yields and their respective rankings are given in the Supporting Information material to this paper. The results of the ranking process are shown for delivered incremental yields for TN and TP in After sorting the HUC8 watersheds on the basis of their delivered incremental yields and summing the yields from the highest ranked watersheds to the lowest, an accumulated yield plot can be created . From thDifferences in the delivered incremental yields from many of the HUC8 watersheds are quite small, especially if the first few watersheds with the largest difference in yields  are not Having confidence limits in the yields and the rankings of the yields, rather than having specific values, makes it much more difficult to determine, which HUC8 watersheds to suggest to implement management efforts in, if only a selected number of contributors are to be considered. For example, if we want to identify which HUC8 watersheds are in the top 150 contributors to the Gulf of Mexico , this uncertainty should be considered. Given the wide 90% confidence intervals in the yields, many of the watersheds that originally ranked near 150, may or may not actually be in the top 150. One approach to incorporate this uncertainty is to use the information from the bootstrapping procedure, which was used to place confidence intervals in the original ranking . If a waThis information was then used to determine which watersheds could be considered to be in the top 150 for delivered incremental yields of TN and TP to the Gulf of Mexico if uncertainty is accounted for in the assessment . With 95Even with the relatively large confidence limits presented here, the results from SPARROW can be very useful in the ranking process for HUC8 watersheds in the MARB or other smaller areas. For example, of the 818 watersheds in the MARB, only 218 for TN and 245 for TP are considered to possibly be in the top 150, the remaining 444 (for TN) and 459 (for TP) watersheds statistically rank above 150 and, therefore, can be removed from special consideration (with 95% confidence), or 600 (for TN) and 573 (for TP) watersheds can be removed from special consideration with 75% confidence. Therefore, this information can be used to remove many areas from special consideration with respect to their importance of their contributions to loading to the Gulf of Mexico.The ranking discussed thus far has focused on the importance of export from specific HUC8 watersheds to the overall loading to the Gulf of Mexico. Actual implementation and prioritization of efforts, however, are often done at a smaller scale, such as in a specific state. An example of the ranking results for a given state, Tennessee, is shown in If water-quality managers are most interested in decreasing the number of locations that were identified as impaired , they woStatistical models require an extensive dataset of stream monitoring sites with information for both the dependent  and independent variables , whereas mechanistic models  can be used to simulate water-quality conditions with only very limited stream monitoring data. A more extensive dataset, however, is required to characterize the complex set of processes and state conditions in applications of mechanistic models to large river basins. Hybrid statistical/mechanistic models, like SPARROW, share many of the benefits of mechanistic simulation models but also allow confidence limits to be reliably estimated for the model’s predictions and watershed rankings. The confidence limits on the individual predictions from SPARROW are generally large but incorporate several sources of error, including error in the estimations of the model due to the reliance on a finite sample, and structural error in the prediction algorithms due to factors that are not accounted for in the model. Most mechanistic models, on the other hand, simulate specific processes, but each simulation usually only has one prediction for a given location and confidence limits can be difficult to estimate. Therefore, the specific rankings provided by many mechanistic models may provide a false sense of accuracy in the final ranking among watersheds. Statistical approaches have been developed that attempt to quantify the confidence limits in the predictions from mechanistic models. For example, parameter estimation methods, such as in the Parameter Estimation model (PEST) , can be The results presented in this paper were based on SPARROW models calibrated using 425 sites distributed throughout the U.S.  in the areas producing the highest nutrient yields. Various types of numerical models can be used to predict yields over large geographical areas, and it is a straightforward process to use the output from these models to rank or prioritize management efforts. However, all numerical models have estimation errors in their predictions; these uncertainties should be considered in efforts to prioritize watersheds for management. Whereas, it is very difficult to estimate the overall uncertainty in many mechanistic models; it is comparatively easier to estimate the overall uncertainties in the predictions from statistical models.In this paper, we used results from the hybrid statistical/mechanistic watershed model, SPARROW, to estimate delivered incremental yields from the 818 HUC8 watersheds in the MARB and then rank the watersheds based on their respective yields. Within the MARB, there was ∼2.5-3 orders of magnitude range in the incremental yields, with even a wider range in the delivered incremental yields, whereas, the 90% confidence in the predictions were only a factor of 2-3 of the predicted values. The highest TN and TP incremental yields and delivered incremental yields were generally in similar regions of the MARB: watersheds in the Central Mississippi, Ohio, and Lower Mississippi River basins. A robust statistical bootstrapping procedure was presented, which uses the uncertainties in the nutrient yield predictions (from SPARROW), to place confidence limits on the individual ranks. Information from the bootstrapping procedure was then used to estimate the probability that each HUC8 watershed is among a collection of watersheds that contributes the largest quantities of nutrients  to the Gulf of Mexico. Because of the wide confidence intervals in the predictions from SPARROW, only a few watersheds could be placed into the top 150 with statistical certainty; however, many more watersheds could be removed from consideration of being in the top 150. This ranking procedure was also demonstrated for use by a specific state to prioritize the targeting of watersheds for local improvements in streams and reservoirs.The information presented here for the MARB could assist managers in prioritizing management efforts based on improving nutrient conditions in local streams and improving nutrient conditions in downstream coastal waters, such as the Gulf of Mexico. This method of ranking is not a fully optimal approach to concurrent management of coastal and inland waters. For example, watersheds that rank near the lowest end of the spectrum in terms of nutrient delivery to the Gulf of Mexico, with management, can be expected to improve the water quality of local receiving waters. These local improvements may be of similar or greater value compared to the incremental improvements in coastal receiving waters.