Reader Variability in Mammography and Its Implications for Expected Utility over the Population of Readers and Cases

Robert F. Wagner, PhD
Craig A. Beam, PhD
Sergey V. Beiden, PhD

The traditional approach to the assessment of a diagnostic modality has been to study the dependence of the true-positive fraction (TPF) on the false-positive fraction (FPF), that is, the receiver operating characteristic (ROC) curve. In the last decade this approach has been extended to account for the variability of reader expertise using an approach called the multiple-reader, multiple-case (MRMC) ROC paradigm in which many readers read many patient cases. The authors used their MRMC ROC algorithm to analyze data from a large random sample from the population of U.S. mammographers, each reading the same random sample of cases from a large screening mammography program—by far the largest data set to date analyzed this way. The analysis quantifies the great range of observed reader skill in that dataset, and the remarkably small error bars demonstrate that the conclusions generalize to the populations sampled there with little uncertainty from the finite sample size. The results were used to study trends in the expected utility or benefit of mammography as a function of the setting (screening or diagnostic work-up), relative utilty of a true-positive to a false-positive decision, and level of reader expertise. Several hypotheses regarding the variable effectiveness of screening mammography and the added value of computer reading aids were formulated for future attention.

The Application of the Heuristic-Systematic Processing Model to Treatment Decision Making about Prostate Cancer

Suzanne K. Steginga, PhD
Stefano Occhipinti, PhD

Current approaches to patient decision making focus on people’s use of systematic decision strategies, such as weighing the pros and cons of different options. However, when making health decisions, people often decide using nonsystematic strategies, such as past memories and beliefs about health. The present study investigated the interrelationship between both types of decision strategies. We assessed the decision strategies and individual characteristics of 111 men recently diagnosed with prostate cancer. Most men preferred that the decision be shared equally between them and their doctor. Men who were uncertain about their decision and who had a positive view of their medical care were more likely to defer their decision to their doctor. Those men who believed maintaining good health was due to their own actions preferred more involvement in their treatment decision. Men tended to think more about the pros or cons of treatment in the presence of a negative memory about cancer and when they were uncertain about the likelihood of cure and side effects. Men who deferred the decision to their doctor tended to think less about the pros or cons of treatment. This approach to decision research provides a way of understanding patient decision making that takes into account intuitive decision strategies.

A Cost-Utility Analysis of Neonatal Circumcision

Robert S. Van Howe, MD, MS, FAAP

When boys who have been circumcised at birth are compared to those not circumcised using a cost-utility analysis, neonatal circumcision and its consequences over a lifetime cost $828.42 more than not circumcising. Neonatal circumcision and its consequences are also associated with more morbidity than not circumcising. When neonatal circumcision was assumed to be cost-free, pain-free, and had no immediate complications, it was still more costly than not circumcising. It was impossible to arrange a scenario of model assumptions that made neonatal circumcision cost-effective. Neonatal circumcision is not good health policy, and support for routine neonatal circumcision, as a medical procedure, cannot be justified financially or medically.

Decision Making in a Multidisciplinary Cancer Team: Does Team Discussion Result in Better Quality Decisions?

Frank Kee, FRCP (Edin)
Tracy Owen, MFPH
Ruth Leathem, RN

It is often assumed that care provided by a multi­disciplinary team will result in improved outcomes for cancer patients. However, there is no evidence that teams make better quality treatment decisions. This paper describes a real-time study of decision making by a regional lung cancer team, comparing the treatment decisions made by individual doctors before and after case discussion by the team. The findings suggest that team discussion did not improve the quality of decision making overall and had limited influence on changing the opinions of individual doctors. The study highlights the need for further research in this area.

An Exploration of Relative Health Stock in Advanced Cancer Patients

Darrell J. Gaskin, PhD
Kevin P. Weinfurt, PhD
Liana D. Castel, MSPH
Venita DePuy, MS
Yun Li, MS
Andrew Balshem, BA
Al Benson, MD, FACP
Caroline B. Burnett, ScD, RN
Sandra Corbett, MS
John Marshall, MD
Elyse Slater, MS
Daniel P. Sulmasy, MD
David Van Echo, MD
Neal J. Meropol, MD
Kevin A. Schulman, MD

We tested whether advanced cancer patients’ relative health stock (RHS) influenced their treatment decisions. RHS refers to the effect on patients’ perceived quality-adjusted life expectancy of learning that they have a life-threatening illness. It measures patients’ sense of loss in health due to their illness. We surveyed 328 advanced cancer patients who were given the opportunity to participate in a phase I clinical trial or receive standard chemotherapy. Controlling for other factors, we found that patients who had lower RHS, i.e., a greater sense of loss in their health, were more likely to participate in a phase I trial. The results have implications for decisions about whether risky treatments, including experimental treatments, are in patients’ best interest. Such decisions made by insurers and drug regulators, among others, may be too restrictive. If we assume that the seriously ill patients are rational (and we have no reason to think they are not), then risky treatment may be more valuable for patients with lower RHS than to healthier people.

Reorganizing the System of Care Surrounding Laparoscopic Surgery: A Cost-Effectiveness Analysis Using Discrete-Event Simulation

James E. Stahl, MD, CM, MPH
David Rattner, MD
Richard Wiklund, MD
Jessica Lester, MM
Molly Beinfeld, MPH
G. Scott Gazelle, MD, MPH, PhD

Surgery departments are often asked to balance patient volume with safety. Currently, individual anesthesiologists are responsible for patient care from the start of anesthesia through recovery. Unfortunately, this means the operating room and surgeon occasionally stand idle while waiting for the anesthesiologist to become free to start the next case. Treating more patients then means either more anesthesiologists or less time with each patient. Neither is an attractive strategy by itself. In this study, we examined a proposed radical redesign of the anesthesia care process. In this new system, anesthesia care is split into 2 parts with 2 anesthesiologists working in tandem. The first anesthesiologist is responsible for the start of anesthesia. The second is responsible for surgery and recovery. With both working together when care is transferred. We used discrete-event simulation to estimate the outcomes of this process change. Discrete-event simulation is a method that allows us to explicitly model all the steps, features, and risks of a process. In the proposed system more patients could be treated per day with an average cost per patient similar to traditional anesthesia care. Regarding safety, the new process would have to have an operative risk more than 4 times that of the traditional anesthesia care for it no longer to be cost-effective. The proposed system also remains cost-effective over a wide range of transfer times and scheduled patient volumes.

Pramipexole v. Levodopa as Initial Treatment for Parkinson’s Disease: A Randomized Clinical-Economic Trial

Katia Noyes, PhD, MPH
Andrew W. Dick, PhD
Robert G. Holloway, MD, MPH
and the Parkinson Study Group

We conducted a 2-year analysis of the impact of a new drug, pramipexole, on quality of life and healthcare expenses of early Parkinson’s patients using data from a randomized clinical trial. We demonstrated that over the 2-year period, pramipexole is likely to be not cost-effective compared to the standard treatment, levodopa, when used as a first-line therapy. Using pramipexole resulted in the gain of about 7 additional quality adjusted life days (as measured via generic quality of life questionnaire EQ-5D) at the cost of $2,138, or about $107,000 per one quality adjusted life year (QALY). Using the recommended $50,000 as a societal willingness to pay for one QALY, there is a 30% chance that pramipexole is cost-effective compared to levodopa. It is important to notice that for treatment of chronic illness like Parkinson’s disease, a 2-year time horizon is too short to make conclusions about therapeutic and economic potential of an intervention. Long-term study is required to assess the cost-effectiveness of pramipexole compared to levodopa in general practice.

The Cost-Effectiveness of Screening Programs Using Single and Multiple Birth Cohort Simulations: A Comparison Using a Model of Cervical Cancer

Sarah Dewilde, MSc
Rob Anderson, PhD

Many model-based economic evaluations of cervical screening have been based on simulations of single birth cohorts of women, in which a single group of identical women are followed through time. These are unrepresentative of the population that will ultimately be affected by the policy choices being modeled. We compared the cost-effectiveness results from a single birth cohort simulation (i.e., the standard approach) with results from a more realistic, multiple birth cohort simulation that more realistically represents population dynamics. We used an established model for evaluating the cost-effectiveness of cervical screening programs. The choice of hypothetical cohort that starts the simulation had a major impact on the cost-effectiveness estimates: compared with a single birth cohort simulation the incremental cost-effectiveness of a shift from biennial to triennial screening was 30% higher when using the multiple cohort simulation. Future modeling-based evaluations of screening policies should reflect the age-range of the population that is targeted, by carefully specifying and justifying the nature of the starting cohort(s).

A Single Mathematical Model Predicts Physicians’ Recommendations and Postmenopausal Women’s Decisions to Participate in a Clinical Trial to Prevent Breast Cancer or Coronary Heart Disease

Clairice T. Veit, PhD

Few eligible postmenopausal women participate in clinical-trial research to prevent breast cancer (BC) or coronary heart disease (CHD), making it difficult to interpret research results for this vulnerable population. To understand how participation might be increased, white and minority postmenopausal women completed questionnaires where they judged the likelihood of participating in prevention clinical trial scenarios with varied characteristics, including the strength of their doctor’s recommendation for or against participation. In addition, 293 white and minority physicians judged what the strength of their participation recommendation to their eligible patients would be in different scenarios. Predictions from mathematical models of the judgments indicate that high expected toxicity and even moderate out-of-pocket expenses for physicians or patients act as severe barriers. White and Hispanic postmenopausal women under 70 years of age are the groups most likely to participate, even under adverse conditions such as high costs, moderate toxicity levels, or a recommendation against participation by their doctor, if they believe they are at high risk for contracting the disease. These groups are followed by white women 70 to 80 years of age and African American women under 70 years of age. African American women 70 to 80 years of age have the lowest predictions. Although perceived risk was the most important factor for women, only 8% of women reported that their doctor had ever told them their risk for developing BC and only 15% their risk for developing CHD; 75% and 48% rated their risk of contracting BC and CHD, respectively, as low to none. Further, the percent of women who reported they had never heard from any source about a prevention clinical trial being conducted was 76% for BC trials and 88% for CHD trials. These results have implications for education, information dissemination, and prevention clinical trial planners.

Refining Estimates of Major Depression Incidence and Episode Duration in Canada Using a Monte Carlo Markov Model

Scott B. Patten, MD, FRCPC, PhD
Robert C. Lee, MSc

Depression is now recognized as being one of the most important medical conditions. There are a variety of options for dealing with depression in the population. These include prevention, early detection, and treatment. However, in order to understand whether such strategies are working, it is necessary to measure the rate at which new cases develop in the population and how long these episodes last. Effective prevention reduces the frequency of new episodes whereas early detection and treatment should reduce the duration of episodes. Mathematical models that can describe such factors in the population are potentially very useful for priority setting, planning, and accountability. If they are to be useful, however, it is important that such models can make use of readily available data. In this study, we obtained depression data from 2 large Canadian health surveys and used it to develop a mathematical model describing the rate at which new cases of depression develop and how long these episodes last in the general population. Using such models, it will be feasible to more accurately monitor this aspect of population mental health over time using available sources of data. This provides a way of monitoring progress, or lack of progress, made by the health system. The methods employed in this project may prove useful for a variety of other episodic conditions.

Decision Aids for Benign Prostatic Hyperplasia: Applicability across Race and Education

David R. Rovner, MD
Celia E. Wills, PhD, RN
Vence Bonham, JD
Gilbert Williams, PhD
Janet Lillie, PhD
Karen Kelly-Blake, BS
Mark V. Williams, MD
Margaret Holmes-Rovner, PhD

Printed or video materials (decision aids) designed to help patients decide about treatment options have not been widely tested in diverse groups. The present study evaluated a video decision aid in a diverse group of 188 men. Approximately one forth were black and college educated, one forth black and not college educated, one forth white and college educated, and one forth white and not college educated. The literacy level of all subjects was measured.

The video discussed treatments for an enlarged prostate with difficulty passing urine. The information was presented by spoken voice, graphs, charts, and animations. The results evaluated were the increase in knowledge and change in readiness to make a decision about which therapy to choose as a result of the video.

Contrary to expectations, the results show no difference by race or college education in either outcome. The video appeared to produce the same increase in knowledge and readiness to make the decision about which therapy to choose, in subjects of both races and education level. These results suggest that a single carefully designed decision aid may be effective for people of different races and literacy levels without making separate ones for each group. Further research is needed to corroborate the findings in different disease states.

Influence of Clinical and Economical Factors on the Expert Rating of Appropriateness of Preoperative Use of Recombinant Erythropoietin in Elective Orthopedic Surgery Patients

Patrick Taffé, PhD
Bernard Burnand, MD
Vincent Wietlisbach, BA
John-Paul Vader, MD

Economic pressures, the need to rationalize procedures and concerns about ensuring the appropriateness of medical treatment has led to the development of clinical practice guidelines.

The RAND appropriateness method, which combines a thorough literature review and expert knowledge, is one of the most established, respected, and studied methods of developing criteria to define appropriate medical care. This method allows, for a particular medical procedure, the development of a long list of plausible clinical scenarios and the assessment of appropriate care for those scenarios. Its common output is the categorization of the procedure as “appropriate” or “inappropriate” for each of the clinical scenarios

We enhance this method by examining the relative weight of various factors, which enter into the expert assessment of appropriateness of care, using statistical regression methods. This approach is illustrated with data from an expert panel examining the appropriateness of erythopoetin (medication used to increase red blood cells and reduce the need for transfusions) in preparation for orthopedic surgery.

Parametric Mean Survival Time Analysis in Gastric Cancer Patients

Shunzo Maetani, MD, PhD
Toshifusa Nakajima, MD, PhD
Toshikuni Nishikawa, MD

The mean survival time (MS) is an ideal outcome measure to assess the efficacy and cost-effectiveness of treatments for cancer and other diseases. It is more comprehensible to patients and clinicians than conventional outcome measures such as 5-year survival and hazard ratio. In order to obtain MS, however, life-long follow-up is needed. To overcome this difficulty we developed a new method by which MS can be predicted from 5-year follow-up data. Using these data a disease-related survival curve (in which only death from the disease is counted as fatal outcome) was obtained and extrapolated beyond 5 years based on a mathematical model. This curve was then combined with survival information obtained from the life tables of age- and sex-matched contemporaries to give MS. The predictability of MS was evaluated using life-long follow-up data from 3597 gastric cancer patients who were divided into 50 prognostically different groups. The results showed that using this method the MS was predictable with reasonable accuracy, the average prediction error being less than 5 % for these groups with slight overestimation of MS. This method may be applicable to clinical decision making, technology assessment, survival analysis, and other related fields.

Cost-Effectiveness of Sibutramine in the Treatment of Obesity

Emma Warren
Alan Brennan
Ron Akehurst

Obesity is a serious health problem in Western society. Weight loss can significantly reduce the incidence of obesity related morbidity. This study reports the cost-effectiveness of one year’s sibutramine treatment alongside diet and lifestyle advice compared to diet and lifestyle advice alone.

Evidence from clinical trials and other research is combined to estimate the extra costs and health benefits associated with sibutramine-induced weight loss, including reduced incidence of coronary heart disease (CHD) and diabetes, and improved general health related quality of life. In 1000 patients, sibutramine provides an estimated 59 extra quality-adjusted life-years (QALYs) at a cost of 4,780 per QALY ($9,299).

US and UK analyses suggest that sibutramine is a cost-effective treatment for obesity and should be considered for use alongside diet and exercise advice when treating the obese.

Estimating the Prognosis of Hepatitis C Patients Infected by Transfusion in Canada between 1986 and 1990

Murray Krahn, MD, MSc
John B. Wong, MD
Jenny Heathcote, MD
Linda Scully, MD
Leonard Seeff, MD

It has been known for decades that transfusion is associated with hepatitis, though the agent responsible was only identified in 1990. Blood banks around the world began using tests for blood abnormalities prior to 1990, although Canada only began to test when the hepatitis C test was available. Federal, Provincial, and Territorial Governments of Canada allocated $1.1 billion to individuals who acquired hepatitis C through the blood supply after 1986, when testing could have begun. However, the degree of compensation was linked to the development of long-term complications of hepatitis C. Individuals who developed more advanced disease received more compensation. The natural history of hepatitis C, particularly in the very long term, was not well understood. This paper describes the application of Markov modeling techniques to develop a prognostic model for hepatitis C to aid in the disbursement of the compensation settlement. The model suggests that the overall death rate in individuals who receive transfusion is high (46% at 10 years), although HCV-related deaths are rare. Only 13% of individuals will develop cirrhosis (liver scarring) at 20 years. One in 4 will develop cirrhosis during their lifetime, and 1 in 8 will die of liver disease.

Cost-Effectiveness of Interventions to Reduce Vertical HIV Transmission from Pregnant Women Who Have Not Received Prenatal Care

Joseph M. Mrus, MD, MSc
Joel Tsevat, MD, MPH

For women presenting in labor without prenatal care, rapid HIV testing and treatment is recommended; however, the optimal drug regimen to use for those with a positive test is controversial, and the feasibility of testing at sites where few women present without prenatal care and/or the prevalence of HIV is low is unclear. Using simulation modeling, our analysis found that for women presenting in labor without prenatal care, the strategy of offering rapid HIV testing and administering HIV drug therapy to women testing positive and the strategy of empirically treating for HIV (not testing first) both prevent cases of HIV and are cost saving compared with no intervention. Given those findings, empiric treatment may be a reasonable strategy at sites where rapid HIV testing is not possible or practical. The analysis also found that if more expensive drug regimens were used, the more expensive strategies would need to prevent mother-to-child HIV transmission only slightly more effectively to make the more expensive strategies relatively cost-effective in comparison with the less expensive strategies, implying that 2 or 3 drug combination regimens should be used.

What Is the Least Costly Strategy to Evaluate Cervical Abnormalities in Rural Women? Comparing Telemedicine, Local Practitioners, and Expert Physicians

David M. Bishai, MD, MPH, PhD,
DaronG. Ferris, MD,
Mark S. Litaker, PhD

An abnormal Pap smear typically requires evaluation using colposcopy where a specialist views the cervix and decides whether to biopsy. For women in rural America, a trip to see a colposcopy specialist at a university can involve costly travel and inconvenience. Some women never take the trouble to make these trips. One alternative is to train local nurse practitioners and general practitioners to perform colposcopy—the danger is that their lack of expertise will lead them to perform too many biopsies. Another alternative is to link rural practitioners through telemedicine to university based experts who can advise them whether or not to biopsy. We found that the lowest cost option was to simply allow local practitioners to perform colposcopy even if it means that some unnecessary biopsies will occur. Telemedicine would have to cost $73 less per patient in order for it to save money by limiting unnecessary biopsies.

Men with Prostate Cancer: Making Decisions about Complementary/Alternative Medicine

Heather Boon, BScPhm, PhD,
Judith Belle Brown, PhD,
Alan Gavin, MSW,
Kathleen Westlake, MSc

Prostate cancer is the most common cancer of North American men. Currently men diagnosed with prostate cancer have access to a variety of conventional treatment options (e.g., surgery). In addition, some men consider using complementary and alternative medicine (CAM). The purpose of our study was to gain a better understanding of men’s perceptions and experiences regarding making decisions to use (or not use) CAM. We spoke with 29 men diagnosed with prostate cancer in five focus groups. We found that decision factors, both fixed (e.g., disease characteristics, demographic characteristics such as age and medical history) and flexible (e.g., perceptions of CAM and conventional medicine, experiences with the health care system and health care practitioners, and perceptions about the need for control (e.g., over the treatment decision), play a vital role in the decision making process. The men in our study were more likely to be “pushed” toward seeking CAM treatment options because of fear of, or actual, negative experiences with the conventional health care system, than to be “pulled” toward CAM by perceptions about its safety. This understanding will help clinicians facilitate patients’ decision making in a manner that empowers them to make choices that ultimately enhance their health and well-being.

Use of Regression Modeling to Simulate Patient-Specific Decision Analysis for Patients with Nonvalvular Atrial Fibrillation

Joseph A. Johnston, MD, MSc,
Mark H. Eckman, MD, MS

Patient-specific decision analysis involves the use of decision support tools to determine the optimal treatment decision for an individual patient. Such an approach allows for incorporation of patient-specific risks and preferences for various disease outcomes and treatment complications. Unfortunately, this process can be time intensive and typically requires the use of special computer software, making it impractical for the average busy clinician. Using a technique called regression modeling, the authors developed a Web-based decision tool that can estimate the benefit in life expectancy that an individual patient with nonvalvular atrial fibrillation (a common heart condition that carries a high risk of stroke) can expect to gain by using warfarin, a blood thinner, for stroke prevention. Their results indicate that such a tool can generate estimates of life expectancy that approximate with a high degree of accuracy those produced using a complex decision model. Such Web-based tools have the potential to improve patient care by making efficient, reliable decision support available at the point of care.

Prediction of Community-Acquired Pneumonia Using Artificial Neural Networks

Paul S. Heckerling, MD,
Ben S. Gerber, MD,
Thomas G. Tape, MD,
Robert S. Wigton, MD

Artificial neural networks (ANNs) are computer programs, based on the neural architecture of the brain. ANNs match patterns in data by learning from examples. The authors used ANNs to identify cases of pneumonia, based on demographic, symptom, sign, and comorbidity data, from among patients presenting to emergency departments with acute respiratory complaints. They found that ANNs, especially those trained using resampling of pneumonia cases, accurately identified patients with and without pneumonia. A high degree of accuracy was seen both among patients on which the ANNs were trained (ROC area = 0.895 on a scale from no identification = 0.5 to perfect identification = 1.0) and among new patients at a different site (ROC area = 0.872). In some cases, ANNs identified patients with and without pneumonia more accurately than did logistic regression, a standard statistical classifier method. These results show that ANNs, trained to recognize pneumonia cases, may assist clinicians in diagnosing pneumonia among patients presenting with acute respiratory complaints.

Visual Acuity following Cataract Surgeries in Relation to Preoperative Appropriateness Ratings

Joanne K. Tobacman, MD,
Bridget Zimmerman, PhD,
Paul Lee, JD, MD,
Lee Hilborne, MD, MPH,
Hansjoerg Kolder, MD, PhD,
Robert H. Brook, MD, ScD

The authors compared preoperative ratings of appropriateness, using the methods developed by investigators from RAND-UCLA, and postoperative visual acuities for 768 patients who had cataract surgery performed at academic medical centers. They found that improvement in visual acuity was more likely to occur for surgeries that were rated appropriate or appropriate and crucial than for surgeries rated uncertain or inappropriate. No change or decline in visual acuity was associated with surgeries that had a rating of either inappropriate or uncertain. These findings support the validity of the application of appropriateness methodology to cataract surgery.

The Relationship between Treatment Objectives and Practice Patterns in the Management of Urinary Tract Infections

Stephen D. Flach, MD, PhD,
J. Craig Longenecker, MD, MPH, PhD,
Thomas G. Tape, MD, Teresa J. Bryan, MD,
Connie Parenti, MD, Robert S. Wigton, MD

Because physicians are responsible for allocating the majority of health care resources, why they select the treatments they do is an important, but not well-studied, question. The authors analyzed a survey of primary care physicians about their treatment of urinary tract infection to explore the relationship between their treatment choices and their stated treatment goals. The most commonly cited goal was patient convenience. Making an accurate diagnosis was next, and avoiding unnecessary antibiotics was third. Physicians who stressed convenience or minimizing patient expenses were less likely to use diagnostic tests and more likely to use telephone treatment. Physicians who stressed making an accurate diagnosis or avoiding unnecessary antibiotics were more likely to use the same resources and less likely to use telephone treatment. Understanding physicians’ goals in selecting among available treatments may help us understand variations in practice patterns and find ways to improve physicians’ treatment choices.

Long-Term HIV/AIDS Survival Estimation in the Highly Active Antiretroviral Therapy Era

Joseph T. King Jr., MD, MSCE,
Amy C. Justice, MD, PhD,
Mark S. Roberts, MD, MPP,
Chung-Chou H. Chang, PhD,
Jennifer S. Fusco, BS

Since 1996, when highly active antiretroviral therapy (HAART) became widely available, short-term mortality for patients with HIV/AIDS has improved. Patients, providers, and policy makers all question how much HAART will prolong long-term survival and whether its benefit depends on the timing of treatment. The authors constructed a computer simulation model to estimate long-term HIV/AIDS survival and its sensitivity to timing of HAART. The model incorporated CD4 cell counts, HAART treatment failure, and death from both HIV/AIDS and non-HIV/AIDS causes using data from a large observational cohort of people with HIV/AIDS receiving HAART and national data on age-associated risk of mortality. Results suggest that an individual gains approximately 4 to 6 additional years of life beyond that observed in pre-HAART cohorts, regardless of the CD4 cell count when starting HAART.

A Comparison of Bayesian and Maximum Likelihood Methods to Determine the Performance of a Point of Care Test for Helicobacter pylori in the Office Setting

Brendan C. Delaney, MD, FRCP, MRCGP,
Roger L. Holder, BSc, Teresa F. Allan, BA, MSc,
Joyce E. Kenkre, MSc,
F. D. Richard Hobbs, FRCP, FRCGP

Accurate knowledge of the performance of a diagnostic test is essential if the results of the test are to be reliably interpreted by clinicians. A test is best evaluated against a reliable reference standard in the setting in which it will be used. Unfortunately, many reference standards are not in themselves reliable, and this introduces significant error into test evaluations. Latent class analysis can be used to provide a theoretical link between the existing data and the unknown “true parameters” for test performance. Tools for applying this solution include maximum likelihood estimation and Bayesian methods. This article develops a Bayesian model for finding the true performance of a diagnostic test compared against 2 other tests that are not good reference standards. The model was developed using a Markov Chain Monte Carlo method and the software WinBugs (MRC, Cambridge, UK). The result was comparable with the results obtained by a (non-Bayesian) maximum likelihood approach. The advantage of the Bayesian method is that it allows for the incorporation of existing data on test performance, using informative priors, and it simultaneously estimates the 95% credible intervals, equivalent to confidence intervals, which the maximum likelihood method does not.

The Reliability of Assessing the Appropriateness of Requested Diagnostic Tests

Rianne Bindels, MSc, Arie Hasman, PhD,
Jan W. J. van Wersch, PhD, MD, Peter Pop, PhD, MD, Ron A. G. Winkens, PhD, MD

Despite a relatively poor reliability, peer assessment is the traditional method to assess the appropriateness of health care activities. In the Maastricht region of the Netherlands, individual feedback is given on the appropriateness of the test ordering behavior of general practitioners. The feedback is based on accepted practice guidelines and is given by expert physicians. The authors were interested in the variability of the provided feedback. Three experts were asked to assess the appropriateness of test ordering. Using their assessments, the authors could determine how much of the observed variability is due to differences in the judgments of these experts. This is expressed in the reliability coefficient. The reason the authors were interested in the reliability is the fact that they planned to develop an automated feedback system. To evaluate the performance of such a system, a gold standard is needed with a high reliability. Using the reliability coefficient, the authors are able to determine how many reviewers are needed to create a reliable gold standard. In addition, the authors verified if feedback on test ordering based by experts is suitable for case-by-case assessments of appropriateness or only for a general assessment over a series of cases. Reporting on the value of the reliability allows other authors to estimate the number of experts needed to create a reliable reference standard for their studies.