|
||
Entire Text
| SUMMARY The Achievable Benchmark of Care (ABC™) system is continually being developed at the University of Alabama at Birmingham (UAB) under an initiative of the Agency for Healthcare Research and Quality (AHRQ). This manual has been written to help practitioners, administrators, payers, and anyone else charged with improving the quality of health care, to use the ABC system. The ABC method is a clinical measure: not a comparison of financial performance. It is not based on arbitrary targets and does not advocate "cook book" medicine. Our ABC system is a tool to facilitate the measurement, comparison and dissemination of benchmarks derived from the process of care practices already being achieved by "best-in-class" providers. It is not a direct outcomes measure. However, because ABCs are typically used to measure the extent of usage of processes of care widely accepted to improve outcomes, it follows that providers achieving high ABC benchmark levels should have better outcomes than providers who do not. The ABC system is statistically sound and simple to use. It's approach is objective, easily updated and readily yields understandable feedback comparisons. ABCs are designed to be used by providers, insurers or government agencies in a wide range of clinical settings from hospitals, physician offices, nursing homes, public health clinics to managed care and regulatory organizations. We hope this manual will encourage many more providers in the United States and internationally to use the ABC method and to share their results as we continuously refine it and validate its value. TABLE OF CONTENTS SUMMARY Page 2 TABLE OF CONTENTS Page 3 EVOLUTION OF THE UAB ABC™ METHOD Page 4 SECTION 1. INTRODUCTION Page 6 SECTION 2. CONCEPTS BEHIND THE ABC™ METHOD Page 10 SECTION 3. IMPLEMENTATION Page 18 SECTION 4. METHODOLOGY AND COMPUTATION Page 23 SECTION 5. INFLUENCING FUTURE CARE Page 31 SELECTED BIBLIOGRAPHY Page 33 APPENDIX I Example of calculation APPENDIX II Illustration of the effect of the Bayesian Estimator APPENDIX III Example of a feedback report. EVOLUTION OF THE UAB ABC™ METHOD In a broad sense, the HCQIP uses principles of continuous quality improvement (CQI) to transform practice patterns. This model involves giving providers feedback about their own practice in a non-threatening manner. Use of clinical guidelines often forms the core of the process of quality improvement. Physicians and health care organizations examine data describing their patterns of care (practice profiles) in comparison with those of their peers and, from these, develop plans for improvement. This new approach raises the average by improving the performance of all providers, as opposed to the older approach of identifying only the worst performers. Whereas, such targeting of statistical outliers usually affected at most 2.5% of providers (i.e., minus two standard deviations from the mean), improving performance of all providers should benefit more patients. Feedback reports typically include statistically derived measures of the performance of individual providers and comparisons with the mean performance of the entire group. Sometimes the feedback includes a benchmark. The benchmark corresponds to the performance of the "best-in-class". The definition of the benchmark or "best-in-class" has typically been a qualitative judgment and not a data-driven measure of performance. In this manual, we describe a developing methodology to identify peer group based, objective, reproducible, data-driven performance measures that we call the University of Alabama at Birmingham's Achievable Benchmarks of Care (ABCs). SECTION 1 INTRODUCTION PURPOSE OF THIS MANUAL This manual has been written to describe the UAB ABC system to the wide range of persons who are charged with improving quality of health care. In addition, we have a purpose other than simply to introduce a new measurement tool. Research on the ABC methodology is continuing at UAB. It is hoped that the manual will stimulate additional use and the interchange of experiences and suggestions for the methods' improvement and wider clinical application. We are cognizant that many questions remain to be answered, and we encourage practitioners and academics to join us in enhancing the ABC methodology.
WHAT IS THE UAB ABC™ SYSTEM? Benchmark performance is measured by the proportion of patients for whom certain clinical processes of care are prescribed or recommended. These processes of care are considered to be indicators (a term used frequently in the ABC method) and their usage indicates differing degrees of excellent care giving. An indicator might, for example, be a recommendation that all post-MI patients take aspirin therapy. The indicator measure for doctor A or hospital Y is the proportion of clinically appropriate patients to whom this recommendation is actually made. In its benchmark calculation, the ABC system ranks comparable providers and computes statistics that can be used as feedback to individual providers to measure their progress towards health care excellence in relation to that of their "best in class" peers. WHO CAN BENEFIT The ultimate beneficiary of course is the patient. The ABC method can be used by providers, insurers or government agencies in a wide range of settings, including hospitals, physician practices, nursing homes, public health clinics, or managed care organizations. ABCs can be applied to groups of community providers, to institutions or departments within them, or to individual practitioners. We firmly believe that if ABCs are used to improve the rates of usage of processes of care known to improve outcomes, a higher proportion of patients and their families will benefit by receiving the "best" care. This, in turn, will benefit those responsible for paying for the nation's health care by leading to a healthier population and ultimately reducing the cost of care. WHAT IS NEEDED IN ORDER TO USE THE ABC™ SYSTEM? WHAT TYPE OF DATA ARE NEEDED? Data to calculate usage rates of selected processes of care must include information that provides both the denominator - the numbers of patients for whom the intervention is appropriate, and the numerator - the number of patients actually receiving the intervention. Data can be obtained from: · the internal information systems of provider organizations such as hospitals, clinics, nursing homes, home healthcare agencies, or laboratories. Data could be in medical records, financial, administrative or clinical information systems already in place to provide daily internal management, billing and decision support information. · managed care organizations' administrative and claims records, or, · public data bases, e.g., the National Health Interview Survey, Medicare claims reports. Lists of processes and outcome indicators can be obtained from AHRQ's CONQUEST database. References to literature to support these are also available. More information can be obtained at www.ahrq.gov. HAVE ABCs BEEN TESTED? Yes. With support from CMS, AHRQ and AQAF, the ABC method has been developed and tested under everyday, clinical conditions. It has been successfully applied in a variety of situations, e.g., management of diabetes mellitus, treatment of post MI patients, and utilization of mammography and pap smear screening. (See Bibliography) DOES THE ABC™ METHOD HAVE ADVANTAGES OVER OTHER METHODS? SECTION 2 · a desire by providers to improve the care they give, These requirements to improve have often taken the form of generalized directives such as ". . improve quality and outcomes of care". It follows that providers and controlling agencies needing more guidance as to expectations will ask " . . .improve in relation to what?" From this it is a short step for them to ask "What are realistic, achievable goals?" "Who is achieving or exceeding target quality now?" "How are they doing it?" "What are the benchmarks of quality, and how are realistic improvement mile posts along the way to improvement set?" "How do we set and measure these in an objective way that will lead to undisputed, consistent, comparisons?" This manual attempts to address all of these issues. The judgment of health care quality and cost-effectiveness has three elements: Each of these multi-faceted elements must be capable of being described by independent, objective, readily comprehensible, data-driven measurements of levels of excellence already being achieved by the best performers. UAB's ABC system does exactly that for processes of care--but it does not attempt to measure effectiveness of care. PURPOSE OF ABCs The fundamental belief of the TQM and CQI movements is that the ongoing process of care measurement and analysis, combined with feedback and process modification, will ultimately lead to improved overall quality. As actions are taken to continuously improve processes, there needs to be a sufficiently sensitive, reliable and valid measurement system to monitor performance and provide informative feedback. The ABC method described in this manual is being developed to achieve this goal. PROCESS INDICATORS AND BENCHMARKS Many recent quality improvement efforts have focused on clinical performance measures consisting of either process of care or outcome indicators or both. Process of care indicators (also called process indicators) are measures of a provider's use of interventions (e.g., diagnostic test ordered, surgical procedure performed, drug prescribed, etc.). Outcome indicators are measures of the patient's health status (e.g., morbidity and mortality, functional capacity, quality of life). In 1996, the Agency for Healthcare Research and Quality (AHRQ), made the CONQUEST database available on their home page (www.ahrq.gov). CONQUEST summarizes information on 1197 clinical performance indicators containing 53 different sets of measures developed by public and private sector organizations such as, AHRQ, CMS, the National Center for Quality Assurance (NCQA), and the RAND Corporation. BENCHMARKS AND ABC™ INDICATORS 1. Many earlier approaches to quality improvement were based only on identifying and eliminating 'poor performers'. ABC benchmarks identify superior performance and encourage others to emulate the practices by which this is achieved. An important tenet of the ABC method is the provision of data-driven, peer-group performance feedback to encourage the improvement of all providers within that group from the best to the worst. Even providers who are already "best in class", and those who are very close to optimal performance, will benefit. 2. The central premise of the ABC system is that its benchmark targets are not 'theoretical', or set by expert panels. They are standards of care already practiced by the top performers. ABC benchmarks are based on actual performance of "best-in-class" providers already using processes of care widely accepted to positively affect outcomes and hence contribute to health quality improvement. The ABC approach is continuous and dynamic: we feel it should become a part of a provider's ongoing philosophy of quality improvement. Repeat measurements are made over time and, as all providers--including even those who are "best-in-class" today--improve, benchmarks will rise. It is possible at each successive stage, that different providers will become "best-in-class" engendering a race to the top. Improvement of health care quality is ultimately achieved by the actions of individual providers. In order to encourage striving towards the high level practices which the best performers already achieve, the ABC system gives periodic data to each provider that indicate a) the performance of the individual provider -- which can be an individual caregiver, a hospital, a department or a clinic - in the cohort group being measured, b) the average performance of the cohort, and, c) the ABC performance of the best providers. ABCs are a measurement tool to aid the spread of the "best practices" of a few providers until these become "average care" by the majority. SOUND BENCHMARKS ABCs AND OUTCOMES. It is important to recognize at the outset that the ABC method does not explicitly measure outcomes--that is not its goal. ABCs are targets for process measurement: they are not intended for outcomes measurement. They are measures of the usage of "best practices of care" by individual providers or groups of providers. Because these "best practices" are widely acknowledged to improve outcomes, their wider use is expected to result in improvements in the quality and effectiveness of care delivery. As overall care improves, benchmarks will rise. Although the ABC methodology could be used to derive target levels for outcomes measurement, we advise that this should be done only with extreme caution, if at all. The reason is that valid comparisons of outcomes across providers presuppose either similar patient populations or adequate risk adjustment. Mortality may be very low at some hospitals because their patients are less severely ill on admission. This low mortality rate may not be achievable by other hospitals that admit a higher proportion of severely ill patients. Therefore, benchmarking outcomes is fraught with hazards that are much more easily addressed when benchmarking usage of a process of care benchmark rather than the outcome of that usage. For example a recommendation of daily aspirin is appropriate for patients with a wide range of condition severity. ARE COMMON PROTOCOLS NEEDED? Benchmarks can be established across a wide range of diagnoses, procedures or settings. When hospitals or individual physicians are compared, ABCs do not require the prior establishment of common clinical protocols in the sense of exact pathways. However, identical indicators and processes of care are important if valid comparisons among providers are to be made. CAN ABC™s BE USED TO MEASURE HP 2010 GOALS? The ABC system determines a benchmark performance based on the present practice of the 'best' care-givers. Because ABCs are based on actual performance data from practicing caregivers -- as opposed to 'expert opinion' targets -- we know they are achievable and realistic. By definition, ABC benchmarks have already been achieved by some providers. UNWISE USE OF ABC™s SECTION 3 Target performance benchmarks can be applied to each of these, but there are important differences in the methodology, and types of data that can be used to measure process compliance. Appropriate management. Here the benchmarks will tend to be related to specific management of target conditions, e.g., proportions of asthmatic, diabetic or post MI patients receiving a particular treatment. Process of care measures will tend to be driven by treatment guidelines. Measurements will often be proportions of a specific sub-population with a specific diagnosis who receive or do not receive particular interventions or therapy, e.g., the percentage of post MI patients advised to take an aspirin a day, or receive regular BP or serum lipid measurement. Multiple interventions may be appropriate for most patients in a diagnostic cohort group (e.g., all post-MI patients), but one or more intervention could be contraindicated in a subset of the cohort, e.g., post-MI patients with asthma. Each intervention would therefore be measured against a different base population, i.e., those patients for whom the therapy is clinically appropriate. The ABC method measures of the use of sound processes of care will be specific to the treatment function, e.g., the proportion of appropriate post MI patients for whom a beta-blocker is appropriately prescribed, the proportion of eligible diabetic patients given a foot examination, or the proportion of patients who smoke counseled to discontinue smoking. ABCs can accommodate all of these different approaches, and thus be useful for a very wide range of settings and clinical or public health objectives. Prevention. Benchmarks will tend to be related to receipt of screening, counseling, educational and public health services for a range of target conditions or risk factors. Population health goals similar to those described in the Healthy People 2010 project will often, but not necessarily, be central. Measures will include, for example, proportions of the population to whom a particular type of screening or intervention was administered, e.g., children under 2 years of age receiving immunizations and vaccinations; proportions of the population who are counseled about the dangers of smoking, not exercising, being over weight, drinking excessively, neglecting prenatal care, etc. Measures of good preventive care will tend to be proportions of target groups to whom a particular screening, counseling, education, or public health service is provided. · Time to treatment. Some benchmarks may be time critical, e.g., that within a certain period after onset of an acute episode the patient was seen by a physician or other appropriate first instance care professional. Performance measures will be the proportion of patients receiving care within the specified time period. For example: time to reach an emergency center. How long before the patient was stabilized and a diagnosis made? How long to transport patients to a tertiary care center, or to receive surgery? Time to receive radiology, lab tests, etc.? Was the patient seen in time for time critical medication, e.g., a thrombolytic for a stroke patient, to be given? An advantage of the ABC method is that it can readily be adapted to measure continuous variables such as time to treatment. ABC™ IMPLEMENTATION DECISIONS 1. Define the appropriate patient group and identify the process (or processes) of care that is (are) to be measured. Use guidelines and assess the weight of the evidence for the process of care selected. This, for example, could be a specific treatment protocol for patients with a specific diagnosis, e.g., that all post-AMI patients should be advised, within a certain number of days after admission, to take aspirin therapy. Are there multiple processes of care within one patient group that need to be measured, e.g., lab tests and prophylactic therapy for a specific diagnosis? Or, are waiting times to see a physician, attentiveness of nursing personnel, ability to understand medical instructions, and time to receive a diagnosis in ER key process of care measurements. They certainly are in patient satisfaction studies. 2. Is consent required? This is usually an issue only if the results are to be used in research or patients are identified for follow-up. Are necessary forms available? Is IRB approval necessary? Is follow-up with a patient's primary care physician necessary? 3. Identify the source of data. Billing records, chart abstracts, etc. Review weaknesses in the sources and determine whether more than one source should be used. Consider the cost/benefit of each source in relation to the accuracy of the data, and the level of statistical precision needed for operational decisions. 4. Sample or total measurement. Decide whether all patients should be measured or only a sample. How shall the sample be drawn to ensure that it is representative? 5. Define how small denominator situations shall be handled. Perhaps only physicians seeing a certain minimum number of patients should be included. This is a very important consideration and is discussed more fully in Section 4 - Calculations. 6. Define the time period over which the process of care will be measured. What is practical? What proportion of a provider's patients are available for testing in the defined period? Are there significant differences between patients who visit frequently or infrequently? Is seasonality a factor? How can loss to follow-up be minimized? 7. How will the data be collected, processed and analyzed? Who will enter the data? What database software will be used? What reports are needed? In Appendix I we illustrate how the ABC benchmark calculations are made. The same principles of performance benchmarking apply to both comparisons of inter- and intra-provider interventions, i.e., between hospitals, or between physicians within the same hospital, clinic or nursing home. However, because the databases used differ in some significant respects (intra-provider data are much richer in their clinical and cost detail) each is described separately in this manual. Several methods have been used to assess rates of patient testing, medications prescribed or procedures performed. These include patient-self report surveys, physician surveys, manual and electronic medical records, chart abstracts, clinical decision support systems, hospital information systems, billing records and claims data. Each method has its limitations: patient self-reports depend upon recollection; physicians may over-estimate provision of services; preventive services may not be recorded if not charged, chart review is time-consuming and expensive. Claims records may contain inaccurate coding. Variations in written terminology, use of abbreviations and spelling errors may be difficult to code consistently without the best, expert system based, encoding software, and data in charts may be difficult to locate or detail be sparse or totally omitted. SECTION 4 METHODOLOGY AND COMPUTATION ISSUES AND DEFINITIONS In the following section we describe what we believe are some unique features and terminology. We also discuss some unresolved issues and shall welcome suggestions for improvement or clarification of the methodology. Important issues and ABC terminology are: The benchmark breakpoint. We define superior performance as that received by an admittedly somewhat arbitrary level of the top 10% of patients. We do this in the following way. After calculating an adjusted performance measure--termed a performance fraction--for each provider (see discussion of Bayesian estimator below), we rank providers in descending order of performance fraction. The number of patients (the denominators) is then cumulated in the same descending order in which the providers were ranked. When we reach a point at which 10% of total patients are included, we call that the benchmark breakpoint. However, if additional providers have performance fractions that tie the provider at the breakpoint, all of the tied providers are included in the group defined to be above the breakpoint. Thus even though we set out to have a breakpoint at 10% of patients, we could, after including all tied providers, have a slightly higher percentage. We recommend that the breakpoint be expressed to at least the third decimal place to determine the cut-off. In the example in Appendix I, 12.3 % of patients were above the breakpoint. It is not a disadvantage of the ABC method that more than the top 10% of patients qualified to be included above the breakpoint. These were all patients of providers giving the defined level of high quality care. Benchmark flexibility. Users can, if they wish, select a different cut-off percentage: substitution of any other percentage e.g., a 5% or 15% level, is theoretically acceptable. The 10% level is based on experience at UAB and is the benchmark decision point used by us in several different applications of the ABC methodology. The importance of the ABC method lies not in the 10% level we have chosen. Rather it rests on the objective and pre-determined criteria with which a cut-off level of excellence is pragmatically rationalized and set. A lower cut-off might be set early in a quality improvement program and, if progress is made, reduced in subsequent measurements. The pared mean. This is the ABC term devised to indicate the benchmarked mean performance of providers in the cohort above the benchmark breakpoint described above. The pared mean is the mean performance fraction of all providers above the breakpoint. It is the performance of providers giving the "best" care for at least the top 10% of patients. Because the pared mean is the mean performance of providers above the benchmark cut-off level, it follows that some providers will be included in the pared mean calculation but be above the benchmark, and some will be included but be below the benchmark. The pared mean is not Bayesian adjusted (see below). This may be made clearer by the diagram below and by the sample calculation included in Appendix I. SMALL DENOMINATORS As described above, a benchmark breakpoint line is drawn at the point at which 10% (or slightly more if there are tied performers) of patients are covered. It is possible that these 10% of patients could be treated by a small or a large number of providers depending on the number of patients per provider. Thus, if 2,580 patients were the base number of cohort patients covered by all HMO providers taking part in an investigation, the 10% patient cut-off is 258 patients. The extreme possibility exists that all 258 patients might be treated by only one provider with exceptional performance and a large patient volume. An opposite potential problem would result if a large number of providers had only one or two patients and all performed at the 100% level. This latter possibility (known as the small denominator problem) has been anticipated and the Bayesian Estimator technique is used to diminish the impact of this. THE BAYESIAN ESTIMATOR The potential problem of small denominator numbers has been referenced several times. If a physician had only one qualifying patient, then clearly the performance of that physician in recommending an intervention, e.g., mammography, can be either 0% or 100%. If providers were ranked on this measurement, a large proportion of providers with 100% performance for only a small number of patients could yield a misleading result because they would artificially inflate the benchmark level. Pragmatically, ABC users must consider whether a physician who recommended mammography for his only eligible patient (i.e., 100% performance), should be ranked higher than a physician who recommended mammography for nine of his ten eligible patients, clearly indicating consistently high, but not perfect, performance for a larger number of patients. The ABC system recognizes this potential problem and makes an approximate adjustment by calculating performance using a Bayesian Estimator technique (described by Agresti, A, 1990, Categorical Data Analysis. Pp 463-464. John Wiley & Sons, New York) which effectively reduces the impact of providers with small numbers of eligible patients. The result of application of this correction is the generation of a number called the Adjusted Performance Fraction (APF) which is calculated as follows: Adj. Perf. Fraction (APF) = (x + 1) / (d + 2). Where x = actual number of patients receiving the chosen intervention, and, d = total number of patients for whom the indicator intervention is clinically appropriate. Thus, in the case of a provider with only one appropriate patient for whom the intervention was actually given (100% performance), the adjusted performance fraction = (1 + 1) / (1 + 2) = 0.67 (or 67%). As the number of appropriate patients (d) increases, the performance fraction calculated using the ABC method and the unadjusted mathematical percentage tend to the same number, e.g., a provider treating 8 out of 10 patients will have a Bayesian Estimator adjusted performance of 0.75 vs an unadjusted percentage performance of 0.80. The two numbers are close to being equivalent (within 3%) at approximately 30 patients. In addition to the advantages of reducing the effect of performance percentages based on small numbers, the Bayesian adjustment allows all data to be used, rather than simply eliminating providers with small numbers. The Adjusted Performance Fraction (APF) is used to rank providers. The effect of the Bayesian estimator on different combinations of numerators and denominators is shown in Appendix II. Note the tendency to push values towards 0.5. The APF is higher than actual performance for values above 50% and lower for values below 50%. The results displayed by the ABC system show both the Bayesian Estimator adjusted performance fraction (APF) and the Unadjusted performance fraction. (UPF) -- see Illustration Data table in Section 5. COMPOSITE INDICATORS Whether and how, multiple interventions (indicators) that are appropriate for the same group of patients, can be combined into a composite indicator is an undecided issue. Problems we have encountered include: a) providers not having eligible patients or having very low numbers of patients for every indicator, and b) disagreement about the relative importance of weighting of indicators. For example, it might be said that an indicator of the overall quality of care for post-MI patients should include the proportion of patients for whom several different processes of care are recommended or performed, e.g., treatment measures such as giving thrombolytic therapy, or prophylactic measures such as advising daily aspirin, prescribing a beta-blocker or ACE inhibitor, measuring BPs regularly, ordering periodic cholesterol and/or triglyceride tests, advising smoking cessation, etc. Each of these is an appropriate indicator of care, albeit for a slightly different sub-population of the post-MI patient cohort. Each should be assessed as an overall indicator of the performance of care by all providers in the cohort group. At the time this manual is being written, we have not determined how composite indicators should be handled. We currently only assess each individual indicator alone. This is valuable because each indicator is a valid process of care known to favorably affect outcomes and, therefore, each is a process of care for which performance goals can be set and measured. However, a composite measure may be of value for a measure of overall quality and this is an area for further research. DENOMINATORS AND THE 'APPROPRIATE' PATIENT If a 'rate' is to be calculated (as a percentage or an index) as a basis for ranking hospitals or physicians early in the ABC calculation, there must be a numerator - the number of eligible patients on whom a specific intervention was performed - and a denominator - the total number of eligible patients for whom that intervention was appropriate. The criteria for 'appropriate' are a) that the intervention not be clinically contraindicated, e.g., beta-blockers are not appropriate for most asthmatics, b) that there should be no physical reason why the intervention cannot be given, e.g., a foot exam for double amputees, and c) that the intervention be clinically indicated e.g., it is possible that there could be no contraindications for an indicator, but that the patient might not benefit clinically from it. 'Best practice of care' indicators are actions that are recommended for most patients of a particular target group, e.g., all diabetics (the target group) should periodically receive a fundoscopic retinal examination (the process of care indicator). A physician who performed an eye exam on only 50% of his or her diabetic patients may be performing sub-optimal care and perhaps encouraged to improve his performance. PATIENT REFUSAL Some patients may refuse a process of care. Our thinking at present is that a documented offer to the patient of the indicator intervention should be credited in the numerator. HOW ARE ABC™s CALCULATED? The example in Appendix I demostrates how we calculated periodic benchmark levels for use of screening mammography among physicians participating in a local HMO. Appropriate patients for screening are defined as all female patients over the age of 50 served by the HMO who are not being treated for, nor are suspected of having, breast cancer. Microsoft Excel- and Access-based calculation models, along with this manual, are available on CDROM by request. (For information on the CDROM contact coere@uab.edu) WHAT IS THE RELIABILITY AND VALIDITY FOR EACH INDICATOR? Reliability relates to the reproducibility of a measure, validity judges whether a measurement actually measures the phenomenon it purports to measure. There are several different types of statistical validity measures, and there are also important considerations of practical and operational validity. A difference between two numbers may have statistical significance, but have no practical difference. A chosen indicator could be a clinically acceptable process, but if the measurement of that indicator is unreliable or invalid, the indicator has no value as a measure. Descriptive statistics and controls set in place when we abstract or collect data in another way, help define reliability: validity is derived from the guidelines and supporting literature. CONFIDENCE INTERVALS Confidence intervals are frequently used to convey a sense of the levels of precision attributable to sampling. ABCs are based on actual performance observed on a pre-determined "universe" of patients rather than a sample. We do not use ABCs to make statistical inferences, but rather to hold up concrete targets of excellent performance. For this reason, we do not feel it is appropriate to place confidence intervals around ABCs. CONCLUSIVE EVIDENCE UAB Investigators continue to test and refine the ABC methodology and invite others to contribute to the process of refinement. In one controlled trial, recently published in JAMA (see bibliography), we group-randomized seventy community physicians participating in Alabama's CMS-sponsored Ambulatory Care Quality Improvement Project (ACQIP). These community physicians represented 2978 fee-for service Medicare patients with diabetes mellitus. We looked at the effect of the ABC feedback versus feedback of unadjusted mean performance as part of a CMS-model quality improvement project on the following performance indicators: influenza vaccination, foot examination, glucose control, cholesterol levels and triglyceride levels between the ABC and non-ABC physicians. We concluded that the use of the ABC significantly enhances the effectiveness of physician performance feedback in the context of a multimodal quality improvement intervention. In addition, the CCP has performed several measurements which do show sustained improvements in mean process of care indicator performance. SECTION 5 COMPARISONS WITH WHOM Early work has suggested that providers are most interested in performance comparisons with other local providers rather than with nationwide averages. Eventually, we envision making comparisons between sub-groups, e.g., urban vs rural providers, teaching vs non-teaching institutions, for-profit vs not-for-profit hospitals, physicians under capitated vs fee-for-service payment schemes. FEEDBACK MECHANISMS We at UAB have designed a simple form which we use to feedback to individual practitioners their own performance and an indication of the level of performance of benchmark providers. An example is given in Appendix III. FEEDBACK EFFECTIVENESS TESTS Several trials of the effectiveness of ABC feedback are ongoing. One such study was discussed in Section 4 and recently published in JAMA (see Bibliography). Similar tests are being conducted by QIOs in other states where providers for whom base levels of performance have been measured are randomly divided into two groups. One group receives only their own performance measure, and no benchmark result, the other group receives both measures. Subsequent practice data will be collected after a period of time. If we find that the future performance of those who received benchmark comparison data will be higher than those who did not we may conclude that the ABC method has played a part in bringing about that improvement. We particularly hope that other users of the ABC method will share their experiences and evaluate its effectiveness. SELECTED BIBLIOGRAPHY Kiefe CI, Allison JJ, Williams OD, Person SD, Weaver MT, Weissman NW. Improving quality improvement using achievable benchmarks for physician feedback. A randomized controlled trial. JAMA. 2001; 285(22):2871-2879. Allison JJ, Kiefe CI, Weissman NW. Can data-driven benchmarks be used to set the goals of Healthy People 2010? American Journal of Public Health January 1999; 89(1):61-5. Weissmann NW, Allison JJ, Kiefe CI, Farmer RM, Weaver MT, Williams OD, Child IG, Pemberton JH, Brown KC, Baker CS. Achievable benchmarks of care: The ABC™s of benchmarking. Journal of Evaluation in Clinical Practice 1999; 5(3):269-81. Kiefe CI, Weissman NW, Allison JJ, Farmer RM, Weaver M, Williams OD. Methodolgy matters-XII. Identifying Achievable Benchmarks of Care: concepts and methodology. International Journal for Quality in Health Care 1998; 10(5):443-7. Boscarino JA, Chang J. Commentary: inaccurate data on the quality of care may do more harm than good--an alternative approach is required. American Journal of Medical Quality 1997; 12:196-200. Hofer TP, Bernstein SJ, Hayward RA, DeMonner S. Validating quality indicators for hospital care. Joint Commission Journal on Quality Improvement 1997; 23:455-67. Palmer RH. Process-based measures of quality: the need for detailed clinical data in large health care databases. Annals of Internal Medicine 1997; 127:733-8. Mohr JJ, Mahoney CC, Nelson EC, Batalden PB, Plume SK. Improving health care, Part 3: Clinical benchmarking for best patient care. Joint Commission Journal on Quality Improvement 1996; 22:599-616. Berkey T. Benchmarking in health care: turning challenges into success. Joint Commission Journal on Quality Improvement 1994; 20:277-284. Berwick DM. Continuous improvement as an ideal in health care. New England Journal of Medicine 1989; 320:53-6. Wennberg J. Which rate is right? [editorial]. New England Journal of Medicine 1986; 314:310-1. |
 |
|