The Scope of Performance Measurement
In New York City, a priest and a taxicab driver died and went to heaven. Saint Peter then showed the priest his eternal dwelling place-a shack. Saint Peter then showed the driver his eternal dwelling place—a mansion. The priest was angry and asked Saint Peter, "Why the difference?" Saint Peter said, "When you preach, people sleep. When riders get into his cab, they pray!"
For this book, RESULTS are what count!
What Is Performance Measurement? Why Do It?
Managers of any sports team need to know the running score so they can assess whether changes are needed for the team to win. Managers of public agencies and private, nonprofit organizations need similar information. For businesses, the running score is data on profits and market share. Costs alone mean little. Measuring that running score and using it for better performance are the subject of this book-a subject popularly called performance measurement.
Performance measurement has many meanings, but it is defined here as measurement on a regular basis of the results (outcomes) and efficiency of services or programs. The new element in this definition is the regular measurement of results or outcomes. Regular measurement of progress toward specified outcomes is a vital component of any effort at managing-for-results,1 a customer-oriented process that focuses on maximizing benefits and minimizing negative consequences for customers of services and programs. The customers may be citizens receiving services directly or citizens or businesses affected indirectly.
If the right things are not measured, or are measured inaccurately, those using the data will be misled and bad decisions will likely follow. As the old saying puts it: garbage in, garbage out.
A major use of performance information is to help develop and then justify budget proposals. But it is at least as important to help managers manage throughout the year. Public and private managers often say that performance information will not help them because their problem is too few resources to do what needs to be done. Yet they need performance information to tell them how to increase their ability to get the job done with the resources they have-and to provide evidence to the budget decisionmakers that they are indeed getting the biggest bang for the buck.
Tracking expenditures and the physical outputs programs produce has been done for decades. Tracking these elements is useful to program personnel but says little about what resulted-how customers and the public benefited. Regular tracking of outcomes is intended to fill this gap. Outcome tracking is the new kid on the block for most public and private services and programs, although ample precedents exist. Police and other law enforcement agencies for many years have regularly tracked and reported data such as crime and clearance rates. Transportation agencies have tracked accidents, injuries, and deaths. Health officials have tracked the incidence of serious diseases and mortality. Fire agencies have tracked the number of fires and the resulting injuries and deaths. Environmental protection agencies have tracked the pollutant content of air and water. School districts have tracked test scores. However, the focus has often been on jurisdictionwide data, not the outcomes related to specific programs, which are essential information.
Regular tracking is a key characteristic of performance measurement. For budget purposes, annual data are usually sufficient, but agencies need more frequent outcome information to assess how successful the activities of their programs have been, identify where significant problems exist, and motivate personnel to strive for continuous service improvement.2 One final point. A particularly crucial outcome characteristic for public programs-often neglected in discussions of the uses of performance measurement-is equity. A well-designed measurement system enables agency managers to assess the fairness of a program and make appropriate adjustments. A good performance measurement system will help officials demonstrate to the public and to policymakers that services are delivered fairly-and this will build trust. (Chapter 8's suggestion of categorizing outcomes by the characteristics of the citizens affected is a major way to help assess equity.)
Limitations of Performance Measurement
All those using performance measurement information, whether inside or outside government or in a private agency, should understand what it can and cannot do and keep their expectations realistic. Performance measurement has three limitations.
1. Performance data do not, by themselves, tell why the outcomes occurred. In other words, they do not reveal the extent to which the program caused the measured results. This is an important point. The analogy to managers of sports teams helps here. Managers may need to make changes in the game plan, but the score does not tell them why it is what it is or what needs to be changed to improve the score. For that information the managers and their coaches need to watch the game films and have postgame conferences to identify specific aspects that went wrong. It is the same for service delivery. Performance measurement is designed primarily to provide data on outcomes (the score). But to be of most help, performance measurement systems also need to have built into them opportunities to assess the details of program performance and seek explanations for the outcome data such systems produce.
This limitation raises a major issue that is a source of controversy in performance measurement: accountability. What should managers be held accountable for? The government of New Zealand has taken the view that responsibility for program outcomes rests solely with officials at the policymaking level, thus removing all accountability for outcomes from the operating departments. Although important outcomes are seldom, if ever, fully under the control of a particular agency (public or private), the agency and its personnel do share responsibility for producing those outcomes. As long as a program has any role in delivering a service intended to help produce particular outcomes, the managers of that program-and their personnel-have a responsibility to track the relevant outcomes and use that information to help improve them.
Agency personnel and other officials are often too ready to believe they lack influence over outcomes. Acknowledging their shared responsibility helps create innovative solutions that can often help agencies improve the outcomes of the services they provide, even in the face of highly limited resources.
2. Some outcomes cannot be measured directly. A major example is success in preventing undesirable events, such as prevention of crime or reduction of illicit drug use. In such cases, surrogates can be used, such as indicators that reflect trends over time in the number of incidents that were not prevented.
3. The information provided by performance measurement is just part of the information managers and elected officials need to make decisions. Performance measurement does not replace the need for basic expenditure data or political judgments, nor does it replace the need for common sense or good management, leadership, and creativity. A major purpose of performance measurement is to raise questions. It seldom, if ever, provides answers by itself as to what should be done.
Outcome-Focused Efficiency Measurement
Efficiency is usually defined in performance measurement as the ratio of the amount of input (dollar expenditure, personnel time, or other physical input) to the amount of product produced by that input. Using unit-cost ratios that relate expenditures to physical outputs has been common in public agencies for years. The trouble with these ratios is that they can be improved by reducing the quality of the output. If outcomes are tracked, a more accurate indicator of efficiency becomes possible. For example, "cost per client served" is an output-based efficiency indicator. Efficiency appears to increase when less is spent per client, even if the condition of the typical client deteriorates. "Cost per client whose condition improved after services" is an outcome-focused efficiency indicator. It gives a much more meaningful picture of a program's real accomplishments.
Some people want to use only output-focused efficiency indicators, arguing that such indicators are completely under the control of the program and enable observers to attribute causation confidently. This approach to efficiency measurement presents problems, however.
Take the example of a program that holds regular sessions to help customers stop smoking. "Cost per session" is under the control of the program. "Cost per customer who quits smoking" is not, because whether someone quits something depends on a host of other factors besides the stop-smoking session. But is "cost per session" a true measure of efficiency? Officials and citizens are more likely concerned with efficiency in producing the desired outcome. Even if the causal link cannot be firmly drawn, the program still has some responsibility for affecting the desired outcome. An outcome-based indicator provides more insight into the extent to which the program is helping accomplish that objective.
Which Organizations Are Suitable for Performance Measurement?
Managing-for-results applies to all agencies that provide services to the public, whether the agency has ample or limited resources, is small or large, is public or private, or is in a developing or developed country. As long as the agency is delivering services to the public, its management and elected officials should be intensely concerned with the quality, outcomes, and efficiency of those services and should measure performance.
Even small agencies with very limited resources should be able to track some aspects of service quality and outcomes, probably more than seems possible at first glance, and make operational improvements with their existing resources. Poorer agencies with fewer resources will have to rely on less sophisticated procedures and, perhaps, more volunteers.
The same principles apply to all. Agency officials and managers need to recognize and support the need for outcome information and be willing to use it to improve the services they are providing, however tight their budgets.
Which Services Are Suitable for Performance Measurement?
The procedures and issues of performance measurement are applicable to most public and private services-ranging from public safety programs, to public works programs, to human service programs, to environmental protection programs, to regulatory programs, and to defense programs. Performance measurement is even applicable to internal support services, such as building maintenance, fleet maintenance, information systems, personnel activities, and purchasing. However, outcomes of these services occur primarily within an organization, and it is usually difficult, if not impossible, to estimate the effect these internal services have on the outcomes of external services. Therefore, this book focuses on external services.
The activities and outcomes to which the regular tracking of performance measurement may not be readily applicable are those whose important outcomes do not occur for years, if not decades. Long-range planning and basic research are primary examples. The federal government's Government Performance and Results Act of 1993 has been applied broadly to every type of federal program. Nevertheless, basic research programs have had only slight success at fitting tracking systems into the annual outcome-oriented performance measurement process. Regular tracking can be used to assess whether timelines have been met, dollars have been kept within budget, and the quality of any interim product is acceptable (such as by using expert panels to rate the quality and progress of ongoing research. For assessing the major outcomes of research, analytical resources are better spent on later, in-depth evaluations.
Performance Measurement in Relation to Other Evaluation Activities Program Evaluations and Other In-Depth Studies
Performance measurement can be considered a field of program evaluation. However, program evaluation usually refers to in-depth, special studies that not only examine a program's outcomes but also identify the "whys," including the extent to which the program actually caused the outcomes. The procedures involved in such in-depth program evaluations are not the subject of this book.3
In actual practice, many of the so-called program evaluations undertaken by government (federal, state, or local) provide information on outcomes but little evidence on the causal linkage between activities and results. Even so, in-depth studies can provide many insights about what happened and why, information ongoing performance measurement cannot provide.
Because of the time and cost involved, in-depth evaluations are usually done much less frequently and only for selected programs. Thus, performance measurement systems and in-depth program evaluations are complementary activities that can nourish and enhance each other. Findings from a program evaluation completed during a given year can add to or even supersede that year's performance measurement data. Data from an agency's performance measurement system can provide program evaluators with data, with useful indications of trends, and raise questions that encourage more in-depth evaluation.4Sometimes evaluators can use performance measurement procedures to collect data.
Relation to Performance Auditing
Performance audits, which are becoming more frequent, are typically conducted by auditors or inspectors general. They are ad hoc studies, often closely resembling in-depth program evaluations, that are applied to a selection of public programs each year. Performance auditors should have considerable interest in performance measurement systems as ways to provide data on outcomes that they can use in their audits.5
Strategic Planning, Budgeting, and Policy Analysis Performance measurement provides information primarily about the past. Budgeting, strategic planning, and policy analysis are primarily about the future. As discussed in later chapters, performance data provide a baseline for decisions and give clues as to what might happen in the future. These future-oriented processes require estimation and judgment skills that performance measurement systems cannot provide by themselves. This book provides an introduction to these issues (especially in chapters 11 and 12) but does not attempt comprehensive coverage of budgeting, strategic planning, or policy analysis. Rather, these topics are discussed only in the context of the role that outcome-focused performance measurement systems play in these activities.
Role of Agency Employees
The employees of agencies undertaking performance measurement clearly have a stake in the process. Later chapters address the roles of this important stakeholder group in helping identify appropriate performance indicators and in using performance information. The book does not address the measurement of employee job satisfaction, however, because employees are here considered suppliers of services, not customers.6
A Guide to This Volume
Chapter 2 completes part I by providing the definitions that are the basic background for the material in the rest of the book.
Part II addresses the performance measurement process. Chapter 3 discusses organizational start-up. Chapters 4 through 6 answer the questions of what the program's objectives are and who its customers are (chapter 4); what outcomes should be tracked (chapter 5); and what the specific outcome indicators should be (chapter 6). Chapter 7 addresses the question of how the data can be obtained.
Part III covers the critical issues of how to examine and use the performance measurement data. Chapters 8 and 9 focus on maximizing the usefulness of the data to program personnel and others. Chapter 8 discusses the importance of procedures for providing more detailed breakouts of outcome data. Chapter 9 discusses benchmarking, that is, what comparisons should be made to help interpret outcome levels. Chapter 10 addresses how to decide what analyses can be undertaken, given data availability, and provides suggestions on an all too frequently neglected key element: reporting the findings. Chapters 11 and 12 discuss major uses of performance information, with special attention to results-based budgeting.
Part IV (chapters 13 and 14) addresses a variety of other important performance measurement concerns, including the long-term problem of quality control of the information produced by performance measurement (chapter 13), political considerations, and the need for personnel training (chapter 14). Part V (chapter 15) provides a summary of the principal points about performance measurement that are important in producing a practical process that is useful in the real world.
References and Notes
- "Governing-for-results" and "results-oriented government" refer to the same process. "Managing-for-results" is preferable here because the process is not restricted to government (executive or legislative) but is equally applicable to private service agencies.
- A distinction is often made between the way in which a service is delivered (such as its timeliness, accessibility, and courteousness to customers) and the results it is intended to achieve (such as actual improvements in the condition of customers). As will be discussed in chapter 4 on the specific content of performance measurement, these service delivery quality aspects are important to customers and should be tracked as intermediate outcomes, but they usually do not indicate how much progress has been made toward service objectives.
- Considerable literature exists describing in-depth program evaluations and how they might be done.
- As Marcantonio and Cook note in their discussion of interrupted time-series designs, regularly collected performance data can be an important source of historical data for evaluators. See Richard J. Marcantonio and Thomas D. Cook, "Convincing Quasi-Experiments," chap. 7 in Handbook of Practical Program Evaluation, ed. Joseph S. Wholey et al. (San Francisco: Jossey-Bass Publishers, 1994).
- In addition, these offices are likely to be given the responsibility for periodically assessing agencies' performance measurement systems, the indicators used, and the data being provided. (This quality control responsibility is discussed in chapter 13.)
- One exception to this is the case of employees of lower-level public agencies through which higher levels of government work. In this context, lower-level agency staff are considered to be intermediate customers of the higher levels of government and their experiences and satisfaction germane to the quality of the higher-level government's program.
