The Scope of Performance Measurement
In New York City, a priest and a taxicab driver died and went to heaven. Saint Peter showed the priest his eternal dwelling place—a shack. Saint Peter then showed the driver his eternal dwelling place—a mansion. The priest was angry and asked Saint Peter, "Why the difference?" Saint Peter said, "When you preach, people sleep. When riders get into his cab, they pray!"
For this book, RESULTS are what count!
What Is Performance Measurement? Why Do It?
Managers of any sports team need to know the running score so they can assess whether changes are needed for the team to win. Managers of public agencies and private, nonprofit organizations need similar information. For businesses, the running score is data on profits and market share. Costs alone mean little. Measuring that running score and using it for better performance are the subject of this book—a subject popularly called performance measurement.
Performance measurement has many meanings, but it is defined here as regular measurement of the results (outcomes) and efficiency of services or programs. The new element in this definition is the regular measurement of results or outcomes. Regular measurement of progress toward specified outcomes is a vital component of any effort at managing-for-results,1 a customer-oriented process that focuses on maximizing benefits and minimizing negative consequences for customers of services and programs. The customers may be citizens receiving services directly or citizens or businesses affected indirectly.
If the right things are not measured, or are measured inaccurately, those using the data will be misled and bad decisions will likely follow. As the old saying puts it: garbage in, garbage out.
A major use of performance information is to establish accountability, so citizens and elected officials can assess what programs have achieved with the funds provided. Another major use is to help programs develop and then justify budget proposals. But performance information is at least as important to help managers throughout the year. Public and private managers often say that performance information will not help them because their problem is too few resources to do what needs to be done. Yet managers need performance information to help them decide how to increase their ability to get the job done with whatever resources they have—and to provide evidence to the budget decisionmakers that they are indeed getting the biggest bang for the buck.
Programs have been tracking expenditures and physical outputs for decades. Tracking these elements is useful to program personnel but says little about what resulted—how customers and the public benefited. Regular tracking of outcomes is intended to fill this gap.
Outcome tracking is the new kid on the block for most public and private services and programs, although ample precedents exist. Police and other law enforcement agencies for many years have regularly tracked and reported such data as crime and clearance rates. Transportation agencies have tracked accidents, injuries, and deaths. Health officials have tracked the incidence of serious diseases and mortality. Fire agencies have tracked the number of fires and the resulting injuries and deaths. Environmental protection agencies have tracked the pollutant content of air and water. School districts have tracked dropouts and test scores. However, such tracking focused on jurisdiction-wide data, not specific program outcomes that are essential information.
Regular tracking is a key characteristic of performance measurement. For budget purposes, annual data are usually sufficient, but agencies and their managers need more frequent outcome information to assess the success of their program activities, identify where significant problems exist, and motivate personnel to strive for continuous service improvement.2
A particularly crucial outcome characteristic for public programs—often neglected in discussions of performance measurement—is equity. A well-designed measurement system enables agency managers to assess the fairness of a program and adjust it appropriately. A good performance measurement system will help officials demonstrate to the public and to policymakers that services are delivered fairly—thus building trust in the program. As chapter 8 discusses, disaggregating outcome data by the characteristics of the citizens affected is a major way to help assess equity.
Limitations of Performance Measurement
All those using performance measurement information, whether inside or outside government or in a private agency, should understand what it can and cannot do and keep their expectations realistic. Performance measurement has three primary limitations.
1. Performance Data Do Not, by Themselves, Tell Why the Outcomes Occurred
In other words, performance data do not reveal the extent to which the program caused the measured results. This point is an important one. The analogy to managers of sports teams helps here. The manager needs to know the running score. If the team is losing, whether an individual game or over the whole season, the manager and other team officials may need to change the game plan. But the score does not tell the officials why the score is the way it is. Nor does the running score tell what specifically needs to be changed to improve the score. For that information, the managers, coaches, and other team officials need to seek explanations before they act.
It is the same for service delivery. Managers and other officials need to track results and use that information to help guide them about what, if any, future actions to take. Performance measurement is designed primarily to provide data on outcomes (the score). But to be most helpful (as discussed in chapters 9 and 10), performance measurement systems also need to have built into them opportunities to analyze the details of program performance and steps to seek explanations for the outcome data such systems produce.
This limitation raises a major issue in performance measurement that generates controversy: accountability. What should managers be held accountable for? In the past, the government of New Zealand had taken the view that responsibility for program outcomes rested solely with officials at the policymaking level, thus removing all accountability for outcomes from the operating departments. Important outcomes are seldom, if ever, fully under the control of a particular agency (public or private). Nevertheless, the agency and its personnel do share responsibility for producing those outcomes. As long as a program has any role in delivering a service intended to help produce particular outcomes, the managers of that program—and its personnel—have a responsibility to track the relevant outcomes and use that information to help improve results.
Agency personnel and other officials are often too ready to believe they lack responsibility over outcomes—in part out of fear that they will be blamed unfairly for poorer-than-desired outcomes. This fear is reasonable. However, recognizing shared responsibility helps agencies create innovative solutions that can improveservice outcomes, even in the face of highly limited resources. And this understanding can lead to more use of performance partnerships among programs, agencies, levels of government, and between the private and public sectors.
2. Some Outcomes Cannot Be Measured Directly
The classic example is success in preventing undesirable events, such as prevention of crime or reduction of illicit drug use. In such cases, surrogates can usually be used, such as indicators that reflect trends over time in the number of incidents that were not prevented. This is not ideal, but this is the real world.
3. Performance Measurement Provides Just Part of the Information Managers and Elected Officials Need to Make Decisions
Performance measurement does not replace the need for expenditure data or political judgments, nor does it replace the need for common sense, good management, leadership, and creativity. A major purpose of performance measurement is to raise questions. It seldom, if ever, provides answers by itself about what should be done.
Exhibit 1-1 presents common objections from agencies and programs required to implement an outcome-based performance measurement process. Each objection is an element for concern. Subsequent chapters will address most of these concerns and hopefully will at least allay them.
Outcome-Focused Efficiency Measurement
In performance measurement, efficiency is usually defined as the ratio of the amount of input (usually monetary expenditures or amount of employee time) to the amount of product created by that input. Unit-cost ratios that relate expenditures to physical outputs have been common in public agencies for years. The trouble with input-to-output ratios is they can be improved by reducing the quality of the output. If outcomes are tracked, a considerably more accurate indicator of true efficiency becomes possible. For example, “cost per client served” is an output-based efficiency indicator. Efficiency appears to increase when a program spends less per client, even if the condition of the typical client deteriorates. “Cost per client whose condition improved after services” is an outcome-focused efficiency indicator. It gives a much more meaningful picture of a program’s real accomplishments.
Take the example of a program that holds regular sessions to help customers stop smoking. “Cost per session held” is considerably under the control of the program. “Cost per customer who quits smoking” is not, because whether someone quits probably also depends on a host of other factors besides the stop-smoking sessions. But is “cost per session held” a true measure of efficiency? Officials and citizens are considerably more likely concerned with efficiency in producing the desired outcome. Even if a causal link cannot be firmly drawn, the program still has some responsibility for affecting the desired outcome. An outcome-based indicator provides more insight into how much the program is helping accomplish that objective.
Which Organizations Are Suitable for Performance Measurement?
Managing-for-results applies to all agencies that provide services to the public, whether the agency has ample or highly limited resources, is small or large, is public or private, or is in a developing or developed country.3 As long as the agency is delivering services to the public, its management and elected officials should be intensely concerned with the quality, outcomes, and efficiency of those services and should measure performance.
Even small agencies with very limited resources should be able to track some aspects of service quality and outcomes (probably more than seems possible at first glance) and improve operations with their existing resources. Poorer agencies with fewer resources will have to rely on less sophisticated procedures and, perhaps, more volunteers.
The same principles apply to all agencies. Officials and managers need to recognize and support the need for outcome information and be willing to use it to improve services, however tight their budgets.
Which Services Are Suitable for Performance Measurement?
The procedures and issues of performance measurement are applicable to most public and private services—ranging from public safety programs, to public works programs, to human service programs, to environmental protection programs, to regulatory programs, and to defense programs. Performance measurement is even applicable to internal support services, such as building maintenance, fleet maintenance, information systems, personnel activities, and purchasing. However, outcomes of these support services occur primarily within an organization, and it is usually difficult, if not impossible, to estimate the effect these internal services have on the outcomes of external services. This book focuses on external services, but the same principles apply to support services.
The regular tracking of performance measurement may not be readily applicable to activities whose important outcomes do not occur for years, if not decades. Long-range planning and basic research are primary examples. The federal government’s Government Performance and Results Act of 1993 has been applied broadly to every type of federal program. Nevertheless, basic research programs have had only slight success at fitting tracking systems into the annual outcome-oriented performance measurement process. Regular tracking can be used to assess whether timelines have been met, expenditures have been kept within budget, and the quality of any interim product is acceptable (such as by using expert panels to rate the quality and progress of ongoing research). For assessing the major outcomes of research, analytical resources are better spent on later, in-depth evaluations.
Performance Measurement in Relation to Other Evaluation Activities
Program Evaluations and Other In-Depth Studies
Performance measurement can be considered a field of program evaluation. However, program evaluation usually refers to in-depth, special studies that not only examine a program’s outcomes but also identify the “whys,” including the extent to which the program actually caused the outcomes. Such in-depth program evaluations are not the subject of this book.4
In practice, many of the so-called program evaluations undertaken by government (federal, state, or local) provide information on outcomes but little evidence on the causal link between activities and results. Even so, in-depth studies can provide many insights about what happened and why. Performance measurement cannot generally provide this information.
Because of the time and cost involved, in-depth evaluations are usually done much less frequently and only for selected programs. Performance measurement and in-depth program evaluations are complementary activities that can nourish and enhance each other. Findings from a program evaluation completed during a given year can add to or supersede that year’s performance measurement data. Data from an agency’s performance measurement system can offer program evaluators data, useful indications of trends, and questions that encourage more in-depth evaluation. Sometimes evaluators can use the existing performance measurement procedures to collect data.
Performance Auditing
Performance audits, which are becoming more frequent, are typically conducted by auditors or inspectors general. They are ad hoc studies, often closely resembling in-depth program evaluations, that are applied to a selection of public programs each year. Performance auditors should have considerable interest in performance measurement systems as ways to provide data on outcomes for use in audits. In addition, these offices are likely to be given the responsibility for periodically assessing agencies’ performance measurement systems, the indicators used, and the data being provided. (This quality control responsibility is discussed in chapter 14.)
Budgeting, Strategic Planning, and Policy Analysis
Performance measurement provides information primarily about the past. Budgeting, strategic planning, and policy analysis are primarily about the future. As discussed in later chapters, performance data provide a baseline for decisions and give clues about what might happen in the future. The future-oriented processes require estimation and judgment skills that performance measurement systems cannot provide by themselves. Subsequent chapters (especially 12 and 13) introduce these issues but do not attempt comprehensive coverage of budgeting, strategic planning, or policy analysis. Rather, these topics are discussed only in the context of the (important) role that outcome-focused performance measurement systems play in these activities.
Role of Agency Employees
The employees of agencies undertaking performance measurement clearly have a stake in the process. Later chapters address the roles of this important stakeholder group in helping identify appropriate performance indicators and in using performance information to help improve services. The performance measurement work described here does not address the measurement of employee job satisfaction, however, because employees are considered suppliers of services, not customers.
Moving Performance Measurement into Performance Management
Performance measurement focuses on measuring outcomes and efficiency. If at least some of the measurement information generated is not used, the effort and cost of the performance measurement process will be wasted. Use of the performance information —whether by program managers, agency officials, officials in the central government, elected officials, members of boards of private nonprofit organizations, or citizens—transforms performance measurement into performance management. The purely measurement chapters of this book are chapters 1 through 7, 14, and 15. Chapters 8 through 11 discuss key components that can greatly enhance usefulness and that reflect the transition from measurement into usefulness. Chapters 12 and 13 discuss the various uses of performance information.
Thus, this book is about both performance measurement and performance management.
A Guide to This Volume
Chapter 2 completes Part I by providing definitions that are the basic background for the material in the rest of the book.
Part II addresses the performance measurement process. Chapter 3 discusses organizational start-up. Chapters 4 through 6 address determining what the program’s objectives are and who its customers are (chapter 4), what outcomes should be tracked (chapter 5), and what the specific outcome indicators should be (chapter 6). Chapter 7 addresses how the data can be obtained.
Part III covers the critical issues of how to analyze, report, and use the performance measurement data. Chapters 8 and 9 focus on ways to make performance data useful to program personnel and others. Chapter 8 discusses the importance of procedures for providing more detailed breakouts of outcome data. Chapter 9 discusses benchmarking—that is, what comparisons should be made to help interpret outcome levels. Chapter 10 discusses analyses that can make the outcome information fully useful. Chapter 11 provides suggestions on an all too frequently neglected key element: reporting the findings. Chapters 12 and 13 identify major uses of performance information, with special attention to results-based budgeting.
Part IV (chapters 14 and 15) addresses various other important performance measurement concerns, including the long-term problem of controlling the quality of the information performance measurement produces (chapter 14), political considerations, and the need for personnel training (chapter 15).
Part V (chapter 16) summarizes the principal points about performance measurement that are important in producing a practical process with real world utility.
References and Notes
1. “Governing-for-results” and “results-oriented government” refer to the same process. We have used “managing-for-results” here to indicate that the process is not restricted to government (executive or legislative) but is equally applicable to private service agencies. Other phrases have been used, such as “results-based management,” “managing by results,” and the like. Recent work on legislatures has used the phrase “legislating for results.”
2. A distinction is often made between the way in which a service is delivered (such as its timeliness, accessibility, and courteousness to customers) and the results the service is intended to achieve (such as actual improvements in the condition of customers). As will be discussed in chapter 4, these aspects of service delivery quality are important to customers (and, thus, we have categorized them “intermediate outcomes”), but they usually do not indicate how much progress has been made toward service objectives.
3. Numerous publications have been written on this subject. A few recent ones are John Kamensky and Albert Morales, eds., Managing for Results 2005 (Lanham, MD: Rowman & Littlefield Publishers, 2005); Barry White and Kathryn Newcomer, eds., “Getting Results: A Guide for Federal Leaders and Managers” (Vienna, VA: Management Concepts, 2005); and Dall W. Forsythe, ed., Quicker, Better, Cheaper? Managing Performance in American Government (Albany, NY: Rockefeller Institute Press, 2001). For those interested in performance measurement in the international scene, some publications are Jody Zall Kusek and Ray Rist, Ten Steps to a Results-Based Monitoring and Evaluation System (Washington, DC: The World Bank, 2004); Anwar Shah, ed., Public Services Delivery (Washington, DC: The World Bank, 2005); Korean Development Institute, “Reforming the Public Expenditure System: Medium-Term Expenditure Framework, Performance Management, and Fiscal Transparency,” (Seoul and Washington, DC: Korean Development Institute and The World Bank, Conference Proceedings, March 2004); Hans de Bruijn, Managing Performance in the Public Sector (London: Routledge, 2002); and Burt Perrin, “Moving from Outputs to Outcomes: Practical Advice from Governments around the World” (Washington, DC: The World Bank and IBM Center for the Business of Government, 2006).
4. Considerable literature exists describing in-depth program evaluations and how they might be done.
