Trained Observer Ratings

by Jake Cowan, Harry P. Hatry, Katharine Mark, Ritu Nayyar-Stone

adapted from "Developing A Performance Management System for Local Governments: An Operational Guideline" by Ritu Nayyar-Stone, Katharine Mark, Jake Cowan and Harry P. Hatry, June 2002, and Performance Measurement: Getting Results

What are they used to measure?

Trained Observer Ratings are used to measure outcomes that can be perceived by the eyes or other physical senses of an observer. Individual trained observers, or a team of trained observers assess outcome conditions using predefined rating scales. This can be a highly accurate and reliable procedure with a clearly-defined rating system, adequate training of the observers, adequate supervision of the rating process, and a procedure for periodically checking the quality of the ratings. If properly done, the ratings can provide measurements that can be compared over time.

Examples of conditions that can be measured using Trained Observer Ratings include

• cleanliness of streets and alleys;
• condition of trash receptacles in public areas;
• presence of offensive odors from solid waste;
• condition of roads (potholes, sidewalks, paved area, etc.);
• condition of facilities such as school buildings;
• condition and visibility of street signs;
• ability of elderly or persons with handicaps to undertake normal activities of daily living;
• safety conditions (e.g. condition of safety equipment in buildings such fire extinguishers, hoses, sprinklers);
• condition of safety equipment in buildings (fire extinguisher, hose, sprinklers); and
• cleanliness of public baths.

How do they work?

Trained observers use three major types of rating systems:

• written descriptions,
• photographs, and
• other visual scales such as drawings or videos.

Written descriptions. This is the simplest and most familiar type of rating system. It depends on specific written descriptions of each grade used in the rating scale. Below is an example of a written set of grades for street litter. These ratings can be made from a car, or by observers on foot.

Rating scale for street litter accumulation

1. Street completely clean—No more than two pieces of litter or broken glass.
2. Street largely clean—A few pieces of litter and broken glass observable, but only in the form of isolated discarded items; A single accumulation less than or approximately equal to the volume of a grocery sack should be rated as 2.
3. Lightly scattered litter—Litter or broken glass along all or most of the street, or one heavy pile of litter, but no accumulations of litter large enough to indicate dumping; A single accumulation greater than the volume of a grocery sack but less than the volume of a garbage can should be rated as 3.
4. Heavily scattered litter—Heavily scattered streets with litter and broken glass; litter accumulation in piles; or heavy litter distributed down all or nearly all the block; A single accumulation greater than the volume of a garbage can should be rated as 4.

Photographic rating scales. Photographic scales can be more precise than written scales in providing clear definitions of each ratings grade, and make ratings easier to understand. Photos are used to represent each of the grades on the rating scale. Observers are given (and trained in the use of) the set of photos, with several representing each grade on the rating scale. Below is a sample photographic rating scale for street surface rideability.

Sample photographic rating scale: Street Surface Rideability

Condition 1: Smooth		Condition 2: Slightly bumpy
Condition 3: Considerably bumpy		Condition 4: Safety hazard

Other visual scales. Visual rating scales can also use drawings or sketches that represent each grade on a rating scale. An example of this is sketches representing conditions of school buildings, or classroom walls. This kind of rating scale was used by the New York City school system to track the physical condition of its schools and to help make decisions about building repairs.

Research examples

Public Use of Urban Parks: A Methods Manual for Park Managers and Community Leaders

Performance Measurement: Getting Results

How Effective Are Your Community Services? Procedures for Measuring Their Quality

"Albania: Performance Measurement: Improving Municipal Service Delivery and Strengthening Local Autonomy"

"Thailand: Performance Measurement for the Government of Thailand"