The nonpartisan Urban Institute publishes studies, reports, and books on timely topics worthy of public consideration. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders.
This report is available in its entirety in the Portable Document Format (PDF), which many find convenient when printing.
Contents
Introduction. Why Use This Handbook?
Chapter 1. Professional Development and Systemic Standards-Based Reform
Chapter 2. Evaluation: Basic Definitions and Steps
Chapter 3. The Evaluation Frame: Understanding the Journey
Chapter 4. Stakeholders: Deciding Who Will Come on the Trip
Chapter 5. Evaluation Goals: Determining Your Destination
Chapter 6. Evaluation Design: Plotting Your Course
Chapter 7. Data Collection: Getting Started
Chapter 8. Data Collection: Choosing the Methods
Chapter 9. Data Collection: Creating and Using the Tools
Chapter 10. Data Analysis: Understanding What Happened
Chapter 11. The Evaluation Report: Telling the Story
Appendix: Guiding Principles for Evaluators
References and Resources
Glossary of Common Evaluation Terms
Introduction. Why Use This Handbook?
In the era of systemic standards -based reform based on challenging new academic standards, school districts, schools, teachers, and students are being held accountable for improved educational outcomes. To help meet these new goals, school administrators have an important responsibility to ensure that teachers possess the skills they need to meet higher expectations. As a consequence, many districts are changing the way they plan and deliver professional development. No longer is professional development seen as simply having teachers complete a specific number of hours of training. Instead, districts now are concerned about outcomes—typically, changes in teachers' knowledge, skills, attitudes, and/or behaviors, and, ultimately, improved student learning, as measured on tests aligned with state or district standards.
The way professional development is delivered to staff is also changing. Instead of offering only short-term in-service workshops, many districts are now providing intensive professional learning opportunities for teachers on a continual basis, such as creating mentoring relationships and building learning communities within schools. Providing these types of opportunities can be costly, however. Consequently, district staff must make informed decisions about the types of training opportunities to offer, for how long, and for which staff. This, in turn, means that districts will need to collect ongoing evidence about the effectiveness of their professional development programs and be able to revise their staff development programs to achieve the intended objectives and meet changing demands.
This handbook is written to help district staff members gain a working knowledge of how to evaluate their professional development programs. A wide range of staff may find themselves conducting or overseeing such evaluations, including directors of research and evaluation, professional development, and federal programs, as well as district staff members in small districts who must wear several hats. All of these individuals can benefit from this handbook.
Throughout, we have assumed that the district is engaged in systemic standards -based reform; that is, that it has already adopted learning standards and is using student assessments that are well aligned with those standards. We also assume that the district's professional development efforts are linked to its overall plan for focusing all components of the educational system on helping all students meet higher academic standards. This professional development is integrated with all aspects of the district's improvement efforts.
In writing this handbook, we have also assumed that the reader is not trained in evaluation, and that he or she needs basic practical information about how to design and conduct evaluations. Whether the district conducts its evaluation in-house or works with an outside evaluator, we assume that the objective is to get as much out of the evaluation process as possible—meaning that the district will use the results to better understand and improve its professional development program.
Some Guiding Principles Used in Writing This Handbook
Several principles guided us in the preparation of this handbook.
Evaluating professional development programs will strengthen a district's entire reform effort.
Many districts, overburdened with administrative and management tasks and short on funds, fail to incorporate evaluation activities into their ongoing program operations. In these situations, staff are hard-pressed to attribute observed improvements in teaching and learning to specific professional development efforts. Although some professional development programs may appear more successful than others, these districts lack a mechanism to link their "hunches" to documented results, other than anecdotal information. The absence of an evaluation can also weaken staff support for a professional development program. Participants may not receive feedback from district leaders on how professional development contributes to positive outcomes and may not understand how decisions about professional development are being made.
Evaluation results can be used to make thoughtful, cost-saving decisions about how to meet a district's professional development needs. Not only does evaluation help answer questions about the particular professional development program under study, but it builds a district's internal capacity for critical thinking, data collection and analysis, and overall decisionmaking related to reform (Taylor-Powell et al., 1998). Evaluation also helps district and school staff and other key stakeholders agree on a clear focus for districtwide reform efforts. For these reasons, we believe evaluation should be viewed as part of the process of building local capacity for reform, rather than merely as an "add-on."
There is no one best way to evaluate a professional development program.
We cannot prescribe a "cut and dry" model for evaluating professional development
efforts, for two reasons. First, there is no one "best" professional development program. The literature on professional development suggests that high-quality professional
development programs consist of a combination of strategies selected by districts based
on individual district goals and contextual factors (Loucks-Horsley et al., 1998). The
process of carefully planning and conducting such a program and reflecting on the
outcomes appears more important to the quality of a professional development program
than the use of a particular strategy or model.
Second, there is no one "best" evaluation plan. Evaluations need to be sensitive to local programs, and therefore no simple recipe exists for how they ought to be conducted.
Those responsible for the evaluation must make a series of choices based on key
features of the program, including its purposes and objectives and local context,
stakeholders' needs, available resources for the evaluation, and practical constraints on
data collection. Our goal with this handbook is to guide you in making these decisions.
To conduct a successful investigation, evaluators of professional development programs will need to balance conflicting demands from stakeholders.
Tension among the needs of different stakeholders may emerge during evaluations of
professional development programs. Some stakeholders may focus on getting to the
"bottom line," seeking to isolate the effects of a specific program on student test scores.
Others may be far more interested in understanding how and why a program is (or is not)
working and the program's relationship to meeting the goals of broader systemic reform
efforts. All of these needs are important, but with limited evaluation resources, evaluators
may struggle to strike a balance.
Another tension inherent in professional development evaluation is between "intermediate" and "end" outcomes. Professional development benefits teachers, students, and in some cases the school more generally. But if the bottom line, ultimately, is the impact of a program on student achievement, the question becomes how much weight to give other intermediate effects, such as changes in teachers' knowledge, attitudes, and behavior. Should changes in instruction be considered an intermediate or an end outcome? The answers to these questions depend on the goals of the professional development program
and the evaluation, as well as on constraints on the evaluation's scope.
Finally, stakeholders may demand evaluation results in a short time frame, such as one
year or less. However, it is likely that the deeper changes anticipated for professional
development that is related to new, higher academic standards—particularly the end
outcomes related to students—will take longer to achieve. During the first year of a
professional development effort, for example, changes in teacher attitudes and behavior
may be a reasonable outcome. On the other hand, a year or two later may be an
appropriate time to begin investigating changes in student achievement. Evaluators must,
therefore, balance the short-term need for results with the overall focus on developing a
systemic reform effort, expecting to measure important overall goals over a longer period
of time.
Overview of This Handbook
This handbook walks the reader through the decisionmaking that is involved in the
evaluation of professional development. Although district staff can use this handbook
throughout the evaluation process, we recommend reading it in its entirety before
beginning to design and implement an evaluation project, as many issues covered later in
the book are relevant to the planning and design phase. Most likely, readers will also want
to consult other resources about evaluation; some suggested resources are listed in the
References section at the end of the handbook.
Chapter 1 provides a brief overview of the role of professional development in systemic
standards-based reform, and chapter 2 provides general definitions and a description of
the steps needed to plan and conduct a professional development evaluation. Starting
with chapter 3, we move through the conceptual issues underlying evaluation, including
designing the evaluation, collecting and analyzing data, and reporting results. To help
illuminate key steps in the process, we use fictitious school—but "real life"—examples of
districts throughout the handbook. Two of the examples are extended: The Ringwood
School District's experience establishing evaluation plans is discussed in chapters 3
through 6, and the Kramer School District's work in designing and conducting an
evaluation is discussed in chapters 6 through 10. Finally, appendices at the end of the
handbook suggest printed and electronic resources that may be useful to you in planning
and conducting evaluations of professional development programs, and the glossary
defines terms that are commonly used to discuss evaluation.
Using this handbook will help users understand the key steps in the design and
implementation of a district evaluation of professional development. Throughout this
handbook, we will compare this process to that of planning and going on a school field trip.
We describe the following key steps:
- Understand the Journey: Describe the program to be evaluated
- Decide Who Will Come on the Trip: Identify stakeholders and involve them in the evaluation
- Determine Your Destination: Establish your evaluation goals and objectives
- Plot Your Course: Write your evaluation design
- Gather Information: Identify your data sources
- Gather Information: Choose your data collection methods
- Gather Information: Create and use data collection instruments
- Understand What Happened: Analyze your data
- Tell the Story: Interpret and report your results
This report is available in its entirety in the Portable Document Format (PDF), which many find convenient when printing.
Latest Research
Browse by Topic:
PDF
Printer-friendly summary
