Are there important determinants of teacher productivity that are not captured by teacher credentials but that can be measured by subjective assessments? And would evaluating teachers based on a combination of subjective assessments and student outcomes more accurately gauge teacher performance than student test scores alone? Using data from a midsize Florida school district, this paper explores both questions by calculating teachers' "value added" and comparing those outcomes with subjective ratings of teachers by school principals. Teacher value-added and principals' subjective ratings are positively correlated and principals' evaluations are better predictors of a teacher's value added than traditional approaches to teacher compensation focused on experience and formal education. Also, teachers' subject knowledge, teaching skill, and intelligence are most closely associated with both the overall subjective teacher ratings and the teacher value added.
The text below is an excerpt from the complete document. Read the full report in PDF format.
Recent research consistently finds that teacher productivity is the most important component of a school's effect on student learning and that there is considerable heterogeneity in teacher productivity within and across schools (Rockoff 2004; Hanushek et al. 2005; Rivkin, Hanushek, and Kain 2005; Kane, Rockoff, and Staiger 2006; Aaronson, Barrow, and Sander 2007). Relatively little is known, however, about what makes some teachers more productive than others in promoting student achievement.
Older cross-sectional studies of educational production functions found that the characteristics that form the basis for teacher compensation—graduate degrees and experience—are at best weak predictors of a teacher's contribution to student achievement (Hanushek 1986, 1997). More recent estimates using panel data have determined that teacher productivity increases over the first few years of experience (Rockoff 2004; Clotfelter, Ladd, and Vigdor 2006; Jepsen 2005; Rivkin, Hanushek, and Kain 2005; Harris and Sass 2008; Aaronson, Barrow, and Sander 2007), but little else in the way of observed teacher characteristics seems to consistently matter.1 In short, while teachers significantly influence student achievement, the variation in teacher productivity is still largely unexplained by commonly measured characteristics.
One possible explanation for the inability of extant research to identify the determinants of teacher productivity is that researchers have not been measuring the characteristics that truly determine productivity. Previous studies in the economics of education literature have focused primarily on readily observed characteristics like experience, educational attainment, certification status, and college major. According to recent work in labor economics, however, personality traits may also play an important role in determining labor productivity (Borghans, ter Weel, and Weinberg 2008; Cunha et al. 2006; Heckman, Stixrud, and Urzua 2006).
Unraveling the factors that determine teacher productivity could yield valuable insights into the most appropriate policies for selecting and training teachers. If teacher productivity is affected primarily by personality characteristics that are measurable ex ante, they could be used as signals to identify the most desired candidates in the hiring process. If, however, valuable teacher characteristics are malleable, determining which teacher characteristics have the greatest impact on student learning could also inform the design of preservice and in-service teacher training programs.
Understanding the factors that affect teacher productivity and the degree to which these determinants are measurable would also inform current policy debates over how best to evaluate and compensate teachers. If it is not possible to measure the characteristics of teachers that determine their productivity, then ex post evaluation of teachers based on their contributions to student achievement or "value added" may be optimal (Gordon, Kane, and Staiger 2006). Currently, many school districts are experimenting with such "pay for performance" systems (Podgursky and Springer 2007), although some are concerned about the precision of these measures, their narrow focus on student test scores, and the fact that they can be calculated for only a small proportion of teachers. Alternatively, if there are particular teacher characteristics and behaviors that influence their productivity and are observable, but perhaps not easily quantified, then reliance on supervisor evaluations and other more subjective assessments may be advantageous. There is movement toward granting principals greater authority in hiring, evaluation, and retention of teachers both through the creation of independent charter schools nationwide and through decentralization reforms in public school districts such as New York City. The downside of subjective evaluations by principals is they may be affected by personal bias toward factors unrelated to productivity and some principals may simply be poor judges of teacher productivity.
To address the related issues of the determinants of teacher productivity and how best to evaluate teacher performance, we analyze the relationship between principal evaluations of teachers and the contribution of teachers to student achievement or teacher value added. Like other recent work, we examine the relationship between teacher characteristics (including typical teacher credentials and personality traits observed by principals) and both subjective and value-added measures of contemporaneous teacher performance. We also go beyond the existing research, however, and compare the ability of past value-added measures and principal ratings to predict future teacher value added.
We begin by estimating a model of student achievement that includes fixed effects to control for unmeasured student, teacher, and school heterogeneity. The resulting estimated teacher fixed effects are our measure of teacher value added. We then analyze the simple correlation between principals' subjective assessments and teachers' value-added scores. We follow that process with a multivariate analysis to examine whether principals are better at judging teacher productivity than traditional approaches to compensation that focus on experience and formal education. Next, we look in detail at specific teacher attributes and how they relate to both principals' overall evaluations of their faculty members and estimates of teacher value added. Finally, we compare the ability of principal ratings and past teacher value-added measures to predict future teacher value added.
In the next section, we describe the small literature on principal evaluations of teachers and their relationship with value added and follow with a discussion of the data used for our new analysis, including how we conducted the interviews with principals and our method for estimating teacher value added. In the concluding section, we discuss our empirical results and possible policy implications.
(End of excerpt. The full report is available in PDF format.)