Parents, district leaders, and policymakers often use educational “outcome measures” such as test scores and graduation rates to make high-stakes decisions. But these measures often reflect underlying differences in income, opportunity, and the effects of structural racism.
Given what we know about a school’s student demographics and social characteristics, what outcomes would we expect that school to achieve, on average? We used regression models to understand a school’s actual performance relative to its expected performance.
Regression analysis is a simple—albeit imperfect—method we can use to evaluate outcomes across geographies, schools, classrooms, or students. By holding constant background characteristics that differ among individuals or groups, we can compare similar cases with each other fairly. Another analytic approach is to use descriptive data to compare various subgroups (e.g., racial and ethnic groups, disability status, limited English proficiency status, migrant status, foster care status, homelessness, military affiliation, and others). But the approach of disaggregating by every group can result in an overabundance of data that makes it hard to draw conclusions. Regression analysis can combine data for all subgroups into a single number using weighted averages and allows us to compare the difference between a school’s expected and actual performance. We demonstrate with publicly available data for high schools in Buffalo.
We examine graduation rates, test scores, and SAT and ACT participation among Buffalo high schools using data from New York State. In calculating the “expected” versions of these outcomes, we incorporate the share of students who are economically disadvantaged, the share of students in each racial or ethnic group, the share of English-language learners, the share of students with disabilities, the share of migrant students, the share of students experiencing homelessness, the share of students in foster care, the average eighth-grade test scores in the district the year before that cohort of students started high school (to capture their academic preparation), and an indicator for charter schools.
Some subgroups may, on average, perform worse than others on our examined outcomes. These differences are not because students from lower-performing demographics are less capable. Instead, the systems that serve these students may be inequitable in the resources provided and the standards set (standards that have often been designed for middle-class white students). For example, poverty is systematically linked to graduation rates, as shown in the figure, as well as other outcomes. Schools that help students achieve desirable outcomes despite these inequities may have honed strategies that other organizations could apply to promote equity in an unequal system.
The figure below shows how actual graduation rates compare with predicted rates for Buffalo’s traditional public and charter schools. The schools are organized by how much they overperform (or underperform) their predicted rate. Compared with the rest of New York State, Buffalo serves more students who are economically disadvantaged and more students of color students who, on average, face greater obstacles to graduation than their white and economically advantaged peers. Here, we see that most high schools in Buffalo perform slightly better than expected.
In an ideal world, regression adjustment would not be necessary. Students would have access to the same resources and opportunities regardless of their background, and any adjustments for demographics and social characteristics would not change the results much. Regression-adjusted measures are not a perfect solution, and they should not fully replace raw metrics in analysis. Regressions, however, incorporate important contextual information in a compact way to provide insight for parents, policymakers, and school administrators.
The Urban Institute has the evidence to show what it will take to create a society where everyone has a fair shot at achieving their vision of success.