Earlier today, the federal government released the 2024 scores from the National Assessment of Educational Progress (NAEP). NAEP is the only nationally comparable measure of student achievement that is reported for every state on a regular basis, but comparing states’ NAEP scores is misleading for many purposes because states serve very different student populations. For example, more than 20 percent of children live in poverty in Alabama and Mississippi, compared with less than 10 percent in New Hampshire and Vermont.
For nearly 10 years, the Urban Institute has published adjusted scores that capture how well students in each state score on the NAEP compared with demographically similar students around the country. We determine these adjustments by calculating how each individual student who takes the NAEP scores relative to students nationwide who are the same gender, age, and race or ethnicity and have the same free and reduced-price lunch receipt status, special education status, and English language learner status.
We calculate adjusted scores for 2024 using our analysis of student-level data from 2022 and applying it to the unadjusted 2024 scores released earlier today. Our prior research has shown that this is a reliable way to adjust the state-level scores before student-level data are available.
These adjustments have important limitations, including challenges with accurately measuring income across different states and differences across states in the implementation of programs like special education. But the adjusted scores come closer than the unadjusted scores to capturing the relative effectiveness of state policies.
2024 Demographically Adjusted NAEP Scores
METHODOLOGY
Because we do not have student-level data for the 2024 NAEP administration, we perform a pseudo-adjustment using the most recent student-level data. We do this by applying the 2022 adjustment—the difference between unadjusted and adjusted scores in 2022—to the 2024 state average scores as released by the National Center for Education Statistics. This method assumes that a given state’s underlying demographics have not changed substantially in the two years since the previous test. A validation analysis using the 2015 and 2017 data showed that an adjustment like this produces estimated adjusted scores that are highly correlated (r = 0.97–0.98) with the actual adjusted scores.
For the 2022 demographic adjustment, we use the estimated relationship between test scores and student demographics to calculate a predicted score for each student who took the NAEP. The state’s adjusted score is the difference between each student’s actual score and their predicted score (which tells us how well they scored relative to students nationwide with similar characteristics), averaged to the state level.
The demographics we use for the adjustment include gender, age, race or ethnicity, receipt of free and reduced-price lunch, special education status, and English language learner status. Because some schools have adopted universal free school meals under the Community Eligibility Provision, and because nearly all students had access to free school meals during the pandemic, reporting of free and reduced-price school lunch eligibility is highly variable across states in 2022. To account for this, we use an imputed school lunch measure for students enrolled in schools that report offering universal school meals. This imputation methodology is not perfect but does improve the correlation between the state average free and reduced-price lunch rates and an income measure from (more accurate) US Census Bureau data from 0.59 to 0.71. The methodology for this imputation is further described in the America’s Gradebook Technical Appendix.