Recently, story after story has emerged documenting long-standing evidence of racial bias in home appraisals across the country. Because appraisals are a key component of the mortgage origination process, they have contributed to racial home value gaps and wealth disparities. Now, federal and state policymakers are seeking solutions and advancing legislation to ban such discrimination.

Automated valuation models (AVMs), a financial technology that applies statistical algorithms to a database of housing activity to calculate a home’s value, have been raised as a way to mitigate racial bias because they reduce reliance on potentially biased human opinions. Analysis suggests AVMs tend to produce smaller bias than appraisals.

But AVMs may not be a surefire solution to fully closing racial inequities in the home appraisal process.

In a new analysis, we find a greater AVM error as a percentage of the property’s sale price in neighborhoods where Black residents make up the majority than in neighborhoods where white residents do, even after controlling for factors like lower home values and worse property conditions. These findings affirm recent calls by federal agencies to address potential bias in the use of AVMs (PDF) to prevent any such biases from becoming hardwired into the technology.

Property condition may contribute to some of the racial disparities in AVM error

When we first investigated AVM bias, we did not find evidence that AVMs systematically undervalue homes in majority-Black neighborhoods. Rather, we found that the degree of error—the inaccuracy of the estimate compared with the sale price, whether too high or too low—was greater in majority-Black neighborhoods, which have systematically lower home values. That, in turn, magnifies AVM error.

But the root causes of that error remain somewhat unclear. Do AVM disparities reflect technological issues, such as data omission or modeling technique? Or are they caused by the role of systemic racism in the determination of home values?

With new property-level data on property condition from the property intelligence firm Cape Analytics, we began answering that question. In the markets we assessed, which each have a significant Black population, we found majority-Black neighborhoods had a larger share of properties in poor condition, whereas majority-white neighborhoods had a larger share of properties in fair or good condition. After controlling for neighborhood conditions, property differences within neighborhoods, and turnover rates, we find properties in poor condition were associated with a higher percentage magnitude of AVM error relative to those in fair or good condition.

In Atlanta and Memphis, Majority-Black Neighborhoods Have More Properties in Poor Condition than Majority-White Neighborhoods

Source: Urban Institute calculations using data from the 2018 American Community Survey, CAPE Analytics, and a major property records provider.
Notes: CBSA = core-based statistical area; ECR = exterior condition rating.

Though a portion of the percentage magnitude of AVM error is attributable to property condition, simply adding it to our model does not explain AVM error. But deeper analysis suggested that our method of analysis—an ordinary least squares (OLS) regression—was not appropriate because the structure of the underlying data violated some of the key assumptions of OLS regressions. To address this modeling mismatch, we turned to a tool based on artificial intelligence.

Machine learning analysis suggests historic racism may be a key factor behind the AVM error

Machine learning (a subfield of artificial intelligence) gives us additional tools to address the key shortcomings of an OLS regression model, including handling interrelated variables and nonlinear relationships. Our LightGBM model, together with additional data on property condition, can help us more accurately assess the percentage magnitude of AVM error. In fact, using this machine learning model reduced the root mean square error for predicted AVM error from our original model by 5.8 percentage points, from 46.2 to 40.4 percent.

Additional data on property condition and our more sophisticated machine learning model also more accurately assessed the underlying contributors. Using a Shapley Additive Explanations (SHAP) test, we calculated the importance of each variable by comparing the predictiveness of the model with and without it.

The SHAP values indicate that compared with majority-white neighborhoods, predicted AVM error in majority-Black neighborhoods is greater, even after accounting for key variables like home value and applying more accurate modeling. In the chart below, properties in majority-Black neighborhoods are reflected by purple dots. In the Black neighborhood row, we see a significant amount of majority-Black neighborhoods on the right side of the vertical line, which signals that the effect of a property’s location in a majority-Black neighborhood is positive, thus boosting the percentage magnitude of AVM error.

This suggests that even though an AVM algorithm does not explicitly factor in a neighborhood’s majority race, it still can produce racial disparities.

To quantify the magnitude of racial disparities, we employed a synthetic control method. We found that if 60 percent of properties currently in majority-Black neighborhoods “move” to majority-white neighborhoods while keeping all other property attributes unchanged, the predicted magnitude of AVM error could decline from 36.2 percent to 31.8 percent—a 4.4 percentage-point difference. And if all properties currently in majority-Black neighborhoods “move” to majority-white neighborhoods, the predicted AVM error could decline by 5.0 percentage points, giving us an upper-bound estimate of the racial disparity in AVMs.

Our findings underscore the role of historic racism in AVM estimates, even with data and modeling improvement. Continued exploration of new techniques in data and modeling will be necessary to further identify the underlying causes of racial disparities in AVM error.

How can policymakers address AVM error and appraisal bias?

To ensure everyone can benefit from the American dream, eliminating AVM error is critical.

Federal policymakers have begun to act on this issue. Recently, the Biden administration’s Interagency Task Force on Property Appraisal and Valuation Equity (PAVE) released an action plan for addressing racial disparities in home valuation. In addition to addressing appraisal bias, the report also highlighted the need to ensure that AVMs do not rely upon biased data that could replicate and reinforce past discrimination.

And earlier this year, the Consumer Financial Protection Bureau took early steps to prevent algorithmic bias in AVMs used for underwriting purposes. Data-informed regulation is one way to mitigate disparate effects from AVM usage.

Meanwhile, understanding how property condition potentially addresses AVM accuracy is key to Fannie Mae and Freddie Mac’s new plans to close racial homeownership gaps.

More broadly, our research also has important implications as the federal government seeks to expand its use of automation and artificial intelligence in other sectors. Racially inequitable inputs, partially rooted in historical racism, can produce racially disparate AVM error. Understanding use cases of artificial intelligence, such as that of AVMs in the appraisal process, can help the federal government protect historically marginalized communities in other policy areas, too.