The availability, reliability, and quality of criminal legal system data is a persistent issue. There are few federal regulations dictating what data law enforcement agencies must collect and how these data need to be maintained and shared with the public. The Federal Bureau of Investigation collects data from local agencies through the National Incident-Based Reporting System, but law enforcement agencies are not required to report to that system: in 2022 only 66 percent of the US population was covered by NIBRS-reporting law enforcement agencies. Although that has increased to 82 percent as of 2024, there remains a lack of reporting from agencies serving smaller populations and a lack of historical data from before 2024. This means people often need to turn to local law enforcement data from agency or city and county websites to understand the criminal legal system, but the lack of data standards means there are no one-size-fits-all instructions for assessing and analyzing them. Even when data are available, issues with quality and completeness are common.

Data systems in the courts exhibit similar issues. Many court systems only allow you to search for specific case numbers or individuals and do not provide a way to download data in bulk for analysis. Others charge fees for access to their full datasets, creating a barrier to access. Moreover, even if you are able to access court data, because court data collection is not standardized, court systems may not have data on all the fields of possible interest in their data systems.

The Catalyst Grant Program helps organizations use data and technology to advance racial equity and reform in the criminal legal system, supporting their efforts to access, analyze, and share their local criminal legal system data. Publishing data is paramount to increasing transparency in the criminal legal system so organizations can leverage data in their approaches to reform. Because of the infrastructure problems outlined above, however, data can be hard to find, hard to access electronically, and incomplete and inconsistent, hindering investigation of criminal legal system issues. The work of several Catalyst grantees has shed light on the value of access to quantitative criminal legal system data and of digging into the context of those data by talking to the people who produce and are affected by them.

Comparing Multiple Data Sources Revealed Discrepancies in Traffic Stop Reporting in Chicago

Equiticity, with Impact for Equity, both nonprofits in Chicago focusing on racial equity in the city, sought to make traffic safety in Chicago racially equitable through an advocacy campaign to limit police interactions with Chicagoans. Their campaign was informed by public safety data from the law enforcement agencies operating in the city, including the Illinois Department of Transportation and the Chicago Police Department. Although the latter is required to report data to the state, the number of stops, the number of searches, and the amount of contraband recovered were much lower in the state records than the city police records, with stops sometimes off by tens of thousands. These discrepancies likely owe to unclear reporting requirements, inaccurate reporting, potential duplicate entries, and failure to report some interactions across agencies (PDF).

If Equiticity and Impact for Equity had only collected Illinois Department of Transportation data, they would have only seen the much smaller number of stops. Collecting data from multiple sources, combined with a working knowledge of traffic enforcement in the city, shed light on discrepancies between these datasets that may reflect undercounting in the state data. In determining how to handle the inconsistent findings, Equiticity and Impact for Equity chose to report the most conservative estimates (the fewest stops as found in the state data and highest rates of found contraband as found in the police data). Even with these conservative estimates, they demonstrated racial inequities in traffic stop enforcement in Chicago. Their analyses allowed them to raise awareness not only of those racial inequities but also of the discrepancies in different sources of traffic stop data.

Lived Experiences of Human Trafficking Survivors Suggested Undercounting of the Criminalization of Sex Trafficking Survivors in New Orleans

Polaris, a nonprofit organization working against human trafficking, sought to improve outcomes for survivors of sex trafficking in New Orleans. Whereas many jurisdictions have stopped arresting for prostitution, New Orleans has continued to arrest and criminally charge sex trafficking victims, particularly women and girls of color. After procuring arrest and prosecution data from the New Orleans Police Department, Polaris found that the low number of arrests in the data didn’t match what they had heard from human trafficking survivors.

To understand this discrepancy, they spoke with court-system representatives and prosecutors doing work related to sex offenses. Through these conversations, they uncovered data on citations that had been issued by law enforcement and recorded in court data systems. This meant that many of the interactions human trafficking survivors had with the criminal legal system were recorded as citations in a different data system than arrests, resulting in significant underreporting of the criminalization of sex trafficking survivors in New Orleans. Through their relationships with advocates and criminal legal system staff, Polaris analysts were able to ascertain that the data they initially received didn’t reflect people’s experiences on the ground. By uncovering a more complete picture, Polaris was able to identify a range of reforms that would improve survivors' experiences and support.

To Get the Full Picture, Pair Criminal Legal System Data with Other Data Contextualizing People’s Experiences

Polaris’s and Equiticity’s work show the potential harm of taking findings from any one source of administrative criminal legal system data at face value without deeper investigation. While administrative data can reveal important patterns, they usually don’t tell the whole story. If the organizations had not taken the time to dig deeper, they would have drawn incomplete or incorrect conclusions.

Anyone analyzing criminal legal system data should take the following four critical steps to triangulate a more accurate picture:

• Seek out and compare multiple sources of data. Supplemental data can be obtained by filing Freedom of Information Act requests and by simply asking criminal legal system agencies.
• Have people and groups who are closest to the issue use people’s experiences to assess whether the data reflect the ground truth. Having informal conversations and reviews with service providers and conducting community surveys and focus groups can fill in the gaps and illuminate the human impact of the issue.
• Seek to understand how criminal legal system agencies record race and ethnicity data and the implications of those choices. This is important because the lack of data standards and issues with data quality can have important implications for understanding racial disparities in the criminal legal system. For example, some law enforcement agencies record people’s race and ethnicity based on officers’ perceptions.
• Maintain ongoing communication with data providers, including asking questions about the data and what might be missing. This connection is valuable in all cases, but particularly when findings are unexpected. Ongoing dialogue with law enforcement, prosecutors, and municipal court workers can uncover the causes of inconsistencies. In addition to providing access to data in the first place, these relationships can make requests for additional data and advocacy for reforms more successful.

While it can be frustrating to receive data and realize they may be incomplete or don’t look as expected, it does not mean that those data are unusable or there is not a story to tell. Sharing what is available but noting the limitations can start an important conversation about the implications of poor-quality data and how to improve them. Even in cases where data are completely unavailable, reporting that can be worthwhile. Informing the community that data don’t exist or aren’t accessible can drive policy changes to fill those gaps.

Catalyst grantees have often taken these approaches when dealing with difficult data. Organizations should take the lessons Equiticity and Polaris learned to heart in their approaches to criminal legal system reform. Being well attuned to the voices in your community of focus and diligent in data collection can help ensure your findings accurately represent the issues and inform approaches to addressing them.

The Catalyst Grant Program is a collaboration between the Urban Institute and the Microsoft Justice Reform Initiative to help nonprofit organizations use data and technology to advance racial equity and reform in the criminal legal system. Visit the Catalyst Grant Program Insights page for more resources and stories about the grantees.