The voices of Urban Institute's researchers and staff
February 14, 2018

Misleading data and visualizations

February 14, 2018

Have you ever looked at a graph and wondered, “What is this trying to tell me?” Or flipped through a report and examined a chart, only to wonder whether the author is trying to hide information?

The questions we ask when we examine data and data visualizations can be crucial to how we understand complicated issues and how we communicate research.

During a recent visit to the Urban Institute, Alberto Cairo, the Knight chair in visual journalism at the University of Miami, spoke about his concept of “graphicacy,” or visual literacy: how we understand data when it is presented in visual means, such as graphs, charts, and infographics. (A recording of Cairo’s talk and a link to his slides can be found on Urban’s events page.)

Cairo believes a literate, numerate, and graphicate citizenry is the best antidote for a world where, in his view, misleading charts, graphs, and data maps run rampant.

Drawing on politics, music, science, health, and justice policy, Cairo showed numerous examples of data being displayed in ways that intentionally or unintentionally mislead the reader.

Take this pair of maps from Andrew Gelman and Deborah Nolan’s book, Teaching Statistics: A Bag of Tricks.

image from textbook showing maps of kidney cancer death rates

Looking first at the map on the left, you might be inclined to draw a conclusion to explain why the darkened counties—which appear to be in more rural areas—have lower rates of kidney cancer death rates.

Perhaps people who live in rural areas eat different kinds of food? Maybe they are exposed to less pollution than people living in urban areas? Or maybe there are other demographic, biological, or genetic factors at play?

But if you then examine the map on the right, you see that counties with the highest rates of cancer deaths also appear to be rural counties. If your explanation for the first map holds true, can it be true for the second map?

Or is there a simpler explanation? Maybe these counties have fewer people and therefore a small change in the number of people who die from cancer affects the mortality rate more than in more densely populated areas?

Cairo argued that as consumers of data, we need to be more discerning in the conclusions we draw and the statements we make when viewing any data visualization. We should take an additional moment to consider whether the visual makes statistical sense and explore the underlying data. Are the data represented accurately, and are enough data being shown?


Reading charts

  1. Is the designer using the right data and disclosing its origin?

  2. Are you reading too much into the graphic?

  3. Is the data represented accurately?

  4. Is the graphic showing an appropriate amount of data?

  5. Is uncertainty relevant? If so, is it revealed?


Cairo also encouraged thinking about whether uncertainty is relevant to the story being shown. Is it clear, is it obvious, is it important?

When election polling results tell us that 45.3 percent of respondents say “No” to a question and 44.5 percent say “Yes”—and the poll’s margin of error is 2.95 percent—we need to think twice before saying the group is leaning toward “No.”

At Urban, we strongly believe that data, facts, and evidence matter. They help us understand complex national and local economic and social problems. Our goal is to use evidence to elevate the debate and to bring data to issues that matter to us. As Cairo writes in his book The Truthful Art, “We should certainly do good with data, but only after we’ve thoroughly made sure that our data is good.”

Alberto Cairo gives a talk at the Urban Institute. Photo by Jon Schwabish.

SHARE THIS PAGE

As an organization, the Urban Institute does not take positions on issues. Experts are independent and empowered to share their evidence-based views and recommendations shaped by research.