Over the past several weeks, social science has been rocked by an alleged case of data fraud by Michael LaCour, a political science graduate student at UCLA.
LaCour’s research, coauthored with Donald Green of Columbia University, purported to experimentally demonstrate that short conversations with canvassers can dramatically change a voter’s opinion on same-sex marriage. The impact was particularly strong if the voter spoke with a gay canvasser. LaCour’s impressive data visualization and exciting empirical result propelled a flurry of media attention. Science published his work, but has subsequently retracted the article. Although investigations are still ongoing, the evidence seems to suggest that LaCour fabricated data and lied about the financial support he received.
However the investigations turn out, the fraud accusations do offer a few lessons for both consumers and producers of social science research.
Consumers of research—educated individuals who care about evidence-based policymaking—should try to follow these guidelines when reading about social science research:
- Understand that news articles only present the tip of the iceberg. The handful of data points that make it into a given article or that go into visuals can never give you the complete picture. Full research reports often contain important caveats and conditions.
- Appreciate the breadth of most research literature. When you read about social science research, it is typically the “exciting” new study that is being reported. Consumers of research need to appreciate that every study lives in a much broader ecosystem of related studies, some of which may have contradictory results. Try to learn what other studies have to say on the same research question and what might be driving differences between study results.
- Understand the data. The key to understanding any study result is to understand the data it came from. Whenever you read about a new study, pay close attention to what the article or post has to say about how the data were collected.
Producers of research also have a responsibility to make a positive contribution to public discourse. At the Urban Institute, we call this “elevating the debate.” Researchers can elevate the debate by doing the following:
- Replicate, replicate, replicate! No science, least of all social science, has data that are so perfect that no replication efforts are required. Even honest, well-designed studies may vary in their results, so replication (perhaps using slightly different methodological approaches in each case) is essential for triangulating the answer.
- Reach out to the authors. The political scientists who uncovered the problems with LaCour’s work were successful in part because they were willing to pick up the phone and call not only LaCour, but also the staff at the survey firm that LaCour allegedly used to conduct his study. Researchers should be open about their research, and maintaining dialogue is the best way to understand the work that went into a particular study.
- Share your findings. Researchers are online just like everyone else, and a good way to get reactions to your work is to tweet it, blog about it, or share it. The Internet allows you to crowdsource peer review, if you know how to use it effectively. LaCour’s principal critics posted their analysis of LaCour’s work online, allowing a wide audience to test the veracity of his claims and explore new angles of the question.
Problems inevitably arise in real world research, ranging from outright fraud to honest mistakes. But both consumers and producers of research can take steps to police these problems.