About 10 years ago, I was part of a scientific team that reviewed the literature on the effectiveness of firearms legislation for reducing violence. Our team, a non-federal task force convened by the Centers for Disease Control and Prevention, published findings in 2003 and 2005. We concluded—as New York Times columnist David Brooks recently mentioned—that there was no way to know whether any of those laws are effective. A panel convened by the National Research Council reached the same conclusion.

Our team examined seven types of gun laws, from bans on particular kinds of weapons to those that mandate easy access to concealed-weapons permits, sometimes called “right-to-carry” laws. Firearms restrictions are advocated to reduce the lethality of violence; right-to-carry laws are advocated as a means to deter criminals. But such laws do several things simultaneously. While intended to deter would-be offenders, right-to-carry may also spark an arms race that leads criminals to carry ever more dangerous firearms. And they also make it easier for conflicts to quickly escalate in lethality. Put such different effects together and it is hard to confidently predict the net benefit or harm.

Why can’t we tell if these laws increase or reduce violence?

There are two main reasons. First, we don’t do controlled experiments and randomly assign some people to be subject to gun laws. Instead, laws are passed in particular places at particular times. But the jurisdictions that pass such laws are different from those that don’t, so simple cross-sectional comparisons—which are very tempting—are not informative about the effects of those laws. Such laws aren’t passed at random times, either, so that simple before-and-after comparisons aren’t much better. Instead, convincing studies of the effects of such policies require hard thinking, good data, sophisticated analyses, and lots of peer review and replication. Even then, the results often prove inconclusive.

Second, U.S. crime data are terrible. Homicide data from public health systems are good, because almost all deaths in the U.S. are recorded. A few studies capitalized on them; results were mixed.

To look at violence generally, researchers rely on crime data. Our main source is the FBI’s Uniform Crime Reports (UCR), based on aggregate reports to the FBI from the more than 18,000 police departments in the country. Across jurisdictions, the variability in what is reported renders comparisons almost meaningless, especially at the county level. Criminologists—who agree on little else—have deemed using the UCR data to rank cities as “invalid, damaging, and irresponsible.” Reporting is also remarkably incomplete, not to mention the all-too-common claims of jurisdictions underreporting crime. (Our other primary other data source, the National Crime Victimization Survey, is nationally representative but too small to disaggregate to jurisdictions.)

Some statistically sophisticated studies have tried to estimate the effects of gun laws on violence, using UCR data. But in the end, you cannot build strong conclusions on unreliable data. The nascent National Incident-Based Report System would be much more reliable but is more expensive. The mission of the Bureau of Justice Statistics is to produce reliable criminal justice statistics, but it is underfunded. Over the past decade, the National Institute of Justice has tried to improve the UCR data, but these are baby steps. It is long past due for the United States to set up more valid and reliable crime reporting systems.

Meanwhile, in the absence of definitive evidence, what should we conclude about gun laws? Personally, I don’t see any reason not to ban assault weapons and large-capacity magazines. Isn’t their only purpose to kill many people quickly? For most other weapons restrictions, the genie may be out of the bottle. So many guns are already in circulation that bans now have little chance of appreciably affecting criminal violence. Surprisingly, perhaps, the number of households with firearms may be declining.

What about right-to-carry laws and other attempts to allow more citizens to arm themselves for defensive purposes? As explained above, I don’t know whether such laws deter more everyday violence than they facilitate. I am confident that they would not deter mass shootings by assailants who expect to be caught and/or killed. Whether many armed civilian moviegoers would have been able to stop the latest shooter without harming many others in the crossfire is a more difficult question. I remain skeptical.