“Free college” and “student loan crisis” make for catchy campaign slogans and headlines, but the focus on affordability and debt may be distracting from more urgent problems in higher education.

Quoting Jack Lemmon’s character in the 1979 film The China Syndrome, Spencer Foundation president Michael McPherson said Thursday that we need to “turn off the goddamn alarm so we can think.” There has been so much panic about debt and college affordability, McPherson said, that it has obscured the real failures of higher education in the United States: faltering completion rates and lengthening times to degree.

McPherson makes this argument in a new book, coauthored by William Bowen, president emeritus of The Andrew W. Mellon Foundation and Princeton University, which he discussed with a panel of higher education experts at an Urban Institute event Thursday. The national conversation, they all said, needs to move beyond mountains or seas of debt to the real barriers.

“Low completion rates among disadvantaged populations have to be addressed effectively in order to address inequality both of opportunity and of outcomes,” McPherson said.

While there is some evidence linking affordability to completion—after all, students who can’t afford college certainly won’t complete it—Jessica Howell, executive director of policy research at the College Board, said that these examples are indicative of a larger problem. She talked about a small college in Kentucky where all the students receive financial aid but the completion rate is still only 60 percent. That low rate, she said, can’t be due to affordability.

The issue of affordability is a heated one. Much of the mainstream dialogue in the media and on the presidential campaign trail has focused on lowering the cost of tuition or making college free. But research shows that free college proposals may not benefit those who need it most, and, as McPherson pointed out, “Free is not nearly cheap enough for low-income people.”

Additionally, low-income college students, who often have grown up with less access to quality academic opportunities, tend to need more support to succeed in college. This means that it costs schools more money to educate low-income students well, but ideas about where that money might come from are typically left out of discussions about affordability.

“If we put all our chips on affordability, we’re going to wind up squeezing the money that’s available for education, and that’s a worry,” McPherson said.

Many low-income students face barriers to graduation not because of their academic performance, but because they don’t receive adequate advising and end up not meeting the prerequisites for graduation. The growing time to degree is a real problem, panelists said, but one that is underappreciated. Edward Montgomery, dean of the McCourt School of Public Policy at Georgetown University, joked about one state legislator, who upon hearing testimony about six-year graduation rates replied that he didn’t know we had six-year colleges.

Though this quip got a laugh from Thursday’s crowd, it highlighted a very real predicament for higher education experts. While they are worried about graduation rates and time to degree, the public dialogue remains focused on loans and free college, partially, panelists said, because of a misguided notion about what the typical college student looks like. Urban Institute senior fellow Matt Chingos said the popular narrative focuses on the 22-year-old liberal arts major living in his parents’ basement, unable to find a job to pay off his loans—but the real problems arise for the 30-year-old single mother who enrolls in school to better her employment prospects, drops out, and ends up with $5,000 of debt and no degree.

“The problem for that person isn’t that they’re living in their parents’ basement and don’t like their loan payment, it’s that they really don’t have a way of paying back their loan,” Chingos said.

Changing the public’s appetite and getting into the nitty-gritty of higher education will mean overcoming the public’s tendency to point the finger of blame at someone. It is easier, Montgomery noted, for people to blame overpaid college presidents or greedy colleges. But the issue, he said, deserves more time and patience than that.

“We don’t talk about the debt crisis in relation to the income crisis,” he said. “But that’s a harder story to tell than, ‘it’s the college who did it to you.’”