MR. ROBERT REISCHAUER: Okay, why don't we get under way?
My name is Bob Reischauer, I'm the president of the Urban Institute, and I want to welcome you all to this First Tuesday gathering. First Tuesdays are opportunities, which we have once a month, to share interesting and policy-relevant Urban Institute research with the broader community and to stimulate discussion and get the reactions of others who are doing similar types of work and to hear the perspectives from those on the front lines who use such research and analysis. The topic today couldn't be more important: education reform in math and science. We all know the importance of math and science skills to functioning adequately in our ever-changing economy and surviving in our ever-more-complex society.
We also know how poorly American students do with respect to mastering math and science skills that are needed, and we also know that there's been very little progress on this front, if we look across time in the United States, and we're embarrassed by the fact that so many other nations in the world seem to do a better job.
This is notwithstanding the fact that there are many, many efforts in the United States to address this problem and we have been trying, although not too successfully, for a number of years to improve the situation.
There has been really a myriad of efforts and what we're going to hear something about today is an Urban Institute review and evaluation of the science and math curriculum and professional development strategies that have been tried and the extent to which they have increased student
achievement. Also, there is a AAAS [American Association for the Advancement of Science] review of systematic reform efforts highlighting critical factors in districtwide reform efforts. Both of these studies were supported by the General Electric Foundation, and we wish to thank them for their support and to acknowledge and welcome Kelly Wells, who is the program manager for U.S. education at the GE Foundation.
We have a particularly knowledgeable, informed panel, which you will hear from and in order, they are Roger Nozaki, who is the associate dean at Brown University for the college and the director of the Swearer Center for Public Service at Brown, which is an institution that supports student and faculty who are interested in community participation, activism, and other kinds of similar activity. He was the executive director of the GE Foundation and senior program officer of the Hitachi Foundation earlier in his career.
The second speaker will be Shirley Malcom, who is the head of the Directorate for Education and Human Resources Programs of the AAAS. She has many honorary degrees and has served on a number of boards, but more—more importantly was the 2003 recipient of the Public Welfare Medal of the National Academy of Sciences, which is the highest award that that organization bestows.
Third will be Toni Clewell, who is director for the Program for Evaluation and Equity Research at the Urban Institute within the education and policy center. Before coming to the Urban Institute over a decade ago, she was with ETS in the education policy research division and is the lead author on the study that I referred to earlier.
Last but not least is Michael Hickey, who is a professor and director of the Center for Leadership in Education in the College of Education at Towson University. He has a wealth of practical experience as a superintendent of schools in Howard County, Maryland, one of the high-performing school districts in this part of the country. Also he is the superintendent in St. Louis Park School system and the deputy superintendent in Seattle, Washington.
Moderating this skilled set of individuals is Kevin Finneran, who is no stranger to those of us at the Urban Institute. He is the editor-in-chief of Issues in Science and Technology, which is the quarterly journal put out by the National Academies.
With that, let me turn it over to you, Kevin, and look forward to hearing the discussion.
MR. KEVIN FINNERAN: Well, good afternoon. I'm just going to start off very briefly by reminding everybody how important this subject is. Just last week, the National Academy, National Academies, released a report called Rising Above the Gathering Storm, about basically the economic plight of the United States, what the future holds, what's important to us. And like many Academy studies, it involved the committee; it involved three Nobel Prize winners, three or four university presidents, the CEOs of three major companies—Intel, Lockheed Martin, so on—and what all of these people came to, and they looked at the future of the United States and what was most important, the first recommendation they made was we have to do something to improve the quality of K-12 science and math education, and for those of you who follow these type of august reports, this is not an unusual recommendation. And the people who serve on these committees know that they are not the first ones to think about this, but I think what they've come to recognize is that in spite of that, they haven't figured out how to do it yet. It's not a job perhaps for CEOs and Nobel Prize winners and college presidents. I think it's a job for the people like the ones we have on this panel who have looked much more closely about what works in classrooms, what works in schools and school systems.
So I think the nation's leaders—this report was done at the request of Senators Alexander and Bingaman, there have been briefings—committee members briefed Secretary of State Rice yesterday, they're going to be meeting with President Bush, people are interested. People want to know what to do and I think that we're fortunate to have the people we have here who can tell us something, what we've learned from some experience we've had. We've been trying to improve science and math education for a while now and we are learning something about how to do it better.
So with that, I'm going to turn it over to the panelists and they can tell you what they've learned and we'll have brief discussion then with the panel—among the panelists and then we'll have it open to questions.
So with that—
MR. ROGER NOZAKI: Thank you very much, Kevin. I want to just start out by echoing some of the sense of urgency that Kevin outlined in his remarks. I'm kind of putting an exclamation point on that with a couple pieces of data, and I know you're all very familiar with the data and hear the data every day, but just to cite a couple of the sets of statistics that I think remind us of that urgency in a little bit more specific sense.
We've seen the data that of every 100 students who start ninth grade, on average in the U.S., only 67 of them will graduate high school in four years. Of those original 100 in ninth grade, only 38 will go directly to college once they finish high school, and only 18 of those original 100 will earn their associate's degree in three years or their bachelor's degree in six after completing high school. And as we all know, the standards that we—we're not necessarily holding students to hugely demanding standards to begin with, and yet we're losing all these students at every step of the pipeline.
So we have at the end of this pipeline, only 18 of the original 100 students that have entered ninth grade. So what's happening to the other 82 students who have dropped out along the way or who have completed some stage of their education, but remain unprepared for the next stage in their life, whether it's education or work or something else?
If you look at the fourth grade math statistics, and we've seen a little bit of improvement at the fourth grade level, although the eighth and twelfth grade levels haven't been doing as well, but if you look at the fourth grade math scores, only about, just over 40 percent of white students are proficient or better in mathematics at the fourth grade, and only about 10 percent of African American students are proficient in math at the fourth grade, and again, we know that that's kind of the peak of math achievement for many students in the United States.
So again, if those were the numbers at fourth grade and that's the best we're doing, again, what happens to those students the farther they go on through the education continuum? So we're losing
thousands of students every year from the education system. What's happening to those students and what do we need to do in order to accelerate our progress on these issues? We really have to find ways to do a better job, to get to more students much more quickly.
I think all of the panelists today will make an argument, make the argument that what we need to do is to start and end with data. We no longer have the luxury of time or money to continually reinvent the wheel, to invest in solutions that come from some of these theories that aren't based on what we already know works or doesn't work. We don't have the luxury of time or money to continue investing in things and getting kind of a feel-good sense of whether it works or not, that that teachers really like the material or that students really like the material. We've got to know at the end of the day whether it's making a difference in those students or attention in their performance in these academic areas.
And we no longer have the luxury of time or money to continue investing in things that appear to be solutions that may—may improve the performance of targeted small numbers of students, but have no chance of being sustainable or scalable, that and when a grant ends, or that and when a charismatic leader leaves a school. We've got to find ways that can be sustained and expanded and scaled to reach the thousands of students that we're losing each year.
So what we need to do is start from the data. Unfortunately, as you look around the educational landscape, there isn't a lot of data that exists out there, and I think equally disheartening, there still isn't a whole lot of demand for that data out there. So you have educators across the country who are daily making decisions about curricula, about professional development, about school improvement strategies, about school reform, all very well-intentioned people, all very well-meaning people, all who are committed to doing the right things for kids, and yet they're only able to make decisions based on their gut instincts or based on textbook publisher pressure or what seems to sound good or look good rather than being able to work from a base of data around what really does work.
We also have beyond the school level, funders, foundations, others providing resources into the systems who are operating on that same kind of basis and we also have policy being made at all levels within the education system in the same kind of way.
So what this research that you'll hear about today does is try to pull together what we do know. We wish there was more data. We wish there was better data, but at least these publications provide
some guidance on what data exists. So we will ask your help today to get this information into the hands of all those people who are making decisions about schooling in this country, how to provide them the resources and the guidance so that they can make their decisions based on data, and we also ask your help today in increasing the demand for data, helping build a culture of data within
the education system. Helping build individuals' capacity and school systems' capacity to understand how to go about a rigorous process of seeking out data, analyzing that data, and making decisions based on that data in an ongoing kind of way.
There's a huge need in terms of the capacity of individual educators in every piece of the school system, the education system in the U.S., to help understand data and apply data in much more rigorous kinds of ways.
So I would argue that at the end of day, unless we take these steps forward, unless we build our capacity and our understanding of the data, unless we start applying these data in much more rigorous kinds of ways, we'll be condemned to continue the very, very slow rate of progress that
we see currently in the education system and will continue to condemn our U.S. students to a very bleak future in an increasingly demanding and global economy as we go forward.
Thanks.
MR. FINNERAN: Okay.
Next is Toni.
No, no, I'm sorry, Shirley.
MS. SHIRLEY MALCOM: Yes. Right, right order.
Thank you so very much. It's really a pleasure for me to be able to come here and to join this panel and especially to be able to follow Roger because he said things that I didn't have to say and therefore he gives me more time to say other things.
I want to acknowledge my colleagues, Joan Abdullah, who's here in the audience, and colleagues Katie Grogan and Darryl Chuben, who are not, who really helped to pull all these—this information together, and the many people, including Toni and Pat, who also participated in a lot of the discussions that we held around this. Because the work that we did in looking at the—looking at systems is very difficult work because the research base is not the greatest in the world.
One of the places that we decided to start was at a place that actually did have some data and that could then undergird the work that we would do in terms of trying to talk to many of the individuals who had been directly involved in the reform and try to bolster our understanding from the literature and from the discussions with people who had been working this area for so many years.
If you want to know what this report is about, I guess that the title says it all. We really are talking about the need for looking at the problems systemically. We are the ones who are kind of flying at 20,000 feet on this one. Toni's going to take you more to ground level.
But in this particular case, we wanted to look at the processes and the systems that were in place that actually could contribute to and support a large-scale system reform. We have had a lot—many examples of heroic efforts by individuals. Roger talked about the charismatic
leader or the special project. You all know about those—we can all start naming them—that would help to support learning or school improvement with amazing results, and in many cases, these include identifying schools with student populations that are usually characterized as having low performance, that some work that Toni did where they had made incredible achievements with regard to student performance in mathematics and science.
But the GE Foundation asked our team a different question and that is what can be done to support school systems in increasing achievement for all its students? So the title, as I said previously, are findings in rejecting any single magic bullet as the answer.
We turned to look to the National Science Foundation's supported urban systemic initiatives, both the urban systemic—USI, urban systemic initiatives—and the urban systemic programs to—as a way of finding systems where we knew that the evaluation base was sufficient to be
able to make a choice of programs that had, in fact, shown improvements across all student groups, and this was important in terms of looking at every student rather than just with some particular subgroups of students.
And the target for those efforts by the National Science Foundation were, in fact, systems—urban systems—that had high numbers of minority and low-income students and, therefore, they were pretty much smack in the middle of the every system kind of question.
They had been required to collect data, to monitor performance, along six parameters or drivers. In addition, there were external evaluations that were done and that were in place to look within and across the various sites. We didn't just go by the numbers, we actually looked at the reports. We asked program officers who had been managing many of those so that we could narrow from the 27 districts that were candidates to 10 that had the data that could substantiate some of the findings that we were looking at.
We then did a review of the—a secondary review—of the literature and talked to people who had been evaluators of a number of highly regarded, major funded programs that had also demonstrated that they had been able to move the numbers. We triangulated, I guess is the best term, across all of these sources and got a number of people to work with us in developing a protocol that we could then use to interview leaders in these districts.
And we found at the end of the day when we pulled the literature and the interviews and the discussions with experts, where we also had expert meetings as well, we found these particular factors, which we felt were those that could support systemwide reform.
The first was that of ownership and accountability. Michael Hickey, I will confess in the interest of full disclosure, was my superintendent the entire time that my kids were in the Howard County Public Schools, and I learned, you know, independently of the study that we did, I learned ownership—community ownership—from Michael Hickey in the sense that there is an absolute essential—it is absolutely essential that the community own the problem, and that if that does not happen, they are not going to sustain whatever effort is put forward in order to carry out the
solutions.
The accountability issues really relate, I think, to the fact that part of the ownership is really understanding the data. The numbers that are there that say how students are doing, disaggregated by group, so that you can, in fact, see where there are problems and where you're having successes, and that you will have a way of being able to keep track of the changes so that you can know how you are doing over time.
The second major theme was that of resources. Many school systems have access to a lot of different assets, but they're not necessarily pulling all in the same direction. Some are actually pulling in opposite directions in which case you get very little movement or very little synergy from those things that you have that are there available to you, but another resource that we found that was essential in our discussions with the people, with the leaders, was that of time—providing sufficient amount of time for the community ownership and planning process to take place. Rosabeth Moss Cantor once said about institutions that change, that if any plan is imposed from the outside that it will be dead in the water upon arrival, but in fact, people will put pretty rigorous requirements on themselves if they have the opportunity to help develop the plan, and that usually takes time.
We don't have a lot of the time, but I think that the one place where we need to take the time sufficiently is to plan and to actually develop something that looks toward our metrics and how we're going to know we are there, and that's an essential part of any kind of a systemwide
reform effort.
The third is just obvious and that is data- and research-based practices. I've mentioned in almost everything the need to know how you're doing and to know how different subgroups and different schools and different classrooms at every level, data really are needed in order to be able to bring reform about, and research-based practice is an obvious one. If you have a choice between two things, we would rather do the thing that works where there is actually some base for believing that you will get the outcomes that you want.
The next issue was that of high expectations and high standards. There is a quote that Benjamin Mays, who was the president of Morehouse [College] and an intellectual and spiritual mentor to Dr. King—who says that most of our problems stem from low aim, and I think that in many cases, especially with the students that we're talking about, that many of our problems are derived from the fact that there are not the highest standards and not the highest of expectations, but we have ways of presenting strategies, curricula, et cetera, that do speak to higher standards and higher expectations for all students.
The fifth issue is related to management and system capacity. These are large industries, I guess you could say, and yet, they are often poorly managed. In some cases, you have security forces. You have transportation. You have technology. You have human resources, and those systems are not really running at optimum and in many cases they are, in fact, not getting the advantages that would go along with, for example, large purchases and things like this. So the need to respond to the management and system capacity questions and to utilize the resources of the community in doing that was one of the major things that was highlighted for us in those interviews. In some cases, the superintendents or what have you would say "well, we were able to utilize industry to help us come in and look at our IT or our human resources or our budgeting or our auditing in order to be able to find resources."
And the last item is implementation and technical assistance, and that is the need to go to scale. And that is an absolutely crucial component of transforming from something that is basically one-shot small project to something that is truly systemic.
MR. FINNERAN: Thank you. Now we'll have Toni.
MS. TONI CLEWELL: Before I start, I'd like to recognize some people who were very influential in putting together our report—that's the report that's in your packet—curriculum, math and science curriculum. Pat Campbell, who with her team at Campbell-Kibler Associates did the review of the math curriculum, and questions on the math curriculum I think I'm going to direct to Pat or deflect to Pat.
(Laughter.)
Also, I'd like to recognize members of my team: Lisa Troy—they're sitting in the back—Lisa Troy, Nicole Detterding, Kara West, and Clemencia Constantino, who worked on the report. I would also—I don't know if Sara Mayes is here? Sara worked on it—no, she's not here.
Also I would like to thank my AAAS colleagues and also Linda Rosen, who provided some very helpful comments on our report.
I want to start out by talking a little bit about how we got started. The GE Foundation sent out an RFP asking for a review of effective math and science curricula and professional development models. They also stipulated that those—that these should be identified—the effective ones—should be identified in terms of actual evaluations that had, as the main outcome, an increase in student achievement. Kind of a radical approach.
(Laughter.)
And we were lucky enough to get awarded the contract or the grant and this is what we did.
First of all, we developed criteria to screen the evaluation studies, and our criteria—there were three criteria. One is that it had to—they had to have rigorous methodological designs. That the measures of impact on student outcomes had to include, but not be limited to, test scores. So we wanted hard, quantitative data on student achievement, and that
the data had to be comparative with experimental or quasi-experimental design preferred.
We set about reviewing 400 studies and our resulting report will identify the curricula and the professional development models that actually have effectiveness studies that met our criteria, and
we've also identified the ones that are the most effective based on our look at these studies. We have a much more comprehensive report that's on our Web site. So if you want to look and see this and more delving into what we found, you can do that. Look at The Urban Institute Web site.
I should present a caveat right here in terms of effectiveness. There are probably a lot of effective curricula out there that are not on our list, but they're not on there because there haven't been any studies. So, this is a call for more studies on curricula and professional development models.
I'm not going to go through our findings because I think our findings are best read in the report where we actually lay them out and summarize them, but I did want to comment on a number of issues that surfaced during the time we were doing the review that we found really striking and worthy of comment and discussion. So I want to bring those up today to you.
The first thing was the lack of studies that met our criteria. I said that we looked at 400 studies. Well, you can see how many I reported in our report. Out of 89 mass curricula, we found 156 studies and those 156 studies were only for 18 curricula. So that's 20 percent of the whole bunch.
Of the science curricula, we looked at 80 science curricula and we identified them. We found 45 studies for only 21 curricula—26 percent of the whole bunch.
And then for professional development models, we only found 18 studies that actually met our criteria.
So what does this tell us? That there is really a lack of evaluation studies and data out there. Roger and Shirley talked about data and the need for data. Well, there aren't very many data. However, another shocking discovery—this project was full of shocking discoveries. Another shocking discovery was that even though there's such little evidence out there, the evidence that was there was not being used.
So, for example—I'll give you an example of professional development. We identified some strategies that research says are really effective, however, few of these were being—few activities were being modeled on these findings, and there were so many models out there that were using strategies that don't have any validation in the research.
Another shocking thing that occurred to us as we were going through this exercise is that most U.S.-educated students are learning math and science from curricula for which there is no evidence of effectiveness in terms of increasing student achievement. Just think about that.
We're pretty convinced that the situation is not going to change until curriculum selection committees make it clear to publishers that unless they provide credible evidence of effectiveness in terms of increasing student achievement, their products will not be considered for use. I really think that that's key because when we looked at the kinds of data that the publishers were collecting, they were marketing studies, who bought their studies and what they looked like. They were not collecting data on how effective their studies were. So it's really up to the school districts and the schools and to I guess all of us to force the publishers to require more research.
Another suggestion that came out of all of this is school districts and schools should be collecting their own effectiveness data, and this, again, speaks to what Roger and Shirley were talking about. They're perfect laboratories to be collecting data on how these curricula and other things are working, and the benefit of this is that they can look at how they're working for their particular target populations.
I think, especially with No Child Left Behind and all of these kinds of data gathering—these policies that are requiring data gathering, I think it should not be that difficult to do, and then also contribute to the body of research and what we know about curricula.
So I'd like to leave you with a few questions for discussion that I hope we can bring up later and talk among the panel and with you.
Why are districts and schools not using the findings of research and data to drive decisionmaking about instructional and the professional development strategies?
What can policymakers do to encourage this practice?
And what can we as researchers do to encourage this practice?
MR. FINNERAN: Very good. And Michael, our final speaker.
MR. MICHAEL HICKEY: Well, Shirley Malcom characterized her remarks as sort of looking at the situation from the 20,000-foot level and she indicated correctly, I think, that Toni would take it down a little bit closer to the ground.
I guess I'd like to offer in advance that the claim or maybe the disclaimer that my perspective is going to be sort of that of ground zero—the teacher in the classroom.
Over the past five or six years since I left the superintendency, I've been spending my time not teaching in a classroom at the university, but spending most of the time in schools and classrooms around this state and around the country working with teams of teachers and principals on standards-based, data-driven improvement of student performance, and as I looked at the focal issues for the presentation here today, it said we were going to be kind of looking at three things—fixing the system, not the kids, and I thought, absolutely. And I recalled W. Edwards Deming's statement that every organization is perfectly designed to get the results that it gets.
(Laughter.)
You think about that in terms of your local school system and the performance results and there's some good food for thought there.
The second focal issue had to do with the pointed content matters.
And the third had to do with using data to make change, and I'm probably going to talk more about the third one than the other two, but I will definitely be commenting on those as well.
I'd like to begin, I guess, by asserting that we use—really, we use data for two purposes. We use it for accountability. In other words, to prove that what we're doing works. We also use it for school improvement, to improve what it is that we're doing. And I think it's important to state what may seem to be the obvious that those two purposes are not the same and that the data required for each of these purposes are likewise not the same. I'm going to come back to that in just a few minutes.
I think we also need to distinguish between school reform and school improvement because I think, personally, they're very different processes.
School reform, as I've seen it and as I've participated in it in various roles, is really more often than not school system reform. School reform tends to result in a systemwide or a systemic strategy that doesn't necessarily trickle down to the school level. School reform is primarily accountability oriented; i.e., you know, meet AYP [adequate yearly progress]. School reform is assessed on an annual basis, typically using summative assessments. And finally, as several of the panelists have already stated, school reform is difficult to sustain, particularly in large and complex systems.
School improvement, I think, is different in these respects. Real school improvement happens at the classroom level. That means that schools improve because the performance of students at the classroom level improve. Schools don't improve at the system level for the most part.
School improvement results from a grade or department team-level process that involves daily interaction and daily monitoring of individual student performance data. School improvement focuses on using that student performance results formatively to adjust teaching practice in response to student needs.
School improvement is assessed on a daily basis using formative assessments of students' classroom work. It's been stated that, you know, students generate a mountain of data over the course of a school year and yet, most of that data is not mined and is not used for the purpose of
improving schools or the instruction that takes place in the classroom.
Roger made a comment earlier about the importance of building a culture of data, and that culture of data is something that really needs to get down to the level of data where changing of
instructional practice—adjustment of instructional practice—can take place, and that's the student data at the individual classroom level.
In my experience, schools and school systems have been engaged in school improvement planning for several decades now, at least in the State of Maryland that I'm most recently familiar with. It goes
back at least to the 1980s and in some cases earlier than that, and the results have been erratic. They've been inconsistent. They've been less than successful. And I would suggest to you that these volumes, these notebooks full of school improvement plans are, in reality, school accountability plans, and that while these are important in some respects, they have to be acknowledged for the fact of their structural inability to bring about sustained improvement of student performance. An accountability plan is not an improvement plan.
So part of our purpose here today is to really focus on the issue of using data to make change, and indeed, this is very important, but we need to understand what data and what change.
At the beginning of my comments a few moments ago, I referred to a distinction between school reform and school improvement and I think that this distinction is very substantive and can be perhaps most readily explained or readily understood by thinking about data in terms of a hierarchy.
If you think about accountability purposes—No Child Left Behind, for example—the hierarchy of data is quite apparent, readily apparent. We have the state assessments for each state across the country and we have other sorts of national data that clearly dominate in that hierarchy.
School system assessments, which are used for benchmarking purposes primarily, certainly have some meaning at the school level, maybe at the state level, but at the national level, for the most part, they drop off the radar screen. When we get down to the level of assessments at the individual school or, in my judgment most importantly, assessments at the individual classroom level, these don't even appear on the radar screen.
So for accountability purposes, it's those state assessments in Maryland, at MSA [Maryland school assessment] and the HSA [high school assessments] that drive what it is the schools do.
For school improvement purposes, though, I'm going to suggest that the hierarchy is exactly the opposite. The most important data that can bring about improvement in student performance in the schools are the data that are generated in the classroom and in the regular course of assessment of student work by the teachers that are in those classrooms, because it's these data that enable the teachers in those classrooms to adjust their instruction in response to student need, and to do that in real time, not to have to wait until a year later or six months later or whatever it is the
interval that the state assessments become available.
So the least useful data in school improvement for school improvement purposes, in my judgment, are those state assessments, and yet, I have worked literally with hundreds of school systems and schools, here and elsewhere in the country, and if you look at what it is that drives their so-called school improvement plan, it's the state assessments.
And so, you know, I'm suggesting that we really need to rethink our understanding of how it is that schools improve and also to understand clearly the difference between accountability—which is important, which is very important—and school improvement, because that distinction is one that I think can lead us to a much more productive strategy toward changing the performance of students in math, science, and the other curriculum areas across the country.
The two studies that we—that brought us here today contain, I think, some very key elements and key findings that are moving us in that direction, and I'm hopeful that we can move very quickly to the next steps in that journey so that we can make it down to ground zero where I think we should be operating.
MR. FINNERAN: Okay. Well, thank you all very much for sticking to the schedule and giving us lots to work with.
I'm going to just try a couple of questions to get the panel members to interact a little bit and then we'll open it up to everybody else.
Shirley, one of the things Michael said is that, well, a lot of this systemwide stuff, systemic reform, doesn't seem to trickle down, doesn't make a difference at the classroom level. How do you see the things you talked about in systemic reform having effect on what happens in the classroom?
MS. MALCOM: Well, one of the things that Mike said was—one of the things that Toni said was—we need to know which things are effective. One of the things that Mike says was that teachers need to be able to actually adjust their instruction on a just-in-time basis as it were, you know, informed by the data, and I think that there's a role at the systems level for helping to make both of those things happen.
Just as an example...
(tape break)
...classroom level, we're going to have a problem when the teacher gets it.
There is also the issue that being able to adjust the instruction at the classroom level is going to depend on teacher capacity and that is something that the system has to have a responsibility to
address.
So these are not mutually exclusive things. They basically—that is, in fact, the nature of systems that all of the pieces hook up together and, in fact, you have to have all of them in play at the same time in order to get what it is that you, you know, that you really are trying to achieve.
MR. FINNERAN: Okay.
Roger, you started us off talking about the importance of data and we heard from several of the others about how either it did or it didn't exist, data when we did get it wasn't used.
When you look at it, who's to blame? Are researchers producing—I mean, in some cases, data's not being produced, but perhaps it's because it's not being funded and we should think about who should be supporting it. When it is produced, is it asking the right questions for systemic level reform, where I think it's probably difficult to acquire data, and for classroom level?
And Toni had talked about the problem of trying to do alternative assessments of students, not just test phase.
So, when you look at this—and also, then, there's receptivity. Are our teachers educated to interpret data in a way that they can use it to improve their classroom performance?
So, looking at all of those questions, how do you define more precisely where we need to go in data collection, presentation, and use?
MR. NOZAKI: In our experience, I think when we look at the obstacles to getting data and applying data, I would say it boils down to three sets of things.
One is at the technical level. So some of the issues around capacity, both in terms of the research side as well as within the schools—do people have the basic knowledge and skills to be able to do evaluation, to collect data in the right kinds of ways? The human subjects review kinds of things that are all these little pieces of the technical aspects around evaluation and data collection and analysis.
So, at one level it's that where I think the field could probably progress farther in terms of having the right kinds of metrics, as Toni was talking about, but I think within the school systems,
there's a technical capability issue there in terms of just understanding how to go about collecting data and using data.
So that's one area.
I think a second is this culture issue. So is it an expectation within the culture that that's how you operate? That whatever your role is within the education system, is it an expectation—do you have the expectation of yourself and do others have the expectation of you that that's how you will operate? That you won't be expected to just make decisions on the fly based on no data, which is often the expectation, but the expectation is rather that you will first try to find what data you can and then make your decisions on the basis of that.
So that's kind of—I would put that in sort of the culture bucket.
And the third, I'd say, is kind of the philosophical category, and some of those heated discussions that I've been in that relate to evaluation have been with people who spend their lives on improving educational outcomes for disadvantaged students.
And the level of energy against evaluation or around evaluation has always been impressive to me from people who really are committed to doing what we would—what I would—consider the right thing for students.
And I think that comes from a couple of sources. One is that, historically, there's been a process of evaluation being imposed from the top, and I think this goes back to Shirley's point about ownership. If people haven't been part of the process of developing the metrics and the
process of evaluation, they will feel that it's imposed from the outside and has no use for them and they will naturally resist that.
I think another piece of it is that relatedly that kind of data has been used against them so that, again, rather than being a part of the process and finding data that they can then apply for their own goals, it's been used to penalize people.
And then I think, thirdly, that there just has not been the opportunity to have candid discussions about what works and doesn't work in terms of evaluation, in part, because the places where research has come from and the places where education practitioners come from and the places where education policymakers come from have very little connection to each other. So they all come up through different tracks. They all tend to attract different kinds of people and those three streams very rarely have any contact.
So I think that feeds that distrust and lack of ability to move forward on some of these basic data issues.
MR. FINNERAN: Okay. Mike, you had talked about the need for classroom teachers themselves to be doing some type of assessment, developing their own data. Do you have examples that that works or, you know, is this one of those cases where it should be done and a few people can do it, but in fact, we haven't trained teachers to do that or given them tools that they can apply to do that?
MR. HICKEY: Well, I think that a couple of the points that were just made pertain here as well.
I think that for the longest time, the teachers felt that they were gathering data for someone else's purposes and they weren't always sure what those purposes were, and once they did whatever had to be done, it disappeared and they seldom saw any payoff for having done that.
There also has been a lack of really useful data available to teachers and we at schools of education have not done even a barely adequate job of preparing teachers to use data and to translate those data, you know, into action related to their teaching.
And so I would strongly agree that—and the answer is teachers are getting better at it. Institutions and various groups are getting better at helping teachers understand the importance of using data and the technology, of course, is getting better as well.
So the issue of capacity building that Shirley mentioned—capacity building—and it's part of their study as well—capacity building is critical and part of that capacity building involves creating that culture of data, the data-driven culture that has to exist within each school. It's not enough to have it permeate the central office or the central administration.
MR. FINNERAN: Toni, you have to actually try and use data to evaluate; that actually says what works and what doesn't, and you talked about, you know, using some assessments that were available and thinking about alternative assessments. What did you find worked? What was informative to you in trying to look at whether or not a curriculum was effective, and what would you like to see in terms of data that would help you make the types of decisions that—that you need to make?
MS. CLEWELL: Well, we would like to see more evaluations that adhere to the criteria that we developed. I should say that our criteria, as rigorous as they might seem, were somewhat watered down when compared to the National Academy criteria and the Department of Education criteria.
If you look at how many curricula, what works, that the clearinghouse came up with, I think there were two, and we didn't want to end up with two. So we kind of—we had to water down our criteria. We had to say, well, we can look at quasi-experimental. We don't have to look at experimental design, because they're just not out there.
I think that it's reasonable that—I think we should set some reasonable criteria. We have to be looking—for example, we definitely want to look at comparative studies. We want to see that when you're looking at the effect of a curriculum, you want to be able to compare the effect of that curriculum and another group that looks like the group you're looking at, but these are just basic research; you know, the rudiments of good research, and so that's what I think we should be doing, and I think it's certainly possible.
What I'd like to see also is, when people are developing the curricula, that they start out by doing that. I'm sure that if you're a developer, you want to know whether your curriculum works. That's a basic question, and one of the things we found, for example, with science is that there were a lot of—there were studies that the developers had done, but they just didn't publish them. So that's another issue. They have to get it out there in the literature. We ended up calling every developer and saying, do you have, in the bowels of your files, a report? And they did, often. So anyhow, that's—that's my response, and I hope I answered it. (Laughs.) I kind of got off on a tangent.
But I want to also mention something that's been bothering me. I am—I am disturbed by the disconnect; the need for accountability measuring and the measuring in the classroom, looking at student progress. Those cannot be separate, without any kind of linkage, because schools and districts are accountable at a very high level to meeting those accountability requirements, and that's what they're motivated to address.
There has to be some connection between that and the kinds of measurements and data that are being collected in the classroom. So I'd like to pose that question to Mike.
MR. HICKEY: Well, I think there very definitely is a connection, and that connection is that if the performance of the students in my classroom improves and the other classrooms that make up my school, then that school is going to meet the AYP accountability requirements, but I think that trying to develop a strategy that's sort of generalized to improve performance on average across the school population is not going to do it, and I think we have years of examples to see the failure of that kind of strategy. It has to go down to a classroom-focused improvement process, and when you improve performance at the classroom level, performance at the school level will improve, performance at the system level will improve. I think it's linked that way.
MS. MALCOM: What about the practice of—Oh, I'm sorry.
MS. CLEWELL: But it also has—it needs to be linked through the standards. I mean, presumably, if you're all measuring against the same thing, that you really ought to be able to have this conversation. And so maybe that's the question; that is, is, in fact, there clear linkage across all of those things to the standards that you're holding students—to which you're holding students?
MS. MALCOM: What about the practice, for example, of using standardized test score data and providing it to teachers and breaking it down with—at the classroom level, and having them respond to that in their—by identifying the gaps in what they're teaching? A lot of districts are using this. Do you think that this is effective?
MR. HICKEY: Sure. I mean, I think that sort of systemwide or statewide testing, if you get the data in the right format, it's going to be helpful but by itself, it's not sufficient.
MR. FINNERAN: All right, well, we're having a good time up here—(laughter)—but perhaps you'd like to join us. We could go on, but at any rate—questions from anyone? We
have—I think we have a microphone that we will—two microphones that we'll pass around. Please give us your name and affiliation. This is being recorded. Thank you.
Yes? Wait for the microphones. Thank you.
Q: My name is Rosauta Igedroveda (sp), and I'm the mother of four daughters and one son, and four daughters who are—three of them are graduating from—well, one has graduated, two of them are graduating, but math was always very difficult for them. And my question is, are there other factors that impact student learning, such as class placement of students, where they're placed in classes, let's say in sixth or seventh grade, because they need special skills, and other children are placed in other classes where they're getting advanced math, and somehow or another it just—the gap doesn't ever close. So how does that impact student learning? That's what I want to know.
MR. FINNERAN: Okay, who's going to step up to the plate? Shirley, you're ready to go.
MS. MALCOM: I'll step up to it, and basically tell you, that is usually the problem, and that is, it's very difficult to learn what you've never been taught, and in many cases, in the school systems where we have worked, often these classes, I tell people, I have to walk down the hall and determine which class is which. And that is too often the case, where there is an expectation that the girls can't do it or the minorities can't do it, and there is a delay, for example, in offering algebra to students. And you never catch up, and you're absolutely right. That's what is meant by tracks. They're parallel, they will go forever side-by-side, but they don't intersect, and that is a real problem.
What are some of the solutions? Essentially to unpack the tracks; that you give students an opportunity to succeed or to struggle, as it were. You're better off struggling with the good stuff than succeeding at something that's not very useful.
MR. FINNERAN: I know that we have to have multiple entry points. I mean, one of the things is you can—you don't have to study calculus when you're 16, you know? If it doesn't happen till you're 19 or 22, there should still be an opportunity.
Go ahead. Wait for the mike, and identify yourself.
Q: Pat—Pat Campbell from Campbell-Kibler Associates. Basically, what we've been finding all along is the pipeline only goes one way. You can only drop out, okay? You can't get back in. Once you leave a math course, there aren't alternatives for you to come back in. When your kids were in sixth grade and a sixth grade teacher said, "hmm, I don't think she should go in advanced math," it didn't really matter a whole lot after that, because there weren't opportunities for her to then come back and do advanced math or get back on the track some way.
So that it's—the tracking, as Shirley says, is really important, but equally important are lots of opportunities for kids to come back on. When you think about all the kids that leave engineering their freshman year in college; tons and tons. How many kids go into college and then, when they're in college, go into engineering? There must be one somewhere. (Laughter.)
MR. FINNERAN: Thank you.
Q: Hi. I'm just going to preface my question with background.
MR. FINNERAN: Name—your name, please?
Q: My name is Amanda Simpson. My background's aerospace engineering. Worked in aerospace engineering, went into teaching, taught several years in inner-city Miami. Taught eighth grade math and also was the head of the math department. So as head of the math department, I not only taught, but I also got to review a lot of curriculum and look over that and also do a lot of data analysis as well.
So my question stems from [my] own data analysis, having a group of teachers who were all teaching with the same curriculum, and we would see student learning gains; AYP, whatever—all these great acronyms we use. We would see 91 percent in one teacher's classroom and 19 percent in another teacher's classroom. We had teachers that were working 16 hours and doing amazing things, and we had teachers that worked 8 hours and there was cursing going on in the classroom and obscene drawings going on and no learning going on.
My question to the audience, and those of you that are parents would know, you would rather have a great teacher with a horrible curriculum than a terrible teacher with a fantastic curriculum. I can see the curriculum making a difference but, you know, I wonder—your mention
about, you know, teachers not liking some of the data stuff, and I, for the first time in my life am falling into that category, but I also feel like there are some other things that we need to take into account.
You talked about charismatic leaders. Why can't we just have charismatic leaders? Why isn't—I mean, are the inner-city schools just destined to have bad teachers and focus on standards?
MR. FINNERAN: Okay. Well, I guess—and a couple of people may want to respond to this, but I think one of the basic data questions is, what do we know about the relative importance of the institutional systems stuff, the curriculum, the teacher. How big a difference does it make? Where does the variation occur?
MR. NOZAKI: I'll just stay on my data track for a second in the first answer to this question, and then I think everybody else will probably want to jump in on this.
But I think one of the most important uses of the data is also in the decisionmaking or is in the decisionmaking around student assignment and teacher assignment. So if you can get at that kind of
data and understand what students are doing well in what areas and which ones aren't, and which teachers that are doing well in which areas and which ones aren't, and try to use that process to get the best teachers with the kids who need them the most. So that's a very Pollyanna kind of thing to say, but there are school districts who are trying to do that, so that they are really trying
to match up the right students with the right teachers, and then continually assess what's happening in those classrooms and with those students.
MS. MALCOM: I want to say that the only thing that's wrong with the charismatic teacher is that there are not enough of them, and therefore, most of us are going to have to have noncharismatic teachers who are effective, and part of the problem that the differences in the curriculum are revealing is the teacher knowledge.
Let me say that even though we fly at—I'm flying at 20,000 feet in this report; I live at ground zero in working with teachers here in the District of Columbia Public Schools, with Joan. And one of the things that we encounter when we have sessions with teachers is that a lot of what you see is their own lack of confidence and knowledge of the subject area.
Many things that look like discipline problems are teacher knowledge problems. And many things that look like student performance problems are teacher knowledge problems. The teacher capacity issues have to be addressed head-on, and in many cases, they're not being addressed head-on, and the principal who is willing to sit there and see some classrooms with 90-some percent of the students being successful and other classrooms with the same curriculum with a very small fraction of the students being successful, all things being equal, then there is an issue here with regard to the responsibility of the leadership to begin to address some of those concerns.
MS. CLEWELL: Amanda, your questions and your comments raise two issues for me. One is that you're saying teachers are very important. How important are they vis-à-vis the curriculums? Is the curriculum more important than the teacher? And my answer is the curriculum is important, the teacher is important. Educating a child is a very difficult task, and you need all the help you can get. You need a good curriculum, you need good teachers, you need good professional development. One is not better than the other, although, I mean, the research does say that teacher quality is the most important. We know that. But without a good curriculum, you—I mean, you're really floundering as well. You need all of these things. So we can't be thinking in terms of one being better than the other. We need them all, and we have to go into education with that mindset.
Your question about are you—do low-income schools, inner-city schools, are they destined to have bad education? My answer is no. Pat Campbell and I have done a study on effective schools—I mean, there are lots of effective school studies out there—where you match demographically, you match schools, one set of which are doing—are performing at the mean in terms of the standardized test scores, and one group of which are performing at one standard deviation above the mean or more, to look at what makes the difference between those schools, and we found that there are a number of really good schools that are in inner cities, that have low-income students, but are performing at the same level as suburban schools; high-achieving suburban schools.
MR. FINNERAN: Just to follow up, one of the findings, I think in your study, was that the relative importance of teaching technique versus subject matter knowledge, and this has been a complaint, particularly in science and math education, that the education schools give you more technique than substance. Did you find that, and also, Michael, just in your interactions with classroom teachers, the relative importance of content knowledge versus technique, general theory of education?
MS. CLEWELL: Well, we certainly found that for successful professional development, you had—all of the models that we looked at that were successful incorporated content, and also, the "Rising Above the Gathering Storm," the latest National Academy of Sciences report, says that, and I was very impressed. (Laughter.)
MR. FINNERAN: We believe in content. (Laughter.) Michael?
MR. HICKEY: Well, I would just say that you have to have both of them and, I mean, there's no way you can fake a lack of content knowledge and cover it up with some flashy skills in the classroom. I mean, if you don't have the content knowledge there, then it's a very empty process, but one of the things, if you really operate on a grade-level team or department-level team model, other members of the team can help supplement some of the lack of knowledge that you might have, and you might be able to bring other kinds of skills to that process. So it's the dancer and the dance sort of thing. I don't think you can really separate them.
MR. FINNERAN: Okay, we had a question here in the red shirt. Hand him a mike.
Q: Andy Schaus from the National Research Council. The discussion has gotten into the complexity of change; that, you know, you don't implement a program and then see astonishing results, that this issue of teacher capacity is huge, and I wonder if, in your analyses, you've been able to track that. I mean, one expectation would be you implement a program; you get some fidelity measures, and the next year you see some change in student learning. I quite doubt that's how it works, and I'm wondering if you had any insights into the time lag between implementation, teacher knowledge development, capacity, and real change in student learning. Any insights into that?
MR. FINNERAN: I know one of the reports actually gave a specific amount of time, but all of you go ahead.
MS. CLEWELL: Well, when we looked at—what Kevin is referring to is when we looked at professional development models, we saw that in order to be successful, they had to at least provide 80 hours of professional development. That's when changes started taking place that could be—that you could measure later on, in terms of student achievement, and I was also impressed to see that the "Rising Above the Gathering Storm" mentions that, too. So I've actually seen it in three research studies, so it must be true. (Laughter.) But 80 hours is what's required in terms of making a change in the teacher and in the teacher's achievement—achievement levels of the students.
In terms of reform, I think I'll let Shirley answer that.
MS. MALCOM: If you look in the report, you'll see a little kind of schematic on page 10 of the report, and the schematic is really just to represent the notion that there really is such a thing as worse before better. If you keep doing what you have done, you know, you can just plug along, but when you, in fact, begin to take on a new strategy in terms of professional development, you're going to get—I mean, you're going to be worse at it, because you're just learning it, and it does, in fact, take you that kind of time before you are able to master whatever strategy it happens to be and reach a new sat point. I mean, that is reality.
But the—I guess the alternative is staying in the same place, and given the overall theme of this session, "Global Stakes, Local Control," we can't afford to stay in the same place. And so that's the, you know, that's that differential between starting something that is, in fact, new and different and getting to the point where we can learn it, fine-tune it, and own it.
A specific example, with regard to teachers who began using hands-on science, for example, materials. When you first see teachers doing it, it's extremely mechanistic. I mean, it's not any part ingrained, it's not a part of who they are, it's not—it is rote, but over time, they have to come to own it.
MR. FINNERAN: Yes, over here.
Q: My name is Michael Neuschatz with the American Institute of Physics. I had a couple of questions. One was that Toni Clewell mentioned that the thing that struck her was, in all the different curricula that she identified, how few of them had data, effective data analysis to evaluate how well they did. I'm struck by how many different curricula she identified. I think it was 60 or 80 in one category, and maybe more than that in the other, and I think about the kind of pharmaceutical trials that are done with NIH [National Institutes of Health] where you compare very carefully, very systematically, two or three different treatments and see how effectively one works against the other, and then you can draw some conclusions. When you have 60 or 80 different possible
treatments, I almost think it's not surprising that you wouldn't get effective comparisons, because there are so many different possible permutations of comparisons, and it becomes such a Byzantine kind of process to make those comparisons.
So maybe one question is, can we ever get to effective accountability, or evaluation of these programs, when we have so many of them? Is there any—I mean, I see a benefit in the creativity of having many, many different treatments, but in terms of understanding what works, wouldn't it be better if a few were identified and compared in a very careful, systematic way before trying to throw them all in a basket and say, let's see which one is the best?
MR. FINNERAN: I'd like to interject a sort of corollary to this question, and that is, was there any attempt to look at what some people identify as the major problem with U.S. mathematics, is that it's a mile wide and an inch deep. In many of the international comparison studies, they find that curricula used in other countries cover far fewer topics per year, but in more depth. So are any of these U.S.-used curricula following that model of really trimming out—I mean, in many cases, you know, there are state standards and you have to treat a certain—you're told to treat a certain number of subjects, but in looking at them, do we need this many, and are any of them really that different, you know, in the style of Singapore or Japan or elsewhere?
MS. CLEWELL: Well, for the math question, I think I'll refer that to Pat, since she was the one who actually did the reviews of the math curricula, and I'll answer the other one.
MR. FINNERAN: Wait—wait for a mike. Yes.
MS. CAMPBELL: On the math, they didn't break down that much in terms of, with one exception, one of the curricula we looked at was very specifically focused on the Singapore kind of model. They did break into all levels across the math wars, though, so that it did include those that were the more traditional curricula in terms of the separation of algebra, geometry, second-level algebra, pre-calculus, and so on, and then those that were more standard spaced in terms of the integrative that is going through at different levels.
I only wish we had all those combinations and permutations of comparing across the 89 or even the 18 math curricula. In only three cases was the comparison against something definable. In every other study, the comparison was against a mythical—not mythical, but—okay, a mythical traditional curriculum. So I think that your point is a really valid one.
There were not that many distinctions between those which fell on different parts of the math wars continuum, so that it looked to me that you could probably break it into maybe four or five different areas, and then work on that, yes.
MS. CLEWELL: In terms of physics, it's pretty much the same. (Laughs.) Yeah.
MR. FINNERAN: Just quickly, we're near the end, and I wanted to give all of the panelists one last chance to talk about what they would ask from the others. I mean, you all have your specific needs; you come from data needs, classroom needs, system needs, curriculum needs. Part of the trouble here is integrating this, all of this—the data, the policy, the implementation, and the take-up at the classroom level. So looking to your colleagues, starting with Michael, what do you think, you know, from the classroom perspective, what would you like from the others? And then—quick, you've got about 30 seconds.
MR. HICKEY: I think my request would be to really pursue with a vengeance, if you will, this whole issue of capacity building, and look at that capacity building not just at the system level, but at the individual school level.
MS. CLEWELL: I would—my question would be how do you create a culture of evidence? How do you, Mike, for example, in a system, how do you get a culture of evidence going so that it's accepted? Shirley, how do you get it into systemic reform? Roger, what are the most important pieces of data that you need for that?
MS. MALCOM: My major issue, I guess, would be directed at Mike and Toni, and that is that if we know better, why don't we do better? And who has to know in order to move us to doing better?
MR. NOZAKI: I guess I'd touch on a point that I made in my earlier comment in terms of trying to connect the various sets of people who are working on these kinds of issues. So, connecting the researchers with the education practitioners, the teachers in the classroom, with the policymakers and also getting the general public engaged in that discussion so that we can
build their data capacity as well and build their demand for the data and the kind of change that we're all looking for.
MR. FINNERAN: Good. Thank you all. And you all made it in 30 seconds, so we finish right on time. Thank you very much. Thank you.
[END]
Latest Research
Browse by Topic:
Printer-friendly summary
