|
Testing Our Schools
Program #2014
Original airdate: March 28, 2002
Produced by
John D. Tulenko
Correspondent
John Merrow
ANNOUNCER: Early this year, president George W. Bush signed into law the most sweeping reform of public education in 40 years.
Pres. GEORGE W. BUSH: The fundamental principle of this bill is that every child can learn, we expect every child to learn, and you must show us whether or not every child is learning.
ANNOUNCER: With near unanimous support in Congress, the president pushed through the main element of his plan: more tests. The law now calls for testing students in every state every year from the 3rd through the 8th grades. Many schools have already embraced the new accountability, but some complain they've been reduced to teaching to the test.
JUDY HIRSCH: It's more important that children do well on a test than they do well in my class. My grade doesn't matter. My comments don't matter. My teaching doesn't matter. It's just this one test.
ANNOUNCER: And test writers themselves have doubts.
AUDREY QUALLS, Test Writer: I want to know whether or not a student asks questions, whether they're part of the conversation. I can't measure that on a test.
JAMES POPHAM, Test Writer: The public in general has bought into the idea that these tests reflect how good schools are. It is simply incorrect
ANNOUNCER: Tonight FRONTLINE and PBS education reporter John Merrow look at how the new tests and standards are remaking public education in America.
PRINCIPAL: To show everyone here that Dumbarton Elementary School has more school spirit, we're going to turn on some championship music, some celebratory music. Boys and girls, your parents probably haven't seen school spirit like this before, so you help them cheer, too.
JOHN MERROW, FRONTLINE: [voice-over] In a public elementary school in Henrico County, Virginia, a celebration is about to begin.
PRINCIPAL: Boys and girls, raise your hand if you know all about school spirit- you know, that team spirit, school spirit? OK, now, here's where we need your help.
JOHN MERROW: It's not for homecoming-
We want you to start cheering-
JOHN MERROW: -or sports-
JOHN MERROW: -or graduation.
JOHN MERROW: This is a celebration over test scores. The teachers at Dumbarton Elementary have crossed an important threshold. For the first time, students here passed Virginia's state test. As a result, this school, once rated near the bottom, has moved all the way to the top.
Outside, a new flag is raised to mark the achievement, bearing the stamp of approval: full accreditation from the state of Virginia. In an effort to improve education, Virginia made passing its test a requirement. Schools that fail may be closed.
Nearby is a school that hasn't passed, Baker Elementary. And here the pressure is on to improve the scores.
KAYE THOMAS: School starts the minute that bus pulls into the driveway or it pulls onto the property.
JOHN MERROW: Kaye Thomas is principal.
KAYE THOMAS: Good morning, boys and girls!
CHILDREN: Good morning, Miss Thomas!
KAYE THOMAS: Boys and girls, why'd you come to school today?
CHILDREN: To learn!
KAYE THOMAS: That's right. That means everybody's ready to go.
JOHN MERROW: Baker Elementary is determined to pass the test and keep its doors open.
KAYE THOMAS: I want to reach full accreditation more than anything I want in the world. One of the teachers said the other day, "Kaye, if I offered you a million dollars or full accreditation, which one would you take?" And I said "Full accreditation," and I truly mean that.
JOHN MERROW: The students at Baker will be tested on a new state curriculum. Everything Virginia students are expected to know and everything their teachers are required to teach is described in the state's academic standards, called the standards of learning, or SOLs. In every subject and in every grade the SOLs spell out exactly what should be taught and what should be tested.
SUSAN PARRISH, Baker Elementary: The SOLs are my curriculum. They tell me exactly what I'm going to teach. They're specific to exactly what I do every day all day long. It gives me a guideline for what I need to teach, and that's something that I really like. I don't have to go out and say, "Well," you know, "I'm teaching famous Americans." The SOLs tell me exactly what 3rd-grade famous Americans I need to teach, what 2nd-grade, 1st-grade and kindergarten famous Americans I need to go back and review to make sure they've retained that information.
JOHN MERROW: To prepare for the state test, Baker Elementary does a lot of testing of its own. Groups of students begin the day with 10-minute speed tests. Computers run students through drills. And teachers are required to give weekly exams. Some of the testing helps track students' progress, some of it is simply practice for the state's multiple-choice test.
ANN HOUCK, Baker Elementary: What we try to do in 1st grade is give them all types of assessments, so that whatever they get on a test is not going to freeze them up. They're going to really be able to do what- you know, give the information that they know.
BRIAN SMITH, Varina High School: The kids are given the exact look of the test. I mean, we don't want there to be any unfamiliars that will, you know, detract from the accuracy of the scores.
JOHN MERROW: Down the road at Varina High School, the focus is also on testing. And here, too, they've had to adopt the state's uniform curriculum.
BRIAN SMITH: The bottom line is results. We throw together a good product, and I think that the kids can sense the difference between a teacher who's showing up to really prepare them and a teacher who's not.
MARK EDWARDS: We need to understand that if the public tax dollars are paying our salaries and providing the resources, we need to understand that there should be a result.
JOHN MERROW: For Henrico County school superintendent Mark Edwards, results are the bottom line.
MARK EDWARDS: High-performance organizations set goals and work hard to reach those goals. And I think that it's about goal setting and really paying careful attention to learning.
JOHN MERROW: Edwards pays close attention to test scores. He's running the schools with a "business-like" approach.
MARK EDWARDS: In any organization, whether it's a business analogy or sports, you know, you hire people and you give them the support, and you say, "This is the job that needs to be done." But if the job isn't done after a period of time, you say, "We've got to find somebody else that can get that job done." There's a clarion message to all: We must be accountable.
JOHN MERROW: But in some places, enormous hurdles stand in the way of passing the state's mandatory test. Across the county line, in the neighboring district of Richmond, virtually all the schools are struggling. Just 5 of the 54 schools here have passed the state test. Test scores at Lola McDowell's school are stuck at the bottom.
LOLA McDOWELL, Woodville Elementary: Passing those tests is the last thing on their mind. They're worried about whether they're going to have food on their plate, whether they're going to have a home to go to after school, whether somebody's going to be shooting a gun in front of their house, drugs, whether they're going to be abused.
JOHN MERROW: Her kindergarteners, she says, are behind from day one.
LOLA McDOWELL: Many of our children are coming to school in kindergarten for the very first time. They have no idea of how to hold a pencil. They have just gotten out of their mother's laps, per se. Some of them don't know how to use the restroom. Some of them are not able to utensils correctly.
KAREN MERIMAN: There's no books in the house. Or if there are books in the house, they are not given to the children and said, "No, turn the TV off. Do this. Do that." Our kids have, right back to the start, no vocabulary. They haven't been exposed to certain things.
JOHN MERROW: Many of Karen Merimam's high school students are also behind.
KAREN MERIMAM: I've had kids coming to school that were living in cars. I've had kids coming to school to eat. And when you've got a 13-year-old kid, or a 14-year-old kid dealing with something that you and I would have a hard time dealing with, and then we pompously hand them these SOLs and say, "This is what you have to do to get out of school," that seems a little ridiculous at times.
BILL BOSHER: I'm not trying to teach young people to feel good about living in the projects. I'm trying to teach them to do things that will get them out of the projects.
JOHN MERROW: Bill Bosher was one of the main architects of Virginia's school reforms.
BILL BOSHER: The kindest thing I could do was put on a little bit more on, not so much to break, but a little more on that will help them to gain the skills that will advance them, not just accommodate them. We do a disservice to young people who've been falling through the cracks by not holding them to the same standards. This may be their only hope to be taught.
LOLA McDOWELL: I am not against standards. We've had standards all along. I'm not against assessment.
JOHN MERROW: [on-camera] What are you against?
LOLA McDOWELL: I am against the pressure and the stress that is put on us for getting children to master these SOLs at the same rate of speed, when everyone knows that all children do not learn the same way or at the same rate.
JOHN MERROW: [voice-over] But back at Baker Elementary, a school that also serves many disadvantaged children, things are looking better. Three years ago, test scores here were at the bottom. They've climbed steadily. Now passing scores are within sight. And so is the test, just a few weeks away.
SUSAN PARRISH: If I'm nervous, the kids can sense it, so I try to hide it. We try to keep them pumped up. We try to keep them motivated. We do a lot of fun things. Before we get our day started, one of the things we do is we huddle, and somebody gives us, you know, a cheer.
CHILDREN: We can do it! We can do it! We can bust this test! We can do it! We can do it! We can bust this test! Go [unintelligible]
SUSAN PARRISH: It's just something that puts a smile on their face before they get started in the morning. But I get nervous. It's the big game. It's just like going to the big game. You get very nervous about it.
JOHN MERROW: [on-camera] What does that mean? You wake up in the morning-
SUSAN PARRISH: Wake up in the morning, and it's Super Bowl time. You know, we're taking the math test today. How are they going to score? Are we going to win the game?
JOHN MERROW: Around the country, in all 50 states, bottom-line reforms like Virginia's are transforming public schools. Often controversial, they're now at the top of the national agenda.
TELEVISION ANNOUNCER: [January 13, 2001] And now we take you live to the East Room of the White House, where President Bush will unveil his education plan.
JOHN MERROW: Mandatory testing is the centerpiece of George W. Bush's education agenda.
Pres. GEORGE W. BUSH: Children must be tested every year not just in the 3rd grade or the 8th grade, but in the 3rd, 4th, 5th, 6th and 7th and 8th grade.
How's your teacher, pretty cool?
CHILDREN: Yes!
JOHN MERROW: Bush campaigned as an education reformer. As president, he pledged to make schools show results.
Pres.GEORGE W. BUSH: [Urban League speech, August 1, 2001] I'm an activist for high standards. I'm an activist for accountability. My administration has set a great goal. We will lift the load of low expectations so that all children will rise!
JOHN MERROW: Not everyone agrees with the president's plan.
TED SIZER, Education Reformer: I don't think that the public is clear enough in the- in the costs of saying they're- "Yeah, we're going to measure everything on test scores.''
JOHN MERROW: Ted Sizer is a leader in education reform with serious doubts about more and more mandatory testing.
TED SIZER: At the worst, these standardized tests provoke a kind of drilling mentality. It's a game. And so you'll learn- you'll have people who will- you can pay to teach you how to figure the test out. Not the substance, the test. So what you're teaching is a kind of cynicism, which is, you get a high test score, then what you got to do- and the message for life is, you want to get ahead in life, this is the way you jiggle the system. And that's pernicious.
[www.pbs.org: Explore the debate over Bush's plan]
ROD PAIGE, Secretary of Education: Why is there such a strong defense of the status quo when the status quo is failing us?
JOHN MERROW: To fight for his testing plan, Bush brought an ally from Texas, Rod Paige, the former superintendent of schools in Houston.
[on-camera] Why has it taken us so long to say, "Hey, we need standards"?
ROD PAIGE: Let's think about that for just a minute. We're saying that in schools, we should have standards. And that turns out to be a controversial discussion. Would that be the case in the hospital district or in the public highways or in the building industry? Everybody else takes standards as the ordinary. And we're taking standards as something that is extraordinary? I think that gives you some clue of how much we need to change.
JOHN MERROW: [voice-over] The argument for change was first made almost 20 years ago by another Texan, billionaire Ross Perot.
H. ROSS PEROT: You can spend any amount of money you want to on this school system, and unless you manage it and control it intelligently, you will not get results.
JOHN MERROW: When Perot spoke, politicians listened. Soon Texas became one of the first states in the nation to put in place a high stakes test, called the "TASS."
TEACHER: You guys really do know this stuff. I want you to be confident in yourselves. We're about to pass out the [unintelligible] booklets.
JOHN MERROW: Schools and teachers were now under pressure to produce results every year.
TEACHER: All right, you guys ready?
STUDENT: Yeah.
TEACHER: All right, let's try that again. You guys ready?
STUDENTS: Yeah.
TEACHER: You guys ready?
STUDENTS: Yeah!
TEACHER: Yeah! All right.
JOHN MERROW: Scores went up in Texas, especially among black and Hispanic students. But critics charged that students who could not pass the test dropped out of school in higher numbers. And there were reports of widespread cheating.
But around the country, dissatisfaction with public schools was on the rise, and momentum for testing continued.
LOUIS V. GERSTNER, Jr., Chairman, IBM: Let me begin by saying good afternoon to all the governors.
JOHN MERROW: Business leaders held a summit with the nation's governors to push for reform. Louis V. Gerstner, Jr., of IBM led the way.
LOUIS V. GERSTNER, Jr.: It's time to set standards and then go achieve them.
We never will know whether anything gets fixed in public education until we establish a system that says, ``Here's what children should learn. Here's what they should be able to do when they graduate. And let's measure their success."
JOHN MERROW: Testing caught on. By 1999, every state had embraced a "business-like" approach to schooling, a fundamental change in American public education. For some educators, these changes just raised more questions.
TED SIZER: Do you want a small group of people deciding the ideas that are going into your children's head, people you can't get at, people who say there's one answer? There's an arrogance to this which- which has been costly.
JOHN MERROW: But pressure to fix public schools overrode concerns about too much testing. Politicians around the country embraced it.
LOUIS V. GERSTNER, Jr.: I think we've educated almost a generation and a half of people who have very, very, poor prospects, and that's awful. We have a very serious problem here.
JOHN MERROW: Nowhere was the problem greater than in California, with the nation's largest public school system. Back in the mid-1990s, California's schools there hit bottom.
MICHAEL KIRST, Stanford University: California finished in 1995 tied with Louisiana for last in reading. We were behind Mississippi in the national assessment of education progress. The California politics was that there was a crisis here.
Gov. PETE WILSON (R), California: California schools must be places of high test scores and even higher aspirations.
JOHN MERROW: Pete Wilson was then California's governor. He called for tougher academics and mandatory testing.
MICHAEL KIRST: I think politicians are responding to a democratic impulse and demand and polls of the people saying, "Kindergarten through 12 education is really messed up in California. You've got to fix it fast!"
JOHN MERROW: Within a year, California had new academic standards in place. But creating its own tests would take much longer.
MICHAEL KIRST: So California said, "Well, we have these content standards over here on what ought to be in the curriculum, but we've got to grab a test off the shelf right away and test the kids, even if it doesn't match the content standards." And it didn't.
JOHN MERROW: The test in California is called the Stanford 9, or SAT-9. It's a widely used, commercially sold, national achievement test. Scores on the SAT-9 are used to publicly rate schools.
JAMES POPHAM, Test Writer: The public in general has bought into the idea that these tests reflect how good schools are. It is simply incorrect.
JOHN MERROW: James Popham, a test writer and a professor emeritus at UCLA, says the first problem with off-the-shelf tests like the SAT-9 is that they're written to be sold all over the country.
JAMES POPHAM: The testing companies are in the business of making money, and they're going to create whatever kind of test seems most acceptable to most people. And as a consequence, the gaps between what is on a test and what is actually taught sometimes are profound.
JOHN MERROW: At Logan High School, near San Francisco, students who recently took the SAT-9 found questions about topics their teachers had not yet taught.
DAVID HUYNH The test is one year ahead of the curriculum. When you're a freshman, you take the world studies test before you even take the course. As a sophomore, you take the American history test before you even take the course. And as a junior, you take American political system and economics on the test before you even take the course. So basically, you're taking a test on stuff you don't even know yet.
MARIE AUSTRIA: Different teachers teach different things. And they teach you different ways to -I don't know - attack problems. And a lot of the problems on there we hadn't learned how to do.
DAVID HUYNH: One question that I really didn't get was that it told me to compare the economic and government systems of New Zealand and, basically, Great Britain. And like, we haven't really gone in depth into political and economic systems. We're really dealing with U.S. history. So I really could not answer the questions to my best ability because I hadn't taken the course yet.
JOHN MERROW: It costs $6 per student to administer the SAT-9 California uses, which is all multiple choice.
AUDREY QUALLS, Test Writer: The reason we tend to go first to a multiple choice is practical. I can measure a lot of things very efficiently, very quickly and fairly cheap.
JOHN MERROW: Audrey Qualls is co-author of the Iowa Test of Basic Skills, another widely-used achievement test. She believes the public reads too much into test scores.
AUDREY QUALLS: It's not an accurate indication of how good the schools are, how well they're performing. That, in and of itself, isn't enough. I want to know whether or not a student asks questions, whether they're part of a conversation, whether they use background information, caring about learning. I can't measure that with a test.
JOHN MERROW: Especially if it's the wrong kind of test. SAT-9 is built to rank students, like the 9th-graders in Judy Hirsch's class, some at the top, some at the bottom, most in the middle.
JUDY HIRSCH: If I'm a good teacher and I'm teaching well and my students are good learners and they're learning well, then they might actually learn the material that I'm teaching them. Well, then they would do better on the test. Well, then they would pull those questions.
JOHN MERROW: By removing questions that most students get right, test writers can produce scores in a bell-shaped curve, with a ranking for every student. But some say its unfair to judge schools with this kind of test.
JUDY HIRSCH: Well, the test is basically saying, "We don't want you to pass." That's what it's saying.
MICHAEL KIRST: Once a lot of pupils know a question, you've got to get rid of it because it doesn't spread the curve.
JUDY HIRSCH: That's one of the reasons why we're- those of us who are opposed to the test are so opposed to the test. It actually doesn't test if you're teaching and learning. It tests what you come to school with, so its really based on your socioeconomic level. It filters out anything a teacher teaches during the school year.
STEVEN WEINBERG: We have no choice. It's like when you play in a crooked game, when it's the only game in town. SAT-9 right now is the only game in town for us.
JOHN MERROW: For California teachers like Steven Weinberg, it's a game with high stakes.
STEVEN WEINBERG: We're in a situation here where we're under the gun, in terms of our SAT-9 scores. Our principal has been notified that he is on probation, based on our improving our scores. So no one here takes lightly the SAT-9.
JOHN MERROW: Weinberg was hired by Frick Middle School in Oakland to figure out a way to get scores up. But in analyzing past results, he noticed a disturbing trend.
STEVEN WEINBERG: OK. Well, If you look down on a single class here, you'll notice the wide swings in scores.
JOHN MERROW: [on-camera] Up 7, up 12, up 46?
STEVEN WEINBERG: Yeah. Here's a student that supposedly gained 46 percentile-
JOHN MERROW: And here's one up 31.
STEVEN WEINBERG: Here's another page. A student just 20- up 22, another student in the same class down 32.
JOHN MERROW: Here's a kid who went up 50 points- down 24. I mean, that's astounding.
STEVEN WEINBERG: Yeah. These clearly are not reliable statistics. The test shows a student down at the 1st percentile one year and at the 51st percentile the next, or the 51st percentile one year and down to the 1st percentile the next. A student couldn't lose that much ability in one year. It's impossible, except if there were brain injury or something.
JOHN MERROW: [voice-over] A more plausible explanation comes from students themselves.
NOWELL NARTE: The majority of this school does not care, does not understand why we take these tests.
MARIE AUSTRIA: They didn't take it seriously, and they didn't care because they know that even if they do poorly on these tests, nothing bad is going to happen to them.
DAVID HUYNH: A lot of students took it as, like, a free week, you know, to goof off.
JOHN MERROW: But for teachers and schools, the test is serious business. Low scores on the SAT-9 can lead to penalties, high scores to rewards.
SCHOOL OFFICIAL: I would like to introduce our governor to all of you, the honorable Gray Davis.
Gov. GRAY DAVIS (D), California: I am very pleased to announce that today we're going to grant almost $47,000 to Disney Elementary. So give yourself a big hand.
JOHN MERROW: Under Governor Gray Davis, California handed out a total of $834 million in cash bonuses to schools and teachers, all based on SAT-9 scores.
JAMES POPHAM, Test Writer: It's really sad that most educational policy makers are well intentioned and install accountability measures involving these kinds of tests in the belief that good things will happen to children. But most of these policy makers are dirt-ignorant regarding what these tests should and not be used for. And the tragedy is that they set up a system in which the primary indicator of educational quality is simply wrong.
JOHN MERROW: California is changing. A custom-made test they began developing several years ago is now coming on line. It's designed to measure what California's students are actually being taught.
In Massachusetts, they tried to do testing right from the very start. But this is not as easy as it sounds.
1st TEACHER: You don't want a kid to get nailed by a question. The kid- again, it's a- it's a question that kid could have a lot of knowledge and now get it wrong because he didn't know that name. That's the kind of question I dont think is a good idea.
JOHN MERROW: Massachusetts calls its test MCAS. It begins here, with committees of teachers choosing the questions for the state test from lists supplied by the test writers. This is the part of testing that's rarely seen.
1st TEACHER: All right, we know we need to have 10 questions that deal with Africa. So let us look from 100 of them, and we'll pick the 10.
2nd TEACHER: If we say this, and we go down this line where we say, "How many questions are you going to have on Africa? How many questions you going to have on Asia? How many questions you going to have on Europe? How many questions you going to have on"- [crosstalk]
1st TEACHER: -how many questions on the test. This is just- we want a bunch of questions to look at. [crosstalk]
2nd TEACHER: Intellectually, I don't think that's necessary.
3rd TEACHER: Well, it is, given the fact that today and yesterday, the questions that we have looked at, we have one on Japan, we have one on Africa, and everything else is European. I'm sorry. I mean, by not doing this, OK, we're not focusing in on other areas that we are telling teachers they need to teach.
SUE SZACHOWICZ, Brockton High School: We were actually dissatisfied as a group with the items we were seeing. So actually, you saw us arguing a bit about saying to the testing company, "You need to give us more questions that we can look at not one at a time, but in fact, more questions around a theme or an area of the world or a big idea."
4th TEACHER: Like the Islamic influence on Africa, the development of Timbuktu- [crosstalk]
5th TEACHER: You know what?
4th TEACHER: I mean, there's a million things- [crosstalk]
6th TEACHER: We don't want 100 questions on Africa, if they're all "trivial pursuit" questions.
7th TEACHER: Right. That's right.
SUE SZACHOWICZ: If I can only pick one item, please let's sure it's the best item we can have.
8th TEACHER: Let's draw up a chart and a- [crosstalk]
9th TEACHER: Yeah. that would be very helpful.
JOHN MERROW: Dissatisfied with the questions, these teachers sent the test writers back to work. Committees like this screen hundreds of questions, weeding out the ones that are vague or poorly worded or that do not relate to what's taught in schools.
BRENDON DILLON, Westwood High School: They have to be directly related. There were a couple that we looked at- and again, we went back and went through the standards and frameworks and said, "This is not in here. Specifically, we cannot ask that question. It wouldn't be fair to the teachers or students." And we have eliminated those questions.
JOHN MERROW: Massachusetts has not cut corners on testing. It develops its own test, built to measure what's taught in school, and it's not all multiple choice.
JEFF NELLHAUS, Mass. Director of Testing: One of the- I think one of the good features of our test, one of the reasons we believe it is a good test, is because it requires students to do a fairly significant amount of writing.
JOHN MERROW: Jeff Nellhaus is Massachusetts director of testing.
JEFF NELLHAUS: We have some multiple choice questions or things that can be tested very well and efficiently with multiple choice questions. But what we call the "open response" or constructed response questions require students to write out an answer. We have one shot here to get at the very central things we want to assess, and we dont want to assess things that are trivial or arcane.
JOHN MERROW: And the scores are going to count heavily for students, not just for schools. If students here cannot pass the test, they will not graduate from high school. In nearly half the states around the country, those will be the new rules.
DAVID DRISCOLL, Mass. Commissioner of Education: The catalyst was the business community. They stepped forward and said, "Look, these kids are graduating without skills." And we refuse- or I refuse to stand by and watch kids given a false diploma, which is what's been happening.
JOHN MERROW: David Driscoll is Massachusetts commissioner of education. He's been a supporter of the graduation test since the idea emerged in the early 1990s.
[on-camera] Have you taken the test?
DAVID DRISCOLL: Yes.
JOHN MERROW: How'd you do?
DAVID DRISCOLL: Well, I didn't score myself, but I would have passed. I literally take it simply because I like the experience of going through it. I find it to be a very valid test.
[www.pbs.org: What is your state's testing policy?]
JOHN MERROW: [voice-over] But it's the high-stakes graduation requirement that makes MCAS so controversial. And the question at the center of the controversy is this: Should one test determine the outcome of 12 years of schooling, decide who gets a diploma and who gets nothing?
GEORGE MADAUS, Boston College: It's bad practice. We don't need tests- we don't need these tests to tell us who's having a hard time and who's in trouble in school. We know- we basically know the answer to that. You can ask any classroom teacher and they can tell you to a faretheewell who the kids are that are having trouble in math, reading, you name the subject.
JOHN MERROW: George Madaus is a testing expert at Boston College.
GEORGE MADAUS: We know that certain populations are poorly served, that there are schools that aren't doing a good job for these students. We know that. But now we've added this test as a quasi-documentary of those problems that we knew- we've known for years exist. And you're not going to test your way out of those problems.
DAVID DRISCOLL: We're trying to correct a wrong, a historic wrong that in this commonwealth means 667,000 young people have been given a false diploma. That's got to stop. And so we're trying to- we're just simply trying to set a standard that says, "Look, this is what you need to do."
JOHN MERROW: MCAS is supposed to tell us which students know enough to graduate, and Massachusetts is spending $20 million a year to develop its custom-made tests. But the tests turn out to have mistakes.
This question appeared on the most recent 10th grade math test. "Which of the following shapes cannot be made by combining equilateral triangles: A trapezoid, a rhombus, a hexagon, or a pentagon?
PETE PETERSON: And you can actually make all those shapes, and that was the problem, because you had to find out which one you couldn't make.
JOHN MERROW: Pete Peterson, a 10th grade student, discovered the mistake. During the test, he spent precious minutes puzzling over the question. He passed, but confusion may have brought his score down, and hurt other students, too.
PETE PETERSON: If you take a lot of time on one problem, you're going to be tired for the rest of the test. I know a lot of people had taken a lot of time for the problem because they knew there was a way to make a pentagon. Like, some of the people that actually wanted to take the test because they knew they had to do it to pass, and so they decided they were going to take the time to try to figure out the problem, and they couldn't.
BILL KENDALL: That test had been screened. People didn't see that you could make a pentagon.
JOHN MERROW: Bill Kendall is Pete's math adviser.
BILL KENDALL: Pete's showing us there's a real flaw in just standardized multiple-choice tests. There's whole types of intelligence that you're not measuring. And in fact, being intelligent and tenacious, as Pete was, can hurt you. Who would you rather hire in a position? Do you want someone who can whip through a multiple-choice test fast and bubble in the correct answers? Or do you want someone, as Pete showed, who's tenacious with problems and who can come up with unusual, creative solutions?
JOHN MERROW: MCAS may not always identify the most creative problem solvers, but can it at least tell us whether students know enough to graduate?
MADELINE VALERA: And I remember my leg was shaking. My knee tends to shake when I get nervous. And I was thinking, "I have to calm down." And while I was thinking it, time was flying by.
JOHN MERROW: When Madeline Valera was a 10th grade student at Boston Arts Academy, everyone had to take the math portion of MCAS. Even though the test results would not count for graduation until the next year, she was nervous.
MADELINE VALERA: It's dreadful. Anything that you might have studied at home, you forget it at that moment. At least for me, I totally blank out when I'm nervous.
JOHN MERROW: Madeline failed. She needed a 220 to pass. She got a 216. But how precise is her score? Can the number be trusted?
JAMES POPHAM, Test Writer: When I first started teaching, I believed in the accuracy of numbers, and many parents still do. They believe that those numbers are so darn precise. Psychometricians, experts in testing, have a term they call the "standard error of measurement," which indicates how likely it is the kid's score could be off by a certain amount every time the kid takes a test. It's a lot. These tests are not as precise as is widely believed.
JOHN MERROW: The public may believe that test scores are always accurate and precise. But tests too are much like political polls. They, too, contain a margin of error.
AUDREY QUALLS, Test Writer: When you're setting a cut score, and if I'm one point below the cut score and I have a standard error of measurement that's plus or minus 3 points, it's very likely that my true ability is above the cut. And if you have absolutely no knowledge of what a standard error means, you're going to assume you have perfect measurement and fail the student. It's a problem.
JOHN MERROW: Could it have been a problem for Madeline Valera? The margin of error on the test she took is plus or minus 6 points.
(on camera): What could that 216 be? How firm a number is that for Madeline?
GEORGE MADAUS, Boston College: She could be as low as 210 or as high as 222, if the standard error is 6.
JOHN MERROW: And so?
GEORGE MADAUS: Well, and so if- if, in fact, it was 222, she should have- she should have passed.
JOHN MERROW: [voice-over] But it's not just one student whose scores are in doubt. In Massachusetts, there are thousands of students like Madeline Valera, students who scored within a few points of the cut.
[on-camera] What can you say about those students' scores?
JEFF NELLHAUS, Mass. Director of Testing: Well, they are what they are. I don't-
JOHN MERROW: Are they- are- how accurate are they?
JEFF NELLHAUS: They're accurate within the level of precision of the test.
JOHN MERROW: Which means that they could be over the cut or under the cut.
JEFF NELLHAUS: That's right. That's right.
JOHN MERROW: [voice-over] Test experts can calculate the number of students who failed when they should have passed. On the 10th grade math test alone, an estimated 3,000 students were misclassified.
DAVID DRISCOLL: The standard error of measure is absolutely a fact, that any time you give a test , any time you have a place, then you worry about that standard of error and what would have happened on another day. My answer to that continues to remain that we are giving four opportunities to retake the test, at a time when kids are growing anyway in their intellectual ability, and that someone who gets a 216 I would feel very confident would be able to score a 220 easily within the next four retakes.
LINDA NATHAN: I don't think ideally we should be setting up situations where we know kids are going to fail. I don't think that failure is a great motivator for most people.
JOHN MERROW: Linda Nathan is principal at Madeline Valera's school, Boston Arts Academy.
LINDA NATHAN: If we think this is important, I think it should be part of a kid's high school record, but not the sole thing. So you can get As, legitimate As, in high school and not pass this test and be told, "Oh, don't worry, John. You still got four more chances." I don't know that I think that's a positive set-up for kids.
JOHN MERROW: Test experts like George Madaus agree that retakes are not the answer. He'd like to include grades and teacher recommendations, along with test scores, in decisions over graduation.
GEORGE MADAUS: The technology has inherent limitations to it. You can get a measure, but there is error in it. And before you make a decision, you should do what a doctor would do. You get other measures of the same thing, put them together and then come up with a clinical judgment about the kid, rather than just take the number and decide that you're going to do something or not do something.
MADELINE VALERA: I don't think they get the real me at all. I don't think a test can tell you- can really tell you what you know.
JOHN MERROW: Soon Massachusetts will allow students who just missed passing to appeal their scores, using grades and other records to show they know enough to graduate. With safeguards in place, some experts say, there is room in our schools for high-stakes testing
JAMES POPHAM, Test Writer: The measurement community is universal in its condemnation of a single criterion, like a single test score, to be used in making an important decisions like denial of a diploma. But to use a test score as one contributor to a variety of evidence to make that decision- that's acceptable.
I believe that properly constructed high-stakes tests, tests that can help teachers teach more effectively, should be used. I think the public has a right to know how well their schools are doing. So to resist any kind of testing I think is disadvantaging the children. You have to create the right kinds of tests, but they can be a powerful force for instructional design, for getting kids to learn what they ought to learn.
JOHN MERROW: In the latest round of testing, MCAS scores improved dramatically, easing fears that great numbers of students would not graduate on time. But there are still questions in Massachusetts and other states about how much testing really tells us about what students are learning.
WILLIAM SCHMIDT, Michigan State University: States have this notion that if they have standardized tests, they're there. And that's not the issue. It's not whether they have them or not, it's what's in them. And that second question is a much more difficult one to deal with, and yet we find states somewhat unwilling to really confront that reality.
JOHN MERROW: William Schmidt teaches at Michigan State University.
WILLIAM SCHMIDT: The real issues have to do with what it is, the substance, the discipline. What are we expecting children to learn at those grade levels? Is it the kind of things that give them a chance in life to compete internationally? And I'm not talking about just on a test, but I mean for their jobs, for their futures. This is what's important.
And this means we have to have a dialogue about what mathematics, what science, what literature, what history, is important for kids to know at 6th, 7th, 8th grade, and then test those accordingly. We don't go there.
JOHN MERROW: Back in Virginia, they are doing the hard work of deciding what all students should know. And in this state, with its rich and painful past, the hardest subject to decide may be history.
COMMITTEE CHAIR: My question is, isn't Robert E. Lee's name essential to the event called the Civil War? I'm just speaking about the name Robert E. Lee.
1st COMMITTEE MEMBER OK, well, we need to go back because we need to clarify how we're going to deal with these names again because if you mention him and you say, "Well, is he essential?" then, of course, Abraham Lincoln's essential. Jefferson Davis is essential. Ulysses S. Grant's essential. Fredrick Douglass is essential. And you are going to get a long list of names again.
JOHN MERROW: This committee is grappling with the difficult task of deciding which historic figures all Virginia students should study. But the process broke down over names- specifically, who to include and who to omit.
MIKE WILDASIN, Fairfax County Public Schools: One measure of our meeting the diversity charge will be the names in the document. And in fact, when I sat at the Fairfax public hearing, I heard public comment that was not favorable because there weren't enough names.
2nd COMMITTEE MEMBER: It is strongly recommended by our committee that the names stay in.
1st COMMITTEE MEMBER: We do not have to put those names in the standards, not before 12th.
COMMITTEE CHAIR: In other words, you would want a document devoid of names completely, Mike?
MIKE WILDASIN: I'm troubled because it seems almost a discussion that's been unfolding for four months, and we seem not to be able to come up with a win-win situation. I wish someone would have an epiphany at this table right now and come up with a win-win situation.
JOHN MERROW: No one did. Arguments over names continued without resolution.
COMMITTEE CHAIR: With regard to a policy statement, I admit that so far, it appears to be a little ragged, but I dont want us to get hung up on that right now. So we'll just proceed with the other questions.
MIKE WILDASIN: People feel- particularly about U.S. history, people feel very strongly, and they feel they know it. It is a very long process to get consensus about that which is regarded as essential understandings for K-12 students. It isn't obvious. The process requires reflection and deliberation. And it's a consensus that's hard-fought, but once realized, I think, will result in standards that are well received.
JOHN MERROW: While the committee struggled over historic figures, behind the scenes, groups of citizens were lobbying hard to add material to the state's history standards. Among the spectators at the meeting was Richard Kervorkian.
RICHARD KERVORKIAN: We would like to see the genocide included in the state SOLs. The Armenian genocide is important because it was the model. It was the first genocide in the 20th century. And so the Armenian genocide has, in a sense, given birth to other genocides.
JOHN MERROW: Kervorkian, a painter and retired professor, is seen by some as a "special interest." It's a term he does not refute.
RICHARD KERVORKIAN: If you don't engage in supporting the people from whom you came, who will?
JOHN MERROW: Armenians were not the only ones seeking to add to the history schools would be required to teach.
ANDREW PROPHETT: People studying the history of very good in grades 3 and 4 might get the impression that African-Americans had no history before 1619, the beginning of slavery.
JOHN MERROW: Andrew Prophett, a high school history teacher, pushed the state to include a study of the ancient kingdom of Mali in the new history curriculum.
ANDREW PROPHETT: Mali is essential because it happened. The people of Mali were a part of the international trade of gold and salt. The people of Mali were a part of the spread of Islam and the exchange of scholars, the building of cities and the exchange of knowledge.
DAVID SAXE: The kingdom of Mali has nothing to do with the beginning of this country. Nothing. Absolutely nothing to do.
JOHN MERROW: The proposal to add Mali provoked strong criticism from David Saxe, who's reviewed history standards in 48 states.
DAVID SAXE: I think you're falling out of what's essential. So instead of looking at that, we look at Greece and Rome. And it's not that these are white people or the history of Europeans. This is the history of ideas, the history of religion, law, and the very things that make this a great nation, the United States.
ANDREW PROPHETT: I want all students to know that what we call progress or the lack of progress is shared by all people. And not any one particular people happen to have dropped out of the sky, with slavery as their beginning around 1619, as the result of a Dutch man-of-war landing off the coast of Virginia. That's not history.
JOHN MERROW: In the end, the committee decided to include Mali. The Armenian genocide was left out.
[www.pbs.org: Challenges of setting standards]
WILLIAM SCHMIDT, Michigan State University: It's hard work to grapple with the fundamental issue of what's really important for all children to learn. We get hung up in the politics, and then we leave our kids out of the equation. And it's for them that we should be having this kind of serious dialogue.
JOHN MERROW: Meanwhile, in Virginia's schools, the debate over standards comes down to test scores. And the latest results are back. In high-poverty districts like Richmond, most schools are still struggling. But for the rest of Virginia, there was reason to celebrate.
The statewide pass rate doubled from nearly 20 percent to 40 percent. There was good news for Baker Elementary. And Varina High School passed the test, too. But behind all the celebrations lie real concerns about the effect of mandatory testing on the quality of teaching and learning.
ROBIN COPPOCK: All right, now, tomorrow we have our six-week test. Remember that the majority of the test is multiple choice.
JOHN MERROW: Robin Coppock teaches algebra at Courtland High School.
ROBIN COPPOCK: Most of the time in the past, I've always given problems where they showed their work. They got partial credit. I'm turning more and more into multiple-choice-type questions because that's the type of question that they're going to have to answer on the test.
All right, the next thing I'm passing out is the SOL practice for the week. This is going to be due on Thursday.
And you know, we have to give them multiple-choice questions because we have to train them how to take a multiple-choice-type test. So I end up doing things like, "OK, guys. Remember you really don't need to know how to solve this problem. You can take all the answers and substitute them in."
So one thing that I see that's happening is the kids, particularly the smarter kids because I've got smarter kids- they know ways around this kind of thing. And instead of solving the problem, they'll take the answers one by one, plug them in and see which one works. So they really- they're manipulating. They really don't know that concept. They know how to substitute a number in.
Remember, I told you when you take this test, they'll have a choice like this. They'll have a choice like this. But why is one better than the other?
JOHN MERROW: When preparing for tests becomes the first priority, teachers say there's a price to pay.
HERBERT MORRIS: What good does it do to have a kid do a project, when he is not going to be tested on it?
JOHN MERROW: To teach the formula for calculating distances, math teacher Herbert Morris used to take students on field trips to the James River in Richmond. There he'd present them with a real-life problem.
HERBERT MORRIS: "How is it across this river? How can you calculate it? I'll give you a sextant or a compass, and you can measure the angle. Do you know how to measure the width of this river without getting wet?"
JOHN MERROW: [on-camera] You don't have time to do that?
HERBERT MORRIS: Not truly.
[www.pbs.org: More on the effects of testing]
JOHN MERROW: [voice-over] Today, instead of going to the river, Morris goes through math problems more likely to appear on the state test.
[on-camera] OK, you miss it. But would the kids learn more doing it that way?
HERBERT MORRIS: They would learn so much more applying it, going out and getting dirty and doing it.
JOHN MERROW: [voice-over] American students are going to be spending less time solving problems down by the river and more time taking tests with number 2 pencils. Congress has passed the president's plan to test all students in grades 3 through 8 in math and reading. But how far along are the 50 states in deciding what every student should learn and then building tests that truly measure what's taught?
WILLIAM SCHMIDT, Michigan State University: It's a chaotic situation, where none of that coherence exists, nothing lines up. And so you have a teacher being torn in so many different directions, standards saying this. The textbook is so big and has so many things in it, they know they can't do it, but they try to do it. The tests are in another direction. That's not a professional environment.
JOHN MERROW: And will the new law come with enough money? In a weakened economy, states might be tempted to go back to using cheap, ready-made tests.
JAMES POPHAM, Test Writer: If we have more of the same kinds of tests we're currently using, good things will not happen in American education. We have to create tests that really do reflect how well teachers have been teaching. Those kinds of tests will allow, I think, public education to survive. The kinds of tests that we're using now is setting up public educators for absolute failure.
TESTING OUR SCHOOLS
PRODUCED AND EDITED BY
John D. Tulenko
CORRESPONDENT
John Merrow
SENIOR PRODUCER
Sharon Tiller
STORY EDITOR
Ken Dornstein
ASSOCIATE PRODUCER
Tania C. Brief
LEARNING MATTERS STAFF
Donald Devet
Christy Helbing
Shae Isaacs
Alexis Kessler
Colin Mutchler
Valerie Visconti
Georgia West
FIELD CAMERA
John Kabasakalis
Bryan Mong
Alfred Shapiro
Kirk Hayden
Ken Willinger
Brian Dowley
David Paredes
FIELD AUDIO
Glenn Piegari
Billy Britt
Morgan Schmidt-Feng
Keith Hindman
Charlie Ethridge
Tom Williams
Bob Freeman
ADDITIONAL EDITING
Terry Cafaro
Arnie Harchik
ONLINE EDITOR
Arnie Harchik
SOUND MIX
Jim Sullivan
SPECIAL THANKS
Ainslie, Saskia, Ian
IN VIRGINIA
Charles B. Pyle,
Virginia Dept of Education
Janet Binns,
Henrico County Schools
Karen Dorgan,
Mary Baldwin College
Mike Bedwell, Principal,
Spotsylvania High School
IN MASSACHUSETTS
Jonathan Palumbo,
Massachusetts Dept of Education
Mary Grassa O'Neill,
Superintendent, Milton Schools
Anne Wheelock
IN IOWA
Patti Verlinga,
Drake University
Kathy Paarde-Kooper,
Drake University
IN CALIFORNIA
California Dept of Education
Anthony Cody
Ruth Ann McKenna, Supt,
New Haven Unified School District
Gerald Wolfe, Frick Middle School
ALSO THANKS TO:
Jackie Babcock,
Michigan State University
David Berliner,
Arizona State University
Meredith May,
San Francisco Chronicle
ARCHIVAL MATERIALS
Armenian National Institute
California State Library
C-SPAN Archives
IBM
New York Public Library
The NewsHour with Jim Lehrer
FOR FRONTLINE
PRODUCTION MANAGER
Tim Mangini
ON-AIR PROMOTION
PRODUCER
M.G. Rabinow
SENIOR EDITOR
Steve Audette
AVID EDITORS
Michael H. Amundson
Arnie Harchik
John MacGibbon
POST PRODUCTION
COORDINATOR
Patricia Giles
POST PRODUCTION
ASSISTANT
David McMahon
SERIES MUSIC
Mason Daring
Martin Brody
COMMUNICATIONS
MANAGER
Erin Martin Kane
SENIOR PUBLICIST
Christopher Kelly
PROMOTION WRITER
Jennifer McCauley
PROMOTION DESIGNER
Dennis O'Reilly
PROMOTIONS ASSISTANT
Jenna Lowe
FOUNDATION GRANT MANAGER
Jessica Cashdan
SECRETARY
Mary Sullivan
ADMINISTRATIVE ASSISTANT
Danielle Gillis
COMPLIANCE MANAGER
Lisa Palone-Clarke
LEGAL
Eric Brass
Jay Fialkov
PARALEGAL
Adrienne Armor
UNIT MANAGER
Douglas D. Milton
BUSINESS MANAGER
Tobee Phipps
WEBSITE PRODUCTION
ASSISTANT
Todd Goldstein
WEBSITE ASSOCIATE
PRODUCERS
Sarah Moughty
Kimberly Tabor
WEBSITE COORDINATING
PRODUCER
Stephanie Ault
WEBSITE PRODUCER/
DESIGNER
Sam Bailey
WEBSITE MANAGING EDITOR
Wen Stephenson
EDITORIAL RESEARCHER
Catherine Wright
REPORTER
Dana Reinhardt
COORDINATING PRODUCER
Robin Parmelee
SERIES EDITOR
Karen O'Connor
SENIOR PRODUCER
SPECIAL PROJECTS
Sharon Tiller
EXECUTIVE PRODUCER
FOR SPECIAL PROJECTS
Michael Sullivan
EDITORIAL DIRECTOR
Marrie Campbell
SERIES MANAGER
Jim Bracciale
EXECUTIVE EDITOR
Louis Wiley Jr.
EXECUTIVE PRODUCER
David Fanning
A Merrow Report produced by Learning Matters in association with WGBH/Frontline
(c) 2002 LEARNING MATTERS, INC. and
WGBH EDUCATIONAL FOUNDATION
ALL RIGHTS RESERVED
FRONTLINE is a production of WGBH Boston, which is solely responsible for its content.
ANNOUNCER: This report continues on our Web site, where you'll find a guide for parents with information on testing in your state, a report on the surprising politics behind the president's landmark education bill. Read the extended interviews, plus links and background readings, and something new. Find out on our Web site if this program is re-airing on your station and when. Then join the discussion at PBS on line, pbs.org, or write an email to frontline@pbs.org, or write to this address. [Dear FRONTLINE, 125 Western Ave., Boston, MA 02134]
Next time on FRONTLINE: As the war between the Israelis and the Palestinians intensifies, FRONTLINE goes behind the lines on both sides. How did a conflict once fought with stones escalate towards all-out war? Battle for the Holy Land next time on FRONTLINE.
To obtain a VHS copy of Testing Our Schools, please call 1-800-PLAY PBS. [$29.98 plus s&h]
National corporate funding for FRONTLINE is provided by Earthlink and by NPR.
In Denver, Skopje, in Teheran, Omaha, in Istanbul, in Hong Kong, Belgrade, in Decatur, in Seattle, Beijing, in Pittsburgh, in Johannesburg, this is NPR News.
Major funding for this program was provided by the Annenberg Foundation, with additional funding from the Pew Charitable Trusts, Carnegie Corporation of New York, and the William and Flora Hewlett Foundation.
FRONTLINE is made possible by contributions to your PBS station by viewers like you. Thank you.
home · no child left behind · challenge of standards · testing. teaching. learning?
introduction · in your state · parents' guide · producer's chat · interviews
video excerpts · discussion · tapes & transcripts · press
credits · privacy policy · FRONTLINE · wgbh · pbs online
some photographs ©2002 getty images all rights reserved
web site copyright WGBH educational foundation
| 
