... If you spend time building your standards, is there a problem with buying a test off the shelf?

You bet there is. One of the key terms of this standards movement is "alignment." And one of the things that really is crucial here is that the tests that one uses to measure progress against the [curriculum] standards be specifically developed to measure progress against those standards. By definition, if you take a test that was developed for another purpose and try to use it to try to measure progress against your standards, you're sending very mixed signals to the field. ... We have a long history in American education of using standardized, off-the-shelf tests for different kinds of purposes. Those tests, by and large, are designed to be independent of curriculum.

We always used to think here that "teaching to the test" in the American context has a negative connotation. If you think about the role the SAT test has played in our system, historically it was deliberately designed to identify talented kids who perhaps were stuck in schools where they weren't being given a very rich curriculum. It was deliberate. Ironically, it was clearly designed as part of an ... access strategy after the Second World War, to try to broaden the pool of kids particularly from small, isolated high schools who were seen as having enough academic talent to be able to come East to the more selective colleges.

What we've been trying to do in the standards movement is make the shifts to a set of tests that really are specifically designed to measure the degree to which kids have actually mastered the curriculum that they have been taught.

So develop the standards, [then] develop a test.

Right. You begin with the standards. Define what it is that we want kids to know and be able to do, so we can send a very clear signal to teachers, to parents, to kids: Here's what's really important in education. Then develop a set of assessments specifically to measure the degree to which kids have mastered those standards, help districts really develop curriculum materials and professional development, again, directly tied to the standards. So all of the pieces of the system line up. That's the notion behind this alignment term that gets thrown around so often in education jargon.

Describe MCAS.

MCAS is the Massachusetts state assessment. It really does fit this criterion of being an assessment that was very specifically developed to measure progress against the Massachusetts state standards.

Is it a multiple-choice test?

It has some multiple-choice elements in it, but it also has some constructed response and more open kinds of responses. ...

So you write essays? You solve problems?

Yes. You write essays. ... You solve problems. And you have, as I say, some multiple-choice items as well. It is also an untimed test; that is, even though there's kind of a designated period for kids who take the test, it's also possible for kids who need more time to have more time on the test.

Another feature of the Massachusetts test -- which is quite unusual, only a few states do it -- is that the Massachusetts test each year is released to the public. All of the items that count for kids are publicly released, so that parents and teachers and the general public can see what was on the test. ...

The theory was, end the mystery, take away the notion that maybe there's some deep, dark secret here.

So that means you develop a new test every year?

Right. You have to develop a new test every year. It's expensive, but the Massachusetts policymakers decided that the tradeoff, in terms of public understanding and public confidence, was worth it. The other thing that Massachusetts does as part of its MCAS strategy is on the state website, [it releases] examples of student answers, student work products at various performance levels. So if you want to know what constitutes passing work, what constitutes proficient work, and what work is not quite at the level that the state is seeking, you can see examples of work with commentary. ...

As you probably know, ... MCAS is controversial in Massachusetts. Everybody has an opinion about it. But the nice thing about this more public, transparent process is that folks like me -- who think that the MCAS is a good test and it's worth teaching to -- can say to people, "Go take a look at it. See for yourself. Take a look at student work, and answer the question: Would you be satisfied granting a high school diploma to a student whose performance is below the cut point?"...

Suppose it becomes apparent that 50 percent of the kids are not going to be able to pass this high-stakes test, whether it's Massachusetts' test or some other state's. What will happen politically?

I think [a] 50 percent failure rate [is] clearly not sustainable, politically. Something will have to give. ... [I]t's hard to generalize about these issues across states. The Massachusetts test is first given to kids in the 10th grade, and they have four additional opportunities to take the test and, obviously, two more years of instruction ... [under] current policy, before a student would be denied a diploma. ...

Let's say you're governor of Massachusetts and you're looking at a 50 percent failure rate on this high-stakes graduation test. What do you do?

There's a lot you need to do between the time you get the early signal that you might be looking at rates like that, and making any final determination. ...

Would you lower the bar?

I would not lower the bar. Again, this assumes that one has gone through a public process of trying to define what it is that we think is reasonable for kids to be able to do, what kind of work -- especially in reading and writing and math -- is reasonable to expect kids to be able to [do] in order to earn a diploma.

I would stick to the bar. I would explore transition kind of strategies, I would make sure that we had provisions for waivers of various kinds or appeals processes, so that students could make a credible case that they actually are able to meet the standard, even if they can't show it on a test. I think there are ways to be able to get at that ... unless one concludes that the standard is an inappropriate standard. ...

Unlike many other states, Massachusetts did not adopt an incremental strategy in setting its standards or in developing its test. That is, it didn't say, "Here's where our students currently are. Let's raise the bar a little bit and proceed incrementally, step by step." Texas was an example of a state that has done that and in political terms, that strategy has really worked very well for Texas. They have not had to face anything like these kind of potentially catastrophic failure rates. They've built confidence in among the teaching professional and among the public that, in fact, their schools can get better. They have reduced racial disparities in performance. So that strategy, in that context, has worked.

Massachusetts adopted a different strategy, perhaps a higher-risk one. But its notion was, "Let's really develop a set of ambitious standards and assessments that reflect where we want our students ultimately to be. And then let's really push very hard to do whatever it takes to try to get virtually all kids over that bar."

I think what Massachusetts policymakers and educators would say now is, "Obviously, we know we've got a problem." But in Massachusetts, the heavy focus right now is on additional intervention, support, remediation.

More help?

More help. For the class of 2003, "There's still time" is the theory. We will get shortly the results from last spring's test. There is another, more focused retest opportunity for kids in the middle of this year. The state has committed $80 million in special remediation and support dollars for kids who are at risk of not being able to get over the bar. And there's a pretty intensive effort focused on trying to design this extra help and support.

So Massachusetts is not backing off?

Massachusetts is not, at this point, backing off. Now ... if Massachusetts gets to early spring of 2003 and is still looking at failure rates that are close to the current level of failure rates, obviously I think people will pause and say, "Hmm, maybe we need to look at some other alternatives."

But the urban superintendents in particular, led by Tom Payzant, the Boston superintendent, have said, "Do not back off. Do not pull the rug out from under our efforts to really get our systems focused on trying to help all kids get over this bar. If there has to be a set of policy modifications, let's do that after we've got more evidence in, after we've had more opportunity to see the remediation and extra help kick in."

... [W]hen California gave the SAT-9, it was using it to evaluate schools ... and then teachers, and gave out money. And we have all kinds of wonderful stories of kids who just didn't give a damn. They just blew off the test.

Yes. I think this is a really tough problem. ... That's less of an issue for younger kids. But by the time you get into middle school and particularly into high school, kids are smart enough to know whether or not doing well on this test is going to have any consequence for them.

If you have an accountability system in which there are big stakes for the adults, but no stakes for kids, that's not fair to the adults. I worry actually more about those states that have attached consequences for kids, but no consequences for the adults, if you follow me. Massachusetts, I would have to say, is so far lagging a bit behind in the development of an accountability system for schools, as distinct from an accountability system that focuses on kids. So the issue here is, how do you get the balance right? ...

My own bias here is that accountability for kids ought to be one of the last pieces of the puzzle that kicks in. In my view, we need to be pretty confident that we have provided kids with an aligned curriculum; that is, a curriculum that's aligned with the standards and with the tests; that kids have access to extra help as they are having in Massachusetts; that parents have been given early notification, so they can use their own resources to make sure that their kids are getting extra help and support. And that there's really been adequate professional development for teachers, so that we can be reasonably confident that teachers have the knowledge and skill to be able to teach to these standards.

If the kids don't make it in large numbers, what do we do about the adults? Do we say, "You're out of here"?

For me, this shouldn't be about blame and punishment. The goal here is to try to not only equip our teachers with the knowledge and the skills ... but to really try to take every step we can to ensure that kids are given the support they need. Now, if schools persistently fail to educate their kids after additional help has been offered, then I think that ultimately there do need to be sanctions for the adults in the system. But again, that's a last step for me -- not an early step. ...

Standards and accountability is a kind of a business model. "Here's what you're supposed to be, and we'll measure it." ...

I would say this has less to do with business, per se, and more to do with a set of principles about organizations and how an organization will change. [Business leaders] have been among the strongest advocates for the notion of, first, at least being clear about what the purpose of the enterprise is -- and that's really what the standards represent. Second, committing to measurement. And third, thinking harder than educators are accustomed to about incentives and rewards and sanctions.

Obviously, I work for an organization that has strong business leadership, as well as strong public sector leadership from governors, but ... the business advocacy has been extremely important for this movement.

I've heard some people say, "Well, educators don't quite get business. And so their notion of a bottom line is maybe a little bit simplistic." ...

Admittedly, the bottom line in education is more complicated than it is in many businesses. And there are limits to this business analogy. Most of the corporate folks that I've worked with, ... who take the time to actually learn about schools and go visit schools, pretty quickly develop an appreciation for the fact that these private sector principles cannot automatically and easily be applied to the public sector. They know that the public sector folks operate under a somewhat different set of constraints, and how difficult it is to actually make change. But what they bring is a sense of urgency and also confidence that, in fact, organizational change is possible. ...

The business people I've talked with do get it. I mean, they understand it's a complicated enterprise. I've just been intrigued by hearing superintendents using the language of business. ...

Yes, I get put off by some of that myself, I have to confess, and I do think one of the not-so-healthy consequences ... is it sometimes tends to devalue the other purposes of education. And again, most of the business people I talk with are very clear that the purpose of American schools is not simply to produce the next generation of workers, but to produce thoughtful, reflective citizens, and people who can lead sort of productive personal lives, as well as be productive economically.

And I do think that sometimes superintendents ... take this business rhetoric and they sound as if they think that the only thing that they're doing is producing people to take part in economic life, if you will. ... I do think it's overly reductionist, in terms of the purposes of education.

Let me get your reaction to the to the Bush proposal for testing all kids grades 3-8 in math and reading. Is that a good thing?

It has clearly worked for Texas. This is a big, varied country and I'm a little cautious about wanting to take a model that has worked in one place, and think it can automatically produce the same kinds of results elsewhere. [My] cautions have a lot to do with my anxieties about the quality of tests and the way in which tests will be used.

If we're not careful, the states may take the sort of easy way out, and go to the least expensive, off-the-shelf tests to fill in the gaps in the current testing programs. As you probably know, most states are not currently testing every grade in reading and mathematics, grades 3-8. So a lot of states are going to be in the business of having to develop or purchase tests to fill in the gaps, if this legislation passes in its current form. ...

So one concern really has to do with making sure that there aren't disincentives to really invest in quality tests that really are aligned with standards. ...

Are tests driving the system?

In some places, tests have taken on an undue part of the load here, if you will. This is supposed to be a movement about ... "standards-based" reform, and in some places, unfortunately, it's becoming "test-based" reform, or "accountability-based" reform.

[We're] trying to help people get back to the centrality of the standards themselves, that is, asking some pretty tough questions: Do we have the standards right? Are they clear? Are they measurable? Are they reasonably economical? Are the standards focused only on what really are the most essential things that kids need to know? ...

In some places, what we've seen is people sort of moving right past the standards, and simply focusing on the tests themselves. And there you do get some distortions in the system. If you've got high-quality tests, tests that in some measure one could describe as worth teaching to, then I worry less about the role that tests have played. What I mean is, if you've got tests that can't be narrowly crammed for, and that don't drive down instruction to a kind of rote drill-and-kill kind of strategy, then ... I'm more relaxed about the fact that tests have become more central.

I should say, no education program should be focused solely on preparing kids for tests. Even the best tests are not worth teaching to exclusively. And the schools that I have been in a variety of states -- some with high-quality tests, some with not-so-high-quality tests, some with high stakes, some with no stakes -- ... the common denominator in good schools is that, yes, people are mindful of the tests. They know there are tests out there, but that's not what they're focused on. They're focused on high-quality instruction for all kids, based on a sort of rich, challenging curriculum, ideally aligned with standards. ...

[On] the tests that are used, [the] questions are, by and large, multiple choice.

Increasingly, states have at least some "open response" kinds of items and ... more and more tests are testing writing. You test writing, not by multiple choice, but by having kids write.

And that's a good thing, moving away from multiple choice?

Absolutely. ... There are two issues [and], as you know, these all get entangled. There's testing time. Obviously, if you're having kids do more challenging kinds of tests that force them to write and explain, that's going to take more time. Those can't be scored by machines. Those require human beings, with judgment. ... And that means, again, there is more of a time lag between the time of a test, and the time that results come back.

And it's more expensive.

And it's more expensive. There are exceptions to that. New York State, for example, uses teachers to read and grade the Regents, so they are able to get quick turnaround. And I think, again, we're in an arena where there are lots of tradeoffs. ... Relying solely on multiple-choice tests -- which are, by definition, the least costly -- they also give you the least useful information. ...

It's important to keep reminding ourselves, tests are only worth doing if they provide valuable, useful information for the people who need it -- for teachers, for parents, for policymakers. ... [B]y and large, I think teacher-administered and scored tests as the kind of tests that teachers routinely do in the course of their work are probably more useful for diagnostic purposes than almost anything you can get from a state test.

But where good state tests are really important is in being able to give you guidance about areas of weakness and strength in the curriculum and, indeed, the instructional program -- helping at the district level, identifying what are some concepts in mathematics that we really need to work harder on? How do we focus our professional development? In other words, they're really for system improvement. ...

We watched some people in Virginia working on standards, actually rewriting the history standards, and arguing, "Well, do we need Frederick Douglass or Robert E. Lee?" And other people arguing, "Well, we didn't have to study the Armenian genocide," or "We need to study Mali." ... Is it always so ... incredibly complicated?

Yes, if you involve more than one or two people in doing it. By definition, it is contentious. And in my view, appropriately so, particularly when you get into these contested areas like history. As we're finding out, even mathematics -- which some of us thought would be relatively easy -- [has] turned out to be not so easy. I view that as healthy, with one big caveat. And that is that somebody in the standards process, at the state level, needs to impose some really tight constraints on the amount of time that any particular piece, a subject area, can occupy in a curriculum.

What I'm driving at is that, all too often, what happens in these debates is they get resolved by including both your pet ideas and my pet ideas. ... [They] solve the argument by adding everything. And you then have standards documents that begin to look rather like our textbooks. They're fat. They're voluminous. They include everybody's pet ideas, and they encourage teachers to kind of skim superficially over [material]. The name of the game becomes coverage, not focusing on a few things and really going at them in some depth. ...

We saw one teacher in Virginia talk about how he was racing to get through everything. And he likened it to cramming food down someone's mouth: No matter how good the food is, they're not going to enjoy it.

This is where some of the more thoughtful critics of the standards movement, I think, have a really important point to make. ... In some ways, the most important criterion [to consider when creating standards] is being able to answer the question, Are these concepts really the most essential concepts, ideas, pieces of knowledge that kids really need? And as I said, in a lot of standard-setting processes, people have not been disciplined enough to say, "We really are going to pare down and pare down and pare down and really try to focus on what's most essential."

So you get a laundry list?

So you get a laundry list. Right. ...

Who gets hurt if you end up with a laundry list?

Well, this comes back to this point about the importance the tests have come to play in some states, at least, having almost cast the standards back off into the background. If the standards documents aren't useful for teachers in guiding curricular choices, in guiding instructional strategies, and useful to system people in guiding professional development programs for teachers, then the tests become in effect the de facto standards. And particularly, if you start attaching consequences to the test, people will say, "Look, I can't get through these standards documents. They're too voluminous. ... They may be nice statements of what we want kids to know and be able to do, but ... what do I do tomorrow in my classroom?"

I should say, by the way, that's been a big missing piece in a lot of this. ...

What's the missing piece?

A big missing piece is curriculum. One of the theories here was that the states would set the standards, would define the goals here, and would set the mechanism for measurement. But they would then leave it to districts and schools to figure out, "OK, what do we do day by day, week by week?" ... And in some districts, which have ... a long tradition of building their own curriculum, that strategy has worked fine.

But we've got 15,000 school districts in the country; a lot of them small, relatively isolated, without much of a tradition of curriculum building other than simply making choices about texts. And to me, one of the most interesting things that's happening in this movement is the increasing cry from teachers, including from the teacher organizations, for more help on the curriculum side. ...

What's your biggest fear?

... My biggest fear is that we won't have the patience to really stay with this strategy long enough to work through a lot of these very difficult, thorny issues: Getting the standards right; getting the tests right; really figuring out what high-quality sustained support for teachers looks like; getting the curriculum supports in place.

And how long will this take?

Some of our critics like to say, "Look, the 'Nation at Risk' report ... was 1983. [It's] almost two decades later. Haven't we given this movement time enough?" If you ask the question, even in the states that have been furthest along and had really good leadership, how many years have there really been changes in classrooms? ... You're talking about three or four years.

My own state, Massachusetts, [is] a good example. A law was passed in 1993. The first set of consequences for kids [would be in] 2003. Ten years looked like ample time. The reality is, changes in classrooms have really only kicked in [during] the last two or three years. ...

I think it's probably going to take a generation before we really can make the judgment. ... It took us a long time to get into the condition that many of our schools are in. This is not something that is going to happen in a year, or two, or three. ...