And the way you're supposed to puzzle this out is--again, it's number 25 on
the map.
Well, I mean, you know what you eliminate? You eliminate 11 because it's
the least. You, the, the thing you think of immediately--5:05, well, the next
time it does it is 6:06. That's the obvious answer. You can quickly go from
12:11 to 1:01 is the right answer. That's the shortest amount--to, so from 12,
from 12:12, 12 minutes after 12 to, 11 minutes after 11. That's less than an
hour. So 61 can't be right. Sixty can't be right. We know 11 can't be right
because it's the smallest. Even if you can't get any further, you're down to
two choices. 30 and 49. And you can guess between the two and you'll increase
your score on the average if you guess when you've eliminated more than one
item.
Again, I have no clue to how to solve that problem anymore. But I've
boosted my score on the average by five points. Each item you get right instead
of wrong is worth 10 points. My odds of getting that right are one half by
guessing between two items.
Are you saying the SAT is a guessing game?
Part of the SAT is a guessing game. Fair Test's position is reasonably new
on it. You know, we've never said that the SAT is not measuring something
meaningful. There's a little part of meaning. There's a little part of
background. There's a little part of schooling. But there's a lot of
test-wiseness. There's a lot of how shrewdly you can play the game. There's a
lot that can taught in coaching courses that has nothing to do with any of the
skills you need to succeed in college or in life.
Wayne Camara says it's a measure of verbal and mathematical reasoning
ability.
Well, Wayne Camara, who I know is a decent person, maybe, he believes that.
But if you talk to representatives of tobacco companies, they tell you their
products are good and healthy and don't lead to diseases. And auto
manufacturers have told you that their cars don't burst into flames. And back
at the beginning of the century, you know, stockyard owners claimed that they
were slaughtering the animals in a clean way. I think we've learned to trust
not to--or not to trust those who profit from manufacturing products, as the
sole source of information.
Yes, one of the things the SAT is measuring in small part, are those
skills. But it's measuring a whole lot of other things. And when you use the
SAT as the major factor--or worse, the sole factor--to make high stakes
decisions, to define what is merit, you're relying not on what most people
think of is merit. But on these very trainable skills that coaching courses and
others help people learn.
Does coaching work?
There's no question that coaching works. The only people in this country
who don't believe coaching works are the employees of the test manufacturers.
Because they've learned from the past that someone who works there and says
coaching works, needs to find another job.
Coaching works anecdotally. I've seen my own son take a coaching course.
And his score went up 150 points. Not because he became smarter or had more
aptitude, was better prepared for college work, but because the coaching course
taught him how and when to guess. And he's a very careful thoughtful kid.
Probably the sort of kid that you want in college or in life. But that doesn't
help you on the SAT. You get points only when you get items right. And that
means you need to complete them all. And the coaching courses teach you how to
pace yourself and teach you how to guess. That's not aptitude. That's not
merit. But it's high scores on the SAT.
John Katzman says, pull the plug. Kill it. It's a bad test. It's a
scam.
Well, we don't disagree with John Katzman. Fair Test's position is that the
SAT should be optional in the admissions process. Right now, kids do all kinds
of wild things to boost their admissions portfolio profiles. They bake cakes.
They submit video tapes and projects. If kids want to take tests on Saturday
morning and fill in bubbles and think that that adds something to their
profile--let them. But a system right now, where a couple million kids a year
are forced to go to the SAT--and another million and some through the ACT, it's
competitor, is a tragic waste of educational resources and money. And ends up
dis-forming the educational process by emphasizing the very narrow range of
things an exam like the SAT or the ACT can measure.
ETS research and reviews the strong evidence that ETS can manipulate
test content to change score averages for males and females.
Well, that's absolutely correct, and probably the strongest provable bias
of the SAT. What Fair Test has proved it a number of times in federal court and
elsewhere, is that it's biased against females. The sole scientific claim made
by the SAT--when you get down to the bottom line and strip away all the
rhetoric and nonsense--is its capacity to predict first year grades. Well,
young women get higher grades than young males across the country in colleges
despite the fact that they earn lower SAT scores by about 40 points on the
average. There's only two ways to square that circle. Either all the colleges
in the country are wrong, they're biased towards girls and give them higher
grades than they deserve. Or there's something fundamentally flawed about the
test.
Well, a number of researchers asked the Educational Testing Service and
College Board--have thought through that problem. And they have shown that you
can--by adding different types of questions to the exam--skew the answers very
differently so that girls get the same or slightly higher score than males. And
in fact, ETS researchers have been able to manipulate other tests to
substantially reduce or reverse the score gap between blacks and whites.
Talk about that.
Well in the example of white and black score gap--which, by the way, is a
red herring that Fair Test is always slapped over the head of with, that it,
some claim that we believe somehow that the whole gap is caused by bias. We
never said it. We don't believe it. Anybody who thinks that the differences
between blacks and whites and test scores in our country, are only the result
of test score bias, is not someone who belongs in our organization.
But to deny that there is some bias, is also ignoring a piece of the
problem. Yes, some of that gap is a result of differences in preparation,
opportunity--going back to prenatal nutrition. But if tests reinforce that by
exaggerating the nature of the gap by the way the tests are constructed, so
that the only items that can added to the SAT are ones in which those who have
already scored high, perform well, so that items that minority groups might do
well on and whites don't do on, they can't be added to the SAT or other
standardized tests based on the definition of how they're put together. So it
becomes a reinforcing cycle.
Is an item performing correctly only if high scorers get the question
right more often than low scorers.
One of the great joys of working at Fair Test is we get to read all the
publications that ETS and the College Board and other testing industry
advocates put forth, because and they have hang themselves with their own
words.
Yes, that is exactly how the test is put together. That means if you tried
out a question on say, you know--what's the major ingredient of a tortilla? And
Mexican American kids said corn quite regularly. And white kids didn't know or
said wheat--but corn is the right answer. That item wouldn't appear on the SAT
because the wrong group got it right. Mexican Americans, on average--because of
language--score lower than whites on the SAT. Such an item doesn't fit the
formula and it's thrown away in the pre-testing. So again, self reinforcing.
It's a vicious cycle.
Test takers were able to answer nearly 40 percent of reading
comprehension questions correctly without reading the passages
themselves...
I read that report when it first came out. I said, well that's interesting.
And then, I did it myself. I actually did it when they launched the new SAT,
before a bunch of reporters at the College Board's annual conference. They
handed out this sample of the new, more complicated reading problems. And I
said, well let's look at this item without any knowledge and see how it works.
The reading problem compared a funeral speech by Pericles and Lincoln's
Gettysburg Address, which most well educated people have some knowledge with. I
didn't read the passages. I read the questions. I got 11 out of 13 correct
without ever reading the passage.
Why is that important through, other than to show how smart you can be, in
a way that the test supposedly isn't scoring? It's because kids who've been
coached know that. So if you're running out of time and you have this lengthy
reading passage in front of you, and then you get to the questions. Instead of
reading the passage--times up. You get no points. You read the questions and
answer them without regard for the passage. And you'll get some points. Every
item you get right--another ten points. You get five of those 13 items right--
you're up 50 points on the SAT. Bang, like that. Does it mean you have more
knowledge or ability or merit? Probably not.
There's a suit that was just filed about ten days ago, on behalf of
eight kids who didn't get into UC Berkeley, by five legal defense groups. What
they're claiming is two things. One, they don't have access to AP courses state
wide in proportional numbers. Two, that UC relies too heavily on SAT's.
Therefore, because of the gap, it's discriminating against them. Can you talk
about that?
Fair Test worked very closely with the civil rights groups that put that
together. And I did a lot of research personally, for that suit. What they're
saying there is that the SAT is actually a very good measure of cumulative
disadvantage. And if that's what we're trying to reward in our systems and
define it as merit--yes, the SAT performs a pretty decent service. But if you
want to define merit as something much more meaningful, the SAT falls way
short.
What they say, and one of the plaintiffs --you, take, a kid in California
who's a child of migrant farm workers, who despite the background, and despite
moving around, and despite possibly having to drink poisoned water in the
formula--still excelled in high school. Got a 4.0. Was a leader in groups. And
probably got quite a decent score on the SAT. That person was not admitted to
UC Berkeley. And in his or her place was someone who grew up in Beverly Hills,
had all the advantages of a nanny who read to them and parents who were college
educated in the best of private schools, and an SAT coaching course. And that
person had similar grades. More AP courses because they're in Beverly Hills and
not in the Valley. And a higher SAT score. Possibly in part because of
coaching. How can you possibly say that the kid from Beverly Hills has more of
this mystical thing called merit? You got to at least say that the migrant farm
worker has at least the same, or maybe more, because he's over come his
background. Yet a system that relies heavily on test scores, ignores all those
other components. And says it's only how well you do on this very narrow
measure that matters.
How can they possibly take a person who scored lower than someone
else?
Well, first of all, the test makers themselves in their technical
manuals--which are far better than their advertising--say a 100 points
difference between two candidates is absolutely meaningless. It's within the
margin of error. Called the standard error of the difference. So that you can't
even really be sure that those two scores are that different. That if they took
the test again, they might possibly reverse. And they warn explicitly about
making decisions based on those kinds of narrow differences.
Now that's one of those rules that is ignored very widely. And of course,
the test makers--for their profit reasons--do absolutely nothing to stop
institutions which over rely on the SAT or use them as a sole criterion for
scholarships. But, why would you argue that that means that you have more merit
rather than the kid who's done nearly as well on the test, equally well in the
classroom and overcome so much? Which one's going to be a better contributor at
UC Berkeley? Which one's going to be a better contributor in life--in
California or this country?
The average difference between whites and Asians and blacks and Latinos
is--for Latinos it's about 150 points...
Well, even a gap in the 150, 200 point range may not be that meaningful if
the kids with the higher score have taken coaching courses which claim to raise
scores in that range. The independent research suggests it's a slightly
lower--more like 100, 120 points. How do you know that the scores that you're
seeing-- whether they're the result of some kid walking in and taking the test
cold on a Saturday morning? And the results of some other kid who's been
tutored for $700 at the Princeton Review or Kaplan, or $1,500 for some tutor
who comes to your house and drills you on the test? Those scores don't mean the
same thing.
The colleges are making judgment based on a host of factors. Which include
grades, and class ranks, and community service, and all the other kinds of
things that particularly a public university system wants to see in its
undergraduates and in its future alumni. And that seems perfectly reasonable to
balance all those factors.
Derek Bok actually feels the SAT is a common yardstick.
Well, as much as I respect Dr. Bok and the excellent book that he wrote,
he's wrong about that. The test makers themselves tell you that high school
performance, either as grades or as test scores, is a better predictor than the
SAT. Now you think about that. And that's a real important indictment of their
own product. Because everybody knows grades vary all across the lot. There are
tough schools and there are really very easy schools, even within the same
school. Despite all that variation, the test makers admit that grades are a
better predictor than their test is. What does it say about the test?
The second answer to Dr. Bok is the experience in Texas and the other
several hundred schools in the country that don't require test scores as part
of their admissions process for some or all of their applicants. You end up
with a better entering class, in terms of other factors--including general
academic performance--if you ignore test scores. You look at the very
competitive elite end of the university spectrum. Places like Bates and Bowdoin
in Maine. Which are not quite at the Harvard level of selectivity but reject
two thirds to three quarters of all their applicants. Both of those schools
dropped test scores back in the 1980's and both are very, very happy with SAT
optional as a way to admit students. And they say they get more and better
applicants from across the board.
The other thing--and I've offered this many times at college admissions
conferences--is that, okay admissions officers, your administration believes
that the SAT is so useful. Let's say the rules were changed. Instead of parents
paying for the test to the tune of over a 100 million dollars a year, colleges
paid for this useful advice. My hypothesis is, Fair Test's hypothesis is this:
If the SAT is gone tomorrow morning. How many of you disagree? I'm waiting for
that first hand to go up.
Another reason, the hidden reason, colleges like the SAT--is they get,
along with the test scores, a ton of other very useful information from the
student demographic questionnaire. Where the kids fill out all kinds of
information about their history, their college preferences. And colleges use it
for recruiting. You got to college admissions conferences here in the late
1990's, the major focus is what's called yield management. Not, how do we be
selective? It's, how do we put buts with tuition checks in the seats? In the
vast majority of the colleges the reality is, they're looking for three real
factors. High school diploma. You're alive. And there's a tuition check. That's
what gets you in at the vast majority of colleges. There's tremendous amount of
recruiting that goes on. Even from name brand little Ivy colleges, for kids who
have decent--not super--academic records. They are desperate for bodies. And
the SAT is a sideline to that. But they get the recruiting information along
with the test scores.
Professor John Yoo says, oh, great, now because we've passed prop
209--which he supported because of views of fairness-- the standards are going
down in the sewer.
Well, if he really said, limited the damage--that sounds like a
fundamentally racist position. Because the data on what affirmative action did
do, in California when it could legally be practiced, was that it diversified
the jewels but it did not at all lower the quality. It's--the notion that
you're playing off equity on the one hand versus excellence on another, is a
false dichotomy. You can have them both.
That there is research done I know on the medical schools in California and
they looked at the career paths of those who had been admitted under
affirmative action in California. I think one could argue that what those young
people did with their lives was more valuable, more meritorious to society,
than those higher scores perhaps who they replaced. These are people who are in
things like community medicine. And in fact, their profiles did not look
significantly different, than people who were not admitted according to
affirmative action.
There's this notion that somehow among the elite you can very delicately
rank order based on small differences in test scores. When in fact, probably
everybody in the top third of test scores for something like Boalt as a law
school or one of the University of California med schools or UC Berkeley, could
do perfectly well. I mean, at MIT where I went to undergraduate school, the
admissions office has admitted that probably half their applicants could do
adequate level MIT work. And yet they're rejecting three out of every four. So
they're throwing away other people who are meritorious based on some other
factors.
What is your feeling about the significance of the black/white, test
score gap?
Well, first of all, that's not--it's another one of those non either/or.
Indeed attention needs to be paid, more attention, by society to some of the
controllable factors that go into the black, white test score gap, everything
from infant nutrition through lousy schools, etc. But when kids through that
system perform well--even though they might not do quite as well no the test
scores for blacks who graduated from high school, as white, their white counter
parts maybe--you maybe do want to take into account the obstacles they have
overcome and give them at shot at the next level, as a way you define merit.
But defining merit solely as this test score, particularly a test score that
does not predict very well at all, ends up reinforcing the biases that led to
that lower test score.
What does the SAT predict?
The sole scientific claim of the SAT is its capacity to predict first year
grades. According to the technical studies done by the Educational Testing
Service and College Board, the SAT predicts about one factor in six--one sixth
of the difference between two kids first year grades. The predictive value
declines after that--looking at four year grades or graduation rates. So even
the test makers agree that five out of six parts of whatever it takes to
predict how well you're going to do in your freshman year, is not their test.
It does correlate extremely highly with an IQ test. It was developed
from the army IQ test...
That's part of the seedy under side of the SAT. The SAT was originally
developed by straight out racists, eugenicists, people who thought my
forbearers--not just people of color--were imbeciles and shouldn't be allowed
in their country because they didn't know the language and couldn't score high
on their test. I wouldn't suggest the current people who run those companies
share those kinds of ugly views. But it's a self reinforcing notion of defining
intelligence as that which whatever the dominant group in society has. Ends up
giving that group higher scores and lower scores. The fact that test scores
correlate with test scores is rather meaningless. The tests are measuring the
same set of factors. What's more important is whether the test accurately
predicts how well you're going to do.
Would you say that we are the only country in the world that administers
a national IQ test?
Well, despite the efforts of the Educational Testing Service-- which is a
global corporation with nearly about half a billion dollars in total
revenues--the US still is the major country that administers a test like this
across the board to college bound seniors. If you take the SAT and its
competitor, the ACT--which about 80 percent as many kids take-- the vast
majority of college bound kids take those tests. And yes, the SAT in particular
has its roots in IQ testing. Which is, are at best controversial and, at worst,
quite, quite poor predictors of anything of value.
So is it an IQ test?
It's a variant of an IQ like test. It is set up somewhat differently. It
begs the question of, what is an IQ test measuring? What is intelligence? And
you talk to test makers. And intelligence is what their test makes. And that's
a circular definition. So to the extent that it's measuring the same that an
intelligence test is measuring--then, yes it is. But there's three fallacies
there: That there is such a thing as intelligence. That it can be measured. And
that you can put the measurements on a linear scale. And other, even people who
believe that there is such construct as intelligence believes that this thing
is not one thing but seven or eight or possibly nine different things. And
Robert Sternberg at Yale says it's three different things.
At best, the SAT is badly measuring one of those parts of what goes into
intelligence.
Should we get rid of it?
You don't need a test at all. People like Katzman--and he's in the test
coaching business. He'd have to retreat to his summer home in the Hamptons if
he didn't have it. He says you don't need tests at all. It's like a
hypochondriac walking around with a crutch even though they're perfectly able.
You say, we're taking your crutch away, and they say, can I have a wheel chair?
But they can walk.
Two hundred and eighty one colleges have shown that they can do admissions
well. Everything from very competitive to not so competitive, without looking
at test scores at all. All the colleges in the country could in fact do
that.
Wayne Camara said, our analysis show that it's 30 points--maybe 43
points this time around...
Maybe. He's talking about a study released early this year. I looked at the
methodology. Here's how they did it. They did take a group of kids. Give them a
pre-test to see their base line. Coach them. And then see the results. Which
other people have done and found 130, 140 points. What they did is, after kids
had taken the SAT and before they got their score back, kids received a letter
on ETS letter head saying, were you coached? How much were you coached? Fill in
the bubbles and tell us. And they expect that they got honest answers from kids
in that kind of setting. When the coaching companies tell them explicitly, if
you're coached, it's up to you to say. they have no idea whether these kids
were coached or not. It's, ah, it's quite despicable research to be making the
claims they did based on their methodology, which is fundamentally flawed.
We've asked Don Stuart, the president of the College Board, to sponsor
that--directly to his face. He's taken it under advisement for a decade. They
wouldn't do it because if the coaching works, they're dead. And they can't take
the risk because coaching works.
I'm hitting hot buttons here. They are just such liars about coaching.
It's, I mean, they are the tobacco companies about coaching.
Because they won't do the research?
They are unwilling to risk endorsing research that could show that their
product doesn't work the way they say it does. Back historically, ETS has fired
researchers who concluded that coaching works. A guy named Lewis Pike was fired
and wrote about it, back in the 70's. It's their soft under belly. And they
can't risk exposing it.
But the difference between them and the tobacco companies is, they're
supposed to be non-profits operating in the public interest.
Let's go back to the History...
Standardized testing, since it came to this country, in the early part of,
of the 20th century, has been used primarily to sort and to
reinforce existing social structures. Everything from IQ test used for
immigration, to the SAT used for college admissions. Some of the earliest
proponents of the SAT were virulent racists and anti-Semites. And it goes
without saying at that stage, misogynists.
Carl Brigham was, at the time he was appointed to be the first secretary of
the College Board, very heavily involved and I believe he became the president
of the American Eugenics Association. Other early testers, Yerkes and other
were, were absolute racists. Brigham recanted on his death bed apparently. And
I believe that's legitimate.
That's how the tool has been used historically. Testing is a tool that is
used to, to funnel black kids, in the elementary schools into dummy classes.
It's used to make sure that even in integrated school districts, that the white
suburban elite has the advanced classes, so it serves that purpose. Is that the
reason why test makers support the product? No, it's not the sole reason, but
it meets a need for others in society.
home |
discussion |
who got in? |
interviews |
the race issue |
sat & test prep |
history of the sat
the screening process |
test score gap |
getting in to berkeley |
bibliography |
links |
tapes & transcripts |
press |
synopsis
FRONTLINE |
pbs online |
wgbh
| 
And what coaching courses like the Princeton Review and others teach you,
is that you can analyze--without knowing how to solve the problem, you can have
a very good chance of getting the right answer. You look at it and say, what is
the greatest number? Well the greatest number in the answer's six. That's
wrong. That's the distracter answer. It's designed to take somebody who doesn't
understand it and make him guess wrong. But you look at it and say, well, the
simplest thing I can do is I can divide it into four pieces by cutting
vertically and cutting horizontally. But that's too easy too. That's got to be
wrong. But I could get four. So, I must be able to get three or two because
they're smaller than four. And six is wrong because it's the easy answer. The
answer's five. That is the right answer. 
