Testing Errors Prompt Calls for Oversight

[George Sheridan <learn@jps.net>]

Today the Sacramento Bee reprinted on its front page this article, which 
also appeared on the front page of the New York Times.
The article reports James Popham's suggestion that testing companies be 
required to report all errors publicly. This sounds like the kind of 
legislative proposal that could pass in the current climate.

Testing Errors Prompt Calls for Oversight
By KAREN W. ARENSON
Published: March 18, 2006
http://www.nytimes.com/2006/03/18/politics/18testing.html?_r=1&oref=login

As the College Board races to score the final 1,600 exams from its 
problem-ridden October SAT test, a string of recent testing errors around 
the country has college and high school officials, testing experts, 
students and parents asking with rising urgency, Who is watching the 
testing industry?

Spurred largely by the No Child Left Behind law, testing has exploded in 
recent years. Educators are now trying to measure factors like whether 
toddlers in Head Start know their letters and whether elementary and high 
school students are making progress in reading and math.

The states alone are administering about 45 million tests this school year.

And for students headed to college, there are the Advanced Placement exams, 
the SAT and the ACT.

The volume is stretching the $2 billion-a-year education testing industry, 
educators say, taxing its ability to draw up enough tests and score them 
quickly and reliably. The states are struggling to find experienced 
officials to provide quality control.

The resulting flurry of errors has educators and lawmakers calling for 
better disclosure and oversight. Some are even proposing a national agency 
like the Food and Drug Administration to regulate testing.

"We need accountability," said George F. Madaus, a research professor at 
the Center for the Study of Testing, Evaluation and Educational Policy at 
Boston College. "I certainly wouldn't get rid of testing, but we need to be 
much more aware than we are now about the shortcomings, the limitations and 
the fallibility of the technology."

The past few weeks have shown the range of the problems. The College 
Board's disclosure that at least 4,600 students out of 495,000 who took its 
October SAT test had scoring errors was followed this week by an $11 
million settlement by the Educational Testing Service of a case involving 
scoring errors for 27,000 people on tests that more than half the states 
use for teacher certification.

The Illinois superintendent of education, Randy J. Dunn, threatened last 
week to recommend ending the state's $44.5 million five-year contract with 
Harcourt Assessment because state tests were delivered late with misprints 
and collating errors. (He later backed away.) And New York State said this 
week that its seventh- and eighth-grade math tests had several questions 
that had been in test preparation materials.

"All of these tests have errors," said John Katzman, chairman and founder 
of the Princeton Review, a test-coaching company that has benefited from 
the testing boom. "The questions might be flawed in some way. The scoring 
might be flawed. The administration is often flawed."

A large part of the problem, policy makers and educators say, is that the 
demand for tests is outstripping the abilities of the industry, and testing 
and scoring quality have deteriorated. Adding to the pressure, each state 
has its own specifications.

Robert L. Linn, a professor emeritus at the University of Colorado at 
Boulder who serves on a College Board advisory committee, said that 
problems like those with the October SAT were not uncommon in testing.

"These things happen now and then in a lot of testing programs," Dr. Linn 
said. "But there are too many now; the industry is stretched pretty thin." 
He added, "It's pretty clear, I guess, that the quality-control issues need 
to be looked at again."

In a recent study for EducationSector, a new education research 
organization, Thomas Toch, the group's co-director, found that the high 
stakes of the No Child Left Behind law, which sanctions schools that do not 
improve, had states trying to administer tests as late as possible so 
children had the most time to prepare. But they still want scoring 
completed in time for summer school placement, giving companies little 
turnaround time.

Mr. Toch said states were having trouble recruiting and retaining experts, 
leaving testing companies largely responsible for their own quality 
control. Companies, too, are struggling with these demands.

"They get so little money for these contracts that they are hard pressed to 
hire all the people they need to do this immense amount of work without 
making mistakes," he said.

Joyce E. Karon, a member of the Illinois State Board of Education, said 
that although she believed Harcourt was "a reliable company," like other 
test providers, it was "working at the brink, and when you work at the 
brink, things happen."

Mrs. Karon said that before the board chose Harcourt, it asked other states 
whether they had experienced problems. "Almost universally," she said, 
"they had all had some problems."

Rick Blake, a spokesman for Harcourt Assessment, said the company regretted 
the hardship that the problems had created for Illinois and was working to 
ensure that the rest of the testing went "smoothly and without further delay."

Still, as the pressure for more tests intensifies, testing experts and 
policy makers are beginning to weigh whether more oversight is needed.

One suggestion would require that testing problems be reported publicly. W. 
James Popham, an emeritus professor at the University of California at Los 
Angeles, said that more exposure might help. "Frankly, because it is in the 
best public-relations interest of both the scoring service and the state 
officials who hire them, many scoring muck-ups are masked," Dr. Popham said.

One step toward disclosure could come from the inspector general of the 
federal Education Department, who is planning to study what states are 
doing about errors and whether there is need for federal oversight.

Another approach being suggested by Mr. Toch, Dr. Madaus and others is to 
have some kind of auditor or oversight board, an independent entity or a 
federal body. "There are all kinds of things in society that get 
monitored," Dr. Madaus said. "Nobody is seriously looking over the shoulder 
of those testing programs."

But some in the industry say regulation is unnecessary and will raise the 
price of testing. "When something like this occurs, you want to make sure 
you don't create regulations or ways of doing things that increase costs 
and don't improve the quality of service," said Gaston Caperton, the 
College Board president.

Kenneth P. LaValle, the New York State senator responsible for the state's 
1979 Truth in Testing law mandating disclosure, said he believed more 
disclosure of test questions and answers might be needed.

"We are now living in a testing culture," said Mr. LaValle, Republican of 
Long Island. "We need accuracy and security and all these things."

Some counselors are urging college-bound students to pay extra to obtain 
answers after a test is administered to check their own performance, and 
for hand scoring.

Eugene Falik, a computer specialist in Far Rockaway, N.Y., said that after 
his daughter signed up recently for an Advanced Placement exam, he spotted 
the hand-scoring option. "The clear implication is that these marked sense 
sheets, or optical mark readers, are not reliable," Mr. Falik said.

More colleges have stopped requiring the SAT. Joanne V. Creighton, 
president of Mount Holyoke College in Massachusetts, said she felt her 
college's decision five years ago to make the tests optional was reinforced 
when she learned of the SAT scoring errors. She said the college had found 
this made "no measurable difference" in quality.

But others say testing is a crucial tool. Richard P. Mills, education 
commissioner in New York State, said recent problems had not changed his 
mind about the usefulness of testing. "It can be done right," Mr. Mills 
said. "Does that mean it's flawless? No. Errors crop up. No error is 
acceptable. But testing is indispensable."


George Sheridan