In-depth news story on Essay Scoring Errors

[Bob Schaeffer <bobschaeffer@earthlink.net>]

Though this story has a Connecticut focus it may have national 
implications since CTB/McGraw Hill holds testing contracts in 22 states. 
Hard to believe that the company scored eveyone else's exams correctly 
after so badly screwing up one state.


FEVERISHLY FIXING AN ERROR: COMPANY RACES TO CORRECT INACCURATE SCORING 
OF CONNECTICUT MASTERY TEST
   Hartford Courang -- March 21, 2004
           by Robert Frahm

Indianapolis -- In cavernous rooms lined with rows of computers, the 
fate of Connecticut's multimillion-dollar student testing program 
depends - literally - on whether Indiana workers like Ethel 
Trice-Sanders know the score.

One of the nation's largest testing companies, racing to fix errors in 
its initial scoring report, has hired an army of workers to rescore 
student writing on this year's Connecticut Mastery Test.

"This supplements my retirement," said Trice-Sanders, 74, one of 
hundreds of temporary employees selected for their ability to maintain 
rigid accuracy as they reduce each written answer to a single number.

No less than CTB/McGraw Hill's new seven-year, $48 million contract with 
Connecticut is on the line as the California-based company tries to 
recover from the worst scoring miscalculation in the 19-year history of 
the annual test.

A flawed initial report by the company not only forced Connecticut to 
postpone its customary January release of scores, but also raised 
questions about whether the company - and the entire testing industry - 
can handle the onslaught of new test requirements under President Bush's 
school reform law, the No Child Left Behind Act.

"There's a lot at stake here," said Barbara Q. Beaudin, acting chief of 
the state Department of Education's bureau of student assessment. "Can 
we do it as a state? Can the industry do it? Have expectations extended 
beyond the capacity to deliver?"

Some experts contend that the glitch is a warning of more problems to 
come for an overburdened industry.

"It's a terrible problem. It's happening all over the United States with 
greater consequences for students and teachers," said Thomas Haladyna, a 
testing specialist at Arizona State University.

Connecticut asked CTB/McGraw-Hill to rescore this year's exams after the 
company, in the first year of its contract, reported scores that were 
mysteriously lower than those of a year ago. About 125,000 fourth-, 
sixth- and eighth-graders took the exam in September.

The problem was traced to questions that required essays or other 
written answers - an increasingly common task on many state achievement 
tests and soon even on the SAT college entrance exam.

Educators believe written answers are crucial in demonstrating student 
skills but, unlike multiple-choice answers, they are far more difficult, 
time-consuming and costly to grade.

Connecticut - which tests students in reading, writing and mathematics - 
was one of the earliest states to require open-ended written answers on 
parts of its exam.

"If we can't score this correctly," said Beaudin, "that will bring into 
question whether this higher level of testing we're doing can be done on 
a broader level."

Brain Aerobics

That question will be answered here at CTB/McGraw-Hill's Indianapolis 
center, a sprawling complex consisting of converted strip mall stores. 
It is the largest of the company's four scoring centers, a bustling 
factory that processes hundreds of thousands of tests from across the 
nation.

Beaudin was in Indianapolis earlier this month with a contingent of 
Connecticut and local school testing officials to monitor the rescoring 
effort.

The company has mounted a massive effort, bringing in more than 900 
scorers who are working two shifts and weekends to meet a series of 
deadlines. The first reports are scheduled to be sent to districts by 
the end of this month, with a final report due June 4.

The scorers, many of them veterans of earlier projects, are recruited by 
Kelly Services, a job placement agency with an office located at the 
center. A bachelor's degree is a minimum requirement for the job. The 
scorers start at $10.50 an hour and range from recent college graduates 
to moonlighting teachers to retirees such as Trice-Sanders.

By the end of her eight-hour shift, Trice-Sanders estimates she will 
finish 300 papers, each consisting of the same block of written answers 
from one small section of the test.

Trice-Sanders, a grandmother and former nutritionist, calls the work "my 
brain aerobics." She reads answers that have been scanned into a 
computer and pop up on her screen in each student's own handwriting.

"We always focus on accuracy first," Trice-Sanders said, referring to 
strict guidelines that require scorers to adhere to a point scale 
designed to take the guesswork out of grading student writing.

State officials believe that faulty monitoring of how scorers followed 
those guidelines is at the heart of what went wrong in the first round 
of scoring.

CTB/McGraw Hill took over from another company, and the transition led 
to inconsistencies and misunderstanding about the guidelines, 
CTB/McGraw-Hill officials say.

"There is an art to scoring," said Bud Hall, CTB/McGraw-Hill's director 
of hand-scoring. "There is subjectivity. ... Our work is to remove as 
much of that variable as possible."

The company insists it has tightened its procedures on the Connecticut exam.

It puts employees through hours of training on Connecticut's scoring 
blueprint - which includes strict point scales for each written answer.

In a cluster of cubicles, dozens of trainees wearing headsets listen to 
instructions as they read a sample essay, looking for elements such as 
organization, fluency and detail.

"We are starting to get more specific detail" in this sample, trainer 
Pam Spear tells them, explaining that the essay would merit a score "in 
the mid-5-point range" on a 6-point scale.

The company rejects scorers who cannot consistently rate papers accurately.

On the writing exam, every essay is read by at least two scorers. The 
company also assigns team leaders to rescore random papers to make sure 
scorers stick to the guidelines. A group of supervisors performs similar 
rechecks on the team leaders.

"You develop a rhythm after a while. You internalize the guidelines," 
said Judy Dye, a veteran team leader who helps realign scorers who, 
after rating hundreds of essays, sometimes "drift away from where they 
belong."

Straining The System

Company officials say they are confident the quality checks will iron 
out earlier problems, but as recently as last week, they still were 
finding errant scoring patterns on sixth-grade writing tests.

State officials are taking a wait-and-see stance.

"I won't be entirely convinced until I see they've met the deadlines," 
state Education Commissioner Betty J. Sternberg said.

The state hired CTB/McGraw-Hill last year, ending a long relationship 
with Harcourt Assessment, a Texas company that had scored Connecticut's 
mastery test since it was first given in 1985. In recent years, state 
Department of Education officials had grown increasingly dissatisfied 
with Harcourt's work.

"They were consistently late, inaccurate," said Abigail Hughes, who 
oversaw the testing program for the state. "These kinds of problems 
occur with all kinds of [test] companies ... a lack of quality control."

Within months, there were signs that the new company, too, was off to a 
rocky start.

"When can we expect to get the scoring tables ... sent to us for 
approval?" asked a Dec. 4 e-mail from the state Department of Education, 
one of a series of increasingly urgent messages to the company about 
procedural delays.

A Boston College study last year found that scoring mistakes have 
occurred at several testing companies. One such error in 1999 by 
CTB/McGraw-Hill resulted in incorrect scores for a quarter-million 
students in Tennessee, Indiana, Wisconsin, Nevada, South Carolina and 
New York City, the study found.

Because of another testing company's error, 50 Minnesota students were 
temporarily denied high school diplomas in 2000, the report said.

Georgia officials limited the administration of their test last year 
after a testing company inadvertently revealed some test questions 
before the exam was given.

"In Connecticut, we've been criticized by testing companies [saying], 
`Why do you have to double-check everything?'" Hughes said. "We've seen 
those cases in Georgia, New York, Maryland, where they were wrong. 
Unfortunately, it seems to be the nature of the industry."

Much of the nation's school testing is done by only a handful of major 
companies, which now must gear up for a heavier workload. Under the 
federal No Child Left Behind Act, most states are expanding existing 
test programs or adding new ones. In Connecticut, the mastery test 
program will add grades 3, 5 and 7, doubling its size by 2006.

"Given No Child Left Behind ... and given the fact that 50 [states] are 
trying to get vendors to do this ... it's straining the system," said 
George F. Madaus, one of the authors of the Boston College study.

Michael Kean, a spokesman for CTB/McGraw-Hill, disagrees. Although 
schools are adding new tests to comply with the federal law, they are 
also dropping older tests, he said. "It's largely a trade-off."

He said CTB/McGraw-Hill has been expanding for more than a decade and 
can handle new demands. It currently holds contracts in 22 states, he said.

Of more immediate concern is the Connecticut mix-up. Unless the company 
can resolve the scoring problem, some educators fear there could be 
pressure to reshape the test itself - by backing away from essays or 
other written answers.

That would not surprise 68-year-old Lucille Heinzmann.

The retired American Express employee has worked on several scoring 
projects for CTB/McGraw-Hill but found the Connecticut test too 
difficult to grade.

"They took me off that project. My team leader and I couldn't agree on a 
number of scores," said Heinzmann, who described the scoring guidelines 
as "loose and inconsistent."

Nevertheless, after her brief stint, she had formed her own opinion 
about Connecticut's students.

"I've done Indiana, I've done Colorado, I've done Missouri," she said. 
"Connecticut students spell better. Some of them write a lot better."

------------------------------------------------------------------------
http://www.ctnow.com/news/education/hc-glitch0321.artmar21,1,2365257.story?coll=hc-headlines-education