Manufacturers of high stakes tests are against high stakes testin g!

[Arthur Hu <ArthurH@TANGIS.COM>]

z42\clip\2000\06\testmark.txt
URL:
http://washingtonpost.com/cgi-bin/gx.cgi/AppLogic+FTContentServer?pagename=w

pni/print&articleid=A6706-2000May25
Testing the Market
By Jay Mathews
Washington Post Staff Writer
Tuesday , May 30, 2000 ; A09
1. Harcourt. The San Antonio-based company has about 40 percent of the
testing market, including the Stanford 9 used by
Virginia and D.C. schools and Virginia's Standards of Learning tests.
Harcourt's rise to the top was aided by a $28 million
contract to test all California schoolchildren, a prize that came when
the state school board could not resolve a bitter fight
between proponents of the other two companies.

2. CTB. Based in Monterey, Calif., its biggest tests have been the
California Achievement Tests and the California Tests of
Basic Skills/TerraNova. It also produces the MSPAP, Maryland's principal
school assessment test, and has won a five-year,
$10.3 million contract to develop Maryland's new high school graduation
tests. It has about 40 percent of the market, slightly
behind Harcourt.

3. Riverside. The company produces the popular Iowa Test of Basic Skills
and the Woodcock Diagnostic Reading Battery. It
is based in Itasca, Ill., just west of Chicago, and has about 20 percent
of the market. (Washington State WASL, ITBS)

Maureen DiMarco (of Riverside who makes the WASL) said. "If all you
are doing is giving the test and putting it up on a scoreboard, then
you have wasted your money. The purpose of these instruments is to
improve student achievement levels. . . . Every single one of us will
say to you that you should not use one single data point to make
decisions."

Read at you own risk.  WARNING:  Gerald Bracey comments ahead!

[NOTE:  In accordance with Title 17 U.S.C. section 107, this material is
distributed without profit or payment to those who have expressed a
prior interest in receiving this information for non-profit research and
educational purposes only.  This material may not be copied or quoted on
any web site or other open forum without the express consent of the
copyright owner.]

URL:
http://washingtonpost.com/cgi-bin/gx.cgi/AppLogic+FTContentServer?pagename=w

pni/print&articleid=A6706-2000May25

Testing the Market

By Jay Mathews
Washington Post Staff Writer
Tuesday , May 30, 2000 ; A09

Five years ago, National Computer Systems did only one thing--score
tests. The Minneapolis-based company had 400,000
square feet of office space devoted to processing the answer sheets that
primary and secondary school students covered with
little black pencil marks every spring.

Then the demand for student testing soared. States raised their
educational standards and instituted rigorous exams to measure
whether students and schools were meeting the new benchmarks. And each
state wanted a different battery of tests,
customized to assess how much students had learned of its particular
curriculum.

Life at NCS changed. Its total office space has jumped 150 percent to
more than a million square feet. Its scanners, computers
and human test-scorers are working at permanent facilities in six states
and temporary ones in 10 more, compared to a total of
three facilities in 1995. And the demand for more tests is so great that
NCS no longer just scores exams--it writes some of
them, too.

The testing boom has brought similar growth at the three companies that
write most of the standardized exams used in public
schools: Harcourt Educational Measurement, CTB/McGraw-Hill and Riverside
Publishing. Although the companies will not
release financial data, industry analysts say the three have reached a
total of about $200 million in annual sales and their
revenues have been growing by about 9 percent a year.

"This new push for higher educational standards in the states is really
growing the market for these testing guys," said Gail
Kalinoski, managing editor of the Stamford, Conn.-based industry
newsletter Educational Marketer.

But with the new business has come an increase in public scrutiny, an
unsettling change for a culture that has tended to hide
behind a screen of bewildering numerical complexities.

All three companies have felt the sting of resentment from many teachers
and some parents who do not like the increased
emphasis on standardized testing. And because the test results often
carry major consequences--sometimes determining
whether students are promoted and whether educators keep their jobs--any
mistakes made by the companies now receive
more publicity.

CTB came under fire last year when it scored student exams in New York
City incorrectly, prompting school officials to send
thousands of children to summer school they did not need. Harcourt last
year botched test scoring in some school districts in
California.

Test company executives say that such mistakes are part of the business
and that their error rate has not increased with the
greater volume of work. Testing experts "are people who agonize over
perfection," said David W. Smith, president of
assessments and testing at NCS. "They don't go into educational
measurement because they are fast and loose. They go into it
in part because they are finicky people."

Gerald W. Bracey, an educational psychologist and testing critic,
disagrees. "I think the fact is the companies have taken on
more than they can handle with the kind of professionalism they had in
the past," he said.

Although NCS and the Toronto-based Thomson Corp. have started to get
into test-writing, Harcourt, CTB and Riverside still
control almost all of the business. It is difficult to distinguish their
expansion from the growth of their much-larger parent
companies, all book publishers, but analysts say they rank in roughly
this order:

1. Harcourt. The San Antonio-based company has about 40 percent of the
testing market, including the Stanford 9 used by
Virginia and D.C. schools and Virginia's Standards of Learning tests.
Harcourt's rise to the top was aided by a $28 million
contract to test all California schoolchildren, a prize that came when
the state school board could not resolve a bitter fight
between proponents of the other two companies.

2. CTB. Based in Monterey, Calif., its biggest tests have been the
California Achievement Tests and the California Tests of
Basic Skills/TerraNova. It also produces the MSPAP, Maryland's principal
school assessment test, and has won a five-year,
$10.3 million contract to develop Maryland's new high school graduation
tests. It has about 40 percent of the market, slightly
behind Harcourt.

3. Riverside. The company produces the popular Iowa Test of Basic Skills
and the Woodcock Diagnostic Reading Battery. It
is based in Itasca, Ill., just west of Chicago, and has about 20 percent
of the market.

Most of the work goes to just three companies because tests, like
automobiles, are expensive to design and produce and can
be profitable only if they have large national markets.

Many testing executives worked for school districts or state education
departments before moving to private industry. Those
contacts are a key selling point for the companies, which work closely
with state and local school officials in developing and
polishing test questions.

Company officials say the rapid growth in testing has put a strain on
their efforts to find more experts in the field, which is
known as psychometrics. "I don't know anyone who says, 'I am going to
college to become a psychometrician,' " said
Maureen DiMarco, vice president of educational and government affairs
for Houghton Mifflin Co., which owns Riverside
Publishing.

But the boom has created after-hours job opportunities for teachers, who
write many of the test questions, and for university
students, who score the essay questions that many of the new state tests
demand.

The company executives are well aware that some educators think the
whole testing process is misguided and takes time away
from good teaching. Margie Jorgensen, a vice president at Harcourt, said
she also has heard people say that the standards
movement is just a way for her company to make money. Such critics do
not understand how important tests are in helping
children learn, she said.

"As a parent," she said, "I believe the single biggest benefit of
testing progress is to define quality work for all children
equitably." If there are no valid benchmarks, she said, teachers will
not know where their children are in the learning process,
nor how much further they have to go.

But as to whether school rankings and student promotion decisions should
be based on test scores alone, testing company
officials sound very much like the educators and parents who have
criticized state officials for attaching such high stakes to the
exams.

"The first people who are concerned about the use and misuse of test
scores are the test publishers," DiMarco said. "If all you
are doing is giving the test and putting it up on a scoreboard, then you
have wasted your money. The purpose of these
instruments is to improve student achievement levels. . . . Every single
one of us will say to you that you should not use one
single data point to make decisions."

www.washingtonpost.com

For more information on standardized testing, go to The Washington
Post's Web site and click on "Education." And visit the
Web site at 1 p.m. tomorrow to join a live online discussion on testing
issues with Maureen DiMarco, vice president of
educational and government affairs for Houghton Mifflin Co.

Anatomy of a Question

Every year, test-making companies churn out tens of thousands of
questions to help schools assess how well America's
schoolchildren are learning their lessons. Here's how one of those
questions made its way from a science teacher's office to a
student exam room.

The mission: Write a question for the Virginia Standards of Learning
(SOL) high school exam in earth science that will show
whether students understand "how geologic processes are evidenced in the
physiographic provinces of Virginia . . ."

(language taken from the state's curriculum standards)

Fall 1996

A science teacher who works at a school outside Virginia and also serves
as an "item writer" for Harcourt Educational
Measurement, the San Antonio-based company that produces the SOL tests
for Virginia, types out this question on a
computer:

Mud cracks found in the limestone road cuts along the Gate City,
Virginia, bypass would show that before the sediments were
buried the Gate City area was ----

A. a migration route for animals

B. an active earthquake zone

C. a wet area exposed to air

D. a sand-covered beachfront

The correct answer is C, the choice that indicates a student understands
the conditions required to form mud cracks. Each of
the three wrong answers, known as "distractors," is designed to trap
students whose knowledge of the subject is incomplete.

Answer A: Would be chosen by students who think that animals caused the
earth to crack as they repeatedly followed the
same trail. The conclusion is somewhat logical, but fails to take into
account the regular pattern formed by mud cracks.

Answer B: Would be picked by students who don't realize that earthquake
damage is much greater than the size of mud
cracks.

Answer D: Would be chosen by students who believe that sand can form and
retain cracks. They fail to realize that drying sand
tends to fill in cracks.

The question is reviewed by a copy editor, a senior content specialist
and several other testing executives at Harcourt.

Winter 1997

The Virginia Department of Education's earth science review committee
meets to consider the Gate City question and about
200 other proposed earth science items. The committee includes high
school teachers, school district science specialists and
college professors.

They approve the question with one change, substituting the word "shale"
for "limestone."

May 1997

The question is included in a "field test" given to 4,262 high school
students across Virginia.

Summer 1997

The earth science review committee meets again to consider the field
test results. They note that the percentage of students
who chose the correct answer fell between 35 percent and 85 percent,
which, to testing experts, shows that the question is
neither too hard nor too easy. The pattern of responses also suggests
that the item is not sexually or racially biased. The
committee approves the question.

Spring 1998

The question is given to 54,000 students taking the earth science SOL
test, and this time the results count. The percentage of
students who answer it correctly drops from 58 percent to 51 percent.

Fall 1999

The item is placed in the bank of used SOL questions that Virginia
officials will release to the public.

Here are comments on the test item from three earth science teachers who
weren't involved in developing it:

"We talk [in our class] about reading the past even though we weren't
there. And we talk about things like mud cracks and we
talk about ripple marks and how they can indicate whether an area has a
steady tide or a steady set of waves. I thought it was
a good question."

Bob Nicholson, T.C. Williams High School, Alexandria

"It does not fairly cover the [curriculum standard] noted above, which
is far more generalized. Taking a tiny sample question
from a curriculum, it is hard to see that a student is achieving a
particular [standard of learning]. It is far better to know that a
student can apply concepts he/she has learned to make better sense of
his world."

Tricia Pease, Yorktown High School, Arlington

"Is the question valid as to its ability to test how geologic processes
are evidenced in the physiographic provinces of Virginia?
Yes, it is. A better question would be whether this is a good tool to
help determine earth science mastery. The Virginia SOL
test is designed to determine a level of learning by using very specific
and structured questions. Wouldn't it be better to test the
application of the standards using open-ended questions?"

Michael D. Dyre, Potomac High School, Prince William

                                  © 2000 The Washington Post Company

POSTSCRIPT:
Remember, this IS the Washington Post, and it's in camp with the 'crats
& NEA.

Best//John

--------------------------------------------------------------------------
To unsubscribe from the ARN-L list, send command SIGNOFF ARN-L
to LISTSERV@LISTS.CUA.EDU.