Re: test misuse: Here we go again

["Allen Flanigan." <Allen.Flanigan@USPTO.GOV>]

George is right, up to a point.  Arguing ethical conundrums with Deanna and
others supportive of the testing companies encourages hair splitting and
conjecture about the exact wording of the contract (was Mod. Angoff
"required" or "recommended", was this part of contract modified by mutual
agreement, etc.), and even in situations where the language regarding
appropriate test use comes from unilaterally expressed guidelines that have
been in place for some time, few companies respond to ethical complaints
absent some perceived threat to the bottom line or other dire consequences.
One recent example that came to mind was the licensing of AIDS drugs in
Africa; pharmaceutical companies maintained a lawsuit designed to protect
their profits through drug licensing until the negative publicity grew to
the point where the drug companies withdrew the suit.  This demonstrates to
me that it is possible for ethical considerations to triumph over the bottom
line, but in the case of AIDS, the negative consequences are widely
percieved to be tragic, conspicuous, and profound.  Unfortunately, the same
cannot be said of the consequences of test misuse.



-----Original Message-----
From: gkc@LOUISVILLE.EDU [mailto:gkc@LOUISVILLE.EDU]
Sent: Friday, September 07, 2001 3:27 PM
To: ARN-L@listsrva.CUA.EDU
Subject: Re: test misuse: Here we go again


List,

The members of this list disagree about many things, but they are joined in
their criticism of state accountability tests.  Their some on this list who
criticize their implementation, but accept their use, while others are
opposed to all testing.  There are other issues that come up here
occasionally and we tend to display a wonderful diversity of opinion on
these.

What is most important is how to change the state imposed policies on
assessment and accountability with which there is disagreement.  Over the
last several days, we have returned to the legality and/or ethics of various
assessment practices as a possible strategy.

In my view, this will not lead very far.  The implementation of
accountability tests is by definition, legal, because it is based on laws
passed by legislatures signed by governors, all of whom are enthusiastic
proponents of these practices.  Attempts to find relief in courts for test
abuses have not met with much success because the courts tend to give
considerable leeway to educators.  The politicians and educators who
implement these practices are doing so within the limits of the existing
laws implementing these programs.  No matter how egregious you may find
these practices, those who implement them are doing so believing that they
represent the best way to educate students.  You may think this is idiotic,
unfair, educationally unsound or whatever.  The courts generally take the
position that educational leaders have been appointed to administer the
educational practices of the state and as such, they are reluctant to
substitute their judgment for that of the educational establishment.  As
those on this list who have become involved in such lawsuits will tell you,
it is awfully hard to win such cases.

Now, concerning the decision in Virginia to not set their cut-scores using
the modified Angoff method specified in their contract with HEM.  That may
or not be a good policy, but contracts are modified all the time on the
basis of a memorandum of record.  If doing so caused some financial harm to
HEM, they could object.  If the legislation that enabled SOL mandated the
use of modified Angoff, there would be a basis for some sort of complaint.
That does not appear to be the case here.  Just because the contract
advocated the use of this method, doesn't mean that the state has to use it.
They can make this sort of change with impunity and states do this all of
the time.

Anyway, there is nothing wonderful about the modified Angoff method.  It is
just one of many standard setting methods.  It differs from the vanilla
version of Angoff in that it includes principles of norm-referenced
assessment.  All standard setting methods and in particular, the Angoff
method have been highly criticized over the last several years.  This
cut-score setting method is the basis for the NAEP performance levels and
the claim that 40 percent of fourth graders are below basic.  Gerald Bracey
has pointed out, on a fairly regular basis, the folly of these assertions.
For a thorough critique of this method see _Grading the Nation's Report
Card_  put out by the National Research Council.  It is an evaluation of
NAEP, which includes an extended critique of the modified Angoff.
Surprisingly, they ended up recommending methods that look a lot like
norm-referenced testing.

Regarding the issue of states or school districts using standardized
achievement tests inappropriately or in violation of the recommendation of
the test publishers.   It seems to me that Chicago's use of the ITBS is a
bad educational practice as are similar practices around the country, but it
is not illegal and you will not get very far trying to hold the publishers
accountable.  You should interpret their recommendation that the test not be
used as the sole criteria for promotion decisions the same way you would
interpret the statement printed on inflatable rafts that states that they
should not be used as flotation devices for safety purposes.  Manufacturers
do not include this admonishment on their products so that they can be held
responsible if someone drowns using the raft instead of a life preserver.
They print this on their products to keep from being sued.  The statements
about how publishers recommend that their tests be used are just CYA for the
publishers.  They are not going to take responsibility for how their tests
are used.

Same holds true for ethical standards such as the Joint Standards on
measurement and testing published by APA, AERA, and NCME.  Ethics are not
laws and are unenforceable except by professional organizations.  APA will
expel members who violate their standards.  This applies primarily to
client-therapist relations.  Most educators are not members of AERA or NCME
and those organizations don't kick people out anyway.

I believe that if you want things to change regarding state accountability,
you have to do it politically.  I have been there, done that.  The old
aphorism about how you shouldn't watch sausages or laws made has real
meaning for me.  However, that is the only way to change things.  The courts
are not going to help you and neither are appeals to ethics.  The California
HSEE is worth 50 million to ETS.   What is Chicago's contract with Riverside
worth for the ITBS?  The publishers and the educational leaders in the
states and districts doing these assessments can parse words and rationalize
their policies better than Bill Clinton defining what "is" is and will never
be pinned down.

This is where I part ways with Deanna.  She agrees with many members of this
list that laws and ethics are relevant she just disagrees with the facts and
the interpretations.  I think it is all about politics and that courts and
ethics are a dead end street.

George K. Cunningham
University of Louisville

--------------------------------------------------------------------------
To unsubscribe from the ARN-L list, send command SIGNOFF ARN-L
to LISTSERV@LISTS.CUA.EDU.

--------------------------------------------------------------------------
To unsubscribe from the ARN-L list, send command SIGNOFF ARN-L
to LISTSERV@LISTS.CUA.EDU.