Re: bias in tests

["Monty Neill" <monty@fairtest.org>]

From: George Hein 
To: ndsgroup@yahoogroups.com 
Sent: Friday, February 02, 2007 10:15 AM
Subject: [ndsgroup] Re:bias in tests


Monty Neill's latest post about test bias (following a long tradition of finding  questionable items) needs to be viewed from a perspective of a broad analysis of the problems with current high stakes tests.


Any careful observer can find questions that reflect biases, technically poor questions, inappropriate questions, etc., in almost every test.  And it's important to keep pointing this out with examples.


But, he essential point is that there is no objective way of selecting questions in the standard format of current testing regimes.  That's one little realized crucial flaw of psychometrics.  The practitioners of this art can provide exquisitely detailed analyses of the validity and reliability of items compared to to other items and tests, but the very choice of any item in the first pace is the issue. That they are "aligned with the standards" is simply a statement.  No one, to my knowledge,  ever provides any empirical evidence, based on experimental work that would let me know what "alignment" means in a quantitative way.   Items are generated by a process that has no theoretical basis and no experimental base that adheres to the kind of "scientific" rigor that  the tests are supposed to exemplify.  Remember when John Silber personally added items to the MCAS because he thought they were important for any high school graduate?  I've been asked to submit items for various science tests with no advice from the (well established)  test designers other than to suggest what I thought was "appropriate." 


And, as the examples Monty refers to illustrate, there are no objective definitions of "bias," "culturally sensitive,"  "racism," etc., against which test items could be measured to  apply a  quantitative standard of  the validity.  So how can the tests actually assign cut off scores, levels of progress, etc?  In science, If you end up with a quantitative result , you're supposed to have an unbroken chain of quantitative experimental data that justifies doing so.


George
-- 
 George E. Hein
 Professor Emeritus
 Lesley University
 
  ----- Original Message ----- 
  From: Monty Neill 
  To: ndsgroup@yahoogroups.com ; arn2-strategy ; ARN-L ; ARN-state@yahoogroups.com ; care@yahoogroups.com 
  Sent: Wednesday, January 31, 2007 6:14 PM
  Subject: [arn-l] bias in tests


  Attached is a fact sheet on test bias from PURE that FairTest collaborated on and that we have used for workshops in Chicago. We chose not to go into the limitations of statistical means for addressing bias, but simply to provide some examples of the kinds of things that continue to get by reviewers or that are introduced after reviewers are done or that are retained even if reviewers flagged them. These examples led to interesting discussions with Chicago parents. 

  It is attached as a pdf file - the link to the pdf version on the PURE website is http://www.pureparents.org/data/files/testbias.pdf

  Monty 
  Monty Neill, Ed.D.
  Executive Director
  FairTest
  342 Broadway
  Cambridge, MA 02139
  617-864-4810    fax 617-497-2224
  monty@fairtest.org
  http://www.fairtest.org
  Donate: https://secure.entango.com/servlet/donate/MnrXjT8MQqk