H Berlak Respose to State of Califonia

[Harold Berlak <hberlak@yahoo.com>]

Harold Berlak notes in response to State of California: Opposition To  
Motion For Preliminary Injunction  re: CAHSEE, California High School  
Exit Exam.

 May 2, 2006

P.1 L9             At the center of the argument in the document  is  
the assumption  or claim  that that CAHSEE is  a test valid test. The  
claim ( P1 L 1-8 )  that CAHSEE is  “dramatically improving the  
quality and equality of educational opportunities in California”  
rests on an  assumption that CAHSEE is a valid test. On P1 L26-27 the  
State asserts that the test’s validity “cannot seriously be  
questioned.”  On P19 L 18 the State asserts that the plaintiffs’  
petition for injunctive relief should be denied because it would  
“lower standards for the Class of 2006 by allowing thousands of   
students who do not possess minimum literacy skills in English and  
math” to graduate without meeting State  standards contrary to policy  
established by the  Legislature.”

 This assertion as well as numerous other assertions about the  
benefits of CAHSEE  presume that CAHSEE  is a  valid or true measure  
of what  the test purports to measure.  Whether a test is a true  
measure of what the test purports to measure is, as the State  
acknowledges, the  nub of the validity question.  In other words,  
does the score a student gets on  the CAHSEE  math and language  
tests  truly reflect that student’s  actual ability to do math and  
use the English language to the level outlined in the State’s math  
and English language content standards.

 IB1&2

P10-12            The State’s case for validity is made beginning on  
P10 L15.  The claim is (L18) that since 2000 CAHSEE has been  
“intensively evaluated” with large scale longitudinal reports…on  
every aspect including content, curricular, and test validity.” Cited  
are studies by HumRRO that ‘concluded that that CAHSEE meets all the  
test standards for use as a graduation requirement

I do not have all the documents to enable me to interrogate fully  
this claim.  But it is vitally important to pay close attention to a  
sleight of hand.  The claim is that the test has “content and  
curricular” validity. (I will return to what the State calls test  
validity later.)

 What curricular or content validity means is that the items are  
constructed so that the content and skills assessed by the test items  
correspond to skills that are specified in the State’s curriculum  
standards. This correspondence is established by panels of experts  
rendering judgments about whether the test items correspond to, or  
are a fair sample of, the content and skills enumerated in the State  
of California’s curricular standards.

 It is important to note that establishing content correspondence  
(or to use HumRRO’s  term “curricular validity”), while it may be a  
requisite  for establishing test validity, can never  in and of  
itself establish the validity of a measure. This is very widely  
accepted by testing professionals  [based on the seminal  work of  S.  
Messick, a former chief test scientist at Educational Testing  
Service. See:  Meanings and Values in Test Validation; the Science  
and Ethics of Assesment. Educational Researcher, March 1989]. To  
establish validity of a test there must be a demonstrable  
relationship between students’ test performance and students’ actual  
performance.  Statistical correlation of a test to forms of the same  
test administered over time, or to other standardized tests cannot  
establish test validity.  To establish the validity of CAHSEE, an  
academic achievement test, would require a validity study that showed  
the connection between test scores  and an independent assessment and  
measure of students’ actual language  mathematical competence.

 That test scales and cut scores established for CAHSEE are not  
grounded in actual performance is fact .  It is not intended here as  
a criticism but as an accurate  statement of CAHSEE’s limitations.   
It is precisely this limitation that led the associations of testing  
professionals (APA, American Psychological Association, AERA,  
American Educational Research Association, and NCME National Council  
for Measurement in Education) to establish ethical guidelines  
requiring multiple measures (not simply multiple opportunities to  
take a different version of the same test).  It also explains why the  
vast majority of educational  researchers , including the recent past  
and current presidents of the American Research Association, oppose  
the use of tests such as CAHSEE alone for making decisions that  
likely will have a  long and permanent effect  on the opportunities  
and direction of a person’s life. The proposition that the  
professional test standards intended to protect  rights of the test  
takers do not apply to graduation tests (P 16 L22-23) and hence  
afford no protection to California’s high school students is highly  
problematic and I believe without any legal or educational  
foundation.  The citation to Phillips does not warrant this claim  
which  cannot be regarded as anything other than  the opinion of  a  
single individual.

 Laying  ‘content and curricular” validity aside, the State’s claim  
for test validity rests on several  studies cited on P 11 &12.  I do  
not have access to all the documents cited.  However it appears in  
none of these studies are CAHSEE scales and cut scores grounded in  
actual performance.  The best I can determine is that the studies  
cited employ a variation on a method called the Angoff method.  I  
will not describe this method in detail.  It is a process much like  
assembling a focus  group.  In this instance the focus group is  
composed  of panels of  subject matter experts and teachers who are  
given  sets of  test items composed  by the test experts and the  
panels are asked to  make a set of  judgments about the items.  There  
is no direct or indirect assessment of student’s actual academic  
performance.  In 1993, the National Academy of Education in a review  
of another test that employed the “Angoff Method” concluded:   
“Discontinue use of the Angoff method… [The ] Angoff method approach  
and other item-judgment methods are fundamentally flawed…” [I cannot  
locate the original source, but I believe it may have been  quoted   
from the written testimony submitted by John Poggio or by Edward  
Haertel  in the 1995 CBEST case filed  by Public Advocates of San  
Francisco (Class Action No. C92-3874-WHO.)  There have also been  
several rulings by the U.S.Civil Rights Commission that also raised  
serious questions.  Unfortunately I do not have the citations.  My  
recollection is that The Harvard Civil Rights Project does.]

 On P8, L6 and P10, L5-7 the State argues that it has  
conscientiously followed the dictates of Education Code 60856 and  
that the CAHSEE advisory committee concluded that no alternative ”has  
met sufficient technical or feasibility standards for full-scale  
implementation in Californis as an equivalent alternative’ to the  
CAHSEE (emphasis in original).”  This conclusion was drawn after  
comprehensive evaluation “of all these alternatives. (those named on  
P.8 L1-3.)

 This claim that there was a comprehensive examination of  
alternatives cannot be sustained. P8 L 1 cites a WestEd  report.  Not  
only does WestEd rely heavily  on the State for its contracts, West  
Ed has relatively little  expertise in alternative assessment. The  
conclusion drawn by WestEd was preordained by the fact that CAHSEE is  
used as the standard of comparison. There is nothing whatsoever in  
the enabling legislation that so restricted the search for  
alternatives.   The court may need to be reminded that the  
information gathering technology employed by CAHSEE is  
extraordinarily limited.  It is an invention that dates to the early  
years of the 20th century when state of the art information  
technology was  limited to the familiar multiple choice test item  
format. The digital revolution in information technology has  
dramatically changed what is possible.   We are still at the dawn of  
the digital age and it is ironic that in a State that leads the  
nation in digital technology resources the CDE chose to limit  its  
examination of  “alternative” test technology to what is  cited in  
the WestEd report.  The technology of the 21st century affords an  
almost unlimited capacity to record store, summarize  and retrieve  
information  and at very  low per unit cost .  The high speed micro  
chip can transform, the ways we can assess intellectual performance  
and (for example) our  capacity to design and employ what we  
currently call  portfolios. Most of us are aware of the limitations  
of current standardized scoring procedures, and there is no  
justification for limiting the pursuit of alternatives that are  
comparable or “equivalent “ to arcane standardized testing  
technology. Any effort to establish cost must surely take into  
account  the  capacity of digital technology to be  interactive   
(i.e. computer games  are an illustration).  This allows not only  
storage and retrieval of information about performance, but would  
enable us specify  in a far more nuanced way the strengths and  
weakness in outcomes and performance and to devise interactive  
educational experiences that correct and improve performance.

 Finally I am puzzled by what appears to be a contradiction that  
undermines the State’s response to the Plantiffs’ petition.  In P20  
section 2 L1-10 headed “An Injunction would harm students” the State  
argues that the injunction would harm student because “graduates  
would continue to suffer in terms of the types of jobs they can  
obtain and the wages that they potentially earn.” In section 4B 1  
(P21 –22) the State argues (L1-6) the students who are denied a  
diploma based on CAFSEE test score will not be hurt economically.  
“Because merely changing an individual’s classification from “no  
diploma to “diploma” will not change the actual skills and knowledge  
that the student possesses, it is unlikely to impact their average  
earnings.”  The State wants it both ways. On the one hand the claim  
is to award diplomas to those who did not pass CAHSEE  hurts students  
who passed CAHSEE economically, but to deny diplomas to the  
plaintiffs has no negative economic effect.