PACT

[Harold Berlak <hberlak@yahoo.com>]

Comments on PACT article in Journal of Teacher Education

Raymond L Pecheone, ., and Ruth R. Chung. "Evidence in teacher  
education: the Performance Assessment for California Teachers  
(PACT)." Journal of Teacher Education 57.1 (Jan-Feb)



What is most disturbing about the article is that it is written as if  
it is presenting evidence that PACT (Performance Assessment for  
California Teachers) is a scientifically promising direction for  
teacher education in California. Such a conclusion is entirely  
unjustified. Dismantling the logic of this article would take many  
pages.  In fact, the paper carefully read supports an opposite  
conclusion, that it  is unwise and dangerous to proceed on this or  
any single model for educating and assessing teachers.

I do agree with the writers that at the heart of the matter is  
whether the PACT assessments are valid measures of teacher quality.  
The article goes on to make the case that PACT meets the test for  
‘content validity’, ‘concurrent validity’, and ‘construct validity’.   
In making their case for construct validity, the authors cite as  
their authority  Lee Cronbach, the widely known psychometrician and  
psychologist who apparently invented the idea and the term ‘construct  
validity’.  In a nutshell this concept was incorporated into the APA/  
AERA /NCME professional test standards as a way to answer the  
question of how to establish whether a test measures what it purports  
to measure. For example, does a test that labels itself a reading  
test (‘reading test’ is the ‘construct’) in fact measure reading  
competence or ability?  In this instance the question is whether   
PACT  truly measures what it purports to measure - – teaching ability  
and competence. In making their case for construct validity the  
authors cite as a source a Cronbach article reprinted in a book  
published in 1988. The definition cited is drawn from Cronbach and  
Meehl’s 1955 seminal article on construct validity.

The confusions over types of validity are numerous and I won’t  
attempt to deal with them here except to note that thirty-four years  
after Cronbach and Meehl published their  article on test validity,  
Sam Messick of ETS, then the acknowledged reigning scientist and  
chief theoretician of standardized educational testing, in an article  
in the Educational Researcher (March 1989)  made a major contribution  
to clearing up the continuing confusions over types of validity and  
how validity is established. He concluded (and it is now widely  
accepted) that there are not several types of validity  but only one  
--construct validity. He argued that content, concurrent and  
predictive validity are all aspects of construct validity.  Further  
that the construct validity of an evaluation instrument cannot be  
established without a full examination of  the possible consequences  
or the effects the assessment may have on individuals, on the  
institution in which the assessment is used, and on society. He also  
argued that assessment instruments to meet the test of validity must  
be grounded in demonstrable connections to real performance.   
Establishing a test’s validity requires study and determination of  
the anticipated and unanticipated effects of the assessments and  
exercise of human judgment about the desirability of the  
consequences.  Do the promised benefits materialize and do they   
outweigh the negatives? These questions are not merely technical but  
moral matters raising fundamental questions of values and priorities.  
What kinds of teachers do we want to prepare for what kind of schools  
and society?  How should teachers be educated and how should they be  
taught to teach and manage classrooms and children clearly rests on  
value questions that in a democracy ought not to be left to  
government officials panels of government anointed ‘stakeholders’ and/ 
or to professional experts. The writers claim that the instruments  
they used are valid yet the most fundamental questions about test  
validity are unaddressed.

The writers do at times acknowledge that the validity of PACT  
assessments are still problematic since the relationship of the PACT  
instruments to teachers’ actual classroom performance over time is  
problematic. Making judgments about, and classifying teacher  
performance (live or on video) using rating scales in conjunction  
with Likert type questionnaire items and portfolios cannot substitute  
for intensive direct observation over time.  The Teacher Performance  
Expectations are complex and are subject to many interpretations.  
Consensus on rating scales can only be obtained by training that in  
effect suppresses what may be valuable differences in perspectives  
and values.

The writers call for multiple measures and caution. This is  
commendable but politically naive. Is there any doubt that if  
standardized assessments are used as measurements of teacher  
preparation and competence that these standardized tests results will  
overwhelm all other assessments (not to mention devouring teacher  
educators’ already limited time). Before forging ahead with a plan,  
we need to identify and study educational impact or consequences and  
ask whether we are prepared to inflict the same damage in higher  
education and teacher education that we are in the process of  
inflicting on public schools with NCLB.

Among the most disturbing (and dazzlingly superficial) statements is  
the last line in the document to the effect that validity of the TPE/ 
TPAs will be established ultimately by whether they enhance pupil  
learning. This apparently innocuous statement raises the prospect  
that the value of teacher education programs will ultimately depend  
upon and be validated by pupils’ scores on standardized tests. The  
damage that this continuing focus on standardization and standardized  
testing will inflict if it goes unchecked will be enormous. If the  
writers agree that this was not their intention then they have the  
moral responsibility to make this absolutely clear.  PACT may be a  
well-intentioned and reasoned response to a bad State law.  
Nevertheless, if PACT goes forward and is used as presently  
conceived, we are at great risk of creating a massive statewide  
bureaucratic system of control in teacher education that in effect  
represses or ignores dissent, issues of race and cultural diversity,  
and fosters docile and compliant teachers and citizenry.

 Harold Berlak