Re: PACT

[Peter Campbell <campbellp@mail.montclair.edu>]

Excellent piece, Harold. Well said.

Peter

On Dec 7, 2006, at 12:38 PM, Harold Berlak wrote:

> Comments on PACT article in Journal of Teacher Education
>
> <x-tad-smaller>Raymond L Pecheone, ., and Ruth R. Chung. "</x-tad-smaller><x-tad-smaller>Evidence in teacher education: the Performance Assessment for California Teachers (PACT).</x-tad-smaller><x-tad-smaller>" </x-tad-smaller><x-tad-smaller>Journal of Teacher Education</x-tad-smaller><x-tad-smaller> 57.1 (Jan-Feb)</x-tad-smaller>
>
>
> What is most disturbing about the article is that it is written as if it is presenting evidence that PACT (Performance Assessment for California Teachers) is a scientifically promising direction for teacher education in California. Such a conclusion is entirely unjustified. Dismantling the logic of this article would take many pages.  In fact, the paper carefully read supports an opposite conclusion, that it  is unwise and dangerous to proceed on this or any single model for educating and assessing teachers.
>
> I do agree with the writers that at the heart of the matter is whether the PACT assessments are valid measures of teacher quality. The article goes on to make the case that PACT meets the test for ‘content validity’, ‘concurrent validity’, and ‘construct validity’.  In making their case for construct validity, the authors cite as their authority  Lee Cronbach, the widely known psychometrician and psychologist who apparently invented the idea and the term ‘construct validity’.  In a nutshell this concept was incorporated into the APA/ AERA /NCME professional test standards as a way to answer the question of how to establish whether a test measures what it purports to measure. For example, does a test that labels itself a reading test (‘reading test’ is the ‘construct’) in fact measure reading competence or ability?  In this instance the question is whether  PACT  truly measures what it purports to measure - – teaching ability and competence. In making their case for construct validity the authors cite as a source a Cronbach article reprinted in a book published in 1988. The definition cited is drawn from Cronbach and Meehl’s 1955 seminal article on construct validity. 
>
> The confusions over types of validity are numerous and I won’t attempt to deal with them here except to note that thirty-four years after Cronbach and Meehl published their  article on test validity, Sam Messick of ETS, then the acknowledged reigning scientist and chief theoretician of standardized educational testing, in an article in the Educational Researcher (March 1989)  made a major contribution to clearing up the continuing confusions over types of validity and how validity is established. He concluded (and it is now widely accepted) that there are not several types of validity  but only one --construct validity. He argued that content, concurrent and predictive validity are all aspects of construct validity.  Further that the construct validity of an evaluation instrument cannot be established without a full examination of  the possible consequences or the effects the assessment may have on individuals, on the institution in which the assessment is used, and on society. He also argued that assessment instruments to meet the test of validity must be grounded in demonstrable connections to real performance.  Establishing a test’s validity requires study and determination of the anticipated and unanticipated effects of the assessments and exercise of human judgment about the desirability of the consequences.  Do the promised benefits materialize and do they  outweigh the negatives? These questions are not merely technical but moral matters raising fundamental questions of values and priorities. What kinds of teachers do we want to prepare for what kind of schools and society?  How should teachers be educated and how should they be taught to teach and manage classrooms and children clearly rests on value questions that in a democracy ought not to be left to government officials panels of government anointed ‘stakeholders’ and/or to professional experts. The writers claim that the instruments they used are valid yet the most fundamental questions about test validity are unaddressed.  
> The writers do at times acknowledge that the validity of PACT assessments are still problematic since the relationship of the PACT instruments to teachers’ actual classroom performance over time is problematic. Making judgments about, and classifying teacher performance (live or on video) using rating scales in conjunction with Likert type questionnaire items and portfolios cannot substitute for intensive direct observation over time.  The Teacher Performance Expectations are complex and are subject to many interpretations. Consensus on rating scales can only be obtained by training that in effect suppresses what may be valuable differences in perspectives and values.
>
>
> The writers call for multiple measures and caution. This is commendable but politically naive. Is there any doubt that if standardized assessments are used as measurements of teacher preparation and competence that these standardized tests results will overwhelm all other assessments (not to mention devouring teacher educators’ already limited time). Before forging ahead with a plan, we need to identify and study educational impact or consequences and ask whether we are prepared to inflict the same damage in higher education and teacher education that we are in the process of inflicting on public schools with NCLB.
>
> Among the most disturbing (and dazzlingly superficial) statements is the last line in the document to the effect that validity of the TPE/TPAs will be established ultimately by whether they enhance pupil learning. This apparently innocuous statement raises the prospect that the value of teacher education programs will ultimately depend upon and be validated by pupils’ scores on standardized tests. The damage that this continuing focus on standardization and standardized testing will inflict if it goes unchecked will be enormous. If the writers agree that this was not their intention then they have the moral responsibility to make this absolutely clear.  PACT may be a well-intentioned and reasoned response to a bad State law. Nevertheless, if PACT goes forward and is used as presently conceived, we are at great risk of creating a massive statewide bureaucratic system of control in teacher education that in effect represses or ignores dissent, issues of race and cultural diversity, and fosters docile and compliant teachers and citizenry. 
>  Harold Berlak
> <x-tad-bigger> </x-tad-bigger>