Not the same test - What is being measured?

[George Sheridan <learn@jps.net>]

Will second grade scores on California's mathematics test rise this  year?

Rise or fall, the scores will tell more about students' knowledge of 
geometry and less about their computational ability than scores on 
past tests. I don't know if any official body like the State Board of 
Education or the Curriculum Commission decided to change the weight 
assigned to various math standards this year, but it appeared to me 
as I administered the test to my students that there were fewer 
problems involving addition and subtraction of two- and three-digit 
numbers with regrouping (borrowing and carrying) than in past years.

At the same time, there were more problems than I recall from 
previous years in which the student had to identify what shapes could 
be constructed using a pair or two pair of identical triangles.

If scores go up, will that be reported as evidence of achievement? 
Will pundits and politicians proclaim that programs are working and 
students are learning more than ever before? Or will psychometricians 
statistically manipulate the scores so that this year's numbers are 
supposedly comparable to last year's?

When scores are adjusted to account for the "difficulty" of the 
items, who can separate the difficulty of the content being measured 
from the difficulty of the particular question and format?

In the end, who knows what really is being measured? Even within the 
artificial world of standardized tests, the various uncertainties 
compound to the point that we can have little confidence that a 
higher score means anything at all.


George Sheridan