Re: Is "Grade Level" A Load of Hooey?
Jerry - thanks for this. I've been meaning to read your "Snookered"
book for a while and will now order it from Amazon.
One more question on this: how does all this -- cut scores, etc. --
relate to "grade level"? What is the relationship between them? In
your running a mile in 10 minutes example, the ability to run a mile
in 10 minutes is the standard; is it also the cut score? And, if you
know that most kids can run a mile in 10 minutes and, lo and behold,
they do, that means you have very low standards, yes? But you'd be
able to say that all kids in your state are at grade level for the
mile. Conversely, in another state, they set the mile time at 6
minutes, something most kids cannot do. The state prides itself on
high standards, but only a few kids are at grade level.
Accurate paraphrase/extension of your argument?
On Apr 7, 2009, at 7:25 PM, gerald bracey wrote:
Standards-based tests are not criterion referenced tests in the
but that is another post for another time.
The test score distributions I've seen from Texas are so skewed to the
negative end, it would be hard to call them norm-reference. Counter
intuitively, "skewed to the negative end" means that most people
In some, there were even suggestions of ceiling effects. There is
distribution of SCORES, but states are not reporting scores any
are reporting only percent meeting standard, percent proficient,
theoretically all students can jump over that barrier.
If your standard says "all students will run a mile in 10 minutes"
get nearly 100% of the kids to do that although there will be some who
cannot do it because of physical infirmities. You will not know,
how many kids ran the distance in 9:59 and how many ran it in 6:00
I used to use for myself 40 years ago). All you know is the
The cut score is not necessarily--I think, not even usually--
the results are in. In the typical situation, say, in mathematics,
people judge each item in terms of the probability that a "minimally
competent person" would get the item right (if you want to spend
running in circles, try to get your mind around the idea of a
competent person). The 20 might be math teachers, supervisors, math
education profs. Summing across items for each person gives you 20
cut scores. The recommendation is to pick a real cut score
somewhere in the
middle of the range. In Virginia, some cut scores were set higher
recommended by a group because it felt the Board needed to look
the other 19 cases, the highest recommended cut score was adopted.
since had to lower the score on several tests because so many kids
Sometimes, another round of judgments is used where judges are given
information on what proportion of students actually got the item
me, this is important because people who only look at an item and
take the test or see such data INVARIABLY think the item/test is
in fact it is.
The most honest way to set a cut score is to decide in advance how
you want to flunk. Really. There is no scientifically defensible or
technically feasible way to set a cut score, something Gene Glass
out in "Standards and Criteria" in a 1978 article in the Journal of
Educational Measurement. People as ideologically opposite as I and
Murray have both pointed out that moving the cut score around can
minimize "achievement gaps." The cut scores in Texas were
apparently set so
that initially, many black and Hispanic kids flunked, but close
their average scores that small improvements in scores produced large
increases in passing rates, thus producing a socially desirable
Setting a cut score is always arbitrary and often politically
can only hope that it is arbitrary in the sense of reflecting
not in the other sense of capricious. But it throws away the most
information--the scores themselves, assuming the test measures
worthwhile in the first place.
Take a look at the long section on tests in "Reading Educational
How to Avoid Getting Statistically Snookered." This is America--
From: Peter Campbell
Date: 4/7/2009 6:58:45 PM
Subject: Re: [arn-l] Is "Grade Level" A Load of Hooey?
On Apr 7, 2009, at 12:20 PM, gerald bracey wrote:
Peter is right that "grade level" has traditionally been defined
as the score of the average child in a given grade on a norm-
(NRT). It is thus a floating standard and might be great or
virtually no one uses NRT's anymore so that def. is not available.
and Spellings changed the word "proficiency" in NCLB to "grade level"
thereby giving us 50 definitions of grade level--grade level is
state defines as proficient--and since everyone must be proficient
it is attainable by all (theoretically).
Jerry - I've read in different places that criterion-referenced
tests/ standards-based tests are just NRT's in sheep's clothing. Any
validity to that claim? With NRT's, my understanding is the results
are distributed along a normalized curve. So there's the the usual
distribution of scores, i.e, half above and half below grade level.
But then the cut score is decided upon after the results of the tests
come in. So you have, on the one hand, a methodology in the creation
of the tests that assures a normalized distribution, but then a
little statistical trickery after the fact that assures a higher
passing rate above the cut score. I'd appreciate getting some clarity
on this. It's still muddy to me.
On the other hand, given some of the scenarios Linda Perlstein
Tested: One American School Struggles to Make the Grade, it is hard
that these kids are on a "continuum" of any kind. They not only
the words associated with the images they're expected to respond
Dibels, many don't have a clue about what reading involves. I
that "they are where they are" but it's not as simple as the
concept of a
continuum would make it seem. "Managing a class means making a
stream of decisions about which misbehaviors to address and which
(p. 111). If you read her chapter where she goes to a nearby
(she spent over a year in the poor one), you come away thinking
worlds", not continuum.
Well, it's still where they are, isn't it? I've read Perlstein's book
and was struck by what the school did to the kids that were "behind."
In reading this book, I was saddened, enraged, and disgusted. After
being exposed to a constant regimen of "BCR's" (brief constructed
responses), decoding drills, and endless test prep, it would be
surprising if these kids ever wanted to read anything ever again.
Thus my point: let's not employ punitive pedagogy that makes the
situation even worse than it already is.
I don't think there is a press that "ALL kids have to be equally
reading." Even NCLB doesn't require that.
NCLB requires that all kids be proficient/at grade level by 2014.
It is hard to know what reading
level is sufficient to cope with the real world because reading is
way adults gather the information they need. That's probably why
don't score high in adult literacy tests (which have many flaws)
report that their reading level causes them real life
the average black 12th grader scored 267 in 2005 compared to 293
That's about a 2 1/2 year difference as NAEP goes and slightly
271 of white 8th graders. It's hard to think of that as "slightly
I don't get it. You usually bash NAEP data, Jerry. But now you're
using NAEP data uncritically to make a point? Even so, what do we do
about this? If these kids really are "behind," then what actions do
we take to help them? Do we take away art and music and recess and
load them up on phonics drills and worksheets? Or do we do something
else? Of course, the "something else" is open for debate. But what
we're doing now is not working. Period.
Post a Message to arn-l: