[ Date Prev][ Date Next][ Thread Prev][ Thread Next][ Date Index][ Thread Index]
Re: Is "Grade Level" A Load of Hooey?
Jerry - thanks for this. I've been meaning to read your "Snookered"
book for a while and will now order it from Amazon.
http://www.amazon.com/Reading-Educational-Research-Statistically-
Snookered/dp/0325008582
One more question on this: how does all this -- cut scores, etc. --
relate to "grade level"? What is the relationship between them? In
your running a mile in 10 minutes example, the ability to run a mile
in 10 minutes is the standard; is it also the cut score? And, if you
know that most kids can run a mile in 10 minutes and, lo and behold,
they do, that means you have very low standards, yes? But you'd be
able to say that all kids in your state are at grade level for the
mile. Conversely, in another state, they set the mile time at 6
minutes, something most kids cannot do. The state prides itself on
high standards, but only a few kids are at grade level.
Accurate paraphrase/extension of your argument?
Peter
On Apr 7, 2009, at 7:25 PM, gerald bracey wrote:
Peter:
Standards-based tests are not criterion referenced tests in the
true sense,
but that is another post for another time.
The test score distributions I've seen from Texas are so skewed to the
negative end, it would be hard to call them norm-reference. Counter
intuitively, "skewed to the negative end" means that most people
score high.
In some, there were even suggestions of ceiling effects. There is
always a
distribution of SCORES, but states are not reporting scores any
more, they
are reporting only percent meeting standard, percent proficient,
etc., and,
theoretically all students can jump over that barrier.
If your standard says "all students will run a mile in 10 minutes"
you can
get nearly 100% of the kids to do that although there will be some who
cannot do it because of physical infirmities. You will not know,
though,
how many kids ran the distance in 9:59 and how many ran it in 6:00
(a speed
I used to use for myself 40 years ago). All you know is the
percent meeting
the standard.
The cut score is not necessarily--I think, not even usually--
decided after
the results are in. In the typical situation, say, in mathematics,
about 20
people judge each item in terms of the probability that a "minimally
competent person" would get the item right (if you want to spend
some time
running in circles, try to get your mind around the idea of a
minimally
competent person). The 20 might be math teachers, supervisors, math
education profs. Summing across items for each person gives you 20
possible
cut scores. The recommendation is to pick a real cut score
somewhere in the
middle of the range. In Virginia, some cut scores were set higher
than any
recommended by a group because it felt the Board needed to look
tough--in
the other 19 cases, the highest recommended cut score was adopted.
It has
since had to lower the score on several tests because so many kids
flunked.
Sometimes, another round of judgments is used where judges are given
information on what proportion of students actually got the item
right. To
me, this is important because people who only look at an item and
who don't
take the test or see such data INVARIABLY think the item/test is
easier than
in fact it is.
The most honest way to set a cut score is to decide in advance how
many kids
you want to flunk. Really. There is no scientifically defensible or
technically feasible way to set a cut score, something Gene Glass
pointed
out in "Standards and Criteria" in a 1978 article in the Journal of
Educational Measurement. People as ideologically opposite as I and
Charlie
Murray have both pointed out that moving the cut score around can
create or
minimize "achievement gaps." The cut scores in Texas were
apparently set so
that initially, many black and Hispanic kids flunked, but close
enough to
their average scores that small improvements in scores produced large
increases in passing rates, thus producing a socially desirable
miracle.
Setting a cut score is always arbitrary and often politically
driven. You
can only hope that it is arbitrary in the sense of reflecting
arbitration,
not in the other sense of capricious. But it throws away the most
important
information--the scores themselves, assuming the test measures
something
worthwhile in the first place.
Take a look at the long section on tests in "Reading Educational
Research:
How to Avoid Getting Statistically Snookered." This is America--
gotta have
a commercial.
Jerry
-------Original Message-------
From: Peter Campbell
Date: 4/7/2009 6:58:45 PM
To: arn-l@interversity.org
Subject: Re: [arn-l] Is "Grade Level" A Load of Hooey?
On Apr 7, 2009, at 12:20 PM, gerald bracey wrote:
Peter is right that "grade level" has traditionally been defined
as the score of the average child in a given grade on a norm-
referenced test
(NRT). It is thus a floating standard and might be great or
awful. But
virtually no one uses NRT's anymore so that def. is not available.
Paige
and Spellings changed the word "proficiency" in NCLB to "grade level"
thereby giving us 50 definitions of grade level--grade level is
whatever a
state defines as proficient--and since everyone must be proficient
by 2014,
it is attainable by all (theoretically).
Jerry - I've read in different places that criterion-referenced
tests/ standards-based tests are just NRT's in sheep's clothing. Any
validity to that claim? With NRT's, my understanding is the results
are distributed along a normalized curve. So there's the the usual
distribution of scores, i.e, half above and half below grade level.
But then the cut score is decided upon after the results of the tests
come in. So you have, on the one hand, a methodology in the creation
of the tests that assures a normalized distribution, but then a
little statistical trickery after the fact that assures a higher
passing rate above the cut score. I'd appreciate getting some clarity
on this. It's still muddy to me.
On the other hand, given some of the scenarios Linda Perlstein
describes in
Tested: One American School Struggles to Make the Grade, it is hard
to argue
that these kids are on a "continuum" of any kind. They not only
don't know
the words associated with the images they're expected to respond
to in
Dibels, many don't have a clue about what reading involves. I
agree, Peter,
that "they are where they are" but it's not as simple as the
concept of a
continuum would make it seem. "Managing a class means making a
constant
stream of decisions about which misbehaviors to address and which
to ignore"
(p. 111). If you read her chapter where she goes to a nearby
wealthy school
(she spent over a year in the poor one), you come away thinking
"different
worlds", not continuum.
Well, it's still where they are, isn't it? I've read Perlstein's book
and was struck by what the school did to the kids that were "behind."
In reading this book, I was saddened, enraged, and disgusted. After
being exposed to a constant regimen of "BCR's" (brief constructed
responses), decoding drills, and endless test prep, it would be
surprising if these kids ever wanted to read anything ever again.
Thus my point: let's not employ punitive pedagogy that makes the
situation even worse than it already is.
I don't think there is a press that "ALL kids have to be equally
good at
reading." Even NCLB doesn't require that.
NCLB requires that all kids be proficient/at grade level by 2014.
It is hard to know what reading
level is sufficient to cope with the real world because reading is
only one
way adults gather the information they need. That's probably why
people who
don't score high in adult literacy tests (which have many flaws)
don't
report that their reading level causes them real life
difficulties. Still,
the average black 12th grader scored 267 in 2005 compared to 293
for whites.
That's about a 2 1/2 year difference as NAEP goes and slightly
below the
271 of white 8th graders. It's hard to think of that as "slightly
better"
for whites.
I don't get it. You usually bash NAEP data, Jerry. But now you're
using NAEP data uncritically to make a point? Even so, what do we do
about this? If these kids really are "behind," then what actions do
we take to help them? Do we take away art and music and recess and
load them up on phonics drills and worksheets? Or do we do something
else? Of course, the "something else" is open for debate. But what
we're doing now is not working. Period.
Peter
Post a Message to arn-l:
|