[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: bad item



DMD: thanks for elaborating on this. Research is needed into this
area. I agree with Victor that kids "should" be able to handle the CD
item, but I put the quotation marks around "should" purposefully. Most
tests demand that kids think in a peculiar, even unnatural way. By the
time most of us are adults, and esp. if we are good test takers, that
unnatural approach comes to seem normal, and we may forget how
acculturated we have become. I still see the kids' CD items comments as
legitimate (and harmful to the item's validity), but there is much to
learn here...

Nelson

DMDesiderata@aol.com wrote:

>Victor, your comments are very interesting, and for the most part I'd
agree
>with your advice on not using this item as an example of a bad item.
I'd
>like to see your examples of bad items and I'd like to see the results
of
>Nelson's research.
>
>Nelson, I'd really like to see the results of your research and I have
a
>question......have you considered doing any "thinking aloud" studies
beyond
>just this one item? I'd recommend doing this more extensively and
carefully
>documenting and coding the responses.
>
>Now back to items.....I recall something similar to an item involving
trains
>and train schedules. I am making up the numbers for illustrative
purposes
>because I do not recall the actual numbers.
>
>A train takes 2 hours to travel from New York City to Trenton. If
Maria
>takes the 8:30 train from New York City, what time will she arrive in
>Trenton?
>
>The correct answer SHOULD be 10:30. The problem here is that, students
know
>that trains can arrive late or not arrive at all.
>
>This same issue occurs wit items that ask students to determine how
much
>change person X. WILL receive. We do not really know enough to get
that
>answer, do we? Better to ask "how much change SHOULD person X receive?"
>
>Even with the train item, it would be better with the change from
"WILL" to
>"SHOULD," but as I recall in the item I am trying to recall, that would
not
>have totally fixed the item. As I recall, the revised item was much
wordier
>and my reaction, at the time, was the item is now so complicated that
even
>those students that had gotten the item correct would no longer do so
for
>reasons of readability. So much emphasis is being placed on developing
items
>that involve real-world contexts that are not contrived, etc. But the
>real-world is not simplistic so what happens is you either get overly
>contrived items or items in which there could be several legitimate
correct
>approaches and answers. These sort of items would do BETTER on a
contracted
>response item or project basis, where the student can state his/her
>assumptions ("givens") and then use the necessary mathematics to
support the
>answer and provide an explanation. These just do not do well on
>multiple-choice tests.
>
>Victor, I agree that this is not the best example of such abuse (and
mine are
>probably not that much better), but if one could find a really good
example
>of a really bad item, and then have someone try to fix the item to
remove the
>problem, you will soon see how complicated the revised item becomes.
At that
>point, you can legally justify the key but the item no longer measures
what
>it is intended to measure because the language and context has become
too
>complex. I recall saying this to a test developer and the answer I got
was
>that it was better to have an item that could be legally defended even
if
>that made the item so complex that few students could get it correct
>(including those that would be expected to get the item correct).
>
>Any thoughts here????
>
>
>
>
>"Miracles happen to those who believe."
>