[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: bad item

DMD: thanks for elaborating on this. Research is needed into this
area. I agree with Victor that kids "should" be able to handle the CD
item, but I put the quotation marks around "should" purposefully. Most
tests demand that kids think in a peculiar, even unnatural way. By the
time most of us are adults, and esp. if we are good test takers, that
unnatural approach comes to seem normal, and we may forget how
acculturated we have become. I still see the kids' CD items comments as
legitimate (and harmful to the item's validity), but there is much to
learn here...


DMDesiderata@aol.com wrote:

>Victor, your comments are very interesting, and for the most part I'd
>with your advice on not using this item as an example of a bad item.
>like to see your examples of bad items and I'd like to see the results
>Nelson's research.
>Nelson, I'd really like to see the results of your research and I have
>question......have you considered doing any "thinking aloud" studies
>just this one item? I'd recommend doing this more extensively and
>documenting and coding the responses.
>Now back to items.....I recall something similar to an item involving
>and train schedules. I am making up the numbers for illustrative
>because I do not recall the actual numbers.
>A train takes 2 hours to travel from New York City to Trenton. If
>takes the 8:30 train from New York City, what time will she arrive in
>The correct answer SHOULD be 10:30. The problem here is that, students
>that trains can arrive late or not arrive at all.
>This same issue occurs wit items that ask students to determine how
>change person X. WILL receive. We do not really know enough to get
>answer, do we? Better to ask "how much change SHOULD person X receive?"
>Even with the train item, it would be better with the change from
"WILL" to
>"SHOULD," but as I recall in the item I am trying to recall, that would
>have totally fixed the item. As I recall, the revised item was much
>and my reaction, at the time, was the item is now so complicated that
>those students that had gotten the item correct would no longer do so
>reasons of readability. So much emphasis is being placed on developing
>that involve real-world contexts that are not contrived, etc. But the
>real-world is not simplistic so what happens is you either get overly
>contrived items or items in which there could be several legitimate
>approaches and answers. These sort of items would do BETTER on a
>response item or project basis, where the student can state his/her
>assumptions ("givens") and then use the necessary mathematics to
support the
>answer and provide an explanation. These just do not do well on
>multiple-choice tests.
>Victor, I agree that this is not the best example of such abuse (and
mine are
>probably not that much better), but if one could find a really good
>of a really bad item, and then have someone try to fix the item to
remove the
>problem, you will soon see how complicated the revised item becomes.
At that
>point, you can legally justify the key but the item no longer measures
>it is intended to measure because the language and context has become
>complex. I recall saying this to a test developer and the answer I got
>that it was better to have an item that could be legally defended even
>that made the item so complex that few students could get it correct
>(including those that would be expected to get the item correct).
>Any thoughts here????
>"Miracles happen to those who believe."