Re: bad item
- To: firstname.lastname@example.org
- Subject: Re: bad item
- From: DMDesiderata@aol.com
- Date: Mon, 5 May 2003 12:36:30 EDT
Victor, your comments are very interesting, and for the most part I'd agree
with your advice on not using this item as an example of a bad item. I'd
like to see your examples of bad items and I'd like to see the results of
Nelson, I'd really like to see the results of your research and I have a
question......have you considered doing any "thinking aloud" studies beyond
just this one item? I'd recommend doing this more extensively and carefully
documenting and coding the responses.
Now back to items.....I recall something similar to an item involving trains
and train schedules. I am making up the numbers for illustrative purposes
because I do not recall the actual numbers.
A train takes 2 hours to travel from New York City to Trenton. If Maria
takes the 8:30 train from New York City, what time will she arrive in
The correct answer SHOULD be 10:30. The problem here is that, students know
that trains can arrive late or not arrive at all.
This same issue occurs wit items that ask students to determine how much
change person X. WILL receive. We do not really know enough to get that
answer, do we? Better to ask "how much change SHOULD person X receive?"
Even with the train item, it would be better with the change from "WILL" to
"SHOULD," but as I recall in the item I am trying to recall, that would not
have totally fixed the item. As I recall, the revised item was much wordier
and my reaction, at the time, was the item is now so complicated that even
those students that had gotten the item correct would no longer do so for
reasons of readability. So much emphasis is being placed on developing items
that involve real-world contexts that are not contrived, etc. But the
real-world is not simplistic so what happens is you either get overly
contrived items or items in which there could be several legitimate correct
approaches and answers. These sort of items would do BETTER on a contracted
response item or project basis, where the student can state his/her
assumptions ("givens") and then use the necessary mathematics to support the
answer and provide an explanation. These just do not do well on
Victor, I agree that this is not the best example of such abuse (and mine are
probably not that much better), but if one could find a really good example
of a really bad item, and then have someone try to fix the item to remove the
problem, you will soon see how complicated the revised item becomes. At that
point, you can legally justify the key but the item no longer measures what
it is intended to measure because the language and context has become too
complex. I recall saying this to a test developer and the answer I got was
that it was better to have an item that could be legally defended even if
that made the item so complex that few students could get it correct
(including those that would be expected to get the item correct).
Any thoughts here????
"Miracles happen to those who believe."
Post a Message to arn-l: