Re: bad item

[DMDesiderata@aol.com]

Victor, your comments are very interesting, and for the most part I'd agree 
with your advice on not using this item as an example of a bad item.  I'd 
like to see your examples of bad items and I'd like to see the results of 
Nelson's research.  

Nelson, I'd really like to see the results of your research and I have a 
question......have you considered doing any "thinking aloud" studies beyond 
just this one item?  I'd recommend doing this more extensively and carefully 
documenting and coding the responses.

Now back to items.....I recall something similar to an item involving trains 
and train schedules.  I am making up the numbers for illustrative purposes 
because I do not recall the actual numbers.

A train takes 2 hours to travel from New York City to Trenton.  If Maria 
takes the 8:30 train from New York City, what time will she arrive in 
Trenton?

The correct answer SHOULD be 10:30.  The problem here is that, students know 
that trains can arrive late or not arrive at all.  

This same issue occurs wit items that ask students to determine how much 
change person X. WILL receive.  We do not really know enough to get that 
answer, do we? Better to ask "how much change SHOULD person X receive?" 

Even with the train item, it would be better with the change from "WILL" to 
"SHOULD," but as I recall in the item I am trying to recall, that would not 
have totally fixed the item.  As I recall, the revised item was much wordier 
and my reaction, at the time, was the item is now so complicated that even 
those students that had gotten the item correct would no longer do so for 
reasons of readability.  So much emphasis is being placed on developing items 
that involve real-world contexts that are not contrived, etc.  But the 
real-world is not simplistic so what happens is you either get overly 
contrived items or items in which there could be several legitimate correct 
approaches and answers.  These sort of items would do BETTER on a contracted 
response item or project basis, where the student can state his/her 
assumptions ("givens") and then use the necessary mathematics to support the 
answer and provide an explanation.  These just do not do well on 
multiple-choice tests.

Victor, I agree that this is not the best example of such abuse (and mine are 
probably not that much better), but if one could find a really good example 
of a really bad item, and then have someone try to fix the item to remove the 
problem, you will soon see how complicated the revised item becomes.  At that 
point, you can legally justify the key but the item no longer measures what 
it is intended to measure because the language and context has become too 
complex.  I recall saying this to a test developer and the answer I got was 
that it was better to have an item that could be legally defended even if 
that made the item so complex that few students could get it correct 
(including those that would be expected to get the item correct).

Any thoughts here????




"Miracles happen to those who believe."