Re: bad item

[Victor Steinbok <Victor.Steinbok@verizon.net>]

Oddly, I did not receive DMD's message.

> DMDesiderata@aol.com wrote:
>  >...Now back to items.....I recall something similar to an item 
> involving trains and train schedules.  I am making up the numbers 
> for illustrative purposes because I do not recall the actual numbers.

We can all agree that most word problems (and almost anything that is 
given in "context") in mathematics teaching is contrived. This is not 
news. One has to consider, however, that students are not asked about 
real life. They are stuck in a world-game called "school" and the 
particular nature of the math questions they have to answer is a part 
of the game. If you want to ask them questions about real-life 
situations don't do it in school context.

>  >A train takes 2 hours to travel from New York City to Trenton.  If 
> Maria takes the 8:30 train from New York City, what time will she 
> arrive in Trenton?
>  >
> > The correct answer SHOULD be 10:30.  The problem here is that, students
> know  that trains can arrive late or not arrive at all.

And there always may be a wise-ass in the class who will pull out a 
schedule and claim that there is no 8:30 train to Trenton. The way 
the school-game is played is that the information given is the 
information that is supposed to be used. The same often applies to 
essay questions concerning a reasoning paragraph students are 
supposed to critique. These make poor teaching items, but they are 
being used.  HOWEVER, in this case, there is very little ambiguity. 
In fact, some of the students' assumption are directly contrary to 
the conditions stated in the task.

>  >This same issue occurs wit items that ask students to determine 
> how much change person X. WILL receive.  We do not really know 
> enough to get that answer, do we? Better to ask "how much change 
> SHOULD person X receive?"

This is simply a difference between a poor and a mediocre item. One 
should not introduce more ambiguity into the question than necessary, 
but no predictive language (e.g., what will happen) should be used 
either. A particularly extreme example of such error is in 
probability questions, something like this:

A game is fair if each player has equal probability of winning. Joe 
and Bob play game Quack 20 times. Joe won 16 times. Is the game fair?

This is an ill posed question since we are dealing with 
probabilistic, not deterministic calculations. Joe could have won all 
20 times and we still could not be certain that the game was NOT 
fair. BUT, in looking at this question, one can say that the game is 
LIKELY to be unfair. A more REALISTIC approach would be to ask 
whether the game Quack involves skill as well as luck, in which case, 
Joe simply could have been a better skilled player. The first 
criticism of the item--that the question is posed incorrectly--is 
valid. The second, although perfectly reasonable in analyzing the 
situation--is not, because it requires suspension of the rules of the 
school-game.

>  >Victor, I agree that this is not the best example of such abuse 
> (and mine are probably not that much better), but if one could find 
> a really good example of a really bad item, and then have someone 
> try to fix the item to remove the problem, you will soon see how 
> complicated the revised item becomes. At that point, you can legally 
> justify the key but the item no longer measures what it is intended 
> to measure because the language and context has become too complex. 
> I recall saying this to a test developer and the answer I got was 
> that it was better to have an item that could be legally defended 
> even if that made the item so complex that few students could get it 
> correct (including those that would be expected to get the item 
> correct).
>  >
>  >Any thoughts here????

One could play this game forever, since no matter how specific you 
make the item, there will always be some conditions that remain 
unsaid (might a comet hit Earth and destroy all of humanity? Does the 
engineer smoke pot on the train, making it more likely to derail? 
etc.) The list is simply endless. The way to decide wether the item 
is reasonable or not--that is, whether the expectations of answering 
the item in a particular manner are reasonable--is to never forget 
that the real context of the question is the school-game, not the 
actual situation.

At 12:50 PM -0400 05.05.03, Nelson J Maylone wrote:
> DMD:  thanks for elaborating on this. Research is needed into this
> area. I agree with Victor that kids "should" be able to handle the CD
> item, but I put the quotation marks around "should" purposefully. Most
> tests demand that kids think in a peculiar, even unnatural way. By the
> time most of us are adults, and esp. if we are good test takers, that
> unnatural approach comes to seem normal, and we may forget how
> acculturated we have become. I still see the kids' CD items comments as
> legitimate (and harmful to the item's validity), but there is much to
> learn here...

I can rephrase the "school-game" in a less philosophical 
language--what students are expected to understand is that the item 
presents a model, not the actual situation. As such, they are 
constrained by the model--the main unstated assumption is that any 
other assumptions that are not stated are irrelevant to answering the 
question. In the case of the CD item, it is much worse--some of the 
assumptions students are making are IMPROBABLE, other assumptions are 
ludicrous, and the assumptions that are reasonable are not followed 
through.

Had this been an item I was discussing with students in class, the 
first two concerns would require further investigation--that is, 
students should be able to follow the line of thought they presented 
to discover that neither has any impact on the outcome (tax is 
proportional, therefore preserving the order from least expensive to 
most expensive; shipping is the same additional cost in each case, 
therefore preserving the order as well). The other two assumptions 
are the ones I identify as IMPROBABLE--there is no suggestion in the 
text of the item to suggest that there even MIGHT be a difference in 
the way the dues are applied, let alone that such difference is 
LIKELY. The concern over the choice of selections is also completely 
ludicrous because it directly contradicts the conditions stated in 
the item--you MUST buy 5 CDs and all the CDs will have the same price.

I have no sympathy for these students--they did not answer the 
question in a reasonable manner. Period. Their explanations have 
absolutely nothing to do with the item and appear to be post 
hoc--that is, this is not something they thought of during the test. 
It is also not something that is likely to EVER lead them to an 
answer. You were not looking at a philosophy class--the test was in 
math, was it not? So why all this digression? If you want these 
students to approach the test item philosophically, find another 
venue to do so--don't do it in a math class.

	VS-)