Re: bad item
Oddly, I did not receive DMD's message.
>...Now back to items.....I recall something similar to an item
involving trains and train schedules. I am making up the numbers
for illustrative purposes because I do not recall the actual numbers.
We can all agree that most word problems (and almost anything that is
given in "context") in mathematics teaching is contrived. This is not
news. One has to consider, however, that students are not asked about
real life. They are stuck in a world-game called "school" and the
particular nature of the math questions they have to answer is a part
of the game. If you want to ask them questions about real-life
situations don't do it in school context.
>A train takes 2 hours to travel from New York City to Trenton. If
Maria takes the 8:30 train from New York City, what time will she
arrive in Trenton?
The correct answer SHOULD be 10:30. The problem here is that, students
know that trains can arrive late or not arrive at all.
And there always may be a wise-ass in the class who will pull out a
schedule and claim that there is no 8:30 train to Trenton. The way
the school-game is played is that the information given is the
information that is supposed to be used. The same often applies to
essay questions concerning a reasoning paragraph students are
supposed to critique. These make poor teaching items, but they are
being used. HOWEVER, in this case, there is very little ambiguity.
In fact, some of the students' assumption are directly contrary to
the conditions stated in the task.
>This same issue occurs wit items that ask students to determine
how much change person X. WILL receive. We do not really know
enough to get that answer, do we? Better to ask "how much change
SHOULD person X receive?"
This is simply a difference between a poor and a mediocre item. One
should not introduce more ambiguity into the question than necessary,
but no predictive language (e.g., what will happen) should be used
either. A particularly extreme example of such error is in
probability questions, something like this:
A game is fair if each player has equal probability of winning. Joe
and Bob play game Quack 20 times. Joe won 16 times. Is the game fair?
This is an ill posed question since we are dealing with
probabilistic, not deterministic calculations. Joe could have won all
20 times and we still could not be certain that the game was NOT
fair. BUT, in looking at this question, one can say that the game is
LIKELY to be unfair. A more REALISTIC approach would be to ask
whether the game Quack involves skill as well as luck, in which case,
Joe simply could have been a better skilled player. The first
criticism of the item--that the question is posed incorrectly--is
valid. The second, although perfectly reasonable in analyzing the
situation--is not, because it requires suspension of the rules of the
>Victor, I agree that this is not the best example of such abuse
(and mine are probably not that much better), but if one could find
a really good example of a really bad item, and then have someone
try to fix the item to remove the problem, you will soon see how
complicated the revised item becomes. At that point, you can legally
justify the key but the item no longer measures what it is intended
to measure because the language and context has become too complex.
I recall saying this to a test developer and the answer I got was
that it was better to have an item that could be legally defended
even if that made the item so complex that few students could get it
correct (including those that would be expected to get the item
>Any thoughts here????
One could play this game forever, since no matter how specific you
make the item, there will always be some conditions that remain
unsaid (might a comet hit Earth and destroy all of humanity? Does the
engineer smoke pot on the train, making it more likely to derail?
etc.) The list is simply endless. The way to decide wether the item
is reasonable or not--that is, whether the expectations of answering
the item in a particular manner are reasonable--is to never forget
that the real context of the question is the school-game, not the
At 12:50 PM -0400 05.05.03, Nelson J Maylone wrote:
DMD: thanks for elaborating on this. Research is needed into this
area. I agree with Victor that kids "should" be able to handle the CD
item, but I put the quotation marks around "should" purposefully. Most
tests demand that kids think in a peculiar, even unnatural way. By the
time most of us are adults, and esp. if we are good test takers, that
unnatural approach comes to seem normal, and we may forget how
acculturated we have become. I still see the kids' CD items comments as
legitimate (and harmful to the item's validity), but there is much to
I can rephrase the "school-game" in a less philosophical
language--what students are expected to understand is that the item
presents a model, not the actual situation. As such, they are
constrained by the model--the main unstated assumption is that any
other assumptions that are not stated are irrelevant to answering the
question. In the case of the CD item, it is much worse--some of the
assumptions students are making are IMPROBABLE, other assumptions are
ludicrous, and the assumptions that are reasonable are not followed
Had this been an item I was discussing with students in class, the
first two concerns would require further investigation--that is,
students should be able to follow the line of thought they presented
to discover that neither has any impact on the outcome (tax is
proportional, therefore preserving the order from least expensive to
most expensive; shipping is the same additional cost in each case,
therefore preserving the order as well). The other two assumptions
are the ones I identify as IMPROBABLE--there is no suggestion in the
text of the item to suggest that there even MIGHT be a difference in
the way the dues are applied, let alone that such difference is
LIKELY. The concern over the choice of selections is also completely
ludicrous because it directly contradicts the conditions stated in
the item--you MUST buy 5 CDs and all the CDs will have the same price.
I have no sympathy for these students--they did not answer the
question in a reasonable manner. Period. Their explanations have
absolutely nothing to do with the item and appear to be post
hoc--that is, this is not something they thought of during the test.
It is also not something that is likely to EVER lead them to an
answer. You were not looking at a philosophy class--the test was in
math, was it not? So why all this digression? If you want these
students to approach the test item philosophically, find another
venue to do so--don't do it in a math class.
- Re: bad item
- From: Nelson J Maylone <firstname.lastname@example.org>
Post a Message to arn-l: