Re: bad item
At 11:27 PM -0400 05.05.03, DMDesiderata@aol.com wrote:
> And there always may be a wise-ass in the class who will pull out a
schedule and claim that there is no 8:30 train to Trenton. The way
> information that is supposed to be used.
the school-game is played is that the information given is the
Your second sentence greatly concerns me. Isn't one of the purpose of
education and learning ... to enable students (who later become adults) to
make judgments as to whether the information presented is reasonable, likely
to be correct, makes sense, etc.? I do not see much value in simply having
students "accept" on any basis that the information presented MUST be correct
simply because we are playing the game of "school". If students are not
presented with opportunities to question information, when will they ever be
able to do that? Consider this, each day telemarketers call and claim that
their product/service is the best one. Should I simply accept that? Or
should I take in this information, do further research, do whatever
calculations might be appropriate and then decide which is really best?
You're right. It is important to enable students in making judgments.
An important skill is understanding when to question the information
and when not to. You seem to be arguing for ALWAYS questioning if the
information is sufficient to answer the question, but in all my
experience, I do not recall this being a real-life situation--we
always make decisions based on limited amount of information even if
we often wish we had more. This particular issue has nothing to do
with a test--this is something that happens in every school, in every
classroom, every hour the school is open. When we require physical
activity of our students, usually in the form of PE classes, it is
also in a contrived, heavily restricted environment. Hitting a ball
with a stick, plowing through a dozen bodies with an oblong hard
object under your arm, kicking a ball roughly the size of a human
head or bouncing one twice the size before lopping it into a basket
are hardly real-life experiences or ones that will translate easily
into such. You can say the same about children-invented games, with a
few exception, such as playing "doctor";-).
Following orders is also a skill, although we would not want to have
that particular skill to be executed unquestioningly. But being able
to make a decision when to follow orders and when to question them is
much finer skill that I would like to see inculcated, so the argument
But let me get back to the issue of modeling. When we create/follow
physical theories, we don't normally include unreasonable
expectations--when working on such problems, we always have to list
the conditions we are concerned with and ignore the rest. Sometimes,
this backfires--the Tacoma Narrows Bridge is one of the most famous
examples of this. There is also the Butterfly Economics theory--a
butterfly flapping wings in Brazil will somehow affect the stock
market in NY (you might have heard the same expressed regarding
weather patterns). But what is important in this theory is that
although the connection may well exist, it is IMPOSSIBLE to trace it,
so, for the sake of modeling, it is not useful.
I think I'd like to see how you would transform my proposed items from
mediocre to acceptable or good or excellent.
I don't know if I would use these particular items or if I would be
tempted to ever turn them into something excellent. It depends on the
particular array of skills I want to see students demonstrate (note
that I am not saying here what purpose this demonstration
serves--that's another discussion). But, if I were to ask a question
concerning train schedules, I would offer a few more details. In
constructing this particular item, it is important to remember that
there is more than simply the perfection of the conditions that is at
stake--the item has to be readable, readable in a limited amount of
time (not "timed", but not unlimited either), doable within a limited
amount of time using skilled students are likely to possess. That
means that I do not want to make the text particularly long, I want
to keep the language simple and, yes, I do have to leave some
conditions unstated because this is how the real world works, even if
the situation is contrived.
For the train (or bus or plane) item, let me try something like this.
You (or "your parents", or "Billie Joe Bob" or "Bubba" or whoever)
are planning a trip from New York to Trenton. The train for Trenton
is scheduled to leave New York at 8:30 A.M. and, according to the
schedule, the trip takes two and a half hours. If you want your
friend to meet you at the station, what time should you tell him to
expect you there?
You'd be amazed how doubt will disappear if the conditions are
properly set. But, as is the case with the items, you should not
expect anything perfect--there might always be a student with doubts
or, quite simply, a bonehead who is clueless about what is being
asked. We may believe that all students are CAPABLE of producing
great work, but it does not mean that we should EXPECT all students
submitting great work. When teaching, I give every opportunity for
this capacity to manifest itself, but if someone refuses to take
advantage of these opportunities or blatantly refuses to learn, I
will fail him just the same--and I am not about to wear sack cloth
and put ashes on my head over him.
My expectations on this particular item would be a single, specific
time (11 A.M.), but I would accept other REASONABLE answers with
partial or full credit, depending on the answer and the circumstances.
But suppose, in the case of my train item, I
actually state that the train actually arrived in Trenton on schedule?
As you can see, my rephrasing eliminates even this condition as necessary.
This removes cases of train derailment, comets destroying Earth,
train delays, etc. While I agree that we will never be able to
address all of the possible
conditions, test developers CAN do a better job of addressing the more likely
unsaid conditions that would distract students from responding to the item.
Of course, they can! I was not saying otherwise. But don't forget
that test usually reflect what is being taught. If you teach your
students to always question the conditions of the problem, you'd be
doing them a disservice.
But, even if the train arrives on schedule, Maria may not be on
it--not alive, at least. She might have been mugged or had a heart
attack on the train. She might have had the urge to jump off, or,
changed her mind, got off somewhere in the middle and took a train
back to New York. Are you suggesting that test developers should also
include wording that Maria is alive and on the train when it arrives
Someone has to make a determination of what are reasonable and
unreasonable conditions to be included in the problem and the
students are not the ones, for the most part, who will be making
those decisions. I expect most of these decision to be made FOR THEM
in advance and I expect students to know this. If they don't, then
the test will reveal this. A better written test will differentiate
between refusing to follow the rules and being unable to do so.
> Had this been an item I was discussing with students in class, the
first two concerns would require further investigation--that is,
students should be able to follow the line of thought they presented
to discover that neither has any impact on the outcome
The above would be not be possible via the multiple-choice item type.
Nonsense! I have no love for multiple-choice items, but what the
student does is up to him, not the item format. The students can
still follow the train of thought even if, in the end, all he has to
do is bubble in an oval.
entire item would be much better and appropriate as a constructed response
item or discovery project, wherein such issues can be further explored and
discovered by the student.
EVERY item would be much better off as a constructed response item or
a discovery project. But another life skill that students should
learn from us is that it is not always up to us to make that decision.
> I have no sympathy for these students--they did not answer the
question in a reasonable manner. Period. Their explanations have
> It is also not something that is likely to EVER lead them to an
absolutely nothing to do with the item and appear to be post
hoc--that is, this is not something they thought of during the test.
answer. You were not looking at a philosophy class--the test was in
math, was it not? So why all this digression? If you want these
students to approach the test item philosophically, find another
venue to do so--don't do it in a math class.
VS.....by any chance are you a test developer? If you teach, please tell me
which level (high school, college, etc.) and the course (math, testing and
measurement, or other)? Just curious ....
No chance at all--I am not a test developer, not even an item writer
(although if someone asks me and pays me to do it, I will certainly
consider it). I generally teach college courses, although the "level"
is another matter--I often teach college courses that simply rehash
high-school math, and even have high-school students in my classes. I
do not teach anything concerning testing and measurement.
I have, however, been involved in some research on assessment
development. Check out the Balanced Assessment Project in Mathematics
(there are several Dale Seymore volumes and BAMP/MARS websites that
distribute the tasks).
I was asked earlier for examples of bad questions. Here are three
from the most recent (December) MCAS retest--the one that determined
that 10% of high-school students in Massachusetts public schools are
not fit to graduate. Each is bad for different reasons.
MCAS ITEM 1(16):
TABLE: Admission Costs: Basketball Hall of Fame
General Admission $9.00 each
Groups of 15 or more $6.00 each
Two groups of high school students visited the Basketball Hall of
Fame in Springfield, Massachusetts. The two groups did not have the
same number of students, but each group paid a total of $108 for
their tickets. How many students were in EACH group?
Correct Answer: 12, 17, and 18
VS: This is bad not so much because of the question itself, although
it is rather contrived (does anyone really care if two groups paid
the same amount?). The answer is just bad, especially as listed. 12
and 18 are easy to see--12 tickets cost 12 * 9 = $108 and 18 cost 18
* 6 = $108. But 17? Apparently, the logic was 17 is two over 15, so
15 * 6 + 2 * 9 = $108. However, this contradicts the conditions of
the problem! All it says is that groups of 15 or more are priced at
$6 per person, so a group of 17 should cost 17 * 6 = $102, not $108!
But even had 17 been acceptable, the "correct answers" are "12 and
17", "12 and 18" or "17 and 18", not "12, 17, and 18". I suppose,
that someone at the DOE was following the line of reasoning in this
thread and decided to give the students who saw 17 as a part of the
answer the benefit of a doubt. They were wrong, as were the students.
MCAS ITEM 2(17):
On a high school football team, the 7 starting players on the
offensive line weigh 192, 217, 235, 224, 218, 210, and 195 pounds.
The 4 starting players in the offensive backfield weigh 180, 169,
182, and 177 pounds.
a. What is the mean weight of the offensive linemen? Round you answer
to the nearest whole pound. Show all your work.
b. What is the mean weight of the offensive backfield? Round your
answer to the nearest whole pound. Show your work.
c. What is the mean weight of the entire team? Round your answer to
the nearest whole pound. Show or explain how you got your answer.
d. Is the mean weight of the entire team closer to the mean weight of
the offensive linemen than to the mean weight of the offensive
backfield? Explain your reasoning.
VS: Let me rephrase the first part of the item.
"On a blog team, 7 starting clogs weigh 192, 217, 235, 224, 218, 210,
and 195 pounds. The 4 starting dlogs weigh 180, 169, 182, and 177
The rest of the text would be changed appropriately. Can this
question be answered in the same manner as the one on the MCAS? If
not, the question is inappropriate--it is culturally biased. There is
no reason why this question should require the knowledge of even a
single football formation. Furthermore, the initial text talks about
the "starting offense", while the parts (c) and (d) talk about the
"entire team". Bad question.
MCAS ITEM 3(18):
What are the solutions to the equation below?
2x^2 - 11x - 21 = 0
A. 7, -1.5
B. 4.5, 1.0
C. 1, -21
D. 7, -3
VS: At first pass, there seems to be little wrong with this. But, if
the goal of the question is to assess student ability to solve a
quadratic equation, the question fails miserably. There is absolutely
no reason to solve the equation. There are two possible approaches,
one is more insightful, the other less so. The more ingenious student
can look at the equation and notice that the product of the two
"solutions" (actually inappropriate language, but common enough to
let pass) is -21/2. The only pair that achieves this is A. On the
other hand, if all a student knows is that a number is a solution if
it can be substituted into the equation to produce a true statement,
one only needs to test a single number from each pair to determine if
the pair qualifies. The choices can be further reduced by noticing
that 7 and 1 are the only numbers that appears twice. So, testing 7
and 1 will reduce the number of possible choices in half, then
testing 3 will eliminate D from consideration and leave only A as the
answer. Alternately, an even weaker student might test the number in
order. Having discovered that A works, he would--and should--stop.
Again, the item is not bad because it leads students astray--it is
not likely to do so. It is bad because it does not address any stated
goal and really does not test much that is meaningful. Even if the
question is answered correctly we have absolutely no idea what the
Note that I listed here one item of each time--short response, open
response and multiple choice, so the format does not determine the
"badness" of the item.
Post a Message to arn-l: