Re: bad item

[Victor Steinbok <Victor.Steinbok@verizon.net>]

At 11:27 PM -0400 05.05.03, DMDesiderata@aol.com wrote:
>  > And there always may be a wise-ass in the class who will pull out a
> >  schedule and claim that there is no 8:30 train to Trenton. The way
> >  the school-game is played is that the information given is the
>  > information that is supposed to be used.
>
> Your second sentence greatly concerns me.  Isn't one of the purpose of
> education and learning ...  to enable students (who later become adults) to
> make judgments as to whether the information presented is reasonable, likely
> to be correct, makes sense, etc.?  I do not see much value in simply having
> students "accept" on any basis that the information presented MUST be correct
> simply because we are playing the game of "school".  If students are not
> presented with opportunities to question information, when will they ever be
> able to do that? Consider this, each day telemarketers call and claim that
> their product/service is the best one.  Should I simply accept that?  Or
> should I take in this information, do further research, do whatever
> calculations might be appropriate and then decide which is really best?

You're right. It is important to enable students in making judgments. 
An important skill is understanding when to question the information 
and when not to. You seem to be arguing for ALWAYS questioning if the 
information is sufficient to answer the question, but in all my 
experience, I do not recall this being a real-life situation--we 
always make decisions based on limited amount of information even if 
we often wish we had more. This particular issue has nothing to do 
with a test--this is something that happens in every school, in every 
classroom, every hour the school is open. When we require physical 
activity of our students, usually in the form of PE classes, it is 
also in a contrived, heavily restricted environment. Hitting a ball 
with a stick, plowing through a dozen bodies with an oblong hard 
object under your arm, kicking a ball roughly the size of a human 
head or bouncing one twice the size before lopping it into a basket 
are hardly real-life experiences or ones that will translate easily 
into such. You can say the same about children-invented games, with a 
few exception, such as playing "doctor";-).

Following orders is also a skill, although we would not want to have 
that particular skill to be executed unquestioningly. But being able 
to make a decision when to follow orders and when to question them is 
much finer skill that I would like to see inculcated, so the argument 
is moot.

But let me get back to the issue of modeling. When we create/follow 
physical theories, we don't normally include unreasonable 
expectations--when working on such problems, we always have to list 
the conditions we are concerned with and ignore the rest. Sometimes, 
this backfires--the Tacoma Narrows Bridge is one of the most famous 
examples of this. There is also the Butterfly Economics theory--a 
butterfly flapping wings in Brazil will somehow affect the stock 
market in NY (you might have heard the same expressed regarding 
weather patterns). But what is important in this theory is that 
although the connection may well exist, it is IMPOSSIBLE to trace it, 
so, for the sake of modeling, it is not useful.

> I think I'd like to see how you would transform my proposed items from
> mediocre to acceptable or good or excellent.

I don't know if I would use these particular items or if I would be 
tempted to ever turn them into something excellent. It depends on the 
particular array of skills I want to see students demonstrate (note 
that I am not saying here what purpose this demonstration 
serves--that's another discussion). But, if I were to ask a question 
concerning train schedules, I would offer a few more details. In 
constructing this particular item, it is important to remember that 
there is more than simply the perfection of the conditions that is at 
stake--the item has to be readable, readable in a limited amount of 
time (not "timed", but not unlimited either), doable within a limited 
amount of time using skilled students are likely to possess. That 
means that I do not want to make the text particularly long, I want 
to keep the language simple and, yes, I do have to leave some 
conditions unstated because this is how the real world works, even if 
the situation is contrived.

For the train (or bus or plane) item, let me try something like this.

You (or "your parents", or "Billie Joe Bob" or "Bubba" or whoever) 
are planning a trip from New York to Trenton. The train for Trenton 
is scheduled to leave New York at 8:30 A.M. and, according to the 
schedule, the trip takes two and a half hours. If you want your 
friend to meet you at the station, what time should you tell him to 
expect you there?

You'd be amazed how doubt will disappear if the conditions are 
properly set. But, as is the case with the items, you should not 
expect anything perfect--there might always be a student with doubts 
or, quite simply, a bonehead who is clueless about what is being 
asked. We may believe that all students are CAPABLE of producing 
great work, but it does not mean that we should EXPECT all students 
submitting great work. When teaching, I give every opportunity for 
this capacity to manifest itself, but if someone refuses to take 
advantage of these opportunities or blatantly refuses to learn, I 
will fail him just the same--and I am not about to wear sack cloth 
and put ashes on my head over him.

My expectations on this particular item would be a single, specific 
time (11 A.M.), but I would accept other REASONABLE answers with 
partial or full credit, depending on the answer and the circumstances.

> But suppose, in the case of my train item, I
> actually state that the train actually arrived in Trenton on schedule?

As you can see, my rephrasing eliminates even this condition as necessary.

> This removes cases of train derailment, comets destroying Earth, 
> train delays, etc.  While I agree that we will never be able to 
> address all of the possible
> conditions, test developers CAN do a better job of addressing the more likely
> unsaid conditions that would distract students from responding to the item.

Of course, they can! I was not saying otherwise. But don't forget 
that test usually reflect what is being taught. If you teach your 
students to always question the conditions of the problem, you'd be 
doing them a disservice.

But, even if the train arrives on schedule, Maria may not be on 
it--not alive, at least. She might have been mugged or had a heart 
attack on the train. She might have had the urge to jump off, or, 
changed her mind, got off somewhere in the middle and took a train 
back to New York. Are you suggesting that test developers should also 
include wording that Maria is alive and on the train when it arrives 
in Trenton?

Someone has to make a determination of what are reasonable and 
unreasonable conditions to be included in the problem and the 
students are not the ones, for the most part, who will be making 
those decisions. I expect most of these decision to be made FOR THEM 
in advance and I expect students to know this. If they don't, then 
the test will reveal this. A better written test will differentiate 
between refusing to follow the rules and being unable to do so.

>  > Had this been an item I was discussing with students in class, the
> >  first two concerns would require further investigation--that is,
> >  students should be able to follow the line of thought they presented
> >  to discover that neither has any impact on the outcome
>
> The above would be not be possible via the multiple-choice item type.

Nonsense! I have no love for multiple-choice items, but what the 
student does is up to him, not the item format. The students can 
still follow the train of thought even if, in the end, all he has to 
do is bubble in an oval.

> This
> entire item would be much better and appropriate as a constructed response
> item or discovery project, wherein such issues can be further explored and
> discovered by the student.

EVERY item would be much better off as a constructed response item or 
a discovery project. But another life skill that students should 
learn from us is that it is not always up to us to make that decision.

>  > I have no sympathy for these students--they did not answer the
> >  question in a reasonable manner. Period. Their explanations have
> >  absolutely nothing to do with the item and appear to be post
> >  hoc--that is, this is not something they thought of during the test.
>  > It is also not something that is likely to EVER lead them to an
> >  answer. You were not looking at a philosophy class--the test was in
> >  math, was it not? So why all this digression? If you want these
> >  students to approach the test item philosophically, find another
> >  venue to do so--don't do it in a math class.
>
> VS.....by any chance are you a test developer?  If you teach, please tell me
> which level (high school, college, etc.) and the course (math, testing and
> measurement, or other)?  Just curious ....

No chance at all--I am not a test developer, not even an item writer 
(although if someone asks me and pays me to do it, I will certainly 
consider it). I generally teach college courses, although the "level" 
is another matter--I often teach college courses that simply rehash 
high-school math, and even have high-school students in my classes. I 
do not teach anything concerning testing and measurement.

I have, however, been involved in some research on assessment 
development. Check out the Balanced Assessment Project in Mathematics 
(there are several Dale Seymore volumes and BAMP/MARS websites that 
distribute the tasks).

	VS-)

I was asked earlier for examples of bad questions. Here are three 
from the most recent (December) MCAS retest--the one that determined 
that 10% of high-school students in Massachusetts public schools are 
not fit to graduate. Each is bad for different reasons.

MCAS ITEM 1(16):
TABLE: Admission Costs: Basketball Hall of Fame
General Admission		$9.00 each
Groups of 15 or more		$6.00 each

Two groups of high school students visited the Basketball Hall of 
Fame in Springfield, Massachusetts. The two groups did not have the 
same number of students, but each group paid a total of $108 for 
their tickets. How many students were in EACH group?

Correct Answer: 12, 17, and 18

VS: This is bad not so much because of the question itself, although 
it is rather contrived (does anyone really care if two groups paid 
the same amount?). The answer is just bad, especially as listed. 12 
and 18 are easy to see--12 tickets cost 12 * 9 = $108 and 18 cost 18 
* 6 = $108. But 17? Apparently, the logic was 17 is two over 15, so 
15 * 6 + 2 * 9 = $108. However, this contradicts the conditions of 
the problem! All it says is that groups of 15 or more are priced at 
$6 per person, so a group of 17 should cost 17 * 6 = $102, not $108! 
But even had 17 been acceptable, the "correct answers" are "12 and 
17", "12 and 18" or "17 and 18", not "12, 17, and 18". I suppose, 
that someone at the DOE was following the line of reasoning in this 
thread and decided to give the students who saw 17 as a part of the 
answer the benefit of a doubt. They were wrong, as were the students.

MCAS ITEM 2(17):
On a high school football team, the 7 starting players on the 
offensive line weigh 192, 217, 235, 224, 218, 210, and 195 pounds. 
The 4 starting players in the offensive backfield weigh 180, 169, 
182, and 177 pounds.
a. What is the mean weight of the offensive linemen? Round you answer 
to the nearest whole pound. Show all your work.
b. What is the mean weight of the offensive backfield? Round your 
answer to the nearest whole pound. Show your work.
c. What is the mean weight of the entire team? Round your answer to 
the nearest whole pound. Show or explain how you got your answer.
d. Is the mean weight of the entire team closer to the mean weight of 
the offensive linemen than to the mean weight of the offensive 
backfield? Explain your reasoning.

VS: Let me rephrase the first part of the item.

"On a blog team, 7 starting clogs weigh 192, 217, 235, 224, 218, 210, 
and 195 pounds. The 4 starting dlogs weigh 180, 169, 182, and 177 
pounds."

The rest of the text would be changed appropriately. Can this 
question be answered in the same manner as the one on the MCAS? If 
not, the question is inappropriate--it is culturally biased. There is 
no reason why this question should require the knowledge of even a 
single football formation. Furthermore, the initial text talks about 
the "starting offense", while the parts (c) and (d) talk about the 
"entire team". Bad question.

MCAS ITEM 3(18):
What are the solutions to the equation below?
2x^2 - 11x - 21 = 0
A. 7, -1.5
B. 4.5, 1.0
C. 1, -21
D. 7, -3

VS: At first pass, there seems to be little wrong with this. But, if 
the goal of the question is to assess student ability to solve a 
quadratic equation, the question fails miserably. There is absolutely 
no reason to solve the equation. There are two possible approaches, 
one is more insightful, the other less so. The more ingenious student 
can look at the equation and notice that the product of the two 
"solutions" (actually inappropriate language, but common enough to 
let pass) is -21/2. The only pair that achieves this is A. On the 
other hand, if all a student knows is that a number is a solution if 
it can be substituted into the equation to produce a true statement, 
one only needs to test a single number from each pair to determine if 
the pair qualifies. The choices can be further reduced by noticing 
that 7 and 1 are the only numbers that appears twice. So, testing 7 
and 1 will reduce the number of possible choices in half, then 
testing 3 will eliminate D from consideration and leave only A as the 
answer. Alternately, an even weaker student might test the number in 
order. Having discovered that A works, he would--and should--stop.

Again, the item is not bad because it leads students astray--it is 
not likely to do so. It is bad because it does not address any stated 
goal and really does not test much that is meaningful. Even if the 
question is answered correctly we have absolutely no idea what the 
student knows.

Note that I listed here one item of each time--short response, open 
response and multiple choice, so the format does not determine the 
"badness" of the item.