Fwd: bad item

[DMDesiderata@aol.com]

Forwarding my response since it appears it never was sent to the list and 
only was sent to Victor.


"Miracles happen to those who believe."

--- Begin Message ---

• To: Victor.Steinbok@verizon.net
• Subject: Re: [arn-l] bad item
• From: DMDesiderata@aol.com
• Date: Tue, 6 May 2003 14:25:52 EDT
• Full-name: DMDesiderata

In a message dated 5/6/03 11:53:42 AM Eastern Daylight Time,
Victor.Steinbok@verizon.net writes:

>
>
> For the train (or bus or plane) item, let me try something like this.
>
> You (or "your parents", or "Billie Joe Bob" or "Bubba" or whoever)
> are planning a trip from New York to Trenton. The train for Trenton
> is scheduled to leave New York at 8:30 A.M. and, according to the
> schedule, the trip takes two and a half hours. If you want your
> friend to meet you at the station, what time should you tell him to
> expect you there?

Gosh, this "revised" items bothers me even more. I initially had
"understood" this item to be asking "what time should I tell my friend to
meet me at the train station?" Obviously, this can result in various answers
that are close to the train's scheduled arrival time. But for a variety of
reasons, to get a parking space, avoid traffic, etc. it might be advisable
to get there 30 minuets to an hour earlier than the train's scheduled
arrival. The use of "expected" certainly does remove those conditions, but
I'd suggest that in a timed testing situation, there might be some issues
here, on a first, and fairly quick, reading. Furthermore, if I know that it
is snowing or if I know that this train is usually late, I would expect my
friend to arrive later by as much as 30 minutes or even an hour. I realize
that you would rule out such responses as my adding irrelevant information,
but this, at least, is my experiences in terms of flying....most of the time
my plane arrives late....so much so that I have now told those that plan to
pick me up to call the airline in advance.

>
>
> My expectations on this particular item would be a single, specific
> time (11 A.M.), but I would accept other REASONABLE answers with
> partial or full credit, depending on the answer and the circumstances.
>

If this were a multiple-choice item, with responses, such as 10AM, 10:30AM,
11AM, 11:30AM, and 12noon, ... (the original item I was recalling was a
multiple-choice item), students would not be able to suggest such REASONABLE
answers. In fact, the options could even be, 10:30 AM, 10:45AM, 11AM,
11:15AM, 11:30AM. Clearly those choices are even worse, unless the INTENT
is for students to pick 11AM and play the school game. But this is not what
I do in real life.

Heck, we can eve do this in reverse. We could have an item that states that
your train (plane, bus, whatever) is scheduled to leave at some time. Give
some sort of condition like it takes so much time to travel to the train
station, airport, etc. and ask what time should you leave your house to
catch the train (plane, bus, whatever). Same problems.....traffic, security
checkpoints, leaving enough time for unexpected delays, etc. I'm one who
generally gets to airports 2 to 3 hours prior to departure.



>
> Of course, they can! I was not saying otherwise. But don't forget
> that test usually reflect what is being taught. If you teach your
> students to always question the conditions of the problem, you'd be
> doing them a disservice.
>

Agreed.

> But, even if the train arrives on schedule, Maria may not be on
> it--not alive, at least. She might have been mugged or had a heart
> attack on the train. She might have had the urge to jump off, or,
> changed her mind, got off somewhere in the middle and took a train
> back to New York. Are you suggesting that test developers should also
> include wording that Maria is alive and on the train when it arrives
> in Trenton?
>

I agree that this can get quite ridiculous. I think that anyone can find
something wrong with any item if they wanted to find fault with an item,
Some of the faults will be quite lame, ridciulous, ludicrous, etc. One area
where it is fairly easy to see how this can happen is in the area of
sensitivity reviews. Depending upon your standards (or those of the test
publisher, etc.) I, at least, could make a case for eliminating most word
problems. As I understand it, most test publishers, have something like a
taboo list containing topics to avoid in item writing. I believe this has
really gotten out of hand. These taboo lists can be as many as 3 pages in
length and make it nearly impossible to write any type of meaningful item.

> Someone has to make a determination of what are reasonable and
> unreasonable conditions to be included in the problem and the
> students are not the ones, for the most part, who will be making
> those decisions. I expect most of these decision to be made FOR THEM
> in advance and I expect students to know this. If they don't, then
> the test will reveal this. A better written test will differentiate
> between refusing to follow the rules and being unable to do so.
>

Agreed. But who gets to decide this? Test publishers?

>
>
>
> EVERY item would be much better off as a constructed response item or
> a discovery project.

Agreed.



>
> No chance at all--I am not a test developer, not even an item writer
> (although if someone asks me and pays me to do it, I will certainly
> consider it).

You would likely be far better as someone reviewing items as opposed to
writing them. I think you would make a great item writer, but you are likely
more needed as an item reviewer or editor.

I generally teach college courses, although the "level"
>
> is another matter--I often teach college courses that simply rehash
> high-school math, and even have high-school students in my classes. I
> do not teach anything concerning testing and measurement.
>

In addition to the quality of items, which is clearly important, I am
concerned about the grading of constructed response/open-ended items, not
just in math, but in writing as well. Does anyone know anyone that has been
involved in reading essays for a state-wide test? I spoke to a few a year or
so ago and was shocked at what I had been told. I've been looking into this
more recently and it has not gotten much better, I'm not referring to
teachers grading their own tests, just the statewide testing programs which
hire contractors who hire ANYONE to grade these tests.

> I have, however, been involved in some research on assessment
> development. Check out the Balanced Assessment Project in Mathematics
> (there are several Dale Seymore volumes and BAMP/MARS websites that
> distribute the tasks).
>

Believe it or not, I am somewhat familiar with this assessment. As I
understood it, I thought CTB had gotten an exclusive contract from the
developers of this assessment to market, promote this assessment. Perhaps, I
am mistake, but I recall someone telling me that CTB was selling it in their
testing catalogue. What was/is your role in the development of this
assessment?

> VS-)




I will look at your proposed "bad" items later. They do look interesting>
>

"Miracles happen to those who believe."


--- End Message ---