Fwd: [eddra] More on: NCLB Support by NAEP????

[Susan Harman <susanharman@igc.org>]

Begin forwarded message:

> From: Richard Innes <70224.434@compuserve.com>
> Date: Wed Jan 10, 2007  8:56:36 AM US/Pacific
> To: eddra <eddra@yahoogroups.com>
> Subject: [eddra] More on: NCLB Support by NAEP????
>
> Long Term NAEP Can't Support NCLB at This Time
>
>
> Gerry's criticism of US Secretary of Education Margaret Spelling's 
> attempts
> to use the Long Term Trend NAEP to claim NCLB progress is right on 
> target,
> but there is even more reason to suspect the validity of the 2004 Long 
> Term
> Trend scores for such analysis.
>
> Very simply, the 2004 Long Term Trend NAEP has fallen victim to the 
> same
> sort of exclusion and testing condition changes that have dogged the
> validity of the Main and State NAEP's for nearly a decade.
>
> Here are a few reasons why the 2004 Long Term NAEP results must be
> considered inflated in comparison to previous years. You will find a
> discussion of these issues beginning on page 58 of NCES 2005-464, "NAEP
> 2004 Trends in Academic Progress, Three Decades of Student Performance 
> in
> Reading and Mathematics."
>
>
> 1) Replacing items.
>
> The explanation reads: "Many of the items in the long-term trend 
> assessment
> were written in the late 1960s. Given changes in context over the past 
> four
> decades, several items needed to be restructured or replaced with items
> more in line with current contexts. For example, reading passages that
> discussed outdated issues were replaced with more current passages."
>
> If this is true, the 1999 scores and scores for some earlier years 
> would
> have been depressed. This serious change to the assessment could not 
> help
> but destroy comparability to earlier versions by creating an artificial
> increase in the 2004 results.
>
>
> 2) Moving all background questions to the end of the administration 
> time.
>
> In the past, these questions were intermixed with the academic 
> questions.
> Certainly, that past testing protocol would have been annoying, if not
> terribly distracting. This, again, would have tended to suppress 
> scores on
> the past assessments. Some testing experts would probably view this
> dramatic change in test conditions as another validity threat.
>
>
> 3) Eliminating "I don't know" as a response option for multiple-choice
> items.
>
> This creates a huge reduction in test difficulty. Clearly, comparison 
> to
> any previous tests is badly compromised by this very notable change in
> basic protocol.
>
>
> 4) Eliminating audio paced tapes.
>
> The explanation reads: "Use of an audio tape, which paced students 
> during
> the assessment session so that they were at the same place in the test
> booklet at the same time, was discontinued for mathematics in 2004. 
> That
> is, in the 1999 and previous assessments, students taking the 
> mathematics
> portion would listen to an audio tape that spoke each question aloud,
> paused to allow the student time to respond, then spoke the next 
> question
> aloud."
>
> I think the administration technique for math questions on the prior 
> tests
> must have been very confusing to students. I doubt students ever
> experienced a similar test presentation in their math classes. Now, 
> with
> this change, the Long Term Trend NAEP clearly and completely trashed 
> its
> trend line, as the new presentation methods will be far more familiar 
> to
> students.
>
>
> 5) Using assessment booklets that pertain only to a single subject 
> area.
>
> The explanation reads: "In the past, a single assessment booklet may 
> have
> contained both reading and mathematics items. Science and writing items
> were also intermingled with the reading and mathematics items. In the 
> 2004
> modified assessment, students received a booklet that either contained 
> only
> reading questions or only mathematics questions."
>
> Again, this strikes me as a very serious change that replaced a more
> jumbled up format of question presentation to a highly ordered 
> structure.
> That, too, would increase the most recent scores.
>
>
> Overall, I view these and other changes in the 2004 Long Term Trend 
> NAEP to
> amount to a very serious departure from the true, "long term" nature of
> previous assessments in this series. The changes outlined above would 
> be
> expected to all increase 2004 scores versus earlier test results, 
> thereby
> severing the trend lines.
>
> Richard Innes
>
>
>
> Yahoo! Groups Links
>
> <*> To visit your group on the web, go to:
>     http://groups.yahoo.com/group/eddra/
>
> <*> Your email settings:
>     Individual Email | Traditional
>
> <*> To change settings online go to:
>     http://groups.yahoo.com/group/eddra/join
>     (Yahoo! ID required)
>
> <*> To change settings via email:
>     mailto:eddra-digest@yahoogroups.com
>     mailto:eddra-fullfeatured@yahoogroups.com
>
> <*> To unsubscribe from this group, send an email to:
>     eddra-unsubscribe@yahoogroups.com
>
> <*> Your use of Yahoo! Groups is subject to:
>     http://docs.yahoo.com/info/terms/