Hoxby Charter Study Turns Hoaxby

[James Horn <ontogenyx@gmail.com>]

I was at Substance News this morning enjoying the work of the Schmidt Team
<http://www.substancenews.net/>when I came across some important news I had
missed. No doubt I would still be missing it if it were not for Substance
News, for the research review by EPIC and EPRU announced below remains
ignored by corporate editorial boards everywhere.

Jennifer Medina did have a brief, buried blog post in the NYTimes Saturday
regional news<http://cityroom.blogs.nytimes.com/2009/11/12/counterpoint-to-study-praising-charter-schools/>,
but you can be sure that the deflation of the gaseous Hoxby hoax will not be
reported as news anywhere--it simply does not conform to the corporate
education reform narrative, particularly since it is, yet, another loud
smackdown to the anti-scientific approach to education policy being taken by
Team Obama/Gates/Broad.

When the history is written on this sad episode of American educational
policy, Arne Duncan will, no doubt, make Margaret Spellings seem, well,
99.44% pure.

The Press Release from EPIC
<http://epicpolicy.org/newsletter/2009/11/headline-grabbing-charter-school-study-doesn%C3%A2%C2%80%C2%99t-hold-scrutiny>at
the University of Colorado:

Headline-Grabbing Charter School Study Doesn’t Hold Up To Scrutiny
Reviewer finds serious statistical flaws in research on NYC charter schools

*Contact: *Sean Reardon, (650) 736-8517 (office); (617) 251-4782 (cell);
sean.reardon@stanford.edu
Kevin Welner, (303) 492-8370; kevin.welner@colorado.edu
Gary Miron, (269) 599-7965; gary.miron@wmich.edu

BOULDER, Colo. and TEMPE, Ariz. (November 12, 2009) -- A recent report on
New York City charter schools found achievement results at the charters to
be better than comparison traditional schools. But that report relies on a
flawed statistical analysis, according to a new
review<http://epicpolicy.org/thinktank/review-How-New-York-City-Charter>.


The report is *How New York City's Charter Schools Affect Achievement* and
was written by Caroline Hoxby, Sonali Murarka, and Jenny Kang. When it was
released in late September, it was enthusiastically and uncritically
embraced by charter advocates as well as media outlets. The *Washington Post
* offered an editorial titled, "Charter Success. Poor children learn.
Teachers unions are not
pleased<http://www.washingtonpost.com/wp-dyn/content/article/2009/09/26/AR2009092602002.html>."
The editorial's first paragraph reads:

"Opponents of charter schools are going to have to come up with a new
excuse: They can't claim any longer that these non-traditional public
schools don't succeed. A rigorous new study of charter schools in New York
City demolishes the argument that charter schools outperform traditional
public schools only because they get the 'best students.' This evidence
should spur states to change policies that inhibit charter-school growth. It
also should cause traditional schools to emulate practices that produce
these remarkable results."

The editorial argues throughout that the study provides unquestionable
evidence that charters result in improved student achievement. It ends, "Now
the facts are in."

The *New York Daily
News*<http://www.nydailynews.com/opinions/2009/09/23/2009-09-23_acing_the_test.html>was
no less effusive: "It's official. From this day forward, those who
battle New York's charter school movement stand conclusively on notice that
they are fighting to block thousands of children from getting superior
educations."

Because of the declared importance of the new report, we asked Professor
Sean Reardon to carefully examine the report's strengths and weaknesses for
the Think Tank Review Project and write a review that would help others use
the study in a sensible way. Reardon, like the report's lead author Hoxby,
is a professor at Stanford University. He is an expert on research
methodology.

The Hoxby report estimates the effects on student achievement of attending a
New York City charter school rather than a traditional public school. A key
finding, repeated in press reports throughout the U.S., compares the
cumulative effect of attending a New York City charter school for nine years
(from kindergarten through eighth grade) to the magnitude of average test
score differences between students in Harlem and the wealthy New York
community of Scarsdale. The report estimates this cumulative effect at
roughly 66% of the "Scarsdale-Harlem gap" in English and roughly 86% of the
gap in math.

In his review<http://epicpolicy.org/thinktank/review-How-New-York-City-Charter>,
Reardon observes that the report "has the potential to add usefully to the
growing body of evidence regarding the effectiveness of charter schools."
New York charter schools' use of randomized lotteries to admit students to
charter schools offers the possibility that the study of those schools can
roughly approximate laboratory conditions.

But Reardon points out that the report's key findings are grounded in an
unsound analysis -- an inappropriate set of statistical models -- and that
the report's authors never provide crucial information that would allow
readers to more thoroughly evaluate "its methods, results, or
generalizability."

Reardon's review notes these shortcomings in the report:

   - In measuring the effects of charter schooling on students in grades 4
   through 12, the study relies on statistical models that include test scores
   from the previous year, measured after the admission lotteries take place.
   Yet because of that timing, those scores could be affected by whether
   students attend a charter school. As a consequence, the statistical models
   "destroy the benefits of the randomization" that is a strength of the
   study's design. (The use of a different model makes the results for students
   in grades K-3 more credible, he notes.)


   - The report's claims regarding the cumulative effects of attending a New
   York City charter school from kindergarten through eighth grade are based on
   an inappropriate extrapolation.


   - It uses a weaker criterion for statistical significance than is
   conventionally used in social science research (0.05), referring to p-values
   of roughly 0.15 as "marginally statistically significant".


   - The report describes the variation in charter school effects across
   schools in a way that may distort the true distribution of effects by
   omitting many ineffective charter schools from the distribution.

Reardon explains that, as a result of the flaws in the report's statistical
analysis, the report "likely overstates the effects of New York City charter
schools on students' cumulative achievement, though it is not possible --
given the information missing from the report -- to precisely quantify the
extent of overestimation." This, as well as the lack of detailed information
in the report to assess the extent of that bias, make it impossible for
readers to know whether the report's estimated charter school effects are in
fact valid.

"Policymakers, educators, and parents should therefore not rely on these
estimates until the bias issues have been fully investigated and the
analysis has undergone rigorous peer review."

According to Professor Kevin Welner, director of the University of Colorado
at Boulder's Education and the Public Interest Center (EPIC): "Readers of
this review will understand that, while Hoxby's charter school study is a
contribution, it has significant flaws and limitations. Unfortunately, the
editorial reaction of otherwise-respectable media outlets trumpeted the New
York City findings as the final and faultless word on charter school
performance. In fact, the study used inappropriate methods that overstate
the performance of the charter schools it studied."

Welner notes that the Think Tank Review
Project<http://epicpolicy.org/thinktank/review-multiple-choice>also
recently
reviewed another charter school study, released in June by Stanford's CREDO
policy center <http://epicpolicy.org/thinktank/review-multiple-choice>. That
study encompassed 65-70% of the nation's charter schools. "Our review
pointed out a number of limitations but also noted the relative strength and
comprehensiveness of the data set, the solid analytic approaches of the
CREDO researchers, and the important fact that the CREDO results were
consistent with a large body of research showing charter schools overall to
be performing no better than (and perhaps worse than) traditional public
schools," Welner says. But he added that "the CREDO and Hoxby reports used
different designs and covered different schools. They are not directly
comparable, nor are we able to say which is 'better.' Neither report is
definitive or without notable weaknesses."

Welner concludes, "the important thing to understand is that if, after an
appropriate reanalysis of the data, we still find that New York City's
charter schools are in fact bucking the national trend, the sensible next
step is for researchers to explore the causes rather than to jump to broad
conclusions that fly in the face of the overall research base. It would be
irresponsible to use the NYC results -- even if they were valid and reliable
-- to drive policy in places throughout the U.S. where charters are
apparently underperforming their competition."

Find Sean Reardon's review on the web at:
*http://epicpolicy.org/thinktank/review*-*How-New-York-City-Charter*<http://epicpolicy.org/thinktank/review-How-New-York-City-Charter>

Find the NYC report by Hoxby and her colleagues at:
*http://www.nber.org/~schools/charterschoolseval/*<http://www.nber.org/%7Eschools/charterschoolseval/>

*CONTACT:*
Sean F. Reardon
Associate Professor of Education and (by courtesy) Sociology
Stanford University
(650) 736-8517 (office); (617) 251-4782 (cell)
sean.reardon@stanford.edu

Kevin Welner, Professor and Director
Education and the Public Interest Center
University of Colorado at Boulder
(303) 492-8370
kevin.welner@colorado.edu

Gary Miron, Professor of Education
Western Michigan University
(269) 599-7965
gary.miron@wmich.edu

*About the Think Tank Review Project*

The Think Tank Review Project (http://thinktankreview.org), a collaborative
project of the University of Colorado at Boulder Education and the Public
Interest Center (EPIC) and the ASU Education Policy Research Unit (EPRU),
provides the public, policy makers, and the press with timely, academically
sound reviews of selected think tank publications. The project is made
possible by funding from the Great Lakes Center for Education Research and
Practice.

EPIC and EPRU collaborate to produce policy briefs in addition to think tank
reviews. Our goal is to promote well-informed democratic deliberation about
education policy by providing academic as well as non-academic audiences
with useful information and high quality analyses.

Visit EPIC and EPRU at http://www.educationanalysis.org/

EPIC and EPRU are members of the Education Policy Alliance
(http://educationpolicyalliance.org).



--
Posted By Jim Horn to Schools
Matter<http://www.schoolsmatter.info/2009/11/hoxby-charter-study-turns-hoaxby.html>at
11/14/2009 09:07:00 AM