Schoenfeld's Criticism of The What Works Clearinghouse

[Richard Hake <rrhake@earthlink.net>]

Some subscribers may not be aware of Alan Schoenfeld's (2006a,b) 
recent criticism of the U.S. Dept. of Education's "What Works 
Clearinghouse."

The abstract of "What Doesn't Work: The Challenge and Failure of the 
What Works Clearinghouse to Conduct Meaningful Reviews of Studies of 
Mathematics Curricula" [Schoenfeld (2006a)], reads as follows 
[bracketed by lines "SSSSS. . ."; my URL inserts]:

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
An early version of this article, discussing curricular interventions 
in mathematics, was written for the What Works Clearinghouse
(WWC <http://www.whatworks.ed.gov/>). The Institute of Education Sciences
(IES <http://www.ed.gov/about/offices/list/ies/index.html>), which 
funds WWC, instructed WWC not to publish it. An expanded version, 
written at WWC's invitation for a special issue of an independent 
electronic journal and a book to be published by WWC, argued that 
methodological problems rendered some WWC mathematics reports 
potentially misleading and/or uninterpretable. IES instructed WWC 
staff not to publish their chapters - thus canceling the publication 
of the special issue and the book. Those actions, chronicled here, 
raise important issues concerning the role of federal agencies and 
their contracting organizations in suppressing scientific research 
that casts doubt on current or intended federal policy.
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

And in "Reply to Comments From the What Works Clearinghouse on 'What 
Doesn't Work,'" Schoenfeld (2006b) wrote [my insert at ". . .[.....]. 
. ."] :

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Let me be clear about the stakes involved in this case. The issue 
here is the suppression of a report that challenges the scientific 
underpinnings of the current federal policy agenda. The WWC website 
claims that WWC was funded by the government "so that you know what 
the best scientific evidence has to say." The evidence suggests 
otherwise.

In recent years there has been mounting evidence of attempts to 
suppress research on global warming . . .[see e.g., Revkin (2006)]. . 
.  that challenges the rationale for federal policy in that area. At 
stake there, and here, are the integrity of the research process and 
the academic freedom on which it depends. Also at stake is the public 
trust in results vetted by federally sponsored groups that claim to 
represent the best that science has to offer. The academic community 
must strive to preserve the integrity of the research process and its 
contributions to our society. In the words of John Philpot Curran, 
"Eternal vigilance is the price of liberty." 
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS


The U.S. Dept. of Education's evident misunderstand of what 
constitutes good science, has also been criticized by psychologist 
Robert Sternberg (2004), whose eighth reason that the NCLB is failing 
is:

"8. THE ASSUMPTION THAT GOOD SCIENCE SHOULD BE POLITICALLY GUIDED.
The [NCLB act <http://www.ed.gov/policy/elsec/leg/esea02/index.html>] 
specifies that educational practice be guided by good, rigorous 
science. But what is good science? The current administration, to an 
unprecedented degree, has decided to play an active role in deciding 
what it means by "good science." Some of the science thus supported 
may indeed be good science. But science has always proceeded best 
when it is left totally independent of the political process, and 
when competing schools of thought are left to slug it out on the 
scientific battlefield free of political influence or interference."

What the administration means by good science is set forth at the
What Works Clearinghouse Evidence Standards at
<http://www.w-w-c.org/reviewprocess/standards.html>, where it is stated
that:

****************************************
WWC Evidence Standards
<http://www.w-w-c.org/reviewprocess/study_standards_final.pdf> 
identify studies that provide the strongest evidence of effects: 
primarily well conducted randomized controlled trials and regression 
discontinuity studies, and secondarily quasi-experimental studies of 
especially strong design 
<http://www.w-w-c.org/faq/what_research_count.html>.

**"Meets Evidence Standards"--randomized controlled trials (RCTs) 
that do not have problems with randomization, attrition, or 
disruption, and regression discontinuity designs that do not have 
problems with attrition or disruption.

*"Meets Evidence Standards with Reservations"--strong 
quasi-experimental studies that have comparison groups and meet other 
WWC Evidence Standards, as well as randomized trials with 
randomization, attrition, or disruption problems and regression 
discontinuity designs with attrition or disruption problems.

"Does Not Meet Evidence Screens"
<http://www.w-w-c.org/reviewprocess/notmeetscreens.html> -  studies 
that provide insufficient evidence of causal validity or are not 
relevant to the topic being reviewed.
****************************************


Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
<rrhake@earthlink.net>
<http://www.physics.indiana.edu/~hake>
<http://www.physics.indiana.edu/~sdi>

"Since the general or prevailing opinion on any subject is rarely or 
never the whole truth, it is only by the collision of adverse opinion 
that the remainder of the truth has any chance of being supplied."
  John Stuart Mill (1806-1873)


REFERENCES
Herman, R., R. Boruch, R. Powell, S. Fleischman, & R. Maynard. 2006. 
"Overcoming the Challenges: A Response to Alan H. Schoenfeld's "What 
Doesn't Work" Educational Researcher 35(2); online at 
<http://www.aera.net/publications/?id=1148>.

Schoenfeld, A.H. 2006a. "What Doesn't Work: The Challenge and Failure 
of the What Works Clearinghouse to Conduct Meaningful Reviews of 
Studies of Mathematics Curricula," Educational Researcher 35(2); 
online at <http://www.aera.net/publications/?id=1148>. For the WWC 
response see Herman et al. (2006).

Schoenfeld, A.H. 2006b. "Reply to Comments From the What Works 
Clearinghouse on 'What Doesn't Work,'" Educational Researcher 35(2); 
online at <http://www.aera.net/publications/?id=1148>.

Sternberg, R.J. 2004. "Good Intentions, Bad Results: A Dozen Reasons 
Why the No Child Left Behind Act is Failing Our Schools," Education 
Week, 27 October, online at
<http://www.edweek.org/ew/articles/2004/10/27/09sternberg.h24.html>. 
A copy of this article is also at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0411&L=aera-l&T=0&F=&S=&P=964> 
(Scroll to the APPENDIX).

Revkin, A.C.  2006. "Climate Expert Says NASA Tried to Silence Him," 
New York Times, 29 January; online at  <http://tinyurl.com/r5auc>. 
Revkin wrote: "The top climate scientist at NASA says the Bush 
administration has tried to stop him from speaking out since he gave 
a lecture last month calling for prompt reductions in emissions of 
greenhouse gases linked to global warming. The scientist, James E. 
Hansen, longtime director of the agency's Goddard Institute for Space 
Studies, said in an interview that officials at NASA headquarters had 
ordered the public affairs staff to review his coming lectures, 
papers, postings on the Goddard Web site and requests for interviews 
from journalists.  Dr. Hansen said he would ignore the restrictions. 
'They feel their job is to be this censor of information going out to 
the public,' he said."