Scientifically Based Research

[George Sheridan <learn@JPS.NET>]

Nancy Creech wrote: You might enjoy doing a little discourse analysis at 
this site.
Click here: Scientifically Based Research -- U.S. Department of Education
http://www.ed.gov/offices/OESE/esea/research/index.html

Susan Neuman, Assistant Secretary for Elementary and Secondary Education 
hosted this gathering. She began her welcoming statement with these words:

     "One of our goals today?we have a very practical goal actually. We're 
no longer debating whether scientifically based research and scientifically 
based evidence is important, we know it now is important and we know it is 
critical. As many of you know, we have counted one hundred and eleven times 
that the phrase "scientifically based research" is in our new law."

Two things struck me on first reading:

First: The presentation by Valerie Reyna, Deputy for the Office of 
Educational Research and Improvement was at a very elementary level. Who in 
that room needed this sort of basic introduction to science?

Second, the scientists they called on to define scientifically-based 
research defined it much less narrowly than Reid Lyon, et.al. have done. 
And they stressed that something is "scientific" not because the government 
says it is, but because it withstands critical scrutiny.

But the scientists were only window-dressing. The core purpose of the 
program was revealed in the afternoon sessions. Look at the presentation on 
reading < http://www.ed.gov/offices/OESE/esea/research/greer.html >--you 
won't be surprised to see phonemic awareness and phonics listed as number 
one and number two of the five elements research has proven essential to 
reading instruction. In fact, the presenter (Dr. Eunice Greer, identified 
as a reading consultant from Illinois) stated, "Most of my comments today 
are drawn from the National Reading Panel Report that was delivered late in 
the year 2000." Whenever someone mentions the National Reading Panel, 
remember that the National Research Council of the National Academy of 
Sciences did a report on Preventing Reading Difficulties in Young Children, 
which did not endorse Phonics-uber-alles. While it is possible to disagree 
with specific emphases in the NRC report, that report was explicitly 
designed to reflect the scientific consensus on reading instruction.Reid 
Lyon put together his panel because he didn't like what science is really 
telling us.

Still, it may be possible to draw on what these scientists said to oppose 
the highly restrictive definitions of "scientific" likely to be embodied in 
proposed U.S. Department. of Education regulations.

Speaking on the basic principles of scientifically based research, Lisa 
Towne of the Center for Education of the National Research Council said:

        "Replicating" is a very core notion in science. It has to do with 
the fact that since in any particular study you're only relying on a 
limited set of observations, to what extent does what you're looking at 
here and now generalize to other times, places and contexts. In education, 
as you know, this is a critical question. Teachers and researchers alike 
have been knowing for years that something that works in a particular 
classroom may not work in the classroom next door and may not work in the 
same classroom a year later. So attention to sort of what's going on in the 
classroom at that time can help you understand the conditions under which 
things tend to work and therefore how to think about how findings can 
generalize from one time to another.

        "I'll go on to the last principle here, which has to do with the 
transparency of the scientific enterprise. Valerie alluded to this as well. 
This just has to do with the role of the scientific community actually 
working together to try and make sense of all of the findings and all of 
the conclusions that come from individual studies. Educators often bemoan 
what there perceive as bickering among the research community and we'll 
grant you that there is some bickering. But there is actually something 
important to say about that and that is that researchers are actually 
trained and employed and paid money to be skeptical observers and to ask 
critical questions. That's their job. So, this critical kind of work, 
critiquing other peoples findings and trying to make sense of them is 
actually an indication of the health of the scientific enterprise, not its 
failure."

        <SNIP> "at one level there is a difference between the so-called 
hard and soft sciences. And, that has to do with differences that emanate 
from studying inanimate objects and studying people, which are complex and 
do crazy things that we often can't understand or predict very well.

        "So, there are some things that are different. Broadly, research 
or control is one of them. Think of it this way, a petri dish of heart 
cells is a heck of a lot better behaved than a classroom of third graders. 
Anyone whose tried to study education research and has done cell biology, 
as one of my committee members did, can attest to this.

        "There's other things that are different. I'll just touch on this 
last one on the slide which has to do with certainty. Valerie said, and the 
committee completely agrees, that science is by definition an uncertain 
enterprise. The key is understanding the degree of uncertainty that is 
associated with what we know. In general terms, in the physical sciences we 
because of this ability to control the environment tend to have more 
certainty associated with them than sciences that have to do with people, 
like education research."

Stephen Raudenbush of the University of Michigan said, "A scientist is 
expected to search for disconfirming evidence, and that's a crucial feature."

<SNIP>

        He continued, "The role of the scientific community is key. It's a 
healthy scientific community who can?and this relates to democracy, being 
able to freely evaluate alternative points of view, not feel that there's 
going to be some censorship.

        "The people who are committed to the principles I just mentioned 
who evaluate this, the process of objectivity really involves this group of 
people engaging in this ongoing debate. Scientists, as was mentioned, are 
trained to be skeptical and that process can really work. What's really in 
the final analysis scientific is what the community of scientists says is 
scientific."

<SNIP>

        Raudenbush went on to say, " Does qualitative research play a 
role? I would say, yes, without doubt. Because we need to not just test the 
impact of things out in the field, we need to do a lot more of that. We 
haven't done enough. But we have to have good things to take into the 
field. We have to have good ideas about how to teach math, how to teach 
reading. Those ideas come from up close, careful study of expert 
practitioners in real settings and how kids learn. So, we need that up 
close kind of research but see we've got to do a better job of connecting 
that research with field trials of what works, and that's what's really 
been missing."

<SNIP>

        "My tenth and final question is: Is there any danger here that we 
are going to be overselling the role of science in education? I think there is.

        "I've got a quote here from E. L. Thorndike who wrote the lead 
article in the founding edition of the Journal of Educational Psychology in 
1910. I won't read the entire quote except to say that Thorndike felt that 
a scientific psychology was about to produce decisive evidence on virtually 
every practical question that arises in education. We know in retrospect 
that he was wrong. Unfortunately, by overselling what science can do, it 
led to a crisis of, you might say, rising expectations that couldn't be 
met. For a long time thereafter science in education fell into disarray.

        "The same thing happened in the '60s with scientific problem 
solving, the idea that we would have kind of a social engineering model. 
We'd try programs, we'd evaluate them, we'd get feedback, the programs 
would get better and the great society was going to be born out of this 
sort of scientific and engineering model. That was an overselling. We 
couldn't really pull that off.

        "So, let's make sure that we have a balanced view this time. I am 
so excited that we have an opportunity to do it, to do it right without 
overselling it this time."