Re: Scriven on randomized control groups

["Sandra M Mathison" <smathison@louisville.edu>]

Just for context, Michael's response is one of many being sent by
evaluators to the request for feedback on the US Department of Education
priorities for funding evaluation. I have given the federal register
link below where you can find the memo from Rod Paige. In addition,
there is an American Evaluation Association Task Force preparing a
response as well--I am on that Task Force, so I will post that statement
when it is complete. Responses need to be to the feds by December 4th. 

http://www.ed.gov/legislation/FedRegister/proprule/2003-4/110403b.html

Sandra
Sandra Mathison
Professor
College of Education & Human Development
University of Louisville
Louisville KY 40292
502 852 0616

To write a text is to release its meanings.
                                         David Hamilton
>>> monty@fairtest.org 11/21/03 10:57 AM >>>
I received this from someone actually seeking my assistance in
addressing the idiocy of reducing scientific investigation to randomized
control groups, as the flat earth "scientists" in the Bush
administration seek to do. I circulate it for anyone else having at the
state or local level to confront this issue and who is not a researcher
familiar with details or research methodologies. 



Michael Scriven on randomized control groups vs other methods

Here are some comments from Michael Scriven, an evaluator who is an
ex-President of the American Evaluation Association and also an
ex-President of the American Educational Research Association. 

 

I state and then prove two key propositions.

1. First, as demonstrated below, randomized control group trials (RCTs),
even when possible, are NOT always superior to other approaches
(non-RCTs) in demonstrating causality.



2. Second, there are many issues of great importance in educational as
well as in social or health research where it is ethically and/or
practically IMPOSSIBLE TO USE RCTs, but nevertheless essential to the
welfare of children and adults that we distinguish between poorly
supported and well supported interventions. Hence, even if the
alternative designs were less reliable for establishing causation, which
is not true, for work in these areas they are superior because ethical
and/or feasible. Since the best alternatives are not inferior in
evidential value, they are doubly valuable in such cases, and it would
be disastrous to cancel or even penalize research or implementation of
such approaches.



3. On the matter of proof of the above propositions, we can begin with
direct refutation by a counter-example. The Federal government pours
millions of dollars every year into the worthy task of saving millions
of life-years of its citizens by opposing the adoption and continuation
of smoking cigarettes. It does so with very solid justification; no-one
today questions the view that smoking causes lung cancer and other
ailments. But there was NEVER a single RCT, let alone replications or
multiple RCT studies, that established that smoking causes lung cancer
(or heart disease, etc.). It wa established by non-RCT studies, FAR
BEYOND REASONABLE DOUBT. One reason for not using RCT design was simple;
one could not ethically ask the non-smokers in a randomly chosen
experimental or control group to take up smoking for a period of years
when there was a serious risk that it would cause illness or death. The
same reasoning applies to asking children to take up the use of drugs,
violent behavior, or the avoidance of reading (or even after-school)
instruction. It would be vicious in the extreme, as well as completely
illogical, for the federal government, allegedly in the interest of
children, to cut off funding of valid studies of such crucial matters on
the grounds that RCTs are not being done there. However, there is no
loss of certainty about the conclusions.



4.   More general points that one should understand in order to grasp
the

magnitude of the confusion in the proposed regulations:

4A. Causation is a unique concept, one of the first that young children
acquire, in that it is based not on mere cognition, nor on affective
experience, but on psychomotor activities. The child acquires control
over its envirnoment as it learns how to throw food on the floor, shake
rattles, and push toys around on the floor. It is learning how to cause
things to happen, and it is entirely correct in its belief that it can
do this. Thus it grows up fully aware of how to cause (the concept of
agency), then how to recognize causation when others are the agents
producting similar effects, and finally how to recognize causation when
naturally occurring events bring about other events. Causation is thus
en integral part of the OBSERVATIONAL vocabulary of all humans with any
control over their limbs (and in fact of many who lack this). In
adulthood, this means that multiple observers can attest to the cause of
death in traffic and other accidents, and--after due attention to the
well-understood causes of error in such judgments--a jury and a court
can often certify causation beyond reasonable doubt. 

4B. In well-conducted case studies, which are essentially
quality-controlled observational studies with corroboration, the same is
true; hence these are another of the important examples of valid non-RCT
investigations. Our knowledge that twelve-step programs do indeed bring
about termination of addiction in some cases, is an example of
life-saving non-RCT research into causation. 

4C. It is unfortunate that the use of the term 'quasi-experimental'
reinforces the naive view that such studies are a kind of poor relation
of RCT studies. In fact, they are often AT LEAST AS VALID AND RELIABLE
as RCTs. If we connect up a ciruit at randomly chosen times, do this a
hundred times, and find that whenever we do it, but on no other
occasions, a light flashes, we have demonstrated beyond reasonable doubt
that the connection causes the flash (in this context). If, on twelve
occasioins, we switch a single class from lectures to the new model of
highly interactive learning, running pre and post tests on both such
occasions, and we find that the learning gains in the interactive mode
are always very close to triple those in the lecture mode, albeit on
different content, and such changes do not occur spontaneously, we have
proven beyond reasonable doubt, for this class at least, that the
interactive approach represents an improvement of great educational
importance, despite the absence of a matched control group. Repeating
the study in twenty contexts, as has been done, shows the conclusion is
robust, and on the way to generality, just as in the case of an RCT. In
fact, for reasons extensively discussed in the literature, this method
arguably provides superior validity to the use of an RCT. 

4D. There are a number of other designs that are just as good as the
interrupted time series design of 4C, of which we only mention one more
here. This is the design used by the forensic pathologist who discovers
and proves the cause of death by working from a master list of possible
causes, plus clues that shorten the list of possible causes in the
present case, followed by the elimination of all but one by exhaustive
search for the key 'marker-indicators' (modus operandi clues) associated
with the presence or absence of each locally possible cause. It is a
method that transfers easily to both individual and group studies of
humans, old and young, learning or suffering. Thinking of the case of
someone shot twice in the forehead with a .357, do we really want to
argue that this approach is always less valid or reliable or in any way
scientifically inferior to the RCT? The simple fact is that RCT design
is one important and clever way amongst many that demonstrate causation,
and in some complex cases it is the best choice; no more and no less so
than the others are in other cases.



5. In short, the suggestion that there is one superior method for
establishing causation in scientific evaluation is based on an extremely
superficial and technically hopeless analysis of the concept of
causation and of existing methodologies. I hope that you will not
support the further starvation and prevention of good research that can
improve the lives and learning of children and adults based on this
naive view. There is an alternative approach for you to adopt that will
do more good, AND less harm, which is to insist on the highest standards
of quality in all evaluation and in particular the element in evaluation
that involves the proof of causation.



Yours sincerely,



Michael Scriven

Professor of Evaluation

Auckland University, New Zealand

            ---and--

Professor of Psychology

Claremont Graduate University

Claremont, California



 

Monty Neill, Ed.D.
Executive Director
FairTest
342 Broadway
Cambridge, MA 02139
617-864-4810    fax 617-497-2224
monty@fairtest.org
http://www.fairtest.org