Scriven on randomized control groups

["Monty Neill" <monty@fairtest.org>]

I received this from someone actually seeking my assistance in addressing the idiocy of reducing scientific investigation to randomized control groups, as the flat earth "scientists" in the Bush administration seek to do. I circulate it for anyone else having at the state or local level to confront this issue and who is not a researcher familiar with details or research methodologies. 



Michael Scriven on randomized control groups vs other methods

Here are some comments from Michael Scriven, an evaluator who is an ex-President of the American Evaluation Association and also an ex-President of the American Educational Research Association. 

 

I state and then prove two key propositions.

1. First, as demonstrated below, randomized control group trials (RCTs), even when possible, are NOT always superior to other approaches (non-RCTs) in demonstrating causality.



2. Second, there are many issues of great importance in educational as well as in social or health research where it is ethically and/or practically IMPOSSIBLE TO USE RCTs, but nevertheless essential to the welfare of children and adults that we distinguish between poorly supported and well supported interventions. Hence, even if the alternative designs were less reliable for establishing causation, which is not true, for work in these areas they are superior because ethical and/or feasible. Since the best alternatives are not inferior in evidential value, they are doubly valuable in such cases, and it would be disastrous to cancel or even penalize research or implementation of such approaches.



3. On the matter of proof of the above propositions, we can begin with direct refutation by a counter-example. The Federal government pours millions of dollars every year into the worthy task of saving millions of life-years of its citizens by opposing the adoption and continuation of smoking cigarettes. It does so with very solid justification; no-one today questions the view that smoking causes lung cancer and other ailments. But there was NEVER a single RCT, let alone replications or multiple RCT studies, that established that smoking causes lung cancer (or heart disease, etc.). It wa established by non-RCT studies, FAR BEYOND REASONABLE DOUBT. One reason for not using RCT design was simple; one could not ethically ask the non-smokers in a randomly chosen experimental or control group to take up smoking for a period of years when there was a serious risk that it would cause illness or death. The same reasoning applies to asking children to take up the use of drugs, violent behavior, or the avoidance of reading (or even after-school) instruction. It would be vicious in the extreme, as well as completely illogical, for the federal government, allegedly in the interest of children, to cut off funding of valid studies of such crucial matters on the grounds that RCTs are not being done there. However, there is no loss of certainty about the conclusions.



4.   More general points that one should understand in order to grasp the

magnitude of the confusion in the proposed regulations:

4A. Causation is a unique concept, one of the first that young children acquire, in that it is based not on mere cognition, nor on affective experience, but on psychomotor activities. The child acquires control over its envirnoment as it learns how to throw food on the floor, shake rattles, and push toys around on the floor. It is learning how to cause things to happen, and it is entirely correct in its belief that it can do this. Thus it grows up fully aware of how to cause (the concept of agency), then how to recognize causation when others are the agents producting similar effects, and finally how to recognize causation when naturally occurring events bring about other events. Causation is thus en integral part of the OBSERVATIONAL vocabulary of all humans with any control over their limbs (and in fact of many who lack this). In adulthood, this means that multiple observers can attest to the cause of death in traffic and other accidents, and--after due attention to the well-understood causes of error in such judgments--a jury and a court can often certify causation beyond reasonable doubt. 

4B. In well-conducted case studies, which are essentially quality-controlled observational studies with corroboration, the same is true; hence these are another of the important examples of valid non-RCT investigations. Our knowledge that twelve-step programs do indeed bring about termination of addiction in some cases, is an example of life-saving non-RCT research into causation. 

4C. It is unfortunate that the use of the term 'quasi-experimental' reinforces the naive view that such studies are a kind of poor relation of RCT studies. In fact, they are often AT LEAST AS VALID AND RELIABLE as RCTs. If we connect up a ciruit at randomly chosen times, do this a hundred times, and find that whenever we do it, but on no other occasions, a light flashes, we have demonstrated beyond reasonable doubt that the connection causes the flash (in this context). If, on twelve occasioins, we switch a single class from lectures to the new model of highly interactive learning, running pre and post tests on both such occasions, and we find that the learning gains in the interactive mode are always very close to triple those in the lecture mode, albeit on different content, and such changes do not occur spontaneously, we have proven beyond reasonable doubt, for this class at least, that the interactive approach represents an improvement of great educational importance, despite the absence of a matched control group. Repeating the study in twenty contexts, as has been done, shows the conclusion is robust, and on the way to generality, just as in the case of an RCT. In fact, for reasons extensively discussed in the literature, this method arguably provides superior validity to the use of an RCT. 

4D. There are a number of other designs that are just as good as the interrupted time series design of 4C, of which we only mention one more here. This is the design used by the forensic pathologist who discovers and proves the cause of death by working from a master list of possible causes, plus clues that shorten the list of possible causes in the present case, followed by the elimination of all but one by exhaustive search for the key 'marker-indicators' (modus operandi clues) associated with the presence or absence of each locally possible cause. It is a method that transfers easily to both individual and group studies of humans, old and young, learning or suffering. Thinking of the case of someone shot twice in the forehead with a .357, do we really want to argue that this approach is always less valid or reliable or in any way scientifically inferior to the RCT? The simple fact is that RCT design is one important and clever way amongst many that demonstrate causation, and in some complex cases it is the best choice; no more and no less so than the others are in other cases.



5. In short, the suggestion that there is one superior method for establishing causation in scientific evaluation is based on an extremely superficial and technically hopeless analysis of the concept of causation and of existing methodologies. I hope that you will not support the further starvation and prevention of good research that can improve the lives and learning of children and adults based on this naive view. There is an alternative approach for you to adopt that will do more good, AND less harm, which is to insist on the highest standards of quality in all evaluation and in particular the element in evaluation that involves the proof of causation.



Yours sincerely,



Michael Scriven

Professor of Evaluation

Auckland University, New Zealand

            ---and--

Professor of Psychology

Claremont Graduate University

Claremont, California



 

Monty Neill, Ed.D.
Executive Director
FairTest
342 Broadway
Cambridge, MA 02139
617-864-4810    fax 617-497-2224
monty@fairtest.org
http://www.fairtest.org