In various contexts, for practical, philosophical, and logical reasons, there is a default assumption. In the criminal justice system, for example, someone is presumed innocent until proven guilty. Therefore we must act “as if” someone is innocent until the burden of proof is met that they are indeed guilty of the specific crime of which they are accused. In medicine, if someone has signs or symptoms sufficiently suspicious for being a deadly illness, like cancer, the clinician must assume it is cancer until proven otherwise.
In science it is very well-established where the starting assumption and therefore burden of proof lies – for any new hypothesis, it is assumed to be wrong until it has met a sufficient bar of evidence to be taken seriously, and then to be considered possibly true, and eventually probably true. Scientific hypotheses are not considered true until proven false. The burden is even more extreme because proponents are tasked with trying to prove their own hypothesis wrong, not trying to prove it correct. A hypothesis must survive repeated attempts at such disproof to be considered viable. At least, this is good science.
Scientific ideas also have another burden – they must demonstrate themselves to be superior to other alternatives, especially those with a more elegant explanation. You cannot make your theory work by simply introducing lots of new assumptions. Each assumption adds to the burden of proof, it does not subtract from it. If an alternative with fewer new assumptions is available, you must also show evidence that this alternative is not correct before your hypothesis can take the lead. This is Occam’s razor.
In scientific studies, including medical studies, we call this assumption that the claim being tested is wrong the “null hypothesis”. The evidence presented in the study must be sufficient to “reject the null” is order to be considered positive. Even then, a single study does not meet the burden of proof for any new hypothesis, but it means that the one study does support the hypothesis, or at least does not argue for rejecting it.
In medical studies (but not necessarily in other fields) the threshold for rejecting the null is traditionally set at a p-value of 0.05. This means that if the null hypothesis were true, there is a 5% or greater chance of the data in the study being what it is. This is not the same thing as a 95% chance of the hypothesis being true. It just means that the data passes the statistical smell test. You still need to meet the burden of Occam’s razor and look for other explanations of the data, the rigor of the study, the plausibility of the hypothesis, and what other studies show. Further, many experts (myself included) feel that the 0.05 threshold is too low a bar, favoring false positives too much over false negatives. But either way, it’s really OK as long as you look at all the factors I mentioned, and look at the entire literature and not just a single study.
All this brings me to the main point I wish to make – doctors and medical scientists must look at the entire scientific literature relevant to any individual question and ask, is the totality of evidence sufficient to reject the null hypothesis. I like to frame this as the following question: is it possible or probable that we are living in a world where the null hypothesis is true and this data exists? If it is anything but extremely improbable, bordering on impossible, then you have not rejected the null hypothesis. You should further ask: is there a simpler explanation for this data that does not require inventing a new and unproven phenomenon? If so, you must first reject that simpler explanation.
These questions can be extremely clarifying. For example, we can take a topic such as ESP and ask – after 100 years of research, is the totality of the evidence sufficient to reject the null hypothesis? The answer, in my opinion, is clearly no. There is no ESP paradigm that replicates a consistent effect. It is all, ultimately, noise and poor methodology.
In medicine there is also some low-hanging fruit. The most obvious such claim that has failed to reject the null hypothesis is homeopathy. Proponents of homeopathy have failed to prove any of its underlying components – there is no law of similars, substances do not carry any meaningful “essence”, water cannot be programmed with complex chemical information, and dilution does not make a potion for powerful. Further, after more than a century of research, proponents have failed to demonstrate that homeopathic products are effective for any specific indication. Homeopathy is a string of null hypotheses, and they have not rejected a single one.
In fact, beyond a certain point we can not only say that the null hypothesis (homeopathy is not real and doesn’t work) is not rejected, we can say that it has been confirmed. Everything in science is relative – there is no absolute proof, just the best inference to the most likely conclusion. But at some point we can, for all practical purposes, reject a failed idea. In medicine this means – it is unethical to use this idea as a basis for treatment. It is also wasteful, and likely unethical, to conduct further clinical research. At best, you would need to go back to the basic science drawing board and find evidence to reconsider taking the idea seriously again.
There are also regulatory implications. For something like homeopathy, which has been soundly rejected by science, insurance should not cover it, the FDA should not approve it, and medical professions should not teach or endorse it. Naturopaths do, which is part of the reason they are not a legitimate medical profession.
The other medical claim which is in a similar boat to homeopathy is acupuncture. After decades of research and thousands of studies, we have not been able to reject the null hypothesis with acupuncture for any of its basic or clinical claims. In other words, all of the evidence is compatible with a world in which acupuncture does not work.
As with homeopathy, proponents have failed the reject the null when it comes to the basic underlying components of acupuncture. There is no evidence that chi or any sort of life force exists. There is no evidence to reject the null when it comes to the existence of acupuncture points. Even under the assumption that acupuncture points exist (they don’t), there is insufficient evidence to determine where they are, how big they are, and what they do. Further, there is no evidence that inserting acupuncture needles (either at alleged acupuncture points or elsewhere) does anything other than create local trauma and all the downstream effects of that.
Clinically there is insufficient evidence to reject the null in terms of acupuncture being effective in treating any condition. This requires a bit more context. In medicine we have very clearly defined criteria for showing that an intervention has efficacy – it must demonstrate a statistically significant effect beyond placebo. Acupuncture cannot do this. There is no single application where acupuncture shows clear benefit beyond placebo, despite the claims of proponents.
But there is a lot of complexity here when you dig into the details. Simply search for “acupuncture” in the SBM search field and you will find hundreds of articles detailing all the many reasons why clinical acupuncture research does not ever reach the level of rejecting the null hypothesis (here is as good a summary as any). The biggest problem is the failure to show benefit beyond placebo is adequately blinded studies.
There is a risk in prematurely rejecting the null in medicine. There are obvious problems with using treatments that do not work. But in addition, treatments in medicine have a tendency to be institutionalized, which gives them a certain inertia. It can then become difficult to get rid of them, even when the evidence is heading in that direction. Homeopathy and acupuncture are poster-children for this effect as well – they are institutionalized despite never having rejected the null hypothesis.
We see this to a less degree with things like facilitated communication and repressed memory syndrome. These things do not exist scientifically either, and are actively harmful when used clinically, and yet they have their own institutions.
There is evidence that overall we need to slow down a bit in medicine. We are collectively a bit too eager to use new treatments, before the null hypothesis has been rejected. This can be appropriate for compassionate or experimental care in desperate situations, but not for general use. We need to be a bit more skeptical, and we need to set our thresholds for acceptance higher. That, in essence, is what science-based medicine is all about.
