Absence of evidence…

A famous axiom in medical research is that the absence of evidence of an effect is not evidence of the absence of an effect. The principle seems self-evident and many authors (including myself) have used it to make the following point: if there is a paucity of data and plausibility is on our side, we might give CAM the benefit of the doubt or at least keep an open mind about it. In this issue Reznik and Ozuah, for instance, employ it in the conclusion of their Focus article. Whenever something seems self-evident, it is worth submitting it to critical analysis. A range of scenarios could make this more explicit.

First scenario

Imagine a CAM modality X for which no trial evidence exists at all. This would be a true case of ‘absence of evidence of an effect’. Because no such evidence is available no-one would be able to say with any degree of confidence that the treatment is effective. Similarly, nobody could be sure that it is ineffective.

So far so good, but what are the implications of this type of scenario? In CAM it is frequently implied that, according to the absence of evidence principle, our judgement and advice regarding modality X should be neutral or even mildly positive: ‘it’s alright to employ X, particularly if patients want it’. Yet, scrutinising this situation more closely, this apparently (and politically) correct position turns out to be guided by double standards.

The overriding principle in medicine (and that should mean all medicine) is that any treatment must be regarded as ineffective until convincing data tell us otherwise. This is not an academic point but a maxim essential for protecting patients and providing the best possible health care. Imagine for a moment what it would mean to abandon it. A drug firm could then market an untested drug (‘absence of evidence’) and claim that, as long as patients want it, healthcare professionals should be supportive or at least neutral. Perhaps the example is extreme but it clearly signals the importance of adhering to the principle that without convincing evidence treatments must be judged very critically indeed.

Second scenario

Now imagine a CAM modality Y for which one or two trials exist and imply that Y is not superior to placebo. Intriguingly, this too is often construed as ‘absence of evidence of an effect’ in CAM. While such an interpretation could perhaps be seen as formally correct, the more realistic conclusion is that, on balance, the evidence is not neutral but negative, i.e. the available evidence fails to show an effect. In this type of scenario it would be overtly misleading to say ‘it’s alright to employ Y’. On the contrary, one would need to recommend that, until better data emerge, Y must be judged not just critically but negatively.

Third scenario

So far we have equated ‘effect’ with therapeutic effectiveness. Yet originally the principle ‘absence of evidence of an effect is not evidence of the absence of an effect’ referred to adverse effects: absence of evidence should not be mistaken for evidence of absence of risk!

For most CAM modalities, we have no adequate systems for monitoring adverse effects. This usually results in a scenario of ‘absence of evidence’. But the crucial point is that such absence could be due either to the safety of the intervention or the lack of adequate monitoring. We have prominent examples of this: hormone replacement therapy is no longer accepted as totally safe (it took decades of research and some of the largest studies ever conducted to get reliable information regarding its risks). Thus many women prefer phyto-oestrogens, which apparently are not associated with nasty adverse effects. Are phyto-oestrogens free of risk or have the risks simply not yet been discovered? In such scenarios ‘absence of evidence’ takes on a different meaning and importance. It serves as a reminder to err on the safe side.

The implications of this can be far-reaching. Absence of evidence cannot be accepted as demonstrating safety. On the contrary, treatments should usually be regarded as unsafe until proven safe. This principle is perhaps the most important axiom in clinical medicine (first do no harm). To deviate from it in CAM (as we often seem to do) not only introduces double standards in medicine, but is a serious disservice to patients and the long-term future of CAM.

Conclusion

The ‘absence of evidence’ principle has become a much-used truism in CAM. Depending on the exact circumstances, it can have several meanings and vastly different implications. It is perhaps a welcome reminder for us to use our faculties of critical analysis more acutely, particularly with truisms, platitudes and clichés.

Acknowledgement

I am grateful to Peter Canter, Max H Pittler and Jelena Savović from my unit for constructive comments on a draft version of this article.