Focus on Alternative and Complementary Therapies
www.pharmpress.com/fact
Focus Alternat Complement Ther©2005 Pharmaceutical Press
Focus Altern Complement Ther 2005; 10: 181–3
Karl-Ludwig Resch
Typically, clinical trials aim to scrutinise the assumption that there is a causal relationship between an intervention (exposure) and a concomitant and/or subsequent change (effect). RCTs, which are very likely to achieve structural equality between two or more groups of individuals under investigation by means of a proper randomisation procedure, are commonly considered to be the methodological gold standard for this type of investigation.
The technique of the RCT was consciously chosen for the first time in the late 1940s in a drug trial in the UK, and has rapidly become standard in this very area of medicine. Recent analyses suggest that up to four out of five RCTs on common conditions deal with pharmacological aspects of therapy.1 Naturally, in this context, the main focus is on the role of a drug, i.e. the ‘specific’ effect of a given pharmacological substance, and RCTs focus on questions of efficacy (‘can it work’) in the first place. The (‘quasi’) standard of comparison is that of an ‘active’ vs. a presumably ‘inert’ substance with the obvious and methodologically inevitable consequence of masking patient as well as therapist and/or observer (double-blinding). Core outcomes are typically validly quantifiable, ‘external’, ‘objective’ variables such as technical measurements, laboratory parameters, quantitative analyses of imaging procedures, etc. It seems that efficacy trials tend to aim more at producing elaborate statistical differences of ‘handy variables’ rather than insights into the clinical relevance (‘…many of which addressed questions of little relevance to current management issues’1).
In ordinary daily clinical practice, however, what really matters is the magnitude of the probable de facto clinical effect, i.e. the overall effectiveness (‘does it work’) of a given intervention. This type of study has been referred to as a ‘pragmatic trial’ or a pragmatic randomised controlled trial (pRCT),2 typically aiming at a comparison of different potential treatment strategies (including the option not to treat, ‘watchful waiting’). Main outcomes include or even focus on perceived and/or subjectively quantified outcomes such as pain, function or health (‘…using the “big 4” measures: length of life, quality of life, discrete negative events, and costs’3). In terms of classification systems, pRCTs tend to correspond with the WHO’s International Classification of Functioning Disability and Health (ICF) more than with its sister classification ICD (International Classification of Diseases) because, unlike the latter, the former does not classify an objective diagnosis in the first place, but rather aspects that affect the activity and participation of an individual, thus determining the individual ‘health outcome’.
In an era in which resources have become increasingly scarce and budgets increasingly tight, politicians and purchasers are being forced to put more emphasis on aspects such as the relevance of an intervention to appropriate healthcare provision or to issues of cost–benefit or cost–utility.4 It is complex health problems, issues of disease management, rehabilitation and secondary and tertiary prevention of chronic conditions, where trials designed as pRCTs often seem to be more appropriate to generate clinically relevant answers than the ‘classical’ efficacy trial. In this context the lack of a placebo arm (and thus the waiver of double-blinding) cannot be considered a methodological drawback. Single or just observer blind studies can be methodologically adequate and sound enough to produce valid evidence.
However, the methodology of pRCTs is still much less sophisticated and elaborate than that of the efficacy trial, although there has been an almost exponential increase of reports of this type of trial over the last decade or so. Increasingly, methodologically profound suggestions, recommendations and guidelines (like, for instance, the MRC’s recent Framework for Development of RCTs for Complex Interventions to Improve Health) assist researchers in choosing an adequate methodology for their respective investigation. It can be expected that the methodology of pRCTs will eventually come to be established as a valuable and appropriate tool to study particular questions, e.g. of relevance to the delivery of health care – and of relevance to the individual consumer, who will increasingly be forced to buy health services out of his or her own pocket.
Edzard Ernst
For decades, a debate has raged about the question of whether or not RCTs are an acceptable way of testing the effectiveness/efficacy of CAM treatments. The debate has often been characterised by generating more heat than light, and progress has therefore been painfully slow. Finally it seems that the majority of CAM researchers are now accepting that the RCT is a valuable tool in CAM research (many CAM practitioners, I fear, have not yet approached this stage). Just as this controversy amongst researchers seems to be calming down, a similar one is coming to the fore: are pragmatic CAM trials valuable?
Pragmatic trials or effectiveness trials, as they are also called, are aimed at approximating clinical practice. Their strength is high external validity, i.e. good generalisability. On the other hand, efficacy trials take place under ideal conditions. Their strength is high internal validity. The main limitation of a pragmatic trial, according to Earl-Slater, is that ‘the area of reality it seeks to approximate may not in fact be common practice.’1
It would be silly to argue that pragmatic trials are without value. Of course, they can provide valuable information on how useful a treatment is under ‘real life’ circumstances. But ‘real life’ is invariably messy and confounded. Relaxing the rigour of RCTs with high internal validity in order to achieve high external validity can therefore be fraught with problems. These problems may become exaggerated when the investigators are ardent believers in the treatment they study. The lack of critical thinking in designing a study can introduce bias and flaws that may later be undetectable in the published report. Evidently this is true for any scientific investigation but it is temptingly easier in pragmatic trials. Science is a means of testing hypotheses; I am repeatedly stunned when reading in the write-ups of pragmatic CAM trials that CAM researchers aimed at ‘proving’ that the treatment has an effect. I fear that pragmatic trials invite such ‘investigator bias.’2 Bias, in turn, will mislead us, and this cannot be good for anybody, least of all the patient.
Theoretical points are of limited use and clarity. Perhaps it is best to use an example (there are certainly plenty to choose from in CAM!). One of the most recent pragmatic CAM study is an investigation aimed at comparing the clinical outcomes of anthroposophic and conventional treatment of acute respiratory and ear symptoms.3 The investigators recruited 1016 patients, who were treated by either a conventional or an anthroposophic physician. The results looked impressive: improvements within 24 h were noted in 31% (anthroposophy) and 17% (orthodox medicine) of the patients and adverse effects were experienced by 3% vs. 6%. The authors conclude that ‘anthroposophic treatment … is safe and at least as effective as conventional treatment.’
This is my main objection to pragmatic trials of CAM: weak data tend to be hopelessly overstretched to the point where we are ‘led up the garden path’. In the above example, one of many interpretations is that conventional doctors quite simply over-treated with antibiotics, which evidently had detrimental effects. In such a scenario even a sugar pill would have generated better outcomes without the alternative treatment having any beneficial effect at all!
It may be unfair to simply use an obviously flawed study as an example against pragmatic trials so let’s also look at one that, by anyone’s standards, is of high quality. Vickers et al.4 randomised headache patients (mostly migraineurs) into either receiving or not receiving acupuncture. The results show a range of benefits, including economic ones, in the former compared to the latter group.4 As far as pragmatic trials go, this must be one of the best. Yet I find that it tells us little of lasting value. Acupuncture was ‘en vogue’ in London when the study was conducted. Thus it is reasonable to assume that the expectations of patients seeing an acupuncturist were high. Conversely those randomised to not having acupuncture were perhaps dispirited. These effects alone could account for the observed group differences without acupuncture having any effect beyond placebo at all. The fact that acupuncture works under ‘real life’ conditions today does not mean that it will work tomorrow (when the enthusiasm for acupuncture has worn off). The fact that it worked in London does not mean that it will work in Berlin, Paris or New York. In such a scenario (I deliberately exaggerated to make my point) generalisability is merely a pipe dream. In this case there was no reason why a sham acupuncture arm could not have been included in the design. In some CAM studies, ‘pragmatic’ appears to be an excuse for lack of precision.
In conclusion, pragmatic trials invite bias. Even when they are well done they tend to tell us little of true, lasting importance. They are not necessarily reproducible, and if a study is not reproducible it fails to fulfil one of the most important preconditions for qualifying as science. If it’s not science, it may well be pure propaganda.
