One of the basic principles of science-based medicine is that a single study rarely tells us much about any complex topic. Reliable conclusions are derived from an assessment of basic science (i.e prior probability or plausibility) and a pattern of effects across multiple clinical trials. However the mainstream media generally report each study as if it is a breakthrough or the definitive answer to the question at hand. If the many e-mails I receive asking me about such studies are representative, the general public takes a similar approach, perhaps due in part to the media coverage.

I generally do not plan to report on each study that comes out as that would be an endless and ultimately pointless exercise. But occasionally focusing on a specific study is educational, especially if that study is garnering a significant amount of media attention. And so I turn my attention this week to a recent study looking at acupuncture in major depression during pregnancy. The study concludes:

The short acupuncture protocol demonstrated symptom reduction and a response rate comparable to those observed in standard depression treatments of similar length and could be a viable treatment option for depression during pregnancy.

Plausibility

The study compared acupuncture designed specifically to treat depression, and in fact tailored to the individual patient, according to principles of Traditional Chinese Medicine (TCM). This was compared to two control groups – a control acupuncture that was not specific to depression and massage. The comparison to massage was obviously not blinded and therefore, in my opinion, of very little value as depression is highly susceptible to non-specific therapeutic effects and both interventions – acupuncture and massage – would be likely to create such non-specific effects.

The interesting aspect of this study is the comparison between treatment acupuncture (targeted for depression) and control acupuncture (not targeted for depression). The purpose of the study was to control, as much as possible, for any other variables so as to determine if the underlying TCM principles have any validity – does it matter where the needles are placed?

We can really only put this study into context if we first consider the prior probability of this claim. I would argue that there is already a large body of acupuncture research that collectively shows needle placement as a variable has no effect on clinical outcome. This one study does little to alter the balance of that evidence.

Further, from a basic science point of view, the TCM principles have essentially no plausibility. The underlying theory is that there is an undetected life force (chi) that is partly responsible for health and illness, that acupuncture needles placed in specific acupuncture points alters the flow and strength of this energy, resulting in a clinical outcome. Chi has no existence in science, however. Vitalistic philosophies such as chi were discarded over a century ago as both unnecessary and without any empirical foundation.

Any modern attempts to explain acupuncture effects with known physiological phenomena might explain non-specific needling effects, but cannot explain any differences due to needle placement, and do not provide any explanation for the location of alleged acupuncture points.

Therefore, given the extremely low prior probability of the claims of this study, nothing short of a large rigorous and replicated study would alter our assessment of validity of acupuncture as a specific intervention.

The Current Study

This new study, published in the Obstetrics and Gynecology, is not of sufficient quality to justify the conclusions of the authors. The authors did do a decent job of trying to rigorously control the comparison between the two acupuncture groups. Subjects were blinded to which group they were in, as were those evaluating the outcome. Standard depressions scales were used. They even made a reasonable attempt to blind the acupuncturists, using a novel method (to my knowledge).

They had experienced acupuncturists design a treatment and control acupuncture regimen for each subject, and then had a “junior acupuncturist” (less than two years experience) perform the treatment without being told which one they were giving.

This, in my opinion, in the crux of the methodology – were the treating acupuncturists properly blinded. The study authors took the very useful step of assessing the degree of blinding of the acupuncturists and the subjects. Unfortunately for the validity of the study, they found that the treating acupuncturists were significantly more likely to have positive expectations for the treatment group than the control group – so their blinding methods failed with respect to the treating acupuncturists. The study was therefore, at best, single blinded. Test subjects did not have any significant difference in expectations.

Because depression is so amenable to non-specific therapeutic effects, the expectations of the treating acupuncturist can plausibly have had a significant effect on the final outcomes. This is the primary weakness of the study – but there are other worth mentioning.

The author also, for some reason, did not stratify the test subject according to race, and there turned out to be significantly more African Americans in the control acupuncture group than the treatment group. Cultural beliefs can have a significant effect on responses to different kinds of placebos, particularly needles. This is therefore a potential, if unknown, confounder.

The results were also not impressive. The study used the Hamilton Rating Scale for depression:

Interpretation of Hamilton Rating Scale for Depression scores is as follows: less than 7, nondepressed; 8–13, mild depression; 14–18, moderate depression; 19–22, severe depression; more than 23, very severe depression.

At 8 weeks the control acupuncture groups has about a 9 point drop in the scale, while the treatment acupuncture group dropped 11.5 points. On this scale that is a modest clinical effect. There was also no difference in remission rates among the three groups. In addition this was a relatively small study (141 treated in total, divided among the three groups) with a 23% drop out rate.

Conclusion

Therefore we have a small and improperly blinded and randomized study showing a modest clinical effect. This does not significantly alter the low prior probability of a treatment effect from needle placement.

This study should also be considered in the context of other trials looking at acupuncture and depression. This very recent Cochrane review concluded:

We found insufficient evidence to recommend the use of acupuncture for people with depression. The results are limited by the high risk of bias in the majority of trials meeting inclusion criteria.

Specifically – there was no difference between verum acupuncture and sham acupuncture in the clinical trials reviewed.

Given the low plausibility and overall negative character of the clinical evidence, it is reasonable to conclude that no further research into acupuncture for any indication is warranted. However, acupuncture is a modality with dedicated practitioners (acupuncturists) and proponents (by contrast, for example, there is no medical specialty dedicated to a particular drug – there are no penicillinists). And therefore it is likely that further research will be conducted.

In that event, given existing research, it would be useful to conduct only highly rigorous trials, using sham and/or placebo acupuncture (where the needle or fake needle does not penetrate the skin) with adequate blinding. Such trials would need to be large with consistent replicated positive results in order to have sufficient weight to overturn the current mass of basic and clinical evidence.