Two recent acupuncture studies have received some media attention, both purporting to show positive effects. Both studies are also not clinical efficacy trials, so cannot be used to support any claims for efficacy for acupuncture – although that is how they are often being presented in the media.

These and other studies show the dire need for more trained science journalists, or science blogging – they only make sense when put into a proper context. No media coverage I read bothered to do this.

The first study comes out of South Korea and involves using acupuncture in a rat model of spinal cord injury. The researchers used a standard method of inducing spinal cord injury in rats, and compared various acupuncture locations to no-acupuncture control. They followed a series of metabolic outcomes, as well as the extent of spinal cord injury and functional recovery. They conclude:

Thus, our results suggest that the neuroprotection by acupuncture may be partly mediated via inhibition of inflammation and microglial activation after SCI and acupuncture can be used as a potential therapeutic tool for treating acute spinal injury in human.

The notion that acupuncture will actually improve outcome after acute spinal cord injury is, of course, extraordinary. This goes far beyond a subjective decrease in pain or some other symptomatic benefit. Therefore similarly extraordinary evidence should be required to support such a claim – and this study does not provide that.

In reading through the details of the study several factors caught my attention. The first is that there is no indication that the researchers were blinded. This alone calls the results into serious question. It is all too easy for researchers to allow personal bias to affect study results, even when they seem quantitative. We need look no further than the homeopathy research of Jacques Benveniste to see this (initial impressive results were investigated by Nature and found to be the result, charitably, if inadequate blinding).

Further, the researchers looked at several acupuncture points and then chose the ones that seemed to have an effect. This allowed for retrospective cherry picking – it is possible, in other words, that they received a scattering of random effects and chose the ones that appeared positive.

The effect sizes themselves, while statistically significant, were not clinically impressive. If they were real they would be useful in the treatment of spinal cord injury, but that is the point. Such small effect sizes are easily the product of randomness or bias.

And finally it should be noted that the study is coming out of South Korea. It is well established that countries where acupuncture is culturally important tend to have a much higher positive outcome rate than the same research in Western countries. The motivation to prove acupuncture seems to be a significant bias. Similarly, we recognize that there is a bias in favor of efficacy for pharmaceutical company sponsored research – the principle is the same. The bias of the researchers, even when well-controlled on paper, is measurable.

The bottom line with this study is that it provides weak evidence for a very extraordinary claim. It is of no practical use unless and until it is independently replicated with proper blinding. If you believe what you read in the media, however, you would be led to the conclusion that spinal injured patients could be made to walk again simply by sticking needles into magical locations on their body.

The second study uses quantitative sensory testing (QST) to look at pain threshold at baseline and after acupuncture and “electroacupuncture”. They conclude:

There were congruent changes on QST after 3 common acupuncture stimulation methods, with possible unilateral as well as bilateral effects.

In other words – acupuncture decreases the perception of pain. This small study suffers from the same primary problem as the other – it is described as only single-blinded. The subjects themselves were not blinded to whether or not they were getting “real” acupuncture vs a sham or placebo. The totality of prior acupuncture research has clearly demonstrated that such unblinded studies are all but useless. There is a significant placebo effect from getting poked with needles, and this is sufficient to explain the results of this study.

While QST is quantitative, it is still subjective. In fact, using QST has fallen a bit out of favor in neurological studies because the elaborate procedure is no more reliable as an outcome measure than straightforward sensory testing. QST is still reliant on the subjective report of the subject.

Further, this study mixed acupuncture with “electroacupuncture.” I strongly maintain that there is no such thing as “electroacupunture” – it is, rather, the application of transcutaneous electrical stimulation through an acupuncture needle. This is no more acupuncture than the application of morphine through a hollow acupuncture needle should be considered acupuncture.

It is possible that needling and electrical stimulation do decrease subjective pain perception (although we can’t conclude that based upon this study). One pain or sensory stimulation can certainly distract you from another. There is also the principle of counter-irritation – the inhibition of pain pathways by activating parallel sensory pathways. Bang your elbow and you will rub it to decrease the sharp pain.

Conclusion

Given the state of the acupuncture literature, such small and insufficiently blinded studies are of little value. It has already been established that there is a significant placebo effect surrounding the ritual of acupuncture and there are mechanisms of non-specific effects, such as counter-irritation. None of this can be logically used to support the underlying assumptions of acupuncture – that there is anything special about the designated acupuncture points, or that they can be used to manipulate “chi” or some other mysterious energy.

We are already well past the stage of preliminary studies in acupuncture. Only rigorously controlled studies are of any use. And the term “electroacupuncture” causes only confusion and cannot be meaningfully used. It is the blurring of variables when good science should endeavor to isolate variables.

Also, in a perfect world, the general press would not report on every preliminary study as if they were a definitive medical breakthrough. Such medical news stories should be covered in more focused outlets that have the space and expertise to put the results into a reasonable context.

It is my pleasure to announce the addition of a new SBM blogger. Impressed by his dedication to applying scientific principles to the profession of pharmacy, we have recruited Scott Gavura, who is currently best known for his work on Science-Based Pharmacy. You can find out a bit more about his background at his new page on SBM, and his first post is scheduled for Thursday, May 13. In the beginning he will be posting approximately once every four weeks.

Please join me in welcoming Scott to the SBM team.

For most of human history, doctors have killed their patients more often than they have saved them. An excellent new book, Taking the Medicine: A Short History of Medicine’s Beautiful Idea, and Our Difficulty Swallowing It, by Druin Burch, MD, describes medicine’s bleak past, how better ways of thinking led to modern successes, and how failure to adopt those better ways of thinking continues to impede medical progress.

The moral is not that doctors once did foolish things. The moral is that even the best of people let themselves down when they rely on untested theories and that these failures kill people and stain history. Bleeding and mercury have gone out of fashion, untested certainties and overconfidence have not.

Burch’s conversation with his rowing coach epitomizes the problem:

“I want you to keep your heart rate at 85% of max for the next hour and a half.”
“Why?”
“Because it’s the best way to improve your fitness.”
“How do you know?”
“Because I’ve done it before and it worked. Because that’s what the people who win the Olympics do. I know, I’ve trained some of them.”
“But has anyone actually done an experiment?”
“What on earth are you talking about?”

This book is Burch’s answer to his coach’s question. Medicine’s “beautiful idea” is that we should test all hypotheses and beliefs using the kind of tests that are reliable for determining the truth. Instead of going by tradition, authority, theory, common sense, or personal experience, we now have effective tools to find out for sure whether a treatment really works.

The scientific method developed slowly and there were a lot of hiccups on the way. Researchers frequently misunderstood what constituted evidence.

In an early Chinese experiment, two people were asked to run together. One was given ginseng; the other, who didn’t get ginseng, developed shortness of breath. They thought that was sufficient evidence to prove that ginseng prevented shortness of breath.

Galen gave one of his potions to a lot of patients: some recovered, some died. He thought that was evidence that the potion worked, because

All who drink of this treatment recover in a short time, except those whom it does not help, who all die. It is obvious, therefore, that it fails only in incurable cases.

Galen’s fallacious reasoning is easy to spot, but a 20th century doctor committed a similar error. He gave all his patients aspirin and asserted it was 100% effective in preventing heart attacks. Some of them did have heart attacks, but he didn’t count them because on close questioning he found that they had omitted doses or otherwise didn’t strictly follow the aspirin protocol (which was probably equally true of all his patients).

Even after the importance of randomization was recognized, there were errors in applying the principle. In early trials, randomization was by alternate allocation, where the first subject to enroll is put in group A, the second in group B, the third in group A, etc. But doctors tended to bend the rules to put certain patients in the treatment group. True randomization had to be forced on doctors who thought they knew what was best for their patients and who didn’t even realize they were cheating.

Humility is required of those who have theories rather than evidence. If they design experiments simply to confirm their prejudices, they are in danger of designing bad ones or misinterpreting results. The more researchers want to prove that the results were due to their favored treatment, the more exhaustive should be their search for alternative and equally reasonable explanations.

Burch’s book is a history of medicine with many intriguing stories about people, personalities, penicillin, opium, thalidomide, and the other usual subjects of medical history; but it is also an explanation of the scientific method and a commentary on modern medicine’s failure to rigorously and consistently apply that method.

Despite our increasing acceptance of the scientific method, the term evidence-based medicine (EBM) didn’t appear in the medical literature until 1991. Critics of scientific medicine have unfairly claimed that less than 10% of treatments are EBM. Burch points out that evidence doesn’t just consist of randomized controlled trials (RCTs), and that we have good evidence that parachutes save lives without having to do an RCT on parachutes. The 10% figure is way too low: a recent study estimated that 80% of current treatments are based on evidence.

Testing and experiment have failed to protect us from deluded cures and poisonous remedies. They can’t be relied upon unless they are carried out with method and rigor. Understanding previous mistakes helps us to avoid them.

Burch has some harsh things to say about current medical research and the processes of drug approval. Many treatments accepted as EBM are actually based on poor quality studies. 62% of studies change the definition of what they are studying between ethical approval and publication. Some studies are stopped prematurely because of apparently clear benefits or risks to patients: this is usually a mistake that diminishes the quality of data. It might be better to finish the study as planned and harm a few patients today than to harm thousands of patients later because of a false conclusion.

People worry about withholding new drugs from needy patients while they undergo testing. They worry about the ethics of offering placebos to patients when a new drug offers an apparently effective treatment. But history has shown that the new drugs in these trials are just as likely to harm as to help.

A drug’s effects, even if they are moderately large, can almost never be reliably figured out on the basis of personal experience.

Doctors are still reluctant to trust science when it goes against their prejudices. He tells how cardiologists strongly supported the first Coronary Care Units (CCUs). A study was done comparing CCU treatment to home treatment for heart attacks. The researchers told the cardiologists that there were fewer deaths in the CCU but that the difference didn’t reach statistical significance. The cardiologists all thought this trend was a strong enough reason to insist on CCUs. Then the researchers admitted they had lied: the numbers were correct but reversed. The trend had actually favored home care. Based on the same quality of evidence, the cardiologists now did not consider the data a strong enough reason to insist on home care!

Medicine is becoming more scientific and more evidence-based every day, but we can and should do better.

What is needed is a culture, regulatory and intellectual, where every attempt is made to ensure new medical interventions are used solely in randomized trials. Only when their effects have been determined should they become available for use outside a trial setting. Until then there is a moral obligation on doctors to use unknown drugs and treatments only in such a way as to come to an understanding of them, and a moral obligation on patients to demand treatments that are either supported by sound evidence or only given as part of a trial which will uncover some.

This is good advice for mainstream medicine, and it is even more important for alternative medicine, which Burch doesn’t address. Since by definition “alternative” medicine is medicine that has not been proven effective, following these guidelines would eliminate any use of alternative medicine outside of a clinical trial. I know, the money isn’t there and it would be difficult to implement, but the principle is irrefutably sound. (That’s assuming that we want to avoid using placebos and find out what really works; but I don’t think the general public wants that. I suspect they would resist and prefer to cling to untested beliefs.)  

Here’s a sampling of some of Burch’s quotable words of wisdom:

There is a bitter joke in modern medicine: the violence with which someone makes an argument is inversely proportional to the amount of evidence they have backing it up.

Trials can be full of statistics; difficult to understand and laborious to undertake. They have a loveliness to them all the same, and it comes from their power to uncover parts of the reality we live in.

[It is] our nature to prefer credulity to doubt, confidence to skepticism. We share a tendency to simplify and confuse things, to slip into mental habits that let us down.

The idea that even the most reasonable-sounding theories should be subjected to tests probably has more potential to make the world a better place than all the drugs that doctors possess. Economics, politics, social care and education are full of policies that are based on beliefs held as a matter of principle rather than because they are supported by objective tests. Humility, even more than pills, is the healthiest thing that doctors have to offer.

I highly recommend this book. It’s well-written, entertaining, and provides much food for thought. It’s a great way to learn about fascinating incidents in the history of medicine and a great way to learn what constitutes truly science-based medicine and how to avoid the errors of the past, the errors in thinking that we flawed humans are all susceptible to.