From Abracadabra to Zombies

• a
• b
• c
• d
• e
• f
• g
• h
• i
• j
• k
• l
• m
• n
• o
• p
• q
• r
• s
• t
• u
• v
• w
• x
• y
• z

Skeptimedia

Skeptimedia is a commentary on mass media treatment of issues concerning science, the paranormal, and the supernatural.

»Skeptimedia archives

Skeptimedia replaces  Mass Media Funk and Mass Media Bunk. Those blogs are now archived.

Evaluating Acupuncture Studies: Laughable vs. Dangerous Delusions

May 2008. In Newsletter 85, I noted that keeping up with  acupuncture studies is very time-consuming because of the number of new studies being done. Keeping up with these studies is also very wasteful because the studies are usually fatally flawed. Meta-analyses of several small studies are most irritating; they take several reports that found nothing significant, lump them together and claim significance.

Yet, acupuncture is the darling of the media and the CAM movement, so the studies can't be ignored. Furthermore, there are thousands, if not millions, of people who walk up to skeptics every day and say: "Well, I got acupuncture and I feel better. So what do you have to say about that, Mr. Skeptic?" (See Mark Crislip's blog on Science-Based Medicine for an example.) I've said about all I have to say about acupuncture and the studies that allegedly support it as an effective medical therapy, but I continue to get requests to evaluate this or that new study. The latest was a request to evaluate a report in ScienceDaily on a study that supposedly found that "acupuncture provides effective relief from hot flushes in women who are being treated with the anti-estrogen tamoxifen following surgery for breast cancer."

Earlier this year the media was all over a British Medical Journal (BMJ) study that claimed acupuncture increases the odds of success with in vitro fertilization (IVF). STATS, an online non-profit that does in-depth analyses of media reports involving statistical data, explained the flawed reasoning of the media that claimed acupuncture fosters successful pregnancies. The authors of the BMJ article did a meta-analysis of seven studies, four of which found nothing positive to report. Even so, the data, according to the authors of the study, indicated that if ten IVF patients were treated with acupuncture, one of them might benefit. Some media stories noted that it was "far from proven" that acupuncture is of any assistance in getting pregnant, but the idea was too good to let pass without making a big deal out of nothing. The Washington Times reported that acupuncture increased the odds of getting pregnant by 65%, which is not shown by the data. [update: March 2010] 'No evidence' acupuncture boosts chances of IVF baby There is no evidence acupuncture or Chinese herbal medicine increase the chance of getting pregnant through IVF, fertility experts say....They analyzed 14 trials involving 2,670 people....No matter at what stage of the IVF process acupuncture was used, it had no impact on the pregnancy or live birth rate. The good news is that acupuncture didn't cause an increase in miscarriage rates.[/update]

It's usually not that difficult to evaluate these studies and it is pointless to refer to them when answering the Mr. Skeptic question. But we should not expect the media to help us here. Here are a few facts:

1. Acupuncture has millions of satisfied customers.

2. There are millions of satisfied customers who have received placebo therapies.

3. There have been hundreds, if not thousands, of acupuncture studies.

4. There have been countless placebo-controlled studies.

5. Some acupuncture studies have found objective physiological changes occur when getting acupuncture.

6. Some placebo-controlled studies have found objective physiological changes occur when getting placebo treatment.

What has become increasingly clear from acupuncture studies that use proper controls is that acupuncture is a placebo therapy: the effects of acupuncture are not significantly different from the effects of placebos.

A study that compares one group given acupuncture and another group given pills, exercise, massage, or some other sort of therapy is not a properly controlled study because such a design cannot measure the placebo effect.

A properly controlled acupuncture study is one that is double-blind, randomized, and uses a control group. The only proper control group for an acupuncture study is a group that receives sham acupuncture.

A double-blind acupuncture study would be one where neither the therapist nor the patient knows who is getting true acupuncture. True acupuncture is acupuncture that sticks needles into traditional acupuncture points on the body to traditional depths. Sham acupuncture is of at least three types. One type inserts needles into non-traditional sites at the same depth as traditional acupuncture. Another does the same but to a shallower depth. The third type uses a method that prevents the needles from actually being inserted into the body. It is important that the patients in the sham group think they are getting true acupuncture. It is equally important that the acupuncturist not indicate to the subjects in any way whether she is delivering true or sham acupuncture. Any study where the patients can easily detect whether they are getting true acupuncture is an invalid study because it cannot measure the placebo effect.

Some researchers obviously do not understand the placebo effect. When they compare true and sham acupuncture groups to a third group getting some other kind of therapy and find that the two acupuncture groups show a significant positive effect over the third group, they don't realize such a result supports the position that acupuncture works by the placebo effect.

A proper acupuncture study would randomly assign at least 50 patients to either a true or sham acupuncture group and not reveal to the patients which group they have been assigned to. Comparing an acupuncture group to a group that does not get acupuncture is an invalid study, unless one is trying to measure the different degrees of effectiveness of placebo treatments.

A proper acupuncture study should not have a high dropout rate and would, if appropriate, involve follow-up reviews to measure long-term effectiveness.

The most difficult thing to evaluate about acupuncture studies reported in the media is how careful the researchers were in disguising the acupuncture from the subjects. Subjects in such studies usually have a high expectation of effectiveness, so if they suspect they are not in the true acupuncture group, the placebo treatment will not be as effective. On the other hand, if those in the sham acupuncture group think they are getting true acupuncture, one should find the placebo treatment to be effective.

So, to the person who asks Mr. Skeptic to explain how acupuncture relieved her back pain or took away her wrinkles or subdued her hot flashes, one should reply: Placebos really do work! Only a person who does not understand the placebo effect would claim that acupuncture can't be a placebo since it was effective. Being effective is a necessary condition for a therapy to be ethically advocated, but it is not a sufficient condition. We know enough from the acupuncture studies that have been done that acupuncture is no more effective than a placebo. This means that acupuncture will have many satisfied customers as long as people are susceptible to suggestion and classical conditioning, believe in the effectiveness of acupuncture, and have it administered by someone who is seen as a knowledgeable and trustworthy healer. We should not forget that snake oil salesmen have always had many satisfied customers and it isn't because their snake oil contains healing properties beyond that of other placebos.

Okay, Mr. Skeptic, one might say. But if acupuncture is just a placebo, how could it help my dog or horse? Ever hear of Pavlov and classical conditioning? You should look it up. It's also possible that it didn't help your dog or horse, but you perceive it that way to confirm your bias.

If an acupuncture study has been properly done and has found that acupuncture is more effective than a placebo for some ailment or condition, one should say: That's great. Now, let's wait and see if it can be replicated. Single studies in medicine should rarely be taken to be significant. Once we get some replication in independent labs, we can feel confident we aren't dealing with a fluke study and start looking for a non-placebo explanation for the effectiveness of acupuncture. We don't have to know why a treatment is effective in order to know that it is effective. But it would be illogical to try to figure out why something's effective before we show that it is effective.

If you read about an acupuncture study in the media, try to get a copy of the published paper or at least an abstract so you can get a better idea of what the researchers tried to do and what they actually accomplished. Stories based on press releases from universities or conferences may not be unbiased and completely informative. The actual data is important to help us judge whether claims like "50% improvement" really mean anything. If the absolute numbers are very small, the percentages can be grossly misleading and of no statistical significance. Also, some news reports are about unpublished papers given at conferences. Such papers may never go through a proper peer review or get published in a peer-reviewed journal.

I suggest that anyone who wants to investigate the ScienceDaily story on acupuncture and hot flushes try to find the source of the data. The article says the research was presented on April 18, 2008, at the 6th European Breast Cancer Conference (EBCC-6) in Berlin. Look up the program and see what it says. The ScienceDaily article says the research was presented by Jill Hervik, a physiotherapist and acupuncturist at the Vestfold Central Hospital in Tønsberg, Norway. (You can view her press conference and slides online. All her data is given in percentages. It may be true that the acupuncture group had 50% fewer hot flushes at night but was the data statistically significant? We can't tell from this percentage alone. We await the actual data and replication in other treatment centers.)

Expect to continue to see news reports about the wonders of acupuncture. Expect few of these reports to be about studies that have been properly done, reached statistical significance, have been replicated in several different labs, and published in several peer-reviewed first-level scientific journals. Above all, expect to be greeted again and again by smiling satisfied customers who are totally oblivious as to why they are smiling and, frankly, don't give a damn.

Who can blame them?

As long as acupuncture is limited to such things as treating nausea from chemotherapy, it is a laughable delusion. When acupuncture is used instead of chemotherapy to treat cancer, it will become a dangerous delusion. The same should be said of other alternative treatments known to be placebos, such as homeopathic remedies. As long as homeopathy is used to treat nausea or insomnia, it is laughable. But when homeopathy is used to treat  AIDS, or to prevent malaria or pregnancy, it becomes a dangerous delusion.

[update: March 2010.] Below is an example of another acupuncture study that looks good upon a cursory review, but which falls apart when looked at more closely:

They had experienced acupuncturists design a treatment and control acupuncture regimen for each subject, and then had a “junior acupuncturist” (less than two years experience) perform the treatment without being told which one they were giving.

The study authors "found that the treating acupuncturists were significantly more likely to have positive expectations for the treatment group than the control group – so their blinding methods failed with respect to the treating acupuncturists."

Novella also raises concerns about the fact that there were significantly more African Americans in the control acupuncture group than the treatment group.

Finally, Novella points out that the Hamilton Rating Scale for depression rates a score of more than 23 as very severe depression.

At 8 weeks the control acupuncture groups has about a 9 point drop in the scale, while the treatment acupuncture group dropped 11.5 points. On this scale that is a modest clinical effect. There was also no difference in remission rates among the three groups.

So, overall, he says, what we have is "a small and improperly blinded and randomized study showing a modest clinical effect." There is no reason to think this study trumps all the earlier studies that indicate needle placement is irrelevant to any clinical outcome. [/update]

further reading

--Bausell, R. Barker. (2007). Snake Oil Science: The Truth about Complementary and Alternative Medicine Oxford. (My review of this book.)

--acupuncture, conditioning, and placebo effect (entries in The Skeptic's Dictionary)

--The trouble with acupuncture by R. Carroll

--Sticking needles into acupuncture studies by R. Carroll