From Abracadabra to Zombies | View All
anecdotal (testimonial) evidence
Testimonials and vivid anecdotes are one of the most popular and convincing forms of evidence presented for beliefs in the supernatural, paranormal, and pseudoscientific. Nevertheless, testimonials and anecdotes in such matters are of little value in establishing the probability of the claims they are put forth to support. Sincere and vivid accounts of one’s encounter with an angel or the Virgin Mary, an alien, a ghost, a Bigfoot, a child claiming to have lived before, purple auras around dying patients, a miraculous dowser, a levitating guru, or a psychic surgeon are of little value in establishing the reasonableness of believing in such matters.
Anecdotes are unreliable for various reasons. Stories are prone to contamination by beliefs, later experiences, feedback, selective attention to details, and so on. Most stories get distorted in the telling and the retelling. Events get exaggerated. Time sequences get confused. Details get muddled. Memories are imperfect and selective; they are often filled in after the fact. People misinterpret their experiences. Experiences are conditioned by biases, memories, and beliefs, so people's perceptions might not be accurate. Most people aren't expecting to be deceived, so they may not be aware of deceptions that others might engage in. Some people make up stories. Some stories are delusions. Sometimes events are inappropriately deemed psychic simply because they seem improbable when they might not be that improbable after all. In short, anecdotes are inherently problematic and are usually impossible to test for accuracy.
Thus, stories of personal experience with paranormal or supernatural events have little scientific value. If others cannot experience the same thing under the same conditions, then there will be no way to verify the experience. If there is no way to test the claim made, then there will be no way to tell if the experience was interpreted correctly. If others can experience the same thing, then it is possible to make a test of the testimonial and determine whether the claim based on it is worthy of belief. As parapsychologist Charles Tart once said after reporting an anecdote of a possibly paranormal event: “Let’s take this into the laboratory, where we can know exactly what conditions were. We don’t have to hear a story told years later and hope that it was accurate.” Dean Radin also noted that anecdotes aren't good proof of the paranormal because memory “is much more fallible than most people think” and eyewitness testimony “is easily distorted”(Radin 1997: 32).
Testimonials regarding paranormal experiences are of little use to science because selective thinking and self-deception must be controlled for in scientific observations. Most psychics and dowsers, for example, do not even realize that they need to do controlled tests of their powers to rule out the possibility that they are deceiving themselves. They are satisfied that their experiences provide them with enough positive feedback to justify the belief in their paranormal abilities. Controlled tests of psychics and dowsers would prove once and for all that they are not being selective in their evidence gathering. It is common for such people to remember their apparent successes and ignore or underplay their failures. Controlled tests can also determine whether other factors such as cheating might be involved.
If such testimonials are scientifically worthless, why are they so popular and why are they so convincing? There are several reasons. Testimonials are often vivid and detailed, making them appear credible. They are often made by enthusiastic people who seem trustworthy and honest, and who lack any reason to deceive us. They are often made by people with some semblance of authority, such as those who hold a Ph.D. in psychology or physics. To some extent, testimonials are believable because people want to believe them. Often, one anticipates with hope some new treatment or instruction. One’s testimonial is given soon after the experience while one’s mood is still elevated from the desire for a positive outcome. The experience and the testimonial it elicits are given more significance than they deserve.
Finally, it should be noted that testimonials are often used in many areas of life, including medical science, and that giving due consideration to such testimonials is considered wise, not foolish. A physician will use the testimonies of his or her patients to draw conclusions about certain medications or procedures. For example, a physician will take anecdotal evidence from a patient about a reaction to a new medication and use that information in deciding to adjust the prescribed dosage or to change the medication. This is quite reasonable. But the physician cannot be selective in listening to testimony, listening only to those claims that fit his or her own prejudices. To do so is to risk harming one’s patients. Nor should the average person be selective when listening to testimonials regarding some paranormal or occult experience.
addendum: from my SD Newsletter, October 2014
Anecdotes Revisited
For many years I've repeated the skeptical mantras: 'the plural of anecdote is not data,' 'anecdotal evidence is unreliable,' and 'anecdotes are useful as a guide to what to study scientifically.' My recent diagnosis of pancreatic neuroendocrine cancer has motivated me to reexamine my view on the value of anecdotal evidence or testimonials.
There aren't a whole lot of us with PNETs, pancreatic neuroendocrine tumors. Many oncologists have little knowledge of NETs. There's a great deal of research being done on breast or prostate cancer and there are some standard forms of treatment available for those and many other types of cancer. There currently is no standard treatment for any kind of neuroendocrine cancer. Neuroendocrine cancer can originate anywhere in the neuroendocrine system: the gut, the pancreas, the thymus, the lungs, the brain. Mine happened to originate in the tail of the pancreas. The tumor in my pancreas, however, is small compared to the tumors seeded there and carried in my bloodstream to my liver. NETs are often slow growing. I was told that I probably had my cancer for several years before I went to my primary care physician with complaints of nausea, loss of appetite, unexplained weight loss, bloatedness, and a hypersensitive olfactory system that made me think the world around me was rotting. I would open the refrigerator and be overwhelmed by the stench of rotting food. I'd walk into a restaurant and want to turn around and walk back out. My wife could smell nothing unusual. And both of us noticed that my voice had changed; it had become raspy.
Late detection--i.e., detection after extensive metastasis--is common with NETS. Many go years being misdiagnosed with things like Crohn’s disease, peptic ulcer disease, or gastritis. By the time my cancer was detected (by a CT scan), my liver was riddled with tumors, too many to surgically remove. By the time my pancreatic tumor was detected, it had spread its network of varices over my spleen, colon, and stomach. A colonoscopy failed to find any growths inside the colon and my lack of symptoms after a few months of chemotherapy tell me that the cancer has not infected the inside of my spleen or stomach to a serious degree, if at all.
Ideally, I suppose, I would be receiving treatment based on many double-blind, randomized, control-group studies of people with PNETs that have metastasized to the liver. In each study, we'd have at least 25 subjects in each group, the experimental and the control. The experimental group would get some drug that looks promising (based, perhaps, on testimonials) and the control group would get a placebo. The experiments might go on for years while scientists tinkered with the dosages and weeded out the drugs that had a high rate of serious side effects or low rate of success in slowing tumor growth. (Success used to be measured by length of survival, but these days success is often measured by progression-free survival. We don't expect to have our cancers cured; we aim at controlling the growth of the tumors and maintaining a decent quality of life.) After years of experimentation, in an ideal world, one or two treatments would be found that are highly effective and have minimal adverse side effects on a large percentage of patients.
I think anyone reading this knows that this is not going to happen. It's obviously unethical, but see note. What is happening, then? Let me start at the end. The drugs I am taking for my cancer were not approved by the FDA or any other formal organization for pancreatic neuroendocrine cancer. One drug--capecitebine--has been approved for Dukes' stage C colon cancer. The other--temozolomide--has been approved for glioblastoma multiforme. There has never been, and probably never will be, an RCT pitting CAPTEM (the combo treatment of capecitebine and temozolomide) against a placebo. There is a randomized clinical trial going on that pits CAPTEM against treatment with temozolomide only. There was a phase II clinical trial of CAPTEM with no control group on a small set (28) of NET patients (the Fine study) that reported very promising results. Stanford University is currently recruiting for a non-randomized, no control group (single arm) phase II clinical trial using CAPTEM and Bevacizumab for Metastatic or Unresectable Pancreatic Neuroendocrine Tumors. I was hoping to get into the Stanford clinical trial, but the principal investigator, Dr. Pamela Kunz, advised me that the bevacizumab might be harmful to me because of the many varices issuing from my pancreas to my spleen, colon, and stomach. I had consulted Dr. Kunz because I was not convinced that the chemotherapy my oncologist wanted me to undergo was right for me. PNETs are classified as grade I, II, or III. My oncologist wasn't sure what grade my tumors are, but one need not spend too much time at Google University to discover that you must know the grade of the tumor before choosing the treatment. I went to Stanford and Dr. Kunz for a second opinion and she told me that the recommended treatment for me (with a grade II PNET) would not be cisplatin and etoposide (as recommended by my oncologist) but CAPTEM. My oncologist agreed to treat me with CAPTEM, though he told me that he didn't think it would work any better than cisplatin/etoposide. We'll never know.
Why would Dr. Kunz recommend CAPTEM and why would I agree to her recommendation? Neither of us could point to large RCT studies. She's one of the few oncologists in the U.S. who specializes in NETs. I trusted her more than I trusted my oncologist. I had read the Fine study and compared to what else was known about treatments for PNETs, it stood out as hopeful and promising despite its being small and having no control group. In a way, the Fine study was like a bunch of anecdotes. I can now add my own anecdote to the mix. I am in my fourth round of CAPTEM. Each round lasts 14 days, then I am off all chemo for 14 days. A CT scan taken about two-thirds of the way through my third round showed that the pancreatic tumor has shrunk about 30% and the largest tumor in my liver had shrunk about 38%. All my symptoms are gone and except for an increase in the volume of the ringing in my ears, I have had no side effects. (Note: one of the common side effects of NETs and chemo is nausea. I take no anti-nausea pills while on capecitebine--which I take with breakfast and dinner--or when off therapy. I take 8 mg of Zofran (ondansetron) 1/2 hour before taking 400 mg of temozolomide [which I take on an empty stomach before I go to bed]. I have no nausea while following this regimen. My appetite is good and I've gained back the seven pounds I lost during the first two rounds of chemo. I haven't gained back the twenty pounds I lost during the first five months of this year, but I'm not looking for them and wish them well wherever they are.)
In recent weeks, I have been following the postings of various PNET patients on a website. Some of those who post have been on CAPTEM. The reports are generally positive (i.e., tumor shrinkage and slowing the growth of the cancer), though there are some reports that indicate CAPTEM hasn't worked. Other common treatments include Sandostatin, Affinitor, and Sunitab. Some have had various kinds of surgeries and nuclear medicine treatments. In every case, the bulk of the data is in the form of anecdotes. I am finding these anecdotes very useful. I realize that it might not be the CAPTEM that has shrunk my tumors or removed my symptoms. It could be all those prayers everybody is saying for me or it could just be a fluke. I might have gone into semi-remission by chance just as I started the CAPTEM. In any case, these anecdotes give me some information that will prove useful when the CAPTEM stops working (or people quit praying for me) and I discuss the next phase of my treatment with Dr. Kunz and my new oncologist.
note: Clinical trials of NET patients do sometimes use control groups getting a placebo. For example, clinical trial NCT01658436"is a Phase II study in 2 stages, evaluating BEZ235 plus best supportive care (BSC) versus placebo plus BSC in patients with advanced pancreatic neuroendocrine tumors (pNET) after failure of mTOR inhibitor therapy." On the other hand, I couldn't imagine anyone newly diagnosed with PNET joining a clinical trial where he might be given nothing beneficial for his cancer. And I couldn't imagine a researcher who could, in good conscience, give a placebo to a newly diagnosed cancer patient when there are treatments available that might be beneficial. The exception would be crossover studies where those in the placebo group get the experimental drug at some point in the study. This may benefit the cancer patient but it can confound the data.
addendum: There have been several Food and Drug Administration (FDA) approved drugs for NETS. In 1982,streptozocin was approved to treat pancreatic neuroendocrine tumors. Although the data support the antitumor activity of streptozocin based regimens, this approach has been limited because of a cumbersome administration schedule and toxicity profile.* On May 20, 2011, the FDA approved sunitinib (Sutent® Capsules, made by Pfizer, Inc.) for the treatment of progressive well-differentiated pancreatic neuroendocrine tumors in patients with unresectable, locally advanced, or metastatic disease.* On May 5, 2011, the FDA approved everolimus (Afinitor® Tablets, made by Novartis Pharmaceuticals Corporation) for the treatment of progressive neuroendocrine tumors of pancreatic origin in patients with unresectable, locally advanced or metastatic disease.*Recently, the FDA granted priority review of Ipsen’s supplemental New Drug Application (sNDA) for Somatuline Depot (lanreotide) 120mg injection in the treatment of gastroenteropancreatic neuroendocrine tumors (GEP-NETs).* In 1988 the FDA approved octreotide, one of the most common drugs given to NET patients. "Octreotide is used in cancer treatment to control some symptoms of carcinoid syndrome (such as flushing, wheezing, and diarrhea) and to treat graft-versus-host disease. It is also used to treat diarrhea caused by chemotherapy, radiation, and AIDS."* Finally, the combination of temozolomide and capecitabine has been reported to have an objective response rate of 70% (Strosberg et al., 2011).
See also ad hoc hypothesis, Occam's razor, cold reading, communal reinforcement, control study, placebo effect, post hoc fallacy, selective thinking, self-deception, subjective validation, and wishful thinking.
further reading
books
Giere, Ronald, Understanding Scientific Reasoning, 4th ed, (New York, Holt Rinehart, Winston: 1998).
Moore, Brooke Noel. Critical Thinking (Mayfield Publishing Company, 2000).
website
"How Medical Facts Are Developed: Why Some Are More Potent Than Others" by Rodger Pirnie Doyle
Alzheimer’s and Diet "Nita Scoggan ... is promoting herself as a 'health and happiness coach' and using her husband as anecdotal evidence of the power of her nutritional advice. The story is now being promoted by other low-carb gurus, including Jimmy Moore. The story is a great example of why anecdotes are so problematic."
Ten Commandments