Skeptic's Dictionary Facebook PageFrom Abracadabra to Zombies | View All
Critical Thinking mini-lesson 10
Control Group Studies
Dr. Alan Hirsch claims to be "The World Expert In Smell & Taste." He is an M.D. - a psychiatrist, in fact - who has developed some magical crystals that will "help you reduce your appetite and food cravings." You can read all about his crystals, which he calls SprinkleThin™, on his website. On July 25, 2005, I found the following testimonial on that website.
Dateline NBC Investigates SprinkleThin
“What Dr. Hirsch discovered might surprise you. [Certain smells] seem to control appetite. Dr. Hirsch studied 2,700 people over six months, like the six people we met. They tried just about every diet imaginable. Dr. Hirsch brought along with him these special, non-caloric, scented crystals and asked the six to sprinkle it on their food.
All the participants kept a video diary for Dateline to prove they were using the product. At the end of three months when we checked in on them, they were all losing weight.”
What is wrong with Dateline's investigation? Among other things, Dateline did not have a control group. Dr. Hirsch says he has been studying eating behavior and weight loss for 25 years. He says he has done many studies, but if his studies were like Dateline's study they are not of much scientific value.
What is a control group and why is having one important?
A well designed study on the diet crystals would use two groups. The group getting the crystals would be called the experimental group. The control group would be a group that, ideally, is identical to the experimental group except that the members of the control group do not use the crystals. The ideal can never be fully achieved with humans, especially when one is doing a study that involves weight loss because (a) weight loss is affected by many factors (motivation, eating behavior, amount of activity - especially exercise - overall health, metabolism, stress, and so on) and (b) experimenters can't lock up humans in cages to make sure they do what they're supposed to do for the study. But, at the very least, a well designed scientific study should use a control group and try to match the members of that group to those in the experimental group for factors that might have a significant effect on the outcome. For example, if you were doing a study that was testing whether prayer has an effect on the longevity of patients dying of AIDS, you should make sure that the ages of the subjects in both groups match up. It would not be a fair study to have 60-year-olds in one group and twenty-somethings in the other group.
Having a control group allows the scientist to test a causal hypothesis. In this case, the hypothesis is that SprinkleThin™ is a significant causal factor in producing weight loss.
Without a control group, a scientist can't be sure that the diet crystals contributed significantly to the weight loss or, if they did, in what way. The placebo effect may be at work here: dieters may believe these crystals really affect their sense of taste and smell to such a degree that their appetites are suppressed. They may be deceiving themselves, but the crystals help them anyway. However, powdered beetle dung might have had the same effect. The diet scientist doesn't just want to help people lose weight. If a product works, she wants to know why it works.
Dateline (and Dr. Hirsch) should not just give the crystals to dieters and observe whether they lose weight. They should have a group of similar people who want to lose weight and give them a placebo, a substance that looks like the diet crystals and is ingested in exactly the same way, but which is inert. They should agree to study the two groups for a set length of time, long enough for any diet to show results (several weeks, at least). At the end of the study they would compare the weight loss of the two groups. If the experimental group shows a significantly greater weight loss than the control group, then the scientists have good evidence that the crystals might be effective. There are various reasons why the results of a single study should not be taken as proof of one's causal hypothesis. We'll return to this issue later.
Having a control group is necessary but it is not sufficient for having a well-designed control group study. The study must use an adequate number of participants. Six people would not be adequate for a control group study. Several hundred would be a better number. Why? With only six people, all it takes is one participant to do really well to elevate the average of the group significantly above the average of the other group. But this one person's success might be a fluke. By having a larger sample, the researcher reduces the chances that a few fluky individuals have skewed the results.
Another way to reduce the chances of fluky results is to randomly assign subjects to the control and experimental groups. Randomization is very important to reduce the chances of biasing the samples. If highly motivated folks are placed in the diet crystal group and a bunch of lazy couch potatoes are in the control group, the results of the study would be biased. It is important that a method of true randomization be used, such as a random number table. You might think that assigning all the dark-haired subjects to one group and the light-haired subjects to the other would be sufficient to avoid having biased groups, but you cannot be sure that there is not something about hair color that is related to a person's weight. It is unlikely, but a scientist should not go with hunches in matters such as randomization.
It is also important that the subjects in this study not know whether they have been given the magic crystals or the placebo. There is much controversy regarding the ethics of deceiving subjects, but from a scientific point of view it might be better if the subjects didn't even know that the study is about weight loss. If they think, for example, that the study is testing the effectiveness of a new blood pressure medicine, you would eliminate such things as motivation to lose weight or belief that the crystals are appetite suppressants as possible causes of any weight loss achieved. However, many, if not most, scientists argue that it is unethical to deceive participants in scientific studies. The subjects in a study don't need to be told which group they are in, but they should be told that they have been randomly assigned to their group and that at the end of the study they will be told which group they were in.
In many studies, not only should the subjects be blind to which group they are in for the duration of the study, but the experimenters should also be blind to which group the subjects have been assigned to. Double-blind studies require at least two experimenters, one who assigns the subjects to their groups and one who keeps track of the data. Had Dr. Hirsch done a double-blind study, an assistant might have randomly assigned the subjects to their groups and kept a record of who is in which group. Dr. Hirsh or another assistant might have weighed all the subjects and kept weight records for each participant. After all the data had been collected, Dr. Hirsch would "unblind" the study and the data for the two groups be compared.
The final step in a well-designed study is the analysis of the data. You might think that the scientists should be able to look at the results and see right away whether the crystals did any good. This would only be true if, say, there were hundreds in each group and the experimental group lost 50 pounds each on average, while the control group gained 2 pounds. If the study had been designed properly, such results would be extremely unlikely to be a fluke. But what if the experimental group lost 2% more weight than the control group? Would that be significant? To answer that question, scientists revert to statistical formulae. By some formula, a 2% weight loss might be statistically significant. If, however, a 2% weight loss meant 4 ounces over six weeks, most of us would say that even if this is statistically significant it is not important and not worth the money or the risk to use these crystals. The crystals might have some wicked side effect that hasn't yet been discovered.
The moral of this story is that while testimonials of six people who use crystals and lose weight might have a powerful effect on a television audience, a critical thinker should recognize that without a well-designed control group study, such testimonials do not have much scientific value.
A critical thinker also knows that information should be put in the proper context, which requires a certain amount of background knowledge. For example, you should know that many well designed scientific studies get significant results that cannot be replicated at all or in a consistent fashion. If there is a causal relationship between diet crystals and losing weight, it should not work sporadically but consistently, unless, of course, there are so many factors that affect body weight as to make it nearly impossible to isolate the true effectiveness of a single item. A single study, no matter how well designed or how significant the results, rarely justifies drawing strong conclusions about causal relationships.
Finally, as mentioned above, there might be some deleterious side effect of these crystals that has not yet been discovered. SprinkleThin™ might help you lose weight but if it kills you in the process, what have you gained?
Postscript
The kind of control group study described above is known as a parallel group study. However, as Dr. Gerard Dallal writes: "It takes little experience with parallel group studies to recognize the potential for great gains in efficiency if each subject could receive both treatments. The comparison of treatments would no longer be contaminated by the variability between subjects since the comparison is carried out within each individual." Such studies are known as crossover studies. They are highly recommended.
lesson 11: false dichotomy Last updated 12/09/10
