SkepDic.com

Click to order from Amazon

Click here for a complete list of the SD critical thinking mini-lessons

Critical Thinking mini-lesson 6

Replication of Scientific Studies

A student who did very well in my Logic and Critical Reasoning course sent the following news item along with the suggestion that I might need to revise my thinking about lunar effects. I replied that I might need to emphasize more strongly what I teach: Look for what is not mentioned in the study, not just at what is mentioned. And don't forget how important replication of a study is.

Aug 11, 2003. (Bloomberg) -- Car accidents occur 14 percent more often on average during a full moon than a new moon, according to a study of 3 million car policies by the U.K.'s Churchill Insurance Group Plc.

The data show a rise in all types of accidents, involving single vehicles or multiple cars, the company said in an e-mailed press release. The next full moon will be tomorrow night.

``We know that the moon is a strong source of energy, as it affects the tides and weather patterns, but were surprised by this bizarre trend,'' Craig Staniland, head of car insurance at Churchill, said in the release.

The company, which Royal Bank of Scotland Group Plc agreed to buy from Credit Suisse Group in June, speculated that eastern philosophy's concepts of yin and yang may explain the accident rate. It cited a feng shui expert, Simon Brown, saying that the full moon radiates more of the sun's yang energy onto earth, making people more aggressive and impatient.

The insurer said it won't change its underwriting criteria to take the full moon into account, the company said.

In addition to yin and yang, there might be other explanations for this data, but before searching for explanations one should make sure there is something that needs to be explained. The study seems to claim that there are 14% more accidents on nights when there is a full moon than on nights when there is a new moon. (When the moon is full, if the weather is clear, it will generally be very bright. When the moon is new, even if the weather is clear, the moon will hardly be visible.)* The results of a single study may be suggestive but they are not usually considered conclusive. This study may have been well-designed but we are not told anything about how it was conducted or how it was designed, so we can't be sure. The Churchill Insurance Group may have a flawless study, but note that they didn't take the results seriously enough to alter their underwriting criteria. Why not? I don't know. What I would like to know is how was the study done?

The press release mentions a study of 3 million car policies but that's a bit vague. Did they analyze 3 million policies and separate those who made accident claims from those who didn't? Then, did they find that claims that involved accidents that happened at night when there was a full moon occurred 14% more frequently than claims that involved accidents that happened at night when there was a new moon? Did they control for weather? That is, did they review their data to make sure that there were about the same number of stormy nights on both full and new moon nights? Otherwise, they might just be measuring an effect of bad weather, not moon phases.

How many accidents are we talking about? Without knowing the numbers we can't determine whether this study had a sufficient number of cases to analyze. But even if it had many thousands of cases, we don't know over how long a period of time this study was conducted. If it analyzed data over a very long period of time, that would be more impressive than if it analyzed data over a very short period of time. Why? Over a short period of time they are more likely to get skewed results. For example, maybe the period they evaluated had two full moons in 30 days and both occurred on Saturdays. With smaller numbers it becomes more important to control for factors like the weather or weekends.

We need to know exactly how many accidents were involved in the study, the beginning date and end date of the data collection, the exact number of nights involved, and the exact number of full and new moons during the study. We should also be assured that only accidents that occurred after the rising and before the setting of the full moon were included in the study. If the accidents happened during the day or before the full moon was present, the likelihood that the moon had anything to do with diminishes significantly.

Finally, even if the study was based on a sufficient number of cases over an adequate period of time and included only data it should include (and didn't include data it shouldn't include), and even if the data were analyzed properly by professional statisticians, we should still wait until it is replicated before worrying about finding an explanation for the 14% statistic. A single study with statistically impressive results should not be taken as sufficient to base any important decisions on.

Now, trying to prove the statistic is due to yin and yang is another matter altogether. I have no idea how anyone could construct a scientific study to test that hypothesis.

But we can at least correct one misconception put forth in this press release: the moon is not a strong source of gravitational energy on earthlings. George Abell claims that a mosquito would exert more gravitational pull on your arm than the moon would. Ivan Kelly put it this way: "A mother holding her child "will exert 12 million times as much tidal force on her child as the moon."*

Why would anyone cite this study favorably? Confirmation bias. If you already believe in lunar effects, this study confirms your belief. You will be less likely to be critical of it than if it goes against your beliefs. Also, the suburban myth that the moon is a strong source of energy continues to be reported in the media, giving many people the impression that it must be true.

lesson 7: fallacy of suppressed evidence