From Abracadabra to Zombies | View All
reader comments: p-value fallacy
23 Nov 2015
In your "P-value fallacy" article, you write "A p-value of 0.05 means that if the null hypothesis is true, it will be rejected in 5% of trials over many trials." This isn't clear to me, and it does not appear to be correct. I think a better and more precise thing to say is "A p-value of 0.05 for parameter beta in trial x means that, over many trials, the parameter beta has a 5% chance of being estimated at a magnitude greater than that estimated in trial x if the null hypothesis is true."
Overall, I like your article, but there is one important aspect to the fallacy that I don't think you mentioned. A p-value applies to the value of a parameter in a regression analysis, but these sorts of analyses assume the parameters themselves. For example, the use of a linear regression analysis implicitly assumes that the covariate(s) and their associated response variables can be parameterized in terms of a linear coefficient for each covariate and an overall intercept term. But clearly there are situations in which other terms (quadratic, exponential decay, and so forth) are needed to represent the relationship between covariates and a response. For problems of this sort, a p-value, aside from being often misunderstood, may be fundamentally meaningless.
Personally, I believe that if a researcher wants to use the p-value to substantiate claims, he should have to first demonstrate that the parameter in question is, in fact, either fundamental to the covariate/response relationship or is a reasonable proxy for a key feature of that relationship. I know that the foregoing is somewhat technical, but I think it's important to point out that the issues surrounding the p-value are not limited to just the researcher's and reader's understanding of it, but include general limitations of its applicability.
Last updated 23-Nov-2015