Alpha

One of the most frustrating things I have seen as a graduate student was a p-value of 0.06. In fact, I have seen this terrible number on more than one occasion. A p-value of 0.06 means that there is a six percent chance that the results I observed were a result of random chance if there really was no effect. Now, I personally think, when it comes to ecology at least, that six percent is low. Think about all the troubleshooting ecological research requires or how we are often limited to small sample sizes. How different is six percent really from 5 percent?

I am not the only one who is suspicious of the alpha values. I have listened to professors proclaim the death of traditional statistics and the end of an era of α < 0.05. Ecologists are moving towards Bayesian statistics. If you have been around FIU long enough you may remember the prestigious Glaser Seminar speaker in 2014 was Dr. David Anderson.

The other day I was surprised to see an article that was advocating for lowering alpha values to 0.005! What??? 

THEY ADDED AN EXTRA ZERO AFTER THE DECIMAL! 

I was shook.

After reading more I began to understand their reasoning. Benjamin et al. are focused on preventing false positives, or type I error. They are concerned with the lack of reproducible experiments. One example of this is in drug research where a new drug is shown to have a significant effect during the trials but then years later, after released, scientists discover that there is no effect or even a harmful effect. This wastes time and money and increases the general public’s mistrust of science.

Overall, Benjamin et al. did a thorough job at explaining their logic and took time to preemptively respond to anticipated objections. They make it clear that the change to 0.005 is a standard for evidence but should not be a standard for publication or policy. I think this might be what saves us ecologists from perishing.

Their justification for lowering alpha reminded me of how natural gas is often presented as the transition fuel. You know those people who say, “natural gas burns cleaner than coal or oil and will help buy time as we transition to renewable energy”. I interpreted Benjamin et al.’s conclusion along the lines of: lowering the alpha helps mitigate some of the systemic issues in modern scientific research but it is only a temporary fix for a much larger issue that will take more thought and debate.

What do you think about lowering alphas to 0.005? How would that change future FCE findings? 

I would love to hear your thoughts on the value of α and p! Comment below!

Comments

  1. I used p=0.01 for my Ph.D. thesis and it did simplify my interpretations around key findings. It's useful to comment in your discussion sections on stronger vs weaker findings based on p-values and other criteria such as AIC for models, e.g. regressions. What is often lacking are stronger / more frequent collaborations between statisticians / statistical ecologists and ecologists who are great at doing empirical work, but not so great with super-quantitative analyses. Having to-go experts on this within departments at all seniority levels can help, especially if there are more opportunities for open discussions on one's data, results, and discoveries.

    ReplyDelete
    Replies
    1. Thanks Luca! I can see how having an alpha of 0.01 would make interpreting results clearer. Do you find it acceptable to discuss results as marginally significant or approaching significance? I was taught to use the alpha as a strict cutoff. The difference is either significant or not. Thanks!

      Delete
  2. It depends on your question. I have argued that for conservation purposes, p of 0.1 should be enough to show a potential effect of contaminants to stream communities. If the question is about conserving a rare species or preventing potential harm to the environment and human health, p of 0.005 or even p of 0.05 may be too conservative.

    ReplyDelete
    Replies
    1. I think that the authors would agree with you here. They conclude by saying that an alpha of 0.005 should not be a standard for publication or management. In my opinion It is better to be safe than sorry when it comes to avoiding environmental consequences.

      Delete
  3. I published work discussing 'slight' or 'not significant' differences as compared to major ones. Given that the p-value threshold is a choice, I think it's good to qualify your statements and you can also put most highly significant differences in bold in a table as another way of guiding the reader to a faster and better (more nuanced) understanding of your results. Also, zero is a result too and sometimes the absence of significant differences or good predictive models may mean that we are not measuring enough variables and factors and this can lead to more in depth analysis and new research, right?

    ReplyDelete

Post a Comment

Popular posts from this blog

Diatom of the month - February 2016: Cyclotella meneghiniana

Researching Algae, the Unsung Heroes of Aquatic Food Webs