NPR – Statisticians Call To Arms – Reject Significance And Embrace Uncertainty

  • Why do studies often flip-flop from one answer to another?
    • one reason for the fluctuations is that scientists have a hard time handling the uncertainty
  • It is being suggest to ban the very concept of “statistical significance.”
    • declaring a result to be statistically significant or not essentially forces complicated questions to be answered as true or false
    • “The world is much more uncertain than that,” – Nicole Lazar, professor of statistics at the University of Georgia
    • entire issue of the journal The American Statistician is devoted to this question, ( a more digestible commentary
      that appears in Thursday’s issue of Nature)
  • In early 20th century, R.A. Fisher (father of statistics), developed a test of significance – involves a variable called the p-value, intended to be a guide for judging results.
    • scientists have warped that idea beyond recognition
      • created arbitrary threshold for the p-value, typically 0.05, and use that to declare whether a scientific result is significant
    • many downsides
      • scientists have been known to massage their data to make their results hit this magic threshold
      • scientists often find they can’t publish their interesting, but ambiguous, results if they aren’t statistically significant – but that information is actually still useful
    • However, others argue “Banning the word ‘significance’ may well free researchers from being held accountable
  • “All statistics naturally bounce around quite a lot from study to study to study,”  – Blake McShane, statistician Kellogg School of Management/Northwestern
    • there’s lots of variation from one group of people to another
    • this phenomenon also partly explains why studies done in one lab are frequently not reproduced in other labs (referred to as the “reproducibility crisis,”)
  • “the change needs to start there, when hypotheses are formulated, experiments designed and analyzed, and when researchers decide whether to write up and publish their work.”
  • what would scientists use instead of statistical significance
    • can still use the p-value test but as part of a broader approach to measuring uncertainty
  • Uncertainty is present always. That’s part of science. So rather than trying to dance around it, we [should] accept it.” – Wasserstein, exec director of American Statistical Association
    • Arriving at a yes/no answer is too simplistic – also need to understand things like “how big is the risk”, “how likely is it to be real”, “what are the costs and benefits”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s