p-value fallacy - The Skeptic's Dictionary

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>p-value fallacy - The Skeptic's Dictionary</title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta name="keywords" content="skepticism, critical thinking"> <link rel="stylesheet" type="text/css" href="_style/style.css" /><link rel="stylesheet" type="text/css" href="_style/printstyle.css" media="print"/> <link rel="shortcut icon" href="s1.ico"> <script>(function(d, s, id) { var js, fjs = d.getElementsByTagName(s)[0]; if (d.getElementById(id)) return; js = d.createElement(s); js.id = id; js.src = "//connect.facebook.net/en_US/sdk.js#xfbml=1&version=v2.0"; fjs.parentNode.insertBefore(js, fjs); }(document, 'script', 'facebook-jssdk'));</script>  <script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-5487a05a7e027f30" async="async"></script> <script>(function(d, s, id) { var js, fjs = d.getElementsByTagName(s)[0]; if (d.getElementById(id)) return; js = d.createElement(s); js.id = id; js.src = "//connect.facebook.net/en_US/sdk.js#xfbml=1&version=v2.0"; fjs.parentNode.insertBefore(js, fjs); }(document, 'script', 'facebook-jssdk'));</script>  <script>(function(d, s, id) { var js, fjs = d.getElementsByTagName(s)[0]; if (d.getElementById(id)) return; js = d.createElement(s); js.id = id; js.src = "//connect.facebook.net/en_US/sdk.js#xfbml=1&version=v2.0"; fjs.parentNode.insertBefore(js, fjs); }(document, 'script', 'facebook-jssdk'));</script>  <script type="text/javascript" src="//s7.addthis.com/js/300/addthis_widget.js#pubid=ra-5487a05a7e027f30" async="async"></script></head> <body><br /> <div class="container"> <div class="header"> <div class="logo"> <img src="_images/logo_01.png" alt="The Skeptic's Dictionary" width="270" height="60" /> <br> </div> <div class="slogan"> <img src="_images/slogan_01.gif" alt="A Collection of Strange Beliefs, Amusing Deceptions, and Dangerous Delusions" width="362" height="30" /> </div> </div> <div class="menu"> <ul> <li><a href="http://skepdic.com/">Home</a></li> <li><a href="http://skepdic.com/news/">Newsletter</a></li> <li><a href="http://skepdic.com/feedback.html">Feedback</a></li> <li><a href="http://skepdic.com/faq.html">FAQ & Interviews</a></li> <li><a href="http://skepdic.com/tour.html">Site Map</a></li> </ul> </div> <div class="middle"> <div class="left"> <h2>Topical Indexes</h2> <ul class="nav_1"> <li><a href="http://skepdic.com/tialtmed.html">Alternatives to Medicine</a></li> <li> <a href="http://skepdic.com/ticriticalthinking.html">Critical Thinking</a></li> <li><a href="http://skepdic.com/ticrypto.html">Cryptozoology</a></li> <li><a href="http://skepdic.com/tialien.html">ETs & UFOs</a></li> <li><a href="http://skepdic.com/tifraud.html">Frauds, Hoaxes, Conspiracies</a></li> <li><a href="http://skepdic.com/tijunk.html">Junk Science</a></li> <li><a href="http://skepdic.com/tilogic.html">Logic & Perception</a></li> <li><a href="http://skepdic.com/tinewage.html">New Age</a></li> <li><a href="http://skepdic.com/tipara.html">Paranormal</a></li> <li><a href="http://skepdic.com/tiscience.html">Science & Philosophy</a></li> <li><a href="http://skepdic.com/tisuper.html">Supernatural</a></li> <li> </li> </ul> <h2>Other Writings</h2> <ul class="nav_2"> <li> <a href="http://59ways.blogspot.com/">Unnatural Acts blog</a></li> <li> <a href="http://skepdic.com/skeptimedia/skeptimediaarchive.html"> Skeptimedia</a></li> <li><a href="http://skepdic.com/refuge/funkarchive.html"> Mass Media Funk</a></li> <li><a href="http://skepdic.com/refuge/bunkarchive.html"> Mass Media Bunk</a></li> <li><a href="http://skepdic.com/refuge/harmarchive.html"> What's the Harm?</a></li> <li><a href="http://skepdic.com/news/">Newsletter Archives</a></li> <li><a href="http://skepdic.com/refuge/wwwbunk.html"> Internet Bunk</a></li> <li><a href="http://skepdic.com/refuge/2good.html"> Too good to be true</a></li> <li><a href="http://skepdic.com/refuge/essays.html">Skeptical Essays</a></li> <li><a href="http://skepdic.com/refuge/randr.html"> Book Reviews</a></li> <li><a href="http://skepdic.com/refuge/suburbanmyths.html"> Suburban Myths</a></li> <li><a href="http://skepdic.com/refuge/memoriam.html"> In Memoriam</a></li> </ul> <h2>Other Resources</h2> <ul class="nav_2"> <li> <a href="http://sd4kids.skepdic.com"> Mysteries & Science for Kids</a></li> <li><a href="http://skepdic.com/UnnaturalVirtue.html">Podcast: Unnatural Virtue archive</a></li> <li><a href="http://skepdic.com/hollow.html">A Skeptic's Halloween</a></li> <li><a href="http://renish.skepdic.com/AllNotes.html">Editor's Notes</a></li> <li><a href="http://skepdic.com/refuge/getinvolved.html"> Get involved</a></li> <li><a href="http://skepdic.com/future.html">Future Topics?</a></li> <li><a href="http://skepdic.com/permissions.html"> Permission to print</a></li> <li><a href="http://skepdic.com/report/">Site Statistics</a></li> </ul> </div> <div class="content"> <div class="it"> <h2>From Abracadabra to Zombies<span class="small"> | <a href="http://skepdic.com/contents.html">View All</a></span></h2> <ul class="alphabet"> <li><a href="http://skepdic.com/contents.html#A">a</a></li> <li><a href="http://skepdic.com/contents.html#B">b</a></li> <li><a href="http://skepdic.com/contents.html#C">c</a></li> <li><a href="http://skepdic.com/contents.html#D">d</a></li> <li><a href="http://skepdic.com/contents.html#E">e</a></li> <li><a href="http://skepdic.com/contents.html#F">f</a></li> <li><a href="http://skepdic.com/contents.html#G">g</a></li> <li><a href="http://skepdic.com/contents.html#H">h</a></li> <li><a href="http://skepdic.com/contents.html#I">i</a></li> <li><a href="http://skepdic.com/contents.html#J">j</a></li> <li><a href="http://skepdic.com/contents.html#K">k</a></li> <li><a href="http://skepdic.com/contents.html#L">l</a></li> <li><a href="http://skepdic.com/contents.html#M">m</a></li> <li><a href="http://skepdic.com/contents.html#N">n</a></li> <li><a href="http://skepdic.com/contents.html#O">o</a></li> <li><a href="http://skepdic.com/contents.html#P">p</a></li> <li><a href="http://skepdic.com/contents.html#Q">q</a></li> <li><a href="http://skepdic.com/contents.html#R">r</a></li> <li><a href="http://skepdic.com/contents.html#S">s</a></li> <li><a href="http://skepdic.com/contents.html#T">t</a></li> <li><a href="http://skepdic.com/contents.html#U">u</a></li> <li><a href="http://skepdic.com/contents.html#V">v</a></li> <li><a href="http://skepdic.com/contents.html#W">w</a></li> <li><a href="http://skepdic.com/contents.html#X">x</a></li> <li><a href="http://skepdic.com/contents.html#Y">y</a></li> <li><a href="http://skepdic.com/contents.html#Z">z</a></li> </ul> <webContent> <h1>p-value fallacy</h1> <blockquote> <p class="quote">Many researchers have labored under the misbelief that the p-value gives the probability that their study’s results are just pure random chance. --<a href="http://www.scientificamerican.com/article/scientists-perturbed-by-loss-of-stat-tools-to-sift-research-fudge-from-fact/">Regina Nuzzo</a></p> </blockquote> <p>The p-value fallacy is "the mistaken idea that a single number can capture both the long-run outcomes of an experiment and the evidential meaning of a single result" (Goodman, 1999). Many people mistakenly think that a p-value of 0.05 means that there is a 5% chance that the <a href="https://en.wikipedia.org/wiki/Null_hypothesis">null hypothesis</a> is true. A p-value of 0.05 means that if the null hypothesis is true, it will be rejected in 5% of trials over many trials. Thus, the p-value for a single trial doesn’t provide conclusive evidence that a hypothesis is correct. We need many trials before we should assert with confidence that the null hypothesis is true or false. </p> <p>In February 2015, editors of the scientific journal <em>Basic and Applied Social Psychology</em> announced that researchers who submit studies for publication would not be allowed to use the p-value. Biostatistician Jeff Leek of Johns Hopkins University says that “the p-value is the most widely known statistic.” He estimates that the p-value has been used in at least three million scientific papers. P-values may be extremely popular, but they are widely misunderstood and believed to provide more information than they do. </p> <blockquote> <p>Many researchers have labored under the misbelief that the p-value gives the probability that their study’s results are just pure random chance. But statisticians say the p-value’s information is much more non-specific, and can be interpreted only in the context of hypothetical alternative scenarios: The p-value summarizes how often results at least as extreme as those observed would show up if the study were repeated an infinite number of times when in fact only pure random chance were at work.</p> <p>This means that the p-value is a statement about imaginary data in hypothetical study replications, not a statement about actual conclusions in any given study. Instead of being a “scientific lie detector” that can get at the truth of a particular scientific finding, the p-value is more of an “alternative reality machine” that lets researchers compare their results with what random chance would hypothetically produce.<a href="http://www.scientificamerican.com/article/scientists-perturbed-by-loss-of-stat-tools-to-sift-research-fudge-from-fact/">*</a></p> </blockquote> <p>What does this mean for all those millions of papers whose grand conclusions were based on the assumption that the p-value was a valid guide to the probability of some hypothesis being true? It means that there are many scientific papers that have been published in tier one peer-reviewed journals that contain--in the words of psychologist Eric-Jan Wagenmakers of the University of Amsterdam--"plain nonsense."</p> <p><strong>further reading</strong></p> <p><a href="http://skepdic.com/comments/pvaluecom.html"><strong>reader comments</strong></a></p> <p><a href="http://www.annals.org/content/130/12/995.abstract">Goodman, Stephen N. 1999. Toward Evidence-Based Medical Statistics. 1: The P Value Fallacy, <i>Annals of Internal Medicine</i>. </a></p> <p><a href="http://www.scientificamerican.com/article/scientists-perturbed-by-loss-of-stat-tools-to-sift-research-fudge-from-fact/">Nuzzo, Regina. 2015. Scientists Perturbed by Loss of Stat Tools to Sift Research Fudge from Fact. <em>Scientific American</em></a>. April 16. The journal <em>Basic and Applied Social Psychology</em> recently banned the use of p-values and other statistical methods to quantify uncertainty from significance in research results.</p> <p> <a href="http://www.sciencebasedmedicine.org/?p=2996" rel="bookmark"> Acupuncture, the P-Value Fallacy, and Honesty</a> by Kimball Atwood</p> <p><a name="new" id="new"></a>[<span class="highlight">new</span>] <a href="https://www.sciencebasedmedicine.org/p-value-under-fire/">P Value Under Fire</a> by Steven Novella "The main problem with P-values is that people use them as a substitute for a thorough analysis of the overall scientific rigor of a study. If the P-value creeps over the 0.05 level then people assume the hypothesis is likely to be true – but that is not what the P-value means."[/<span class="highlight">new</span>]</p> <p><a href="https://en.wikipedia.org/wiki/P-value">p-value Wikipedia</a></p> <p></webContent>Last updated 15-Mar-2016 </p> </div> </div> <div class="right"> <div class="right_it"> <div class="search"> <form method="get" action="http://www.google.com/custom" target="_top"> <input type="hidden" name="domains" value="skepdic.com" /> <input type="text" name="q" maxlength="255" value="" size="20" /> <input type="submit" name="sa" value="Search" /> <br /> <div class="search_2"> <input type="radio" name="sitesearch" value="" /> Web     <input type="radio" name="sitesearch" value="skepdic.com" checked="checked" /> Skepdic.com <br><img src="_images/logo_google.gif" alt="Google" width="48" height="19" /> </div> <input type="hidden" name="client" value="pub-3542920166946725" /> <input type="hidden" name="forid" value="1" /> <input type="hidden" name="ie" value="ISO-8859-1" /> <input type="hidden" name="oe" value="ISO-8859-1" /> <input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFFFFF;LBGC:336699;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;" /> <input type="hidden" name="hl" value="en" /> </form> </div> <p align="right"><a href="http://skepdic.com/authorpage.html"><strong>Books by R. T. Carroll </strong></a></p> <p align="right"><a href="http://skepdic.com/authorpage.html"><img src="http://skepdic.com/graphics/CTDcover140.jpg" alt="cover The Critical Thinker's Dictionary" width="140" height="212" /></a></p> <p align="right"><a href="http://www.amazon.com/exec/obidos/ISBN=0471272426/roberttoddcarrolA/"><img src="_images/book_01.jpg" alt="The Skeptic's Dictionary" border="0" width="140" height="171" /></a> <br /> <a href="http://skepdic.com/orderinfo.html">Ordering information</a><br> <br> <br /> <a href="http://skepdic.com/ctinfo.html"> <img src="_images/book_02.jpg" alt="Becoming a Critical Thinker" border="0" width="140" height="171" /></a> </p> <h2 align="center">OTHER LANGUAGES</h2> <ul class="nav_4"> <li><a href="http://nederlands.skepdic.com/">Dutch</a> </li> <li><a href="http://sdkinderen.skepdic.com/">Dutch voor kinderen</a></li> <li><a href="http://dictionnaire.sceptiques.qc.ca/">French</a> </li> <li><a href="http://skepdic.com/German/morgenwelt.html">German</a></li> <li><a href="http://skepdic.gr/">Greek</a></li> <li><a href="http://www.szabadgondolkodo.hu/szkeptikus/szotar/">Hungarian</a></li> <li><a href="http://www.vantru.is/efahyggjuordabokin/">Icelandic</a></li> <li><a href="http://italiano.skepdic.com/">Italian</a></li> <li><a href="http://www.genpaku.org/skepticj/">Japanese</a></li> <li><a href="http://www.rathinker.co.kr/skeptic/index.html">Korean</a></li> <li><a href="http://brazil.skepdic.com/">Portuguese</a></li> <li><a href="http://skepdic.ru/">Russian</a></li> <li><a href="http://slovnik.skepdic.com/">Slovak</a></li> <li><a href="http://skepticus.org/dicc/">Spanish</a></li> <li><a href="http://www.vof.se/skepdic/">Swedish</a></li> <li><a href="http://turkish.skepdic.com/">Turkish</a></li> </ul> <p>Print versions available in <a href="http://www.lannoo.be/content/lannoo/wbnl/listview/1/index.jsp?titelcode=24816&fondsid=8" >Dutch</a>, <a href="http://skepdic.com/russian.html">Russian</a>, <a href="http://www.rakkousha.co.jp/info/index.html#2" > Japanese</a>, and <a href="http://www.aladdin.co.kr/shop/wproduct.aspx?isbn=8995876441" >Korean</a>.</p> </div> </div> </div> <div class="footer"> <a href="pufedorfhoax.html"> <img src="graphics/larrow.gif" alt="larrow.gif (1051 bytes)" border="0" align="bottom" width="14" height="26" /> Pufedorf hoax</a> | <a href="pyramidiocy.html"> <font face="Arial">pyramidiocy <img src="graphics/rarrow.gif" alt="rarrow.gif (1048 bytes)" border="0" align="bottom" width="13" height="26" /></a></div> </div> <div class="bottom"> © Copyright 1994-2016 <a href="http://skepdic.com/refuge/bio.html">Robert T. Carroll</a> - This page was designed by <a href="http://www.cristianpopa.com/portfolio-2/">Cristian Popa</a>.</div> </body> </html>

CINXE.COM

p-value fallacy - The Skeptic's Dictionary