The Skeptic's Dictionary

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" type="text/css" href="../_style/style.css" /> <link rel="shortcut icon" href="../s1.ico"> <script>(function(d, s, id) { var js, fjs = d.getElementsByTagName(s)[0]; if (d.getElementById(id)) return; js = d.createElement(s); js.id = id; js.src = "//connect.facebook.net/en_US/sdk.js#xfbml=1&version=v2.0"; fjs.parentNode.insertBefore(js, fjs); }(document, 'script', 'facebook-jssdk'));</script>  <title>The Skeptic's Dictionary - Skepdic.com</title> </head> <body><br /> <div class="container"> <div class="header"> <div class="logo"> <img src="../_images/logo_01.png" alt="The Skeptic's Dictionary" width="270" height="60" /> <br>by <a href="http://skepdic.com/refuge/bio.html">Robert Todd Carroll</a> © Copyright 1994-2009 </div> <div class="slogan"> <img src="../_images/slogan_01.gif" alt="A Collection of Strange Beliefs, Amusing Deceptions, and Dangerous Delusions" width="362" height="30" /> </div> </div> <div class="menu"> <ul> <li></li><li><a href="http://skepdic.com/">Home</a></li> <li><a href="http://skepdic.com/news/">Newsletter</a></li> <li><a href="http://skepdic.com/feedback.html">Feedback</a></li> <li><a href="http://skepdic.com/faq.html">FAQ & Interviews</a></li> <li><a href="http://skepdic.com/tour.html">Site Map</a></li> </ul> </div> <div class="middle"> <div class="left"> <ul class="nav_1"> <li><a href="http://skepdic.com/tialtmed.html">Alternatives to Medicine</a></li> <li><a href="http://skepdic.com/ticriticalthinking.html">Critical Thinking</a></li> <li><a href="http://skepdic.com/ticrypto.html">Cryptozoology</a></li> <li><a href="http://skepdic.com/tialien.html">ETs & UFOs</a></li> <li><a href="http://skepdic.com/tifraud.html">Frauds, Hoaxes, Conspiracies</a></li> <li><a href="http://skepdic.com/tijunk.html">Junk Science</a></li> <li><a href="http://skepdic.com/tilogic.html">Logic & Perception</a></li> <li><a href="http://skepdic.com/tinewage.html">New Age</a></li> <li><a href="http://skepdic.com/tipara.html">Paranormal</a></li> <li><a href="http://skepdic.com/tiscience.html">Science & Philosophy</a></li> <li><a href="http://skepdic.com/tisuper.html">Supernatural</a></li> <li> </li> </ul> <h2>Other Writings</h2> <ul class="nav_2"> <li><a href="http://sd4kids.skepdic.com/">Mysteries & Science for Kids</a></li> <li><a href="http://skepdic.com/news/">Newsletter</a></li> <li><a href="http://skepdic.com/refuge/essays.html">Skeptical Essays</a></li> <li><a href="http://skepdic.com/refuge/randr.html"> Book Reviews</a></li> <li><a href="http://59ways.blogspot.com/p/index-of-unnatural-acts-that-can.html">Unnatural Acts blog archive</a></li> <li> <a href="http://skepdic.com/skeptimedia/skeptimediaarchive.html"> Skeptimedia</a></li> <li><a href="http://skepdic.com/refuge/funkarchive.html"> Mass Media Funk</a></li> <li><a href="http://skepdic.com/refuge/bunkarchive.html"> Mass Media Bunk</a></li> <li><a href="http://skepdic.com/refuge/harmarchive.html"> What's the Harm?</a></li> <li><a href="http://skepdic.com/refuge/wwwbunk.html"> Internet Bunk</a></li> <li><a href="http://skepdic.com/refuge/2good.html"> Too good to be true</a></li> <li><a href="http://skepdic.com/refuge/studyskills1.pdf">Student Success Guide: Study Skills</a></li> <li><a href="http://skepdic.com/refuge/writingskills.pdf">Student Success Guide: Writing Skills</a></li> <li><a href="http://skepdic.com/refuge/suburbanmyths.html">Suburban Myths</a></li> <li><a href="http://skepdic.com/refuge/memoriam.html"> In Memoriam</a></li><li></li> </ul> <h2>Other Resources</h2> <ul class="nav_2"> <li><a href="http://www.facebook.com/pages/The-Skeptics-Dictionary/195265446870?ref=ts"><img border="0" src="http://skepdic.com/graphics/fbsquare.jpg" alt="facebook" width="36" height="35" /> Skeptic's Dictionary Facebook Page</a></li> <li><a href="http://skepdic.com/refuge/courses.html">For Teachers</a></li> <li><a href="http://skepdic.com/halloween.html">A Skeptic's Halloween</a></li> <li> <a href="http://renish.skepdic.com/AllNotes.html">Editor's Notes</a></li> <li><a href="http://skepdic.com/refuge/getinvolved.html">Get involved</a></li> <li><a href="http://skepdic.com/future.html">Future Topics?</a></li> <li><a href="http://skepdic.com/permissions.html"> Permission to print</a></li> <li><a href="http://skepdic.com/report/">Site Statistics</a> (600,000+ visitors a month)</li><li></li> </ul> </div> <div class="content"> <div class="it"> <h2>From Abracadabra to Zombies<span class="small"> | <a href="http://skepdic.com/contents.html">View All</a></span></h2> <ul class="alphabet"> <li><a href="http://skepdic.com/contents.html#A">a</a></li> <li><a href="http://skepdic.com/contents.html#B">b</a></li> <li><a href="http://skepdic.com/contents.html#C">c</a></li> <li><a href="http://skepdic.com/contents.html#D">d</a></li> <li><a href="http://skepdic.com/contents.html#E">e</a></li> <li><a href="http://skepdic.com/contents.html#F">f</a></li> <li><a href="http://skepdic.com/contents.html#G">g</a></li> <li><a href="http://skepdic.com/contents.html#H">h</a></li> <li><a href="http://skepdic.com/contents.html#I">i</a></li> <li><a href="http://skepdic.com/contents.html#J">j</a></li> <li><a href="http://skepdic.com/contents.html#K">k</a></li> <li><a href="http://skepdic.com/contents.html#L">l</a></li> <li><a href="http://skepdic.com/contents.html#M">m</a></li> <li><a href="http://skepdic.com/contents.html#N">n</a></li> <li><a href="http://skepdic.com/contents.html#O">o</a></li> <li><a href="http://skepdic.com/contents.html#P">p</a></li> <li><a href="http://skepdic.com/contents.html#Q">q</a></li> <li><a href="http://skepdic.com/contents.html#R">r</a></li> <li><a href="http://skepdic.com/contents.html#S">s</a></li> <li><a href="http://skepdic.com/contents.html#T">t</a></li> <li><a href="http://skepdic.com/contents.html#U">u</a></li> <li><a href="http://skepdic.com/contents.html#V">v</a></li> <li><a href="http://skepdic.com/contents.html#W">w</a></li> <li><a href="http://skepdic.com/contents.html#X">x</a></li> <li><a href="http://skepdic.com/contents.html#Y">y</a></li> <li><a href="http://skepdic.com/contents.html#Z">z</a></li> </ul> <WebContent> <h1 style="text-align: center">Critical Thinking and Control Groups</h1> <h4> </h4> <h4><img src="rtc.jpg" alt="Bob Carroll" width="131" height="150" align="right" /></h4> <h4>by Robert T. Carroll<br>  </h4> <p> </p> <p> </p> <p style="text-align: left"><font size="2">(Note: this essay first appeared on the now-defunct Inquiring Minds Newsletter website in the fall of 2005. )</font></p> <p style="text-align: left"><font size="3">Dr. Alan Hirsch claims to be "The World Expert In Smell & Taste." He is an M.D. - a psychiatrist, in fact - who has developed some magical crystals that will "help you reduce your appetite and food cravings." You can read all about his crystals, which he calls SprinkleThin™, on his <a target="_blank" href="http://www.scienceofsmell.com/">website</a>. </font></p> <p style="text-align: left"><font size="3">On July 25, 2005, I found the following testimonial on his website:</font></p> <blockquote> <p><span class="style4"><em><font size="3">Dateline NBC</font></em><font size="3"> Investigates SprinkleThin™ </font></span></p> <p class="style2"><font size="3">“What Dr. Hirsch discovered might surprise you. [Certain smells] seem to control appetite. Dr. Hirsch studied 2,700 people over six months, like the six people we met. They tried just about every diet imaginable. Dr. Hirsch brought along with him these special, non-caloric, scented crystals and asked the six to sprinkle it on their food. </font></p> <p class="style2"><font size="3">All the participants kept a video diary for Dateline to prove they were using the product. At the end of three months when we checked in on them, they were all losing weight.” </font></p> </blockquote> <p style="text-align: left"><font size="3">What is wrong with <em>Dateline</em>'s investigation? Among other things, <em> Dateline</em> did not have a control group. Dr. Hirsch says he has been studying eating behavior and weight loss for 25 years. He says he has done many studies, but if his studies were like <em>Dateline</em>'s study they are not of much scientific value. </font></p> <h3 style="text-align: left"><font size="3">What is a control group and why is having one important? </font></h3> <p style="text-align: left"><font size="3">A well designed study on the diet crystals would use two groups. The group getting the crystals would be called the experimental group. The control group would be a group that, ideally, is identical to the experimental group except that the members of the control group do not use the crystals. </font></p> <p style="text-align: left"><font size="3">The ideal can never be fully achieved with humans, especially when one is doing a study that involves weight loss because: </font></p> <ul> <li> <p style="text-align: left"><font size="3">(a) weight loss is affected by many factors (motivation, eating behavior, amount of activity - especially exercise - overall health, metabolism, stress, and so on.)</font></li> <li> <p style="text-align: left"><font size="3">(b) experimenters can't lock up humans in cages to make sure they do what they're supposed to do for the study. </font></li> </ul> <p style="text-align: left"><font size="3">But, at the very least, a well designed scientific study should use a control group and try to match the members of that group to those in the experimental group for factors that might have a significant effect on the outcome. For example, if you were doing a study that was testing whether prayer has an effect on the longevity of patients dying of AIDS, you should make sure that the ages of the subjects in both groups match up. It would not be a fair study to have 60-year-olds in one group and twenty-somethings in the other group. </font></p> <p style="text-align: left"><font size="3">Having a control group allows the scientist to test a causal hypothesis. In this case, the hypothesis is that SprinkleThin™ is a significant causal factor in producing weight loss. </font></p> <p style="text-align: left"><font size="3">Without a control group, a scientist can't be sure that the diet crystals contributed significantly to the weight loss or, if they did, in what way. The <a href="http://web.archive.org/web/20071124042112/http://skepdic.com/placebo.html" target="_blank"> placebo effect</a> may be at work here: dieters may believe these crystals really affect their sense of taste and smell to such a degree that their appetites are suppressed. They may be deceiving themselves, but the crystals help them anyway. However, powdered beetle dung might have had the same effect. The diet scientist doesn't just want to help people lose weight. If a product works, she wants to know why it works. </font></p> <p style="text-align: left"><em><font size="3">Dateline</font></em><font size="3"> (and Dr. Hirsch) should not just give the crystals to dieters and observe whether they lose weight. They should have a group of similar people who want to lose weight and give them a placebo, a substance that looks like the diet crystals and is ingested in exactly the same way, but which is inert. They should agree to study the two groups for a set length of time, long enough for any diet to show results (several weeks, at least). At the end of the study they would compare the weight loss of the two groups. If the experimental group shows a significantly greater weight loss than the control group, then the scientists have good evidence that the crystals might be effective. There are various reasons why the results of a single study should not be taken as proof of one's causal hypothesis. We'll return to this issue later. </font></p> <p style="text-align: left"><font size="3">Having a control group is necessary but it is not sufficient for having a well-designed control group study. The study must use an adequate number of participants. Six people would not be adequate for a control group study. Several hundred would be a better number. Why? With only six people, all it takes is one participant to do really well to elevate the average of the group significantly above the average of the other group. But this one person's success might be a fluke. By having a larger sample, the researcher reduces the chances that a few fluky individuals have skewed the results. </font></p> <p style="text-align: left"><font size="3">Another way to reduce the chances of fluky results is to randomly assign subjects to the control and experimental groups. Randomization is very important to reduce the chances of biasing the samples. If highly motivated folks are placed in the diet crystal group and a bunch of lazy couch potatoes are in the experimental group, the results of the study would be biased. It is important that a method of true randomization be used, such as a random number table. You might think that assigning all the dark-haired subjects to one group and the light-haired subjects to the other would be sufficient to avoid having biased groups, but you cannot be sure that there is not something about hair color that is related to a person's weight. It is unlikely, but a scientist should not go with hunches in matters such as randomization. </font></p> <p style="text-align: left"><font size="3">It is also important that the subjects in this study not know whether they have been given the magic crystals or the placebo. There is much controversy regarding the ethics of deceiving subjects, but from a scientific point of view it might be better if the subjects didn't even know that the study is about weight loss. If they think, for example, that the study is testing the effectiveness of a new blood pressure medicine, you would eliminate such things as motivation to lose weight or belief that the crystals are appetite suppressants as possible causes of any weight loss achieved. However, many, if not most, scientists argue that it is unethical to deceive participants in scientific studies. The subjects in a study don't need to be told which group they are in, but they should be told that they have been randomly assigned to their group and that at the end of the study they will be told which group they were in. </font></p> <p style="text-align: left"><font size="3">In many studies, not only should the subjects be blind to which group they are in for the duration of the study, but the experimenters should also be blind to which group the subjects have been assigned to. Double-blind studies require at least two experimenters, one who assigns the subjects to their groups and one who keeps track of the data. Had Dr. Hirsch done a double-blind study, an assistant might have randomly assigned the subjects to their groups and kept a record of who is in which group. Dr. Hirsh or another assistant might have then weighed all the subjects and kept weight records for each participant. After all the data had been collected, Dr. Hirsch would then "unblind" the study and the data for the two groups be compared. </font></p> <p style="text-align: left"><font size="3">The final step in a well-designed study is the analysis of the data. You might think that the scientists should be able to look at the results and see right away whether the crystals did any good. This would only be true if, say, there were hundreds in each group and the experimental group lost 50 pounds each on average, while the control group gained 2 pounds. If the study had been designed properly, such results would be extremely unlikely to be a fluke. But what if the experimental group lost 2% more weight than the control group? Would that be significant? To answer that question, scientists revert to statistical formulae. By some formula, a 2% weight loss might be statistically significant. If, however, a 2% weight loss meant 4 ounces over six weeks, most of us would say that even if this is statistically significant it is not important and not worth the money or the risk to use these crystals. The crystals might have some wicked side effect that hasn't yet been discovered. </font> </p> <p style="text-align: left"><font size="3">The moral of this story is that while<a href="http://web.archive.org/web/20071124042112/http://skepdic.com/testimon.html" target="_blank"> testimonials</a> of six people who use crystals and lose weight might have a powerful effect on a television audience, a critical thinker should recognize that without a well-designed control group study, such testimonials do not have much scientific value. </font></p> <p style="text-align: left"><font size="3">A critical thinker also knows that information should be put in the proper context, which requires a certain amount of background knowledge. For example, you should know that many well designed scientific studies get significant results that cannot be replicated at all or in a consistent fashion. If there is a causal relationship between diet crystals and losing weight, it should not work sporadically but consistently, unless, of course, there are so many factors that affect body weight as to make it nearly impossible to isolate the true effectiveness of a single item. A single study, no matter how well designed or how significant the results, rarely justifies drawing strong conclusions about causal relationships. </font></p> <p style="text-align: left"><font size="3">Finally, as mentioned above, there might be some deleterious side effect of these crystals that has not yet been discovered. SprinkleThin™ might help you lose weight but if it kills you in the process, what have you gained? </font></p> <p> </p> <h4>About the Author</h4> <blockquote> <p><strong><font size="2">Robert T. Carroll</font></strong><font size="2"> is a Philosophy professor emeritus at Sacramento City College. He is the founder and creator of the<a target="_blank" href="http://skepdic.com/"> Skeptic's Dictionary web site</a>, an invaluable resource that has been available online since 1994. Bob is also author of <em> <a target="_blank" href="http://skepdic.com/orderinfo.html"> The Skeptic's Dictionary</a></em> (2003) and<em><a target="_blank" href="http://skepdic.com/ctinfo.html"> Becoming a Critical Thinker</a></em> (2005).</font></p> </blockquote> <h2><br> <font face="Arial"><small><small> <strong style="font-weight: 400"><small></webContent><small>Last updated 12/09/10 </small></small></strong></small></small></font> </h2> <p> </div> </div> <div class="right"> <div class="right_it"> <div class="search"> <form method="get" action="http://www.google.com/custom" target="_top"> <input type="hidden" name="domains" value="skepdic.com" /> <input type="text" name="q" maxlength="255" value="" size="20" /> <input type="submit" name="sa" value="Search" /> <br /> <div class="search_2"> <input type="radio" name="sitesearch" value="" /> Web     <input type="radio" name="sitesearch" value="skepdic.com" checked="checked" /> Skepdic.com <img src="../_images/logo_google.gif" alt="Google" width="48" height="19" /> </div> <input type="hidden" name="client" value="pub-3542920166946725" /> <input type="hidden" name="forid" value="1" /> <input type="hidden" name="ie" value="ISO-8859-1" /> <input type="hidden" name="oe" value="ISO-8859-1" /> <input type="hidden" name="cof" value="GALT:#008000;GL:1;DIV:#336699;VLC:663399;AH:center;BGC:FFFFFF;LBGC:336699;ALC:0000FF;LC:0000FF;T:000000;GFNT:0000FF;GIMP:0000FF;FORID:1;" /> <input type="hidden" name="hl" value="en" /> </form> </div> <p align="right"><a href="http://skepdic.com/authorpage.html"><strong>Books by R. T. Carroll </strong></a></p> <p align="right"><a href="http://skepdic.com/authorpage.html"><img src="http://skepdic.com/graphics/CTDcover140.jpg" alt="cover The Critical Thinker's Dictionary" width="140" height="212" /></a></p> <div class="center"> <div align="right"><a href="http://www.amazon.com/exec/obidos/ISBN=0471272426/roberttoddcarrolA/"> <img src="../_images/book_01.jpg" alt="The Skeptic's Dictionary" border="0" width="140" height="171" /></a> <br /> <a href="http://skepdic.com/orderinfo.html">Ordering information</a><br> <br> <br /> <a href="http://skepdic.com/ctinfo.html"> <img src="../_images/book_02.jpg" alt="Becoming a Critical Thinker" border="0" width="140" height="171" /></a> </div> </div> <h2>Other Languages</h2> <ul class="nav_4"> <a href="http://nederlands.skepdic.com/">Dutch</a><br /> <a href="http://sdkinderen.skepdic.com/">Dutch voor kinderen</a> </li> <li> <a href="http://dictionnaire.sceptiques.qc.ca/">French</a> </li> <li> <a href="http://skepdic.com/German/morgenwelt.html">German</a></li> <li> <a href="http://skepdic.gr/">Greek</a> </li> <li> <a href="http://www.szabadgondolkodo.hu/szkeptikus/szotar/">Hungarian</a> </li> <li> <a href="http://www.vantru.is/efahyggjuordabokin/">Icelandic</a> </li> <li> <a href="http://italiano.skepdic.com/">Italian</a> </li> <li> <a href="http://www.genpaku.org/skepticj/">Japanese</a> </li> <li> <a href="http://www.aladin.co.kr/shop/wproduct.aspx?isbn=8995876441">Korean</a> </li> <li> <a href="http://brazil.skepdic.com/">Portuguese</a> <br> <li> <a href="http://skepdic.ru/">Russian</a> </li> <li> <a href="http://slovnik.skepdic.com/">Slovak</a> </li> <li> <a href="http://spanish.skepdic.com/">Spanish</a><br> <a href="http://www.vof.se/skepdic/"> Swedish</a><br> <a href="http://turkish.skepdic.com/">Turkish</a> </li> </ul> <p align="right">Print versions available in <a href="http://www.lannoo.be/content/lannoo/wbnl/listview/1/index.jsp?titelcode=24816&fondsid=8" >Dutch</a>, <a href="http://skepdic.com/russian.html" >Russian</a>, <a href="http://www.rakkousha.co.jp/info/index.html#2" > Japanese</a>, and <a href="http://www.aladdin.co.kr/shop/wproduct.aspx?isbn=8995876441" >Korean</a>.</p></div> </div> </div> <div class="footer"> <span class="quote">Skepticism is a virtue.</span> --Steven Brill </div> </div> <div class="bottom"> This page was designed by <a href="http://www.cristianpopa.com/portfolio-2/">Cristian Popa</a>.</div> </body> </html>

CINXE.COM

The Skeptic's Dictionary - Skepdic.com