How to Spot Pseudoscience
This 15-point checklist will help you tell science from pseudoscience.
You should follow me on twitter here.
| Skeptoid #37 April 06, 2007 Podcast transcript | Listen | Subscribe Also available in Greek |
|
Carl Sagan was one of the first to bring to the mass public a toolkit for deciphering science from pseudoscience when he published Demon-Haunted World in 1996. His "baloney detection kit" was a set of guidelines to help lay people spot logical fallacies and other common flaws in popular pseudoscientific claims. Many other great minds have continued in this tradition, and today we're flush with similar lists. In an effort to increase the confusion, I've taken it upon myself to compile some of the best, mix them all up, pick and choose the best points, and distill them down into a single list. In compiling this list, I owe the biggest debts to Dr. Stephen Barrett of Quackwatch.org, Dr. Tom Perls of AntiAgingQuackery.com, Dr. Michael Shermer of the Skeptics Society, and of course Dr. Carl Sagan. All four of these guys have put in many years of hard work protecting the innocent public from harmful and untruthful pseudoscientific claims. It's no accident that common themes run through all of their work.
I've created a 15-point checklist that I call "How to Spot Pseudoscience." When you hear any claim about a new product, a new discovery, or some paranormal ability, run it through these fifteen questions and you'll get a pretty clear idea of whether or not it has any merit.
1. Does the claim meet the qualifications of a theory?
Very few claims that aren't true actually qualify as theories. Let's review
the four main requirements that a theory must fulfill. 1) A theory must originate
from, and be well supported by, experimental evidence. Anecdotal or unsubstantiated
reports don't qualify. It must be supported by many strands of evidence,
and not just a single foundation. You'll find that most pseudoscience is
supported by only a single foundation. 2) A theory must be specific enough
to be falsifiable by testing. If it cannot be tested or refuted, it can't
qualify as a theory. And if something is truly testable, others must be able
to repeat the tests and get the same results. You'll find that this feature
is truly rare among pseudosciences; they'll generally claim some excuse or
make up a reason why it can't be tested or repeated by others. 3) A theory
must make specific, testable predictions about things not yet observed. 4)
A theory must allow for changes based on the discovery of new evidence. It
must be dynamic, tentative, and correctable. You'll find that most pseudoscience
does not allow for changes based on new discoveries.
2. Is the claim said to be based on ancient knowledge?
This is a sure sign that the claim is not based on scientific evidence, and
it's intended to fool you into thinking that because the ancient Chinese
believed it, it must have merit. In fact many true theories are not very
old at all, because they've replaced older theories as knowledge has increased.
Generally, the more recent the evidence, the better scientific foundation
it has.
3. Was the claim first announced through mass media, or through scientific
channels?
Real discoveries go through an unbiased peer review process, which results
in publication through scientific journals. When a belief is first announced
through the mass media, like Pons and Fleischman's cold fusion experiments
or like the Stoern perpetual motion machine, there's generally a reason its
proponents chose not to subject it to the scrutiny of peer review.
4. Is the claim based on the existence of an unknown form of "energy" or
other paranormal phenomenon?
Loose, meaningless usage of a scientific-sounding word like "energy" is
one of the most common red flags you'll see on popular pseudoscience. Terms
like energy fields, negative energy, chi, orgone, aura, psi, and trans-dimensional
energy are utterly meaningless in any scientific context. Approach with extreme
caution.
5. Do the claimants state that their claim is being suppressed by
authorities?
This is usually a really frail excuse for why mainstream scientists don't take
their claim seriously, why the product is not approved by the FDA, or why scientific
journals won't publish their articles. You'll often hear this in the form of
a conspiracy of the medical establishment to suppress a quack cure because
it's in the interest of the medical industry to keep you sick. In fact, any
doctor or pharmaceutical company that could develop a new cure would make a
huge fortune; they'd never suppress it. The same goes for auto manufacturers
worldwide who are said to be "suppressing" new efficient engine technologies.
6. Does the claim sound far fetched, or too good to be true?
When something sounds too good to be true, it usually is. Extraordinary claims
require extraordinary evidence. Does the claim truly fit in with what we
know of the way the world works? How often do claims that turn the world
upside down really turn out to be true? Approach such claims with extreme
skepticism, and demand evidence that's as extraordinary as the claim.
7. Is the claim supported by hokey marketing?
Be wary of marketing gimmicks, and keep in mind that marketing gimmicks are,
by themselves, completely worthless. Examples of hokey marketing that should
always raise a red flag are pictures of people wearing white lab coats, celebrity
endorsements, anecdotes and testimonials from any source, and mentions of
certifications, colleges, academies, and institutes.
8. Does the claim pass the Occam's Razor test?
Is there a simpler, natural explanation for the claim that does not require
any supernatural component? Are results consistent with the placebo effect
or the body's natural healing capacity? Can a stage magician duplicate the
psychic's feats? The Law of Large Numbers states that a one-in-a-million
event usually happens to everyone about once a month, and since Occam's Razor
says that the simpler of two possible explanations is usually the right one,
don't leap for a supernatural explanation just because you happened to dream
about your grandmother on the night she died.
9. Does the claim come from a source dedicated to supporting it?
Science works by starting with a null hypothesis and searching for evidence.
Pseudoscience starts with a positive hypothesis and supports it with questionable
research and anecdotal reasoning. It's unlikely that an institution dedicated
to the promotion of any given claim will present any type of evidence other
than that which supports their claim, and its bias should be given serious
consideration.
10. Are the claimants up front about their testing?
Any good research will outline the testing that was done, and will present
all evidence that did not support the conclusion. Be skeptical of any claims
that do not detail testing methodology that was thorough and responsible,
including external verification and duplication, or that do not provide evidence
unsupportive of the conclusion.
11. How good is the quality of data supporting the claim?
Watch out when testing data might be susceptible to observational selection,
which is the counting of hits and not the misses, like we see with television
psychics. Watch out when sample sizes are too small to have statistical significance,
as with most clinical trials of homeopathy. And especially watch out for
hastily drawn causal relationships: the assumption that because the relief
occurred after the remedy, the remedy must have caused the relief.
12. Do the claimants have legitimate credentials?
Be aware that there is a huge number of unaccredited institutions (which are
often just bedroom offices) giving out degrees in just about anything. Be
aware that some institutions claiming to be accredited received their accreditation
from unrecognized accreditation bodies. Finally, be aware that genuine accredited
universities often have programs in unscientific fields such as chiropractic,
naturopathy, and acupuncture. You must be vigilant. To see just how vigilant,
go to ThunderwoodCollege.com and
get your own Ph.D. in the field of your choice in seconds, for free.
13. Do the claimants state that there's something wrong with the norm?
When real research is presented, it consists of the evidence that was discovered
and the conclusion. It does not go off on alarmist rants about how the food
we eat is dangerous, how we're destroying the planet, how the government
covers up its evils, or how you're going to hell if you accept evolution.
When a claim is presented as an alternative to the wrongs of the status quo,
it's a sign that the claim is probably based on ideology or philosophy rather
than science.
14. Is the claim said to be "all natural"?
As we've see time and time again, by no definition can "all natural" mean
that a product is safe or healthy. Consider the examples of hemlock, mercury,
lead, toadstools, box jellyfish neurotoxin, asbestos — not to mention
a nearly infinite number of toxic bacteria and viruses (E. coli, salmonella,
bubonic plague, smallpox). In many cases, synthetic versions of natural compounds
have been engineered to make them safer, more effective, and able to be produced
in large quantities.
15. Does the claim have support that is political, ideological, or
cultural?
Some claimants suggest that it's moral, ethical, or politically correct to
accept their claims, to redirect your attention from the fact that they may
not be scientifically sound. In some cases, such as creationism, proponents
use the court system to force schools to teach their claims as fact. Generally,
when a theory is scientifically sound, even if it's brand new it will eventually
find its way into the educational curriculum. Good science is done in the lab — not
in the courts, not in protest marches, not in blogs, and not in church. A political
or cultural campaign to legalize or promote some product or claim is a major
indicator that it's bogus.
There you have it. With this checklist, anyone is well equipped to filter out the chaff from the wheat. Questions like these are what should be taught in schools, encouraging young people to begin looking at all the crazy misinformation in our world with critical analysis. The ability to tell fact from fiction is essential to our progress as a species as we search for the next great discoveries in medicine, space exploration, computing, power generation, and every other scientific field.
You should follow me on twitter here.
Drop a tip in my tip jar? or become a supporter
© 2007 Skeptoid.com
References & Further Reading
Gardner, Martin. Science: Good, Bad and Bogus. Amherst, New York: Prometheus Books, 1989.
Hines, Terrence. Pseudoscience and the Paranormal. Amherst, NY: Prometheus Books, 2003.
Randi, James. Flim-Flam! Psychics, ESP, Unicorns, and Other Delusions. Amherst, New York: Prometheus Books, 1982.
Sagan, C., Druyan, A. The Demon-Haunted World. London, UK: Headline Book Publishing, 1996. 189-206.
Shermer, Michael. Why People Believe Weird Things: Pseudoscience, Superstition, and Other Confusions of Our Time. New York: Henry Holt and Company, LLC, 1997. 63-123.
Reference this article:
Dunning, Brian.
"How to Spot Pseudoscience." Skeptoid Podcast. Skeptoid Media, Inc.,
6 Apr 2007. Web.
11 Mar 2010. <http://skeptoid.com/episodes/4037>
Discuss!
5 most recent comments | Show all 76 comments
Remember, you should always read with skepticism the comments of anyone too lame to put their real name & city.
This is a useful list.
I think many people who participate in the internet do not know some of the terminology or concepts used in the list. The effectiveness of this list (getting people to be less gullible )could be improved by explaining things at a lower level.
IMHO, having seen how people react to such lists it could be helpful to emphasize that the items on the list are not smoking guns, or laws.
Only warning signs of potential BS. Such an emphasis can help the list avoid seeming dogmatic.
beforewisdom, Washingtong, D.C.
September 06, 2009 7:17am
To add to the "toolbox" of bias-detection indicators, here are two I use:
1) When a perspective is presented with a tone of fear or great urgency, warning lights should go off. Example: government vaccine conspiracy assertions.
2) Anytime someone starts out saying "I find it interesting..." or "It's curious that..." or other similar phrase, it almost always means they are mad at you, and have quite likely compromised scientific detachment. They say "interesting" or "curious" or another similar word to try to disguise or balance the anger they have.
(An aside: Finding something to be "interesting" when you are really just mad reminds me of calling problems "issues." While calling problems "issues" does establish a clinical, objective tone to solving problems, it is also a way to try to avoid the grim meathook reality in many cases that if the problem is not solved, you will be fired or dumped or painfully ostracized or whatever.
I digress.)
Aloysius, Santa Cruz, CA
October 03, 2009 10:51am
Just wanted to comment on Eric from Oklahoma City's bit there on tests 5 and 9. Clearly this information wasn't available at the time but it's now been shown that some studies that had run contrary to the scientific consensus were actively rejected from peer-reviewed journals for political, not scientific, reasons. The whole ordeal has been nicknamed "Climategate."
Also, it's interesting to compare the "theory" of anthropogenic global warming against this list. I believe that most unbiased readers will find it fails nearly every test listed here.
Tim Wickstrom, Winnipeg, MB
December 17, 2009 12:41pm
One question... How does mention of "certifications, colleges, academies, and institutes." equate to hokey marketing if these are legit certifications, colleges, etc..
Personally I'm opening my own clinic and research center. I advertise that I'm a board member of my professional association, I have a degree from one of the top universities in the country and further diplomas and certifications in my field of expertise. I encourage anyone to check out those if they are skeptical of their credibility, but simply advertising my achievements doesn't make my clinic Hokey. Marketing yourself doesn't lessen your service. Seriously, no one would survive in business without some initial marketing to develop word of mouth references, or other medical referals. AS well just because your selling a service or product doesn't mean your out to pull the wool over anyone's eyes. We all have to pay bills, and the only way to do so is selling our services or products (whether as a self-employed person to a market, or as an employed person to their employer.). Be sketical of claims but if someone can back them up, you can't discredit them based on silly notions that someone else who was a fake used the same marketing tactic. Everyone would be discredited on that alone.
Josanne, Canada
January 19, 2010 9:22pm
Make a comment about this episode of Skeptoid (please try to keep it brief & to the point). Anyone can post:
You can also discuss this episode in the Skeptoid Forum, hosted by the James Randi Educational Foundation.
Join the Skeptalk email discussion list.
What's the most important thing about Skeptoid?
![[Valid RSS]](/images/valid-rss.png "Validate my RSS feed"){kind=small}
It is interesting that, just two episodes after this, you support the anti-anthropogenic global warming (anti AGW) crowd and make them seem as though they are not on the fringe.
Test 5: How often do we hear these conspiracies that anti-AGW promoters babble about. In fact, Climate Audit does precisely this. It spreads around non-peer reviewed claims about peer reviewed articles in order to promote the idea of a "scientific conspiracy." No wonder these anti-AGW guys tend to be friends with Creationists and IDers (ex. Discovery Institute and AAPS).
Test 9: Looking at all the comments on episode 39, i see that all the anti-AGW crowd get their data from anti-AGW sources and many times advertise them. Other than unbiased peer reviewed journals (which all side with AGW, check!), if u want to look at these sides, look at both sides and weight their sources. AGW sites (realclimate and cliamteskeptic) all directly source peer reviewed articles while others (climateaudit and GlobalWarming.org) use either original non-peer reviewed analysis (unverified by experts and highly likely BS) or source questionable biased sources that arent listed in peer reviewed search engines (ex Energy and Enviornment).
Test 15: Obviously, there is no doubt the anti-AGW focuses on politicians (ex al gore) and not science.
Read my comment for episode 39. Check both sides and see what side fails what tests. Anti-AGW is sold alongside tons of other anti-science positions (ex: creationism anti vaccines).
Eric, Oklahoma City
July 26, 2009 11:18pm