How to Spot Pseudoscience

This 15-point checklist will help you tell science from pseudoscience.

by Brian Dunning

Filed under Logic & Persuasion

Skeptoid #37
April 6, 2007
Podcast transcript | Listen | Subscribe
Also available in Greek
 

Carl Sagan was one of the first to bring to the mass public a toolkit for deciphering science from pseudoscience when he published Demon-Haunted World in 1996. His "baloney detection kit" was a set of guidelines to help lay people spot logical fallacies and other common flaws in popular pseudoscientific claims. Many other great minds have continued in this tradition, and today we're flush with similar lists. In an effort to increase the confusion, I've taken it upon myself to compile some of the best, mix them all up, pick and choose the best points, and distill them down into a single list. In compiling this list, I owe the biggest debts to Dr. Stephen Barrett of Quackwatch.org, Dr. Tom Perls of AntiAgingQuackery.com, Dr. Michael Shermer of the Skeptics Society, and of course Dr. Carl Sagan. All four of these guys have put in many years of hard work protecting the innocent public from harmful and untruthful pseudoscientific claims. It's no accident that common themes run through all of their work.

I've created a 15-point checklist that I call "How to Spot Pseudoscience." When you hear any claim about a new product, a new discovery, or some paranormal ability, run it through these fifteen questions and you'll get a pretty clear idea of whether or not it has any merit.

1. Does the claim meet the qualifications of a theory?
Very few claims that aren't true actually qualify as theories. Let's review the four main requirements that a theory must fulfill. 1) A theory must originate from, and be well supported by, experimental evidence. Anecdotal or unsubstantiated reports don't qualify. It must be supported by many strands of evidence, and not just a single foundation. You'll find that most pseudoscience is supported by only a single foundation. 2) A theory must be specific enough to be falsifiable by testing. If it cannot be tested or refuted, it can't qualify as a theory. And if something is truly testable, others must be able to repeat the tests and get the same results. You'll find that this feature is truly rare among pseudosciences; they'll generally claim some excuse or make up a reason why it can't be tested or repeated by others. 3) A theory must make specific, testable predictions about things not yet observed. 4) A theory must allow for changes based on the discovery of new evidence. It must be dynamic, tentative, and correctable. You'll find that most pseudoscience does not allow for changes based on new discoveries.

2. Is the claim said to be based on ancient knowledge?
This is a sure sign that the claim is not based on scientific evidence, and it's intended to fool you into thinking that because the ancient Chinese believed it, it must have merit. In fact many true theories are not very old at all, because they've replaced older theories as knowledge has increased. Generally, the more recent the evidence, the better scientific foundation it has.

3. Was the claim first announced through mass media, or through scientific channels?
Real discoveries go through an unbiased peer review process, which results in publication through scientific journals. When a belief is first announced through the mass media, like Pons and Fleischman's cold fusion experiments or like the Steorn Orbo perpetual motion machine, there's generally a reason its proponents chose not to subject it to the scrutiny of peer review.

4. Is the claim based on the existence of an unknown form of "energy" or other paranormal phenomenon?
Loose, meaningless usage of a scientific-sounding word like "energy" is one of the most common red flags you'll see on popular pseudoscience. Terms like energy fields, negative energy, chi, orgone, aura, psi, and trans-dimensional energy are utterly meaningless in any scientific context. Approach with extreme caution.

5. Do the claimants state that their claim is being suppressed by authorities?
This is usually a really frail excuse for why mainstream scientists don't take their claim seriously, why the product is not approved by the FDA, or why scientific journals won't publish their articles. You'll often hear this in the form of a conspiracy of the medical establishment to suppress a quack cure because it's in the interest of the medical industry to keep you sick. In fact, any doctor or pharmaceutical company that could develop a new cure would make a huge fortune; they'd never suppress it. The same goes for auto manufacturers worldwide who are said to be "suppressing" new efficient engine technologies.

6. Does the claim sound far fetched, or too good to be true?
When something sounds too good to be true, it usually is. Extraordinary claims require extraordinary evidence. Does the claim truly fit in with what we know of the way the world works? How often do claims that turn the world upside down really turn out to be true? Approach such claims with extreme skepticism, and demand evidence that's as extraordinary as the claim.

7. Is the claim supported by hokey marketing?
Be wary of marketing gimmicks, and keep in mind that marketing gimmicks are, by themselves, completely worthless. Examples of hokey marketing that should always raise a red flag are pictures of people wearing white lab coats, celebrity endorsements, anecdotes and testimonials from any source, and mentions of certifications, colleges, academies, and institutes.

8. Does the claim pass the Occam's Razor test?
Is there a simpler, natural explanation for the claim that does not require any supernatural component? Are results consistent with the placebo effect or the body's natural healing capacity? Can a stage magician duplicate the psychic's feats? The Law of Large Numbers states that a one-in-a-million event usually happens to everyone about once a month, and since Occam's Razor says that the simpler of two possible explanations is usually the right one, don't leap for a supernatural explanation just because you happened to dream about your grandmother on the night she died.

9. Does the claim come from a source dedicated to supporting it?
Science works by starting with a null hypothesis and searching for evidence. Pseudoscience starts with a positive hypothesis and supports it with questionable research and anecdotal reasoning. It's unlikely that an institution dedicated to the promotion of any given claim will present any type of evidence other than that which supports their claim, and its bias should be given serious consideration.

10. Are the claimants up front about their testing?
Any good research will outline the testing that was done, and will present all evidence that did not support the conclusion. Be skeptical of any claims that do not detail testing methodology that was thorough and responsible, including external verification and duplication, or that do not provide evidence unsupportive of the conclusion.

11. How good is the quality of data supporting the claim?
Watch out when testing data might be susceptible to observational selection, which is the counting of hits and not the misses, like we see with television psychics. Watch out when sample sizes are too small to have statistical significance, as with most clinical trials of homeopathy. And especially watch out for hastily drawn causal relationships: the assumption that because the relief occurred after the remedy, the remedy must have caused the relief.

12. Do the claimants have legitimate credentials?
Be aware that there is a huge number of unaccredited institutions (which are often just bedroom offices) giving out degrees in just about anything. Be aware that some institutions claiming to be accredited received their accreditation from unrecognized accreditation bodies. Finally, be aware that genuine accredited universities often have programs in unscientific fields such as chiropractic, naturopathy, and acupuncture. You must be vigilant. To see just how vigilant, go to ThunderwoodCollege.com and get your own Ph.D. in the field of your choice in seconds, for free.

13. Do the claimants state that there's something wrong with the norm?
When real research is presented, it consists of the evidence that was discovered and the conclusion. It does not go off on alarmist rants about how the food we eat is dangerous, how we're destroying the planet, how the government covers up its evils, or how you're going to hell if you accept evolution. When a claim is presented as an alternative to the wrongs of the status quo, it's a sign that the claim is probably based on ideology or philosophy rather than science.

Tip Skeptoid $2/mo $5/mo $10/mo One time

14. Is the claim said to be "all natural"?
As we've see time and time again, by no definition can "all natural" mean that a product is safe or healthy. Consider the examples of hemlock, mercury, lead, toadstools, box jellyfish neurotoxin, asbestos — not to mention a nearly infinite number of toxic bacteria and viruses (E. coli, salmonella, bubonic plague, smallpox). In many cases, synthetic versions of natural compounds have been engineered to make them safer, more effective, and able to be produced in large quantities.

15. Does the claim have support that is political, ideological, or cultural?
Some claimants suggest that it's moral, ethical, or politically correct to accept their claims, to redirect your attention from the fact that they may not be scientifically sound. In some cases, such as Young Earth Creationism, proponents use the court system to force schools to teach their claims as fact. Generally, when a theory is scientifically sound, even if it's brand new it will eventually find its way into the educational curriculum. Good science is done in the lab — not in the courts, not in protest marches, not in blogs, and not in church. A political or cultural campaign to legalize or promote some product or claim is a major indicator that it's bogus.

There you have it. With this checklist, anyone is well equipped to filter out the chaff from the wheat. Questions like these are what should be taught in schools, encouraging young people to begin looking at all the crazy misinformation in our world with critical analysis. The ability to tell fact from fiction is essential to our progress as a species as we search for the next great discoveries in medicine, space exploration, computing, power generation, and every other scientific field.

Brian Dunning

© 2007 Skeptoid Media, Inc. Copyright information

References & Further Reading

Gardner, Martin. Science: Good, Bad and Bogus. Amherst, New York: Prometheus Books, 1989.

Hines, Terrence. Pseudoscience and the Paranormal. Amherst, NY: Prometheus Books, 2003.

Randi, James. Flim-Flam! Psychics, ESP, Unicorns, and Other Delusions. Amherst, New York: Prometheus Books, 1982.

Sagan, C., Druyan, A. The Demon-Haunted World. London, UK: Headline Book Publishing, 1996. 189-206.

Shermer, Michael. Why People Believe Weird Things: Pseudoscience, Superstition, and Other Confusions of Our Time. New York: Henry Holt and Company, LLC, 1997. 63-123.

Reference this article:
Dunning, B. "How to Spot Pseudoscience." Skeptoid Podcast. Skeptoid Media, Inc., 6 Apr 2007. Web. 2 Sep 2014. <http://skeptoid.com/episodes/4037>

Discuss!

10 most recent comments | Show all 110 comments

Brian, if you are alluding to a text called "dragons of Eden" please appropriately identify it...

I am sick of chasing th myriad of biblical criticism texts that are identified in paasing.

Yes, all biblicsl critique of historical verification is worth noticing in the first book of the anthology we call "the bible". Obviously, archaeology and textual criticism allows for comparative literature of the writing period.
Still an article of dragons in eden still refers extra textually to dragons and not Eden..

As far as I can see.. dragons or all their ilk belong in the rest of the Torah and Psalms...

Sure.. a seraph may be the wisest animal in the world... but only by extrapolation.

Sure.. a snake may be the wisest animal in the world... but only by interpolstion by passing statement.

Mud, Virtually Missing point, NSW, Oz
December 21, 2012 9:48pm

Having said all that..If dragons in eden is a Brian docco or skeptoid.. please appropriate it correctly...

Religion is a form of culture expressed as an atheism...

I require something better than that quip.. even if it were a Brian Quip!

Mud, missing point, NSW, Oz
December 21, 2012 9:53pm

the signature that certified my Thunderwood Degree looked very familiar..

Boognish, Mexico, China
March 15, 2013 7:51am

Typical, the new NAFTA rep complaining that his diploma and 1000,000,000 peso note has the same signature..

Mud, sin city, Oz
April 9, 2013 10:05pm

Thanks for the Ph.D in Feng Shui, Brian!

Yabeen Sees, Tooleedoo
April 18, 2013 8:44am

If anyone had a PhD in Feng Shui it would be on the tradition around it..

Maybe a D.Lit but I can see a faculty awarding that..

To date, marking skeptoid? Sure, I'd give Brian a masters if he provided a reliable thesis.

Magnanamous Dinoflagellate, sin city, Oz
July 5, 2013 3:18am

OMG, Sagan was this flawed in his logic?? Most of these points are riddled with the most preposterous assumptions.
It would take an essay to tear this crap apart. Not a very difficult one, but I have more important things to do. Sad for the people who fall for this. Science has become the new religion. Absolute knowledge in a relative universe? Good luck.

Lex, SF
September 13, 2013 6:12pm

Lawrence Krauss is a MUCH better source of scientific information than EITHER Carl Sagan or Karl Popper.... Sagan and Popper are outdated.

Science has moved on from their pseudo-logic. We now know that falsification is NOT a very accurate picture of how scientific knowledge ios built. Also, there is far more to life than science.
Matters science cannot investigate are STILL WORTHWHILE to look into- Dr. Krauss himself admits this. Science is NOT everything- another thing Dr. Krauss went out of his way to mention. J. Robert Oppenheimer acknolwdged this too.

The only one "full of baloney" is Carl Sagan himself.....
Jane

Michael Fedorski, La Jolla, California
October 14, 2013 1:22pm

Now what internet garbage dump did you get that from Michael?

Lawrence Krauss is a profound science communicator/ researcher and Sagan was a profound science communicator/researcher.

Sagan had his science fiction models as devices for teaching, Krauss still uses startrek at times.

To compare the two profound contributions in your fashion is a bit parsimonious with the facts and a stance on some bizarre parochialism.

Maybe one could say Sagan operated in a world where science was beyond most because people didnt have the facilities to education in science.

But then, Krauss operates in a world where everyone has the opportunity towards a scientific education but chooses to do something a lot less worthwhile.

Both have had to teach a lot of people who self impose their own limits and relish in ignorance.

Both have been fantastic and both we can use their teachings to exite the young to a good balance of science in their lives when it comes to lifes expectations.

After all, its religion, flattery (ie alternative modalities) and ignorance that will tell you this earthly life is insignificant or irrelevant.

Thats woo..

Magnificent Devon, Ich bin ein Berliner?, Pho's diner, greenacres by the sea, Oz
January 1, 2014 9:33pm

Dear Brian,

I'm a bit late to the discussion, seeing as how this has been posted seven years ago, but there is one thing that I can't let pass, and this is the mention of negative energy.

Now, I'll be the first to agree that it's probably woo if you use "negative energy" to describe emotions, but in physics, there's nothing mysterious about negative energy.

Gravitational potential energy is always measured with respect to a reference level, which is arbitrarily set to zero. Depending on where you are w.r.t. that level, your gravitational potential energy can be zero, greater than zero or less than zero. (In rocket physics, that level is set at infinity, so your gravitational potential energy is *always* less than zero.)

The reason why the absolute value of your gravitational potential energy doesn't matter is that all that matters is the work you do when going from one place to another, and that is calculated using the *difference* in potential energies. Therefore it doesn't matter if the energy at place A is -5 and at B it's 6, or if at A it's 1 and at B it's 12, as long as the differences 6 - (-5) and 12 - 1 (or of course -5 - 6 and 1 - 12) are the same.

This is not the first time you slip up at physics. Why not get a physicist to check what you write? (Not me though, I'm not a physicist.) Otherwise a splendid podcast, as always.

Best,

Stephan

Stephan, Zurich
August 7, 2014 6:44am

Make a comment about this episode of Skeptoid (please try to keep it brief & to the point).

Post a reply

 

What's the most important thing about Skeptoid?

Support Skeptoid
 
Skeptoid host, Brian Dunning
Skeptoid is hosted
and produced by
Brian Dunning


Newest
Harry Houdini and Sir Arthur Conan Doyle
Skeptoid #430, Sep 2 2014
Read | Listen (13:30)
 
The Many Voices of Frank's Box
Skeptoid #429, Aug 26 2014
Read | Listen (13:31)
 
The Haunted Dybbuk Box
Skeptoid #428, Aug 19 2014
Read | Listen (11:26)
 
The Legend of the Flying Dutchman
Skeptoid #427, Aug 12 2014
Read | Listen (11:49)
 
The Baldoon Mystery
Skeptoid #426, Aug 5 2014
Read | Listen (11:44)
 
Newest
#1 -
The JFK Assassination
Read | Listen
#2 -
Asking the Socratic Questions
Read | Listen
#3 -
5 False Arguments for Raw Milk
Read | Listen
#4 -
Fukushima vs Chernobyl vs Three Mile Island
Read | Listen
#5 -
The Riddle of the L-8 Blimp
Read | Listen
#6 -
Who Discovered the New World?
Read | Listen
#7 -
Listeners Have Another Say
Read | Listen
#8 -
6 Problems with Wind Turbine Syndrome
Read | Listen

Recent Comments...

[Valid RSS]

  Skeptoid PodcastSkeptoid on Facebook   Skeptoid on Twitter   Brian Dunning on Google+   Skeptoid on Stitcher   Skeptoid RSS

Members Portal

 
 


Follow @BrianDunning

Tweets about "skeptoid"

Support Skeptoid

Name/Nickname:  
City/Location:
Email: [Why do we need this?]To reduce spam, we email new faces a confirmation link you must click before your comment will appear.
Comment:
characters left. Abusive posts and spam will be deleted.