Approaching a Subject Skeptically

My process for examining a new topic, to learn whether it's fact or fiction.

by Brian Dunning

Filed under Feedback & Questions, Logic & Persuasion

Skeptoid #290
December 27, 2011
Podcast transcript | Listen | Subscribe
 

One of the questions I get asked a lot is how I go about approaching a new subject. When you hear about something new, what's the best way to think about it? What's the best way to determine whether it's science or pseudoscience? Well, I'm not sure that there is a "best" way, and I don't think there's one methodology that everyone can follow that's going to work in every circumstance, but I'll try to give the best answer I can. It's probably not the same answer you'd hear from others, but this is what works for me.

First of all, and perhaps most important, is that there's a separation between my daily life and working on Skeptoid. I don't walk around demanding peer-reviewed scientific evidence for everything that I see. I don't have a crazed, obsessive drive to know the validity of every new product for sale at the mall. I'd never get through my day without a certain amount of tolerance for pseudoscience. Fad products, marketing campaigns, greenwashing, and even straight-up fraudulent claims surround us, all day, every day. I accept that. Trying to be a full-time challenger of pseudoscience would not only be hopelessly quixotic, it would also annoy everyone around me, and rob me of the freedom to enjoy my day.

So I let virtually everything slide. A coworker is wearing a magic bracelet? Great, good for him. Neighbor talks about her great visit to the reflexologist? Bully for her. Overhear some people discussing what Nostradamus said about the 2012 apocalypse? Whatever floats their boat.

But what if I'm out with friends and somebody asks me my thoughts on something? This happens all the time. You're on the spot, you don't have access to research materials, you don't have time to look into it. Now, oftentimes I've already done an episode about the subject in question, or something really similar, that gives me a pretty good foundation. Sometimes I haven't, and like most people, have to rely on a journeyman's knowledge of a subject area that's outside of my core competence. This provides a pretty good overview of whether or not the new claim is in line with what's generally known about the subject. Usually it's not; otherwise it wouldn't be on the news or wherever it was that my friends heard about it.

So there you are. You're given something that raises your skeptical radar, it's outside your core competence, and your friends just saw it on television or the Internet. Despite the fact that most people say they take TV or Internet reports with a grain of salt, few actually do. There's something deeply compelling about hearing a claim from an authoritative source; we all have a voice in the back of our heads that wants the new claim to be true, and this desire gets confirmed by the belief that the story wouldn't have made it all the way to the TV news without having been pretty well substantiated. What are you going to do?

The first thing I'd do is take out my phone and track down the original source of the story, using keywords from the report to search Google. I'd want to know if it was reported in any journals, or if it skipped this process and went straight to the mass media. This is the simplest and fastest way to see if a new claim or phenomenon has come from the world of legitimate research, or if it comes from a crank, charlatan, or manufacturer operating outside of science. You always have to remember that the mass media doesn't care; they're interested in the sensationalism of the story, not in its validity.

That's it. That's probably all I'm going to do when I'm out in the world and get a question that's worthy of looking into. It's not a perfect process, but nine times out of ten this will correctly tell you whether there's something there, or whether it's just more noise from media clamoring for eyeball share.

It's only when I take my seat in the Skeptoid office that I assume the mantle of proper separator of fact and fiction. This is when I take each week's topic and give it my honest best effort at a good skeptical treatment. The best topics are those that are popularly misunderstood, but with facts behind them that, when properly understood, are way cooler than the popular version. This isn't as hard as it might sound; nearly every popular myth has some history that puts its genesis into a fascinating new perspective.

Sometimes finding this perspective takes me back in time, to an out-of-print book, or to a newspaper article a century old. Tracking these down requires a lot of eBook purchases, Google Books downloads, newspaper archive searches, and occasionally even the coveted trip to a real library to find a real book. Of course, even the relevant pages from the real book end up as electronic files on my computer, photographed with the iPhone and then OCR converted to searchable text. Getting brand new information, like current research, is almost exactly the same process; it's all available when you have the right accounts to access online research libraries. But none of that compares to the few chances to actually go in person to a place where something strange is said to have happened: to smell the dusty desert wind across Death Valley's Racetrack Playa, to touch the cold granite of the Georgia Guidestones, and to photograph a Fata Morgana mirage such as the ones responsible for so many legendary ghost lights.

I've been doing this show every week for five years now, and on the one hand, you might assume that I've developed a certain aptitude for smelling rats, and have pretty good radar for science vs. pseudoscience. That's true to a degree; but at the same time, I've learned that I can easily be surprised. I often learn that something that sounded pretty hokey is actually true, and something I took for granted turns out to be false. So rather than having developed a supersense for fact and fiction, I've actually picked up a more acute awareness of my own ignorance. Kind of the opposite of what one might hope for; but as we see so often, magically easy solutions to complex problems are a fool's gold.

The process is different every time, but it always starts with a quick survey of the most popular sources, followed by delving deeper into the roots. If it's homeopathy, I want to know what led Samuel Hahnemann to his original conclusions. If it's a conspiracy theory, I want to know who came up with it and what question they were trying to answer. If it's a ghost story, I want to know who first wrote about it and what their relationship was to the hauntee. It's critical to allow for the possibility that the story may or may not be as reported; and to follow up the leads in both directions. Frequently this requires some pretty detailed departure from the popularly known core of the story.

For example, say you find a reference to the mayor of an old town. First you find out if the town actually exists, where it was, whether it's still there, and find it on Google Earth to see if it makes sense within the context of the story. Then find out if the person listed as the mayor actually was the mayor. Find out when he was born, see if the timing is right. There are myriad details you can drill down through, to be as thorough as possible validating the story. Sometimes there are an endless number of these leads, and with only a week between episodes, I often have to simply stop following them, thus making many episodes necessarily incomplete and open to error.

But when you have the time, how far do you go tracking these leads? I've found that there's never a point of diminishing returns. Every time I've made a discovery or connection that (to my knowledge) no other researcher has found, it's always in one of these fine tails of data. The unturned stones are rarely in the middle of the road most traveled. They're in the obscure newspaper article that never got syndicated; they're in the out-of-print interview with the expert who was misquoted in the popular version of the story; and more than anywhere else, they're in the actual published research that was omitted from the mass media reports because it did not support a sensational revisioning of the story.

I don't mean to sound cynical about the mass media. There are many, many excellent reporters and news bureaus who conscientiously produce exceptional material. But I think you'll find that the better they are, the more likely they are to give you an honest assessment of the industry's overall goal, which is to be profitable. The easiest way to do this, as practiced by a probable majority of editors, is to be sensational. I don't think it's a cynical assessment, and it has certainly proven itself to me time and time again through my work validating mass media reports.

So take the road most traveled, as presented in Wikipedia, to get the lay of the land. But to truly learn anything new, you must explore those obscure details that nobody else had time for, or that they overlooked.

$2/mo $5/mo $10/mo One time

Interestingly, I'd say that my process — though it's much more thorough — is probably no more accurate than the quick trick in the restaurant with a smartphone and Google. The more information I collect, the more possibility for error. The more obscure threads I follow, the more are likely to be unreliable. And the more time I spend trying to be thorough on one part of the story, the less time I have for the other parts: an unfortunate exigency of producing a weekly show. I'd say that errors of omission are my most common mistakes, followed by errors that I just didn't catch because of limited time. And like every fallible biological entity, I also make errors by misinterpreting, misreading, and failing to see beyond my own personal biases.

You'll make these same errors in your own research. The best defense against them is to acknowledge your blind spots, compensate for them, and honestly qualify remarks that you can't be sure of. First I try to be right more often than I'm wrong, but second I try to emphasize the process over the conclusions. Being right nine times doesn't guarantee that you'll be right the tenth time, but trying hard all ten times guarantees that you'll at least be as right as your process is capable of.

Brian Dunning

© 2011 Skeptoid Media, Inc. Copyright information

References & Further Reading

Nickell, J. Real or Fake: Studies in Authentication. Lexington: University Press of Kentucky, 2009.

Plait, P. Bad Astronomy: Misconceptions and Misuses Revealed, from Astrology to the Moon Landing Hoax. New York: Wiley, 2002.

Radford, B. Media Mythmakers: How Journalists, Activists, and Advertisers Mislead Us. Amherst: Prometheus Books, 2003.

Randi, J. Flim-Flam! Psychics, ESP, Unicorns, and Other Delusions. Buffalo: Prometheus Books, 1982.

Sagan, C. The Demon-Haunted World: Science as a Candle in the Dark. New York: Random House, 1995.

Shermer, M. Science Friction: Where the Known Meets the Unknown. New York: Times Books, 2005.

Reference this article:
Dunning, B. "Approaching a Subject Skeptically." Skeptoid Podcast. Skeptoid Media, Inc., 27 Dec 2011. Web. 21 Aug 2014. <http://skeptoid.com/episodes/4290>

Discuss!

10 most recent comments | Show all 24 comments

The problem skeptics have is when someone points out something that seems to contradicts what is "accepted fact", defies a given studies, or condraticts a "expert" they either.....

1. have to go out of their way to give evidence "behond reasonable doubt".

2. or shut up and accept the "expert" knows better.

Instead of saying "why is this going on/exist"?

At times this "I am an expert" goes to the point of beyond reason.

Due to space I will again use the humble bumble bee example.

Up untill 20 or so years ago in all accepted areas of aerospace science (wing length, body mass, wight to thrust, ect) that something made to EXACT REPLICA of a bumblebee cannot (not shouldnot but CANNOT) fly. Well someone forgot to tell the bumblebee that.

It took decades to come up with (IMO face saving) a "discovery" of a thin channel in its wing giving the extra lift needed.

Now some scientific types did not accept the theory and kept digging KNOWING accepted fact was wrong somewhere.

We see countless areas of archeology, to JFK, to UFO's, to even ft77 where clear evidence flies in face of "presented and acceptable" facts.

But all the "experts" want to do is belittle and deny whats is presented (to the point in some cases of sillyness) that research WHY.

That is why I say if something stands out as "wrong" then one must be a skeptic and demand answers. No matter if the skeptic is a scientis or someone who has common sense enough to say "that aint right".

Eric, Northern IL USA
May 4, 2013 2:21am

I agree Eric.

The problem I've noticed on this site is that many posters rightly use proper scientific rules of enquiry and research for the appropriate subjects, but then proceed to use those same tenets to deny anything that contradicts official versions, of events which are impossible to be brought under the same rigour as science.

What is laughable is that someone may even come up with the very evidence that these "skeptics" demand, but because it's against the official story, the "skeptics" then ignore it or simply continue to deny that the presented alternative may have a measure of truth, even when they themselves have no evidence to present in support of the official story.

In the JFK assassination, even the authorities don't seem to have any firm conclusions about who was responsible, at least publicly, and the case will forever remain wide open to speculation and CT's.

But the 9-11 Comm. report was the final word, and left no doubt in its findings that Osama Bin Laden and co. conspired and carried out the 9-11 attacks essentially alone, using 19 recruits and achieving 75% or their targets.

We've presented clear evidence against the official account of Fl77, and also pointed out that there is no public evidence for the official version whatsoever. None.

Yet the "skeptics" still believe the official story, "believe" being the operative word, because the absence of public evidence of Fl77 renders the official story nothing more than popular legend.

Macky, Auckland
May 4, 2013 4:46am

Eric, the bumblebee story has been known to be an urban myth for a long time. It was an entomologist, with a limited understanding of aerodynamics, who first suggested it and nobody ever took it seriously.

And do you really believe that any scientist who uncovers new information is just 'saving face'? Doesn't that rather defeat your argument that science should dig deeper and do more research?

Macky, I'm not going to dive back into the Hanjour argument again - you've got more stamina in that department than I have- but what do you mean by 'public evidence'? You keep using that phrase and I've no idea what it's supposed to mean.

Darren, Liverpool, UK
May 4, 2013 10:29am

"public evidence" is simply what is in the public domain as to what happened on 9-11 Flight 77.

There is no evidence pro-official story than what an NTSB and corrupt and flawed (self-critized, I might add) report, where top govt officials had to be dragged kicking to testify, and even then in some cases refused to swear on oath to tell the truth.

That's all the evidence there is in the public domain that supports what is at present only a popular version that included some inept pilot managing to perform like a fighter pilot and perform a totally unnecessary turn from a hijacker's point of view.

Also NOT in the public domain is further evidence in the form of withheld cam footage of a contested fly-in where multiple witnesses saw an airliner, or large plane of some sort, fly towards and collide with the Pentagon.

And there is NO evidence (public) outside the official story (Comm. report) that the Pentagon was actually hit by Fl77, or that it was flown by Hanjour.
No public evidence of any sort that he was capable of doing it, and no public evidence of any sort that he was even on the flight.

It's no good saying that the official story of Fl77 sounds plausible, given the "facts".
There are no facts, other than what we are told by Authority. Even many of those so-called facts have been proven contradictory, from both within and without the Report.

So far, the offical Fl77 story is nothing more than a "traditional" popular story, an urban American myth.

Macky, Auckland
May 4, 2013 6:22pm

Darren.

The bumbleee example (of which there are others) is NOT an urban myth.

Evidence of this is the amount of study given to explain why and how they fly.
If it were reptutable scientist would NOT have devoted so much time and effort studying the wing and finding the slat in it.

Now for "face saving" you missed the point entirely.

Not in the specific definition of finding any reason to save a reputation.

But trying to find something scientifically based that they can claim "a new discovery" and use that to avoid (in this case) them looking foolish (in common sense terms) to support the view of the bumblebee they supported for decades.

But unfortunately in many cases they still cling to either their "scientific credentials" or continue to keep trying to search for a "known cause" that fits their dogma instead of admitting they are wrong.

Example of this is the wow signal.

Eric, Northern IL USA
May 4, 2013 10:51pm

The bumblebee thing is totally and completely a myth. You harm yourself and your arguments by even referencing it.

Another Nick, Alexandria VA
July 9, 2013 10:14am

Nick, Eric is a bit (way -way) out of style on this one.. Its a googler thing.. You go to an argument site and voompf you have a view.

Theres a tradition of exacerbation here, ignore it!

Mud, sin city, Oz
August 6, 2013 10:42am

Actually another nick and mud your both wrong on the bumblee "myth".

Wiki article on bumblebees stated that the idea of a bumblebee should not be able to fly was a myth.

HOWEVER as with may topics the two of you seem to defend the old adage "the devil is in the details" comes to play here in spades.

When you follow those numbers that give the citations to where the facts come from in articles a more stunning and inconvient truth comes to light.

All those studies they quote from have a date of 2001 or later that disproved the "myth" of the bumblebee.

Now I know you are going to go nuts over FACTS but common logic shows that untill these studies PROVED that the ORIGIONAL SCIENTIFIC VIEW that a bumblebee could not fly was held to being FACT.

It only became a "myth" after the fact and after (as I pointed out) it was proven false.

In other words in an attempt to change history instead of saying "in the light of new facts we were wrong in our origional scientific facts" you attempt to come off as saying you were right all along.

That is arrogance of the highest order.

That is the arrogance that for some reason people cling to when skeptics bring about facts that contradict the status quo.

So my bubblebee principle not only stays in tact, but shows the vary self serving arrogance that may here claim "true scientist/archeologists/establishes medical/ect" say they are not.

Skeptic 1, experts 0

Eric, Northern IL USA
September 22, 2013 1:54am

I am always impressed by the round about ways you present admissions Eric.

PS who said anything in science was about common sense? I abhor the term..Were science about common sense we'd still be in the trees worrying if our close relatives (the rabbits) would move into the forest.

Multi Dimensional, Greenacres, Big Il USO
September 30, 2013 3:08pm

Brian mentioned googling for journals, to see if the claim went straight to mass media.

What am I looking for as I google search? The word "journal" included in the publication name won't be enough.

Do I simply learn to recognize responsible journals over time?

Robert Hale, Mesa, AZ, USA
October 18, 2013 8:39pm

Make a comment about this episode of Skeptoid (please try to keep it brief & to the point).

Post a reply

 

What's the most important thing about Skeptoid?

Support Skeptoid
 
Skeptoid host, Brian Dunning
Skeptoid is hosted
and produced by
Brian Dunning


Newest
The Haunted Dybbuk Box
Skeptoid #428, Aug 19 2014
Read | Listen (11:26)
 
The Legend of the Flying Dutchman
Skeptoid #427, Aug 12 2014
Read | Listen (11:49)
 
The Baldoon Mystery
Skeptoid #426, Aug 5 2014
Read | Listen (11:44)
 
Listeners Have Another Say
Aug 1 2014
Listen (4:42)
 
Albino Facts and Fiction
Skeptoid #425, Jul 29 2014
Read | Listen (13:49)
 
Newest
#1 -
The JFK Assassination
Read | Listen
#2 -
Asking the Socratic Questions
Read | Listen
#3 -
5 False Arguments for Raw Milk
Read | Listen
#4 -
Fukushima vs Chernobyl vs Three Mile Island
Read | Listen
#5 -
The Riddle of the L-8 Blimp
Read | Listen
#6 -
Who Discovered the New World?
Read | Listen
#7 -
6 Problems with Wind Turbine Syndrome
Read | Listen
#8 -
An Enthusiast's Primer on Study Types
Read | Listen

Recent Comments...

[Valid RSS]

  Skeptoid PodcastSkeptoid on Facebook   Skeptoid on Twitter   Brian Dunning on Google+   Skeptoid on Stitcher   Skeptoid RSS

Members Portal

 
 


Follow @BrianDunning

Tweets about "skeptoid"

Support Skeptoid

Name/Nickname:  
City/Location:
Email: [Why do we need this?]To reduce spam, we email new faces a confirmation link you must click before your comment will appear.
Comment:
characters left. Abusive posts and spam will be deleted.