Approaching a Subject Skeptically
My process for examining a new topic, to learn whether it's fact or fiction.
Filed under Feedback & Questions, Logic & Persuasion
| Skeptoid #290 December 27, 2011 Podcast transcript | Listen | Subscribe |
|
By Brian Dunning, Skeptoid Podcast
Episode 290, December 27, 2011
http://skeptoid.com/episodes/4290
One of the questions I get asked a lot is how I go about approaching a new subject. When you hear about something new, what's the best way to think about it? What's the best way to determine whether it's science or pseudoscience? Well, I'm not sure that there is a "best" way, and I don't think there's one methodology that everyone can follow that's going to work in every circumstance, but I'll try to give the best answer I can. It's probably not the same answer you'd hear from others, but this is what works for me.
First of all, and perhaps most important, is that there's a separation between my daily life and working on Skeptoid. I don't walk around demanding peer-reviewed scientific evidence for everything that I see. I don't have a crazed, obsessive drive to know the validity of every new product for sale at the mall. I'd never get through my day without a certain amount of tolerance for pseudoscience. Fad products, marketing campaigns, greenwashing, and even straight-up fraudulent claims surround us, all day, every day. I accept that. Trying to be a full-time challenger of pseudoscience would not only be hopelessly quixotic, it would also annoy everyone around me, and rob me of the freedom to enjoy my day.
So I let virtually everything slide. A coworker is wearing a magic bracelet? Great, good for him. Neighbor talks about her great visit to the reflexologist? Bully for her. Overhear some people discussing what Nostradamus said about the 2012 apocalypse? Whatever floats their boat.
But what if I'm out with friends and somebody asks me my thoughts on something? This happens all the time. You're on the spot, you don't have access to research materials, you don't have time to look into it. Now, oftentimes I've already done an episode about the subject in question, or something really similar, that gives me a pretty good foundation. Sometimes I haven't, and like most people, have to rely on a journeyman's knowledge of a subject area that's outside of my core competence. This provides a pretty good overview of whether or not the new claim is in line with what's generally known about the subject. Usually it's not; otherwise it wouldn't be on the news or wherever it was that my friends heard about it.
So there you are. You're given something that raises your skeptical radar, it's outside your core competence, and your friends just saw it on television or the Internet. Despite the fact that most people say they take TV or Internet reports with a grain of salt, few actually do. There's something deeply compelling about hearing a claim from an authoritative source; we all have a voice in the back of our heads that wants the new claim to be true, and this desire gets confirmed by the belief that the story wouldn't have made it all the way to the TV news without having been pretty well substantiated. What are you going to do?
The first thing I'd do is take out my phone and track down the original source of the story, using keywords from the report to search Google. I'd want to know if it was reported in any journals, or if it skipped this process and went straight to the mass media. This is the simplest and fastest way to see if a new claim or phenomenon has come from the world of legitimate research, or if it comes from a crank, charlatan, or manufacturer operating outside of science. You always have to remember that the mass media doesn't care; they're interested in the sensationalism of the story, not in its validity.
That's it. That's probably all I'm going to do when I'm out in the world and get a question that's worthy of looking into. It's not a perfect process, but nine times out of ten this will correctly tell you whether there's something there, or whether it's just more noise from media clamoring for eyeball share.
It's only when I take my seat in the Skeptoid office that I assume the mantle of proper separator of fact and fiction. This is when I take each week's topic and give it my honest best effort at a good skeptical treatment. The best topics are those that are popularly misunderstood, but with facts behind them that, when properly understood, are way cooler than the popular version. This isn't as hard as it might sound; nearly every popular myth has some history that puts its genesis into a fascinating new perspective.
Sometimes finding this perspective takes me back in time, to an out-of-print book, or to a newspaper article a century old. Tracking these down requires a lot of eBook purchases, Google Books downloads, newspaper archive searches, and occasionally even the coveted trip to a real library to find a real book. Of course, even the relevant pages from the real book end up as electronic files on my computer, photographed with the iPhone and then OCR converted to searchable text. Getting brand new information, like current research, is almost exactly the same process; it's all available when you have the right accounts to access online research libraries. But none of that compares to the few chances to actually go in person to a place where something strange is said to have happened: to smell the dusty desert wind across Death Valley's Racetrack Playa, to touch the cold granite of the Georgia Guidestones, and to photograph a Fata Morgana mirage such as the ones responsible for so many legendary ghost lights.
I've been doing this show every week for five years now, and on the one hand, you might assume that I've developed a certain aptitude for smelling rats, and have pretty good radar for science vs. pseudoscience. That's true to a degree; but at the same time, I've learned that I can easily be surprised. I often learn that something that sounded pretty hokey is actually true, and something I took for granted turns out to be false. So rather than having developed a supersense for fact and fiction, I've actually picked up a more acute awareness of my own ignorance. Kind of the opposite of what one might hope for; but as we see so often, magically easy solutions to complex problems are a fool's gold.
The process is different every time, but it always starts with a quick survey of the most popular sources, followed by delving deeper into the roots. If it's homeopathy, I want to know what led Samuel Hahnemann to his original conclusions. If it's a conspiracy theory, I want to know who came up with it and what question they were trying to answer. If it's a ghost story, I want to know who first wrote about it and what their relationship was to the hauntee. It's critical to allow for the possibility that the story may or may not be as reported; and to follow up the leads in both directions. Frequently this requires some pretty detailed departure from the popularly known core of the story.
For example, say you find a reference to the mayor of an old town. First you find out if the town actually exists, where it was, whether it's still there, and find it on Google Earth to see if it makes sense within the context of the story. Then find out if the person listed as the mayor actually was the mayor. Find out when he was born, see if the timing is right. There are myriad details you can drill down through, to be as thorough as possible validating the story. Sometimes there are an endless number of these leads, and with only a week between episodes, I often have to simply stop following them, thus making many episodes necessarily incomplete and open to error.
But when you have the time, how far do you go tracking these leads? I've found that there's never a point of diminishing returns. Every time I've made a discovery or connection that (to my knowledge) no other researcher has found, it's always in one of these fine tails of data. The unturned stones are rarely in the middle of the road most traveled. They're in the obscure newspaper article that never got syndicated; they're in the out-of-print interview with the expert who was misquoted in the popular version of the story; and more than anywhere else, they're in the actual published research that was omitted from the mass media reports because it did not support a sensational revisioning of the story.
I don't mean to sound cynical about the mass media. There are many, many excellent reporters and news bureaus who conscientiously produce exceptional material. But I think you'll find that the better they are, the more likely they are to give you an honest assessment of the industry's overall goal, which is to be profitable. The easiest way to do this, as practiced by a probable majority of editors, is to be sensational. I don't think it's a cynical assessment, and it has certainly proven itself to me time and time again through my work validating mass media reports.
So take the road most traveled, as presented in Wikipedia, to get the lay of the land. But to truly learn anything new, you must explore those obscure details that nobody else had time for, or that they overlooked.
Interestingly, I'd say that my process — though it's much more thorough — is probably no more accurate than the quick trick in the restaurant with a smartphone and Google. The more information I collect, the more possibility for error. The more obscure threads I follow, the more are likely to be unreliable. And the more time I spend trying to be thorough on one part of the story, the less time I have for the other parts: an unfortunate exigency of producing a weekly show. I'd say that errors of omission are my most common mistakes, followed by errors that I just didn't catch because of limited time. And like every fallible biological entity, I also make errors by misinterpreting, misreading, and failing to see beyond my own personal biases.
You'll make these same errors in your own research. The best defense against them is to acknowledge your blind spots, compensate for them, and honestly qualify remarks that you can't be sure of. First I try to be right more often than I'm wrong, but second I try to emphasize the process over the conclusions. Being right nine times doesn't guarantee that you'll be right the tenth time, but trying hard all ten times guarantees that you'll at least be as right as your process is capable of.
© 2011 Skeptoid Media, Inc.
References & Further Reading
Nickell, J. Real or Fake: Studies in Authentication. Lexington: University Press of Kentucky, 2009.
Plait, P. Bad Astronomy: Misconceptions and Misuses Revealed, from Astrology to the Moon Landing Hoax. New York: Wiley, 2002.
Radford, B. Media Mythmakers: How Journalists, Activists, and Advertisers Mislead Us. Amherst: Prometheus Books, 2003.
Randi, J. Flim-Flam! Psychics, ESP, Unicorns, and Other Delusions. Buffalo: Prometheus Books, 1982.
Sagan, C. The Demon-Haunted World: Science as a Candle in the Dark. New York: Random House, 1995.
Shermer, M. Science Friction: Where the Known Meets the Unknown. New York: Times Books, 2005.
Reference this article:
Dunning, B.
"Approaching a Subject Skeptically." Skeptoid Podcast. Skeptoid Media, Inc.,
27 Dec 2011. Web.
21 May 2013. <http://skeptoid.com/episodes/4290>
Discuss!
10 most recent comments | Show all 19 comments
What's the point of providing data ?
Even if I could, it's almost certain that rigid positions on the subjects of my assertions and speculations would not change.
Answering posts to mine that comprise of nothing more than ad hominem attacks, diversions, strawmen, and pronouncements of propriety as to the suitability of my sources of evidence, rather than addressing my arguments themselves, are no intelligent counters to my assertions and speculations.
They simply serve as mechanisms of denial
I've learned in my life that the hardest people to get through to are the religious and the educated, those that have a university degree, or its equivalent.
Religious people are already fixed in their beliefs and any questions about their tenets etc are often regarded as an heretical attack on their belief system.
Many Educated seem to believe,in my experience, that because they have a university degree comprising three or four subjects, they are somehow qualified as an authority on everything else under the sun, when in fact while they were spending all those years channeling their efforts down said three or four subjects, others were outside the academic bubble of varsity working in the real world and learning about life and its often illogical and perplexing twists and turns.
Data ?
What's the point when the last nine months on Skeptoid have provided proof to me that even scientists often do not understand plain English sentences, and in some cases, do not want to.
Macky, Auckland
March 08, 2013 2:52pm
First paragraph..
yes, you are speculating..We know that. You have told us this about 10-20 times during a retraction.
Thanx, I am promoted back to scientist from "so called scientist".
Can we have some more fantasy about me? I feel a little bit alone when you arent thinking about me..
Baaaah
Mud, Sin City
May 02, 2013 12:22am
I look out the window every day and assign you the title of "scientist" or "so called scientist" depending on which way the wind is blowing, and as a measure of the rationality of your posts.
I feel my fantastic descriptions of your personal and academic attributes are more than justified, given the nature of your ability to make sudden and unpredictable sideways mental leaps as instruments of diversion from main points of argument submitted, and your chronic tendency to cast ridicule on otherwise very sensible sentiments.
Best wishes
Kiwi Baaaahstard
Macky, Auckland
May 03, 2013 9:23pm
Thanx Macky and hope that your wind problem improves..but less face it, you have the compass and wind.
As to the second, Thank you for the massive compliment. I am touched.
I love the interaction as well.
Cheers
Hang One,
Sin City
you know I still get boards you can hang ten on? A bizarre retro thing for the general skeptoid readership.
Mud, Sin City
May 03, 2013 10:18pm
The problem skeptics have is when someone points out something that seems to contradicts what is "accepted fact", defies a given studies, or condraticts a "expert" they either.....
1. have to go out of their way to give evidence "behond reasonable doubt".
2. or shut up and accept the "expert" knows better.
Instead of saying "why is this going on/exist"?
At times this "I am an expert" goes to the point of beyond reason.
Due to space I will again use the humble bumble bee example.
Up untill 20 or so years ago in all accepted areas of aerospace science (wing length, body mass, wight to thrust, ect) that something made to EXACT REPLICA of a bumblebee cannot (not shouldnot but CANNOT) fly. Well someone forgot to tell the bumblebee that.
It took decades to come up with (IMO face saving) a "discovery" of a thin channel in its wing giving the extra lift needed.
Now some scientific types did not accept the theory and kept digging KNOWING accepted fact was wrong somewhere.
We see countless areas of archeology, to JFK, to UFO's, to even ft77 where clear evidence flies in face of "presented and acceptable" facts.
But all the "experts" want to do is belittle and deny whats is presented (to the point in some cases of sillyness) that research WHY.
That is why I say if something stands out as "wrong" then one must be a skeptic and demand answers. No matter if the skeptic is a scientis or someone who has common sense enough to say "that aint right".
Eric, Northern IL USA
May 04, 2013 2:21am
I agree Eric.
The problem I've noticed on this site is that many posters rightly use proper scientific rules of enquiry and research for the appropriate subjects, but then proceed to use those same tenets to deny anything that contradicts official versions, of events which are impossible to be brought under the same rigour as science.
What is laughable is that someone may even come up with the very evidence that these "skeptics" demand, but because it's against the official story, the "skeptics" then ignore it or simply continue to deny that the presented alternative may have a measure of truth, even when they themselves have no evidence to present in support of the official story.
In the JFK assassination, even the authorities don't seem to have any firm conclusions about who was responsible, at least publicly, and the case will forever remain wide open to speculation and CT's.
But the 9-11 Comm. report was the final word, and left no doubt in its findings that Osama Bin Laden and co. conspired and carried out the 9-11 attacks essentially alone, using 19 recruits and achieving 75% or their targets.
We've presented clear evidence against the official account of Fl77, and also pointed out that there is no public evidence for the official version whatsoever. None.
Yet the "skeptics" still believe the official story, "believe" being the operative word, because the absence of public evidence of Fl77 renders the official story nothing more than popular legend.
Macky, Auckland
May 04, 2013 4:46am
Eric, the bumblebee story has been known to be an urban myth for a long time. It was an entomologist, with a limited understanding of aerodynamics, who first suggested it and nobody ever took it seriously.
And do you really believe that any scientist who uncovers new information is just 'saving face'? Doesn't that rather defeat your argument that science should dig deeper and do more research?
Macky, I'm not going to dive back into the Hanjour argument again - you've got more stamina in that department than I have- but what do you mean by 'public evidence'? You keep using that phrase and I've no idea what it's supposed to mean.
Darren, Liverpool, UK
May 04, 2013 10:29am
"public evidence" is simply what is in the public domain as to what happened on 9-11 Flight 77.
There is no evidence pro-official story than what an NTSB and corrupt and flawed (self-critized, I might add) report, where top govt officials had to be dragged kicking to testify, and even then in some cases refused to swear on oath to tell the truth.
That's all the evidence there is in the public domain that supports what is at present only a popular version that included some inept pilot managing to perform like a fighter pilot and perform a totally unnecessary turn from a hijacker's point of view.
Also NOT in the public domain is further evidence in the form of withheld cam footage of a contested fly-in where multiple witnesses saw an airliner, or large plane of some sort, fly towards and collide with the Pentagon.
And there is NO evidence (public) outside the official story (Comm. report) that the Pentagon was actually hit by Fl77, or that it was flown by Hanjour.
No public evidence of any sort that he was capable of doing it, and no public evidence of any sort that he was even on the flight.
It's no good saying that the official story of Fl77 sounds plausible, given the "facts".
There are no facts, other than what we are told by Authority. Even many of those so-called facts have been proven contradictory, from both within and without the Report.
So far, the offical Fl77 story is nothing more than a "traditional" popular story, an urban American myth.
Macky, Auckland
May 04, 2013 6:22pm
Darren.
The bumbleee example (of which there are others) is NOT an urban myth.
Evidence of this is the amount of study given to explain why and how they fly.
If it were reptutable scientist would NOT have devoted so much time and effort studying the wing and finding the slat in it.
Now for "face saving" you missed the point entirely.
Not in the specific definition of finding any reason to save a reputation.
But trying to find something scientifically based that they can claim "a new discovery" and use that to avoid (in this case) them looking foolish (in common sense terms) to support the view of the bumblebee they supported for decades.
But unfortunately in many cases they still cling to either their "scientific credentials" or continue to keep trying to search for a "known cause" that fits their dogma instead of admitting they are wrong.
Example of this is the wow signal.
Eric, Northern IL USA
May 04, 2013 10:51pm
Make a comment about this episode of Skeptoid (please try to keep it brief & to the point). Anyone can post:
You can also discuss this episode in the Skeptoid Forum, hosted by the James Randi Educational Foundation, or join the Skeptalk email discussion list.
What's the most important thing about Skeptoid?
![[Valid RSS]](/images/valid-rss-rogers.png "Validate my RSS feed"){kind=small}
Hmm the above has been asked since July 24 at least..
Cough up... cough up...
Data, data...
Mud, At virtually missing point, NSW, OZ,
December 17, 2012 10:44pm