Today's episode is about a subject that's as old as debating itself, and it
was prompted by the following email:
I would love to look a lot of this up on my own, but am unsure about
what sources can be trusted. I know you talked about how scientists are not
created equal, but as an average person without the background to fully understand
the primary sources or the ability to synthesize a consensus without reading
meta-analyses, where can I go for reliable information?
This listener brings up a great point. If you follow any of the conversations
on the Skeptalk email discussion list, you've probably heard us banter this
back and forth. One guy says "Hey, leprechauns are real, here's an article
in a peer reviewed scientific journal that says so," and then someone else
replies "No they're not, because there haven't been any such articles in my peer reviewed
scientific journals." Almost any debate can degrade into "My peer reviewed
scientific journal is better than yours."
Now, a really satisfying answer to this question would be "Here,
go to www.legitimate-scientific-journals.com, and you can see at a glance if
your source is a credible one." Surely there must be some register like
that, right? I will dash your hopes with a simple answer: No. There is no such
thing as an authoritative list of reputable scientific journals. There can't
be. And the reason is that word "authoritative". Who is qualified
to be the authority? No one is. No one must be. The moment that any one group
is anointed with the ability to declare a source to be legitimate or not, is
the moment that we lose objectivity and impartiality.
It is very important to be aware that there is any number of bodies who do
presume to be such an authority. Approach with extreme skepticism! The
only reason anyone would compile such a list is to promote an agenda. Someone
once commented on one of my episodes, and they tried to shoot me down by pointing
out that one of my sources was discredited on a web site called sourcewatch.org.
Sourcewatch.org, sounds pretty legitimate, sounds like they do good work, sounds
like they're out there looking out for our best interests by rubber stamping
some sources and discrediting others. But according to critics, Sourcewatch
is a two-man operation that endorses only publications following their own
narrow political bias. This is a perfect example of what you should expect
from any source that attempts to identify itself as a rubber stamping authority.
Be skeptical of
any group you find whose purpose is to identify reputable
As long as we're throwing around the word reputable, I might as well
give the somewhat disappointing answer to the listener's question, and tell
where you can find a reliable journal. Scientific journals achieve their status
only through long histories and good reputations. To be broadly accepted within
the mainstream scientific community, a journal must have established a long
history of responsible reporting, good quality articles detailing well performed
research, and exhaustive peer review. Long standing reputation among the scientists
who matter the most. If you're not one of those scientists, it can be difficult
to know which research is good, which editors and referees are good, and which
journals have a long history of publishing them in good standing. For this
reputation to have any meaning, it must stand on its own and not be supported
by appearing on some simple list. Unfortunately, listener, you
just have to know; but I will give you a starting point in a moment.
While it is essential that good journals be peer reviewed, you should be aware
that almost every publication hoping for prominence describes itself as
peer reviewed. When you hear someone defend their source by stating that it's
peer reviewed, be skeptical. By itself, that's meaningless. Think back to our
old example of the guy writing a UFO newsletter in his basement who has a couple
of his UFOlogist buddies endorse his writing. Suddenly he's "peer reviewed".
This is not the type of peer review that carries any meaning within the mainstream
scientific community, since the peers have a clear agenda and have not established
long histories of scientific acumen by the legitimate community at large. This
is an extreme example, but it does accurately portray a lot of what's out there.
When you don't know anything else about a journal, the fact that it calls itself
peer reviewed cannot, must not, be allowed to carry any weight.
One source that a lot of laypeople are turning to these days is Wikipedia.
What about Wikipedia? It's new and it's a very different animal from anything
previously available, and is something of a paradigm shift. Wikipedia is not
perfect, but it is generally very good. Its principle weakness, so often pointed
out by critics, is also its principle strength. Critics of Wikipedia love to
charge that any old Joe Blow can go in there and edit any article to say whatever
the heck he wants. And this is true, to a point; but they do have multiple
layers of redundant checks and balances in place. Every topic has editors,
and every edit eventually makes it past several sets of eyeballs. Every article
is read by untold zillions of eyeballs, and tempered with suggested edits by
many of them. Most of these suggestions are good, and some of them are bad.
The volunteer editors include some of the foremost authorities on the subject,
and they include crank nitwits, and everyone in between. Wikipedia has tens
of thousands of regular editors, over a thousand administrators who enforce
the behavior of those editors (eventually weeding out the crank nitwits), and
even a judicial committee which resolves any disputes that are not otherwise
handled by the process. The underlying software provides a powerful nerve center
from which the editors and administrators can track history and changes. This
open source process leads to an inevitable effect: Many Wikipedia articles
end up being the closest thing to an authoritative consensus that
we have on a given subject. Each article continually improves over time until
it becomes what Wikipedia describes as the "ideal" article: "balanced,
neutral and encyclopedic, containing notable, verifiable knowledge."
When Wikipedia was first conceived, it was a brand new idea that had never
been tried on such a scale. No doubt, it had plenty of growing pains. But they've
had years to improve the system. They've been dragged through the media more
than once over high-profile errors resulting from vandalism, and
every day since inception, they've worked to address those loopholes. The process
still isn't perfect, but it's one of the most amazing compendiums in human
So, I'm going to give our listener a simple answer to his simple question.
Start with Wikipedia, or any other encyclopedic resource with a good reputation
like Britannica or MSN's Encarta. Nearly always, good articles will include
links to additional references (especially in Wikipedia), but these links are
of tremendously varying quality. Be careful of their external links, and carefully
note why each external reference is being cited. Good articles will often include
a section on criticism or skepticism of the subject. Read it.
Note that I'm no doubt going to be criticized for pointing laypeople toward
Wikipedia as a starting point for research, mainly due to the usual criticisms
of Wikipedia. But, as I said before, Wikipedia's weakness is also its strength,
and I do stand by this recommendation, especially for laypeople of a given
subject who don't otherwise have the experience to choose a good starting point.
What about identifying which scientific journals are reliable? Since we're
not all scientists in the chosen field with the education and experience to
know which are the most reputable publications in our field, we need some kind
of list. But, as we've discussed, lists are bad things when they come from
a source with an agenda. So we turn again to our source with no agenda, Wikipedia.
Search Wikipedia for "List of Scientific Journals" and you'll find
that they have a page listing a few hundred reputable journals in most scientific
fields. Generally, this is an excellent list. The fact that it comes from Wikipedia,
and is constantly being revised for objectivity and quality by experts in each
field, is its strongest recommendation. Some fields are not listed, and most
subsections are partial. You can drill down to find more. But beware: the further
you drill down, the broader the quality control becomes, and the more journals
of lower repute are included, and more non-scientific fields are listed. If
you use this list to gauge the reliability of a source that you see referenced,
you are in as good of hands as are available to the inexperienced journeyman;
but you must use the list wisely. Stay at the top level, or as close to it
as you can. With each click that you drill down, reputation for the listed
journals is generally lower.
Again, this recommendation will no doubt be criticized, and the criticism
is generally valid. But I maintain that for the average guy off the street,
this is the best way to gain a "good enough" grasp of a journal's quality,
and to find good research that truly represents the current scientific consensus.
Garfield, Eugene. "Citation indexes for science. A new dimension in documentation through association of ideas." International Journal of Epidemiology. 19 Sep. 2006, Volume 35, Number 5: 1123-1127.
Library Staff. "How do I evaluate journals?" College of Staten Island Library. College of Staten Island, 1 Jan. 2009. Web. 12 Nov. 2009. <http://www.library.csi.cuny.edu/faqs/169-evaluatingjournals>
Little, John W., Parker, Roy. "How to Read a Scientific Paper." The Department of Chemistry and Biochemistry Department at The University of Arizona. The University of Arizona, 19 Sep. 2009. Web. 12 Nov. 2009. <http://www.biochem.arizona.edu/classes/bioc568/papers.htm>