I Said 'Show Me The SCIENCE'
by Eric Hall
December 7, 2013
Writing this blog has proved to be quite the challenge. I enjoy doing it as it is greatly helping me work on being a better writer. It also is a fantastic way to educate people on the process of science, inform people about science, and on how to be properly skeptical in the scientific sense. However, each time I read the comments, I have to wonder if I am making some assumptions or have missed some piece in getting readers to properly understand the process for evaluating someone's scientific conclusions. Let's look at one piece today.
One of my posts from earlier this year now has over 300 comments on it, in which I repeatedly ask one commenter to provide scientific evidence for the claim being presented. After significant prompting, I did get a few studies. The source was The Journal of Alternative and Complementary Medicine (JACM). While I think most in the skeptic community probably know this is pretty much garbage information, maybe just by the name alone, I would like to examine why this should be a red flag for anyone trying to present a scientific argument. I will approach this by comparing to a more mainstream journal, The Journal of the American Medical Association (JAMA).
As a point of comparison, let's look at the submission instructions for JACM. The first part of the page reads much more like a sales pitch. It gives a list of benefits you will receive for submitting your article. It then further explains there is a small processing charge - but hey - look at these benefits again! On the other hand, JAMA has no charge to submit a manuscript. However, the guidelines are much, much longer and more specific.
Another thing I found in the instructions for JACM was the confusion over what is accepted. Here's how they describe themselves:
The Journal of Alternative and Complementary Medicine: Research on Paradigm, Practice, and Policy is a peer-reviewed journal. It includes observational and analytical reports, ideas, commentary, and opinions on therapies currently thought to be outside the realm of conventional Western biomedicine that are gaining interest and warranting research to assess therapeutic value. The Journal includes current concepts in clinical care, including case reports, which will be valuable for physicians and other health care professionals who are seeking to evaluate and integrate these therapies into patient care protocols. (emphasis mine).This alone is fine. It is their journal, they can accept what they want. But then just a few paragraphs later:
Case Reports may be submitted, but it is important to note that JACM will consider only truly extraordinary cases with contain exceptional findings that will contribute significantly to the field.Um, OK. Which is it? This would seem to confuse what is acceptable for submission.
Here's another point within this that is interesting. Apparently there is a political (or at least an ethical) agenda in JACM is pushing as well. By not accepting animal studies, they are absolutely leaving out an important piece in research that benefits humans. While the ethical guidelines should be constantly reviewed and the animals should be treated humanly, there is at this point, still no substitute for that step in many cases. JACM misses a big piece of medical knowledge.
Though JACM says it is peer-reviewed, there is no guideline as to what the reviewers are looking for, other than formatting issues. JAMA has guidelines on data, tests, ethics, etc. The section on disclosure is extensive, and they even provide resources to for authors to make sure they are fully and properly disclosing any possible conflicts. JACM simply requires reporting conflicts which might affect the manuscript.
One other huge difference is the impact each journal has on the medical field. One way to measure this is the impact factor, which is a measure of how often articles are cited in other journal articles. JACM has an impact factor of 1.46, varying between 0.93 and 1.63 over the last decade. JAMA has an impact factor of 29.98, varying from 15.40 to 30.03 over the last decade with a fairly level factor over the last few years.
JACM is published by Mary Ann Liebert, Inc., a publisher focused on publishing material. That is their business and how they make money. This would suggest that accepting more articles would be in their interest as more publications means more revenue. JAMA is published by the American Medical Association. The focus of the AMA is, "To promote the art and science of medicine and the betterment of public health." As a nonprofit, it doesn't seek a profit, only enough revenue to cover expenses. The other focus area I personally like with the AMA is, "AMA is one enterprise, highly capable, well coordinated and focused on high impact results," with the keyword here being results.
In fact, the JAMA gives facts about how many submissions actually make it through the peer-review process:
"...JAMA's acceptance rate is approximately 9% of the more than 6000 manuscripts received annually."No such disclosure is given in JACM. This is typical for an open-source journal, because the rejections are typically due to form only, and a high degree of submissions are simply published as long as they meet some very basic requirements.
All of the information above is only secondary to the actual content of each article. I would never suggest judging a piece based on its source alone. As I have stated in other articles, to do so would be a logical fallacy. It doesn't matter if the article comes from D.D. Palmer or from Steven Novella, each article needs to be evaluated for its accuracy in the methods used to reach a conclusion. However, clues such as the source are important for how carefully one should evaluate the data and the methods. I have much more confidence in an article from JAMA than I do from JACM. Thus if presented with an article from JACM as "proof" of some hypothesis, I am going to spend much more time evaluating the methods and the conclusions.
It is a careful balance. Certainly even the best of authorities make errors. For example, The Lancet has a higher impact factor than even JAMA. Yet, The Lancet sat on the Wakefield paper for over a decade before finally retracting it. The design of the study had a small number of patients. It was the first time results like Wakefield found were published. Because of that, it certainly didn't serve as "proof." It at best could have served as a starting point for more study. And indeed it did, far beyond what was probably necessary, especially with hindsight to find out not only was it just his results couldn't be repeated, but it turns out there was fraud involved in his results as well. It is important not to accept (or reject something) solely based on the source.
Please understand that the source of information does play a role in the evaluation of the validity of its applicability to science. While using that as the sole evaluation would be a logical fallacy, ignoring it would be equally illogical. If anyone wants to comment on any of my posts and use JACM as "proof" I am wrong, please understand you have a higher level of burden in showing the study being posted is valid. As always, I invite other viewpoints and enjoy having my conclusions be challenged. However, I don't change my conclusions based on any information, but instead on valid, scientific information.
by Eric Hall
@Skeptoid Media, a 501(c)(3) nonprofit