White Hat Journal Hoaxes

by Brian Dunning
November 1, 2016

The idea of having scientific papers published in respected journals is a good one. Not only does it provide published access to the research, but association with a respected journal tells readers that this research paper is high-quality science. Accordingly, the publishing industry has flourished, as researchers have scrambled to have their papers published, and editors have kept the publication bar high to maintain their all-important reputation. And, quite predictably, this thirst for publication has been viewed as a business opportunity to some. Predatory journals have appeared, charging authors money and performing little serious review of their articles. Readers aren't necessarily able to know the quality of a journal and its contents, so the problem of predatory journals has become a serious one for academia at large. One response conjured up by some authors to call attention to this is white hat hoaxing: submitting a deliberately bad paper to journals, hoping to get it approved and published. And when it does, the excrement can hit the rotor.

The reasoning goes that if a journal can be publicly exposed as having published terrible work, it will serve both to steer researchers away from using that publication as a resource, and — hopefully — motivate that journal to improve its rigor.

The classic instance of a white hat hoax was in 1996. Physicist Alan Sokal had grown weary of the content of a progressive humanities journal published by Duke University called Social Text. He figured that they would publish pretty much anything, so long as it sounded sufficiently anticapitalist and was pumped up with stuffy academic jargon, the so-called intellectual elitism of the progressive left; or as he put it himself, "flattered the editors' ideological preconceptions." It turns out he was right. Sokal wrote a nonsense article titled "Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity" in which he argued that quantum gravity is politically liberal. It was a coup of postmodernism, a dismissal of all science and reality as mere social constructs, and an embrace of intellectual idealism as a better explanation for anything than what mere scientific theory could ever hope to achieve.

When Sokal revealed the hoax following its publication, it was quite embarrassing for the Social Text editors; they hemmed and hawed, and condescendingly claimed that they knew the article was terrible but published it as a gesture of kindness to an author seeking their approval. Even their excuse showed that their publication criteria were anything but what they should have been.

Inspired by his success, Sokal co-authored a book titled Fashionable Nonsense: Postmodern Intellectuals' Abuse of Science. Conversely, the editors of Social Text were awarded the 1996 Ig Nobel Prize in Literature for "eagerly publishing research that they could not understand, that the author said was meaningless, and which claimed that reality does not exist." In a later explanation of why he perpetrated the hoax, Sokal wrote:

My goal isn't to defend science from the barbarian hordes of literary criticism (we'll survive just fine, thank you), but to defend the Left from a trendy segment of itself... There are hundreds of important political and economic issues surrounding science and technology. Sociology of science, at its best, has done much to clarify these issues. But sloppy sociology, like sloppy science, is useless, or even counterproductive.

And thus did white hat hoaxing become a real (and effective) tool to improve the quality of our scientific journals.

Inspired by this, a team of three Serbian academics, dismayed with the declining quality of Serbian journals, wrote a brilliantly nonsensical paper in 2013 titled "Evaluation of Transformative Hermeneutic Heuristics for Processing Random Data" and had it actually published in the journal Metalurgia International, a publisher with a poor reputation but that had been in business, actually publishing monthly issues full of real papers, for a long time. The authors posed for their photos (published in the article) wearing goofy wigs and fake mustaches. In their references, they cited porn star Ron Jeremy, king of pop Michael Jackson, fictional character Borat, someone named A. S. Hole, and of course, Alan Sokal — just in case anyone was paying attention, and in case their use of the word "hermeneutics" in their title wasn't an obvious enough homage.

Although Sokal and the Serbians caught their editors sleeping on the job, most white hat hoaxing is directed at journals that are not merely lazy, but actually predatory. Open-access journals are those that offer their content free online. Since they don't charge for subscriptions, they usually have to charge the authors to publish their work. Good open-access journals still employ top standards and have thorough peer review; but there's now a growing segment of predatory open-access (pOA) journals who aren't really interested in anything but taking money from researchers eager to publish. Their quality can be shockingly abysmal.

No discussion of this subject can omit the most famous case. Back in 2005, two guys at NYU, David Mazières and Eddie Kohler, grew tired of receiving spam conference invitations, so they responded with an academic paper that consisted solely of the phrase "Get me off your f**king mailing list" repeated 772 times. It was nicely formatted and referenced, and even included a flowchart and a graph. Their paper was hanging around, sort of a fun novelty, until 2014 when Peter Vamplew at Federation University Australia thought he'd submit it to a pOA to test their review process. It was the International Journal of Advanced Computer Technology located in India. Almost immediately, he received an email advising him that the Mazières/Kohler paper had been accepted, and giving him all the banking information he'd need to transfer US$150 to a Mr. Tej Pal Singh at the State Bank of India in Delhi. The paper had even been peer reviewed, apparently, with scores of Good, Very Good, or Excellent in all nine categories, and the following comments from the reviewer (who was, sadly, not named):

a. Please Use latest references in order to increase your paper quality.
b. Introduction part is precisely explained.
c. Kindly prepare your camera ready version of paper as per IJACT paper format.
d. Use high resolution images in order to increase the appearance of paper.

Needless to say, Vamplew elected to keep his $150; but the paper, and the responses from the alleged journal, have become somewhat legendary.

Correction: An earlier version of this said the "International Journal of Advanced Computer Technology has since folded and disappeared", but Peter Vamplew reports it is still alive and well, and still sending him spam. —BD

One of the gurus of the pOA phenomenon is Jeffrey Beall, an academic librarian at the University of Colorado. He's best known for Beall's List, an online archive of hundreds of "Potential, possible, or probable predatory scholarly open-access publishers". Beall keeps it as current as possible to make sure that all those on Beall's List are currently in business and taking money from anyone who will give it. Some appear to be surprisingly sophisticated.

Update: Beall was forced to deactivate his list after a number of legal threats were made against him and the University of Colorado. Sometimes the bad guys win. —BD

In one particularly sobering case, mechanical engineer Alex Martin tried to see if he could get a paper written by his 7-year-old son published. It was a fun collection of basic facts about bats. Martin selected a publisher from Beall's List called International Journal of Comprehensive Research in Biological Science and submitted the paper, reformatted to look scholarly. After complying with a request for a minor revision, Martin received an email advising him that the article was accepted for publication in the forthcoming January, 2015 issue of the journal, along with an invoice for $60. In his writeup of this adventure, Martin said:

There was no need to potentially tarnish the reputation of a 7-year old by having him published in a non-reputable journal.

...and elected not to pay. But the editor was insistent, repeating his request for payment, and including a galley proof of the article. But — incredibly — Martin saw that the entire text of the article had been replaced; and by searching the web, he found that the replacement text was copied verbatim, in its entirety, from two legitimately published articles about bats from two different authors, from 2001 and 2002.

It didn't even stop there. Martin looked up a number of other articles published in the journal, and found that 100% of those he examined consisted entirely of content previously published in other journals. It appeared that this journal's practice was to take checks from anyone, and then publish copy-and-pasted text from legitimate sources.

So how easy is it to get past the review processes of open-access journals? In 2012, the top-rated journal Science engaged Harvard biologist John Bohannon to perform a sting operation. He wrote a research paper that appeared good enough to a layperson, but contained gross errors that any competent biologist would readily spot; errors clearly bad enough that it rendered the paper unpublishable. He made up fictitious African authors and research institutions, as African scientists are often under the radar and it wouldn't raise suspicion if someone failed to verify them online. Bohannon then went to the Directory of Open Access Journals (DOAJ) which lists thousands of credible open-access journals. He also went to Beall's List. In the end, he filtered his list down to 304 pay-to-publish journals. He spent months carefully submitting the paper, randomizing everything he could. When reviewers requested changes, he made them. He followed each submission through until it was either rejected or accepted. The results were ugly.

84% of Bohannon's submissions went all the way through to either acceptance or rejection. Of those, 60% gave no evidence of having been peer reviewed. Of those publishers that did legitimately peer review the paper — a paper that should not have survived any peer review process — 70% accepted it. Only 36 of those 304 submissions came back with comments recognizing the grave scientific errors; and of those 36, 16 were accepted anyway. Not surprisingly, 82% of those publishers that came from Beall's List were accepted; but it is surprising that 45% of those from DOAJ were accepted also. Those are supposed to be the legitimate open-access journals. Invoices for publication (none of which were ever paid) rained upon Bohannon, many charging thousands of dollars. About a third of the invoices were from India — the evident hotbed of pOA activity — but it's fair to note that there were also rejections from Indian publishers.

Obviously, there are still plenty of Martin's bats in the open-access attic. And as long as there are, I'm going to continue to stand by white hat hoaxing as a valuable force in the academic publishing industry. The more papers get rejected for the right reasons, the stronger our scientific literature becomes as a whole. The reverse is also true: our literature becomes weaker when poor papers are not rejected. So long live the hoaxers, and the genius of their nonsense.

By Brian Dunning
Follow @BrianDunning