The Laboratory Fallacy
April 1, 2016
woo-filled nutrition claims. Looking at most nutritional and "complementary and alternative medicine" (CAM) research one finds predominantly laboratory data and poor-quality correlational studies, meaning that claims for CAM and nutrition science are typically overly dependent on the weakest possible medical evidence.Medical treatment is based on data. Clinical data, statistical data and laboratory data are the main points of interest, though laboratory data is a kind of scientific anomaly in medicine. Although controlled laboratory conditions are the most rigorous kind of scientific data, they're the least useful type of data for clinical practice—the obvious reason being that there are strict ethical restrictions for human experimentation in the lab. Nonetheless, petri dishes are poor substitutes for clinical data. Laboratory data can be very convincing to the lay public when evaluating a medical treatment or
There's a multitude of reasons why this is true. In vitro environments—such as test tubes or petri dishes—are wonderfully controlled, but such testing fails to address the complicated nature of effect in the human body. The human body has defenses, compensatory mechanisms and barriers that greatly influence the effect of any given substance. But they're a good first step. Animal models are a useful substitute to human testing, but they produce sub-par evidence. Each step is one piece of the puzzle towards safety and effectiveness. You eventually learn enough to safely test in humans.
Human experimentation has three steps. The FDA summarizes the process of human trials this way:
Phase 1 trials try to determine dosing, document how a drug is metabolized and excreted, and identify acute side effects. Usually, a small number of healthy volunteers (between 20 and 80) are used in Phase 1 trials.Additionally, there is sometimes a fourth step to find out what the long-term risks of an approved treatment are.
If you skip the steps and assume that lab results are representative of real-world effectiveness you will fail. There are a lot of examples of this, and one of my favorites is Streptococcus pneumoniae. It's one of the more virulent forms of pneumonia, though you can easily kill this organism in a petri dish by adding lime juice, Cointreau and tequila—the basic ingredients for a margarita. The alcohol and acidic environment are deadly to a bacteria that evolved to survive on animals. But you can't treat pneumonia with a cocktail. Isopropyl alcohol is a common, extremely effective disinfectant, but taking it internally would likely kill a patient well before it had any effect on an infection. While these are extreme examples, they still illustrate why laboratory data often doesn't translate into useful therapies. Despite its obvious failings, in vitro experimentation is often offered as evidence for the effectiveness of alternative medicines. You see it extensively in the diet and herbal-supplement markets.
When we examine proven clinical research, only a tiny fraction of plausible compounds are found safe and effective enough to become useful drugs. In vitro experimentation allows researchers to find out if something is plausible. It says nothing about its effect in living things. Those experiments provide only the very crudest filtering of utter junk from unfounded possible treatments. Furthermore, the average rate of successful translation from animal models to clinical trials is less than 8%, which use the animal model as a baseline safety and dosage. Additionally, almost 16% of all approved new drugs are withdrawn from use in 10 years due to low efficacy or unexpected side effects.
So what we are talking about? A tiny fraction of a tiny fraction of a tiny fraction of plausible compounds ever result in something useful. If at any point researchers skip the steps, essentially rushing to market a product (such as a supplement) they're cutting corners and taking risks with people's money and health. Animal and human testing are expensive and difficult, and they're required and scrutinized by the FDA. No such protective regulatory system exists for supplements, vitamins, and other products, so purveyors don't use them, relying almost exclusively on in vitro studies for their claims. Realistically, if you stop at the plausible compound level you are given a 0.0002% chance that it is helpful. A .0002% chance of effectiveness is something you won't see on herbal supplement bottle, and you won't see it in an infomercial or a promoter's website either.
You must also ask yourself the really pointed question: "Why?" Herbal and diet supplements are a billion-dollar business. If you have something that really works—and, it appears, the money to make sure it's safe and effective—why not take to the next step? Why stop if the data is all that promising? Drugs are expensive, but the monetary rewards can be much greater. The herbal supplement business is predicated on the fact that you don't actually pay attention to results, so you will pay a premium price for something that has really, really little in the way of evidentiary support. In reality, those vendors want your money and you don't care much about whether their product works or not. (Many don't even care if their product contains the labeled ingredient, as a recent investigation by the New York Attorney General found some supplements and vitamins filled with sand, houseplants, and ground-up peanuts.) Some throw these same accusations at pharmaceutical companies, which is warranted to some extent. Drug companies are not altruistic bastions of human health and the FDA review system has flaws. Nonetheless, they at least have to prove a decent level of proof before hawking their wares.
The takeaway from this is simple: if someone is selling you a remedy, a supplement, or dietary advice based on a potential effect derived from a laboratory experiment (or less), without controlled human trials, you have good reason to be skeptical.Take a minute and support Skeptoid. The money doesn't go to me, but instead goes to keep Skeptoid running as a resource of science and skepticism. Remember: all donations and gifts to Skeptoid Media, Inc. are tax deductible under section 501(c)(3) of the Internal Revenue Code (sections 170, 2055, 2106, 2522).
You can follow me at Twitter @steveproacnp for a daily dose of skeptical nursing.
Disclaimer: This post is my personal opinion, it is not a substitute for medical care. It is for informational purposes only. Information on the Skeptoid blog is not intended nor recommended as a substitute for medical advice, diagnosis, or treatment. Always seek the advice of your own physician or other qualified healthcare professional regarding any medical questions or conditions. This post does not reflect the opinion of my partners, professional affiliates, or academic affiliations. I have no financial conflicts of interest to disclose.
@Skeptoid Media, a 501(c)(3) nonprofit