The Laboratory Fallacy

Medical treatment is based on data. Clinical data, statistical data and laboratory data are the main points of interest, though laboratory data is a kind of scientific anomaly in medicine. Although controlled laboratory conditions are the most rigorous kind of scientific data, they'rethe least useful type of data for clinical practiceâ€"the obvious reason being that there are strictethical restrictions for human experimentation in the lab. Nonetheless, petri dishes are poor substitutes for clinical data. Laboratory data can be very convincing to the lay publicwhen evaluating amedical treatment or woo-fillednutritionclaims. Looking atmost nutritionaland "complementary and alternative medicine" (CAM) research one finds predominantly laboratorydata and poor-quality correlational studies, meaning that claims for CAM and nutrition science are typicallyoverlydependent on the weakest possiblemedical evidence.

There'sa multitude of reasons why this is true. In vitro environmentsâ€"such as test tubes or petri dishesâ€"are wonderfully controlled, but such testingfails to address the complicated nature of effect in the human body. The human body has defenses, compensatory mechanisms and barriers that greatly influencethe effect of any given substance. But they're a good first step. Animal modelsare a usefulsubstitute to human testing, but they produce sub-par evidence. Each step is one piece of the puzzle towardssafety and effectiveness. You eventually learn enough to safely test in humans.

Human experimentation has three steps.The FDA summarizes the process of human trials this way:

Phase 1 trials try to determine dosing, document how a drug is metabolized and excreted, and identify acute side effects. Usually, a small number of healthy volunteers (between 20 and 80) are used in Phase 1 trials.

Phase 2 trials include more participants (about 100-300) who have the disease or condition that the product potentially could treat. In Phase 2 trials, researchers seek to gather further safety data and preliminary evidence of the drug's beneficial effects (efficacy), and they develop and refine research methods for future trials with this drug. If the Phase 2 trials indicate that the drug may be effective--and the risks are considered acceptable, given the observed efficacy and the severity of the disease--the drug moves to Phase 3.

In Phase 3 trials, the drug is studied in a larger number of people with the disease (approximately 1,000-3,000). This phase further tests the product's effectiveness, monitors side effects and, in some cases, compares the product's effects to a standard treatment, if one is already available. As more and more participants are tested over longer periods of time, the less common side effects are more likely to be revealed.

Additionally, there is sometimes a fourth step to find out what the long-term risks of an approved treatment are.

If you skip the steps and assume that lab results are representative of real-world effectiveness you will fail. There are a lot of examples of this, and one of my favorites isStreptococcus pneumoniae.It's one of the more virulent forms of pneumonia, though you can easily killthis organism in a petri dish by addinglime juice, Cointreau and tequilaâ€"the basic ingredients for a margarita. The alcohol and acidic environment aredeadly to a bacteria that evolved to survive on animals. But you can'ttreat pneumonia with a cocktail. Isopropyl alcohol is a common, extremely effective disinfectant, but taking it internallywould likely kill a patientwell before it hadany effect on an infection. While these are extreme examples, they still illustrate why laboratory data often doesn't translateinto useful therapies.Despite itsobvious failings, in vitroexperimentation is often offered as evidence for the effectiveness ofalternative medicines. You see itextensively in the diet and herbal-supplement markets.

When we examineproven clinical research, only a tiny fraction of plausible compounds are found safe and effective enough to becomeuseful drugs. In vitro experimentation allows researchers to find out if something is plausible. Itsays nothing about its effect in living things. Those experiments provide only the very crudest filtering of utter junk from unfounded possible treatments.Furthermore, the average rate of successful translation from animal models to clinical trials is less than 8%, which usethe animal model as a baseline safety and dosage.Additionally, almost 16% of all approved new drugs are withdrawn from use in 10 years due to low efficacy orunexpected side effects.

So what we are talking about? A tiny fraction of a tiny fraction of a tiny fraction of plausible compounds ever result in something useful. If at any point researchers skip the steps, essentially rushing to market a product (such as a supplement) they're cutting corners and taking risks with people's money and health. Animal and human testing are expensive and difficult, and they'rerequired and scrutinizedby the FDA. No such protective regulatory system exists for supplements, vitamins, and other products, so purveyorsdon't use them, relying almost exclusively onin vitrostudies for their claims.Realistically, if you stop atthe plausible compound level you are given a 0.0002% chance that it is helpful. A .0002% chance of effectivenessis something you won't see on herbal supplement bottle, and you won't see it in an infomercial or a promoter's websiteeither.

You must also ask yourself the really pointed question:"Why?" Herbal and diet supplements are a billion-dollar business. If you have something that really worksâ€"and, it appears, the money to make sure it's safe and effectiveâ€"why not take to the next step? Why stop if the data is all that promising? Drugs are expensive, but the monetary rewards can be much greater. The herbal supplement business is predicated on the fact that you don't actually pay attention to results, so you will pay a premium price for something that has really, really little in the way of evidentiarysupport. In reality, those vendors want yourmoney and you don't care much about whether their productworks or not. (Many don't even care if their product contains thelabeled ingredient, as a recent investigation by the New York Attorney General found some supplements and vitamins filled with sand, houseplants, and ground-up peanuts.) Some throw these same accusations at pharmaceutical companies, which is warrantedto some extent. Drug companiesare not altruistic bastions of human health and the FDA review system has flaws. Nonetheless, they at least have to prove a decent level of proof before hawking their wares.

The takeaway from this is simple: if someone is selling you a remedy, a supplement, or dietary advicebased on a potential effect derived from a laboratory experiment (or less), without controlled human trials, you have good reason to be skeptical.

Take a minute and support Skeptoid. The money doesn’t go to me, but instead goes to keep Skeptoid running as a resource of science and skepticism. Remember: all donations and gifts to Skeptoid Media, Inc. are tax deductible under section 501(c)(3) of the Internal Revenue Code (sections 170, 2055, 2106, 2522).

You can follow me at Twitter @steveproacnpfor a daily dose of skeptical nursing.

Disclaimer: This post is my personal opinion, it is not a substitute for medical care. It is for informational purposes only. Information on the Skeptoid blogis not intended nor recommended as a substitute for medical advice, diagnosis, or treatment. Always seek the advice of your own physician or other qualified healthcare professional regarding any medical questions or conditions. This postdoes not reflect the opinion of my partners, professional affiliates, or academic affiliations. I have no financial conflicts of interest to disclose.