Anecdotes and Science -- Part 1

On my other blog, the anti-aspartame advocate Betty Martini left left some comments in relation to some statements I made about the misuse of anecdotes as evidence. Her comments sparked an interest into diving a bit deeper into the use of anecdotal evidence versus solid “scientific evidence”. Her statements are representative of something we are actually all prone to, and must consciously fight, and that is a major reason behind the popularity of various medical claims not-backed-by-evidence.

This article is Part I of a two part discussion on anecdotal evidence. This part will discuss anecdotal evidence in general and why we are so susceptible to it. The second part will discuss how anecdotal evidence is used in practice.

Should we treat “tons” of anecdotal evidence as if it were actual data?

I will quote from the relevant parts of the comment thread, the interested reader can read the full thread in its full context on my other blog here.

ME: “Anecdotal reports of individual people simply do not address the scientific question of whether aspartame should be considered dangerous to the general population...”

MARTINI: Dr. [HJ] Roberts writes.. “Disparaging comments about ‘anecdotal evidence’ by editors of medical journals and reviewers in academia have contributed significantly to the ongoing lack of awareness of aspartame disease. The great tradition of the clinical anecdote, dating back to Hippocrates, has been forsaken. Borgstein (1999) observed: “Have you noticed that the clinical anecdote has almost disappeared?….We have statistics now, and no case is worthwhile unless we can collect a series and apply some complex statistical formula to it to make it significant somehow.”

My radar goes off any time somebody harkens back to the “good old days” of medicine and science, forgetting that advances are called “advances” for a reason. It’s a Good Thing that we don’t simply rely on personal testimony as the primary basis of evaluating claims, especially when generating guidelines that apply to the larger population. It wasn’t until the 19th century that the “germ theory” of disease really took off, and Robert Koch published his four “postulates” (Wikipedia) for determining whether a particular microorganism is the cause of an illness.

As to the significance of individual cases. Of course single cases are important to the particular person who is affected (or believes they are affected) and therefore they should consult their doctor. But the “complex statistical formula” applied to larger samples is the only method we can reliably use to be able to say that some chemical, process, method appears to have the intended, or unintended, effects across a large population. Even when a drug, device, food additive, etc is approved for general use, they frequently still say “Speak to your doctor before starting or if you experience symptoms”. Those single cases are certainly useful to be aware of, to look for the start of a trend, or a rare disorder or situation that leads to a reaction.

“Pro-industry physicians and investigators have consistently attempted to put a favorable spin on aspartame safety by denigrating published clinical observations as “merely anecdotes.” Corporate-sponsored critics reflexively scorn “anecdotal evidence” as “invalid and unreliable.”

To thousands of people who read “Killer Kola” who were using aspartame and got off of it, their story cannot be considered just an anecdote. It is their life. MS victims walked out of wheelchairs, blind people regained their sight, those crippled with fibromyalgia were able to walk, depression disappeared, and physicians started giving out pages to their patients.

The reader can easily fill in the blank replacing the safety of aspartame with the “effectiveness of homeopathy/astrology/acupuncture”. Essentially: “It works (or hurts, in this case) for me, I don’t care what the scientific evidence says”. And it’s absolutely understandable why this is convincing. What else is more available and convincing than our own personal experience? Thousands of people? May as well be a peer-reviewed study, right? Well, no. It would actually have been great if they were part of a well-controlled study. Thousands of people would be a gold mine. It would be great if the medical histories of all these people could be looked at, to see what they had previously and when and how the symptoms went away. But we don’t have any of that data. We actually know nothing about them except the second-hand claim that they were “healed” by removing aspartame from their diet.

As to multiple sclerosis (MS) and aspartame, here is what the National Multiple Sclerosis Society has to say about aspartame as a cause of MS:

“No scientific evidence supports the claims on several Web sites that aspartame, an artificial sweetener used in many diet soft drinks and other foods, causes MS.”

If there was even a large amount of convincing anecdotal evidence, it would be reasonable for the NMSS to at least issue a warning. Maybe the anti-aspartame folks will respond that the NMSS is in the pockets of Anjimoto (previously, of Monsanto), or that they have been “deceived” or are “blind to the evidence”. But more likely, despite the loud voices on the Internet, there isn’t even enough anecdotal evidence to convince an organization that exists solely to assist people with MS.

“Killer Kola” seems to have been a 16-page brochure distributed by an organic food store in Georgia named “Return to Eden” based on the material from Betty Martini’s anti-aspartame site. There is little reason to think that it contained any additional convincing evidence on top of the unsubstantiated claims that already populate anti-aspartame websites.

Why are anecdotes so convincing?

While I make an effort to look for solid scientific evidence of things, my natural tendency is to trust my own experiences or my friends and family when they say that something “really works for me” or that they “only” experience some side effect when eating one thing or another. Why is anecdotal evidence so convincing?

A likely candidate is built into our brains in the form of cognitive biases. Cognitive biases are essentially the evolutionarily wired tendencies in our brain which help us make quick decisions based on patterns and experience. Unfortunately in today’s world, those biases can lead us astray.

I highly recommend you check out the list of cognitive biases available on Wikipedia. You will most certainly identify many biases that you -- that we all -- do on a daily basis. This makes us human. The problem comes when we are unable to accept that we have been “fooled” by our own bias and insist that the scientific evidence, regardless of how strong, is wrong, a hoax, part of Big Pharma/Food, and so on.

The three I want to discuss that are particularly relevant to the acceptance of anecdotal evidence as “proof” are the Availability cascade, the Congruence Bias, and the well-known Confirmation Bias.

Availability Cascade

A large number of anecdotes might lead to a cognitive bias known as the “Availability cascade” where, regardless of the truth of a claim, its frequent citation and “availability” in discourse (especially on the internet) makes the idea seem plausible. Even friends of mine who drink diet soda will think there is “something” to the notion that aspartame is bad for you. And then there are friends who avoid diet drinks altogether (but don’t necessarily avoid the thousands of other products that contain aspartame, because they don’t realize just how prevalent the usage of aspartame is). The notion that aspartame is dangerous has been floating around for so long that it is “available” as being obvious, despite there being little to no evidence supporting the notion.

Congruence Bias

The “Congruence bias” is when someone only tests their hypothesis directly, but makes no effort to check check alternative hypothesis. For example, a person may decide that aspartame is the cause of some ailment or other (headaches, seizures, general malaise, depression). So they stop “taking aspartame” (most likely, cutting out diet soda but if they are diligent they might cut out any of the multitude of low calorie items) and the feel better. But what else did they stop taking? Are they perhaps just more aware of their diet and replace previous “low calorie” snacks with healthier snacks (fruits and vegetables)? Did they try having “blind” (perhaps with a friend) replacement of sugar-containing items with their low calorie equivalents and check the effects? Did they take objective measures of their symptoms before and after “cutting out” aspartame? Did they have a doctor diagnose a particular ailment before and after? Did they confirm that the number of days of the test is enough to establish a statistical effect? Perhaps, perhaps not. But that is what happens with a properly-controlled, randomized, (double) blinded trial.

Confirmation Bias

Confirmation Bias is when a person favors information which confirms their beliefs and ignores evidence that disconfirms. For example, a person might only remember the days that they avoided a diet drink and didn’t get a headache, but will not recall the days they didn’t have headaches while also having a diet drink, or had headaches while avoiding. Fill in your favorited unsupported belief: a believer in astrology completely forgets all the “misses” of the astrologic/fortune teller, a believer in homeopathy doesn’t realize that before they found out about homeopathy their cold still only lasted 2-4 days.

Conclusion

Unless we make a conscious effort, we are all easily persuaded by the personal experiences of friends, family and ourselves. Without looking for outside explanation of some issue or benefit, or somehow performing a controlled study, we cannot be sure that our experience represents “reality”. Add up a lot of people experiencing this situation and we amass a large body of anecdotal “evidence” that might not in fact mirror reality. Being aware of our own biases is one method of avoiding being overly persuaded by evidence-that-isn’t. It may very well turn out that our experience is actually one of the rare situations where some particular effect actually occurs (e.g. allergies, drug combinations/interactions). But self-diagnosing based on Internet pages is not the right approach to determine this.

In Part 2 I will discuss more specifics about how anecdotal claims are used by the FDA and CDC as they relate to post-marketing surveillance (in some cases confirming anecdotes), as well as a brief discussion of where anecdotes have proven to be way off.

REFERENCES

National Multiple Sclerosis Society. “Old Theories that have been Disproven”. Visited January 2013.
Wikipedia. "List of biases in judgement and decision making". Visited January 2013. http://en.wikipedia.org/wiki/List_of_biases_in_judgment_and_decision_making