What’s covered in this blog:
- Anecdotes are not evidence.
- Anecdotes are valuable to the scientific process, but they are not conclusions.
- The number of anecdotes does not matter. They can, at most, serve as a basis for forming a hypothesis that can then be tested.
Why I wrote this:
I don’t need to go on very long in this post, since fellow blogger Josh DeWald covered this topic really well last year in his two-part series on the topic of anecdotes and science (Follow these hyperlinks for Part 1 and Part 2). While Josh took a more detailed approach to various ways in which anecdotes are used in science, how they don’t fit into science, and why people still believe their anecdotes over science, I wanted to provide a summary because it seemed after using Josh’s links as an explanation, there were some that didn’t seem to get it.
This came about because of an online discussion I was having with a few people who are against vaccines. Their proof or initial argument was a YouTube video of parents describing their experiences watching, as I will paraphrase here, their children lose the life in their eyes shortly after getting vaccinated and developing autism. In their appeal to pity, the claim is that these hundreds of stories are being ignored and science has chosen to ignore these stories. This is actually the opposite of what happened.
The definition and origin of the word anecdote is interesting. The definition is, “a usually short narrative of an interesting, amusing, or biographical incident.” The origin of the word is Greek, and when roughly translated means “unpublished.” And that’s the problem with an anecdote: no one can check your data because it is unpublished.
The other feature of an anecdote that makes it unreliable is it is a reconstruction of a memory. This leads to several problems. One of those Josh acknowledged, which is the cognitive biases which permeate our memories:
A likely candidate is built into our brains in the form of cognitive biases. Cognitive biases are essentially the evolutionarily wired tendencies in our brain which help us make quick decisions based on patterns and experience. Unfortunately in today’s world, those biases can lead us astray.
I highly recommend you check out the list of cognitive biases available on Wikipedia. You will most certainly identify many biases that you — that we all — do on a daily basis. This makes us human. The problem comes when we are unable to accept that we have been “fooled” by our own bias and insist that the scientific evidence, regardless of how strong, is wrong, a hoax, part of Big Pharma/Food, and so on.
Another problem related to this issue is the way memories are recalled. In essence, each time a memory is recalled, it is passed through our filters of current knowledge, and then needs to be recommitted to memory after it is used. Therefore, each time an anecdote is told, it gets distorted. As Dr. Steven Novella put it:
Memories are updated to bring them into line with our current knowledge. If we are told that the person was wearing a blue jacket, then our memory might change so that it is consistent with what we now believe to be true.
My favorite summary from that same blog post by Dr. Novella is his introduction. Here he states:
When someone looks at me and earnestly says, ‘I know what I saw,’ I am fond of replying, ‘No you don’t.’ You have a distorted and constructed memory of a distorted and constructed perception, both of which are subservient to whatever narrative your brain is operating under.
In other words, these parents might know they saw their child suddenly slip into autism after vaccination, but there is no way to judge without unbiased and unaltered data to know for sure if this was the case.
I’m not saying I distrust what these parents are saying. In fact some of the basic information can be trusted. Their child was likely vaccinated. They have an autistic child. Where things get more fuzzy is determining the details of the behavior before and after the vaccination. How long was the time frame? Were there signs they missed? If it was indeed a very fast regression of behavior, is it possible there was another cause?
This is where science and anecdote intersect, and the important distinction I tried to make with the anti-vaccination crowd: science is not ignoring the anecdotes. The difference is where these anecdotes fit in the scientific process. It doesn’t matter if there is a single anecdote, or 1,000 anecdotes with the same story — it is never used as scientific data. It can’t be. Those that want to go by anecdotes over science are making the outcome of the anecdotes the conclusion. Science considers them basic observation. They can be valuable in forming a hypothesis to be tested. As Josh DeWald put it, “Anecdotes can be a precursor to evidence.” But they are not evidence.
Related to anecdote is the notion of “doing what feels right for you.” I often see this as a way to sell all sorts of anti-science ideas from diet products to parental advice (including skipping vaccinations — Jenny McCarthy’s mommy instinct). While we all probably make decisions on what feels right many times a day, it should never be used for important decisions such as life changes, health decisions, or anything else where good science exists. The data is much less biased than the mind.
I want to thank Josh DeWald for his awesome two-part blog on this subject. I hope this serves as a good supplement and summary of his material.