How to Assess a Documentary

It's no secret that a lot of the documentaries that studios throw our way — both on the television and in the cinema — are bloated with misinformation. Most of them are either propaganda or just pop sensationalism. Many of us enjoy watching these for the entertainment value; it's fun, just as it was intended. But there are those of us who hope to actually be enlightened and informed; but unless we happen to have actual expertise on the subject being presented, we have no practical way to know whether what we're watching is good education or just propaganda. Today we're going to cover some tips that will help you acquire this elusive superpower.

I've just read many articles from filmmakers talking about how they make a good documentary film, and there are a number of universal rules they all follow. Don't forget that a movie is art. A movie tells a compelling story. Don't make a documentary, make a movie. Grab your audience emotionally. No matter the subject, at its core it's really about people. Tell a story, don't give a lecture. Don't tell the facts, tell the truth. Movies are art.

There probably are filmmakers out there who are more interested in the material and in helping people than in the art & craft of commercial storytelling, but most of them are not the ones successful enough to be interviewed for these articles. Chances are that if you're watching a documentary, it was made with commercialism as its main goal. The others weren't commercial enough to get onto your screen.

I found one of these director's notes particularly telling. Michael Moore gives one of his 13 rules for making documentaries as "Film only the people who disagree with you." It's a simple insight, and it's been crucial to his tremendous success. Create conflict and drama on the screen. And it's as far as you can possibly get from "Present good information." That's simply not a priority for a commercial filmmaker. The chilling truth is that what we can infer from Michael Moore, and all these other filmmakers trying to make as good a movie as possible, is that they have your entertainment in mind, and not your enlightenment.

So in reality, we need to be gravely skeptical of every documentary, even those that pass all of the assessments I'm about to discuss. And on that note, here they are: the tricks you can use to gauge the validity of a documentary.

Is the film emotionally powerful?

Beware of documentaries that tug at your heartstrings. Appealing to emotions is a powerful way to persuade people to a point of view. Great science can be fascinating and engaging without being emotional; but an emotional presentation is a cheap way to buy the audience's fascination and engagement. It doesn't prove the movie's science is wrong, but it is a red flag that suggests the filmmakers are trying to persuade you without necessarily having the content to do so.

Gasland, the 2010 anti-fracking film, was a great example of this. Even though I agree completely that we urgently need to get off of all fossil fuels, and filmmaker Josh Fox is releasing a new film making that point, Gasland was wrong from beginning to end. But it told emotional stories about people with mystery diseases and showed random pictures of dead wildlife with no context — all to get you emotionally engaged. It worked, and people left the theaters loaded with false information about fracking, and hating it for all the wrong reasons. Beware of emotional content.

How are the contrarians portrayed?

Here's an easy tip that can often give you some great insight into a film's reliability. Most documentaries include experts from "the other side", who disagree with the filmmaker's premise, in an effort to make it appear balanced. Do those experts say something stupid? Because if they did, I can virtually guarantee you that the editors sifted through three hours of tape to find that one stupid sentence. A stupid comment or goofy behavior or anything ludicrous from the "alternative viewpoint" experts is a big red flag that deceptive editing played a significant role in this film's development.

A great example of this was the way the Christian producers of the film Expelled! introduced the atheist Richard Dawkins: they showed him having makeup applied before the interview, and being fussy about it. It was a blatant poisoning of the well, to make him appear ridiculous even before his opened his mouth. In fact everyone wears makeup on camera; you have to. With the harsh lighting and staging, you look weird if you don't.

Here's what always happens to me. While I'm sitting there on camera, the director asks me tons of questions about some urban legend, about its history, about what the people who support the woo-woo version believe, and then I explain what actually happened and how we know that. Then when you watch the edited show, you see me telling about the origin of the legend, and I'll say something like "There are people who believe that x happened, and some people even believe something as crazy as y, or even z." Then everything I said after that, where I give the facts of what really happened, never made it into the show. The viewer gets the impression that the extent of my knowledge on the subject is that I looked it up on Wikipedia, saw a few really strange potential explanations, and then looked no further, believing it to be a mystery without a rational solution. If I was actually clueless, they wouldn't have chased me down to have me on their show. So always pay special attention to how the opposing viewpoint people are presented.

Is the film's protagonist controversial?

By protagonist I don't mean who the movie is about, necessarily, I mean the main person who is trying to make a point. Maybe this is the filmmaker, maybe this is the main expert who is featured on camera and making most of the film's points. Responsible and objective scientists, historians, journalists, whomever, are rarely controversial figures.

As an example, take the movie Blackfish, which was an exposé about killer whales in theme parks. It was highly successful, and had a significant impact on the industry. It's also become iconic for being perhaps the most dishonest film documentary ever made, with virtually all of its claims either misrepresented or wholly fabricated, and the film has been savaged by virtually every marine science publication. But audiences never knew this, they accepted it wholeheartedly, and it won awards and critical acclaim.

The protagonists in Blackfish are the experts it presents who make what appears to be a science-based case against SeaWorld. Its three most prominent are Howard Garrett, Lori Marino, and Ken Balcomb. Although they're all legitimate cetacean scientists or researchers, they're best known as controversial animal rights activists with some pretty far-out views. A responsible science film would have steered well clear of them, because their presence fatally taints the film's reputation.

So that's one thing you can do — Google to see if the protagonists are controversial. If they are, it adds a bit of weight to tip the scales toward the film being unreliable.

Does the film stroke Western Luxury Guilt?

I would like to call out one specific genre as being a huge red flag, and it's what I call western luxury guilt. Those of us who live in western industrialized nations enjoy an extraordinary standard of living. We have comfortable homes filled with gadgets, we drive amazing cars, we eat at restaurants and buy food at grocery stores that have more choices than were available in the entire world a century ago. We make a lot of money and buy whatever we want. We have 75-inch 4K televisions and iPhones.

Although most comfortable people in western nations don't feel any conscious guilt about it, it's still common for some to compensate for it. They tend to project the outward appearance of disdain for such luxuries. It becomes politically correct to show contempt for those things that separate their own wealthy culture from the world's poorest cultures. Thus, themes like anti-corporatism, anti-globalization, anti-materialism, the rejection of biotech and the embrace of esoterica such as alternative medicine, yoga, raw and organic foods, all become badges of enlightenment and social responsibility.

Now, to be clear, I'm not bagging on people who enjoy these things. I'm pointing out that the phenomenon of western luxury guilt is not lost on the producers of documentaries, and they often choose these themes, as they are virtually guaranteed to be successful. Generally, western audiences will love movies that claim the food industry is corrupt or chain restaurants will poison you, or Big Pharma is corrupt and an organic diet cures all disease, or we're all cruel to animals or each other, or humans are destroying the planet with greenhouse gases and plastic — you get the idea. I'm not saying those ideas are wrong — if you listen to Skeptoid you know that I'm a huge proponent of climate science — I'm just pointing out what kinds of themes film producers know their audience wants.

Generally speaking, if the theme of the documentary is one that strokes western luxury guilt and talks about why we horrible westerners should hate ourselves, then the chances are very high that the film is not a good science documentary. Even if its basic claim is one that's generally true, it's likely to be absurdly exaggerated and misrepresented, because it's trying to appeal to the audience's western luxury guilt much more than it's trying to educate.

I repeat: Generally. This is not an absolute rule, but when it comes to assessing a documentary, this is a rule of thumb that will steer you right more often than it will steer you wrong.

Don't worry about a lack of alternative viewpoints.

Here's something that I often hear described as a way to gauge a documentary, but I don't agree. Does it present alternate viewpoints? Does it give a fair shake to opposing views? In most cases, I don't miss it, depending on the type of documentary. If it's about a historical event or a biography or something factual, then introducing opposing viewpoints is harmful — just like math classes don't bring someone in to make the case that 2 + 2 = 5. The process of finding a conclusion can be like a scientific investigation: focus on what we know and how we know it.

If a documentary is about a controversial topic for which there is popular misinformation, then you have to present that misinformation in order to address it. But a good documentary that informs its audience with good information will always be clear about what it's telling you and why. If it's just bringing in alternate viewpoints and saying "Now you decide what's true", then that's a disservice to the facts. A well-done documentary should never allow its audience to leave the theater thinking that some unsupported, unscientific view should remain worthy of their consideration.

So there we have it: a few rules of thumb which may help some of us emerge from theaters a bit less misinformed. But even these will not help you solve the greatest movie mystery of all: What the heck is that yellow fluid they pour over our popcorn?

By Brian Dunning
Follow @BrianDunning