Forensic (Pseudo) Science

by Brian Dunning
March 1, 2022

Something we've pointed out on Skeptoid many times over the years is that science is not done in the courtroom. People are constantly mistaking legal findings for scientific findings. They are not. Judges and juries can be persuaded of things by any number of methods, and solid science is rarely one of those. Even when good science is presented in a courtroom, lawyers are free to argue over its validity, thus wrongly creating the impression that the science itself is shaky. But just as harmful is the opposite process: the introduction of bad science in a courtroom alongside a deep-rooted cultural belief that the science is a valid one, and here we are talking about forensic science, the bits of evidence that link a killer to a crime scene, believed by many to be irrefutable. Today we're going to check in on the current status of the tidal wave of reform that should be crashing down upon forensic pseudoscience, but which has, oddly, stalled offshore somewhere and is nowhere to be found.

In 1987, the very first criminal conviction based on DNA evidence was secured, that of a Florida rapist. Then in 1989, the first exoneration of a previously-convicted man due to DNA evidence took place. This blossomed throughout the 1990s, as DNA was used more and more often to prove with scientific certainty that convictions had been falsely obtained; convictions which were based on other types of evidence. Most of these overturned convictions had been obtained based on eyewitness misidentification, false or coerced confessions, or through the testimony of informants; but of all of these, half also involved false or misleading forensic evidence. The wonderful Innocence Project, a US-based nonprofit legal organization that works to overturn improper criminal convictions, maintains a white paper breaking all this down. Among the most-offending junk sciences they point out are hair analysis, bite mark analysis, shoe print comparisons, bullet lead analysis, and certain types of evidence from a fire allegedly indicating arson. Sadly these are just the tip of the iceberg.

But wait a minute, we're tempted to respond; I've been watching all my favorite cop investigation dramas on television for decades, and I know all those things are rock solid scientifically. Well it turns out we may believe that, but it's in surprisingly sharp contradiction with the facts. Because when all of these convictions started getting overturned due to DNA evidence, prosecutors started getting upset and nervous. Advocates for criminal justice reform started getting louder and bolder. Courts and lawmakers started taking notice. And what has followed, so far, has been a giant wave of nothing. The status quo of bad and misrepresented science still reigns.

The basic root of the problem is that prosecutors do not want scientists in the courtroom. They're called prosecutors because their job is to convict. It's not to investigate, not to learn the truth, not to bring about justice — those are the domains of investigators, grand juries, judges, and others in the chain. Prosecutors want the tools of the courtroom to be those that result in convictions, so the majority of these forensic evidence techniques were not developed by scientists, they were developed by prosecutors and by law enforcement. Therefore we should expect that such forensic techniques would have minimal, if any, scientific basis. And at every level of our criminal justice system, it is the prosecutors — from local to federal — who are primarily responsible for introducing evidence into our courtrooms. Given this structure, the surprise would be if the evidence in our courts did pass scientific scrutiny.

Addendum: It should be noted that the above paragraph triggered volumes of criticism from prosecutors who completely disagree with my characterization. It also drew feedback from public defenders, a mixture of praise and criticism that I did not go far enough. I'll include representative snippets from both in a future feedback episode. —BD

All of this new disgrace that DNA analysis had brought to the other types of forensic evidence prompted Congress to pass a new law in 2005 which, in part, directed the National Academy of Sciences to conduct a study on forensic science. They were to make an assessment and then create recommendations and best practice guidelines. The result of this study came in 2009 with the 300+ page book Strengthening Forensic Science in the United States: A Path Forward, freely available online as a PDF. This book goes into great detail on just about every forensic evidence discipline you've heard of, and it generally finds that the scientific validity of each varies widely from one to another. Although this was nice to have, unfortunately little was done with it; and innocent people continued to be convicted based on pseudoscience.

And this was when politics entered the scene. The United States had a series of two presidential administrations with polar opposite views on criminal justice. The first, the Obama administration from 2009-2016, focused on reform. It was followed by the Trump administration from 2017-2020, which concentrated on prosecution, and this series of two administrations took forensic science on a wild ride. Under Obama, a commission called the President's Council of Advisors on Science and Technology researched and wrote a 2016 report called Forensic Science in Criminal Courts: Ensuring Validity of Feature-Comparison Methods which found many techniques, such as bite mark and shoe print analysis, so devoid of foundational validity they should be prohibited. But what was to be done with these findings would be left to the Trump administration. What that turned out to be was that the Obama commission's charter was allowed to immediately expire; the new Attorney General outright dismissed their report and said it would not be adopted; and Trump's own forensic science review was announced, which consisted of one former prosecutor hired to write one release stating the Obama report was in error and should be ignored. As of this writing, that's where we remain today.

But let's change gears now and have a closer look at some of these specific forensic disciplines. All we've got time for on this short-format show is a quick look, so by no means are these summaries intended to be comprehensive. As always, if you want the full details, come to the transcript page for this episode and look up the references for the full commission reports on these and more.

DNA

It's best to open by pointing out that "DNA evidence" by itself is not some holy panacea. DNA evidence can be used and it can be misused, and often is. One type of DNA analysis has been found to lack foundational validity, and this is called combined probability of inclusion methods on complex mixtures. This is where a complex mixture, potentially containing genetic material from multiple individuals, is given a probability that the suspect may be a contributor. It's a sort of lightweight statistical analysis that depends on the examiner to make subjective choices which drive the final result, and subsequent re-analyses have found the probabilities given in court to be way off in many cases, sometimes by factors of billions.

Ballistics

One of the most common types of forensic evidence we've gotten used to is looking at the scratch marks on a bullet, and supposedly proving that it was fired by this exact gun. They take a bullet from a crime scene and compare it to one test-fired from the gun under suspicion. Well it turns out that when examiners' results are tested against known results, their error rates are crazy high. Various papers I read found error rates as low as 10% and as high as 66%. In 2008, the National Research Council published a 344-page report called Ballistic Imaging and one representative snippet from it says "The validity of the fundamental assumptions of uniqueness and reproducibility of firearms-related toolmarks has not yet been fully demonstrated."

Bullet Lead

In some cases where ballistics hasn't been possible, such as when no gun was recovered, crime labs would do a "bullet lead examination" to show that, for example, the bullet from the victim's body came from the same batch of bullets found in the suspect's garage. The lab would find that the makeup of the lead was a match, and the suspect would be convicted. However, what they never mentioned is that there are probably countless batches of bullets with that exact same chemical makeup. This one has been a rare victory, with the FBI Laboratory announcing in 2005 that they would no longer do this type of examination and present it as evidence.

Bite Marks

Bite mark comparison, called forensic odontology, is particularly unreliable. The expert looks at bite marks on a victim, and then compares them to the teeth of a suspect. The fundamental problem is that while it can reasonably be used to exclude certain suspects whose teeth are totally different from the bite mark, it is typically used to pin the blame on a suspect. There is no basis at all for this, as it there is no evidence that dentition is unique, no evidence that dentition transfers a unique pattern onto skin which will retain it free of distortion, and strong evidence that different experts make different findings from the same data. No large scale study has ever been performed to see whether bite marks are unique, or to establish what percentage of the population could have produced any given bite mark.

Hair Analysis

Microscopic hair analysis, distinct from DNA analysis which can't be done when you don't have the root, is predicated on the notion that hair from different people looks visually distinct under the microscope. The Department of Justice has supported its use largely on the strength of a series of studies from the 1970s which found a false positive rate of less than 1 in 40,000. They've always ignored that the studies were savaged in the scientific journals for being deeply flawed methodologically, even making basic arithmetic errors in their conclusions. When the FBI did a large study in 2002 to assess their own accuracy, they found an 11% false positive rate, and the Department of Justice's response was to simply ignore this as well.

Fingerprinting

Among the oldest and most famous forensic techniques, latent fingerprint analysis is considered to be foundationally valid, but has weaknesses jurors are unaware of. First, the number of properly designed studies of the validity of fingerprinting is — guess how many? — exactly two. One found false positives occur 1 in 306 matches; the other found 1 in 18. The methodology used is sound, but researchers found its practice is plagued with confirmation bias; examiners working backward from a suspected match and matching more patterns than are actually present in the latent print.

And to reiterate, that was just one very brief paragraph on six forensic disciplines. There are many more, and the commissions have similar things to say on each of them.

My favorite summary of the state of forensic science is this lone sentence from the introduction to the 2009 National Academy report: "With the exception of nuclear DNA analysis, no forensic method has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source." That is an extremely bold, and weighty, statement. And as such, it really underscores the value of having a basic science literacy, and exposure to the value of skeptical analysis of pop-culture beliefs like this one. The average person on the street probably doesn't have the slightest inkling that there's a thing wrong with all that fancy forensic science they see on their favorite TV dramas; and if that person gets onto a jury, he'll be the easiest rube in the room for some prosecutor who's hanging his case on carpet fibers and shoe prints, but who won't happen to mention that half the town could be convicted on that same basis. Forensic science is yet one more example of why, whenever you run into something that everyone just assumes is true but that has even the slightest red flags, you should always be skeptical.

By Brian Dunning
Follow @BrianDunning