An Enthusiast's Primer on Study Types

by Brian Dunning
September 24, 2013

We don't all work professionally in the hard sciences; in fact, most of us don't. Yet those of us who are fans of science and its applications are often confronted with evidence and articles citing studies and other research, and we can sometimes feel at a bit of a loss to understand some of the lingo. One thing I've had to do a lot of in my years of Skeptoid is reading studies. There are all kinds of different studies: cohort studies, controlled trials, uncontrolled trials, longitudinal studies, retrospective studies, and many more. The uninitiated student of studies quickly becomes awash in a sea of study types.

Today we're going to take a quick helicopter view of study types. As with most lists, this one is not exhaustive, but it does cover the basics that you're ever likely to encounter in your reading. People have actually published taxonomies of study types; the subject can get that complicated. Just about every type of study has various subtypes, each of which may have its own variants and so on. And because this subject has the appearance of potentially being incredibly boring, we shall use the example of aliens from Planet Zero studying the Earth.

In broad strokes, there are two basic kinds of studies: observational and experimental. In an observational study, the Zeroids just watch the Earthlings to see what happens without intervening in anything; in an experimental study, they watch to see the results of an intervention, such as medical probing or planetary destruction. Experimental and observational studies are sometimes called interventional and non-interventional.

Observational Studies

A common observational study is the cohort study. Cohorts refer to a group of people who have some trait in common, and are thus chosen for observation. The Zeroids may be interested in studying a cohort of Earthlings consisting of astronauts. They may choose this particular group because they might, for example, be interested in seeing how the astronauts respond to the increased radiation of the space environment.

Since they choose the group first and then sit back to watch what happens to them, this would be a prospective cohort study, which means they're going to watch the cohorts in real time, with no idea of what might happen to them in the future. Another way to go would be if they beamed down into the Earth Astronaut Headquarters and secretly stole the record tapes showing archival history of astronauts and how well they responded to radiation, then it would be a retrospective cohort study, where the outcomes are already known, and it's a case of working backwards to see their histories.

If the Planet Zero scientists wish to compare the astronauts to another group — for example, people in Ecuador who are well shielded from radiation by the atmosphere — they might choose to make it a concurrent cohort study, which is when two or more groups of cohorts are studied to compare their results to one another. This might allow the Zeroids to learn that people in Ecuador suffer fewer ill effects from radiation than astronauts.

Now suppose that one of the Planet Zero researchers points out that evolution in spaceship design over the decades might be giving the astronauts more or less protection. They might elect to perform a non-concurrent cohort study, in which they look at astronaut data from their stolen tapes to compare astronauts from the 1970s with astronauts from the 2000s.

As it may turn out, the Zeroids might discover that some humans spontaneously sublimate from a solid to a gaseous cloud of molecules, a phenomenon that they've seen before on some of their other planetary travels. They might like to find out the cause of this, because the ability to induce such a reaction might come in very handy for their invasion plans. So the Zeroids might well take one cohort of humans who spontaneously sublimated, and a second cohort who did not. Then they would perform a case-control study, in which they'd analyze the histories of the two cohorts to see what the sublimators had in common that the non-sublimators did not.

To improve the quality of their results, they might choose to make it a nested case-control study where a particular variable is added to focus both cohorts. Perhaps the invasion coordinators are most concerned with guys from Montana, so to make the study more efficient for their purpose, they don't bother to study dancers from Ireland or weavers from India, but narrow both groups to just the target variables.

There are two final types of observational studies that we're going to gloss over superficially, as we're doing with all of these. They have to do with time. The ship from Planet Zero may have limited time or lots of time, or they may be interested in one type of data or another. These two form a crosshairs in time. A cross-sectional study looks at only a single snapshot in time, across all subjects. This would give the Zeroids a data set showing how many humans are astronauts at the given moment, how many are from Ecuador, how many are not yet sublimated, and how much radiation each shows signs of having been exposed to. Perpendicular in time to the cross-sectional study is the longitudinal study, in which multiple observations are made over time of the same subject or subjects.

By running all of these different observational studies, the Zeroids were able to make and confirm a number of conclusions. It turned out that being an astronaut and being exposed to extraterrestrial radiation, especially in the earlier, less-protected space capsule designs, greatly increase humans' chances of suddenly bursting into disassociated molecules at some future point in their life. But what nobody knew was that as the Zeroids were making these quiet, non-interventional observational studies, another more terrible species was making experimental studies. They were the Nulliton.

Experimental Studies

...in which we don't merely observe; we intervene to see what happens.

The Nulliton were far less patient than the Zeroids. They were not content to merely observe, and their quasidimensional starships allowed them to bring more resources to their study. They had the means and the desire to perform direct interventions, not only on cattle, but on humans as well.

The Nulliton wanted a quick and easy way to destroy all life on Earth. Based on their many experiences on other worlds, there were any number of ways to accomplish this. Weapons could be used, but that's expensive and time consuming, even with their newest Moebius matter translators. Better if it were possible to catalyze natural processes that produce conditions under which life could not endure. To find out which would work best on humans, they contrived to test.

The most familiar experimental study, and the gold standard, is the randomized controlled trial. In this, the Nulliton gathered humans from all over the Earth, selected randomly, and brought them into their saucer's lab by lifting them in a bright beam of light. They decided to create three test groups: humans left alone, humans warped into nonexistence with the matter translator, and humans exposed to conditions similar to a supernova. Subjects were randomly assigned into one of the three groups, and everything was blinded so that the statisticians didn't know anything about each subject's age, background, etc.

A variation on this might be the non-randomized comparative trial. This is a way to see what happens when multiple interventions are performed, and the idea is to compare the results of one intervention with subsequent ones. The first trial is likely a randomized controlled trial, and then followup interventions would be non-randomized comparatives.

These Nulliton elected to perform a slight variation on the gold standard: the before and after study, in which the precise condition of each human subject was measured before and after the intervention. In a regular controlled trial, some of the humans may have been already sick or about to explode for some other reason; the before and after study allowed the Nulliton to better measure exactly how much damage was inflicted by their experiment itself. If one particular subject had 0 damage before, and another had +5 damage, then having performed the before and after let the Nulliton know how much damage was pre-existing.

The study could have been controlled or uncontrolled. The Zeroids had found that astronauts were more likely to be damaged than Ecuadorians. The Nulliton could have controlled for this factor, by counting astronauts and Ecuadorians separately; or their analysis could have been uncontrolled for this variable, by lumping everyone into a group and only looking at the cumulative data. Either a controlled or uncontrolled study might have been best for the Nulliton, depending on whether they intended to try and inflict extra damage on those who were least susceptible, or whether they wanted a single attack type to take care of everyone.

Using historical controls would have been another option. Say the Nulliton had discovered through observation that astronauts typically explode 1/10 of the time after a spaceflight. They can use this as a historical control, against which they can compare their experimental group, all of whom get exposed to the matter translator, in which 90% of astronauts explode. Since they have the historical data, there's no need to run a current control group.

They could also run this using external controls, which is the employ of controls that are outside of the study. For example, they could look at astronauts currently flying on missions, and watching how many of them explode afterward. Their choice might be based on the availability of the historical records, or based on the expense and resources needed to make the external observations, but either way that control group is compared to their experimental group. Either way, the Nulliton discover that their weapons systems should have no trouble disabling the bulk of the human race.

But anyway, what ended up happening is that the Zeroids and the Nulliton both arrived about the same time from opposite sides of the Earth, and as soon as they detected one another, they opened fire and fell victim to Mutually Assured Destruction. The few survivors who floated to Earth in their escape capsules quickly died of viral infections, and thus the Earth was saved.

It all goes to show that even with the best study design, there's no guarantee that your intervention is going to work, or that your observational studies will end up matching the eventual outcome. Had either the Zeroids or the Nulliton listened to Skeptoid, both might have been saved.

By Brian Dunning
Follow @BrianDunning