Experimental Design

Four parts of a well-designed experiment

Suppose a teacher wants to know whether a 10-minute walk just before a math quiz raises scores. Sixty students volunteer. A well-designed experiment has four features.

1. Comparison of at least two treatment groups. Here, students either walk or sit quietly for the same 10 minutes. The sitting group is the control group, a group created for comparison.
2. Random assignment of treatments to experimental units. Number the students from 1 to 60 and use a random number generator to pick 30 different numbers. Those students walk, and the other 30 sit.
3. Replication: more than one experimental unit gets each treatment. With 30 students per group, a difference is much less likely to come down to one or two unusual students.
4. Direct control of other possible sources of variation in the response, by keeping them the same for every unit: the same quiz, the same room, the same time of day, the same walking route.

Control groups, placebos and blinding

A control group may receive a treatment that differs from the one being tested, such as a placebo, an inactive treatment that looks like the real thing. Then both groups believe they were treated, and believing alone cannot explain a difference between them. The placebo effect is the difference between the average response to a placebo and the average response to no treatment at all.

Blinding keeps expectations out of the results. In a single-blind, or single-masked, experiment, the participants do not know which treatment they received but the researchers who interact with them do, or the other way around. In a double-blind, or double-masked, experiment, neither the participants nor the researchers who interact with them know. The walking study cannot blind the students, who know whether they walked, but a second teacher, who gives the quiz and grades it without being told who walked, can be kept unaware, which makes it single-blind.

Why random assignment matters

An extraneous variable, or extraneous source of variation, is known or believed to affect the response but is not one of the explanatory variables being studied. For the quiz, the amount of sleep the night before, earlier math grades and test anxiety all qualify.

The purpose of random assignment is to make the treatment groups as similar as possible on every extraneous variable, including the ones nobody thought to measure. When it works, the distribution of each extraneous variable is about the same in every group. Here is one random split of the 60 volunteers, compared on the hours they slept the night before.

Confounding in an experiment

A confounding variable in an experiment is related to the explanatory variable in a way that makes it hard to tell which of the two is changing the response. Suppose the teacher let students choose whether to walk. The walkers might be the energetic morning people who would have scored higher anyway, and energy would be tangled up with walking. Or suppose every walker took the quiz at 8 a.m. and every sitter at 2 p.m.; then time of day is confounded with the treatment. Random assignment and direct control are what keep the potential for confounding low in a well-designed experiment.

Three designs

In a completely randomized design, the treatments are assigned to the experimental units completely at random. The walking study is one. The groups are often the same size, but they do not have to be.

Sometimes you know in advance that a particular variable will affect the response. A blocking variable is such a source of extraneous variation. In a randomized block design, the units are first sorted into groups called blocks, so that the units within a block are alike on the blocking variable. Then the treatments are randomly assigned within each block, so that every treatment appears in every block. Blocking separates the variation caused by the blocking variable from the rest of the variation in the response, which makes the comparison of treatments more precise: inside a block, the treatments are compared on similar units.

A matched pairs design is a randomized block design with only two treatments. Units are paired by matching them on variables that affect the response, and a random choice decides which member of each pair gets which treatment. Alternatively, each unit receives both treatments in a randomly chosen order, such as a runner testing two brands of running shoes, one brand each week, with a coin flip deciding which brand comes first.

Which design fits depends on the goal of the study, the experimental units and the variables involved. If nothing known in advance divides the units, a completely randomized design is enough. If a known variable affects the response, block on it. If there are only two treatments and the units can be matched, or can each receive both treatments, a matched pairs design compares the treatments within pairs of similar units, or within the same unit.

What the results can show

Random assignment of treatments to experimental units is what allows a cause-and-effect conclusion, because it reduces the potential for confounding variables. If the treatment groups differ clearly in their response, the treatments can be credited with causing the difference.

Who the conclusion applies to is a separate question, answered by how the units were selected. It is often unethical or impractical to randomly select people for an experiment, so most experiments use volunteers. Their results then apply to units similar to the volunteers. The walking study could show that walking caused a change in quiz scores, but only for students like those 60.