Replication Through a Kaleidoscope
When people hear that we are conducting a large international replication of the famous Loftus and Palmer (1974) study (see my earlier posts on this topic), they often assume the task is straightforward: find the original materials, follow the original procedure, collect new data, compare the results, and you’re done!
I probably thought something similar before starting this project.
What I have discovered instead is that reconstructing a classic experiment feels less like following a recipe and more like looking through a kaleidoscope. Every time you turn it slightly, a new pattern emerges.
The original study seems very simple and this is also how it is described in textbooks or talked about in intro psych lectures. Participants watched a traffic accident and were later asked a question. Some were asked how fast the cars were going when they “hit” each other. Others were asked how fast they were going when they “smashed” into each other. The wording affected the estimates people gave and, in a later experiment, even affected what they remembered.
Simple enough, right?
Right. Until you start asking questions.
Recently, I interviewed Elizabeth Loftus for my podcast Research as it Happens, where I document this project as it develops—from the first ideas to the eventual results. One thing I wanted to know was how closely we should try to recreate the original study. That sounds like a practical question, but it quickly turned into a theoretical one.
For example: how fast were the cars actually going? And were they both going the same speed, which the question seems to imply? Should we focus on one of the two cars, depending on the video we use? In a conversation a few days ago (some weeks after the podcast recording), Elizabeth and I watched some candidate videos for the replication and agreed that focusing on one car might be the best idea.
More than fifty years have passed since the study was conducted, and Elizabeth no longer remembers a lot of the details. And who could blame her this many years and experiments later? I’m sure I’d do a lot worse, if I even make it to the stage where I have experiments that I conducted over 50 years ago.
What Elizabeth does remember, though, is that it was a relatively minor accident. That answer immediately generates more questions. How minor? Ten miles per hour? Twenty? Thirty? And does it matter? I bet it does. Thirty miles per hour seems pretty fast for a minor accident.
But the detail that surprised me most concerned the famous broken-glass finding.
Most textbook summaries give the impression that participants later remembered broken glass that was never present in the scene. During our podcast conversation, however, Elizabeth recalled something that was news to me: although participants did not see broken glass, they may have heard the sound of breaking glass.
Suddenly, the study looks a little different. Perhaps participants were not constructing a memory entirely from suggestion. Perhaps they were integrating information from multiple sources: what they saw, what they heard, and what the later question implied.
A single detail opens an entirely new vista. Should a replication preserve that sound? Would the effect be different without it? Should we systematically vary the absence vs. presence of the sound? Does the finding tell us something about false memories, source monitoring, or the way people combine information from different modalities into a coherent recollection?
And this is just one example.
Another concerns perspective. From where did participants see the accident? From the side? At an angle? From another vehicle? Perspective influences perceived speed, attention, and memory. Once again, a seemingly simple detail becomes a scientific question.
Then there is language, which, as it happens, is the topic of an upcoming episode of Research as it Happens.
The original study was conducted in English. Our project will be conducted in about 20 languages. How do you translate “hit” and “smashed” in each of them? Not linguistically, but psychologically.
The issue is not whether a dictionary provides an equivalent. The issue is whether the translated word conveys the same sense of force and severity. A perfect linguistic translation may be a poor psychological translation.
And here is another wrinkle: the original participants estimated speed in miles per hour. Many of ours will use kilometers per hour, something we hadn’t thought of—even though we are metric people ourselves— until very recently, in a conversation with Lisa de Bruine and Anita Eerland.
At first glance, the distinction between miles and kilometers seems trivial. But psychological judgments are informed by experience. A speed of 50 miles per hour may not evoke exactly the same intuition as 80 kilometers per hour, even though they describe roughly the same physical reality.
But how should we measure this? If we use a slider, 80 looks a lot faster than 50—regardless of whether we're talking about kilometers or miles per hour. Lisa suggested using a slider with both scales displayed on it. The idea was so good that we promptly made her head of our statistics team!
Each question seems small. But if you start putting them together, they reveal something important.
Replication is often portrayed as an attempt to reproduce the past. In practice, it is also an attempt to understand the past. The process forces us to think carefully about which aspects of an experiment are essential for testing the hypothesis and which are incidental.
The closer we look at Loftus and Palmer, the more the study seems to unfold into new questions. What initially appeared to be a simple experiment becomes a landscape of inferences and decisions.
Maybe this is part of the value of replication. It doesn’t just tell us whether a finding holds up, but it helps us understand what the finding really was to begin with.
Or perhaps that's just what the world looks like when you spend enough time inside a fifty-year-old experiment.