Reflections on replication

It started with a laptop. Not mine (nor Hunter Biden’s), and not under ordinary circumstances.

I’d been appointed chair of a university committee investigating scientific fraud. One day, we had to retrieve the suspect’s laptop from his home in Belgium. He had a knee injury and couldn’t get it himself. “You’re free to look anywhere in the house,” he told me. I felt very uncomfortable—this certainly was not part of my academic training. I climbed the stairs, went straight to the study, picked up the laptop, and returned to the living room without glancing at anything else.

It feels surreal now, looking back all those years later. But the feeling stayed with me. I had crossed an invisible line—not just into someone’s home, but into a version of science that was no longer built on trust.

It was my unceremonious introduction to the open science movement.

Several years later, I found myself at an open science conference in Charlotte, North Carolina, organized by Brian Nosek and Simine Vazire. The atmosphere was unlike any academic meeting I’d experienced. Most conferences look backward: people present what they’ve already done, and others politely nod along—or, less politely, head to the nearest pub. This one was different in that it looked ahead. It was about building something new. As it turned out, it would become the starting point for what would later be known as the Society for the Improvement of Psychological Science (SIPS).

I joined a breakout group titled “Making Replication Mainstream” with Brent Donnellan and Rich Lucas, both strangers to me at the time, and we began sketching the outlines of a paper. Later, with Alex Etz, we developed it into a full article, which prompted an intimidating number of commentaries—36, if I recall correctly—all of which we had to respond to. The title more or less chose itself. Published in 2018, it became one of the first tangible outcomes of that memorable meeting.

Aside from the genuinely pleasant and constructive collaboration, one moment from that process still stands out to me: Dan Gilbert’s review of our paper. He’s often portrayed as a critic of the replication movement, but his review was one of the most thoughtful, incisive, and generous I’ve ever received.

I have joined and initiated several large-scale replication projects. They were enlightening. It was exciting to work with large and diverse groups of collaborators (methodologists, social psychologists, cognitive psychologists, psycholinguists). It was also exhausting. I remember Anita Eerland and I coordinating with editors Alex Holcomb in Sydney at dawn and Dan Simons in Illinois by midnight. After that, I made a decision I’ve stuck with since: no work outside of working hours.

Looking back, some of those replication projects may have bordered on overkill—dozens of experiments to confirm what a few might have sufficed to show. We might have put some of our collective effort toward more exploratory work. But part of the exercise was to demonstrate just how robust a finding can be when subjected to rigorous testing. You could call it conviction by accumulation.

Meanwhile, I began writing blog posts. Some went semi-viral. I experimented with satire, metaphor, anecdote—whatever helped make the point. Those posts felt personal in a way journal articles never do. I was able to express a part of myself that usually stays hidden behind academic prose.

For a while, it felt like we were building something, a kind of intellectual commons. We were exchanging thoughts, sometimes clashing, and often learning. Over time, the conversation shifted. It became more technical, more about infrastructure. Necessarily so, because this is how progress becomes mainstream. But I also started to feel I had less to add.

I’m not a methodologist. I never really set out to become a meta-scientist—though I briefly was, in my mid-twenties, working at the precursor to Leiden’s Centre for Science and Technology Studies. My core interest has always been comprehension: how people understand language, stories, the world. I eventually returned to that, carrying with me what I’d learned along the way—the value of transparency, of saying “I don’t know,” and of doing the slow, careful work of getting things right.

These days, I’m more likely to be writing about conspiracy thinking than about pre-registration protocols. But I still have the mindset from that period: skepticism, transparency, intellectual humility. It informs how I work and how I teach.

And perhaps this isn’t the final chapter. We’re working on a new replication proposal. If it gets funded, I’ll share it here. I might even ask for your help at that point.

Some replication references

Alogna, V. K., Attaya, M. K., Aucoin, P., Bahnik, S., Birch, S., Birt, A. R., ... Zwaan, R. A. (2014). Registered replication report: Schooler & Engstler-Schooler (1990). Perspectives on Psychological Science, 9, 556–578.

Eerland, A., Sherrill, A.M., Magliano, J.P., Zwaan, R.A., Arnal, J.D., Aucoin, P., … Prenoveau, J.M. (2016). Registered replication report: Hart & Albarracín (2011). Perspectives on Psychological Science, 11, 158-171.

Morey, R.D., [and 44 other authors, including Zwaan, R.A.], (2022). A Pre-registered, Multi-lab Non-replication of the Action-sentence Compatibility Effect (ACE). Psychonomic Bulletin & Review 29, 613–626.

Van Kuijk, I., Verkoeijen, P., Zwaan, R.A., & Dijkstra, K. (2018). The effect of reading a short passage of literary fiction on Theory of Mind: A replication of Kidd and Castano (2013). Collabra: Psychology.

Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., Jr., . . . Zwaan, R. A. (2016). Registered Replication Report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11, 917–928.

Zwaan, R.A., Etz, A., Lucas, R.E., & Donnellan, M.B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, E120. doi:10.1017/S0140525X17001972

Zwaan, R.A., Etz, A., Lucas, R.E., & Donnellan, M.B. (2018). Improving social and behavioral science by making replication mainstream: A response to commentaries. Behavioral and Brain Sciences, 41, E157. doi:10.1017/S0140525X18000961

Zwaan, R.A., Pecher, D. (2012). Revisiting Mental Simulation in Language Comprehension: Six Replication Attempts. PLoS ONE 7(12): e51382. doi:10.1371/journal.pone.0051382

Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., & Zeelenberg, R. (2018). Participant nonnaiveté and the reproducibility of cognitive psychology. Psychonomic Bulletin & Review, 25, 1968-1972.