Attending a psychology conference this month, I was struck by an unsettling trend in social-science research. The abstract book revealed that nearly all of the 1,500 posters and talks related to some kind of confirmatory work — using statistical analyses to test an existing hypothesis. Hardly any reported the sort of open-ended exploratory research needed to come up with those hypotheses.
In my view, it’s crucial for the social sciences to embrace a research culture that balances hypothesis testing with hypothesis generation, embracing exploratory studies alongside confirmatory ones.
Replication games: how to make reproducibility research more systematic
To understand the difference between exploratory and confirmatory studies, consider a research group that is interested in links between children who watch violent films and aggressive behaviour. In an exploratory study, the group might ask parents about the children’s aggressive behaviour and collect information about household habits, individual characteristics and media content that could influence the children. The authors would inspect the data in many ways, looking for any patterns. The patterns that emerge, alongside theoretical background knowledge, could then be used to build specific predictions — for instance, that parents’ explanation of violent content might decrease media-induced violence in adolescents, but not younger children.
Confirmatory studies can seek to support or question these hypotheses using a fresh data set. To conduct a confirmatory study properly, best practice dictates that researchers outline their analysis plan before beginning their work, often through preregistration — recording it in a public repository. Preregistration helps to prevent researchers from constructing their hypotheses and analyses after peeking at their data. In the example above, for instance, if a group’s analyses do not confirm the general hypothesis, the researchers might be tempted to tweak the type of analysis they perform until it shows a statistically significant effect for, say, early adolescent boys, after controlling for family income. This broadly condemned practice opens the door to false-positive findings.
Like many in the social sciences, I was trained to think of hypothesis testing as the main job of a researcher, with exploration hardly worthy of publication. But I changed my mind as I began to lose faith in my own ability to preregister my studies properly. I wholeheartedly believe that preregistration is crucial to transparent science. But often, I would find that I hadn’t done enough exploration to plan my hypothesis-testing analyses properly.
Reward research for being useful — not just flashy
For example, how could a researcher tackling the aggression question know, without exploration, which of many variables — age, gender, socio-economic background and more — might be linked to behavioural change, and how? Similarly, in my own work, I’ve found myself asking: should I have devised my statistical model differently? Would a different analysis bring different conclusions? Did I miss something more interesting in my data? I felt unable to let my data speak for themselves. I should have done more exploration before committing to my confirmatory analysis.
Yet, despite the importance of exploratory studies, they are not valued highly enough. Hundreds of journals promote registered reports — a way to preregister confirmatory research with the journal — yet exploratory science is rarely endorsed. And in the current reproducibility drive, with questions raised about how to make the social sciences more rigorous, robust and reproducible, most solutions have involved improving how we test our hypotheses, with little thought given to exploratory work.
Source link
