An introduction to Comprehensive Results in Social Psychology: The Why and How of peer-reviewed preregistration
Comprehensive Results in Social Psychology (CRSP) is our field's first preregistration-only journal. We have now been accepting submissions for just about a year, making this a good time to reflect on how the preregistration process has been going. Given that we have discussed elsewhere the benefits of preregistration to the field and to science in general (see Jonas & Cesario, 2015) here we focus instead on the main benefits for the individual researcher submitting to CRSP, addressing common questions and misconceptions that have come up along the way.
Quickly, first the basics on the peer-reviewed preregistration process. (For more details, see the CRSP editorial or the Submission Guidelines for Authors) Authors submit an introduction, proposed methods, and proposed analyses. This gets peer-reviewed like any other submitted manuscript. If the review process is positive, the author is given an In-Principle Acceptance (IPA), meaning that the manuscript is guaranteed to be published if the proposed protocol is followed, regardless of the outcome of data analysis.
Here we highlight three major benefits for the individual researcher submitting to CRSP.
The first is that the peer-review process provides you with feedback from fellow experts in the field before you collect data. We have all had the experience of a review that reads like a laundry list of everything the reviewer thinks we should have done differently. With the traditional publication process, many reviewers start with the question, Why should this research be published or not published? and, with this in mind, review the submitted paper with the primary goal of identifying every problem that makes the research unpublishable.
In contrast, reviewers at CRSP serve a different role, one that is much more enjoyable and beneficial for you, the researcher. CRSP reviewers have in mind the question, What is needed for this research to meet its full potential? This certainly does not mean that reviewers are uncritical--your research will be criticized and, if its "full potential" is not good enough, it will be rejected! The difference is that, at CRSP, criticisms are made with the explicit goal of increasing the quality of the research. This allows you to modify the research process while there is still time, before the costs of data collection have been incurred. Indeed, we have so far seen much more contact between editors and authors compared to what one finds in a typical journal, with a lot of "back and forth," questions, suggestions, etc. We welcome wholeheartedly contact with authors!
Perhaps more important, the second major benefit for the researcher is that you can finally explore and make discoveries like you dreamed of doing when you became a scientist. No more pretending that you predicted this interaction but not that one, no more trying to coax p-values to meet some arbitrary publication threshold, no more trying to hide messy results or burying unsupportive studies in the file drawer. You are engaged in the scientific process in a more pure form, in which you set up conditions that allow for scientific discovery and advance, and off you go! This is really quite a freeing sensation for many in the field and puts the excitement of discovery back into the research process.
And finally, at CRSP, this process of scientific discovery is rewarded as it should be. If reviewers agree that the scientific advance is meaningful and the methodology sound, the researcher is recognized and rewarded for his or her scientific acumen. More broadly, preregistration is getting supported these days, more and more, by associations beyond CRSP. For example, the European Association of Social Psychology (EASP) awards grants to run your studies (up to 1000€, click here for more details), once you have been given an IPA at a journal such as CRSP.
But What About...
Learning about preregistration for the first time usually prompts a number of questions. One of the most common is, What are review criteria for a preregistered study? In truth, this is analogous to asking, How can someone review a manuscript without knowing the answer in advance? The preregistered manuscript gets reviewed along the same criteria that we as scientists claim to value: The degree of scientific advance from the outcome of the proposed research and the quality of the methodology and analysis.
Another common question to arise is, Is CRSP only for confirmatory experimental lab studies? Notwithstanding the nuances of different types of research, a general answer is: If it can be published, it can be preregistered and reviewed. Typical confirmatory research studies, meta-analyses, secondary data analysis, and studies that allow for discovery without any clear confirmatory predictions can all be preregistered, reviewed, and published.
Two Common Misconceptions
Having spoken by now with many researchers about CRSP, we have heard two concerns frequently raised that are, in our view, misconceptions of the process.
The first misconception generally comes in some form of the question Does CRSP inhibit discovery? or, How can I explore my data and learn from it if I have to preregister? First, CRSP does allow for exploratory analyses and non-preregistered studies to be included in the final manuscript; these are simply marked as exploratory. Second, one can always use an initial set of exploratory analyses as pilot data for subsequent, preregistered confirmatory studies. Indeed, this is very much encouraged (though not required) in the initial submission. Finally, though, we believe the exact opposite is true about CRSP. As we described above, it allows for more, not less, discovery because you are not constrained in publishing and caring about only "statistically significant" effects.
The second misconception is the idea that it is overly burdensome to submit to CRSP. It is true that there is some initial work that must be done prior to data collection, including writing the introduction, methods, etc., and waiting for peer review. But keep in mind that you expect to do this work anyway. Certainly if you think the data collection is worth the resources and effort, you believe the research to be publishable and so you will eventually have to write an introduction and methods section. The difference is that you will know, up front, whether such efforts will likely result in a publication. That's a strength, not weakness, of CRSP.
However, we do recognize that CRSP is a different way of doing the publication process. Therefore we suggest that researchers simply start with preregistering a single study or set of studies, rather than trying to transform your whole lab overnight. We believe you will find that it is not as burdensome as you may initially think, and indeed all faculty researchers have already done this--you did exactly this for your dissertation.
For More Information:
For more information about CRSP and the preregistration process, you can visit the CRSP website, watch the video above, or email us. You can also find us at the upcoming SPSP convention in San Diego and talk in person. We welcome questions and contact from authors!
Jonas, K.J. & Cesario, J. (2015). How can preregistration contribute to research in our field? Comprehensive Results in Social Psychology, DOI: 10.1080/23743603.2015.1070611