Task Force on Publication and Research Practices
Narrative Report to the SPSP Executive Committee
June 18, 2013
Summary of Charge
The Task Force on Publication and Research practices was charged to make recommendations to the EC for publication policies and research standards to improve the reliability of research in social and personality psychology. It was also specifically charged with making recommendations on how to handle a number of articles published in PSPB that were co-authored by individuals who had other articles, in PSPB and elsewhere, retracted on the basis of apparent research fraud.
The Task Force was chaired by David Funder (2013 SPSP President). The other members were John Levine, Diane Mackie, Carolyn Morf, Carol Sansone, Steve West and Simine Vazire. (Mackie is the current chair of the Publications Committee and Sansone and Levine are the other members; Morf is past chair of the Committee). Wendy Wood was originally appointed to the Task Force but because of an injury was unable to attend the meeting.
The Task Force met in Santa Monica, California, on April 26-28, 2013. We were joined on the morning of April 26 by Sara van Valkenburg, the Corporate Counsel for Sage Publications. The opportunity to meet and discuss these complex issues in person for an extended period was very helpful.
The recommendations from the Task Force are in four categories: (1) Publication of a report in PSPB concerning non-retracted PSPB articles by authors who had other papers retracted from PSPB and other journals. (2) Changes in the "Instructions to Authors" for PSPB and PSPR, intended to improve the reliability of published research, and publication in PSPR of an associated article on "best practices," authored by the committee. (3) Adoption of an SPSP data sharing policy pertaining to articles published in SPSP journals. (4) Other actions by the EC and other SPSP standing committees to improve the reliability of research in social and personality psychology
1. Non-retracted PSPB Articles
Based largely on research by the previous Publications Commitee Chair, Carolyn Morf, the Task Force prepared a report on what could be learned concerning the status of non-retracted articles in PSPB by three authors -- Diedrik Stapel, Dirk Smeesters, and Lawrence Sanna -- who had other articles retracted in PSPB and and other other journals. Stapel's articles were the subject of extensive reports by two Dutch committees, and the Task Force did not attempt to duplicate their work. Extensive efforts were made to contact all of the co-authors of non-retracted PSPB articles by Smeesters and Sanna, and a draft of the report was also sent to all of them who could be reached for factual review. The report was reviewed by legal counsel for Sage Publications. The Task Force recommends that this report (attached) be published in PSPB as soon as is practical.
2. Changes in Instructions to Authors and Explanatory Article
The Task Force drafted revised "Instructions to Authors" for PSPB and PSPR. The principal changes in these instructions are (1) to make them clearer and more consistent than the currently published instructions and (2) to set standards for reporting of power, effect size, and confidence intervals, as well as methodological transparency. Additionally, the Instructions specifically mention meta-analyses and replications among the types of articles PSPB will consider.
The Task Force also drafted a brief article summarizing "best practices" for research in personality and social psychology and outlining how these practices led to changes in the Instructions to authors. The Task Force recommends that this article be published in PSPR.
The recommended revised Instructions to Authors and the summary article are attached.
3. SPSP Data Sharing Policy
The Task Force recommends that SPSP adopt a formal policy on sharing data associated with publications in SPSP journals. (This proposed policy is cited in the proposed Instructions to Authors.) The recommended text is adapted from APA's policy with some clarifications.
Proposed SPSP Policy:
"The corresponding author of every empirically-based publication is responsible for providing the raw data and related coding information underlying all findings reported in the paper to other competent professionals who seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose, provided that a) the confidentiality of the participants can be protected; b) legal rights concerning proprietary data do not preclude their release; and c) those requesting data agree in writing in advance that shared data are to be used only for the purpose of verifying the substantive claims through reanalysis or for some other agreed-upon use. "
For comparison, the current APA policy is stated as follows:
"8.14 Sharing Research Data for Verification
(a) After research results are published, psychologists do not withhold the data on which their conclusions are based from other competent professionals who seek to verify the substantive claims through reanalysis and who intend to use such data only for that purpose, provided that the confidentiality of the participants can be protected and unless legal rights concerning proprietary data preclude their release. This does not preclude psychologists from requiring that such individuals or groups be responsible for costs associated with the provision of such information.
(b) Psychologists who request data from other psychologists to verify the substantive claims through reanalysis may use shared data only for the declared purpose. Requesting psychologists obtain prior written agreement for all other uses of the data."
4. Other Recommendations to the Executive Committee
The Task Force also formulated a number of other recommendations for action by the Executive Committee and other standing committees of SPSP.
1) The Task Force Recommends that the EC ask the Publications Committee to:
i. Ensure that editors and members of editorial teams for SPSP journals are aware of and supportive of the new publication guidelines.
ii. Recommend to the editor of PSPR that a Guest Editor be recruited to publish a special issue on best practices for research articles, including topics such as statistical power, reporting of research, article writing, meta-analysis, and replication studies.
2) The Task Force recommends that the EC ask the Training Committee to:
i. Consider using on-line resources to spread knowledge on methodological issues that are neglected in the statistical training received by many psychologists. Consider commissioning professionally-produced educational modules to be made available on-line. Examples of appropriate topics include statistical power, effect size, interpretation of replication, and alternatives to Null Hypothesis Statistical Testing (NHST).
ii. Add a plenary course to the SISP summer school, focusing on research practices and research ethics. The intention is that this course would address basic good practices and principles, in order to help socialize the next generation of researchers.
iii. Support workshops at SPSP and elsewhere on methodological issues, research practices and ethics. Support for such workshops (and other means of education) may be available from NIH. For example, see http://grants.nih.gov/grants/guide/rfa-files/RFA-OD-13-009.html. (Letter of intent due date is October 14, 2013, and full proposals are due November 14, 2013.)
iv. Develop guidelines for improved quantitative education in social/personality psychology including issues of effect size, power, replication, Bayesian methods, and other emerging methods.
3) The Task Force recommends that the EC ask the Awards Committee to:
i. Establish multiple new awards to young investigators whose articles serve as exemplary models of best research practices (including, but not limited to, adequate power, clear reporting of effect size, methodological transparency, established replicability). Such awards might be based on single articles or consistent excellence in an early career. They might come from open nominations or a survey of the literature by award committee members. Such awards might help -- to some degree -- to counteract the strong pressure in hiring and tenure decisions to maximize the quantity of research articles, in some cases perhaps at the expense of quality.
4) The Task Force recommends that the Executive Committee:
i. Support "expert meetings" (perhaps by another name) to assemble small groups of knowledgeable researchers to consider and prepare reports, guidelines or articles concerning methodological topics such as replication, power/effect size, and so forth. For example, standards for adequate replication, the basis of deciding that replications have "succeeded" or "failed," and the appropriate venues for publishing replication studies are all complex and controversial issues which deserve detailed discussion and analysis.
ii. Maintain contact with NIMH and NSF program officers and review panels, offering to participate in any efforts they may initiate for improving research practices, as well as other strategic initiatives.
iii. Provide resources to facilitate communication among the editors of PSPB. The large number of "action editors," and the lack of consistent training of them and communication among them, raises the risk of inconsistency in editorial standards and enforcement of methodological guidelines. As PSPB continues to grow this issue is of increasing importance. Communication could be improved through such methods as dinner meetings at SPSP, conference calls, and in-person meetings at SPSP and elsewhere.
iv. Support a follow-up meeting of this or a successor Task Force to assess progress one year from now.
The SPSP Task Force on Publication and Research Practices
David Funder (Chair)
Actions by EC in Response to this Report (excerpt from the Meeting Minutes)
July 17-19, 2013
Based on the prior discussion, the EC endorsed (in a vote of 10-0-0) the article on retractions, with the suggested changes including expanded keywords and all relevant articles listed in the reference section.
Concerning the proposed policy on data sharing, EC members had an extensive debate, raising questions about the feasibility or advisability of sharing certain sets of data and expressing concerns about reinforcing "a growing culture of policing.” It was, however, noted that the proposed SPSP data sharing policy was similar to the APA policy, but with what the Task Force believed was better and more precise wording. David Funder explained that the APA policy explicitly applies to papers published in APA journals, and implicitly applies to other journals that adopt APA publication guidelines. Adopting the proposed policy allows SPSP to "own” the policy that applies to our journals (currently PSPB and PSPR), including the ability to change it later if circumstances warrant. The EC voted to endorse the proposed data sharing policy with a vote of 6-3-1.
Concerning the proposed new Instructions to Authors for PSPB and PSPR, David noted that the Task Force accepted as a friendly amendment the additional wording suggested on Thursday concerning including measures in an appendix. There was further discussion about including a narrative about what materials were used but not included in the appendix, but the EC did not decide to include such wording in the instructions. The EC voted 9-0-1 for the revised Instructions to Authors for PSPB and PSPR to "include in an appendix the verbatim wording of all independent and dependent variable instructions, manipulations, and measures underlying all findings reported in the paper. If the article is published, the appendix will be made available on-line.”
The proposal was amended to include an additional sentence, which the Task Force may refine, that reads: "Examples of factors that may address the choice of N and consequent issues of power include desired statistical power for a given effect size, practical constraints on sample size, empirical precedent, limitations in assessments of power for certain statistical techniques, or issues uniquely relevant to the research context.” The EC voted 9-0-1 to support the proposal as amended.
Two additional votes related to the changes in policy for manuscripts submitted to SPSP journals supported the proposals: (a) that there be a hot link to the explanatory article that provides a context for the requirements and further description of the intent of each point (9-0-1); and (b) that the Publication Committee monitor the effects of these changes on submissions to the journal, the responses of people who submit or consider submitting to the journal, and the views of the Editors (supported by a vote of 10-0-0). The Publication Committee will report their assessment at the EC Winter Meetings in 2014 and 2015.
The EC voted (9-0-1) to endorse the explanatory article accompanying the change in submission guidelines, subject to final revision and editing by the Task Force.
Although not specifically endorsing the content of specific recommendations by the Task Force, the EC supported that the Task Force communicate the additional issues for future consideration to the appropriate SPSP committees (9-0-1).