Evaluation and Selection Process for Symposia and Single Presenter Submissions
Have you ever wondered how SPSP science submissions are evaluated? Below we describe the evaluation process in an effort to improve transparency surrounding the convention.
All single-presenter symposia, data blitzes, and symposia submissions will be evaluated by four or more independent reviewers. The posters (both single presenter and undergraduate), research spotlight, and round table unconference submissions will be reviewed by two reviewers. Who are those reviewers and how are they selected?
- SPSP members with a PhD are eligible to self-nominate to be considered as a reviewer. During the nomination process, reviewers indicate their areas of expertise using keywords.
- The Science Program Review Co-Chairs decide on the final pool of reviewers from the self-nominated group, seeking to ensure a range of expertise areas.
- Once the submission portal has closed, all submissions are matched to reviewers using keywords, primarily matching the first keyword listed. We aim to give reviewers an equal number of submissions to review.
All submissions are evaluated on a set of criteria pre-specified by the Science Program Review Co-Chairs. What are those criteria and how are they used?
- Submissions are masked so the reviewers can't identify the authors when evaluating a submission; submissions are not evaluated based on the names of the people involved.
- Reviewers consider several dimensions when evaluating submissions:
- Strength and rigor. Does the research reflect best practices, including issues of statistical power? Are studies well-designed to answer the research question(s)? If the session includes applied or non-empirical talks, do these present strong arguments or clear evidence toward the goals of the session?
- Contribution. Does the submission address a question or set of questions that substantially advances our knowledge of a theoretical and/or practical/applied contribution in social and/or personality psychology?
- Interest-value. Will the submission cut across subfields or bridge basic and applied work in an integrative way? Will it spark or make a meaningful contribution to conversations in social and/or personality psychology? Will the audience think they have learned something new? Are there other reasons to expect it to strongly appeal to social-personality psychologists?
- As a pilot program last year (for 2022 Convention Submissions), we asked people to write these statements during the submission process, but did not give them to reviewers (we required only the symposia, single presenter, and PD submissions to fill this out, posters and Undergraduate posters were exempt from this). We requested the submitters to please explain whether and how this submission advances the equity, inclusion, and anti-racism goals of SPSP. This may include, but is not limited to: The research participants in the sample; the methods used in the research; the members of the research team(s) involved in the work (e.g., background, diversity, career stage, affiliation type); the content of the presentation (e.g., critical theories, prejudice, equity, cross-cultural research).
Both the Convention committee and the Professional Development committee agreed that these statements were helpful in making final decisions.
We are now rolling the DEI statements out as part of the full review process this year (again posters, etc. will be exempt). We will have the reviewers rate the DEI separately. Instructions for the reviewers are below):
Please read and evaluate each abstract with an eye toward the strength and rigor, contribution, interest-value of the submission in light of the general SPSP audience. Separately, please evaluate the extent to which the submission advances SPSP's goal of promoting diversity, equity, inclusion and anti-racism. You can find a statement on those goals here: https://spsp.org/events/equity-anti-racism-goals.
- Equity & Anti-racism:
Evaluate the extent to which the submission advances SPSP's goal of promoting equity, inclusion and anti-racism. To do so, please consider the equity statement as well as the submission as a whole. Submissions advancing equity, inclusion, and anti-racist goals may include (but are not limited to):
- Diverse research participants (e.g., understudied or underserved populations)
- Diverse research methods (e.g., methodology that promotes equity or engages underserved communities or scholars).
- Diverse members of the research team (e.g., those from underrepresented sociodemographic backgrounds, from an array of career stages, from outside the United States, or with professional affiliations that are not typical at SPSP such as predominately undergraduate serving institutions, minority serving institutions, or outside academia)
- Presentation content (e.g., prejudice and discrimination, critical theories, cross-cultural research).
- Rating Scale:
- The system allows you to make ratings on a 3-point rating scale for each dimension:
- 3: Exceptional- The submission clearly and strongly advances SPSP's goal of promoting equity, inclusion and anti-racism
- 2: Satisfactory- The submission slightly to moderately advances SPSP's goal of promoting equity, inclusion, and anti-racism. We expect that this rating will be the most commonly applied rating.
- 1: Not Applicable- The submission does not advance SPSP's goal of promoting equity, inclusion, and anti-racism
- Equity & Anti-racism:
- Reviewers provide individual scoring on each dimension when evaluating each submission on a scale of 1-5. (Note: We use a different review scale for presentation types due to the quantity of submissions and space available for acceptances per submission type.)
- For symposia, these levels correspond to weaker (1), good (2), very good (3), very strong (4), and exceptional (5). Reviewers are instructed to have 25% of their ratings fall within each category (i.e., a rectangular distribution).
- For single presenter submissions, these levels correspond to unacceptable/should be rejected (1), weak (2), good (3), and exceptional (4); reviewers are instructed that only the top 1-2% of submissions should be rated as exceptional.
- The review process is independent each year, so the reviewers use information from the current year's submissions only; the content of prior conventions does not factor into the evaluation process. Exception: If there is a significant amount of feedback during the post-convention survey indicating attendees would like to see more representation around a particular topic, the committee may try to have that topic represented more at the following year's convention.
- When making a final decision about which submissions to accept, the Science Program Review Co-Chairs rely primarily on the reviewers' rubric scores. Also, submissions are unmasked, at this stage. This allows the Science Program Review Co-Chairs to ensure that a diverse range of speakers is represented in the program, both in terms of the content of the talks and the demographics of the speakers (from the demographic data submitted by potential speakers). A broad goal in making the final selection is to accept high-quality submissions while creating a balanced and diverse program. There are no strict quotas in place to specify that X number of submissions from topic Y need to be accepted, however there may be general targets for specific topics to ensure various subfields are adequately represented. Beyond that, any topic-related themes that may emerge do so organically based on having a high number of high-quality submissions that year.
- Single presenter submissions that opt-in to be considered for data blitz and single-presenter symposia that receive the highest scores from reviewers are examined carefully. In many cases, several high-quality submissions will cluster together in terms of content; these will be collected into a symposia and the presenters will be invited to work together to select a chair, title, and description of the program. The other highest-rated submissions that don't cluster into a symposium set will be included in one of several data blitz sessions scheduled for the convention (provided the speaker is eligible for a data blitz).
- We are currently planning for the 2022 convention to include both virtual and in-person sessions.
How is the final schedule determined?
- SPSP staff create a grid that ensures primary keywords do not overlap blocks of the schedule.
- Accepted presenters will be asked if they have any time zone conflicts with the schedule-at-a-glance. When the programming schedule is developed, we will aim to schedule presenters at a more appropriate time.
- The Convention Committee and Science Program Review Co-Chairs adjusts the grid if they see sessions with overlapping content areas that are scheduled at the same time.
- SPSP staff sends a list of all selected symposia for that year's convention to the symposium chairs. The chairs are asked to indicate up to three sessions that should not be scheduled at the same time due to content overlap. This information is considered when creating the final grid.