SPSP Transparency Efforts for 2020 Convention

Part of my mission as convention committee chair is to improve transparency surrounding the convention. My hope is to give people a peek behind the scenes so members have a better sense of how convention is run and how decisions are made. Accordingly, the convention committee developed the following information, describing how science submissions are evaluated, the location for conference is selected, and registration costs are determined respectively. Please share this information with other SPSP members or attendees so we can spread this information widely.

1) Have you ever wondered how SPSP science submissions are evaluated? Below we describe the evaluation process in an effort to improve transparency surrounding the convention.

All submissions are evaluated by 3 independent reviewers. Who are those reviewers and how are they selected?

• Anyone with a PhD can self-nominate to be a reviewer. During the nomination process, they indicate their areas of expertise using keywords
• The symposia and single-resenter co-chairs of the selection panels decide on the final pool of reviewers from this self-nominated group, seeking to ensure a range of fields of expertise
• Once the submission date has passed, there are two strategies used for assigning reviewers to specific submissions, depending on the type of submission
• Symposia submissions are matched to reviewers using keywords, primarily matching the first keyword listed for the symposia and the first keyword listed by the reviewers, sometimes using the second keyword if necessary. During the matching process, we also attempt to give each reviewer an approximately equal review-load.
• Single presenter submissions (e.g., posters and single presenter talks) are assigned randomly to reviewers because the quantity of these submissions is too great to use a matching process

All submissions are evaluated on a set of criteria pre-specified by the science committee. What are those criteria and how are they used?

• Reviewers are blind to the authors’ identity when they evaluate the submission, and thus they do not evaluate the submissions based on the names of the people who are involved
• Reviewers consider several dimensions when evaluating submissions:
  • Importance: Does this symposium address a question or set of questions that substantially advances our knowledge of a theoretical and/or applied issue in social and/or personality psychology?
  • Strength: Does this research reflect best practices in research, including issues of statistical power? Are studies well-designed to answer the research question(s)? If the session includes applied or non-empirical talks, do these present strong arguments or clear evidence toward the goals of the session?
  • Novelty: Does this symposium represent the “cutting edge” of psychological science? Will the audience feel that they have learned something new from this symposium?
  • Interest-value: Will this symposium session cut across subfields in an integrative way, have a clear impact on future conversations about social and/or personality psychology, or otherwise be likely to be well-attended?
• Reviewers provide a single holistic evaluation on a scale of 1-4. For symposia, these levels correspond to “weaker”; “good”; “very good”; and “exceptional”, and reviewers are instructed to have 25% of their ratings fall within each category (i.e., a rectangular distribution). For single presenter submissions, these levels correspond to “unacceptable/should be rejected”, “weak”, “good”, and “exceptional”.
• The review process is independent each year, so the reviewers only use information from this year’s submissions; the content of prior conferences does not factor into the evaluation process. The only exception is if there is a lot of feedback during the post-convention survey that people want more of a particular topic. In these instances, the committee might try to have more of that topic represented during the following years conference.
• When making the final decision about which submissions to accept, the selection committee primarily relies on the rubric scores from reviewers. In addition, at this stage the submissions are unblinded. This allows the selection committee to ensure that a diverse range of speakers are represented in the program, both in terms of the content of the talks and the demographics of the speakers (for the demographic data we have on potential speakers). A broad goal in making the final selection is to accept high quality submissions and also create a balanced and diverse program. There are no strict quotas in place that X number of submissions from topic Y need to be accepted, although there may be general targets for specific topics to make sure various subfields are adequately represented. Beyond that, any topic-related themes that may emerge do so organically based on having a high number of high quality submissions that year.
• Single-paper submissions that receive the highest scores from the reviewers are examined carefully. In many cases, several high-quality submissions will cluster together in terms of content; these will be collected into a symposia and the presenters will be invited to work together to select a chair, title, and description for the program. The other highest-rated submissions that don't cluster into a symposium set will be included in one of the several data blitzes scheduled during the conference (if the speaker is eligible for a datablitz).

How is the final symposium schedule determined?

• First, the SPSP staff create a grid that doesn’t have overlap at the same time for the same primary keyword
• Second, the convention committee looks at the grid and makes changes if they see symposia with overlapping content areas scheduled at the same time
• The SPSP staff also sends a list of all selected symposia for that year’s conference to symposium chairs and has them indicate if there is overlap with theirs and others. They have a quick turn around to identify 2-3 that might have topic overlap. When making the final grid, we also take this information into account

What are historic rates of acceptance for prior conferences? Since we have significantly revised our submission evaluation process in recent years, we provide data for 2018 and 2019 only.

2) Have you ever wondered how the SPSP convention committee and SPSP board select the location for each conference? Below we describe the selection process in an effort to improve transparency surrounding the convention.

Selecting a location is a multi-step decision-making process that originates at the SPSP board meeting every year. The entire board (e.g., the president, president-elect, treasurer, etc) and all SPSP committee chairs have a lengthy discussion about location. They take many factors into account, described below, and vote on top locations. If needed, the SPSP staff then investigate further to get the information needed to choose between the locations that received the most votes. The convention committee then makes the final decision about where the conference will be held.

In making a decision, we take into account a million different factors! The size of our conference limits our options quite a bit right off the bat. We are large enough that many cities can’t accommodate us, but we are simultaneously too small for some of the really large venues. Out of the options that can accommodate our size, cost is also a really important factor, since this is a critical ingredient of whether people can attend. We have to consider both the cost of the conference itself (e.g., reserving the space), and the cost it will require for people to travel there. Unfortunately, sometimes those don’t match up, and a place that would be cheap to reserve would be expensive to fly to or vice versa. This also means that we don’t want to only have the conference in one location (e.g., San Diego), because that means traveling is more expensive for people on the East Coast than the West Coast. So, we try and move from west to central to east coast for this reason. Another important factor in selecting the location are the laws in each state. For example, some states have anti-LGBTQ laws in place that would directly affect LGBTQ members in attendance (e.g., bathroom bills), and we will not have a conference in a location that is unfriendly towards attendees. We also take into account (in no particular order): weather, accessibility to international members, marketability, and any special opportunities that may exist in a city.

Needless to say, once we have taken all of these considerations into place, we are left with relatively few potential options. This is a complex process, and a lot goes into the decision for each year.

3) Have you ever wondered how we determine the registration cost for SPSP? Below we describe the process in an effort to improve transparency surrounding the convention.

Registration costs are determined by a wide variety of factors as we attempt to balance affordability for members (i.e., the cost members will pay) with needs to pay for the conference itself (e.g., reserving space, catering, etc). The cost of running a conference rises every year with inflation and a booming convention industry. However, we did not raise registration rates between 2013-2019 to keep the conference as affordable as possible. We strive to keep registration rates as low as possible each year. In fact, the society often loses money each year by hosting the conference in order to keep the cost as manageable as possible for members.

Lisa Jaremka
Chair, SPSP Convention Committee