Peering Behind the Curtain of the SIOP Program Building Process

Scott Tonidandel, SIOP 2020 Conference Chair, & Elizabeth McCune, SIOP 2020 Program Chair

The purpose of this article is to shine some light on various aspects of the program building process so there is more transparency about what actually happens. We have organized this article around a series of questions that we are often asked in our roles as conference chair and program chair.

How does the acceptance decision get made?

Each year, the SIOP program is assembled by the SIOP Program Chair executive committee, which consists of the outgoing program chair, the current program chair, and the incoming program chair. This committee sets cut-score thresholds based on the reviewer ratings for the different submission types (types are symposiums, panels, debates, etc.). These cut scores are established to balance a variety of constraints, such as the amount of available program time, need for 50 versus 80-minute sessions, desire to have a diversity of formats, etc. Once those cut-scores are established, any session above the cut-score for that submission type will be accepted, and any submission below that score will be rejected. No other factors are considered in making the acceptance decision. This ensures a certain degree of fairness in the process as all submissions of the same type are treated equally. Importantly, the program chairs don’t have any discretion to accept individual sessions that are below the cut score. In the end, the acceptance decision comes down to whether the session was evaluated favorably by reviewers.

The reviewers’ comments seem positive; why was my session not accepted?

The acceptance decision is entirely driven by the reviewers’ ratings and not the comments themselves. With over 1,400 submissions and 6,000 reviewer comments, it is impossible for this three-person committee to individually vet submissions and reviewer comments. Our hope is that reviewers provide constructive comments to the submitters that also reflect their numerical ratings, but in terms of the acceptance decision, the reviewers’ numerical ratings drive that process.

What sessions don't go through the peer review process?

There are a very limited number of invited sessions that do not adhere to the above process. The incoming program chair is responsible for organizing the five sessions that comprise the presidential theme track. Similarly, there are separate committees that are responsible for the 6 Friday seminars, the 12 communities of interest, and a special sessions committee that is allocated 5 hours of program time. The Alliance of Organizational Psychology (a strategic partnership between SIOP, EAWOP, and ICAP) is granted 4 hours of program time, and the SIOP Executive board can use up to 6 hours for sessions related to the business of SIOP (e.g., conversation hour with SIOP leadership). Finally, certain distinguished award winners (e.g. Distinguished Professional Contributions and the Distinguished Scientific Contributions) are given program time. Though these sessions don’t go through the same review process as a regular submission, they are all vetted by a committee. What is important to recognize here is that with over 900 accepted submissions and 450+ hours of programming over 3 days, these sessions represent a tiny fraction (approximately 3%) of the conference program.

How can I participate in an invited session?

Outside of the invited sessions that are reserved for award winners, anyone is welcome to contribute ideas for any of the session categories above. The process for doing so would be to contact the program chair who could then connect you to the appropriate subcommittee chair. However, what is important to realize is that these sessions are often determined well in advance of the submission deadline. All of these committees actually started their work at the conclusion of the prior year’s conference. These sessions are largely pinned down by the time the call for proposals is released in July. If you have an idea for a session that you want to contribute for the next SIOP conference, you must begin those conversations well in advance of the typical submission cycle (usually during or soon after the SIOP conference from the prior year).

Because submissions are accepted based upon reviewers’ scores, how are reviewers assigned?

Four reviewers are assigned to every submission, and no more than one student reviewer is assigned to a submission. There is always one expert reviewer on each session. Reviewers are assigned based on session content match to reviewer background (as indicated by the reviewer upon registering to volunteer for this service).

Why are there so many sessions on topic X, but so few sessions on topic Y?

The program is not designed to have a certain level of representation of content areas. Acceptance decisions are driven entirely by reviewers’ ratings of submissions and are independent of the content area. If reviewers provide high ratings to lots of sessions on a particular topic, those sessions will be accepted regardless of how many other sessions on that topic are also accepted. If there aren’t very many sessions in the program on a particular topic, this means that we either didn’t receive very many submissions on that topic or that the submissions received low ratings by the reviewers. Essentially, the SIOP audience (the submitters and reviewers) determine the overall content of the program.

Why was I put in that room?

When sessions are assigned to rooms, an attempt is made to match room size to perceived demand. Reviewers, when rating submissions, are asked to indicate whether a session should be placed in a small, medium, or large room. The conference meeting spaces are similarly categorized as small, medium, and large, and attempts are made to match the reviewers’ ratings to the size of the spaces.

Why did I get that timeslot? Why are there two or more sessions on a similar topic at the same time?

Outside of a few obvious exceptions (theme track, Friday seminars, etc.), the process of assigning sessions to time slots is largely pseudo-random. A computer algorithm assigns all of the accepted sessions to time slots with the constraint that there cannot be any presenter conflicts. This algorithm also considers other variables (e.g., reviewer estimates of how large an audience the session may draw, matched to appropriate room size) when doing the assignments, but the only constraint that is strictly enforced is the presenter-conflict constraint. The program chair committee does screen for obvious content conflicts and tries to reduce them when possible, but it is impossible to eliminate all content conflicts. For example, with over 100 accepted sessions on diversity and inclusion, there is no way to create a schedule that doesn’t have multiple diversity-and-inclusion sessions happening at the same time. Additionally, though we may wish to relocate a session, oftentimes we are unable to because moving it would conflict with other features of the program such as presenter conflicts, room requirements, and so on.

Why are there so many sessions at the same time?

Having multiple sessions at the same time serves two primary goals: it maximizes the number of sessions that are accepted, which in turn helps to increase the likelihood of a diverse range of sessions. Every year, the SIOP conference sets a new record for conference submissions. Moreover, SIOP attendees seem to be more diverse in terms of their backgrounds and interests. In order to accommodate the increasingly large number of submissions and to try to balance the very different interests of our attendees, we offer a large variety of sessions throughout the day.

Why wasn’t my special request approved?

When considering requests from submitters (e.g., a specific room layout), one of the guiding principles is fairness. We want to make sure that we treat all submitters equally. Thus, a request from the SIOP president will be treated identically to a request made by a first-time SIOP submitter (sorry, Eden). There are a few notable factors that sometimes restrict our ability to honor particular requests. One factor is the clarity of the request provided by the submitter in the submission form. Given the volume of the submissions, we need to rely on what is in the submission form, and at times the request cannot be approved simply because what is being requested is not clear. Another factor is budget. For example, the cost of changing the configuration of a single room is exorbitant. Yet another factor may be our contractual obligations with the hotel and/or audiovisual services. We can assure you that the program chair carefully considers these requests and makes every effort to approve those that are clear and within the bounds of our budget and contracts.

What is being done to accommodate the increasing number of SIOP attendees?

The process of selecting a conference site usually happens 6 years in advance of the conference. As we have seen, attendance at the conference has grown considerably. Although SIOP tries to forecast the anticipated growth of the conference, these attendance estimates are far from perfect. Attendance is expected to grow, but there is considerable year-to-year variability driven by a host of factors (conference location, time in April the conference is held, state of the overall economy, etc.). Because the conference sites are chosen so far in advance, it is impossible to adjust to more recent changes in demand, which sometimes results in space constraints at the conference. These new demands on space can be taken into account when selecting future sites, but because of how far in advance sites are chosen, it will take years for attendees to see these changes implemented.

Because of the continual growth in conference attendance, the site selection committee has started to consider other type of venues (such as conference centers) to hold the conference. SIOP 2019 was a great example of such a venue. The Gaylord National had an attached conference center that was able to easily accommodate what was one of the largest SIOP conferences.

Are data from the Whova app used in conference and program planning?

SIOP 2019 was our third year using the Whova app and usage has grown each of those years. As usage continues to increase and as we are able to customize the app to meet the unique needs of SIOP, the session attendance and rating data coming from Whova will be particularly valuable, but to date are not being used in the creation of the program. As of now, Whova is a great tool for conference attendees to manage their schedules, connect with other conference goers and stay updated on conference events and notifications, and we look forward to leveraging the data in the future to make improvements to the conference experience. We would encourage everyone to consider using the Whova app at SIOP 2020 in Austin!