Sign Up for the SIOP/CARMA Open Science Virtual Summer Series!

Christopher M. Castille, Nicholls State University; & James Grand, University of Maryland

Author Note: I’d like to thank all the members of SIOP’s Open Science and Practice Committee, especially George Banks, Lillian Eby, Tammy Allen, Kevin Murphy, Larry Williams, and Fred Oswald, for providing editorial guidance on this manuscript.

In our last “Opening Up” column, we shared an ongoing discussion regarding the creation of a series of workshops for building open science skill sets. This discussion, which has involved members of both SIOP’s Committee on Open Science and Practice (OSP) and the Consortium for the Advancement of Research Methods (CARMA) has materialized! We are proud to announce the SIOP/CARMA Open Science Virtual Summer Series! By attending the summer series, you will learn critical principles and how-tos of open science practices that can be introduced into your research pipeline as well as learn about the perspectives of journal editors and associate editors hoping to encourage open science practices and enhance the robustness of our work (e.g., Lillian Eby of the Journal of Applied Psychology, Steven Rogelberg of the Journal of Business and Psychology). Equally exciting is that the entire workshop series will be offered free of charge to all SIOP members! We’d like to thank the members of SIOP’s Committee for Open Science and Practice, and especially CARMA’s director, Larry Williams, for helping shape this effort and making it a reality. The event truly fits with the moment that we find ourselves in as a science.

The overall focus of the workshop series is to introduce and teach attendees about open science practices that are widely believed to help researchers produce studies that are better planned and understood by all collaborators involved; more transparent and reproducible; and more accessible, useful, and impactful to the research and practice communities interested in the research. The virtual workshops will be hosted via CARMA’s resources (i.e., Zoom), and attendees can choose to attend any or all virtual workshops.

We’ve established two broad goals for this series. The first is to connect our attendees with leaders in the community who want to practice and promote open science. Such social support will be crucial to making the kinds of productive changes that make our science even stronger and better. The second is to offer sessions that facilitate the development of knowledge and skills for making open science a reality for both research and practice. Example topics covered in the workshop include

• Learning key open science practices that range from using resources available via the Open Science Framework to the smaller and quicker takeaway skills (e.g., how to annotate scripts/analysis code in SPSS, MPLUS, R/RStudio) to make work more independently reproducible
• How to slowly integrate open science into your workflow and overcome common challenges with open science
• How to engage in open science activities when you work with organizational data
• The basics of preregistration and building a registered report
• How to review scholarly works with open science in mind
• How to navigate and implement open science practices in the publication process
• Tips from the editors

Although plans are tentative, we currently anticipate the following structure for the SIOP/CARMA Open Science Virtual Summer Series. Classes will be held each Wednesday for 4 or 5 weeks starting on May 19 and ending on June 16. Sessions will begin at 10am ET and go till 4:30pm ET. The workshops will be divided into morning and afternoon sessions and led by a fantastic slate of speakers, panelists, and former CARMA presenters. Morning sessions will typically consist of skill-building sessions and extend into the afternoon as needed. There will be time each day (breaks, meals) to reflect and network with fellow attendees, presenters, and workshop organizers to promote building SIOP’s open science community. For instance, we’ll break out into small groups, reflect on the practices highlighted that day, and discuss small ways each of us can commit to opening up our work. We’ll discuss questions such as

1. Of the practices highlighted in the workshop, which would you want to use and why?
2. How might you go about successfully implementing the open science practices we’ve covered to make your research even more open or robust?
3. There are many reasons for adopting open science. Reflecting on what was discussed in the workshop(s), what are the best reasons you can think of for using open science practices? How important is it for you to leverage these practices in your research?
4. Do you need any support from this community we’ve built? What do you think you’ll do after leaving the summer series?

After our breakouts, we’ll return to the larger group, sharing stories and ideas about committing to open science. As facilitators, we’ll—where appropriate—highlight some key findings emerging from the meta-science literature that shed light on the value and impact of open science practices where they have been utilized. For instance, a recent study comparing the registered-report publishing format to traditional formats was judged by peer reviewers as resulting in manuscripts that scored slightly better on outcomes such as quality, rigor, novelty, creativity, and inspiring new research (see Soderberg et al., 2020).¹ Although Soderberg et al.’s review contains scholarly work that falls outside of the I-O literature, we wonder if similar benefits might emerge in I-O psychology and related areas. Calls for openness, transparency, and replicability may go too far, such as in the case of qualitative research where the unique experiences of participants as interpreted by the investigators may not be replicable in principle. This can undermine a core strength of qualitative research (see Pratt et al., 2020).

The proposed topics for each workshop meeting are provided below and indicate the specific focus of discussion for the (a) morning and (b) afternoon sessions:

• Workshop 1 (Wednesday, May 19): (a) What is open science? (b) Accelerating robust research in the organizational sciences
• Workshop 2 (Wednesday, May 26): (a) What is the Open Science Framework? (b) An ounce of prevention is worth more than a pound of cure: The various forms of preregistering research
• Workshop 3 (Wednesday, June 2): (a) An open science workflow template; (b) Reviewing with open science in mind (e.g., are findings reported transparently and in a reproducible manner?; not all results need to be significant for science to move forward; also how to review in a results-blind manner, consulting preregistrations; reviewing registered reports)
• Workshop 4 (Wednesday, June 9): (a) The many ways of ensuring analytic reproducibility: From open code, to open data, to full computational reproducibility; (b) Promoting open science and replication work
• Workshop 5 (Wednesday, June 16): (a) How to have better conversations when making authorship decisions; (b) Transparency and openness guidelines, preprints, and our publishing model

Following each workshop, we’ll hold discussions with invited panelists that will include journal editors, associate editors, and reviewers of major journals in the organizational sciences. These individuals hope to enhance the rigor and credibility of our science, and their perspectives are pivotal to the incremental improvements we hope to spur with our summer series. For these sessions, we’ll discuss questions such as

1. Of the practices highlighted in the workshop, which do you see as valuable to promote and reward (e.g., communicate to your team, associate editors, reviewers)? Notably, if your journal has adopted any of these practices, are there any lessons that you wish to share with this community (e.g., linking funding opportunities to registered reports)?
2. Getting into more concrete specifics, how would you go about encouraging the adoption of open science practices, when applicable, to make research submitted to your journal more robust? Are there any tactics that you would encourage your team/reviewers to employ to help authors open up their work?
3.  What are the most important reasons you can think of for encouraging authors to try an open science practice?
4. What support would you like to see from the community we are building with our summer series? What do you think you’ll do after leaving the summer series?

We certainly see potential for open science to advance science more quickly if appropriately implemented in our field. The SIOP/CARMA Open Science Virtual Summer Series will highlight many practices that might describe this future (e.g., how to appropriately preregister key details of your research, how to leverage the registered-report publishing model at journals adopting this approach, how to make raw data shareable when permissible, how to annotate your analytical work to facilitate transparency and reproducibility, how to leverage preprints to spread credible ideas more widely and rapidly, and how to make research computationally reproducible). Ultimately, what we wish to reinforce is not the practices that describe this future per se but our common identity as scientists who hope to bring robust science for a smarter workplace to as many people as we can bring it.  Our aim chimes with the chorus of other scholars in our field (e.g., Grand et al., 2018) who promote incremental thinking regarding making our science even more robust than it has historically been. Small gains over time will add up, further accelerating the credibility that our field has long enjoyed in both academic and applied settings. In keeping this aim, we want to encourage any small step that each of us can commit to making. Getting started with small steps, a few of which we covered in our previous entry (see Castille et al., 2021), sets the stage for more moves later. In this sense, implementing open science practices is not a decision to be “in or out” with open science; it’s figuring out how to take a step in the right direction so that the next one is even easier to commit. With respect to the summer series, then, any ideas, tools, examples, and resources we can give to people for how to improve their next project in ways that are consistent with open science would be awesome! Our plans are still preliminary, and we are always open to suggestions about these workshops! What ideas do you have to more effectively open up our science or make our science even more robust than it has been historically? Please share your thoughts openly with Chris Castille (christopher.castille@nicholls.edu).

Registration Instructions

• Login as a Website User to your CARMA account. (If you do not have a CARMA account, please click here.)
• Once you login, in the middle section of your User Area, you will see an option to “Register for Live Events”. (If you have not registered your device yet, please register your device first. Then, click “User Area.”)
• Select “SIOP/CARMA Open Science Virtual Summer Series-2021” and checkout.
• You will receive an email to confirm your registration that will provide more information about the events.

Hoping to see you at the SIOP/CARMA Open Science Virtual Summer Series!

Note

¹  A registered report is like a dissertation in that you gain commitment from a committee, which in the case of academic publishing would involve the reviewers and editors of a journal, to have a manuscript published regardless of the findings that do (or do not) materialize so long as the agreed upon methods were utilized. Several journals are adopting this format, including the Leadership Quarterly, Journal of Business and Psychology, Human Resource Management Journal, and the Journal of Personnel Psychology, and adoption has been increasing (Chambers, 2019).

References

Castille, C. M., Oswald, F. L., Banks, G., & Williams, L. (2021). Opening up: Small wins in open science: Things you can do today to improve research in I-O psychology. The Industrial-Organizational Psychologist, 58(3).

Chambers, C. (2019). What’s next for registered reports? Nature, 573(7773), 187–189. https://doi.org/10.1038/d41586-019-02674-6

Grand, J. A., Rogelberg, S. G., Allen, T. D., Landis, R. S., Reynolds, D. H., Scott, J. C., Tonidandel, S., & Truxillo, D. M. (2018). A systems-based approach to fostering robust science in industrial-organizational psychology. Industrial and Organizational Psychology, 11(01), 4–42. https://doi.org/10.1017/iop.2017.55

Pratt, M. G., Kaplan, S., & Whittington, R. (2020). Editorial essay: The tumult over transparency: Decoupling transparency from replication in establishing trustworthy qualitative research. Administrative Science Quarterly, 65(1), 1–19. https://doi.org/10.1177/0001839219887663

Soderberg, C. K., Errington, T. M., Schiavone, S. R., Bottesini, J. G., Singleton Thorn, F., Vazire, S., Esterling, K., & Nosek, B. A. (2020). Initial evidence of research quality of registered reports compared to the traditional publishing model [Preprint]. MetaArXiv. https://doi.org/10.31222/osf.io/7x9vy