SIOP Select Session on Open Science

By Robin Gerrow

How should I-O psychologists engage in the open science movement that is all around us? That’s the subject of an important and informative panel discussion at 35^th Annual SIOP Conference slated for this April 23-25 in Austin, Texas. This session will be Thursday, April 23, 3:30-4:20 in Room 402-403 at the JW Marriot hotel.

Open science is relatively new to the research and practice of I-O psychology and  includes activities such as preregistering research plans, providing access to study materials and data, and making key aspects of one’s workflow, such as procedures and analyses, transparent to third parties.

The first part of the SIOP panel will be a tutorial on what open science is—and is not—and to provide major components of the subject. Attendees will then be able to hear more about what SIOP and the American Psychological Association (APA) have been doing around open science, followed by an in-depth discussion about the relevance of open science to I-O psychology in particular.

 “Open science refers to a set of practices that make our research more transparent, reliable, and reproducible, and therefore more understandable and appreciated by graduate students, our colleagues, and other external parties,” said Dr. Frederick Oswald, professor at Rice University, co-chair of the SIOP Open Science and Practice (OSP) Committee, and member of the SIOP panel discussion. He finds it a priority to educate members about open science, working toward the goal of improving organizational research by cultivating a positive culture and diverse community around open science.

“Open science practices are intended to support the integrity of cumulative science,” said Dr. Steve Kozlowski, professor at Michigan State University, chair of the APA Open Science and Methodology Committee, and a fellow panelist for the session. “Good science builds knowledge accumulation. We need to be confident in the veracity of research findings that underpin evidence-based policy decisions in organizations, government, and society. The key issue for open science is transparency.”

“By applying open science practices, our work becomes more transparent and more replicable,” Oswald said. “It is one thing for me to reproduce somebody’s research findings, but it is another to take those research materials and run a similar study to see if I can replicate it in a new sample, or at a different point of time, or in a new setting to see how my results might be similar or different. On this latter point, if you want to reproduce a set of findings and make them more generalizable to a wider range of circumstances, it helps to have a set of practices for being more transparent, reproducible, and replicable.”

Kozlowski added, “Transparency and integrity are important values for ensuring high quality science. SIOP is working to advance the policy application of I-O science in organizations, government, and society. We need to ensure that I-O science is at the forefront of producing high quality actionable knowledge and open science practices will help accomplish that goal.”

One aspect of open science is the idea of preregistration of research projects. Oswald said it is important to understand the value of preregistration, but there are a number of misperceptions around it.

“Preregistration is a new practice for I-O psychology, and so there are a lot of questions about it. But it is simply about clearly stating what your plans are before going into a research project,” he said. “It hopefully shows that you are planning carefully—getting your ducks in a row. If you deviate from those plans, that’s okay, you just need to be transparent in your research about what was planned versus what’s new and what’s changed. That’s how science works.”

He also hopes the panel will be able to alleviate other concerns about open science and preregistration.

“There can be a negative association with preregistration,” Oswald said. “That if you share your plans with the world, you’re locked into a plan or that you’re going to be criticized for it before you even start. But preregistration shouldn’t inspire a fear culture or a culture of policing researchers. Instead, it’s about helping researchers plan and helping external parties understand our research, sharing information that will illuminate and advance the scientific process.

There are some issues OSP and SIOP members in general will need to address, in particular how to share materials and methodology.

“There is a need for thoughtful deliberation,” Kozlowski said. “There are a lot of suggestions and practices being proposed and promoted, particularly those centered on data sharing. But there are potential pitfalls if issues pertaining to what data are shared, to whom, and how they are shared are not carefully deliberated so that unintended consequences do not undermine open science. For example, there are documented cases where de-identified data that was made publicly available was, in effect, reverse engineered by combining other public data to identify individuals. Confidentiality is the bedrock of consent for human research. If proposed open science practices breach confidentiality or undermine public trust in research, the harm will be substantial. So, we want to facilitate transparent, open science, but we want to do it thoughtfully and with our eyes wide open.” 

When it comes to data sharing another issue that must be considered is the sharing of proprietary data.

“Proprietary data and research within organizations can certainly be at odds with open science principles of giving people access and information to the key aspects of the scientific process, for verification or replication purposes,” Oswald said. “SIOP researchers and practitioners have legitimate legal, ethical, and organizational concerns here, and negotiating this tension is key to how the open science movement plays out in I-O psychology.”  

For more information about open science, conference attendees will find this session April 23, 3:30-4:20 in Room 402-403 at the JW Marriot hotel.