Your Guide to Reproducible Research (RR) at SIOP 2017
Zack Horn, Rob Stilson, and Daly Vaughn
The future has arrived for Reproducible Research (RR) at SIOP 2017, with over 60 presentations and a dedicated Saturday RR track (in room Asia 4). This open-source approach to research has caught the tech world and scientific communities by storm, and I-O psychology is quickly catching up! In poster and non-poster presentations alike, researchers across SIOP 2017 are taking a community-oriented approach to advancing robust and reliable research in I-O. So before you finalize your conference schedule, be sure to scan the program for sessions containing the “RR” logo, as at least one presenter in that session is sharing their data, analysis code, or both—for YOU to use!
Why Is RR at SIOP?
Although our community has shown great excitement about bringing RR to I-O, we know that many questions remain—and we would like to address the most common questions here. In short, the ability to replicate studies, reproduce analyses, and share methodologies as a community will help us accelerate the evolution of our science. With the widespread use of open source coding languages such as R and Python, sharing analysis code has become the norm in many analytical sciences. We also understand that as a newly organized initiative at SIOP, there many questions about our community’s norms and expectations for sharing research. To this end, we have created an RR Hub on SIOP.org (www.siop.org/rr) to begin answering many of these questions and provide resources for best practices.
Quickly, What Is RR?
Reproducible research simply refers to the process of documenting the steps taken to conduct a research study so that others can better understand/reuse selected research methods and replicate findings (if data is made available). Researchers are by NO means asked or encouraged to share anything proprietary. Confidential data or proprietary algorithms, for example, are not expected to be shared with the larger community. Data and analysis code can easily be anonymized, however, so that others can reuse methodologies and practice on “mock” data sets. Whether for replication or reuse, participating in RR is an easy way to foster a culture of robust and reliable research across our science.
How Do I Get Started?
The recipe is easy: Prepare your research documents, host them online, and share the link to those documents for others to use. First, visit SIOP’s RR Hub to learn some best practices for RR in I-O, and find links to materials shared by RR presenters at SIOP 2017. Second, this post on SimplyStatistics.org gives a great example of how researchers in any field can prepare share research with the world, but of course it’s up to you to determine where and how much to share. A few examples of file sharing providers include GitHub (recommended), Google Drive, Dropbox, figshare, dryad, zenodo, and Academic Torrents. For those who want to dig in further, the Center for Open Science (COS) provides the Transparency and Openness Promotion (TOP) Guidelines, which discuss three levels of transparency across eight categories.
Where’s This Headed?
We’re just beginning to solidify some best practices for RR in I-O, but we already have great resources for getting started with RR as well as guidelines for presenting RR at SIOP Conferences. Moving forward, the pursuit of RR in I-O will be part of SIOP’s new Robust and Reliable Research initiative, enabling a more strategic approach to formalizing norms and expectations for RR in I-O psychology. The introduction of RR was a surprise to many SIOP members in this year’s Call for Proposals, but RR is forecasted to become mainstream in I-O over the next couple of years. Until then, enjoy the RR presentations in Orlando and make great use of those shared materials!