Talking About Reproducibility in ReproducibiliTea

Haley R. Cobb & Candice L. Thomas, Saint Louis University

As members of the I-O psychology community, we are constantly learning, whether in class, in the lab, at our jobs, with podcasts and Twitter pages, or through conversations with our colleagues and peers. One important topic we are learning about is open science. Open science has been called “disruptive” (Vicente-Saez & Martinez-Fuentes, 2018, p. 428), an “irreversible paradigm shift” (Friesike et al., 2015, p. 598), and “a turbulent yet exciting” change (Arabito & Pitrelli, 2015, p. 1). Practicing open science oneself involves learning and adopting new skills (Arabito & Pitrelli, 2015), and the technologies that make open science possible are ever evolving (Scheliga & Friesike, 2014). Thus, it is important that graduate students and early career researchers in particular are well versed in open science principles and practices.

Open science is “shared and developed through collaborative networks” (Vicente-Saez & Martinez-Fuentes, 2018, p. 434) and is characterized by open access to and sharing of data, preregistration of hypotheses and study methodology, replication, reproducibility, and removing paywalls and increasing free access to information (including published, peer-reviewed articles; Banks et al., 2019). It is a direct challenge to the reproducibility crisis (e.g., Maxwell et al., 2015), which has resulted in part from incentives that favored new and exciting (read: statistically significant) findings (e.g., Nosek et al., 2012). By making study materials (e.g., data, methodology, hypotheses) more openly available, transparency, replicability, and self-correction in the psychological sciences, including I-O psychology, should be strengthened (e.g., Resnik & Shamoo, 2016; Vazire & Holcombe, 2020).

As the I-O community grows to value the benefits of open science, it is crucial that I-O psychologists and students of I-O psychology are equipped to adequately engage in open science practices. For students and early career researchers who are interested in being part of a community and engaging in discourse on open science, ReproducibiliTea might be an appropriate group to join.

I Know About Reproducibility, But What Is ReproducibiliTea?

ReproducibiliTea (https://reproducibilitea.org/) is a grassroots, global community of students and early career researchers that aims to discuss and learn about open science. As a formal member of the ReproducibiliTea community, individuals can create their own journal clubs, which are local groups, usually within one’s university, that meet regularly to read articles, share materials, and ideate on open science topics. These topics can be philosophical (e.g., What does publication bias mean for the psychological sciences? Ferguson & Heene, 2012), cover the reasons why open science is needed (e.g., Why does replication even matter? Maxwell et al., 2015), or skill building (e.g., What are best practices for data sharing? Meyer, 2018).

Although researchers can independently read these articles without a journal club, the conversations, resource sharing, and community building of an open-science-focused journal club provides an easy springboard for enacting open science in your own work and translating general open science principles into I-O psychology research. It can be especially helpful for students and early career researchers who value the structure and community afforded to them. Through our local chapter, we have both access to a global community (whose aim is to improve the psychological sciences) and a personalized structure that allows us to learn within our own community about open science as it relates to our specific research and education. Here, we will share our experiences in creating and participating in a local chapter of ReproducibiliTea and provide some tips on how you can create or get involved in your own local open science community.

The Saint Louis Area Journal Club

The first author organized the development of a ReproducibiliTea journal club in St. Louis, MO, in January 2021, and the group’s first meeting occurred in February 2021. We reflect on our experiences during the first year of this new journal club, and, stemming from our personal experiences, we offer some advice for starting your own journal club or getting involved with the ReproducibiliTea community.

The First Year

Our local chapter of ReproducibiliTea (the St. Louis Area Journal Club) was founded by the first author in the spring of 2021. In addition to membership with this global community, we have helped foster our own internal community with our journal club. The first author created a reading list for the first year’s gatherings after reviewing reading lists from other journal clubs and messaging with ReproducibiliTea members. Because most of the individuals in our journal club would likely be within I-O psychology, the reading list was geared toward this field of study. However, the journal club was accessible to anyone at Saint Louis University who wanted to join. The reading list covered myriad topics, including an introduction to open science (e.g., Ferguson & Heene, 2012), researcher degrees of freedom (e.g., Wicherts et al., 2016), and questionable research practices (e.g., Flake & Fried, 2020). We also invited a guest speaker, Dr. Julia Strand, to discuss her experience with transparency, the Society for the Improvement of Psychological Science (SIPS), and open science generally. (She has a great popular press article on handling an error in her own published work; Strand, 2020).

Creating an Inclusive Community

As we mentioned, the journal club we are part of is composed mainly of people within our I-O program (e.g., students, faculty), but we are not exclusive. We believe that we have much to learn from other students and early career researchers, and we have active participants from other programs (e.g., experimental psychology). Thus, we aimed to capture open science broadly to be inclusive of members from other disciplines, and the readings and discussions typically focused on open science in general.

However, our discussions do capture experiences pertinent to I-O psychology. Many of our discussions focus on conducting research that aligns with open science practices, and we read articles published in I-O-focused journals (e.g., Banks et al., 2019). Thus, each of us is well versed in open science practices and how they pertain to research in I-O. Furthermore, many of our journal club members who are in I-O psychology are not interested in academic careers and plan to pursue applied careers. These journal club members force us to think about and find resources on how open science can benefit practitioners, such as finding articles that are open access or preprints (e.g., Vazire & Holcombe, 2020) and understanding validity implications that arise from open science (e.g., Vazire et al., 2020). Overall, regardless of a journal club member’s path in academia or the applied world, we have a safe space to be curious and ask questions about open science, thus facilitating our learning and education. 

Next Steps

For the authors of this article and members of the St. Louis Area Journal Club, we are starting our second year of our journal club meetings. (Readers can access last year’s materials here: https://osf.io/n5438/.) Many of the readings from 2021 were geared towards getting us to think about open science, learning the fundamentals, and considering how open science impacts all parts of the research process. Our aim for 2022 is to help journal club members learn more actionable steps for open science, such as learning new tools and technologies, creating mock registered reports, and sharing resources in an online repository. With last year’s journal club as a foundation to our knowledge of open science, we hope to gain more tangible, practical skills to facilitate our involvement in open science. We also hope that our group will continue to grow, perhaps to include faculty members from other programs or undergraduate students.

Tips for Growing Your Own Local Open Science Community

ReproducibiliTea is accessible to anyone interested in building or being part of the open science community. We have found that having the local chapter helps us to tailor our journal club to our needs, which includes content, scheduling, and safety precautions during the ongoing pandemic. Perhaps most importantly, having the local journal club facilitates a sense of camaraderie and learning about other members’ interests outside of formal class time.

Start or Join a Local Chapter of ReproducibiliTea

To start the local journal club, the organizer of the group coordinates with ReproducibiliTea to create an OSF page specific to their journal club. (To see the St. Louis Area Journal Club’s OSF page: https://osf.io/n5438/). As there are journal clubs all over the world, started by graduate students and early career researchers across disciplines, OSF pages for many journal clubs are available (https://osf.io/cfby7/). Members of this community, therefore, have access to resources, advice on how to create a structure for journal clubs, and opportunities to interact with people interested in open science all over the world. (We even have a Slack channel!) ReproducibiliTea offers other ways to learn about open science, including joining an existing journal club or independently reading from a selected list of sources.

Know Your Audience

As I-O psychology students, researchers, and practitioners, it can be helpful to tailor the readings and conversations specifically to I-O. For example, we found it helpful to read Banks et al. (2019) and Bergh and Oswald (2020), as this helped us keep up with trends within our field. These readings are also encompassing enough to be relevant to our non-I-O journal club members. Our group also consisted of students who were interested in applied jobs and not just academic research. Although the authors of this paper focus on research, several students in the group expressed that they would find more benefit if we could also discuss applied I-O and open science. This point also helped us to realize that conversations on open science and I-O might benefit other members of SIOP, and we were galvanized to write this article, as well as create an alternative session for this year’s SIOP conference covering education and open science. Thus, understanding the needs of our journal club members allowed us to tailor the readings and conversations to I-O while still being inclusive of members from other disciplines and with various career ambitions. We encourage anyone interested in starting their own journal club to consider the interests of all journal club members and to be as inclusive and accessible as possible.

Accessibility and Extracurriculars

Accessibility is important to us. We know time is a valued resource, and not everyone was able to attend sessions live via Zoom. To make the group more accessible, we started a Slack channel for our own local journal club, which anyone who is interested in can join. We share various resources, articles, questions, and announcements with this Slack channel, and this helps us keep in touch in-between meetings. Another extracurricular activity we provided was the meeting with our guest speaker, Dr. Julia Strand. Journal club members were encouraged to watch a recorded lecture where Dr. Strand discusses transparency in detail (Strand, 2021); this allowed our members to learn about Dr. Strand’s experience before attending the meeting (which was formatted as a Q&A) or as a replacement if they could not attend the meeting. Thus, we suggest supplementing the journal club meetings with activities beyond the readings to help further create a sense of community, including for members who are unable to attend meetings.

Conclusion

Open science isn’t going anywhere. Learning about open science, including how to do open science, is important for I-O psychology students and early career researchers in particular, and we have found that being part of the ReproducibiliTea community has benefited us in numerous ways. We have gained two communities: a global community and a local community, both of which are invested in open science and making the psychological science, including I-O psychology, more robust, transparent, and accessible. We encourage the readers of this article to consider creating informal practices to learn about open science, including joining the ReproducibiliTea community and starting your own local journal club.

References

Arabito, S., & Pitrelli, N. (2015). Open science training and education: Challenges and difficulties on the researchers’ side and in public engagement. Journal of Science Communication, 14(4), 1–4. https://doi.org/10.22323/2.14040301

Banks, G. C., Field, J. G., Oswald, F. L., O’Boyle, E. H., Landis, R. S., Rupp, D. E., & Rogelberg, S. G. (2019). Answers to 18 questions about open science practices. Journal of Business and Psychology, 34(3), 257–270. https://doi.org/10.1007/s10869-018-9547-8 

Bergh, D. D., & Oswald, F. L. (2020). Fostering robust, reliable, and replicable research at the Journal of Management. Journal of Management, 46(7), 1302–1306. https://doi.org/10.1177%2F0149206320917729

Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: Publication bias and psychological science’s aversion to the null. Perspectives on Psychological Science, 7(6), 555–561. https://doi.org/10.1177%2F1745691612459059

Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465. https://doi.org/10.1177%2F2515245920952393

Friesike, S., Widenmayer, B., Gassmann, O., & Schildhauer, T. (2015). Opening science: Towards an agenda of open science in academia and industry.  Journal of Technology Transfer, 40(4), 581–601. https://doi.org/10.1007/s10961-014-9375-6

Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70, 487–498. https://doi.org/10.1037/a0039400

Meyer, M. N. (2018). Practical tips for ethical data sharing. Advances in Methods and Practices in Psychological Science, 1(1), 131–144. https://doi.org/10.1177/2515245917747656

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–631. https://doi.org/10.1177%2F1745691612459058  

Reproducibilitea. (2021). ReproducibiliTea. Retrieved January 24, 2022, from https://reproducibilitea.org/ 

Resnik, D. B., & Shamoo, A. E. (2016). Reproducibility and research integrity. Accountability in Research, 24(2), 116–123. https://doi.org/10.1080/08989621.2016.1257387  

Scheliga, K., & Friesike, S. (2014). Putting open science into practice: A social dilemma? First Monday. https://doi.org/10.5210/fm.v19i9.5381

Strand, J. (2020, April 15). Scientists make mistakes. I made a big one. Medium. https://elemental.medium.com/when-science-needs-self-correcting-a130eacb4235

Strand, J. (2021, March 25). “Toward a more robust science”—Julia Strand's lecture at the CCSN Methodology Workshop, MAR2021 [Video]. YouTube. Retrieved January 24, 2022, from https://www.youtube.com/watch?v=BMlZYWB4crg

Vazire, S., & Holcombe, A. O. (2020). Where are the self-correcting mechanisms in science? PsyArXiv Preprints. https://doi.org/10.31234/osf.io/kgqzt

Vazire, S., Schiavone, S. R., & Bottesini, J. G. (2020). Credibility beyond replicability: Improving the four validities in psychological science. PsyArXiv Preprints. https://doi.org/10.31234/osf.io/bu4d3

Vicente-Saez, R., & Martinez-Fuentes, C. (2018). Open science now: A systematic literature review for an integrated definition. Journal of Business Research, 88, 428–436. https://doi.org/10.1016/j.jbusres.2017.12.043

Wicherts, J. M., Veldkamp, C. L., Augusteijn, H. E., Bakker, M., Van Aert, R., & Van Assen, M. A. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. https://doi.org/10.3389/fpsyg.2016.01832