SIOP Award Winners: Meet Louis Hickman, Douglas W. Bray and Ann Howard Research Grant Winner

Liberty Munson, Microsoft

As part of our ongoing series to provide visibility into what it takes to earn a SIOP award or grant, we highlight a diverse class of award winners in each edition of TIP. We hope that this insight encourages you to consider applying for a SIOP award or grant because you are probably doing something amazing that can and should be recognized by your peers in I-O psychology!

This quarter, we are highlighting SIOP’s 2022 Douglas W. Bray and Ann Howard Research Grant for research into assessment center methods: Louis Hickman.

Louis and his daughter Harper

Share a little a bit about who you are and what you do.

I am currently a postdoc at The Wharton School of the University of Pennsylvania, and in the fall, I will be an assistant professor at Virginia Tech. I research the implications of technology, including machine learning, for workplaces. The focal project is a collaboration with Christoph Herde, Filip Lievens, and Louis Tay.

Describe the research/work that you did that resulted in this award. What led to your idea?

In this work, we investigate the feasibility of applying machine learning to automatically score assessment center simulations. I learned of some archival data held by our collaborators (Filip Lievens and Christoph Herde) that could be used to investigate this.

What do you think was key to you winning this award?

There is tons of interest in machine learning applications, yet we know very little about what makes a given assessment or dataset amenable to machine learning scoring. This particular assessment center included 18 “speed” role plays that were 3 minutes each, allowing us to investigate not only the validity of machine learning scores for the role plays but also investigating how various factors influence the validity of the machine learning scores. I think that our focus on factors that affected the validity of machine learning helped us a lot.

What did you learn that surprised you? Did you have an “aha” moment? What was it?

We were surprised to find how well machine learning worked considering the small sample size—this assessment center included only 96 assessees. Yet, we were able to capture the majority of variance in the assessment center scores by applying machine learning and natural language processing to all of the 18 role plays. I knew that this process *could* work but did not expect it to work this well.

What do you see as the lasting/unique contribution of this work to our discipline? How can it be used to drive changes in organizations, the employee experience, and so on?
Well, first, it does seem that assessment center simulations can be automatically scored. This is exciting because it could give many more people access to the formative and developmental feedback provided by assessment centers. Second, even though other factors had minor effects, interrater reliability was by far the largest effect on machine learning score validity. As the machine learning assessment area matures, this can hopefully be a foundational piece of evidence that researchers build upon to understand the factors that influence machine learning model validity.

To what extent would you say this work/research was interdisciplinary? 

I believe that all machine learning research is interdisciplinary. What’s challenging is that there are no standards yet for appropriate machine learning methods, so you may get opposite perspectives from two reviewers on the same paper (e.g., I love that you used this method; why didn’t you try method X). I encourage everyone to engage in interdisciplinary research because many other fields are researching the same topics as us, and we ignore their findings at our own peril.

Are you still doing work/research in the same area where you won the award? If so, what are you currently working on in this space? If not, what are you working on now, and how did you move into this different work/research area? 

Many more projects to come on machine learning… Additionally, I am beginning to look at new areas where technology has a major influence on workplace experiences, such as electronic performance monitoring of remote workers.

What’s a fun fact about yourself (something that people may not know)?

I love to see/hear live music and have attended about 30 camping music festivals.

What piece of advice would you give to someone new to I-O psychology? (If you knew then what you know now…)

Be patient with and kind to yourself. “I do enough, I am enough.” - Dopapod (one of those bands I like to go see live).

About the author:

Liberty Munson is currently the director of Psychometrics of the Microsoft Worldwide Learning programs in the Worldwide Learning organization. She is responsible for ensuring the validity and reliability of Microsoft’s certification programs. Her passion is for finding innovative solutions to business challenges that balance the science of assessment design and development with the realities of budget, time, and schedule constraints. Most recently, she has been presenting on the future of testing and how technology can change the way we assess skills.

Liberty loves to bake, hike, backpack, and camp with her husband, Scott, and miniature schnauzer, Apex. If she’s not at work, you’ll find her enjoying the great outdoors, or she’s in her kitchen tweaking some recipe just to see what happens.

Her advice to someone new to I-O psychology? Statistics, statistics, statistics—knowing data analytic techniques will open A LOT of doors in this field and beyond!