Closing the Scientist-Practitioner Gap: Studies From 2016 With Significant Practical Utility

Alyssa Perez, Corey Grantham, Izabela Widlak, and Shreya Sarkar-Barney

Introduction

Industrial-organizational psychologists have long touted the scientist–practitioner model.  Descriptions of the field (including the description on SIOP's own website) emphasize the application of research to improve performance across individual employees, teams, and organizations. Therefore, academic research should be of exceptional value to organizations to guide the design of effective workplace interventions. There is a disconnect, however, between scientific knowledge and workplace application. Researchers often conduct studies without considering their utility in the workplace, and organizational leaders and practitioners frequently base decisions on intuition or guesses without first consulting science. When applied effectively in organizational settings, scientifically rigorous I-O research enhances employees’ lives and boosts organizational performance (Huselid, 1995; Sverke, Hellgren, & Naswall, 2002). Top quality studies fill academic journals every year but often go unnoticed by practitioners (Rynes, Colbert, & Brown, 2002; Sanders, van Riemsdijk, & Groen, 2008).  This may be due to the mismatch between what researcher’s study and the topics that are of importance to practitioners. In fact, a recent study by SIOP's SCi taskforce found that of the 687 accepted submissions for the 2016 annual SIOP conference, only 35% were related to the top workplace trends (Thornton, Poeppelman, Sinar, Armstrong, & Blacksmith, 2017).  Our study was partly motivated to understand to what extent published studies match trending issues in the workplace. This article seeks to highlight a shortlist of I-O psychology research articles from 2016 that demonstrate tremendous potential for application in organizations. Additionally, we examine if these articles further understanding of the topics presented by the 2016 workplace trends. By showcasing these important studies, we hope to contribute to closing the scientist–practitioner gap and guide future research endeavors to advance the field while improving employees' lives and organizational performance.               

Method

Our review of the I-O psychology literature began with a focus on academic journal articles published in 2016 in the top 15 most prestigious I-O journals as ranked by SIOP (Zickar & Highhouse, 2001; see Appendix). This focus yielded 955 articles to review across the 15 journals. Articles were reviewed for topics related to practices in the four key areas: hiring, managing, developing, and supporting employees. The team reviewing and ranking the literature consisted of four scientist–practitioners: two graduate students in I-O psychology, a scientist–practitioner with a PhD in I-O psychology, and an HR practitioner with significant field and scientific research experience. The literature review process consisted of two main phases. First, the 955 articles were reduced to a shortlisted set of 47 articles following the creation of article selection criteria. To be selected for the shortlist, an article needed to meet all the following four criteria:

1. Tackles a pressing problem in organizations, including but not limited to SIOP’s list of Top 10 Workplace Trends 2016 (SIOP Administrative Office, 2015)
2. Describes a meta-analysis or an empirical study comprising multiple methods or multiple studies
3. Results of the study can be applied across multiple types of workplace settings (as opposed to only advancing theoretical understanding)
4. Effect sizes of the study/studies are significant and practically meaningful             

Next, a Research Applicability and Practical Utility scale was created by the team to further narrow the 47 shortlisted articles to the final 10. The scale consisted of four dimensions that rated each study’s importance, potential for action, generalizability, and statistical significance.  Each of the dimensions was rated using a four-point scale.

Importance.  Measured the extent to which the empirical study examines current workplace trends (1 = study will lead to incremental improvement in the field; 2 = the study could further understanding on existing practices; 3 = study brings clarity to debated or less understood topics; 4 = the study addressed trending/emerging problems [as described by SIOP's 2016 Workplace Trends Survey]).

Potential for action. Measures the extent that the results of a study can be used in applied settings to design workplace interventions, rather than simply advancing theoretical understanding of a topic (1 = study has future potential but is not applicable; 2 = study only explains a phenomenon; 3 = results of the study can be used primarily for making workplace predictions including employee selection; 4 = study is predictive and can be used for interventions that lead to improvements at the individual, team, and organizational levels).

Applicability. Measures whether the study’s findings have utility across all work settings (1 = study is useful but not generalizable; 2 = study’s results are generalizable to organizations in single industries [i.e., nursing, technology, etc.]; 3 = results are generalizable across multiple industries; 4 = studies are highly generalizable across all work settings).

Statistical significance. Measures the magnitude of the study’s effect size. Cohen (1992) guided the classification of effect sizes into four categories. (1 = extremely small effect size [R² < .02]; 2 = small effect size [.02 < R² < .05]; 3 = moderate effect size [.05 < R² < .25]; 4 = significant effects [R² > .25]). 

Once ratings were provided by each rater, interrater reliability was calculated using the intraclass correlation (ICC) statistic to determine consistency among the four raters on each rating dimension (Shrout & Fleiss, 1979). The interrater reliability is considered poor for ICC values less than .40, fair for values between .40 and .59, good for values between .60 and .74, and excellent for values between .75 and 1.0 (Cicchetti, 1994). All ICC (two-way random) values were in the acceptable range. For the dimension of importance, the ICC = 0.65 (p < .0.001), 95% CI (0.440, 0.790). On the potential for action, ICC = 0.81 (p < 0.001), 95% CI (0.697, 0.886); for applicability, ICC = 0.81 (p < 0.001), 95% CI (0.705, 0.888) and ratings of statistical significance, ICC = 0.90 (p < 0.001), 95% CI (0.832, 0.939).

Results

While there were many significant and impactful studies published this past year, the shortlist below shows the 10 most applicable articles of 2016 based on the Research Applicability and Practical Utility scale described above. Each of the articles is summarized below along with a brief note on the practical implications. We also note how the study relates to a 2016 SIOP trend.

The Struggle With Employee Engagement: Measures and Construct Clarification Using Five Samples (Byrne, Peters, & Weston, 2016)

SIOP trend(s). Employee engagement

Summary/key findings. The authors examined the ability of the Utrecht Work Engagement Scale (UWES) and the Job Engagement Scale to assess the measurement and conceptualization of employee engagement as an independent construct. Results show that the two scales are correlated but not interchangeable and that the UWES assesses engagement with overlap from job attitudes. This study also found that the engagement construct is not the opposite of burnout as previously thought, and engagement is often confounded with the construct of employee commitment.

Practical implications. Promoting employee engagement is an important priority for most organizations. However, current understanding is limited due to lack of construct clarity and measurement challenges. This study offers three guidance points to practitioners: (a) burnout, which is an indicator of disengagement, is not the opposite of engagement, and this must be measured using a separate scale; (b) although the constructs of engagement and job attitudes (i.e., satisfaction, commitment, involvement) overlap, they are not the same; and (c) engagement and job performance share a small correlation. Engagement may be more important than performance in promoting well-being.

Social Media for Selection? Validity and Adverse Impact Potential of a Facebook-Based Assessment (Van Iddekinge, Lanivich, Roth, & Junco, 2016)

SIOP trend(s). Using social media to make employment-related decisions

Summary/key findings. This study explored organizations' use of Facebook to screen job applicants. The authors examined recruiter ratings of Facebook profiles with respect to criterion-related validity and subgroup differences.  Recruiter ratings of applicants' Facebook profile information were unrelated to supervisor ratings of job performance, turnover intentions, and actual turnover. Finally, there was evidence of subgroup differences in Facebook ratings such that White female applicants were preferred over other subgroups.

Practical implications. Social media pages of job candidates are readily available and inexpensive, but their value in informing hiring has been largely unknown. This study demonstrates that social media information offers no value beyond the tools typically used to predict job performance. Although recruiters may be good in sourcing candidates, their judgment about candidate quality based on Facebook profiles tends to be inaccurate. Recruiter ratings may instead do harm and result in subgroup differences that result in adverse impact.

Tethered to Work: A Family Systems Approach Linking Mobile Device Use to Turnover Intentions
(Ferguson et al., 2016)

SIOP trend(s). Trends in technology are changing the way work is done; increasing focus on health and wellness in the workplace

Summary/key findings. This study explored the effects of mobile work (mWork) during family time on both employees and their spouses. The results indicate that mWork is associated with greater work– family conflict, burnout, and declines in employee commitment. The employee’s spouse was also impacted by mWork, and the spouse showed less organizational commitment to the employee’s organization when the employee worked during family time. Employee turnover intentions increase in response to the diminished organizational commitment of employees and their spouses.

Practical implications. In some countries (e.g., France), labor agreements mandate that employees disconnect from mWork after certain hours. This study demonstrates that asking employees to refrain from doing work on their mobile devices during nonwork time may have positive effects on well-being and commitment. Although this study focused on mobile work, it lends support to the broader idea that supportive organizational practices lowers intentions to leave and increases organizational commitment.

The Influence of Family-Supportive Supervisor Training on Employee Job Performance and Attitudes: an Organizational Work–Family Intervention
(Odle-Dusseau, Hammer, Crain, & Bodner, 2016)

SIOP trend(s). Increasing focus on health and wellness in the workplace; employee engagement

Summary/key findings. This study examined the benefits of family-supportive supervisor behaviors (FSSB) on employee physical health, job satisfaction, and turnover intentions in a healthcare setting.  Results demonstrate significant beneficial effects of FSSB training. However, the relationship between training and the outcomes were observed only when employees perceived their managers engaging in family-supportive behaviors. The use of creative work–family management was shown to be the most beneficial among the FSSB strategies that were taught.

Practical implications. The postrecession economy has required doing more with less, resulting in an overwhelmed workforce. There is growing awareness of the need to promote employee well-being, particularly through work and family integration programs. Besides channeling such programs through HR, training managers on the mechanisms of family-supportive behaviors can have significant benefits. Such an intervention requires establishing a clear case of family supportive programs, developing a menu of options, and educating managers on their delivery.

Help or Hindrance? Work–Life Practices and Women in Management
(Kalysh, Kulik, & Perera, 2016)

SIOP trend(s). Work–life balance across generations; building healthy, diverse workforces

Summary/key findings. The study examined the relationship between work-life practices in 2002–2006 and the proportion of women in management in 2010, 2012, and 2014.

Work-life practices had a positive effect on women in management after an 8-year time lag, though the effect was not found in organizations that were highly male dominated. Leave arrangements and direct provision of services such as childcare or elder care had the strongest association with women in management.

Practical implications. There is widespread recognition that increasing the representation of women in the workforce is good for women, society, and the economy. Work–life balance programs are proposed as one solution to achieving this goal. Many organizations invest in and then cut such programs before seeing results. This study provides guidance on the investment period before which organizations can expect to see results. In the case of work–life programs improving gender diversity, the investment period is approximately 8 years.  Access to services such as childcare can be particularly influential in attracting and retaining female employees.

Overconfidence in Personnel Selection: When and Why Unstructured Interview Information Can Hurt Hiring Decisions
(Kuausel, Culbertson, & Madrid, 2016)

SIOP trend(s). Although missing from the list of 2016 trends, considerable research and organizational resources are spent on selection processes. Knowing whether such expenditures have significant ROI in attracting top talent remains a perennial problem despite this topic’s absence from the list of 2016 trends.

Summary/key findings. This study examined overconfidence in predictions of job performance for participants who were responsible for hiring decisions. Participants who received results from an unstructured interview showed more overconfidence in their selection than those who were only presented with applicant test scores. Counter to current organizational practices, managers who used objective test scores of intelligence and conscientiousness were more accurate in their prediction of job performance than those who used test scores and applicant performance in unstructured interviews. 

Practical implications. Unstructured interviews are the most common selection tool used in organizations. Even though managers have access to more valid tools, hiring candidates without an interview is deemed sacrilegious. This study has important implications for selection practices, and organizations can save on managers’ time and resources by utilizing scores from validated predictive tests.

Multisource Feedback, Human Capital, and the Financial Performance of Organizations
(Kim, Atwater, Patel, & Smither, 2016)

SIOP trend(s). Changing nature of performance management and development

Summary/key findings.  This study investigated the relationship between multisource feedback (MSF) programs and the organization's financial performance. This study found that multisource feedback used for administrative and developmental purposes has a moderate effect on a firm’s financial performance by elevating employee ability and promoting knowledge sharing. The implementation of MSF programs was also associated with increased workforce productivity four years later. 

Practical implications. Although there is increasing adoption of MSF programs in organizations, the debate continues whether it should be used for development only or administration only, or whether it can serve both purposes. Many would agree on the developmental benefit of such tools on individual employees; however, the organizational benefits are unknown. This study warrants the use of MSF for dual purposes. The real value of MSF is unleashed when it generates exploration of the ratings by the feedback recipients to reduce discrepancy between current and ideal behavior. The collective outcome is an improvement in overall firm level performance. Organizations should ensure employees are actively seeking clarification, coaching, and mentoring utilizing the information gathered through the MSF process.

How Does Leader Humility Influence Team Performance? Exploring the Mechanisms of Contagion and Collective Promotion Focus
(Owens & Hekman, 2016)

SIOP Trend(s). Changing nature of performance management and development

Summary/Key findings. This study examined the effects of leader humility on team interaction patterns, emergent states, and team performance. The study found that leader humility induced followers’ humility and produced an emergent state that enhanced team performance.

Practical implications. Many believe leaders should be confident and self-righteous to avoid the appearance of weakness or incompetence. The findings of this study suggest the opposite and find that humble leaders help their team members overcome their competitiveness and shift their attention to achieving the highest potential for the team. The ability to create such a selective focus on the team’s potential enables goal achievement. The study has several implications for leader selection and training. Organizations can benefit from using humility as a selection criterion and train leaders on transformational leadership.

A Meta-Analytical Integration of Over 40 Years of Research on Diversity Training Evaluation

(Bezrukova, Spell, Perry, Jehn, & Melbourne, 2016)

SIOP trend(s). Building healthy, diverse workforces

Summary/key findings. This meta-analysis investigated the effects of diversity training on four training outcomes: reactions, cognitive learning, behavioral, and attitudinal/affective learning. Reactions to training and cognitive learning had the largest effects, whereas behavioral and attitudinal learning have smaller effects. Effects on reactions and attitudinal learning decayed over time, but cognitive learning remained stable or increased.

Practical implications. There have been conflicting findings on the role of diversity training in reducing prejudice and stereotypes. The results of this study indicate that diversity training is more impactful when embedded as part of a larger diversity initiative rather than as a stand-alone activity. While training may successfully lead to knowledge gain about diversity, we still do not have effective ways to change attitudes and behavior. Furthermore, it is still unclear whether diversity training should be mandated or if participation should be voluntary. According to this meta-analysis, behavior change was greatest for those for whom diversity training was mandatory. Unfortunately, the meta-analysis did not investigate the effects of unconscious and implicit bias, or the efficacy of practices that are currently popular in addressing bias.

Building Work Engagement: A Systematic Review and Meta-Analysis Investigating the Effectiveness of Work Engagement Interventions (Knight, Patterson, & Dawson, 2016)

SIOP trend(s).  Employee engagement

Summary/key findings. This study aimed to determine which interventions most effectively promote employee engagement. Engagement interventions conducted in groups were most efficacious in improving engagement levels.  However, it should be noted that there were fewer studies investigating individual-level interventions.  This may have limited the ability to fully test the efficacy of such interventions. This meta-analysis assessed the effectiveness of four types of work-engagement interventions: (a) personal resource building; (b) job resource building; (c) leadership training; and (d) health promotion. Each of the interventions had a small but significant effect on work engagement. 

Practical implications. Organizations can achieve measurable improvements in engagement levels by utilizing an intervention strategy that is segmented and matched by subgroups. Such improvement efforts are likely to be more sustained when they are geared toward improving facets of work engagement (i.e., absorption and dedication vs. vigor) rather than on improving overall engagement.

Conclusion and Recommendations

The studies we have highlighted provide rigorous evidence in several areas that are important to practitioners. Here are three key implications for organizations: (a) To see improvements in gender diversity, organizations must be willing to invest in work–life programs for up to 8 years before seeing results; (b) organizations can more accurately select high performing job candidates, but they must be willing to rely on objective tests measures more than unstructured interviews and social media (Facebook) profiles; and (c) to measure disengagement accurately, organization are better off using separate scales designed to measure engagement and disengagement. Utilizing these results, practitioners can provide evidence-based guidance on what works and avoid investments in low-return practices. The most encouraging part of our study was the overlap between the 2016 SIOP trends and the published studies. Seven of the 10 trends were represented in the examples we have highlighted. Areas markedly missing were studies utilizing big data, as well as an exploration of behaviors and mechanisms to promote business agility. We see opportunities for researchers and practitioners to collaborate in studying workplace issues utilizing big data. With the rapid adoption of digital technologies, data collection is no longer the challenge. The success of such efforts is likely to rest on effective utilization of big data to ask meaningful questions and in ensuring that studies are replicable and applicable.

References

Bezrukova, K., Spell, C. S., Perry, J. L., & Jehn, K. A. (2016). A meta-analytical integration of over 40 years of research on diversity training evaluation. Psychological Bulletin, 142(11), 1227-1274. doi:10.1037/bul0000067

Byrne, Z. S., Peters, J. M., & Weston, J. W. (2016). The struggle with employee engagement: Measures and construct clarification using five samples. Journal of Applied Psychology, 101(9), 1201-1227. doi:10.1037/apl0000124

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155.

Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284.

Ferguson, M., Carlson, D., Boswell, W., Whitten, D., Butts, M. M., & Kacmar, K. (2016). Tethered to work: A family systems approach linking mobile device use to turnover intentions. Journal of Applied Psychology, 101(4), 520-534. doi:10.1037/apl0000075

Huselid, M. A. (1995). The impact of human resource management practices on turnover, productivity, and corporate financial performance. Academy of Management Journal, 38, 635–672.

Kalysh, K., Kulik, C. T., & Perera, S. (2016). Help or hindrance? Work–life practices and women in management. The Leadership Quarterly, 27(3), 504-518. http://0-dx-doi-org.library.alliant.edu/10.1016/j.leaqua.2015.12.009

Kausel, E. E., Culbertson, S. S., & Madrid, H. P. (2016). Overconfidence in personnel selection: When and why unstructured interview information can hurt hiring decisions. Organizational Behavior and Human Decision Processes, 137, 27-44. http://0-dx-doi-org.library.alliant.edu/10.1016/j.obhdp.2016.07.005

Kim, K. Y., Atwater, L., Patel, P. C., & Smither, J. W. (2016). Multisource feedback, human capital, and the financial performance of organizations. Journal of Applied Psychology, 101(11), 1569-1584. doi:10.1037/apl0000125

Knight, C., Patterson, M., & Dawson, J. (2016). Building work engagement: A systematic review and meta‐analysis investigating the effectiveness of work engagement interventions. Journal of Organizational Behavior. doi:10.1002/job.2167

Odle-Dusseau, H. N., Hammer, L. B., Crain, T. L., & Bodner, T. E. (2016). The influence of family-supportive supervisor training on employee job performance and attitudes: An organizational work–family intervention. Journal of Occupational Health Psychology, 21(3), 296-308. doi:10.1037/a0039961

Owens, B.P., & Hekman, D.R. (2016). How does leader humility influence team performance? Exploring the mechanisms of contagion and collective promotion focus. Academy of Management Journal, 59(3), 1088-1111. doi:10.5465/amj.2013.0660

Porr, B., Axton, T., Ferro, M., & Dumani, S. (2016). Areas in need of more science/research: Results from the 2015 practitioner needs survey. The Industrial-Organizational Psychologist, 53(4), 113-120.

Rynes, S. L., Colbert, A. E., & Brown, K. G. (2002). HR professionals' beliefs about effective human resource practices: Correspondence between research and practice. Human Resource Management, 41(2), 149-174. 

Sanders, K., van Riemsdijk, M., & Groen, B. (2008). The gap between research and practice: A replication study on the HR professionals’ beliefs about effective human resource practices. The International Journal of Human Resource Management, 19(10), 1976-1988.

SIOP Administrative Office. (2015, December 16). Top 10 Workplace Trends 2016. Retrieved from http://0-www-siop-org.library.alliant.edu/article_view.aspx?article=1467

SIOP Administrative Office. (2016, December 20). Top 10 Workplace Trends 2017. Retrieved from http://0-www-siop-org.library.alliant.edu/article_view.aspx?article=1610

SIOP Administrative Office. (2015, December 16). Top 10 Workplace Trends 2016. Retrieved from http://0-www-siop-org.library.alliant.edu/article_view.aspx?article=1467

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420.

Sverke, M., Hellgren, J., & Näswall, K. (2002). No security: a meta-analysis and review of job insecurity and its consequences. Journal of Occupational Health Psychology, 7(3), 242.

Thornton, J, Poeppelman, T, Sinar, E, Armstrong, M, & Blacksmith, N. (2017).  How Do Our Conference Sessions Track With SIOP's Top Workplace Trends?  Retrieved from http://0-www-siop-org.library.alliant.edu/tip/jan17/sci.aspx

Van Iddekinge, C. H., Lanivich, S. E., Roth, P. L., & Junco, E. (2016). Social media for selection? Validity and adverse impact potential of a Facebook-based assessment. Journal of Management, 42(7), 1811-1835. doi:10.1177/0149206313515524

Zickar, M. J., & Highhouse, S. (2001). Measuring prestige of journals in industrial-organizational psychology. The Industrial-Organizational Psychologist, 38(4), 29-36. Retrieved from http://0-www-siop-org.library.alliant.edu/tip/backissues/tipapr01/03Zicker.aspx

Appendix

List of journals reviewed for the study

1. Journal of Applied Psychology
2. Personnel Psychology
3. Academy of Management Journal
4. Organizational Behavior and Human Decision Processes
5. Journal of Vocational Behavior
6. Administrative Science Quarterly
7. Journal of Management
8. Journal of Organizational Behavior
9. Organizational Research Methods
10. Academy of Management Review
11. Leadership Quarterly
12. Human Resource Management Journal
13. Journal of Personality and Social Psychology
14. Psychological Bulletin
15. Journal of Occupational Health Psychology