Lost in Translation: Visually Communicating Validity Evidence

In our previous column, we discussed the complexity and nuances of measuring unobservable psychological phenomena and the importance of verbally communicating the value of reliability and validity evidence to non-I-O psychologists. Our interviews with Fred Oswald, Jeff Jolton, and Don Zhang were insightful, impactful, and extremely well-received by the SIOP community. However, we also received some feedback from I-O psychology practitioners that emphasized how much more frequently they communicate in the forms of charts, figures, and PowerPoint decks than simply in conversation, making it difficult to apply the lessons learned from our last column to their current roles. In fact, it is common in the business world to have your PowerPoint decks “walk” around the organization after your presentation, meaning that you must create your presentation to be interpretable and easily understood even without talking points to accompany the slides.

The focus of the current column is just that: How does one effectively communicate validity evidence using only visualization? Despite the interesting and relevant topic, we must express how difficult it was for us to obtain visualizations that effectively represented validity evidence. We had several academics and practitioners volunteer contributions that either (a) we had difficulty understanding, (b) mirrored an output you might see in the Journal of Applied Psychology rather than a business meeting, or (c) weren’t representative of validity evidence. For example, although some business audiences may generally be able to interpret a scatterplot (see Don Zhang’s contribution below), we as a field can and should do a better job of taking a validity coefficient and presenting it such that the value can be grasped by anyone.

The goal of this column is to show how conducting rigorous research on the back end is important, but only so far as the evidence/results are effectively communicated to those who decide whether and how to act. We hope to demonstrate how one might effectively visualize validity evidence, including how one might visually present a rigorous analysis or complex model in such a way that non-I-O audiences can understand and appreciate its value. Like previous columns, our hope is that this shift in focus will provide anyone who finds themselves “lost in translation” with resources that not only helps the individual I-O psychologist, but also builds the awareness and use of I-O psychology in organizations. In the sections that follow, Don Zhang, Michael Litano, AJ Thurston, and Daniel Hawthorne each share some of the ways that they have effectively presented validity evidence.

Don C. Zhang

In a recent conversation with a director of human resources, he (the director) lamented about the low prediction of most selection methods. “Only about 50%,” he said, referring to the meta-analytic validities reported in the seminal Schmidt and Hunter (1998) paper. Two problems: first, a correlation of 0.50 translates to 25%—not 50%—of variance explained; second, only 50%? As a field, we are not doing a good job of communicating the value of our work when our most treasured selection methods are received with an unimpressed “meh” by people in positions to use them.

Traditional validity indices such as the correlation and coefficient of determination do not convey the practical impact that a selection method has on organizations. Percent of variance explained is not the kind of metric business professionals care about, and 25% is not a number that will impress many. But, 25% is about as high as we can get due to the “validity ceiling” of predictions (Rundquist, 1969). Clearly, there is a need to communicate our evidence in a way that resonates with the public.

There are many alternative graphical and nongraphical displays of validity information (Kuncel & Rigdon, 2012). But, not all data visualization techniques are created equal. The scatter plot, for example, displays the relationship between a predictor and criterion variables on the X and Y axis. The plot below illustrates the relationship between ACT and college GPA, which has a validity of r = .30: not particularly impressive. At a glance, the relationship depicted in the scatter plot appears equally unconvincing. Yet, scatter plots are the primary method for visualizing linear relationships.

Alternatively, one can use an expectancy chart (Lawshe & Bolda 1958). Expectancy charts communicates the relationship between two variables (e.g., ACT score and GPA) by presenting the proportion of the sample with score above a cut-off criterion (e.g., GPA above 3.5) at a given score interval on the predictor (e.g., ACT score between 25 to 27). Believe it or not, the expectancy chart below is generated with the same data as the scatter plot. Based on the expectancy chart, one can easily see the predictive efficiency of the ACT. Students in the top quintile of ACT have approximately 70% chance making the Dean’s List (GPA above 3.5) at most universities, whereas students in the bottom quintile of ACT have a less than 25% chance (25% takes on a whole new meaning in this context). 

 

Sometimes decision makers are interested in dichotomizing the predictor. For example, an academic administrator might be concerned with choosing an appropriate ACT cut-off score for admitting students into the honors college. To visualize the distribution of GPAs between individuals above or below a cutoff, we can use an overlapping density plot. The figure below illustrates the distribution of GPAs for students with ACT scores above and below 26. One way to translate the results is with the Common Language Effect Size (CLES, McGraw & Wong, 1992). The CLES for the validity of the ACT can be described as, “a randomly chosen person with ACT greater than 26 has a 62% chance of obtaining a higher GPA than a random person with ACT less than 26.”

Nontraditional effect size indices and displays are particularly useful when communicating the practical value of selection tests. Research from my lab has found that the lay public tend to judge nontraditional displays as easier to understand than traditional validity statistics; and more importantly, they had much more favorable judgments about selection instruments (e.g., ACTs, structured interviews) when validity information was presented with nontraditional graphical displays.

Despite the benefits of these nontraditional displays of validity, there are—unfortunately—no accessible tools for generating these displays. Common statistical packages (e.g., SPSS) do not readily produce these alternative displays. In order to facilitate the calculation of nontraditional effect size displays, I’ve created a free-to-use web application that allows scholars and practitioners to easily generate and visualize a variety of nontraditional effect sizes such as expectancy charts, CLES, and binomial effect size displays with their own data (https://dczhang.shinyapps.io/expectancyApp/).

Michael L. Litano

Entering the applied world, I incorrectly assumed that senior leaders and other organizational decision makers would be able to understand and interpret analyses of “people” data that extended beyond simple descriptives, such as favorability scores and agree percentages (i.e., percent of employees who “agree” or “strongly agree” to a single or set of questions). That’s not to say these leaders didn’t care or were incompetent, but because the way that validity evidence tends to be presented is too dense and unintuitive. If we assume the perspective of a senior leader with no I-O background and/or a limited understanding of statistics, what would we do with correlation coefficients, coefficients of determination, or (un)standardized parameter estimates?

I have conducted numerous interviews and focus groups with senior leaders and three themes generally emerge when it comes to digesting and interpreting survey results: they want (a) easily interpretable findings, (b) to see “trend”—how much scores have improved or declined since the last survey, and (c) what to focus on and how to drive meaningful change. As I-O psychologists, we are well-equipped to meet these needs, and I have found the best way to present this data is by doing my due diligence as a scientist in the background, then presenting findings that are easily interpretable even if it’s not the exact analysis that I conducted. After all, in people analytics, the goal is to use data-driven approaches to inform people and organization-related practices, programs, and processes, not to publish in a peer-review journal.

Here’s a scenario: Employee engagement is a high-value metric at Organization X. Employees are surveyed quarterly and each survey includes ten 3-item scales to measure constructs research suggests are primary antecedents of engagement (e.g., leader–employee relationship, professional development opportunities, etc.). Here’s one way I have analyzed such data in the background and then simplified the presentation of findings for a non-I-O audience.

Analysis: Every company has their own guidelines for presenting descriptive data (see translation section below), but when analyzing relationships between variables it’s still essential to methodically clean data, check assumptions, assess reliability of your measures, and demonstrate construct validity (correlations are fine with low N, but use confirmatory factor analysis with high N). At this point, we have some level of confidence that we are measuring distinct constructs that we are intending to measure. I don’t present any of these analyses to leaders outside of my team.

Because we have already identified employee engagement as the high-value metric of interest (let’s assume engagement is related to important HR and/or business outcomes), our goal is to be able to tell senior leaders with some degree of confidence what they should focus on to drive meaningful change. Especially when collected at one point in time, variables in organizational surveys generally share moderate-strong correlations with one another. Rather than rely on correlations (which do not provide accuracy metrics and consider only the focal relationship of interest) or multiple regression (which can provide misleading results when independent variables are highly correlated), I typically use relative importance analysis. This analysis treats independent variables as orthogonal so that you can more reliably assess the unique contributions of each. It easily identifies the most important “predictors” and sets us up well to translate.

Note: Scott Tonidandel & James LeBreton developed free programs for calculating relative weights in multiple, multivariate or logistic regression: http://relativeimportance.davidson.edu/

Finally, it’s important to consider the magnitude of the findings you are sharing. For example, if I was only able to explain 2% variance in engagement, how much of an impact will these variables really have? Conversely, in large sample sizes, everything is statistically significant, so it is important examine the practical significance (direction and magnitude) of effect sizes. These and other considerations must be made as part of your duty as a scientist.

Translation: Considering what I told you leaders care about, I could present this analysis in two very easy-to-understand slides:

1.       They want easily interpretable findings. Raw mean scores and standard deviations aren’t meaningful to most people, so there is a data transformation aspect involved with this step. Each organization has a “preferred” way of communicating results, so you can be more effective if you can cater to their norms. In Organization X, managers are used to seeing “agree percentages.” Therefore, I re-code each individual survey response variable to be binary (1 = agree or strongly agree on 5-point scale).

 

2.       They like to see “trend.” This isn’t validity evidence, nor is it something we often care about as I-O psychologists. But again, if you can take the leader’s perspective, it’s logical to wonder, “how were my scores this survey?”, “how do my scores compare to the organization?”, and “how do my scores compare to my historical scores?” Generally, Slide 1 answers these questions for the senior leader and prepares him/her for our “validity story”:

 

3.       They want to know what to focus on and how to drive change. Here, we take the general results of our relative importance analysis and translate it for senior leaders. In this analysis, assume we’ve identified two variables to have the strongest relationships with employee engagement: leader–employee relationship quality and professional development opportunities.

Even when I am not conducting a random forests or other tree-based analyses, presenting this in a decision tree has been one of the more effective ways to communicate this finding to demonstrate what employee engagement scores look like in instances where these two variables are rated favorably and unfavorably. There are a lot of rules that you—as an analyst, team, and company—have to decide upon (e.g., how many tree branches to present, what are the cut-offs for “yes” and “no” decisions, etc.). Once those rules are determined, you can use your stats program of preference to codify each of the four decision tree buckets (high-high, low-low, high-low, low-high) and create a pivot table that generates unique engagement scores for each. I always present these in a footnote so that even when the deck “walks,” questions about the analysis can be answered.

As you can see in the chart below, it’s easy to see the percentage of associates who are engaged in each of the four scenarios. Again, I’m trying to communicate that these are the two things senior leaders should act on—not publish in an academic journal—so I decided to cut the branches off at two layers and codify predictors into binary agree scores so that the “yes” and “no” decisions are more easily interpretable (i.e., “yes” means an associate “agrees” or “strongly agrees” on average to that set of questions). Slide 2 highlights that 85% of employees that have high-quality relationships with their leaders and have their professional development needs met are engaged, compared to only 15% who have neither.

 

For those senior leaders who are more statistically and analytically proficient, you can also try to present the results of a relative importance analysis like the below:

Daniel R. Hawthorne

When using this data visualization method, it is important to explain different ways that this shows the importance of each variable to the job(s) in question. Green shows the direction of the desired scores and the row-height indicates the strength of the influence. A lay person can rapidly look down the scores and see what the most important attributes are to a particular job.

After a lay person understands how the data is visualized, I can use this to get feedback about our data-driven regression equations and know if I need to consider adjusting weights based on attributes important to the client that we may not have picked up in data collection. We can also use this method to have new clients give feedback about transportable solutions and how they might fit with their jobs and, ultimately, how they might need to be adjusted to account for organizational differences between clients.

Additionally, if we’ve conducted a concurrent validation solution, this data visualization method can be used to adjust for an organizational change, where an organization wants to select for different employees than they currently have. For example, incumbent employees might show a strong association with problem solving and ingenuity, and a client might want to shift more to a model where caring and compassion attributes are the strongest weighted predictors. This method of data-visualization would show the existing employees’ current organizational profile and make it easy to have a conversation with the client about how new employees might be selected to push the organization toward a new organizational competency model in line with the organizational change.