The Bridge: Connecting Science and Practice

Sarah Layman, DCI; Jen Harvel, Amazon; & Apryl Brodersen, Metropolitan State University of Denver

"The Bridge: Connecting Science and Practice” is a TIP column that seeks to help facilitate additional learning and knowledge transfer to encourage sound, evidence-based practice. It can provide academics with an opportunity to discuss the potential and/or realized practical implications of their research as well as learn about cutting-edge practice issues or questions that could inform new research programs or studies. For practitioners, it provides opportunities to learn about the latest research findings that could prompt new techniques, solutions, or services that would benefit the external client community. It also provides practitioners with an opportunity to highlight key practice issues, challenges, trends, and so forth that may benefit from additional research.

In this issue, Megan Paul, Michelle Graef, and Robert Blagg discuss how using I-O psychology research and best practices improved workforce outcomes across 36 child welfare agencies. They share insights from a series of Quality Improvement Center projects on how to develop and test workforce interventions, use organizational data to improve workforce outcomes, and share learnings and resources for practitioners to use to further advance I-O research through practice.

The Quality Improvement Center for Workforce Development:
Bridging the Research–Practice Gap in Child Welfare

Megan Paul, Research Associate Professor, University of Nebraska–Lincoln, Center on Children, Families and the Law

Michelle Graef, Research Professor, University of Nebraska–Lincoln, Center on Children, Families and the Law

Robert Blagg, Research Scientist, University of California Los Angeles, Agile Visual Analytics Lab

Introduction

Challenges in staff recruitment, hiring, and retention are common in the field of child welfare (Graef & Potter, 2002; Graef et al., 2009). Responding to and addressing allegations of child maltreatment is difficult work, and it can exact an emotional toll on employees. The average length of tenure nationally is approximately 2.5 years. State, county, and tribal child welfare agencies struggle with high workloads and often insufficient funding. As in many other employment sectors, high levels of staff turnover can be costly (Graef & Hill, 2000). In addition, staff turnover in child welfare agencies imperils the helping relationship between the worker and the at-risk families they serve.

This article describes work done to address these challenges through a federally funded project called the Quality Improvement Center for Workforce Development¹ (QIC-WD). By collaborating with several child agencies to do applied research, the QIC-WD advanced the available evidence on a variety of workforce interventions. In addition, the QIC-WD advanced the use of organizational data in child welfare agencies through two projects: a cohort-based learning experience called the Child Welfare Workforce Analytics Institute and the development of customized workforce analytics dashboards in several agencies. Finally, the QIC-WD advanced the use of industrial-organizational (I-O) research evidence by creating resources that summarize the most robust findings so that both child welfare practitioners and researchers can capitalize on the many contributions that I-O psychology has made in the pursuit of effective organizations and worker well-being.

Overview of the Quality Improvement Center for Workforce Development (QIC-WD)

The QIC-WD was a cooperative agreement funded by the U.S. HHS/Administration for Children/Children’s Bureau (2016–2023) to serve public and tribal child welfare agencies throughout the country, with a goal of addressing these complex challenges using research to advance practice.

The QIC-WD was led by the University of Nebraska–Lincoln’s (UNL) Center on Children, Families and the Law, in partnership with academics from the University of Colorado Denver; the University of Louisville; the University of Tennessee, Knoxville; and the University of California, Los Angeles, along with several consultants. In addition to I-O psychology, this multidisciplinary project team brought expertise in child welfare practice, implementation science, evaluation research, dissemination, law, public policy, social and educational psychology, and social work.

At its inception, the QIC-WD solicited applications from agencies nationally and selected eight jurisdictions to participate in its primary multiyear projects to test interventions to improve workforce outcomes (e.g., employee performance, retention, and well-being). Several other initiatives were undertaken at various points in the life of the 7-year project, including some additional shorter term projects to generate more research evidence, efforts to advance the use of organizational data within child welfare agencies, and development of resources to advance the use of research findings in practice and in future research. Altogether, the QIC-WD served over 36 distinct agencies through one or more initiatives, with many agencies participating in more than one.

Building Science Through Practice

For the eight primary projects, each selected site was assigned a multidisciplinary QIC-WD WIE team comprising three people, each with expertise in workforce (W), implementation (I), or evaluation (E). The workforce specialists were all I-O psychologists, and the implementation and evaluation specialists had a range of professional and educational backgrounds, including social psychology, educational psychology, social work, social policy, organizational decision science, child welfare, and public administration. The QIC-WD provided financial support for sites to hire or appoint a site implementation manager and a data coordinator to ensure that agencies had sufficient commitment, capacity, and accountability throughout the project. Further, site implementation teams comprising key stakeholders were developed to provide guidance and approval of important recommendations. In addition to including child welfare professionals from all levels, it was important to also include representatives from Human Resources (HR), Information Technology, Training, and Quality Assurance/Continuous Quality Improvement. Membership evolved over time as the project needs changed, and additional stakeholders sometimes participated on a short-term basis. This broad and deep governance structure was instrumental in ensuring informed decision making and thorough communication, planning, and implementation.

The approach for each project was a multistage process of exploration, installation, initial implementation, and full implementation, with evaluation across all stages (Permanency Innovations Initiative Training and Technical Assistance Project, 2016). During the exploration stage, a needs assessment was conducted to examine workforce data and processes in many domains. A variety of workforce metrics were calculated, such as the number and distribution of positions in target jobs, vacancies, demographics, tenure, recruitment, selection, and types of turnover. Other workforce processes and programs were discussed, such as work arrangements, internship programs, onboarding, training, supervision, and performance management. If the agency had any recent employee survey data, those were examined, and the QIC-WD administered a survey to assess organizational culture and climate, job satisfaction, and organizational commitment. Thus, a wealth of information was gathered to understand strengths and opportunities. This extensive, deliberate process helped agencies to slow down and be more thoughtful and data driven, which they typically didn’t have the time and resources to do. After a thorough review of available information and identification of key problems and their potential causes, the WIE teams offered suggestions of potential interventions to test. Each site identified one intervention, which was either custom developed for the agency, an existing intervention, or an adaptation of an existing intervention.

Altogether, the following interventions were selected: job redesign, case-supportive technology, telework, an organizational change process called ARC (Hemmelgarn & Glisson, 2018), competency-based selection, onboarding, a supportive supervision coaching model (ACCWIC, 2013), and Resilience Alliance—a program to address secondary traumatic stress (ACS-NYU Children’s Trauma Institute, 2011). The research designs varied in rigor, from case studies and quasi-experimental designs to cluster randomized control trials. Intervention implementation followed an implementation science framework with prescribed stages and supports (Bertram et al., 2015). I-O psychologists who are not familiar with this body of work may find some useful guidance for deploying new programs in organizations. Finally, a myriad of process and outcome variables were measured, some specific to a site and some across all sites. Common types of variables include learning, reactions, job attitudes, stress indicators, performance, and turnover. Data for these variables were drawn from agency administrative data, surveys, interviews, and focus groups.

Through this highly collaborative approach, participants develop valuable buy-in and learn how they can do some of this work on their own in the future. For federal projects of this nature, organizational capacity building is an important goal. Agency partners invest significant time and effort and make accommodations to allow for rigorous field research, but they also gain knowledge and skills along the way. We believe this approach is a mutually beneficial way to do applied research.

Advancing the Use of Organizational Data

Because of the widespread need for child welfare agencies to better capitalize on their personnel data, we were asked to expand our reach in this area, which we did by creating the Child Welfare Workforce Analytics Institute. Small teams of representatives from selected agencies participated in several webinars and a 3-day workshop to learn about a variety of workforce analytics topics and develop an action plan to improve an aspect of their workforce analytics capacity and practice. In our previous work with agencies, we had seen a trend of disconnects between HR and those doing the direct work of child welfare, which struck us as a missed opportunity. Often, both parties were aware of and trying to address workforce challenges but not always in coordination. We felt that they were each missing the relevant subject matter expertise of the other. Thus, teams deliberately included a mix of people with roles in HR or in child welfare. Existing relationships varied across teams; some already worked together regularly, and some teams had very limited exchanges. The use of workforce data also varied across teams. Smaller agencies with fewer resources tended to capture more limited data, stored in spreadsheets, whereas larger agencies had more sophisticated software. Some had very little means to systematically examine their data, whereas others had significant means, sometimes untapped and sometimes resulting in outputs that were too overwhelming for users.

Each team was assigned a QIC-WD representative to serve as a liaison and coach, who met with the team multiple times throughout the project. In the webinars and workshop, participants learned about a variety of topics, including creating a workforce analytics team, data dictionaries, data quality, workforce metrics, linking data, workforce scorecards, and data visualization. In addition to explaining and discussing concepts in these areas, we provided written resources, which are now publicly available on our website, along with webinar video excerpts. Teams identified a focus area and an action plan, which they presented to everyone at the conclusion of the workshop. In two follow-up webinars, teams reported back on the successes and challenges of their projects. A total of 20 child welfare agencies participated via two cohorts over the course of about 1 year.

Another means of advancing the use of organizational data involved supporting data visualization. The use of data visualization in workforce analytics has become increasingly essential in recent years. Public child welfare agencies collect vast amounts of workforce data related to recruitment, hiring, development, and retention but often have difficulty unlocking the potential of this data. To support these agencies, the QIC-WD partnered with eight public child welfare agencies to develop workforce analytics dashboards that visualize workforce analyses and metrics.

Again, each jurisdiction brought partners to the table they deemed important to furthering the workforce analytics process. The QIC-WD supported these teams in different ways, depending on each agency’s data needs and analysis capacities, to turn new or existing data into visual reports infused with key workforce metrics, with a focus on data products that were dynamic (i.e., meeting varied information needs), accessible, utilized in making informed decisions, and sustainable. In some cases, the QIC-WD supported agencies throughout the process of data identification, preparation, analysis, and visualization development. In other cases, the QIC-WD provided consultation and technical assistance to agency staff building visual reporting.

The QIC-WD worked collaboratively with each public child welfare agency to make better use of organizational data so as to enhance evidence-based management practices through data visualization. These more dynamic ways of reporting workforce metrics are already helping public child welfare agencies make more informed decisions regarding recruitment, hiring, development, and retention efforts. Agencies we have supported in the past year have already made changes (i.e., to recruitment, hiring, retention policies and procedures, and measurement, data collection, analysis, and reporting practices) based on insights derived from our workforce analytics dashboards. For example, one agency has integrated review of these dashboards into the regular processes of their employee “wellness roundtable” and other workforce development-related standing workgroups. One agency partner described that “we have a tool that people can actually rely on and trust that they can in real-time look at the data and make informed decisions.” Another partner noted their dashboards will “help to serve as the basis to hypothesis test our strategies.”

Advancing the Use of Research

We used two primary strategies to advance the use of past I-O research evidence. The first approach was to use it to inform our work, and the second was to share research findings with child welfare practitioners and researchers. Previous research regularly informed our project goals, processes, and products. We modeled evidence-based management with our project sites by drawing on the literature to drive planning and decision making. In developing and selecting measures, we chose the most well-established scales that fit our purpose and target populations. All of this not only strengthened the quality of our work but also forged a connection between I-O research and child welfare organizational and personnel practices. Because of the long-standing workforce challenges in child welfare, there has always been great interest in better understanding and addressing those issues. There are hundreds of articles in child welfare, human services, and social work journals about chronic concerns related to recruitment, selection, training, supervision, workload, burnout, stress, satisfaction, and turnover, among others. Though there is occasional recognition of relevant research and measures outside of the child welfare domain, most of the literature does not draw on I-O research or best practices. Without the benefit of this strong foundation, research is often fragmented, making steady advancements difficult to achieve. This compounds the problems that practitioners already have with the typical research-to-practice gap: Not only is research difficult to access, but there are fewer reliable findings available.

All of these issues led us to develop resources to distill the most relevant and conclusive research for child welfare practitioners and researchers. We searched 29 journals and located over 1,500 meta-analyses, which were then filtered for applicability and sorted into over 100 topics. A product series called Umbrella Summaries was developed to briefly describe the most up-to-date meta-analytic findings on a given topic. The summaries are typically about three to four pages in length and describe (a) the construct or practice; (b) the most common or recommended way of operationalizing and measuring it; (c) the key findings related to a set of core outcomes, such as learning, job attitudes, stress indicators, performance, and turnover; and (d) a bulleted list of takeaways. Example topics include realistic job previews, situational judgment tests, organizational socialization, emotional labor, behavior modeling training, and organizational justice. For those who want more in-depth information, further details are presented via the Workforce Research Catalog, which is an interactive data visualization that includes effect sizes, confidence intervals, and the number of studies and participants. Users can filter by topics, topic categories, predictors, and outcomes to see how the results compare. There is a researcher version with more details and a practitioner version with fewer details. The collective goal of the summaries and the data visualizations is to bridge the gap between research and practice. Practitioners can use evidence to enhance their workforce practices, and researchers can advance the evidence base by capitalizing on the strengths of past research, identifying limitations and needs, and testing new hypotheses. Finally, practitioners and researchers can work together to develop evidence that is grounded in the practical realities, needs, and interests of agencies and the workforce.

Though the topics were chosen with child welfare workforce issues in mind, the summaries and catalog are likely also useful for other practitioners and researchers outside of child welfare. For I-O psychologists, the most noteworthy takeaway is that though there are hundreds of meta-analyses, there aren’t nearly enough that cover interventions that can be implemented to address workforce or workplace challenges. A large number of meta-analyses describe correlational findings among many variables, some of which are suspected antecedents and others that are suspected outcomes. These are no doubt valuable findings, but we need to know more about what can be done to intervene and make reliable changes to those variables, as evidenced by experimental or quasi-experimental designs. The areas that most include concrete strategies are selection and training, though many of the latter come from educational psychology research. I-O practitioners surely have many personnel and organizational solutions to offer their customers, but there are many employers, particularly in the nonprofit and public sectors, that do not have the resources to obtain consulting services, and many have very similar problems that do not necessarily require custom solutions. Organizations in the field of human services, along with other helping professions, are most likely to need I-O solutions and least likely to be able to access them. Thus, there is a need to make information more widely available about strategies that employers can use to be more effective. That is not to say that additional expertise and resources would not possibly be needed to adapt and successfully implement such strategies, but many organizations are eager to have more evidence-based practices and programs available to help them cope with chronic problems related to employee performance, retention, and well-being, and our research review indicates opportunities for further advancement.

Accomplishments and Resources

The accomplishments of the QIC-WD have been many. The QIC-WD built the capacity of child welfare agencies to appreciate the importance of relying on research evidence to inform agency workforce decision making. Agencies that participated in the primary site interventions learned how to conduct a workforce needs assessment and the value of “pumping the brakes” to become less reactive and more deliberative in response to their workforce challenges. Several agencies elected to sustain and scale up their tested intervention agency wide once results of their evaluation were available; in addition, several of the tested interventions are under consideration by other child welfare agencies across the country.

The QIC-WD produced a wide range of resources, including short videos, webinars, guides, conference presentations, data visualizations, and research summaries, which are all available via the website (www.qic-wd.org). Results of the evaluation research are still being developed into academic journal articles. In addition, all resources are available through the UNL Digital Commons to ensure ongoing open access internationally.

Note

^[1] Note that in child welfare, the term workforce development refers to efforts to improve such things as the recruitment, selection, performance, retention, and well-being of employees—much of the typical work done by I-O psychologists. In contrast to how the term is typically used elsewhere, workforce development does not refer to enhancing the skills of unemployed or underemployed individuals.

References

Atlantic Coast Child Welfare Implementation Center (ACCWIC). (2013). Coaching in child welfare: Two day training curriculum. https://www.qic-wd.org/sites/default/files/ACCWIC%20coaching%20curriculum%20r.pdf

ACS-NYU Children’s Trauma Institute. (2011). The Resilience Alliance: Promoting resilience and reducing secondary trauma among child welfare staff. https://www.nctsn.org/resources/resilience-alliance-promoting-resilience-and-reducing-secondary-trauma-among-welfare-staff

Bertram, R. M., Blase, K. A., & Fixsen, D. L. (2015). Improving programs and outcomes: Implementation frameworks and organization change. Research on Social Work Practice, 25(4), 477–487. https://doi.org/10.1177/1049731514537687

Graef, M., & Hill, E. (2000). Costing child protective services staff turnover. Child Welfare, 79(5), 517–533. https://www.qic-wd.org/sites/default/files/Graef_Hill_2000.pdf

Graef, M. I., & Potter, M. E. (2002). Alternative solutions to the child protective services staffing crisis: Innovations from industrial/organizational psychology. Protecting Children, 17(3), 18–31. https://ccfl.unl.edu/docs/Graef_Potter.pdf

Graef, M. I., Paul, M. E., & Myers, T. L. (2009). Recruiting and selecting child welfare staff. In F. Alwon, S. Steib, & B. Schmitt (Eds.), On the job in child welfare: Recruiting, retaining, and supporting a competent workforce. Child Welfare League of America.

Hemmelgarn, A. L., & Glisson, C. (2018). Building cultures and climates for effective human services: Understanding and improving organizational social contexts with the ARC Model. Oxford University Press. https://doi.org/10.1093/oso/9780190455286.001.0001

Permanency Innovations Initiative Training and Technical Assistance Project. (2016). Guide to developing, implementing, and assessing an innovation. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Children’s Bureau.