Society for Industrial and Organizational Psychology > Leading Edge Consortium > Program

Program Block 1: Artificial Intelligence Applications to Assessment

Explain the key components of a conceptual framework of the relationships between job-relevance and large language model (LLMs).
Describe at least one method for exploring explain-ability in AI models used for assessment.
Explain the process for training a machine learning model to replicate expert scoring of audio-based constructed response assessments.
Compare and contrast large language models, both in general and in relation to specific software, with other types of artificial intelligence and machine learning used in assessment.
List and describe major areas of personnel assessment that have been and could be affected by using large language models, both by assessment professionals and by those completing assessments
Engineer ChatGPT prompts to generate high-quality draft assessment materials.

Describe how to integrate three- to five-DEIAB definitions into psychological assessment procedures to enhance the validity, fairness, and cultural responsiveness of those assessments.
Describe the use of at least two statistical methods to detect bias in computer adaptive assessment.
Describe how to implement accommodations to facilitate candidate accessibility--including compatibility, scheduling, and proctoring--to enhance fairness and diversity for pools of candidates.
Describe how to revise assessment procedures to ensure accessibility for the autistic community.
Describe how to implement item-sensitivity procedures to promote inclusion of underrepresented individuals in the test development process.
Describe how to evaluate, select, implement, and govern selection assessments from the perspective of diversity, equity, and inclusion goals.

Illustrate at least one concern with how the range restriction corrections were conducted in the seminal Schmidt and Hunter (1998) article and how validity estimates of various selection procedures change when more conservative corrections are used.
Describe current meta-analytic estimates, based on 21st century validation studies, of general cognitive measures in predicting overall job performance.
Identify three examples of implications of recent research on assessment validity when considering the validity diversity tradeoff.

Analyze and evaluate two assessment applications and processes for compliance with data privacy regulations.
Discuss the complexity and privacy implications of collecting data from participants who may be testing at home or another remote location (where one might inadvertently capture more than the intended responses from examinees).
After weighing business and program variables, discuss which of three assessment modalities would be the most successful for a situation.
Describe how to create a plan for implementing remote proctored assessments.
Identify at least three challenges to assessment security posed by generative AI.
Describe two assessment strategies that take the implications of generative AI into consideration.

Analyze the impact of aligning talent and assessment strategies for developing leaders across all levels of the organization.
Explain the role of assessments in coaching and advancing leaders and the impact they can have on leadership effectiveness.
Identify two innovative assessment techniques and tools that have proven impactful in coaching and leadership development.
Explain how a multi-level approach can be used to evaluate effectiveness of organizational talent models.
Describe how a multi-level approach helps organizations evaluate employee growth and development over time.
Identify how current thinking about leadership, teams, and teambuilding might be preventing organizations from building more effective teams.
Describe how to measure and improve team effectiveness.
Describe how to create a high-performing team culture using the tools described during the session.

Explain the implications of new city, state, and federal regulatory guidelines related to AI-based assessment practice.
Explain how to plan preliminary compliance strategies for existing assessment and selection programs related to current city, state, and federal AI-regulations and/or guidance.
Discuss the implications of the Supreme Court rulings related to (1) educational admissions, (2) affirmative action in employment, and (3) diversity equity and inclusion initiatives.