New Guidelines for Technology-Based Assessment

John Weiner, Lifelong Learner Holdings, & Stephen Sireci, University of Massachusetts Amherst

Technology has become an essential part of assessment throughout the testing lifecycle. Test/item design, development, delivery, scoring, reporting, data storage, evaluation, and maintenance are all heavily technology dependent. This is true in the realms of workplace, educational, clinical, and professional licensing certification testing.

The International Test Commission (ITC) and the Association of Test Publishers (ATP) released in November 2022 a comprehensive set of guidelines specifically for technology-based assessment. The Guidelines for Technology-Based Assessment are the result of a multiyear collaboration by the ATP and ITC to fill an important need: to supply guidance and best practices for the design, delivery, scoring, and use of digital assessments while ensuring the validity, fairness, accessibility, security, and privacy of such assessments. Although other testing standards and guideline documents are available, these new Guidelines are unique in their comprehensive discussion of issues regarding the use of technology in assessment.

Note:  SIOP’s January 2023 publication of Considerations and Recommendations for the Validation and Use of AI-Based Assessments for Employee Selection (SIOP AI Considerations) is an excellent complement to the ITC/ATP Guidelines.

Background

Technology applications in assessment have become commonplace, such as technology-enhanced items, internet-based testing, remote online proctoring, data forensics, and biometric measures to authenticate examinees. More recent applications include game-based and gamified assessment, mining “big data” bases, digital social networks, and use of artificial intelligence and machine learning in assessment, all of which are becoming leading-edge practices.

Yet, as technology advances assessment practices, fundamental measurement concepts and concerns remain. It is critical to ensure that the use of technology in testing adds value without introducing irrelevant variance in scores or unintended consequences. In other words, technology-based assessments (TBAs) must remain valid for their intended purposes, yielding reliable and meaningful measurement in a manner that minimizes bias and ensures fairness. Further, as these new and enhanced technologies increase the global reach of assessment programs, accessibility and cross-cultural assessment and adaptation considerations become increasingly important.

The Guidelines were developed by an expert team of over 100 authors, technical reviewers, and advisers representing a range of practice areas and regions around the globe.

Content

The Guidelines document comprises four parts. Part I describes the background of, purpose for, and approach to developing the Guidelines, and outlines key related documents and references. In Part II, foundational concepts in measurement are discussed, such as validity, fairness, reliability and the need to manage against threats to measurement that may be introduced in technology-based assessments. Part III contains specific guidelines that are divided into 11 chapters. Each chapter begins with an introduction and discussion of important considerations for the relevant topic, followed by guideline statements. In most cases the guidelines are accompanied by commentary to explain and illustrate their relevance and application. Finally, Part IV provides a discussion of emerging applications of technology in assessment that are rapidly evolving and for which the ATP and ITC expect best practices and guidelines will be developed in the future.

Guideline Chapters

The 11 chapters comprising the Guidelines are outlined below to provide an overview of their breadth:

1. Test Development: Planning for technology-based assessments; technology-enhanced items; gamification, game-based assessment, and virtual performance assessments; universal test design; integrating assessment with instruction; item authoring.
2. Test Design and Assembly: Linear test design; adaptive test design.
3. Test Delivery Environments: Web-based delivery; offline, local, and mobile delivery; locked-down browsers; interoperability; test disruptions.
4. Scoring: Automated scoring of selected responses; automated scoring of constructed responses; technology-assisted human scoring; scoring in the event of technology disruption and   incomplete assessments; using item response time in scoring.
5. Digitally Based Results Reporting: Results reporting; quality control in score reporting; maintaining confidentiality of score reporting.
6. Data Management: Data storage; data maintenance, integrity, and security; integrating assessment data with learning systems.
7. Psychometric and Technical Quality: Score precision, comparability, and equating; measuring change and growth; validation of technology-based assessments.
8. Test Security: Technology-enabled test security.
9. Data Privacy: Privacy in technology-based assessment.
10. Fairness and Accessibility: Accessibility; equity issues in technology-based assessment.
11. Global Testing Considerations: Translation and adaptation; availability of technology resources; candidate preparation, practice, and orientation to technology.

As a review of these chapter titles indicates, the scope of the Guidelines is incredibly broad and should be of interest to those developing, selecting, evaluating, and conducting research on educational and psychological assessments.

To obtain a copy of the Guidelines. The Guidelines for Technology-Based Assessment are available in electronic format at no charge on the ATP and ITC websites.