The Bridge: Connecting Science and Practice

Kimberly Adams, Independent Consultant, & Stephanie Zajac, UT MD Anderson Cancer Center

“The Bridge: Connecting Science and Practice” is a TIP column that seeks to help facilitate additional learning and knowledge transfer to encourage sound, evidence-based practice. It can provide academics with an opportunity to discuss the potential and/or realized practical implications of their research as well as learn about cutting-edge practice issues or questions that could inform new research programs or studies. For practitioners, it provides opportunities to learn about the latest research findings that could prompt new techniques, solutions, or services that would benefit the external client community. It also provides practitioners with an opportunity to highlight key practice issues, challenges, trends, and so forth that may benefit from additional research. In this issue, the 2019 M. Scott Myers awardees describe the rigorous interdisciplinary approach drawn from the fields of I-O psychology, biomechanics, and exercise physiology that was used to develop a valid, fair, gender-neutral physical test battery for selection into U.S. Army combat arms jobs previously closed to women.

Development and Validation of the Occupational Physical Assessment Test (OPAT) for Army Combat Arms Military Occupational Specialties

Deborah L. Gebhardt^a, Todd A. Baker ^a, Marilyn A. Sharp^b, Jan E. Redmond^b (not pictured), Stephen A. Foulis^b, Peter N. Frykman^b, Edward J. Zambraski^b

^a Human Resources Research Organization (HumRRO)

^b U.S. Army Research Institute for Environmental Medicine

We at HumRRO and the U.S. Army Research Institute for Environmental Medicine were honored to receive SIOP’s M. Scott Myers Award for Applied Research in the Workplace in 2019 for research that provided the U.S. Army (hereinafter Army) with an assessment that resulted in substantial return on investment and that was expanded to all personnel in the Active Army, Army National Guard, Army Reserve, West Point, Officer Candidate School (OCS), Reserve Officers’ Training Corps (ROTC), and Direct Commissions (Army Medical Department, Judge Advocate General’s Corps).

In 2013 women formally were provided the opportunity to serve in direct combat roles in all military branches when the United States Secretary of Defense and the Chairman of the Joint Chiefs of Staff for the Department of Defense (DoD) rescinded the 1994 Direct Ground Combat Definition and Assignment Rule that banned women from combat roles. In addition, the U.S. Congress mandated that the standards for each combat arms job be based on documented job requirements and that there should be no “artificial barriers” to women’s entry into a combat arms career field. Thus, the DoD and Congressional act opened the door for women to pursue combat arms jobs.

To address the DoD decision and Congressional mandate, the Army conducted a 4-year study to provide physical-assessment procedures for individuals entering combat arms Military Occupational Specialties (MOS). The Army’s goal was to develop a valid, legally defensible, and safe physical assessment to predict a soldier’s ability to serve in a combat role. Paramount to the goal was ensuring that physical capabilities of soldiers are adequate to meet the demands of the combat arms MOS shown below:

• 11B Infantryman                                              
• 11C Infantryman-Indirect Fire                      
• 12B Combat Engineers                               
• 13B Cannon Crewmember
• 13F Fire Support
• 19D Cavalry Scout
• 19K Armor Crewman

A unique challenge to conducting the research was the absence of women in the seven jobs at all phases of the study. Thus, large-scale innovative efforts to recruit women soldiers were undertaken while adhering to strict human-subjects’ criteria. The project was carried out in six phases: (a) job analysis, (b) physiological assessment of job tasks, (c) criterion development, (d) predictor test identification, (e) concurrent validation, and (f) predictive validation. Over 6,300 soldiers participated in the study. Table 1 shows the number of soldiers participating in each study phase.

Table 1
Soldier Participation

^* Phase 4 did not involve use of human subject.

     Job analysis.

The project team conducted job analyses for the seven MOSs involving document review, on-site observations, interviews, and a job analysis questionnaire (JAQ). The Army identified 32 arduous tasks across the seven MOSs. Research staff observed 440 soldiers performing the 32 tasks and received input during focus groups from 105 enlisted subject matter experts (SME) who were deployed one or more times. Over 3,700 soldiers in the seven MOSs completed a JAQ to rate the tasks and provide ergonomic information related to task performance. The results showed that all 32 tasks were important to multiple combat arms MOSs.

     Biomechanical and physiological assessments.

Combat arms male soldiers (n = 152) and female soldiers (n = 90) performed the 32 tasks to obtain physiological measurements such as aerobic demand (oxygen uptake), heart rate, and ratings of perceived exertion. During task performance, soldiers wore the required clothing and equipment (e.g., 24-hour sustainment load of 109 pounds) when performing each task in combat situations. Examples of tasks performed included preparing a fighting position, Bradley Fighting Vehicle (BFV) casualty evacuation, and transfer of 55lb ammunition rounds. Results showed muscular strength and aerobic demand (VO₂) varied across tasks (12–36 ml • kg^–1 • min^–1). Review of the aerobic data showed men and women working at approximately 60% of their maximum VO₂.

     Criterion development.

Project staff reviewed job analysis, biomechanical, and physiological data to identify redundancy across the tasks in the combat arms jobs in terms of movement categories (e.g., lift). This resulted in five movement categories: repetitive lift and carry, heavy lifting, load carriage, agility, and drag. We used the biomechanical/physiological and job analysis results to group the 32 tasks into the five categories and identify the tasks with high importance, frequency, and aerobic and strength demands. Eight of the 32 tasks represented the demands of the jobs and were used for development of criterion measure task simulations (CMTS). Examples of top-ranked aerobic tasks included “foot march” and “build a fighting position.” Examples of top-ranked strength tasks included “the casualty drag,” “Bradley Fighting Vehicle (BFV) casualty evacuation,” and “stow ammo on tank.”

To assess the reliability of the criterion measures, 149 soldiers completed each CMTS three to four times on different days. We employed intraclass correlations (ICC), standard error of the mean (SEM), and limits of agreement (95% LOA) to assess reliability. The ICCs between Trials 2 and 3 were 0.85 and higher for all CMTSs, except the foot march (0.76). The SEM and 95% LOA assessed score variability between trials and identified the number of practice trials needed for consistent performance. These findings determined the number of CMTS practice trials used in the validation study.

     Predictor-test identification.

 The Army’s goal was to administer a physical predictor-test battery at the recruiting stations to identify recruits capable of performing in the combat arms MOSs. Due to safety and logistics parameters at recruiting stations (e.g., space, resources, replicability), the Army decided to use basic-ability tests. Project staff identified physical tests that assessed the abilities required to perform the 32 tasks and eight CMTS. The predictors included 14 basic-ability tests that assessed muscular strength, muscular endurance, aerobic capacity, and anaerobic power.

     Concurrent validation.

The research team conducted validation studies for each MOS to establish the relationship between the predictor tests and CMTS. A total of 838 soldiers (608 men, 230 women) completed the predictor tests and CMTS across all combat arms MOSs. The team generated multiple test batteries based on multiple-regression results and the Army’s testing needs. The final test battery, the Occupational Physical Assessment Test (OPAT), consisted of four tests (seated power throw, squat lift, beep test, and standing long jump) and had a multiple R² of 0.79–0.80 (p < .01). Across all combat arms MOSs, this battery correctly classified CMTS performance (successful/unsuccessful) for 77% to 90% of the soldiers.

     Predictive validation and implementation.

A predictive validation study was conducted to assess OPAT performance in a recruit setting. The OPAT served as the predictor and CMTS, injuries, and attrition served as the criterion measures. Recruits completed the OPAT at the beginning of initial entry training (IET) and CMTS at the completion of IET. Injury and attrition data were collected across their first enlistment. Seven hundred forty-one recruits completed the OPAT and CMTS. Regression analysis using the CMTS as criteria yielded an R² of 0.70. The research team selected a multiple-hurdle-scoring approach and established passing scores using job analysis, validation results, Army standards for the CMTS (e.g., 15 meter casualty drag in 60 seconds), and OPAT improvement during IET.

Implementation impact.

In the predictive study, soldiers who graduated from IET had significantly higher OPAT scores than nongraduates. Further, noninjured recruits performed significantly better (p < .001) on each test in the battery (d = 0.25–0.40), and injury risk for low-scoring recruits was 1.5 to 1.8 times (p < .01) higher comparted to recruits who met the cut scores. Of the recruits who attrited from IET (n = 168), 31.6% were due to injury/medical reasons.

To calculate annual cost savings for implementation of the OPAT, we used several approaches based on the Army’s cost of $73,700 (in 2017) for a recruit to complete IET. Using only injured recruits who failed to meet the OPAT standards (31.6% of 92), the Army would save approximately $2.14 million from the predictive validation sample. Due to the success of the OPAT to provide qualified individuals for Army jobs and a substantial return on investment (ROI), OPAT was expanded to jobs with less physical demands and all enlisted personnel. Cut scores were established for the three additional demand levels yielding four levels (e.g., Level 1 = combat arms jobs; Level 4 = not ready to ship/not entering Army regardless of job).

The Army inducts 120,000 recruits annually and projects a 10% yearly attrition from IET (9.6%–11.6%) at an average cost of $50K. The Training and Doctrine Command (TRADOC) estimated reducing attrition just 1% would result in a cost savings of $60 million annually (1200 * $50k). For an injury-attrition ROI, we used the percentage of attritions due to injury/medical reasons from the OPAT study (31.6%) and applied it to the total extrapolated attritions in the 4^th level OPAT category (n = 1,859). Using a median cost for attrition of $47.85K (range of $23K–$73.7K at different points of the study), the savings were $88.94 million. Regardless of the method used to calculate potential savings, these values are conservative when one considers costs such as additional recruiting the next year for slots not filled due to IET attrition.

Summary

The U.S. Congress mandated that the military services develop fair, gender-neutral evaluation procedures for selection into combat arms MOS that predict critical task performance. To meet this mandate, the Army stipulated that the assessment be valid, legally defensible, and in compliance with federal statutes and guidelines (EEOC Uniform Guidelines, 1978) and professional principles, even though they are not subject to these statutes.  They also required that the assessment be administered at recruiting stations using limited equipment. To meet these requirements the Army and HumRRO conducted a detailed 4-year study to develop and validate a physical-test battery that predicts performance of arduous tasks in combat arms jobs previously closed to women. We used a rigorous interdisciplinary approach to design a comprehensive methodology that drew from the fields of I-O psychology, biomechanics, and exercise physiology. One challenge this study overcame was conducting data collections in a field setting while maintaining standardized procedures used in a laboratory setting. A second was recruiting soldiers in compliance with the Army’s strict protocols for studies with human subjects. Particularly difficult was recruiting women soldiers in noncombat MOSs to participate in the numerous data collection phases. Strategies used to recruit volunteers for participation appealed to intrinsic factors such as knowledge of results, opportunity to practice tasks to improve, helping the Army improve, and being part of opening combats arms MOSs to women.

Our integrated approach identified the critical tasks for the combat arms MOSs and the actual physical demands of these tasks based on physiological data in terms of aerobic capacity and force production. These combined data resulted in selection of tests and criterion measures that addressed the physical demands of the critical tasks. Physical assessment research guided test selection in terms of reliability, prior evidence of validity, use of objective scoring, and differences in performance. Typically, there are no performance standards for most physical jobs. A unique aspect of this project was having Army performance standards for the many CMTS (e.g., 15 meter casualty drag in 60 seconds). In addition to the job analysis data and concurrent validation study, these standards provided evidence supporting test passing scores and classification effectiveness.

Our integrated approach resulted in the OPAT, which met Congress’ mandate and the Army’s technical and administrative needs. Further, the OPAT’s predictive validation study showed reduction in attrition from basic military training and an estimated cost savings of $2.14 million and higher attributed to injury/medical attrition. The Army estimated the yearly savings would be $60 million from an attrition reduction of just 1%.

Reference

Uniform Guidelines on Employment Selection Procedures (1978). Federal Register, 43 (August 25, 1978), pp. 38290–38315; Code of Federal Regulations, 41 CFR 60-3.