History Corner: John C. Flanagan's Contributions Within and Beyond I-O Psychology

 

 

 

 

 

 

 

John C. Flanagan’s Contributions Within and Beyond I-O Psychology

 

Jeffrey M. Cucina

(202) 863-6298; jcucina@gmail. comand Jeffrey. Cucina@dhs. gov

U. S. Customs and Border Protection

1400 L Street, NW, 7^thFloor

Washington, DC 20229-1145

& 

Nathan A. Bowling

(937)775-2483; nathan. bowling@wright. edu

Department of Psychology

Wright State University

303C Fawcett Hall

3640 Colonel Glenn Highway

Dayton, Ohio 45435-0001

Article for History Corner section of the January 2016 issue of TheIndustrial-Organizational Psychologist. 

Note. The authors would like to thank Susan Lapham for her valuable comments and suggestions on this paper. The views expressed in this paper are those of the authors and do not necessarily reflect the views of U. S. Customs and Border Protection or the U. S. Federal Government. 

In the SIOP 2015 closing plenary session, President Kozlowski shared his vision of expanding I-O psychology’s horizons to other fields, such as education, medicine, and STEM disciplines. One notable example of a historical SIOP figure expanding his or her horizons to other fields is John C. Flanagan(Figure 1). This edition of the History Corner discusses Flanagan’s career. We begin with a brief biography of Flanagan’s life, which is largely drawn from the American Psychological Association (APA; 1977), Clemans (1997), and Freeman (1996). 

 

 Biography

 

John C. Flanagan was born in Armour, South Dakota on January 7, 1906 to a Baptist minister and a schoolteacher. At the age of 3, his family relocated to Washington State, where he spent the rest of his childhood and attended college (APA, 1977). He began his undergraduate education studying electrical engineering and physics but eventually discovered an interest in research on human behavior. After several years working as a high school math teacher, he enrolled in the PhD program in mental measurement at Harvard University, working in Truman Lee Kelley’s laboratory. He graduated with a PhD in only 2 years and then began working at the American Council on Education’s Cooperative Test Service where he directed a large-scale annual achievement test. 

 

While working at the Cooperative Test Service, Flanagan directed a research project that involved interviewing pilots in the U. S. Army Air Corps. [1]He was later recruited to become a commissioned officer in the Air Corps, a position that led to his most important professional contributions and one that would shape the course of his career. He then became a psychology professor at the University of Pittsburgh and started the American Institutes for Research (AIR). Later in his career he served as president of four divisions of APA, the National Council on Education, and the Psychometric Society. He continued to pursue his interest in developing test batteries as he published two tests that bear his name: the Flanagan Industrial Tests(1975) and the Flanagan Aptitude Classification Tests(Flanagan, 1957). Flanagan is also one of five SIOP members who won APA’s Distinguished Professional Contributions Award (SIOP, 2015). Six years later, he received SIOP’s Distinguished Professional Contributions Award. His legacy lives on today as the namesake of the award for the best student SIOP poster. 

 

Army Air Corps

 

In 1941, Flanagan joined the U. S. Army Air Corps as a commissioned officer. At the time of his recruitment, the U. S. was on the verge of entering World War II and the Army wanted a plan to grow the Army Air Corps. In 1941 there were only 51,000 servicemen in the Air Corps. Aware that the number would have to increase substantially (it swelled to nearly 2. 3 million by 1945), the Army knew it needed a team of psychologists to develop a selection system. After enlistment, Flanagan immediately formed the U. S. Army Air Corps’ Aviation Psychology Program and hired a group of 36 psychologists, which eventually grew to approximately 150 psychologists and 1,500 psychology assistants. The group’s responsibility was to devise a system to select military recruits and place them into the different positions needed to run an air force (e. g., pilots, copilots, navigators). 

 

To address this challenge, Flanagan proposed a multiple-hurdle assessment strategy (Clemans, 1997; Flanagan, 1946, 1947). His plan, which used an initial qualifying test followed by a more extensive 20-test placement battery, was adopted less than a week after Congress declared war and was operational in less than a month. Enlistees first completed the Army General Classification test (a measure of general mental ability; Jensen, 1998) as an initial screening test. Those who passed the initial screening—approximately half of the 1. 25 million who were tested—completed a subsequent battery of 20 psychological tests. This subsequent battery included both “printed tests” (e. g., measures of mechanical reasoning and map reading) and “apparatus tests” (e. g., measures of reaction time and finger dexterity; see Flanagan, 1946, 1947). The scores for this test battery were on a nine-point scale and were the original example of the stanine (or “standard nine”) scale. 

 

Flanagan provided substantial evidence for the validity of this assessment strategy (see Flanagan, 1946, 1947, 1948). In one study, enlistees (N= 1,143) were sent to pilot training regardless of their performance on the assessment tests (see Flanagan, 1946). Of note, 42% of these enlistees would normally have been excluded from pilot training due to low assessment scores. Performance on the assessment battery was positively related to successful completion of pilot training and negatively related to involvement in aircraft accidents. This effort was later hailed as “the largest and most successful applied psychology program ever undertaken” (Carter as quoted in Clemans, 1997). It was also credited with “sav[ing] a lot of money and…lives in training” (Clemans, as quoted in Freeman, 1996). Upon leaving the military in 1946, Flanagan was awarded the Legion of Merit for his contribution to the war effort. 

 

American Institutes for Research

 

At the close of World War II, Flanagan had amassed a large cadre of psychologists in the U. S. Army Air Forces. With combat over, there was a need for them to return to civilian life. Flanagan realized the potential for applying the principles of psychology that were used in the Air Corps to areas outside the military (APA, 1977). So, he decided to create a nonprofit civilian research organization, and he recruited many of his U. S. Army Air Forces colleagues to join him. In 1946, the American Institutes for Research (AIR) was born. Flanagan and his team adopted many of the methods they developed in the military for use in their work at AIR (Flanagan, 1984). Indeed, the work Flanagan’s team conducted developing selection systems for aircrew members was directly relevant to AIR’s first customer, Trans World Airlines, for whom they developed a pilot selection system (AIR, 2015). Other early projects include job analyses for medical interns, orthopedic surgeons, and police officers, as well as human factors studies, health care studies (e. g., smoking, cancer patients), and selection test development and validation (Flanagan, 1984). 

 

While at AIR, Flanagan also made contributions to the field of medicine. His work at AIR led to the improvement of the validity of licensing examinations for medical doctors and the development of a first-aid training, which was cited as being “especially effective and efficient” (APA, 1977, p. 75). He also led nationwide studies of the quality of life of different age groups of Americans (APA, 1977). Most significant, however, was his work studying individual differences in America’s youth. 

 

Project TALENT[2]

Planning

 

In 1956, Flanagan was in the midst of planning one of the largest (if not the largest) empirical psychological studies ever to take place. He wanted to conduct a census of the talent of America’s youth by developing and administering a battery of mental ability and achievement tests, interest inventories, personality tests, and other measures (e. g., home background, activities, future plans). Flanagan originally came up with the idea for Project TALENT many years early. He proposed a similar study to the American Council on Education in 1939; however, the technology that would be needed for scoring and processing the tests and data for such a large study was not yet developed (Susan Lapham, personal communication, November 24, 2015). He was later reminded of the need for such a study while developing one of his test batteries—the Flanagan Aptitude Classification Tests—which became a commonly used personnel selection and counseling test. Flanagan was unsatisfied with his efforts to develop norms for the battery, and he came to the realization that one of the best parties to sponsor the development of nationwide norms would be the federal government (Shaycoft, 1977). According to Shaycoft, Flanagan’s reasoning was that once the nationwide norms were established on a wide variety of tests, future test developers could use the Project TALENT tests as anchors for equating to the nationwide norms. [3]In addition, the study could identify the correlates of success in learning, high school academics, and college. 

 

Flanagan began his efforts by informally meeting with officials from various federal agencies to gain support for the project (Shaycoft, 1977). Afterward, a working group was formed with representatives from multiple organizations to plan the project (AIR, 2011; Project TALENT, 1960). He finally obtained funding for Project TALENT from the U. S. Office of Education in 1959 (AIR). At that time this type of federal grant could only go to an academic institution, so the money was officially granted to the University of Pittsburgh and then passed through to AIR. 

 

Sampling and Development

 

Because the timing of Flanagan’s “talent census” was to coincide with the 1960 Census (New York Times,1959), the team only had 1 year from obtaining funding to launch the project. Work immediately began on forming the sample of respondents and creating the test battery and questionnaires. Flanagan’s team set about inviting a random sample of high schools (including public, private, and religious) to participate in the study. The response rate was overwhelmingly positive: 93% of the invited high schools agreed to participate in the study. 

 

Nearly all of the measures for Project TALENT were custom developed by Flanagan’s team of psychologists (Shaycoft, 1977). These measures can be roughly grouped into seven categories. First, all students took a low-difficulty screening test that was designed to identify students who had cognitive disabilities or were not completing the measures in a diligent fashion. Next, students completed a booklet containing general information questions on 36 topics. These questions can be viewed as measuring crystallized intelligence or achievement. They were developed to assess knowledge that students would not necessarily learn in school and included topics such as farm work and home economics. Afterwards, students completed language aptitude and ability tests covering memory, grammar, punctuation, and reading. Students also completed tests focusing on additional cognitive abilities such as math, perceptual speed and accuracy, reasoning, creativity, and spatial visualization. 

 

In general, these cognitive tests were inspired by the Aircrew Classification Test battery that Flanagan’s team developed during World War II (Flanagan et al., 1962). The data obtained from the cognitive tests were robust. A recent confirmatory factor analysis indicated that the cognitive tests measure five broad factors (Information, English/Math, Spatial/Reasoning, Mechanical/Science, Speed, and Math) with adequate fit (CFIs were in the. 94–. 96 range; Major, Johnson, & Deary, 2012). After completing the cognitive tests, students completed a non-cognitive student information blank. This 394-item questionnaire asked students about their families, experiences, health, hobbies, and dreams. A Student Activities Inventory covered personality, which was later shown to have excellent reliability and validity (Pozzebon et al., 2013), leadership, and social skills, and an interest inventory asked if students were interested in various occupations and hobbies.  

 

Administration and the Big Dataset

 

After the measures were developed and the schools were selected, it was time to administer the tests. This is where the size of the project becomes staggering. The measures were administered to approximately 450,000 students[4]at 1,353 high schools (roughly 5% of the high schools in the United States) over a 4-day period[5]in classrooms such as the one shown in Figure 2 (Flanagan, 1961; Flanagan, Caludy, Richards, Shaycoft, & Davis, 1970)[6]. The paper-and-pencil materials for the administration consumed “18 freight carloads” (Shaycoft, 1977, pp. 5–13). 

 

After testing, the team was under enormous pressure to quickly process part of the data so that individuals score reports could be mailed back to the schools before the 1960 senior class graduated (AIR, 2011). The nearly 2 million answer sheets first had to be manually inspected and corrected for errors (e. g., incomplete erasures, stray marks) and other issues. A group of part-time clerks were hired to do this work; they were selected using a subset of the Project TALENT tests and were informally known as the “TALENT Laundry” (Shaycoft, 1977, pp. 7-26). In order to process the laundered answer sheets, a team of computer scientists at the University of Iowa created a new computer for scanning them (shown in Figure 3; AIR, 2011). The new computer converted over 2 million individual answer sheets into 5 million punch cards (the more than 2,000 pieces of information in each student’s record took up 11 cards) that were shipped to the University of Pittsburgh (Flanagan et al., 1964; The 7070 computer, 1961). There another new computer, the IBM 7070, converted the one billion bits of data on the punch cards to nine-track magnetic tape that was over seven miles long (Talent Census, 1962; The 7070 computer; Achorn et al., 2013). The data then moved to a third computer, the IBM 7090 (see Figure 4) on which the data analysis was conducted (Shaycoft). The dataset was so large that many researchers were forced to use random samples of it for their analyses. Although the score reports were issued on time, it took years to finish merging the data for research purposes and to make tape records for each student. 

 

Data collection for the study did not cease after 1960. A one-year follow-up survey was administered to the graduating class of 1960 in 1961. Because students in all four grades of high school were tested in 1960, this effort continued for 3 additional years. Five and 11-year follow-up surveys were later administered in the 1960s and 1970s. Unfortunately, over time it became difficult to contact the respondents after the 1970s, and additional data collection ceased until AIR conducted a feasibility study in 2009, which revealed that recent advances in technology have made it easier to track participants (Achorn et al., 2013). Consequently, work on a follow-up study is now underway (AIR, 2011). 

 

Findings

 

Flanagan was an early pioneer of publicly sharing data (Simonsohn, 2012). He made the dataset (American Institutes for Research, 1960a, 1960b) available to researchers on magnetic tapes and to date over 356 publications have been generated from it. In fact, researchers are continuing to use the dataset and AIR staff have worked recently on several grants related to Project TALENT. Research on the dataset led to several interesting findings, some of which were published in Flanagan et al. ’s (1964) report, TheAmerican High School Student. Many of the key findings were shared with the general public and the media was quite supportive of the study. Flanagan’s work was the focus of articles in the New York Times(Census of 1960,1959) and Time Magazine(Talent Census, 1962), which hailed him as one of the “nation’s top testmakers” (p. 56). In addition, Vice President Lyndon B. Johnson commended the work of the Project TALENT team (see Figure 5). 

 

One key finding was that for tests of achievement/information, the top 25% of 9th-graders outperformed the average 12^th-grade student (Flanagan, 1979). In addition, only 1% of students could produce a 5-minute themed written piece without writing errors and the average 12^th-grade student could only comprehend 35% of what he or she read in Time Magazine. Corazzini, Dugan, and Grabowski (1972) discovered that family income, not ability, was the best predictor of enrollment in college, especially for those students of average ability. Follow-up studies revealed participants’ career choices often conflicted with their interests and abilities. Although participants were most satisfied when they chose a career that matched their interests and abilities, few high school graduates had plans to do so. Recently, Wai, Lubinski, and Benboe (2009) reported that of those Project TALENT participants with a PhD in a science, 45% were in the top stanine on spatial ability. However, current talent searches for STEM positions miss many of these individuals as they focus on math and verbal skills rather than spatial skills. 

 

Another of TALENT’s key findings was that students were lacking in individualized support for their education. Specifically, there was a need for greater alignment between the instruction that students receive in school and their career goals (Flanagan, 1967). Flanagan was especially struck by this finding and it led to him to pursue a new project: Project PLAN. 

 

Project PLAN

 

Flanagan developed a computer-assisted instructional program for educating students in grades 1 through 12. This effort was known as Project PLAN—Program for Learning in Accordance with Needs. AIR launched Project PLAN in 1967, with funding from the Westinghouse Learning Corporation. The project used placement tests to frequently assess each student’s needs within multiple academic areas (Flanagan, 1968; Freeman, 1996). This information was used as the basis to continuously modify the student’s academic program, thus creating an education program tailored to each student. In many ways this project was ahead of its time as the modifications to each student’s academic program were managed by a central computer system. In addition, a lack of available computer technological and logistical issues prevented the project from making the huge impact that it could possibly have made today. 

 

Critical Incidents

 

Perhaps Flanagan’s most lasting impact in I-O psychology was his creation of the critical incidents methodology. In this approach, a job analyst collects descriptions of employees’ on-the-job behaviors. The descriptions include a statement of the antecedents of the situation, the behavior itself, and the consequences of the behavior. Typically, the critical incidents represent examples of low or high performance. Flanagan and his colleagues spent nearly a decade developing and refining this methodology, culminating in his 1954 Psychological Bulletinarticle, which was labeled as one of “the most important personnel selection milestones of the past 60 years” in the Annual Review of Psychology(Dunnette & Borman, 1979). 

 

Work on critical incidents began in an attempt to understand why, in less than a year, nearly 1,000 pilots had failed pilot training in the U. S. Army Air Corps during World War II. When a pilot was in the process of being removed from training, his case went before an elimination board. One of Flanagan’s staff members reviewed the written proceedings from this board to identify specific behaviors that were related to failures. The second critical incident study examined “Group Mission” reports of bombing runs and identified possible reasons for bombing failures. Both studies led to changes in the selection system for the U. S. Army Air Forces. Over time, the critical incident approach was used experimentally for leadership positions, spatial disorientation while flying, flying in general, commercial airline pilots, laboratory scientists, air traffic controllers, plant workers at General Motors, and dentists. In his 1954 article, Flanagan presented a sample critical incident collection protocol, which looks much like those used today. It asked “What were the general circumstances leading up to the incident,” “What [did] this person [do] that was so helpful at the time,” and “Why was this so helpful in getting your group’s job done?” (Flanagan, 1954, p. 342). It also made inquiries about tenure, age, and time at which the incident occurred; these items are less commonly collected today. 

 

Flanagan later expanded the critical incidents methodology to use in employee performance appraisals at General Motors (Flanagan & Burns, 1955). Foremen were asked to record critical incidents concerning their employees on a daily basis and the incidents were used as the basis for the employee’s performance appraisal. Focusing on factual events (i. e., critical incidents) as opposed to general impressions (e. g., ratings on a multipoint rating scale) led to better acceptance of the performance appraisal. Asking foremen to record the critical incidents on a daily basis led to a larger quantity of incidents, as corroborated by a field experiment. 

 

Conclusion

 

John Flanagan is perhaps best known among SIOP members for his significant contributions to I-O psychology (e. g., creating the critical incidents methodology, pioneering the development of aviation selection systems, founding AIR). However, his contributions outside of I-O psychology—particularly his work with Project TALENT—were equally immense. In many ways, his ideas were well ahead of his time. He encouraged data sharing and pioneered the use of computer-assisted learning in the 1960s. He adapted applied psychology in new nonwork settings and he helped to “grow-I-O” (Allen, Behrend, Bell, & Smoak, 2014) by founding large teams of I-O psychologists in the Air Force and AIR. We hope that Flanagan’s story will encourage future I-O psychologists to apply their ideas in diverse settings, to be creative in the application of technology, and to expand our field. 

 

Notes

 

[1]At this time, what is now the U. S. Air Force was still technically under control of the U. S. Army. From 1926 until 1941 it was known as the U. S. Army Air Corps and from 1941 until 1947 it was known as the U. S. Army Air Forces. 

2Although the word TALENT appears in all caps, it does not stand for anything; desiring to have a name for the project that would easily roll off the tongues of the general public, the project team came up with the phrase “Project TALENT” (Shaycoft, 1977). Indeed, in the acronym list on the last page of her historical account of Project TALENT, Shaycoft states that TALENT is “Not an acronym!” (p. 274). 

3According to Shaycoft (1977), Flanagan was partly inspired by Herbert Toops, who (25 years before) suggested creating norms on a nationwide sample of 1 million individuals. Flanagan “settled” for half of that figure. 

4Including rock stars Janis Joplin and Jim Morrison. 

5The testing lasted the equivalent of 2 full schooldays; however, most schools decided to spread out the testing over 4 days (Susan Lapham, personal communication, November 24, 2015). 

6The sample size seems quite large, given what is known about power analysis. The research team decided to generate such a large sample so that, after the follow-up studies were conducted, they would yield enough students who entered specialty occupations such as mathematician, nuclear physicist, and so forth. In addition, they anticipated conducting multiple breakout analyses (Flanagan et al., 1962).