The Cream of the Crop: Student and Alumni Perceptions of I-O Psychology Master's Degree Program Quality

Yalcin Acikgoz, Timothy J. Huelsman, Jessica L. Swets, Amanda R. Dixon, Stephanie N. Jeffer, D. Ryan Olsen, and Amanda Ross Appalachian State University

Author Note: First three authors are listed based on contribution. Remaining authors are listed based on alphabetical order. Correspondence concerning this paper should be addressed to Yalcin Acikgoz, Department of Psychology, Appalachian State University, Box 32109, 222 Joyce Lawrence Ln., Boone, NC (28608). Email: acikgozy@appstate.edu. Phone: +1-828-262-8926

There is little dispute that reputation is a factor in a prospective student’s decision to apply to a specific graduate program in industrial-organizational (I-O) psychology. Consequently, there have been many attempts at ranking graduate programs. However, the vast majority of these have examined doctoral-level programs and have used research productivity as the main criteria (e.g., Beiler, Zimmerman, Doerr, & Clark, 2014; Gibby, Reeve, Grauer, Mohr, & Zickar, 2002; Oliver, Blair, Gorman, & Woehr, 2005; Payne, Succa, Maxey, & Bolton, 2001; Winter, Healy, & Svyantek, 1995). Especially for master’s level graduate programs, subjective program experience may be an important consideration for prospective students when evaluating the desirability of a graduate program. An example of this kind of program evaluation is Kraiger and Abalos (2004), but it has been more than a decade since the publication of their results, and an update is long overdue. This project attempts to remedy past neglect and provides rankings of master’s level I-O graduate programs using subjective ratings provided by students and alumni (a companion report in this issue, Vodanovich, Morganson, & Kass, provides a ranking of master’s degree programs based on objective criteria).

Our project was prompted by a call for proposal by SIOP (Salter, Allen, Gabriel, Sowinski, & Naidoo, 2016). A major goal of this effort was to identify more comprehensive ways of ranking programs using the criteria that would be important to prospective students. We took this as a call to further develop subjective criteria to rank master’s level programs and decided on three main indicators of program quality that would be evaluated by each program’s current students. First, there has been a wide range of research supporting the importance of culture in educational success (Pritchard, Morrow, & Marshall, 2005; Tichnor-Wagner, Harrison, & Cohen-Vogel, 2016) as well as business success (Sirota, Mischkind, & Meltzer, 2008; Xenikou & Simosi, 2006). Similarly, we believe having a supportive and warm culture in graduate school is important in creating a superior professional development experience. Accordingly, the first subjective indicator is program culture. Second, we measured program resources via students’ perceptions of resources such as program faculty, breadth of program curriculum, funding opportunities, and availability of academic and professional mentoring. This offers a more in-depth examination of resources beyond typical research and faculty availability. Finally, the third indicator of program quality examined in this study is student satisfaction. We believe that a good program results in a high level of overall satisfaction with the experience it offers.

In addition to expanding the scope of criteria to include subjective evaluations, we believe that another way to ensure comprehensiveness of graduate program rankings is by making sure numerous stakeholders provide input in the process. Although current students offer an important viewpoint, they cannot provide feedback for all relevant graduate program characteristics. An additional group of key stakeholders are program alumni, as these individuals are in a better position to evaluate how their programs prepared them for the “real world.” Accordingly, we included alumni perceptions of the program as a separate dimension of graduate program rankings. However, in order to reach alumni who had a good memory of their programs and following job search experiences, we limited our alumni sample to those who graduated in the last 5 years. This also allowed us to collect data from alumni who have been in the program relatively recently and thus are likely knowledgeable about the current state of the program (i.e., their perceptions would better reflect the outcome of the program in its current form). Finally, two separate surveys were created, one for each group of informants. The first survey, for current students in I-O master’s degree programs, included items assessing program culture, program resources, and satisfaction. The second survey, for recent program alumni, included items assessing whether the program was successful in preparing them for employment. 

Method

 Procedure

After reviewing the existing literature on organizational culture, performance, and satisfaction measures, the research team developed surveys targeted at I-O programs’ current students and recent alumni. For each dimension, the team created a list of items based on other measures when possible (e.g., Bierer, Fishleder, Dannefer, Farrow, & Hull, 2004; Kraiger & Abalos, 2004; O’Reilly, Chatman, & Caldwell, 1991). Survey items used in the current study are in Appendix A. Email addresses for 117 master’s degree program directors were obtained from SIOP, and each program director was sent an email containing information about the project and a request to forward the appropriate survey link to current students and recent alumni (i.e., former students who graduated within the last 5 academic years, as of spring 2017).

Student Survey              

Program culture.

Student perceptions of program culture was measured by 14 items written for this study based on organizational culture profile items described in O’Reilly et al. (1991). Participants were asked to indicate their level of agreement regarding program cultural elements on a five-point scale ranging from 1 = strongly disagree to 5 = strongly agree. Sample items include: “I am able to achieve a balance between my work in the program and life outside the program” and “This program is accepting of people from various backgrounds and perspectives.” Cronbach’s alpha for internal consistency is .92 in the current sample.

Program resources.

Student perceptions of program resources were measured by 13 items adapted from Bierer et al. (2004). Participants were asked to indicate the extent to which their program offers access to a variety of resources on a five-point scale ranging from 1 = strongly disagree to 5 = strongly agree. After the prompt “My program offers me access to…,” the survey referenced a variety of resources including career development services, statistical software, and program alumni. Cronbach’s alpha for internal consistency is .88 in the current sample. 

Program satisfaction.

Students’ satisfaction with their programs was measured by 18 items developed for this study. Participants were asked to indicate the extent to which they are satisfied with several aspects of the program on a five-point scale ranging from 1 = very dissatisfied to 5 = very satisfied. The survey noted program characteristics such as class sizes, course offerings, and internship opportunities. Cronbach’s alpha for internal consistency is .93 in the current sample.

Alumni Survey

Alumni perceptions of the program were measured using nine items developed for this study. Participants were asked to indicate the extent to which they agree with statements on a five-point scale ranging from 1 = strongly disagree to 5 = strongly agree. Sample items include “I feel the program has prepared me well for my career” and “I would encourage others to apply to this program.” Cronbach’s alpha for internal consistency is .87 in the current sample.

Results

As described above, the goal of this study was to produce rankings of I-O master’s degree programs by using subjective evaluations of students and alumni. A total of 975 participants provided the data for the study: 594 current students from 44 programs and 381 alumni from 36 programs. However, some programs had very little data on which to base the rankings. For these programs, one or two individuals with very high or very low evaluations could potentially skew their mean score and unduly influence their program ranking. To avoid this possibility, we excluded programs from the rankings if they had a sample size less than one SD below the mean for each informant group (Kraiger & Abalos, 2004). The mean sample size across programs was 13 for students (SD = 8.4) and 11 for alumni (SD = 8). Thus, programs were included in the rankings if they had at least five student respondents and at least three alumni respondents, which led to the elimination of six programs from the student survey and 12 programs from the alumni survey.

The rankings for student evaluations of program culture, program resources, and satisfaction with the program appear in Tables 1-3, respectively. We only included the 25 highest-ranked programs in the results (see Appendix B for a complete, nonranked list of all programs from which we received any responses). As seen in Table 1, the top three schools in program culture are Indiana University-Purdue University Indianapolis (M = 4.83, SD = .15), Appalachian State University (M = 4.77, SD = .19), and Xavier University (M = 4.75, SD = .26). As see in Table 2, the top three schools in program resources are Appalachian State University (M = 4.63, SD = .32), Minnesota State University–Mankato (M = 4.55, SD = .54), and Indiana University-Purdue University Indianapolis (M = 4.54, SD = .27). As seen in Table 3, the top three schools in student satisfaction are Appalachian State University (M = 4.69, SD = .28), Missouri State University (M = 4.66, SD = .27), and San Diego State University (M = 4.60, SD = .37).

Next, we examined evaluations by program alumni. As seen in Table 4, the top-three programs in this ranking were The University of Georgia (M = 4.65, SD = .25), University of Maryland–Baltimore County (M = 4.63, SD = .17), and Minnesota State University-Mankato (M = 4.61, SD = .44). Finally, we examined the extent to which ratings by students on different dimensions and alumni ratings overlapped. This was done by creating a separate dataset of program means on each dimension (with programs being the level of analysis) and examining the correlation between means of ratings in separate dimensions. As seen in Table 5, all the correlations were strong and statistically significant, especially among student-rated dimensions. The most noteworthy correlations were correlations between alumni ratings and different dimensions of student evaluations. The correlations between alumni evaluations and student evaluations of program culture, resources, and satisfaction were .49 (p < .01), .39 (p < .05), and .45 (p < .01), respectively.

Table 1

Program Rankings by Student Perceptions of Program Culture

Table 2

Program Rankings by Student Perceptions of Program Resources

Table 3

Program Rankings by Student Satisfaction

Table 4

Program Rankings by Alumni Perceptions

Table 5

Means, Standard Deviations, and Correlations of Study Variables

* p < .05, ** p < .01., *** p < .001

Discussion

The present report diverges from previous ranking attempts by incorporating subjective criteria such as culture, satisfaction, and program resources rather than relying only on research productivity outcomes. By doing so, the current study accomplishes two important objectives. First, it addresses issues that

have a direct relevance to how students choose a master’s program but have received little attention in the extant literature. For example, assessing students’ perceptions of their program resources may provide an important context for whether prospective students believe they will be successful in a certain program. Second, it provides a more comprehensive examination of the overall effectiveness of master’s programs by examining how programs prepare their students for their future careers based on subjective evaluations of program alumni. As described above, subjective evaluations of current students and alumni may be more important to prospective students compared to objective outcomes such as research productivity. Therefore, we believe these rankings are likely to be of significant value to prospective I-O graduate students who are engaged in the important decision of where to apply for graduate school.

It is important to note that the top three schools as ranked by their current students do not appear in the top three alumni ranked programs. This demonstrates the importance of evaluating program effectiveness from the perspectives of multiple stakeholders. Current students may have differing opinions about their master’s programs while they are in attendance as compared to after starting their careers. It is likely that satisfaction with their program after graduation may be influenced by factors other than those that are important for student experience. Factors such as the ability to find a job, how quickly they are hired, how well the program prepared them for work, the kinds of jobs they were able to find, or getting into a PhD program after graduating from a master’s program may influence alumni perceptions of their programs. Future ranking efforts should continue including this important perspective in evaluations of graduate programs, perhaps even considering the kinds of jobs alumni take.

There were some limitations to this study, which largely focus around the availability of respondents and the use of subjective measures. Some programs have not been established long enough to produce a sufficient number of alumni to provide rankings. Some other programs were smaller in size and therefore had fewer current students to provide rankings. Accordingly, program age and size are important limitations that should be acknowledged. Another limitation is that due to the subjective nature of the evaluations sought, some program directors may have refused to provide data. Kraiger and Abalos (2004) mention concerns by program directors that some criteria would put them at an inherent disadvantage (e.g., program cost for schools in metropolitan areas) as one reason for some programs not being represented in the evaluations. Even though we did not receive any feedback from program directors with such concerns, this may be one reason for some programs not providing data to our evaluations. Regardless of reason, of 117 programs to which our efforts were directed, we received data from only 44 programs. However, despite these limitations, we believe the rankings reported here provide valuable information for potential graduate students and others interested in I-O master’s degree program quality. We recommend that these rankings be used in conjunction with others, including those developed in response to SIOP’s recent Call for Proposals (Salter et al., 2016). Together, these rankings provide a very comprehensive base for evaluating program success.

References 

Beiler, A. A., Zimmerman, L. M., Doerr, A. J., & Clark, M. A. (2014). An evaluation of research productivity among I-O psychology doctoral programs. The Industrial-Organizational Psychologist, 51 (3), 40-52.

Bierer, S. B., Fishleder, A. J., Dannefer, E., Farrow, N., & Hull, A. L. (2004). Psychometric properties of an instrument designed to measure the educational quality of graduate training programs. Evaluation & the Health Professions, 27, 410-424.

Gibby, R. E., Reeve, C. L., Grauer, E., Mohr, D., & Zickar, M. J. (2002). The top I-O psychology doctoral programs of North America. The Industrial-Organizational Psychologist, 39 (4), 17-25.

Kraiger, K., & Abalos, A. (2004). Rankings of graduate programs in I-O psychology based on student ratings of quality. The Industrial-Organizational Psychologist, 42 (1), 28-43.

Oliver, J., Blair, C. A., Gorman, A., & Woehr, D. J. (2005). Research productivity of I-O psychology doctoral programs in North America. The Industrial-Organizational Psychologist, 43 (1), 55–63.

O'Reilly, C. A., Chatman, J., & Caldwell, D. F. (1991). People and organizational culture: A profile comparison approach to assessing person-organization fit. Academy of Management Journal, 34, 487-516.

Payne, S. C., Succa, C. A., Maxey, T. D., & Bolton, K. R. (2001). Institutional representation in the SIOP conference program: 1986-2000. The Industrial-Organizational Psychologist, 9 (1), 53-60.

Pritchard, R. J., Morrow, D., & Marshall, J. C. (2005). School and district culture as reflected in student voices and student achievement. School Effectiveness and School Improvement, 16, 153-177.

Salter, N. P., Allen, J. A., Gabriel, A. S., Sowinski, D., & Naidoo, L. (2016). Call for proposals for I-O graduate program rankings. The Industrial-Organizational Psychologist, 54 (1).

Sirota, D., Mischkind, L. A., & Meltzer, M. I. (2008). Enthusiastic employees. In R. J. Burke, C. L. Cooper, R. J. Burke, C. L. Cooper (Eds.), Building more effective organizations: HR Management and performance in practice (pp. 35-56). New York, NY: Cambridge University Press.

Tichnor-Wagner, A., Harrison, C., & Cohen-Vogel, L. (2016). Cultures of learning in effective high schools. Educational Administration Quarterly, 52, 602-642.

Winter, J. L., Healy, M. C., & Svyantek, D. J. (1995). North America’s top I-O psychology/doctoral programs: U.S. News and World Report revisited. The Industrial-Organizational Psychologist, 33 (1), 54-58.

Xenikou, A., & Simosi, M. (2006). Organizational culture and transformational leadership as predictors of business unit performance. Journal of Managerial Psychology, 21, 566-579.

APPENDIX A 

Student Survey-Program Culture

Please indicate the extent to which you agree or disagree with the following statements (1 = strongly disagree, 5 = strongly agree)

• I am able to achieve a balance between my work in the program and life outside the program
• Students in the program are supportive of each other
• I have meaningful relationships with program faculty
• The faculty in my program care about me as a person
• Faculty are engaged in the program and its students
• Faculty in my program are motivated to provide the best environment for students’ professional development
• I have been treated fairly by the faculty in my program
• I am proud to be a student in this program
• Faculty take graduate student ideas seriously
• Students are invested in the success of other students
• Faculty have reasonable expectations of students
• I am given timely and constructive feedback
• There is unhealthy competition within the program
• This program is accepting of people of various backgrounds and perspectives

Student Sample-Program Resources

My program offers me access to… (1 = strongly disagree, 5 = strongly agree)

• Career development services
• On-campus study spaces
• Dedicated spaces for graduate students
• Counseling services
• Statistical software
• Adequate library resources
• Mentoring
• Conferences
• Certifications and training (outside of classes)
• Funded assistantship
• Scholarships
• Funding for professional development activities (conferences, training, etc.)
• Program alumni

Student Survey-Satisfaction With the Program

Please indicate the extent to which you are satisfied with the following aspects of your program. (1 = very dissatisfied, 5 = very satisfied)

• Faculty support and accessibility
• Quality of instruction
• Balance between applied and academic emphases
• Quality of research in the program
• Connection with I-O, HR, and related communities
• Variety of course offerings
• Class size
• Culture of the program
• Availability of educational resources
• Internship and other professional opportunities
• Alumni engagement
• Engagement with the program during application process
• Student diversity
• Faculty diversity
• Student relationships
• Financial support
• Website and social media presence
• How well the program is preparing you for your career

Alumni Survey

Please indicate the extent to which you agree or disagree with the following statements. (1 = strongly disagree, 5 = strongly agree)

• I like to stay updated about current events in the program
• I like to participate in available alumni events/opportunities
• I would like to donate money to the program
• The program is keeping me updated about current events/developments
• I have been provided with the necessary skills to succeed in my current career
• I like to keep in touch with faculty
• I feel the program has prepared me well for my career
• I feel the program has helped me develop my soft skills (e.g. leadership, communication, public speaking)
• I would encourage others to apply to this program
• I am proud to be an alumnus of this program

APPENDIX B

Listing of All Programs Submitting at Least One Response