Best Practices for Demographic Survey Items

Alexandra I. Zelin & Sertrice Grice

In October 2019, SIOP published its first “Better Practices in Surveying Demographic Information” report and proposed question set, chaired by Lindsay Dhanani and coauthored with Enrica Ruggs and Jenna Eagleson. They were tasked with developing an inclusive demographic question set that can be used by the community and explained why the proposed questions and response choices were presented and phrased a certain way. As they stated in their report

Collecting demographic information is a ubiquitous practice among scientists and practitioners alike. Surveyors may collect this information from participants to ensure their sample is representative of the population or because they are interested in how psychological phenomena vary as a function of demographic characteristics. Practitioners working within organizations may similarly be interested in gathering this information from employees to monitor the demographic composition of their applicants and/or workforce, to accurately prepare Equal Employment Opportunity reports, or to internally assess the effectiveness of diversity efforts. Regardless of the purpose, it is important to carefully consider the way in which demographic information is collected from participants for several reasons. First, the way demographic data is collected can influence the accuracy of participant responses. For example, describing identities in ways that do not comport with how individuals self-identify might cause participants to choose an option deemed incorrect by the surveyor or to opt out of responding to the question altogether (Bauer et al., 2017; Moody et al., 2013). This can obscure the true demographic composition of the sample or lead to inaccurate conclusions about the effects of demographic group membership.

In addition to reduced accuracy, demographic items can also harm participants if they are not inclusive or use terminology that is unintentionally derogatory. Identities and the way individuals represent them can be complex and survey items often ignore these intricacies by adopting simplistic demographic questions. As an example, many surveys often use binary gender items that only present respondents with the options of “male” or “female.” This item would not recognize the many individuals who identify outside of the gender binary, and these participants may feel as though their gender identity is being delegitimized when it is not included as a response option. Having to select a response option that does not accurately reflect one’s self-concept or that uses offensive language can also create frustration for participants (Hughes et al., 2016). For these reasons, we believe it is paramount to create demographic surveys that accurately capture the ways individuals personally represent their identities, are inclusive of the full range of identities individuals may hold and are free of potentially offensive terminology. The purpose of this report is to provide specific recommendations for the best practices in measuring and assessing demographic information as well as to highlight limitations with some commonly used demographic items.

However, demographic verbiage is in consistent flux, and after discussions among the authors, members of the Diversity & Inclusion portfolio, and feedback from the SIOP Climate Survey, we determined an update to the “Better Practices in Surveying Demographic Information” was needed. Chaired by Alexandra Zelin and in conjunction with Lindsay, Enrica, Jenna, Brenton Wiernik, Theresa Hernandez, Sabrina Volpone, Isaac Sabat, Susan D’Mello, and Tiffany Poeppelman, and reviewed by Sertrice Grice of the Institutional Research Committee (IRC), a 2021 update was published. There are two associated documents: The first includes the list of questions to include if asking demographic variables within a survey. The second focuses on the “why” and provides background information on the current state of demographic best practices.

Historically, the IRC would provide both documents to researchers looking to survey SIOP membership. After this update, the Membership Committee and the IRC discussed the value of requiring researchers to use the recommended demographics as a standard across any survey that wants to include demographic items. With the items standardized it would provide consistency, allowing for easier cross-project analyses. However, and most importantly, it would demonstrate recognition of and support for all SIOP members. The motion was brought to the Executive Board in the September 2021 meeting, and the motion passed. As a result, the recommended demographics were added as an appendix to the IRC guidelines, along with the following note that can be found under Section 2 in the guidelines:

Note: SIOP has provided a pre-approved list of demographic items that can be included in surveys (see Appendix E). Any additional demographic items or changes to these items will require an explanation and are subject to IRC approval.

Overall, we anticipate that these recommendations for demographic best practices can serve as guidance for researchers and practitioners alike, both for SIOP- and non-SIOP-related surveys, data management, and general inclusive practices. As a note, this is a living document that will continue to be updated as changes happen in this space. We encourage you to check the website often to ensure you are using the most up-to-date version.

Resources:

• Recommended Demographic Survey Items
• Better Survey Practices Technical Report
• SIOP Surveys Webpage

References

Bauer, G. R., Braimoh, J., Scheim, A. I., & Dharma, C. (2017). Transgender-inclusive measures of sex/gender for population surveys: Mixed-methods evaluation and recommendations. PLoS ONE, 12, e0178043. https://doi.org/10.1371/journal.pone.0178043

Hughes, J. L., Camden, A. A., & Yangchen, T. (2016). Rethinking and updating demographic questions: Guidance to improve descriptions of research samples. Psi Chi Journal of Psychological Research, 21, 138–151.

Moody, C., Obear, K., Gasser, H., Cheah, S., & Fechter, T. (2013, December 5). ACPA standards for demographic questions. Retrieved from https://archive.myacpa.org/docs/proposal-demographic-questions-and-responses-2pdf