Survey Design in Social Science Research: Practical Tips
Survey design is an essential skill for social science researchers who want to collect reliable and valid data from their target populations. However, designing a good survey is not as easy as it seems. There are many factors to consider, such as the goals of the survey, the measures to use, the platform to choose, the ethical issues to address, and the data quality to ensure. In this blog post, I will share some practical tips on how to design a survey in social science research based on my own experience.
Here are some useful references if you are looking for more academic resources!
- Stantcheva, S. (2022). How to run surveys: A guide to creating your own identifying variation and revealing the invisible. Annual Review of Economics, 15.
- Gideon, L. (Ed.). (2012). Handbook of survey methodology for the social sciences. New York: Springer.
- Chiang, I.-C. A., Jhangiani, R. S., & Price, P. C. (2015). Constructing survey questionnaires. In Research Methods in Psychology – 2nd Canadian Edition. BCcampus.
Specify your goals
The first step in survey design is to specify your goals. What are the main research questions you want to answer with your survey? What are the hypotheses you want to test? What are the variables you want to measure? These questions will guide you in choosing the appropriate measures and sampling methods for your survey. You should also consider how you will analyze your data and what kind of results you expect to find.
Read other examples from ICPSR
https://www.icpsr.umich.edu/web/pages/
One way to learn how to design a survey in social science research is to read other questionnaires from ICPSR. ICPSR is a data archive that provides access to thousands of studies and datasets from various disciplines and topics, such as politics, economics, health, education, and more. You can browse or search for studies that are relevant to your research interests and see how other researchers have designed their surveys, what measures they have used, how they have collected and analyzed their data, and what results they have found. You can also download the data and documentation for your own use or replication.
To find survey examples from ICPSR, you can visit their website and use the search box or the browse options to look for studies that match your keywords or criteria. You can also filter the results by subject, geography, time period, series, principal investigator, or data format.
Find Measures
The next step is to find measures that can capture your variables of interest.
A. You can start by reading the literature on your topic and going directly to the Measures section of relevant papers. You can collect the measures that have been used by previous researchers in a spreadsheet, along with their sources, references, and scoring methods. For example in the following picture, you can see the name of the scale and the response options. You might need to go to the reference to find the full measure.
B. You can also use the APA PsycTests archive to search for psychological tests and measures that have been published or reviewed by experts. For example, you can go to APA PsycNet (you might need to go to this website via your institutional access) and then search terms with “Test Available” button clicked. Then, you can see the right screen, with the “Test” pdf file that you can collect.
Here is the sample spreadsheet, with the construct, scale name, the number of items, priority yes/no, collect yes/no, reference, and notes. The collected column means whether I have collected the statements for that scale (e.g., the pdf file from APA Psycnet).
After you have collected some potential measures on the spreadsheet, you can discuss with others (collaborators, coauthors, or your mentors) to finalize the items that best suit your goals. You should also check the validity and reliability of the measures, as well as their suitability for your target population and platform. For example, some measures may require translation if you want to survey people from different linguistic backgrounds.
Design your survey
Once you have decided on the measures, you can start designing your survey. You can create your own document with the full list of items in the order of the survey. You can divide the section as well and then discuss by section with your collaborators. Here is an example from the design document, including the scoring procedure and reference. You can just copy and paste them to the platform when you finalize.
Here are my few tips.
- You need to specify the time and budget for your survey, as well as the target population and sampling method.
- You need to choose a platform that can host your survey and provide you with the features and functions you need. Some popular platforms are [Qualtrics] and [SurveyMonkey], which offer various options for creating and distributing surveys online. Lots of schools subscribe to Qualtrics as an institution, so you can check that out.
- For multilingual surveys, both Qualtrics and SurveyMonkey support multilingual surveys. You might need to hire professional translators or think about the scale validation in multiple languages.
- One thing to keep in mind when designing your survey is to add some attention-check items throughout your survey. These are items that can help you identify respondents who are not paying attention or answering randomly, such as asking them to select a specific option or answer a simple question. Please find the post from Qualtrics on the best practice of attention check.
- You can also consider adding some social desirability items if you are surveying some sensitive topics, such as stereotypes, health, or mental health. These are items that can measure how much respondents are influenced by social norms or expectations when answering your questions.
- Another thing to consider is the ethical aspect of your survey. If you are surveying vulnerable populations or sensitive topics, you should make sure that you have obtained informed consent from your respondents, that you have protected their privacy and confidentiality, and that you have minimized any potential harm or discomfort they may experience. You should also follow the ethical guidelines and regulations of your institution or organization when conducting your survey.
- Finally, you should think about the comparability of your survey with other surveys. If possible, you should use the same measures that have been used by other researchers in similar contexts or populations. This will allow you to compare your results with theirs and increase the generalizability of your findings.
Collect the data
After you have designed your survey, you can start collecting your data on your preferred platform (e.g., Qualtrics). There are a few things to note here.
- Before you distribute your survey to your target population, you should do a pilot test with a small sample of respondents who are similar to your target population. This will help you estimate the time it takes to complete your survey, check if the skip logic works well, and identify any errors with that.
- You can distribute your survey through various channels, such as email, social media, online communities, or websites. However, you should be careful about the quality of your respondents, especially if you use online platforms that are open to anyone. There are many bots or fake respondents who may try to access your survey and provide invalid or unreliable data. You should use some methods to prevent or detect them, such as using captcha codes, IP address filters, or email verification.
Cleaning data
After you have collected enough responses for your survey, you should check the quality of your data before analyzing it. My tips are here:
- You should clean your data by removing any incomplete or duplicate responses, any responses that failed the attention check items.
- You can also check the internal consistency of the well-known scales that you used.
- You can also read the answers to the open-ended questions, if any. There are some people who just write random things, which makes their case unreliable.
You can refer to this post on getting validity and reliability scores in STATA using validscale command:
Creating the codebook
It would be great if you also created a codebook that describes your data and the variables that you measured. A codebook is a document that provides information about the source, structure, content, and coding of your data. It can help you, and others understand your data better and use it for further analysis or research.
Here is an example of a codebook. You can put the demographic characteristics on the first page and then put the information for every single measure, including the descriptive statistics and internal consistency (Cronbach’s alpha).
I hope this blog post has given you some useful tips on designing a survey in social science research. Thank you for reading!
