Writing Good Survey Questions - Part 1. Base Questions

In this episode, we delve into the art of crafting effective survey questions. We discuss the importance of clear and precise wording, avoiding biases, and the impact of question formats on data quality. Learn practical tips for designing questions that yield accurate and meaningful data, whether for academic research, market analysis, or feedback collection. Join us as we explore examples and demonstrate how to refine a simple question into a well-crafted survey item. Perfect for anyone looking to improve their survey design skills!

Find us online at: instagram.com/totalsurveydesign/
https://taplink.cc/totalsurveydesign
Contact us at: totalsurveydesign@gmail.com

Transcript

SYL: In this episode, we will discuss writing good survey questions. 

SYL: Writing a good survey question means measuring the thing that we want to measure as accurately and truthfully as possible. Whether you're conducting academic research, market research, or just gathering feedback, the quality of your survey questions can make or break your data. Gathering good data means that it will be more likely that all of the work that is being done to create the survey will result in good data.

AC: Before we get into writing good survey questions, I want to spend a minute talking about operationalizing abstract concepts. 

Sociologists and other social scientists often need to measure things that cannot be seen. Concepts like happiness, or cultural capital, or power, or alienation, or identity, or the sense of belonging, are all things that are important to measure and study, but that are not easy to measure.

Operationalization is the process of defining and measuring abstract concepts in practical, quantifiable terms to enable empirical observation and analysis in research. 

We will dedicate another episode sometime to this process of operationalizing, but for now, I just wanted to bring your attention to the importance of ensuring that the thing you are measuring in your survey matches the concept or ideas that you want to match – in other words, that your survey has internal validity.

But for now, we will go over some of the principles and guidelines of good survey design.

SYL: The way you phrase your questions directly impacts the quality and reliability of your data. Poorly worded questions can lead to confusion, misinterpretation, and ultimately misleading results. This might seem like common sense, but if we are not consciously aware of the need to match the question with what we think we are asking, it becomes very easy to drift away from the goals of the survey.

AC: My approach to writing survey questions begins with a holistic approach to the entire survey. I consider the goals of the survey, the purpose of the survey, and what it is trying to accomplish. Then I consider things like who is conducting, and the population being surveyed. Finally, I would also like to think about how the data will be used. If the survey is going to be used in an academic study, I want to know what variable each question is collecting and how that data will be used in the study. 

SYL: I’m sure most of you have already figured out that survey questions come in various forms, such as open-ended and closed-ended questions. 

Closed-ended questions provide predefined response options, making them easier to analyze large numbers of responses but potentially limiting the depth of responses. 

Open-ended questions, on the other hand, allow respondents to answer in their own words, providing richer, more detailed responses. However, they can be harder to analyze.

Often, surveys employ a mix of open-ended and closed-ended questions to balance out the pros and cons of each form of questions.

SYL: Now, let's dive into some principles to take good survey questions and make them even better.

AC: First, before the respondent even reads the question, you need to give clear directions. Whether it is instructions before each individual question, or at the beginning of a set of questions, it is important that it is not assumed that the respondent knows what is being asked.

You shouldn’t over-explain, but giving clear instructions is important. For example, if you are asking a series of questions asking respondents to rate their support for federal legislation, you may want to give some context to the respondent. Or if you are beginning to ask a set of demographic questions, it is helpful to add a brief explanation that says something like “Now we are going to ask a few questions about who you are.”

SYL: Next, it is essential to use clear and precise language to ensure that respondents understand exactly what you're asking. For example,if you ask, 'Do you exercise often?', the idea of what is ‘often’ may be different from person to person. To avoid this issue, a better question might be, 'How many days per week do you exercise?' 

AC: Don’t forget to write questions in a question format. Often, we encounter questions in surveys that are not in the form of a question. For example, instead of just writing “Age” or “Name” and a blank space and expecting people to respond, it is better to ask these questions in full sentences, such as “What is your age?” and “What is your name?” While it may be obvious to most respondents that “name” implies that the survey is asking the name of the respondent, we should not be making these kinds of assumptions.

SYL: Next, pay attention to ambiguous and jargon words. Keeping your language simple and easy to understand helps avoid confusion. Avoid technical jargon and complex sentence structures. Although you should be cautious not to talk down to your audience as if they were children or unintelligent, you need to maintain your trust with them. For example, instead of asking something like, 'What is your frequency of utilizing online banking services?', you could ask something like, 'How often do you use online banking?'

AC: Similarly, avoid mentally taxing questions, or questions that require the respondent to do a lot of thinking or complicated math. For example, asking respondents to rank the order of four or more items is shown to return poor data quality. Or, math questions, like those that ask respondents to calculate how many calories their last meal was, instead, you may want to ask questions which get to the purpose of why we want to know those questions, such as whether the last meal was a healthy meal. Or instead of asking how much the respondents’ income was in the last three months, since you might have a research question that looks at something on a three month timespan, ask about income in increments that people are familiar with, such as per year, or per pay cycle. “Check all that apply” style questions are also shown to be cognitively taxing. While “check all that apply” questions can be useful under certain circumstances, you might want to design the question in a way that makes responding easier, such as by having respondents answer yes or no to each item in the list. Finally, you can counter mentally taxing questions by using visual design techniques to make questions easier to read and answer.

SYL: A common problem we see with questions is due to bias. And avoiding bias is crucial. Your questions should be neutral and not lead respondents toward a particular answer. For instance, a recent survey by ‘US bank’ asked “How much would you agree with the following statement: It was easy for me to accomplish my task.” It may not be apparent, but asking how much you agree with a statement is biased. Instead, a question should be balanced to avoid bias. For example, “How much would you agree or disagree with the following statement:” This phrasing is neutral and doesn’t suggest a preferred answer. Furthermore, bias can come from questions being interpreted differently by diverse demographic groups, hence why questions need to be written in a culturally sensitive way that takes into consideration the different interpretations.

AC: Avoid double-negatives, such as “Don’t you dislike the proposed environmental legislation?” Instead, you can simply ask, “Do you like the proposed environmental legislation?” 

SYL: Leading questions also happen among a lot of motivated surveyors, such as asking a question like “Which one do you prefer: Wendy's Fresh, Never Frozen 100% Beef patties or the competitor’s frozen, gross, fake-beef patties?” Such contrasting and biased descriptions could be minimized to reduce the bias from the surveyor’s end.

AC: Double-barreled questions are a frequent mistake. These questions ask about two things at once, which can confuse respondents. For example, with the question, 'How satisfied are you with our customer service and product quality?' It might be possible that a customer is satisfied with the service but not the product quality, and vice Versa. Instead, the question should be split into two separate questions: 'How satisfied are you with our customer service?' and 'How satisfied are you with our product quality?'

SYL: Loaded questions contain assumptions that might not apply to all respondents. For example, 'How much do you enjoy our new website?' which assumes that the respondent has used the website. A better approach may be to first ask, 'Have you used our new website?’ and then ask about their satisfaction with the new website. Always one question at a time.

AC: We would also caution against asking questions that involve recalling a memory. The further in the past you are asking a respondent to recall something, the more difficult it is for the respondents to answer – and as a result, the lower the quality of the data would be. For example, it would be easier to get good data asking what someone made for lunch the previous day, then on a particular day a few weeks in the past. 

SYL: These are just a few of the principals to keep in mind to help you create better survey questions. But, beyond just thinking about these concepts theoretically, one of the best tools that we have to create good survey questions is pretesting our surveys. Before finalizing your survey, it's essential to pretest the survey, first yourself from top to bottom, then with a small sample of your target population. But also pretest the questions with other people you may know even if they are not exactly in your target population, ask people who are not technical to take your survey, or people who are academics, or your spouse, or your siblings, or your best friend, or whoever is willing to take the survey and give you feedback. 

When pretesting, ask respondents to point out what was unclear, if there were unclear questions or specific words, or if they were unsure how to answer something, or if they stumbled on a question and spent too long on it. Asking pre-testers to verbally give you feedback might be a way to get them to reveal issues with the surveys which might have not been revealed in written feedback. 

So, whatever you do, don’t just design a survey and launch it to the world. Remember to pretest the survey and take the time to refine your survey. 

AC: Asking good questions is a constant work in progress. We want to acknowledge that it is much easier to critique an existing question, than it is to write a good question. Therefore, the best thing to do is to just get a first draft on paper. A questionnaire is never perfect, but we can improve on it bearing in mind all the principles and guidelines discussed in this episode.