The Four Sources of Survey Error

In this episode of the Total Survey Design Podcast, we explore the four main sources of survey error through the engaging story of Mario, a local pizzeria owner aiming to conduct an ethical survey. We delve into coverage error, sampling error, measurement error, and non-response error, explaining how each can distort survey results. By using Mario's quest to honestly determine if his pizzeria is the best in town, we illustrate practical strategies for reducing these errors, offering listeners valuable insights into improving survey accuracy and reliability. Join us as we unravel the complexities of total survey error and its impact on survey design.

Transcript

AC: Hello everyone. Welcome back to the Total Survey Design Podcast.

In this episode, we talk about the four sources of survey error.

SYL: Every survey seeks to measure something. The question is, how far from or close to the truth are the measurements in that survey? This difference between what is measured and what is the truth is called total survey error. And every survey has some total survey error. Our goal as survey designers is to reduce that error as much as possible.

SYL: The entire goal of learning about survey design is to reduce the perceived gap between what is measured and what is the truth or reduce the sources of error. 

AC: To talk about Total Survey Error, we are going to use the example of Mario owning a Pizzeria. 

AC: Mario has seen other local businesses that have hung banners that advertise their business as being number one in their field, and Mario thinks his pizzas are pretty good. 

At first, Mario thought to just hang a banner that said something like “Voted the best pizza in town.” But Mario has some good ethics and morals and cannot be dishonest that way. So, Mario decides to conduct an honest survey to find out if the people in his hometown actually think his pizzeria is the best in town. 

Mario decided to contact us to help him conduct an ethical survey that measures the views of the people in town as accurately as possible. He wants to win the competition of public opinion, and he wants to do it honestly, so we are going to help him.

AC: There are four main sources of total survey error: Coverage error, sampling error, measurement error, and nonresponse error.

SYL: We will cover each type of error briefly in this episode, but we will also go into each of these types of errors in their own episodes.

SYL: Coverage error means that the sample frame, or the people who have a chance to be selected to take your survey, is not exactly the same as the target population to which you are trying to apply the findings. In Mario’s case, the target population is the whole population of the town. 

A survey with zero coverage error would mean having a complete list of every single resident in Mario’s town and giving an equal chance to anyone from the list randomly selected to take part in the survey.

On the other hand, an extreme example of really bad coverage error would be if Mario decided to only share the survey about his pizzeria on his private social media account, where only his friends could access the website.

However, since Mario does not have a complete list of every single resident of his town, he thinks of ways to distribute his survey as broadly as possible.

He is thinking of printing a link to the survey right on top of the pizza box. Therefore, all the people who came to his pizzeria can take the survey. But, the issue with this is that there will still be plenty of survey error, since people who are his customers are probably meaningfully different from people who are not his customers – especially in a survey trying to figure out the best pizzeria in town. 

AC: For this survey to honestly say that it represents the views of the whole town, he needs to have a sample frame that includes everyone in the town, or at least does not include certain types of people or exclude only certain types of people. He might consider distributing the survey via a flier to 1000 random houses, with a coupon on the next purchase included if they complete the survey. This might produce less coverage error. However, coverage error might still exist, since a lot of customers might be local businesses, with workers who might not even live in Mario’s town. 

Finally, it is important to mention that coverage error also arises when the mode of the survey inherently excludes certain segments of the population, so if the survey was done by land-line phone, or by cellphone, with each mode, people will be excluded.

AC: The second source of total survey error is Sampling error. Having no sampling error means that everyone in the target population is surveyed, therefore there is no error that could come from the process of sampling. But since the act of sampling means that you are only surveying a portion of the total population, it is important that the group sampled is as close to being representative of the total population as possible, in order to reduce sampling error.

SYL: Ideally, a survey would give an exactly equal chance to everyone in the population to be surveyed. So that even if you just survey a sample, it is representative of the whole.

Even if there is no coverage error, and the survey is delivered to every resident of the town, sampling error could occur because something about how people are invited to take the survey might exclude a certain proportion of the population. 

For example, if Mario’s survey was distributed through a flier with a Q.R. code link to the survey, there is an automatic risk that the survey will be accessed by younger respondents since there is a risk that older respondents might be less familiar with how to use and access a survey through a Q.R. code. 

AC: To meaningfully reduce sampling error, there should be no significant difference on average between the group of people who decide to take your survey and those who don’t. In other words, having no sampling error means that the sample of respondents who are taking Mario’s survey is like the population of the town.

AC: The third source of total survey error is Measurement error, which is when the survey itself is not accurately measuring what the survey is trying to measure. 

For example, if the survey is not anonymous, and if Mario’s friends knew that Mario could see their answers, they might be reluctant to be critical of the pizza, even if that is how they truly felt. This is due to something called social desirability bias, which essentially means that, when someone is watching you, you might act or respond differently from what you would do in a private setting. 

But this is just one source of measurement error! Measurement error can happen because a question is poorly phrased or formulated, or maybe the question asks something slightly different than what the researcher thinks they are asking, or the response options to a question are not exclusive, or maybe because of other biases involved. Measurement error could happen even if the question is worded perfectly, because people will still misunderstand a question. It might be that they did not read it carefully enough, or maybe that English is not their first language. Even if someone accidentally skips the question by miss-clicking on the website, this will also result in measurement error. 

SYL: Reducing measurement error completely is difficult, but there is plenty of research out there that shows us ways that we can effectively reduce measurement error.

SYL: The fourth source of total survey error is non-response error, and thisis when the people who respond to the survey are different from the people who do NOT respond to the survey. This source of error can happen in different ways.

Some people drop out of the survey and do not complete it, and their opinions can be different from those of those who finish the survey. Some people do not answer certain questions – like they skip them or leave them blank. And they can be the important questions in the survey. 

For example, if Mario offers a 10% coupon to anyone who takes the survey, people who like Mario's pizza will be more likely to complete it than those who do not. 

Or, if Mario decides to ask about the best pizza in town, and he asks about what can be improved with his pizzeria, and whether people like his store’s hours, and if people like the customer service… the survey is way too long, and young people and working people may drop out before answer all the questions in contrast to older or unemployed people. People who don’t think they will visit the pizzeria might not feel invested in helping Mario improve his business, so they drop out from the survey, and those people are meaningfully different than those who complete the survey.

In other words, a nonresponse error happens when the people who answer the survey are somehow different from those who do not.

AC: At the end of the day, it is not possible to completely avoid all forms of survey error. We can just do our best to minimize total error. 

Total survey design is an art that takes a lot of balancing in weighing the pros and cons of each decision in your survey design. You might have the urge to over-explain a question so that people do not misinterpret it, but then you risk people not reading your question completely. Or you might increase the incentives for people to take your survey, but it might attract different kinds of people that are substantially different from the average person in your target population. 

Using the art and science of survey design means working towards reducing the four sources of survey error as best we can. It is important to note that excessively focusing on one type of error, like sampling error, can inadvertently increase other errors, like nonresponse or measurement error. The art of survey design lies in minimizing all these errors in unison, not just one of them.