Sample Design - Part 1. Sample Size - or why just over 1000 responses is enough to generalize findings to millions of people!

In this episode of the Total Survey Design Podcast, we explore the fascinating concept of sample size and its impact on survey accuracy. Discover why just about 1000 responses can represent the views of millions, debunking common misconceptions about survey sampling. We'll dive into the key parameters for calculating sample size, such as margin of error, confidence level, predicted variance, and target population size. Learn how a small, well-chosen sample can provide precise and reliable data, making large-scale surveys both practical and cost-effective. Tune in to understand the math behind the magic!

Here's a general formula commonly used for calculating sample size for simple random sampling:

𝑛=𝑍^2*𝑃*(1−𝑃)/𝐸^2

Where:
𝑛 = required sample size
𝑍 = Z-value (the number of standard deviations from the mean corresponding to the desired confidence level)
For a 95% confidence level, 𝑍≈1.96
𝑃 = estimated proportion of the population (if unknown, 𝑃 is often assumed to be 0.5 to maximize the sample size)
𝐸 = margin of error (the desired level of precision, expressed as a decimal)

Find us online at: instagram.com/totalsurveydesign/
https://taplink.cc/totalsurveydesign
Contact us at: totalsurveydesign@gmail.com

Transcript

SYL: Hello everyone. Welcome back to the Total Survey Design Podcast.

In this episode, we talk about determining an appropriate sample size and why sometimes only about 1000 responses are needed to provide representative data for the entire United States!

SYL: I have always wanted to do a podcast episode on this topic. The inspiration came from listening to some YouTube commentaries, which expressed almost in a mocking way that a certain study in a certain article has only gathered 1000 responses as if that was enough to get a good sense of the opinions of all Americans. 

AC: Sure, for most people unfamiliar with sampling methodology, 1000 might seem like a ridiculously small sample size. Some might think that to get a good national poll, you would need to get tens or even hundreds of thousands of people to respond to a survey to get accurate representation. But in this episode, we will explore how a sample size is calculated based on what you want to achieve with your survey.

AC: Let’s begin by talking about what a sample of a population is. 

SYL: A sample is just a part of the population that you want to survey. So, imagine that your target population is the United States; your sample is the number of people you survey to get the data that is meant to be generalizable to the entire population. If you happen to sample everyone in the population, that is called a census. The U.S. Census conducts a survey of the United States of America once every ten years, but it also costs tens of billions of dollars to conduct. 

You can imagine why it doesn’t make sense to survey everyone in a population for every survey. Not only is it extremely expensive and impractical, but it is also inefficient and unnecessary. Often, surveying just a small proportion of the population could give us enough information to act on whatever the purpose of the survey might be.

If you think of various populations that people might want to learn about, such as university students, business customers, or city citizens, surveying just a proportion of the population is cheaper and more efficient. You can collect the data much faster. 

AC: Sometimes, you want to attempt to survey everyone in your target population, such as all the people who attended an event that you want to assess. It might be possible since you have a list of the contact information of all attendees. It might also be possible to survey the entire population of a university student body. But, in situations like all the customers of a business or all the citizens of a city, it is realistically almost impossible to have any complete lists of the whole population. So, to survey the whole population, it will take a tremendous amount of effort and resources.

But even when we have the complete list of the population, it is not easy to get responses from everyone. In fact, it is often impossible. This is okay because surveying just a proportion of the population still gives us good data. 

You might be curious at this point: How good could the data be if it is just a proportion of the population? Well, with a simple mathematical calculation, we can say a lot about how confident we can be in our findings and how precise they are. 

One way to think about the sample size is that when you can’t survey everyone, you need to figure out how many people need to be surveyed so that you can say with a good level of confidence and precision that the data reflects the population. This is what it means to calculate the survey sample size.

The formula for calculating the sample size is a bit too complicated to discuss in detail, but we will include the formula in our episode notes.

To me, it is more important that you understand the idea behind the formula. The most important thing to keep in mind, without having the formula in front of you, is that several key parameters matter when calculating sample size. Now, let’s discuss those key parameters.

SYL: First, you need to determine your desired margin of error or the level of precision of your findings. 

Then, you need to determine the confidence level.

Furthermore, you also need to have an idea of the predicted variance of your questions. 

And finally, the target population size. I will elaborate on each variable now, starting with the margin of error.

SYL: The margin of error is essentially how close your findings are to the true average of the population. In other words, the smaller the margin of error, the closer to the true mean of the population are your findings.

·         Margins of error can be plus or minus 3%, 5%, and sometimes even 10%. The right percentage depends on how precise you need your data to be.

·         Another way of imagining the margin of error is to imagine an arrow shot at a target with a series of circles surrounding the center. A smaller margin of error would mean that your arrow is closer to the center of the target.

The other factor you need to consider is how often you can be sure that the arrow will fall in the area you estimate it will hit. This is known as the confidence level.

·         In other words, confidence level tells us how often you expect your data to fall within the precision range your results have shown.

·         The scientific rule of thumb for an acceptable minimum is a 95% confidence level. This means that, if you conduct the same survey 100 times, 95 out of those 100 times, your findings will be representative of the population.

Once you have determined the margin of error and the confidence level, you need to know your predicted variance.

AC:

·         What this predicted variance means is that different research questions have a different proportion of responses. For some questions, you can expect that most people will answer response option A, whereas only some will answer B. For other questions, you might expect a more even 50/50 split. 

·         This even 50/50 split is the most conservative estimate of the predicted variance. It means that you expect half your respondents to answer A and half to answer B.

·         If you want to obtain sufficient statistical power for every group being surveyed, you need to predict your expected variance. With a 50/50 split, you need more respondents since they are equally split between the two options. But with a 20/80 split, you get more people in one box (the box with 80%), so you actually gather more statistical power to describe the population based on their answers with the same number of respondents.

·         So, a survey with a 50/50 split might need over 1040 respondents to describe the population within +-3% with a 95% confidence interval, whereas a 20/80 split only needs 672 respondents to achieve the same level of precision and confidence. 

The final variable to consider is Population size. But Population size only matters to a certain extent. For very low numbers, you need a greater proportion of the population to confidently say that the mean results are representative of the population, but for very large numbers, the sample size is about the same.

SYL:

·         Consider first that you want to precisely figure out the opinions of three people. If you only ask one of the three people, the views can vary wildly, so it is not possible that you can be confident that surveying just one person will be very precise. Surveying two of the three is also not enough to get the view of the three. Therefore, you really should be asking all three people. As the target population size grows, the proportion of the population that needs to be surveyed goes down.

·         For example, if you want to survey the views of the workers of a small office of 100 people on an issue that might be split 50/50, you still need to survey 92 people—or 92% of your population—to be able to say with 95% confidence that your findings are within 3% of the true mean of the population.

·         But if you are surveying an office of 1000 workers, you now need only 517 responses, or 51.7% of the population, for the same precision and confidence.

·         If you are surveying a large company of 10,000 workers, you now need to survey only 965 people, or 9.65% of the population, for the same level of precision and confidence.

·         As the target population size increases, the proportion that needs to be sampled goes down until, eventually, the number of people you need to survey reaches about 1,067, which is enough to describe a population of 1 million or more! So, if you survey 1 million people, or 567 million, you will only need 1,067 for the same level of statistical precision.

However, I would imagine that the idea of needing just over 1000 responses to be representative of the whole population would still be a bit hard to accept for many. That is why I want to spend some time focusing on this concept. 

AC: Here’s a great example from Ashley Amaya, who is now an Associate Director at Pew Research Center. She posted this on behalf of the American Association for Public Opinion Research on a Reddit forum about four years ago.

Let’s talk about M&M’s. Imagine that you want to get a sense of the ratio of colors produced. According to mms.com, about 400 million M&Ms are produced each day. 

We don’t need to observe all 400 million M&Ms to determine the ratio of colors. Instead, we can observe just a sample of M&Ms. Intuitively, since there are six colors, we need to have a sample that is big enough so that each color has a chance to appear in the ratio that it exists in production. We should probably examine a sample that is bigger than just a handful. I would guess that if you took three or four large bags of M&M’s, you would get a pretty good sense of what the ratio of M&Ms is in every bag produced. It's the same with gauging the attitudes of the entire U.S. population.

If you conduct the survey properly, and you give the entire population an equal chance of being randomly selected to complete the survey – also known as a simple random sample or a probability sample – and if your survey does not suffer from other major survey design flaws, then statistically, the variations of the answers between those 1067 respondents will vary in such that in total, you can with 95% confidence say that your results are within 3% of the true population mean. 

You can add another 1000 respondents, and if your previous responses were collected soundly, then the next 1000 respondents will not significantly influence the means of your findings. It is like with the M&Ms. Once you have gone through a few large bags, you can then add another large bag, but counting the new bag will not significantly influence the ratios of colors that you have already counted. The mean findings might change by a percentage point or two, but not more.

That’s why, with about 1000 respondents, you can get relatively accurate data that could potentially represent a population of the size of the United States of America.