The Costs of Conducting Surveys - Part 1

In this episode, we explore the costs associated with conducting surveys. We start by debunking the common misconception that surveys are free, highlighting the significant expenses involved in various survey methods—from internet to mixed-mode surveys which are needed to collect useful data. We explore the astronomical costs of large-scale efforts like the U.S. Census, which can reach billions, and contrast these with more modest academic research surveys, which can still run into thousands of dollars. Special attention is given to the balance between cost and quality, underscoring that a higher investment often means better data. Through specific examples and a detailed breakdown of costs in both large and small-scale surveys, this episode provides a comprehensive overview of what it truly costs to gather reliable data.

Find us online at: instagram.com/totalsurveydesign/
https://taplink.cc/totalsurveydesign
Contact us at: totalsurveydesign@gmail.com

Transcript

SYL: Today we are going to talk about the costs of conducting surveys.

AC: When I first started thinking about conducting surveys, I used to believe it was a lot like writing an email—just like Yahoo and Gmail is for emails, free services exist for surveys like Google Forms or SurveyMonkey, and anyone could make a survey, and it should be free. 

However, once I began learning more about survey design, I quickly learned that there are so many factors to consider that can significantly hike up the cost. And with good reason.

Today, we're going to dive into the various costs associated with conducting surveys, covering surveys of different sizes and costs, everything from the grand scale of the U.S. Census to more modest surveys for academic research, discussing the real costs with those surveys. 

SYL: As you know, surveys can be conducted through various mediums, each with its own cost implications. These mediums include internet surveys, phone surveys, mail surveys, and mixed-mode surveys. 

In general, Internet surveys are the least expensive. Costs mainly include software subscriptions and incentives for respondents.

On the other hand, Telephone Surveys are more expensive due to the need for call centers, ideally with a CATI system, trained interviewers who are usually paid hourly, and phone bills.

There are also mail surveys, which have costs including printing, postage, stamps, the labor cost of assembling the envelopes, the sampling frame, and incentives to encourage responses.

And, finally, mixed-mode surveys can potentially be the most expensive, as they combine the costs of more than one mode among those that we just mentioned, but also are more likely to maximize response rates and coverage.

AC: Let’s look at some specific examples to give you a sense of the range:

The U.S. Census is a massive population-wide survey that costs about 16 billion dollars. The Census involves extensive planning, thousands of field workers, and multiple modes of data collection to ensure every person is counted. In 2020, you probably had either received a physical copy of the survey in the mail or had an interviewer come by your doorstep.

One might think that 16 billion dollars is a lot of money. Ideally, the way the US census would work is that every 10 years during the decennial census count, everyone would log into a website set up by the census and fill out their demographic information. This would reduce the cost of the census by about $10 billion, which is about 80% of the budget, and is the amount of money set aside for enumerations operations, infrastructure, and information technology. You will still have infrastructure and technological costs, but without the need of field enumerators, you will have a huge saving. 

SYL: But, the world is not ideal, and most of us don’t just automatically participate in the census. And since the enumeration is written into the constitution, and rightly so, it must be done. 

So, here are the breakdowns of some of those costs. Every county in the United States has dozens of enumerators to count people, walking from home to home. Enumerators are paired, and they spend several months covering all of the places that are missing data. Those enumerators need to be paid. they are making about $18 an hour on average. Each enumerator makes about $20,000 for their role in the census. To cover an average county population of about 100,000 people, each county has an average of 200 enumerators. That comes out to $4 million per county just for the enumerators. That’s $1.26 billion in all the US just for hourly wages. Then you have Census field supervisors, who make slightly more. And regional managers, and national managers, and so on. So, just the human cost of running the census is in the billions of dollars. But the US census also needs to pay their workers for miles driven, and for training, and for the technology that they use, and this is all the costs of just collecting the data. Then you need experts who can clean the data, organize it, analyze it, and then you have teams of scientists writing reports which will be used by everyone. 

If you add everything up, it becomes understandable as to why the US census costs billions of dollars. 

AC: Luckily, most surveys don’t cost billions of dollars. But they still cost something.

The cost of conducting academic surveysvary significantly depending on the approach, potentially ranging from a few thousand dollars to more than 50 thousand dollars.

The 50,000 dollar plus version is what I like to call the Cadillac Version, and it includes using the full address-based sampling list, which can be quite costly to access. Additional costs include Printing, postage, survey software, incentives, data analysis tools, and possibly hiring research assistants. This is perhaps the gold standard of science research to produce nationally representative generalizable findings.

The costs can be brought down significantly, if a lot of the work is done by the researcher, such as instead of hiring people to stuff envelope, you buy a few friends some pizza and gather then around one night to stuff a few hundred envelopes each. Costs can be reduced by minimizing incentives, but this comes to the sacrifice of the response rate. Fliers and letters for recruitment can also be printed in black and white instead of color, and the paper quality can be lower. And instead of sending a letter to potential recruits, you can send a flier instead, which reduced the cost of postage and paper. This shoe-string budget may still cost in the range of 15 thousand dollars or more.

SYL: An alternative to a mail or mixed-modes survey might be using platform services like YouGov, which promise nationally representative random samples, and which can potentially cost around 7 thousand dollars for the same number of respondents as the previous two versions mentioned. A panel maintenance and the quality of the data provided. Although, I am unfamiliar with the quality of their sampling frame. 

Moving down on the cost tiers, Prolific is a platform that hosts surveys from researchers all over the world. They are based in the UK, but have a large number of participants in the US and the rest of the world. For the same number of respondents, you can get a convenience sample from them for about 3000 dollars.

MTurk offers similar services as Prolific, and has been around Prolific longer, but while Prolific is designed with research in mind, MTurk had a wide variety of tasks that people can pay workers to complete. Research shows that while the costs of administering surveys on MTurk are lower than Prolific, the quality of the data is also considered to be lower, and more recently MTurk appears to have problems with bots and other issues. 

AC: It is not an absolute rule that the more expensive a survey, the better the data. But it is still a good indicator of the quality of the data. This is why it is important to consider how the data was collected and potential issues with the cost aspect of the survey, whether you are the one conducting research, or you are reading someone else’s research. 

Since a large proportion of social science research today is conducted on platforms such as Prolific, I want you to be aware of how Prolific works. Prolific allows the researchers to set their own pay rate. While they do suggest a certain rate, there are many researchers who pay way below even minimum wage. A study that pays below minimum wage could turn away a lot of respondents. Also, even if participants respond, you can guess that they are mostly either economically desperate or not paying attention, because of the low pay. As a result, a low-paying survey has a greater risk of having lower quality data than a survey that pays respondents a fair rate. 

SYL: Finally, it IS possible to conduct surveys for “free”. But you might think conducting a survey can be free, hidden costs include your time and effort. Also, due to lack of incentives, there are always risks such as lower response rates and poorer quality data. So, if you are a reviewer for an academic journal, or just a reader of a scientific paper, you should see if the pay rate for the study is good enough, and also to be cautious when the authors do not tell you how much they paid their respondents.

AC: Looking beyond just personal and scientific research, some surveys are conducted by teams doing program evaluations. Program evaluations are meant to make sure that government-funded projects or privately funded projects are achieving the goals they are setting out to achieve. Programs funded by the National Science Foundation often involve extensive data collection and analysis phases, hiring external evaluators, and ensuring rigorous methodologies are followed, and because of this the costs tend to go up significantly higher, often ranging between 10 and 90 thousand dollars per project. While these costs might seem high, they are in fact a small fraction of the total budget of the entire budget, and these evaluations make sure that projects achieve the goals that they set out to achieve year after year.

SYL: Why don’t we discuss the cost of conducting surveys from an actual study? Azdren, you have a great case study to talk about the cost of surveys. Could you tell us more about your research?

AC: My dissertation research examines public support for climate change mitigation and adaptation legislation through a survey of voting aged Americans. And for this topic, I came to the conclusion that a survey would be the best methodology to capture people’s attitudes. I began planning for this push-to-web survey, hoping to have a nationally representative sample. However, the more I began to break down the budget with more detail, the more I came to realize about the cost. 

I spent a significant amount of time trying to figure out the budget for my survey, so that I could apply for funding, and get on with my dissertation. Here is a line-by-line breakdown of the cost of my push-to-web survey, with the goal of collecting 1067 good responses. 

SYL: And if you are wondering why 1067 respondents, we talked about this in our previous episode, Survey Design Part 1, Sample Size. 

AC: To collect the 1067 responses, I first figured that I would need to build in some kind of a buffer for bad data, of incomplete data, or of responses that were not completed in good faith. So I added another 25% to the total, for a required number of responses of about 1333 completions. 

SYL: 25% attrition rate is considered acceptable in today’s day and age. And could you tell us more about how you’re going to conduct the survey?

AC: I decided to conduct the survey in three phases, expecting that some people would respond to the survey in each phase, reducing the amount of mailings in each phase gradually. In the first phase, 4000 letters would go out.

The cost of collecting a random sample of 4000 addresses is about $700 from Marketing Systems Group. Then the letters need to be printed, on color, which would cost me about $2,663. The paper itself costs about $68. And the services of someone ensuring that each letter goes to the right address throughout the printing process costs $414. The cost of actually cleaning the addresses and making them ready to print is about $30. The scoring and folding of the letters cost about $166. Then, having someone insert the letters in envelopes cost about $368. 4000 color windowed envelopes cost about $792. The stamps bought in bulk through the USPS for my nonprofit work cost about $876. Finally, a $1 incentive is inserted into each envelope, for a $4000 cost. The total cost for wave one comes up to about $9,378, which does not include any costs of actually creating the survey itself. 

SYL: That’s the first wave, costing about over a whopping 9000 dollars. Then what about the second and third wave?

AC: The second wave consists of sending postcards to 2000 people who had not yet completed the survey. The costs associated with this include about $740 for the postcards, $398 for the postage, and about $444 for address cleaning and sorting, for a total cost of phase two of $1,582. 

The third wave would involve sending 1000 mailed survey letters, for a total cost of about $1,177. 

The total cost of these three waves comes up to about $12,816. This also does not include the costs of hosting the survey on Qualtrics, which is a service which is provided to me by my university, but which might otherwise cost about $420 per month, or just over $5000 per year. Although, I would still need a more expensive plan, since their $420 plan is only for up to 1000 responses. 

SYL: That sounds like a lot of money to spend as a graduate student. What if you were to conduct this survey but on a smaller budget? 

AC: If I were to conduct this survey on a smaller budget, I would recruit my participants through an online panel platform such as Prolific. Platforms like Prolific provide you access to a panel of users; you just need to determine how many users you want to recruit and what are the criteria of recruiting. 

If I recruit 1333 respondents for a 6-minute survey that pays 1 dollar 50 cents per person, that comes out to a pay rate of about $15 per hour, which is what is considered an ethical minimum pay rate. That means that the total pay just for the respondents would be about 2000 dollars. Prolific advertises their fee as costing you 25 percent of your total cost, or the way you determine how much that will be is by taking what you will be paying participants and adding 33 percent. So, for my study, Prolific’s fee will be about 666 dollars, making the total budget about 2,666 dollars. 

Again, this does not include the costs of hosting the survey on Qualtrics. 

SYL: Also, in the beginning of this episode we mentioned how there are other platforms that let you host surveys for free. While there are free websites to host surveys, such as Google Forms and Survey Monkey, other websites such as Qualtrics offer platforms to host surveys that have more complex features, advanced tools, including analytical tools and sampling tools.

Even for free platforms like SurveyMonkey, they also offer paid premium services that are worth mentioning and quite useful. For example, for their lowest paid tier called the Advantage Annual plan, it costs about $468 per year, and it offers unlimited number of surveys, up to 15,000 responses, data exporting such as through CSV, skip logic, and so on. We are not paid by surveymonkey to advertise their paid services, maybe they should. But we just want to highlight that they do offer services which can be very useful to surveyors.

It’s like with emails. Many of us might be upset if someone told you to pay 500 dollars per year to have access to your email. And a lot of people who are unfamiliar with surveys might think the same for people paying to conduct survey. 

To conclude, we hope that this episode gives you an insight into the costs of surveys. Whether it is the cost of fielding a survey, designing a survey, hosting a survey, or analyzing the data from a survey, there are many important elements to conducting a survey, all of which should be valued on their own.