FROM THE BLOG

Survey sample size – how big is big enough?

Crowd

When designing and running surveys, the question often arises, ‘How large should the survey sample be?’ It is easy to be seduced into thinking bigger is better. If time and budget were limitless, we would interview everyone in a population to guarantee reliability and accuracy. But that is never possible. There comes a point of diminishing returns over a certain sample size. This means that the more time and money you spend on creating a larger sample size gives you less and less return on your investment. There are unlikely to be differences in survey findings as you increase your sample size beyond a certain point, so you have to have an idea where to stop.

Most importantly, using an appropriate sample size is critical for research.

There are three main considerations when determining your sample size:  margins of error, response rates and the way the sample is selected. To understand these factors, it is often helpful to think of using a sample to look at a community as rather like using a magnifying glass to look at small objects. The more powerful the magnifying glass (the larger the sample), the more detail you can see and the more clearly you can see these details (the margin of error). So, there is no ‘correct’ sample size. The questions that need to be asked are:

How fine are the details I need to see?;
How clearly do I need to see them?; and
Will my budget allow me to buy a magnifying glass that meets these needs?

Margins of error

The margin of error in a survey is rather like a ‘blurring’ we might see when we look through a magnifying glass.  The larger the sample the smaller the margin of error (the clearer the picture). The story gets complicated when we think about dividing a sample into sub-groups such as male and female. The sample size for each of these groups will, of course, be smaller than the total sample and so you will be looking at these sub-groups through a weaker magnifying glass and the “blur” will be greater around any percentages we base on the sub-samples. The smaller the margin of error, the more confident we can feel that a percentage estimated from our sample is close to the percentage we would find if we could talk to everyone in a community.

Response rates

It can be costly to achieve a large sample size. This is where the response rate has to be taken into account. For example, a sample size of 1,000 is a statistically robust size.  But if 100,000 people had to be contacted to achieve that size, this low response rate (1%) could have a dramatic effect on the representativeness of the sample and the accuracy of the results.  In terms of our magnifying glass analogy, a low response rate is like having a magnifying glass that can only see the blue parts of an object and not all of the other colours. So, a survey with a low response rate is much more likely to provide a biased view of what you are looking at.

Let’s consider a survey of a small regional town where we hope to get a sample size of 300.  Response rates in small towns are usually higher than in larger metropolitan areas, as people are better connected. It is not unreasonable to anticipate a response rate of 50% in this situation, which is relatively high. With a response rate of 50%, if we are aiming for sample size of 300 we would need to approach 600 people to participate in the survey. We would still need to be conscious of the fact that the 50% who didn’t take part in the survey might have opinions or demographics that differ from those who did participate and look for clues to whether this is so or not.

If your survey is to be conducted by telephone, your budget should allow for making at least 1,200 calls in order to achieve a sample of 300 people who represent the population. For an online survey you need to consider a range of channels through which your survey may be distributed. This may include links on websites that the community is likely to access (such as a Council website), residential email lists and social media. In a small town of 1,500 people it may be most efficient to turn up at community events and activities such as sporting events to approach people on the spot.

The selection of the sample

A given sample can never match the characteristics of a population. It is important to use the best technique to ensure the sample is as randomly selected as possible to ensure the best representation of the population.

With all projects, there are time and budget constraints. If it is unlikely you will be able to achieve a sample size of 300, it is better to focus on the quality of a smaller sample. To put it simply, it is better to have a smaller sample of, say, 100 or so relevant respondents than a larger sample of people who are of little or no importance to you.

If respondents are approached at community events, it is important to approach people with a range of demographic and personality characteristics. It is not uncommon for interviewers to tend to approach certain types of people who are most like themselves, but this will only bias the nature of the survey. Don’t just approach people at sporting events, as this will likely result in a sample bias. If your survey is about broad community attitudes to the arts, it is important you don’t just rely on networks within the arts communities, as your sample is already skewed.

With online surveys, it is the responsibility of the researcher to monitor the demographic profile of respondents as they come in. If there seems to be a bias towards a particular type of respondent it will be necessary to find ways to approach other members of the community or population to ensure the sample is representative.

Conclusion

There is no magic solution to determining the right sample size for a survey. Good judgement is required. It is important to aim to achieve the best possible outcome with your available budget. If, through trial and error, you do not obtain the sample size you feel is required for your study, it is important you don’t attribute meaning to your results that do not exist. It is far better to acknowledge that you have a limited sample size and that the findings are likely to be indicative only.

Thanks to Rob Hall, Director of Environmetrics, for input into this blog post.
Back to All Posts