Typically, evaluation and research is about human beings. Our focus of interest is often users of programs or initiatives. They may be individuals, social groups, special interest groups, clubs, companies, cities or countries. We need to identify people willing to provide their experiences, thoughts, ratings, assessments, or other input into the topic of interest. But how do we decide which individuals to include in our study? Is it just a case of picking those who put up their hand?
The first step is to have a clear idea about who makes up the research population. At times we may be able to collect information from every individual we are interested in, such as all students in a class, or all users of a specialist program. In such cases, these are people who make up the entire population of the study; and we would be conducting a census.
When conducting quantitative research, in particular, there will be too many people who make up our population of interest. Or we may not be interested in all the views of all types of people within that population group. For example, we may decide that the policy or program is only likely to impact on people living in regional Australia, so we may exclude from our study people living in metropolitan locations. It would then be reasonable to assume we will not be able to involve everyone in regional Australia in our study. So, we need to consider consulting with a smaller, more manageable group. This smaller group is termed a sample and, as far as possible, it should be representative of the much larger population.
There are two main options for selecting a sample from a larger population:
1. Probability sampling
Also called random sampling, to use this approach in the selection of respondents, every individual in the population has the same chance–or probability–of being included in the sample. A common example of how probability sampling works is the running of a lotto draw. As lotto balls are drawn at random from a barrel, each ball has an equal chance of being selected.
Random sampling is the only approach a researcher or evaluator can use to be sure of obtaining information from a relatively representative group of the population. It may be most useful in the conducting of large-scale evaluations of policies or programs that impact on a broad sector of the population. Simple random sampling allows evaluators to make generalisations, or statistical inferences. This is a major advantage because such generalisations are more likely to be considered to have external validity that is, they can be replicated in other contexts. However, probability sampling is expensive, time consuming and not always desirable, possible, or even necessary.
2. Non-probability sampling
Non-probability sampling involves using a subjective procedure of selection, whereby some people will have a greater chance of being selected than others. A large proportion of evaluation and social research studies use non-probability sampling as they can be executed more quickly and easily than probability sampling. In small samples such as these, a carefully chosen judgement sample may be better to represent the mix of potential respondents in a population than even a probability sample as you can balance your sample to be in keeping with known market characteristics. It is important, when using non-probability sampling techniques, to eliminate any recruiter bias in the selection of individuals.
How to decide which approach to use?
There are a number of specific probability and non-probability sampling methods, each presenting advantages and disadvantages. The choice between probability and non-probability sampling methods should be based on considerations such as the nature of the research, what questions you are trying to answer and from whom, the degree of error tolerance, and other commercial realities including time, resources and budget. Conversations with your evaluator or researcher will help you determine who you should pick and how you are able to find them.
More information about sampling methods can be found on the Australian Bureau of Statistics site, here.