FROM THE BLOG

Quantitative research in evaluation – for what purpose?

Screen Shot 2016-03-21 at 1.52.14 PM

Evaluation is a field in which evaluators are looking for mechanisms within program that bring about effects. The practice of evaluation involves making a judgement and also often testing a hypothesis of explanation of a theory. In order to make a sound judgement or test a theory, the evaluator needs to gather evidence, either in the form of qualitative or quantitative research.

Whereas qualitative research involves exploring underlying reasons and motivations for people’s preferences, attitudes or behaviour, quantitative research involves the collection of numerical data to tell us about the size or scope of an issue.

The philosophical position in which quantitative research sits is a positivist one; one that argues that the objective accounts of the world may be given by gathering empirical data that contribute to the development of nomothetic, or universal, knowledge (Punch, 2014).

Quantitative methodology can play a range of roles in evaluation. It is underpinned by the theory that there are relationships between variables, which can manipulated or controlled in order to determine any cause and effect relationship. Some applications to the field of evaluation include:

Causal influence

It is possible to use quantitative research methods to manipulate one variable in a program’s operations to identify the crucial program mechanism that causes outcomes, and determine the direction of the causal influence of a program or service.

Extent of an impact

A quantitative approach can explore the extent to which people do something of interest, in relation to the evaluative inquiry. Quantitative methodology can enable an evaluator to compare groups, such as user experiences of a situation. It may allow an evaluator to determine whether the impact of a social intervention on a specific population cohort is a likely predictor of its impact on the broader population. For example, if a welfare-to-work program rolled out in NSW successfully assisted young people gain employment, quantitative methodology may be designed to ascertain whether the program is likely to have the same outcome in other states.

Replicability

Quantitative methodology is also about replicability: the ability to replicate the study objective (Bryman, 2004). This is particularly important in a long-term evaluation, in which data collection tools need to be re-used, possibly by a range of researchers over time.

Scientific plausibility

Brouselle and Champagne (2011) argue of the importance of using quantitative methods to test the scientific plausibility of a program’s theory. Quantitative methodology may be used to examine measures that may work in different ways or trigger different mechanisms within programs. This is important as the evaluative process typically explores what brings about effects.

Measuring learning

Consider a program or initiative that has been rolled out in schools, with the aim of increasing students’ learning in a particular curriculum area. It is possible to measure whether any learning took place from the program by setting up a pre-test post-test design for all participating students. Then, a significance test may be conducted to see whether there was a significant difference in the scores from the pre-test to post-test, and the Effect Size may be measured to explore the extent of the difference.

CONCLUSION

Quantitative research can plays an important role in the evaluative process. But the extent to which quantitative methods may be suitably applied to an evaluation inquiry depends on the situation. There is no one approach that is appropriate for all contexts; the strengths and weaknesses of various instruments of measurement need to be considered, as well as what each approach is likely to achieve. The best quantitative approach will be determined by the nature of the evaluation and the questions of the specific inquiry.

References:

Brousselle, A., & Champagne, F. (2011). Program theory evaluation: Logic analysis. Evaluation and Program Planning, 34(1), 69-78.

Bryman, A. (2004). The Nature of Quantitative Research. In Social Research Methods. Old University Press

Punch, K. (2014). Introduction to Social Research. Quantitative & qualitative approaches (3rd ed.). Thousand Oaks, CA: SAGE

Back to All Posts