You’ve rolled out a government or community program with specific objectives in mind and ask yourself:
“What was the impact of this initiative on the community? How can an evaluation answer this question?”
Impact Evaluation goes beyond describing or measuring outputs of a program or initiative. An Impact Evaluation is unique in two ways:
1. You need to be patient
Impact is a long-term concept. The impact evaluation will explore, after time, the types of impacts that are seen in a community of interest, and particularly, what sorts of changes occurred for what types of people and in what circumstances.
To understand the impact of a program means waiting to evaluate months or sometimes years beyond the end of the initiative, because it can take this long for the impact to become evident. But it is often not feasible to wait years to conduct an evaluation. So it may be necessary for the client and evaluator to negotiate the timing.
For example, I am currently in the midst of a two year evaluation of Small Town Transformations for Regional Arts Victoria, evaluating long-term concepts of community transformation and lasting legacy. For these areas of inquiry to be explored, time needs to transpire between the end of the funded period and the final evaluation field work phase. Yet my client couldn’t wait many years for the evaluation, as they have their own reporting requirements to fulfil for their funder. So I will conduct the final phase of the evaluation six months after the end of the funded period so that any evidence of these long-term impacts can be explored with some confidence.
2. Impact evaluations explore cause and effect
It is crucial to know whether or not it was the program or initiative that caused these changes. Without exploring the notion of cause and effect, it can be too easy assume it was the program that caused observed changes, as that is what we are looking for and expecting. To jump to those conclusions without evidence can have dire consequences. For example, a recommendation may be made to continue or extend a program, when in fact it may have been other things that caused the changes to take place. Or, if undesirable long term outcomes were observed, a recommendation to discontinue a program can be a mistake, when, in fact the negative outcomes were caused by something unrelated, such as other programs, local influences, the political situation, natural disaster or anything else.
There are three well accepted approach for exploring cause and effect: experimental designs, which construct a control group through random assignment; quasi-experimental designs, which construct a comparison group through matching, regression discontinuity, propensity scores or other means; or non-experimental designs, which look systematically at whether the evidence is consistent with what would be expected if the intervention was producing the impacts, and also whether other factors could provide an alternative explanation*.
In most situations in which it is cultural, regional or community initiatives that are being evaluated, non-experimental designs are the most appropriate method to adopt.
When would you conduct an Impact Evaluation?
You may consider commissioning an Impact Evaluation if:
1. You want to know whether long term changes occurred within a community of interest as a result of a program; or
2. You rolled out a program or initiative many years ago and despite conducting an outcome evaluation at the time, it would be helpful to know what long-term impacts the program may have had on the community.
To conduct an Impact Evaluation takes patience. Time needs to pass after the end of the funded period if you really want to explore the notion of impact on a community of interest.
* http://betterevaluation.org/en/themes/impact_evaluation#ImpactEvaluation_1