You’ve delivered a government program or initiative with specific objectives in mind and now you ask yourself:
“What was the impact of this program? How can an evaluation answer this question?”
You’re right in leaning towards evaluation to answer your question about impact. But is this the right time? Have you let enough time pass for impacts to become evident?
Impact Evaluation goes beyond describing or measuring outputs of a program or initiative, which can be measured as soon as the program has been delivered.
An Impact Evaluation is unique in two ways:
1. The timing needs to be right
Impact is a long-term concept. The impact evaluation will explore—after time—the types of ripple effects that are seen in a community of interest, and particularly, what sorts of changes occurred for particular types of people in specific circumstances.
To understand the impact of a program means waiting months or sometimes years beyond the end of the initiative, because it can take this long for the ripple effects to become evident. An impact evaluation explores things like new jobs created, new partnerships, increased tourism or community activity, and other unexpected changes to a community of interest as a direct or indirect result of the program or initiative in question. These things don’t happen straight away; they take time. But it is often not feasible to wait years to conduct an impact evaluation to explore these things. It may be necessary to negotiate with your evaluator what is feasible and realistic within the given time frame.
For example, I recently completed a three year evaluation of Small Town Transformations for Regional Arts Victoria, evaluating long-term concepts of community transformation and lasting legacy. For these areas of inquiry to be explored, time needs to transpire between the end of the funded period and the final evaluation field work phase. Yet, my client had their own reporting deadlines and couldn’t wait for years for this evaluation. So I conducted the final phase of the evaluation six months after the end of the funded period so that evidence of any long term impacts may be explored with some confidence.
2. Impact evaluations explore cause and effect
For any changes observed in a community of interest, it is crucial to know whether they were caused by the program or initiative in question. This is also known as cause-and-effect. Without exploring the notion of cause-and-effect, it can be too easy to assume that it was the program that caused observed changes, particularly as that is what we are looking for and expecting.
But to jump to those conclusions can have dire consequences. For example, a recommendation may be made to continue or extend a program, when in fact it may have been other things that caused the changes to take place. Or, if undesirable long term outcomes were observed, a recommendation may be made to discontinue the program, when, in fact the negative outcomes were caused by something unrelated, such as other external factors such as the political situation, local influences, the political natural disaster or anything else.
When would you conduct an Impact Evaluation?
You may consider commissioning an Impact Evaluation if:
1. You want to know whether long term changes occurred within a community of interest as a result of a program; or
2. You rolled out a program or initiative many years ago and despite conducting an outcome evaluation at the time, it would be helpful to know what long-term impacts the program may have had on the community.
To conduct an Impact Evaluation takes patience. Time needs to pass beyond the end of the funded period if you really want to explore the notion of impact on a community of interest.