I’m currently ensconced in report writing for an evaluation project. That is not an uncommon situation to be in, as I work as a program evaluator and consultant. But this project is different; it is more challenging than many, as it poses some big theoretical questions. I am evaluating Small Town Transformations; an initiative of Regional Arts Victoria, funded by the Victorian State Government. Small Town Transformations was a program that offered six towns across Victoria of less than 2000 people $350,000 to develop significant arts or cultural projects to transform their town over a period of two years. This followed the rollout of the pilot project in 2012 in which an original cohort of five towns aimed to do the same thing.
I have been grappling with the idea of whether art can transform small towns since 2012. Whether art can transform a small town is a big question. To evaluate the impact of such an initiative requires more than just reporting outcomes and impacts and measuring engagement and participation levels. It involves asking global questions about what transformation means, and whether the concept of transformation be measured at all. Only if we arrive at the conclusion that it can be measured are we able to determine whether or not it has been achieved.
The premise of measuring transformation is currently being widely discussed and debated within evaluation circles. At last year’s Australian Evaluation Society Conference held in Launceston, Tasmania, esteemed American evaluation scholar, author and practitioner, Michael Quinn Patton, delivered a keynote address posing the question about how we measure transformation. He drew on four dimensions of transformation, as identified by the Independent Evaluation Group of the World Bank. I have adapted this thinking into my evaluation of Small Town Transformations evaluation.
1. Relevance
To think about transformation one must consider relevance. One must assess whether there were significant changes that addressed a major community challenge, such as poverty, equity, climate change. If the recorded changes are not of critical importance to a community, then they cannot be considered to have been transformative.
2. Depth of change
Depth of change refers to whether the transformation either causes or supports fundamental change in a system or market, addressing a root cause or supporting a change in trajectory. It may be about behavioural change in a community or the systematic change in which that community functions.
3. Scale of change
It is important to ask whether the change caused large-scale impact on a more regional (or even national) level. There needs to be a demonstrative effect, obvious spillovers and externalities. Perhaps there has been a discernible acceleration or discontinuation of a crucial indicator of how the community functions. That is, something that changes significantly, but in isoloation, is unlikely to have been transformative.
4. Sustainability
The impact of the change is economically and environmentally sustainable in the long term. That is, the transformative effects on these dimensions remain after the engagement of the program ends. Within the arts cultural sectors, this is often referred to as the ‘legacy’ of the initiative.
Theory-based evaluation
In order to incorporate this big-picture theoretical thinking into my evaluation of Small Town Transformations, I adopted a theory-based approach to the project. Theory-based evaluation is a widely accepted and understood evaluation approach by which the impact of programs are systematically assessed to help us underhand how they work, for whom, and in what circumstances. It is an ideally suited approach to evaluating an initiative like Small Town Transformations that is a complex structure and that aims to achieve long-term change.
I am delving into the big questions of what transformation means in the context of this initiative, and whether there is evidence that it has been achieved. I am starting with a broad hypothesis about how the initiative was intended to work, then whether we can either prove—or disprove—that theory using empirical evidence. Having tracked and monitored the progress of the initiative over the years, I am exploring the cause-and-effect mechanisms within its complex system, and, importantly, looking for other external factors that may have contributed to any significant or transformational change. Only then will I be able to say whether the program model has acted as an enabler of transformation, and, as outlined by Michael Quinn Patton, whether observed changes can actually be defined as transformational or not.
If you’re trying to contact me over Easter, you’ll most likely find me at my desk!