Evaluations conducted in the field of arts and cultural initiatives in Australia are often critiqued for their methodologically weakness. Concerns amongst peers and academics have included their reliance on small sample sizes, anecdotal evidence, limited hypothesis testing and little attention to the examination of internal program mechanisms with the aim of looking for causal factors (Dunphy, 2015).
With often limited budgets, little longitudinal data to use, and few industry benchmarks to draw on, it is not uncommon for evaluators to jump straight to thinking about methods of research and data collection to shape their evaluation.
Methods-based designs rely on verifying outputs (such as arts engagement or participation) rather than broader-based outcomes such as measuring community-wide benefits and looking for deeper causal influences on what generates success or otherwise of an initiative. The reasons for this include:
* Simple counts like engagement and participation are straightforward forms of empirical evidence for funders of programs to interpret.
* To link engagement and participation to the success of a program seems logical. Surely the more people engaged or participating over time is likely to be a good thing?
* There may be political sensitivities or constraints in exploring deeper mechanisms of the workings of programs.
But there are limitations and problems in this approach of methods-based evaluation, and critics claim it is lazy thinking. We need to ask ourselves: Just because engagement and participation levels may have increased, how can we conclude this is beneficial for the community? And how do we know what actually caused the positive outcomes for future program planning?
Theory-based evaluation is the practice of going beyond looking at engagement and participation data towards more longitudinal components, literature, ways of measuring outcomes rather than just outputs, and incorporating global thinking and greater empirical evidence. This approach looks for internal mechanisms that make arts and cultural mechanisms work. They are often about testing hypotheses, looking at likely causal influences and what are likely to be the real and anticipated benefits of the initiative on the community. This is about more than engagement and participation.
* bases the evaluation on an account of what may happen, as understood by key stakeholders, community members and participants;
* considers contextual influences of initiatives, which includes stakeholders’ and key community members’ environments and public service culture and behaviour;
* utilises all methods of data collection and evidence building that may be suitable, without depending on, say, just qualitative or just quantitative research;
* is clearly committed to looking for causality (internal validity). It asks: Was it the program itself that caused the outcomes, or some other more plausible influences?
How can theory-based evaluations help remove the evaluation deficit in the arts and cultural sector?
Theory based evaluation moves beyond measuring outputs such as participation and engagement levels to addressing outcomes. It also explores intangible concepts that are likely to be part of the picture, including community change, transformation, perceptions, attitudes and values. These intangible concepts are crucial components of a theory-based evaluation yet they presents a further complexity of measurement precisely because of their intangible nature. But the challenges of measuring intangible concepts must be addressed if we are to undertake quality evaluations of arts initiatives that are designed to make a more meaningful contribution than simply aligning success or otherwise with participation and engagement levels.
I’m starting to thinking beyond outputs towards theory-based outcomes in order to address the identified evaluation deficit, particularly in the arts and cultural sector. I’m keen to explore ways to identify causal links within program mechanisms, and to assess the success of a program beyond its engagement and participation levels. I’m looking forward to the journey and encourage you to come along this long country road with me.
Dunphy (2015). A Holistic Framework for Evaluation of Arts Engagement. Making Culture Count: The Politics of Cultural Measurement. Palsgrave Macmillan, UK.
Stame, N (2004). Theory-based Evaluation and Types of Complexity. SAGE Publications, Vol 10(1): 58-76