Last week I was a presenter and member of a discussion panel at the Spectres of Evaluation Conference, held at the Footscray Community Arts Centre in Melbourne. This international conference aimed to explore issues surrounding the way the arts are evaluated, including competing claims and judgements about limits, uses and value of the arts. Delegates included policy makers, artists, evaluators, arts organisations and government representatives.
I presented in a team of four with Esther Anatolitis, Director of Regional Arts Victoria, (also our panel Chair), Serana Hunt and Helen Kelly, both leaders of an arts-led project being rolled out in the small Victorian town of Dookie. We four are involved in an 18-month project in regional Victoria: the Small Town Transformations initiative. In our 90-minute session we discussed Small Town Transformations from the perspective of the project director, participants and independent evaluator.
During the Q&A session, I was asked the following question about evaluation:
“To what extent do you involve the local regional communities in the framing of your evaluation questions?”
I responded by saying that although the regional communities have designed and shaped their own projects and have taken full ownership of them, they do not have direct input into the design of the evaluation questions. There was some dissent amongst a number of audience members who rightly believe in the power of community, the importance of collaboration and the necessity of project ownership; a kind of ‘grass roots’ approach to community arts which my evaluations have demonstrated is crucial for the longevity and legacy of community-based projects.
This raises an interesting question about how much input local communities should have in the evaluation process and when it is appropriate. The level of involvement of community members in a typical evaluation processes can vary from no involvement to being involved throughout the process.
Benefits of community engagement in developing evaluation questions
Engaging the community in the evaluation process is beneficial for arts bodies, government agencies and policy makers. Community representatives and stakeholders can bring a great deal to an evaluation project. They are in a unique position to identify and contribute to the evaluation process and their involvement can lead to deeper and more accurate data interpretation. Involving key community members in the evaluation process builds capacity with the community which is likely to contribute to the long-term legacy of the project.
Whilst there are many benefits to engaging stakeholders and community members in developing evaluation questions, there are some limitations.
Limitations of community engagement in developing evaluation questions
Limited objectivity
An evaluation is carried out to assess, inform, critique and judge, based on evidence. Evaluation is all about asking and answering questions that matter. If members of the local community are involved in the development of evaluation questions it can compromise the objectivity of the evaluation process, as, understandably, they will have particular areas of interest that matter to them which are likely to be more narrow than the evaluation brief.
Lack of specialist skills
To conduct an evaluation requires a range of specialist skills. Community members have different levels of experience or understanding about evaluation. They will have a range of assumptions about its purpose, design and what is expected of them in the engagement process. A skilled evaluator knows what to look for, what questions to ask and what tools to use to ask them. They know how to design questions suited for surveys, in-depth interviews, discussion groups and a range of other measurement tools.
Different priorities and information needs
Community members often have different interests and needs in terms of what they hope an evaluation will address. To invite all members of the community to input into the design of the evaluation framework can become unmanageable. Instead, it can be beneficial to have a single community member provide who acts as a community representative and works as a conduit for the flow of ideas between the evaluator and the community.
Not privy to the project brief
Evaluators are commissioned directly by government agencies or funding bodies in response to specific project brief. The project brief outlines to the evaluator the purpose of the evaluation. This, combined with discussions held between the client and evaluator, uncovers a range of sometimes complex and subtle contextual influences that need to be taken into account in the evaluation process. The evaluator is in a good position to know what questions need to be asked and of whom, in order to uncover the important and relevant findings. Without this broad understanding of the policy environment, it can be difficult for community members to provide valuable input into the design of evaluation questions.
Conclusion
I have seen evidence that when local communities are fully engaged in regional arts projects, from their inception to their delivery and beyond, the benefits are long-lasting. I believe the most valuable role the community can play in the evaluation process is to participate in the range of consultation activities that seek their views, ideas, and experiences and feed into the evaluation findings. Then the evaluation outcomes will be truly representative of the community.
There is more than one way to conduct an evaluation. I was pleased to see such passion displayed by conference delegates about the importance of community input in the evaluation process. The question in the end revolves around the best way for that input to be captured and directed so that we achieve what I am sure we all agree on: creative projects in regional communities are owned and managed entirely by their local communities.
Let the discussion continue!
Image from left: Natalie Fisher, Helen Kelly, Serana Hunt, Esther Anatolitis (image courtesy of Regional Arts Victoria)