Last week the annual Australasian Evaluation Society’s International Conference was held in Launceston, Tasmania. This year’s theme was Transformations, which was more than fitting, given the rate of change in our industry with new theories, approaches and technology changing the way we work.
I left the conference with transformed thinking, my four main take-outs were these:
1. We can measure transformation
The opening keynote address was delivered by respected evaluation consultant, author and former President of the American Evaluation Association (AEA) Michael Quinn Paton. Among other things, Michael discussed how transformation can be measured. This was of particular interest to me, given my current ongoing evaluation of Small Town Transformations for Regional Arts Victoria. Michael argued that the concept of transformation firstly needed to be understood and well defined. He articulated the concept of transformation as highly contextualised, visible or palpable change to a community or place. Think of the fall of the Berlin Wall, social movements over time, changes to the environmental quality of the Great Barrier Reef. To measure transformation is to look at the change along four key dimensions: relevance, depth of change, scale of change and sustainability. Easy, right?
2. We should transform our mindsets about our capabilities
As evaluators, our job is to identify or articulate problems, work with clients to develop key questions for inquiry, identify the best way to collect empirical evidence to be able to arrive at robust evaluative conclusions. Yet there is so much we feel we need to know. There are ever-changing evaluation theories, approaches, methods of data collection and new technologies. Sometimes we feel we need to be expert at everything, which can be daunting and can even lead to unmet client expectations. Perhaps we need to be more confident and clear about where our skills and expertise lie, and where we may need to outsource or learn. We can’t possibly know everything but we sure can find out!
3. Transforming the power balance
Virtually every evaluation involves working with stakeholders and with any evaluation process there is a power dynamic that needs to be established. Evaluators, clients and stakeholders need to establish and agree upon the power balance for a project. This is particularly important when working with Aboriginal communities or evaluating Aboriginal programs or initiatives. Typically, evaluators working in remote or Aboriginal communities maintain all power and control, perhaps just as part of the project management process, or possibly due to inexperience or insensitivity.
Professional and academic, Sharon Gollan, argues that evaluators need to re-think and transform the balance of power to one of control in the project to one of inclusion and cultural accountability. Sharon is a descendent of the Ngarrindjeri nation of South Australia, with family and cultural connections to many communities within and beyond South Australia. Her message resonated with me and confirmed consistent and strong research findings from my consultations from throughout Australia, as part of the preparation of a regional engagement guidelines project for Regional Arts Australia. My consultation findings clearly pointed to a need for an urgent transformative approach in collaborative regional projects. Indigenous protocols need to be considered, and a more equitable power balance achieved between a research/evaluator and the Aboriginal communities in which they are working.
4. Transforming how we write reports
Boring, long reports are killing evaluation, argued Liz Smith, Partner at NZ evaluation organisation, Litmus. If we really want our evaluation recommendations adopted, our reports need to be short and succinct. This is something that respected evaluation consultant Dr Jane Davidson has also been arguing for a long time. In the interest of blog brevity, in keeping with this theme, that is all I will say about that!
These four themes—and many others—played out throughout the conference in highly effective ways. The conference challenged us to explore what transformation means for evaluators and evaluation commissioners , and did so in a highly effective experimental interactive way. Technology was embraced and used well through live video links, real-time polls and questions, and interesting and effective interactive activities. And, of course, it was wonderful catching up with colleagues and making new contacts. These are all key ingredients for a great conference.
See you in Sydney next year!