2 key points to allay the fears of evaluation respondents.
Evaluations are conducted to collect, analyse and use information to answer questions about initiatives, projects, policies and program. Typically, they are used to enable funding bodies to understand the effectiveness and efficiency of their programs. Evaluations outcomes drive organisational learning and lessons from the work evaluated.
The process of evaluation itself creates an opportunity to share insight and knowledge. However, sometimes stakeholders fear that if an evaluation is being conducted one of two things must be likely:
FEAR #1: Our program budget is likely to be cut;
or
FEAR #2: My individual performance is under review.
The evaluation process typically involves collecting and analysing both quantitative data and qualitative information. The quantitative data comprises the cold hard facts and numbers, typically numbers of participants, levels of engagement, spend patterns, investment and tourism data, visitation levels, economic measures, and survey findings. Much of this information can come from program records, program acquittal reports, data spreadsheets or emails; information sources that often require little personal interaction.
Qualitative measures, however, are typically gathered by talking with people – program participants, stakeholders and program client target groups. The aim of these consultations is to delve deeper than numbers can describe to explore motivations, barriers to participation, ways of engaging, personal experiences, ideas, challenges, strengths and weaknesses of the program being evaluated.
I evaluate programs within the arts, cultural and human services sectors. Typically, program participants are people who are passionate about their work and may give a great number of unpaid hours to the program. It is understandable that they may fear possible budget cuts or the discontinuation of their positions. But for them to hold these unfounded concerns can impact on the quality of our interview, as they may withhold information from me for fear of the implications.
To allay their fears it is important to anticipate and address them up front in initial correspondence. I tend to make the following key points in what is usually an introductory email:
I outline the purpose of the evaluation and how the findings will be reported and used. I emphasise that it is not an audit or individual performance review.
and
I include a statement about confidentiality, namely that what they say to me in the course of our conversations will be held as confidential. They will not be identified individually against anything they say to me unless they give their consent to do so.
These two points are included in the The Australasian Evaluation Society’s Guidelines for the Ethical Conduct of Evaluations. The guidelines not only describes how professional evaluators must conduct themselves to carry out their work, but they also provide some ideas about how to allay any concerns held by research respondents. There is no simple recipe for ethical practice. Ethical principles need to be applied to a range of situations and procedures may vary depending on the situation.
In my experience, outlining these two key points up front allays the fears commonly held by evaluation participants. But ultimately, my oral and written reports to my clients are direct, honest and comprehensive in the disclosure of findings and the limitations of the evaluation, another point outlined in the AES guidelines. Of course, this means that it is impossible to promise to respondents that their budgets won’t be cut as a result of the evaluation findings, nor will their positions be guaranteed if the program proves to be under-performing. But it is important they know that this is not the reason the evaluation is being conducted.
I have found that if I make those points up front, the air is cleared and the evaluation interviews can take place in a relaxed and open way. In fact, to the surprise of most respondents, they enjoy the whole process as it gives them a chance to have their say, talk through some of the challenges, and focus on the future. They realise that the process doesn’t hurt one bit!