If you run a program, project or initiative that you intend to evaluate, it’s natural that you hope the evaluation will show evidence of positive outcomes.
Favourable evaluation outcomes may be needed for your initiative to leverage ongoing funding, perhaps from government, philanthropic funders, or other foundations. As an evaluator, there have been many times when this has been explicitly articulated in the project brief.
It seems natural for evaluators to be tempted to provide exclusively positive findings to deliver on the brief, and to make their lives as easy as possible, including nurturing good client relations, which could lead to future work. Such temptations rest on the shoulders of consultants who rely on future contracts. But I believe it is important to develop a reputation for unswerving honesty and competence as a long-term professional goal.
It is easy to sympathise with the client’s situation. Who doesn’t want positive results? Who doesn’t want to leverage ongoing funding? But if this is really the only thing you expect, perhaps you need to reconsider the role of the evaluator and think about the true value they can add if you are open to a range of outcomes.
Principles of honesty and transparency
As members of the Australasian Evaluation Society, we have a commitment to operate according to the following principles of honesty and transparency:
1. Be clear at the outset
At the start of a project, evaluators have a responsibility to discuss the project brief with the client. We should understand the purpose of your evaluation, how you intend to use it, and the intended audience for your report. We need to make distinctions between the interests of the commissioner and other stakeholders in the evaluation, including the general public and taxpayers.
2. Tell the truth
Independent evaluators must fully report negative findings. We are obliged to adhere to professional standards of practice which include reporting results that truthfully and fully reflect our findings. We must not influence outcomes, or tamper with results that breach the integrity of our research, regardless of any subtle or even unspoken pressure we may feel to report positive findings.
If findings are negative, we are obliged to explain with as much certainty as possible what the results are and what they mean. For example, there are a number of reasons why results may be negative:
1. Targets have been missed. If targets have been missed, we are obliged to be clear about that fact that also clarify how negative the results are. We must report as carefully as possible when things have gone wrong so that similar errors in programming or delivery approach can be avoided in the future.
2. Data may be inaccurate or inconclusive. It is possible that the results are inaccurate due to faulty data collection or analysis, or other evaluation mis-steps. Due to project limitations determined by budget or scope, sometimes data collection may have been inaccurate or incomplete, which means results may not indicate clearly what happened or what the next steps should be.
3. Not fit for purpose. A program or activity may be seen to be working well, but for the wrong audience or community. We need to be able to determine if a program has not addressed the needs of the community of interest.
Evaluation is not marketing
A truthful, transparent, independent evaluation is more valuable than one that is designed as a marketing tool. Good summative evaluation is an essential source of useful in-house information for learning about what works, building competitive advantage, and making prudent decisions (Davidson, 2012). It is the not responsibility of the evaluator to prepare a marketing tool for a client without the freedom to openly discuss any weaknesses or issues associated with the program or activity being evaluated.
What can a client do with negative findings?
Of course, a client may take evaluation findings and do with them what they will. That is beyond the scope or control of an evaluator. That is not to suggest we don’t care. What we care about is telling a client what they really need to know so that if they wish to leverage more funding or support for a program, they will do so with eyes wide open, with a full understanding of what aspects of their program may be worth supporting and what elements are less prosperous.
Our evaluation findings may suggest that they do not spend time trying to garner more support for this initiative, but instead, invest your time and resources into something else. We can work through that thought process with you.
Good evaluation involves open communication between a client and evaluator. We are independent for a reason. We will tell you what you need to know and you need to be open to that process. Then it is up to you to remedy any situations that require attention, or use our evaluation report in any way you see fit. The more areas we present to you for improvement, the better your program or initiative will be in the future.
Don’t get me wrong… I am a positive person, which I hope transfers to my work and my client relationships. But in the world of evaluation, too much positivity can be a bad thing.