Evaluation is a field in which we encourage judgement. Rather than living by the ‘Please don’t be so judgemental’ mantra, evaluation is about judgement. It involves pulling all of the empirical data together and making a call. Making a judgement in evaluation goes from the ‘what is so’ (typical of research), to the ‘so what’. It involves making and judgement and discussing the implications of those conclusions. The process of evaluation—and making associated judgements— is becoming increasingly central to the practice of policymaking.
Evaluative reasoning is the piece that makes evaluation fundamentally different from descriptive research.
Evaluations can be conducted for a number of reasons, but they are often associated with exploring the merit, worth or significance of something, and that often involves assessing how well something has performed against its intended outcomes. There are a number of different approaches to setting up evaluations to make good and well founded judgements, but all evaluations firstly need to establish values up which a judgement will be made.
1. Establish criteria of merit
This is an important first step. On what dimensions must the the evaluand (the thing we are evaluating) so well? The most straightforward approach here is to consider the objectives of the program. This is commonly used for valuing both implementation and effectiveness, although it is often lacking when valuing a program’s impact. Discussions with key stakeholders are one of the best ways to ensure that suitable criteria of merit have been determined so that the program or activity is being judged in a way that is appropriate and useful.
2. Construct standards
How well should the evaluand perform on each of the criteria of merit? What is optimal? How good is good? How excellent is excellent? What defines poor? Posing these questions together with a client is the process of setting up the evaluative reasoning process.
3. Measure performance
Performance can be conducted through a range of research methods, often including qualitative and quantitative approaches. How well did the evaluand perform on each criteria of merit? Our criteria tell us what evidence we will need to gather to answer these questions.
4. Synthesize and integrate data into a judgement of merit or worth
One we gather evidence, we pass it though our evaluative interpretive channel, work through interpretations and draw evaluative conclusions. Our definitions of ‘how good is good’ tells us how to interpret that evidence. Here we make the judgement, based on the empirical evidence gathered as part of the evaluation. A judgement on our evaluand is one that defines value and one that defines the state of the thing we are judging.
To arrive at our judgement we need to conduct synthesis. Synthesis is the formal organisation of our criteria of merit. We look at what to focus on, how to weight values or scores and the order in which to judge them. It tells us how to define good and bad and everything in between. Jane Davidson, in her book, Evaluation Methodology Basics: The Nuts and Bolts of Evaluation is one of my favourite go-too books to brush up on this method.
In a recent project NSF conduced for Create NSW, an evaluation of a Western Sydney arts funding model, I adopted a rating gauge used to illustrating judgements made on each of our criteria of merit. The gauge was simply a visual tool used in the evaluation report to illustrate the conclusion we arrived at for each area of enquiry – each criterion of merit – and judgements were arrived at based on the synthesis of our qualitative and quantitative data.
To conduct an meaningful evaluation involves putting yourself in a situation in which you are prepared for your program or activity to be judged, which goes beyond description. It can be a powerful way of understanding what works, for whom, where and why, and what areas of your program may be weak or under performing.
But don’t worry… we are only being judgemental to help you make better decisions. And that can only benefit you, your organisation and your community of interest.