KPIs: Key Performance Indicators. Surely they’re the most relevant indicator when conducting an evaluation, right?
KPIs are widely known and used in the business world. They are known as measurement values that demonstrate how effectively a company is achieving its key business objectives. KPIs are typically quantifiable measures that are used to track and assess the status of something. So, surely KPIs would be considered the most fundamental metric in an evaluation process, wouldn’t they? Well, not necessarily.
Evaluation is a discipline or process that is about measuring. It can be easy for evaluators (and program funders and managers) to gravitate towards the aspects of a program or initiative that can be easily measured. To do this results in the acquisition of easy data. And data is often revered by program funders, as data is perceived as providing the hard evidence of how a program or initiative is performing. But more important that easy data in evaluation is the presence of good, relevant data: data that addresses the important questions and sheds light on a program or initiative that is meaningful.
Good evaluators will spend some quality time thinking about whether the KPIs that are already in place are going to be useful metrics in order to assess the success or otherwise of a program, and to tell a meaningful story. As internationally recognised evaluation thought leader Jane Davidson recently told our Masters class:
“It is more important to get an approximate answer to the right question than to get to the sixth decimal place using the wrong KPIs”.
What Jane means here is that just because you can collect data on a range of measures doesn’t mean they are going to be useful. Instead, it can be more prudent to find other ways of answering the right questions, even if it means having slightly less certainty. Because in the end, there is no point conducting an evaluation if it doesn’t tell us what is going to be meaningful for strategic decision making.
Jane highlighted a few important questions that program managers, evaluation commissioners and evaluation practitioners should ask themselves that are arguably more important to explore in the evaluation process than standard KPIs.
1. Relevance
Is this program or initiative still needed? Is it the most important thing in the suite of options available in order to solve the problem it was designed to address? Sure, this can be a sensitive topic to explore, as program commissioners may not want evaluators considering the redundancy of a program that their jobs may depend on. But at the same time, it is important that publicly funded programs can demonstrate their relevance to the community for the benefit of all tax payers.
2. Implementation
Is the program/ initiative delivering the right thing to the right people in the right way and efficiently? There is a range of metrics that can be used to explore these things, and many of them may not include the range of KPIs that are more commonly associated with the program by those who work with it.
3. Outcomes
How worthwhile are the outcomes of the program/ initiative? If outcomes can be observed in a community, how do we know they were because of the initiative? This is where causal chains need to be explored, and that involves the consideration of a range of indicators and metrics that would certainly not include standard KPIs.
To conduct a meaningful evaluation involves more than just measurement. Measurement can be set up as an automated system, given the right resources. Evaluators make meaningful, well informed judgements about the merit or worth of a program (among other things) based on empirical evidence, which is garnered by firstly thinking through what the key areas of inquiry should be. These high level questions are more important to the process than simply grabbing a default list of KPIs.