At this beginning of a new year it is possible you are wondering whether your programs and initiatives are providing a good return on investment, or how your budget may be better spent. In the not-for-profit and government sectors throughout Australia, value for money and how to assess it has become important. Determining whether programs or activities provide value for money has become a policy imperative and is often an important objective of a program evaluation.
The concept of value for money is sometimes debatable. Most commonly, it includes an assessment of the cost of running the program, its efficiency (the outputs it achieves for its inputs) and its effectiveness (the extent to which it has achieved programs outcomes).
A recent BetterEvaluation paper written by Farida Fleming identifies six main methods that can be used to evaluate value for money. Each approach has benefits and limitations and examines the relationship between costs and benefits in a particular way. The full paper can be seen here.
1. Cost Effectiveness Analysis (CE Analysis). This approach involves the evaluation of two or more alternatives, based on the relative costs and outcomes (effects), in reaching a particular goal. It can be used when comparing programs that aim to achieve the same goal. It cannot, however, compare alternatives with different goals, nor make an overall assessment of whether a program is worthwhile in an absolute sense.
2. Cost Utility Analysis (CU Analysis). This type of evaluation takes two or more alternatives and compares their costs to their value. It is most commonly used when the evaluator needs to consider individual preferences. A CU analysis can result in a large number of potential outcomes. However, results are often difficult to reproduce among different evaluators because of the sometimes conflicting methodologies used to estimate importance weights.
3. Cost Benefit Analysis. A cost benefit analysis is an evaluation of alternatives by identifying the cost and benefits of each alternative in money terms, and adjusting for time. This method can be used to identify if a course of action is worthwhile in an absolute sense; that is, whether the costs outweigh the benefits. It is best used when the majority of benefits can be converted to monetary values or when those that cannot be converted are unimportant or are similar among the alternatives considered. Conducting a cost benefit analysis requires considerable specialist skills.
4. Social Return on Investment (SROI). This approach involves measuring social, environment, and economic costs and benefits. Like Cost Benefit Analysis, SROI can be used when comparing programs with different goals or in different sectors. A limitation of this method is that cost data can be disputed as different evaluators may use conflicting methodologies to derive value. Data collection and analysis is very time consuming, so it is best done as a discrete evaluation approach rather than within a mixed method approach.
5. Rank correlation of cost vs impact. This approach allows for the relative measurement of value for money across a portfolio of initiatives. This method can help determine a comparison between alternatives with different objectives. It can be useful to multi-unit programs and can contribute to participatory decision-making as stakeholders are called upon to identify and value program outcomes.
6. Basic Efficiency Resource Analysis (BER analysis). A BER analysis provides a framework for evaluating complex programs by comparing impact to resources and offering a relative perspective on performance where units analysed are judged in comparison to other peer units. This approach simplifies complex information and should not be relied on alone. It should be used in conjunction with other data and never as the only analytical approach.
An evaluation may employ one or more of these methods, or even a combination of some aspects of each approach. It is important for you to have a conversation with your evaluator about what you hope the evaluation may achieve, what you hope it may uncover and what you intend to do with the findings. Only then can an evaluator choose the best approach to the evaluation to help you understand whether your program is delivering value for money.
Regardless of which method an evaluator chooses for assessing value for money, it is important that they ask themselves and their client the following questions:
– How will value be measured?
-Will it include cost, efficiency and effectiveness?
-Will it include equity?
-Who will decide how to define value?
– Is the evaluation assessing the value of a stand alone program or comparing a number of projects?
– Will the evaluation measure in monetary terms or will is use a proxy measure of value?
– How will the evaluation process make sure costs and benefits are agreed and transparent?
– Will these methods be used in ways that enable the participation of stakeholders and the community in the evaluation process?
Conclusion
There are benefits and limitations to each of the methods of assessing value for money. It is important for the evaluator to consider a range of options for determining whether an activity is value for money. They should think through the benefits and limitations of each approach and ensure key points are included in the design of the evaluation that may be important. It may be necessary for the evaluator to source input from other specialist consultants in particularly complex evaluations of this nature.
This main content of this blog summarises key points made in a paper published by BetterEvaluation developed by a Working Group convened through the Australasian Evaluation Society.