Conducting an evaluation of a program or initiative is crucial to ensure its success. Although it can feel like a burden to go through the tender process and choose a program evaluation consultant, there are a number of benefits to the process of evaluation, including identifying strengths and weaknesses of the program, creating a foundation for strategic planning, allowing you to demonstrate program outcomes to grant providers and other constituents, and producing credibility and vision.
It may be tempting to consider conducting a program evaluation in-house by your internal staff. There are some advantages to internal program evaluation. It can be the less expensive option. Additionally, internal staff are likely to have a more detailed knowledge of the program or initiative. Their close involvement in the program can help to build capacity of your organisation to work with evaluations in the future.
However, there can be some false economies of scale here based on a number of disadvantages to conducting an evaluation in-house. Internal staff are unlikely to have the skills or experience in program evaluation. Because they are so closely involved in their work, they cannot be objective and it can be difficult to identify any true problems or challenges. Funders of your program are unlikely to view the evaluation as objective enough for the same reasons and the validity of the evaluation findings may not be highly regarded by those outside the program.
An external evaluator will be objective and look at the program dispassionately. There may be some issues uncovered in the process of the evaluation you may not want to hear, but for the future success of the initiative it is important that you know. An external evaluator brings technical expertise to the process and will be more efficient in getting the evaluation done. They will bring fresh insights and a fresh perspective to your program that can only benefit you, your organisation, and the future of your initiative. The process can be expensive, depending on the size and complexity of your program, how you intend to use the findings, and the range of program objectives you wish to measure.
Choosing the right program evaluator requires careful consideration of a number of things. The process itself may come about by putting a brief out to open tender, invited tender, or you may take word of mouth recommendations. When considering who to commission for your program evaluation, you should look for a number of qualities and skills in the consultant. Apart from the standard skill set that good consultants required such as solid project management skills and relevant experience, the skills of a good program evaluator include these things.
Ability to choose the right approach
A good evaluation consultant should be aware of the range of evaluation approaches and choose the one that is best suited to your objectives. For example, a Formative Evaluation is usually undertaken during the implementation of a program to gain further insight and contribute to its learning process. A Process evaluation documents the process of the establishment of the program or initiative and monitors how it has been implemented. It focuses on aspects relating to efficiency of the program and the quality of its activities and services. A Summative evaluation is usually conducted when the initiative has been in place for some time (such as after the pilot period) and is designed to examine its effectiveness and overall value. Each of these three types of evaluation are carried out in subtly different ways, and a good consultant will know which is the most suitable.
Selecting the right evaluation tools
Although you know what you want to evaluate and hopefully why, a good evaluator must decide which evaluation method(s) to use. For example, how do you measure attitudes towards the environment? Is a questionnaire appropriate? Might personal in-depth interviews be better? What other methods could be used? A good evaluator must have a toolbox from which are able to draw a range of research tools, such as questionnaires, interviews, focus groups, and other more cutting edge techniques, and be able to justify which tools should be used and why.
Understanding your sector
The evaluator needs to demonstrate a solid appreciation and understanding of your sector and the likely contextual considerations that need to be taken into account in the evaluation process. For example, it is likely that busy business owners will not have the time to come to focus groups or meetings. There are a range of sensitivities that must be employed when consulting with people with disabilities or mental illness, as well as people recovering from natural disasters. It is not necessary for the consultant to necessarily have conducted evaluations in your sector before, as long as they are aware of the range of possible sensitivities, practical considerations and contextual influences that are likely to arise.
Finding clarity and meaning
Evaluation differs from research in the way that findings are turned into meaningful implications or recommendations. It is important to try to select an evaluation consultant who can demonstrate they are able to make sense of complex data and detailed information and can disseminate it in a way that is relevant to your needs. The information that has been collected through the research should be presented to directly respond to the range of stated program objectives and be used to justify the extent to which the program has met its intended objectives.
As well as having the full range of solid consultant skills, combined with the evaluation qualities discussed, it will be important that you feel you will be able to work well with your evaluator. This individual or individuals will be spending time talking with you and your staff, probing into the ins and outs of a program that you know well. There are a number of things that you––the client––can do to make the process run more smoothly, and this will be discussed another time.
Ultimately, if your program, initiative or service is worth doing, it is worth understanding fully. The more consideration you can give to what you want to achieve from your evaluation, and what you intend to do with the findings, the easier it will be to select a consultant who is most suitable for the job.