No one needs reminding we’re entering challenging financial times and need to practice social distancing. It’s possible that your evaluation plans may need to be pared back or reconfigured. But there are ways that your evaluation project can still be carried out without compromising its validity.
1. Modify the evaluation design
Ask your evaluator if they can use a simpler evaluation design.
It will be important to think really carefully about what you need to know, versus what you want to know. This is good practice whether budgets are tight or not and evaluations will always be stronger as a result. If you work with your evaluator to identify and prioritise the critical issues that must be addressed, the evaluator will be able to integrate all of their evaluation tools to focus on the critical questions.
2. Clarify client information needs
Again, good practice at all times, but being clear about what you need to know is particularly important when finances are tight and work needs to be conducted remotely. You may be able to significantly reduce your evaluation costs by being crystal clear about the kinds of decisions you would make with the evaluation findings and how you expect the evaluation to contribute to those decisions.
3. Use existing data
Also called ‘secondary data’, existing data can be identified in a desktop research to reduce the need for as much data collection. This also reduces the need for face to face consultation. Typical examples of secondary data include findings from similar studies, census or survey data covering the project and comparison communities, data from project monitoring, records, newspapers and other mass media, records from community organisations and dissertations from academic studies.
4. Reduce sample size
Even in the best of times, sample sizes should always be appropriately related to the purpose of the evaluation. There are often misconceptions about how large a sample size needs to be. But whether you are using qualitative or quantitative research in your evaluation, mindful sampling is often more important than the size of the sample. In times of tight budgets and the need to minimise face to face contact, purposeful sampling of stakeholders can be useful.
5. Reduce costs of data collection and analysis
Reducing the length and complexity of the data collection instrument can keekp costs down and minimise face to face consultation hours. This relates to the earlier points about being very clear about the purpose of your evaluation and differentiation between what you need to know and what you want to know.
Here are some ideas:
- Reduce the range of interview topics to those of the greatest priority;
- Scale back the number of interviews;
- Cut down on the number and types of documents to be analysed; and
- Consider using self-completion diary methods for respondents to record information over a period of time.
Things to keep in mind
When considering any of these methods to keep your evaluation costs down and minise face-to-face consultation, it is important that you avoid posing a threat to the validity of your evaluation, particularly if you intend to make important decisions as a result of the findings.
For example, you need to ensure that your data remains rich and meaningul, that triangulation is used by adopting complementary methods of data collection to produce converging conclusions. You need to use enough data collection methods to answer evaluation questions, so don’t just rely on stakeholder interviews. Be sure to include program users, internal staff, and other key stakeholders. All of these people can be consulted remotely if necessary.
These things relate to ensuring the reliability, dependability, statistical validity and external validity of your evaluation.
Conclusion
Even with budget constraints and the need for social distancing, it can be possible to commission a useful and robust evaluation without threats to its validity. The key to this is spending time with your evaluator in preliminary discussions to design your evaluation. This is good practice in any situation, but is particularly in the unprecedented times we find ourselves in.
Reference
Bamberger, M., Rugh, J., Mary, L. (2012). Real World Evaluation. Working Under Budget, Time, Data and Political Constraints (2nd edition). SAGE.