You may have seen this meme recently, doing the social media circuit. Its origin is questionable, but seems to be credited to Frank Kotsonis, an American pharmacologist, although I don’t know from when. In my work as an evaluation and research consultant, I deal with anecdotes and data. When I saw this meme recently my heart leapt with a little bit of joy, as it succinctly illustrates a common misunderstanding about the limitations of research findings.
Anecdotes are stories. In my work they are personal accounts given by research or evaluation ‘respondents’; that is, people responding to questions, surveys or evaluation activities. Respondents may be project participants, stakeholders or members of the public. Collecting anecdotes is extremely valuable in a research or evaluation project. Anecdotes bring evaluation findings to life; they build up the qualitative component of a project, that is, the aspect that provides depth of meaning to the findings. Anecdotes provide real life examples of how people have been impacted by an initiative, or perhaps how they experienced a program or service. They can be used to create personal case studies that highlight champions within programs or initiatives and demonstrate what has been possible to achieve within the program funding.
A collection of stories, in my view, provide anecdotal evidence. They do not provide data. ‘Data’ is most commonly considered to refer to facts such as values, statistics or measurements gathered together for reference or analysis. By definition, this may include a number of anecdotes, but the value in this is limited by sample sizes as well as by the range of qualitative nuances contained within multiple anecdotes. These two limitations make it troublesome to present anecdotes as scientific data.
SAMPLE SIZES
Small sample sizes are inherently problematic when looking for statistical evidence. When funding bodies commission evaluation projects, they are ideally looking for evidence in the form of numbers and facts to help them make important decisions that may include whether to continue or discontinue a program or initiative. For this reason it can be alluring to look for ways to assign percentages, graphs and bar charts to evaluation findings. But when we’re talking about qualitative research this does not prove anything; and in fact, it is misleading.
Sometimes sample sizes must be small. There may be a limited number of program stakeholders and participants, or the focus of the evaluation may lie within a small local community. When population surveys are not necessary or relevant, it becomes important to consult with as many stakeholders and program participants as possible to feed into the evaluation. But these findings are qualitative rather than quantitative. They should not be considered to be data. And further, when sample sizes are small, if the anecdotes are to be tallied, they should be presented in raw numbers. This still does not make them data, in my view.
QUALITATIVE NUANCES
One pertinent anecdote in an evaluation project can be strong. More than one can start to provide powerful and compelling evidence to support a finding. But even if a collection of anecdotes and stories convey a consistent theme and seem to provide strong evaluation evidence, this does not mean they can legitimately presented as data. Rather, the compelling evidence should be presented as a compelling narrative, with insightful quotes where appropriate.
Conclusion
Organisations collect lots of data. But much of that data isn’t particularly useful. It is more important that decisions are based on convincing evidence. A collection of well-sourced and analysed anecdotes can be more useful than irrelevant data. Despite the allure of statistics and percentages, a group of anecdotes can play an important role in contributing to knowledge and evidence. An anecdote is a form of qualitative evidence, to be given its due weight either alone or combined with other forms of evidence. But a group of anecdotes contribute to the literature of the evaluation, rather than the mathematics.
I think it’s more useful––and arguably obvious–– to consider that the plural of anecdote is not data; the plural of anecdote is……anecdotes.
Image: Nestler Analytics