To conduct a program evaluation is fundamentally about exploring the extent to which the initiative meets its intended objectives.
We are currently evaluating an arts-based program Creative Capricorn being rolled out in Rockhampton over a period of three years. It involves the integration of artistic and cultural programs to help diversify the local economy and to increase social cohesion and quality of life. We are using a range of research and evaluation methods including surveys, in-depth interviews, observations and data analysis. We are also using social media monitoring. It is the first time we have used social media monitoring in an evaluation project. It has been a steep learning curve, there have been frustrations along the way, but ultimately, a lot to be gained for this and future projects. But what is social media monitoring and why is it useful for this evaluation project?
In basic terms, social media monitoring is the act of using a tool to monitor what is being said on the internet. One of the important measures of success of Creative Capricorn is the extent to which it engages people in thinking, participating and attending arts and cultural activities in the region. These days people use social media and the internet to exchange ideas, share information, broadcast events and activities, write blogs, write reviews, give feedback, and create interest and excitement.
We felt it would be valuable to monitor social media activity to track the extent to which Creative Capricorn generated interest in the arts over time, what was being said about the arts in the region, who was saying it, who they talked to and influenced. To answer these questions, we needed to carry out both quantitative and qualitative analysis. Specially, we are measuring:
- The numbers of relevant online and social media mentions (quantitative analysis);
- Types of people following or getting involved in conversations (qualitative analysis); and
- The numbers of people the followers are influencing, that is, the broadcast reach of the program (quantitative analysis).
The social media monitoring market is a crowded one and reviews are emerging all the time by people assessing the merits of each one. There are a number of tools available; some are paid, some are free of charge. New tools are appearing all the time, and others die out. In order to find the most suitable tool for this project has been no easy task. One of the main things we have discovered is that there is no one magic bullet; not one tool can do everything you need. To monitor social media in a meaningful way takes some research, time and skills in meaningful analysis.
Some of the questions and challenges we have encountered in the process have included the following:
1. How do you weed out irrelevant information?
Most tools work by using keywords or writing a query to find the mentions they are interested in. It could be the name of a brand –– in our case Creative Capricorn –– or more general keywords such as Rockhampton arts. This means that as well as searching for specific terms and words, the tools search for mentions that relate in any way. To use a keyword ‘Creative Capricorn’ means you have to be sure you don’t receive findings that relate to horoscopes! It is important to know how to do accurate keyword searches to eliminate completely irrelevant data. Then, there will always be mentions that are still not entirely relevant. It is important to read through results to determine their relevance and eliminate irrelevant results. This is time consuming and is not dissimilar to how qualitative researchers sift through lists of verbatim responses looking for patterns and themes. This is an important part of the process of good social media monitoring.
2. If Google says there are thousands or millions of results, why do the social media tools find less?
Google searches do not take into account spam, duplicates and date ranges. You want your social media tools to give you quality data, so this is another reason why it is important to spend time analysing the results and determining their relevance.
3. What do you do with the data?
Some social media monitoring tools give you an option of pressing a button to export the results into an Excel spreadsheet. This is a very useful first step in order to be able to analyse and sort the results. You can then sort them by source (eg. Facebook, Twitter, blog, Google, YouTube etc), by date, or by topic. If you have enough content it will be possible to analyse the data by topic or sentiment.
4. How can you be sure you’re trawling everything?
Most social media monitoring tools crawl all sorts of websites, including forums, blogs, Facebook, Twitter, Google+, YouTube and so on. Coverage varies between tools and regions, some tools have strict rules that make it impossible to cover all of the content online. It is not possible to be sure you’re trawling everything, but the best tools trawl most reliable sources. It is likely that in the process of social media monitoring some online mentions will be missed. That is the nature of technology and the limitations of humans. But by setting up these tools and scheduling in a regular time to monitor them you will be sure to pick up most online mentions on your topic, and you will know that you won’t miss the most influential ones.
5. Other sources of information
Some social media monitoring tools pick up online content in Twitter, Facebook, Pinterest etc. but others do not. It can be just as efficient to monitor Facebook and Twitter feeds manually and conduct more detailed analyses on those platforms, particularly if they are the main social media platforms that are being used in your project. In our case we are using a social media monitoring tool, as well as manually monitoring Facebook and Twitter so that we can be sure to have picked up as much as we can.
6. The benefits of using Twitter hashtags #
Twitter hashtags act as keywords; they can be created by anyone and when used in Tweets they are ways of grouping information by topics. As Twitter followers see these being used they are encouraged to pick them up and use them too. For example, if a hashtag such as #Rockyarts starts to be used widely, it will make it possible to search for Tweets on this topic that don’t necessarily talk directly about Creative Capricorn, but are nevertheless relevant to the study. I will be encouraging the program team to set up hashtags and use them to issue messages. This is likely to influence others in their sphere of influence to use them too, which will ultimately increase the broadcast reach as well as make the process of social media monitoring more efficient.
Social media monitoring is a technique that can add considerable value to an evaluation project. In order for it to be most effective it is important to work closely with your client to put in place social media strategies. The monitoring and charting of findings should be conducted systematically and regularly. Then, the real value is added in the process of making sense of the results and finding real meaning that can enable to you answer the following question: Has this program or initiative contributed to increased online conversations about the topic over time, and if so, to what extent and in what way?