As we emerge from the pandemic and start to evaluate COVID- response programs and initiatives, it’s likely that you may need to collect data in the form of online surveys. Online survey software is available to everyone at little or no cost. But if your survey is not well designed, the internal validity (accuracy) of your data may be compromised, which can impact on the credibility of your findings.
Here are 5 top tips for good survey design:
Tip #1. Decide how you will use the information
It’s a common misconception to think it’s a good idea to pack as much into a survey as you can. It won’t cost you anything, so why not ask respondents everything you’ve always wanted to know? If your surveys is longer than about 12 minutes, respondents may feel drained and exploited; with many not completing the survey. When you think your survey is ready, scrutinise every question.
If you don’t know what you intend to do with the findings to each and every question, drop those questions from the survey.
Tip #2. Resist asking two questions in one
Two questions are asked as one when you conflate two ideas that are not mutually exclusive. When you look at the results, it will be impossible to know which aspect of the questions respondents are referring to. For example:
“How satisfied were you with the speed and accuracy of the delivery of our COVID response?”
Speed and accuracy are different concepts; some respondents may be satisfied with the speed, others just with accuracy. So, instead, ask two questions: one about speed, the other dealing with accuracy.
Tip #3. Avoid leading questions
Questions are considered leading if respondents feel they are being guided to a particular response. It may be unintentional and can come about from you being too close to the subject matter of your survey. For example:
“How important do you think it is for our important COVID-response program to continue to support and arts and cultural sector?”
Here, respondents are being subtly manipulated into feeling a positive disposition towards the initiative, which they may not believe. It’s important to remain neutral and avoid using persuasive language.
Tip #4: Leave demographics till the end
I believe that demographic questions (age, sex, place or residence etc) should be asked at the end of a survey.
Compare it to the etiquette of meeting someone in person. Many of us know how inappropriate it can be when you meet someone who begins with ‘Nice to meet you. Are you married? Do you have children?’ (I’ve experienced this!) I think the same thing applies to a survey.
Tip #5. Avoid too many open-ended questions
This tip is intended to help respondents but also you, the researcher, who will be analysing the findings!
When a topic is complex, open ended questions provide respondents with the freedom to explain things in depth. But too many open ended questions in a survey can be a burden to those filling it in. In effect, they’re being asked to write multiple short essays.
But open ended questions are often used by survey designers as a default question type. Rather than provide a drop down menu with options (including ‘other’ for those you may not have anticipated), a lazy survey designer can simply use an open field and ask respondents to use their own words in response to a question. Although easier to set up, open ended questions take considerably more time and effort to analyse, particularly with a large sample size. You need to pull out themes and patterns, then code and count. Instead, consider thinking through a range of likely responses and provide them in statements or options, adding the option ‘other’ with an open field. This will save you a lot of time at the other end.
So, use open ended questions sparingly. I recommend no more than two or three per survey.
As we all get busy researching and evaluating our COVID response, it can be tempting to design online surveys without too much thought. If you embark on it yourself, try to avoid these common mistakes to give yourself a good start. Otherwise, get in touch if you’d rather outsource the whole process. I would work with my colleague Pete Wilson at Pragmatic Research to help you get it right and save you time and angst.