FROM THE BLOG

10 common mistakes in survey design

Screen shot 2013-07-21 at 2.54.24 PM

The availability of free online survey software like Survey Monkey means that anyone can design and run a survey.  It’s great that these tools are so readily available. It means that gathering important data is easier and cost-effective. On the other hand, surveys are being designed by people with little or no experience. Here are some common mistakes I have seen in surveys in my evaluation and market research work.

Mistake #1: Leading questions

Based on their structure, questions can ‘lead’ respondents to a particular response. This is often unintentional, and is a common mistake when a survey is designed by someone who is too closely associated with the project.

Example:  Our organisation was recently recognised for its excellence in stakeholder communication. As a stakeholder, how satisfied are you with your experience with us?  [1- very dissatisfied to 5- very satisfied].

Alternative:  As a stakeholder, please rate your level of satisfaction with our organisation [1- very dissatisfied to 5- very satisfied].

Mistake #2:  Assuming prior knowledge or understanding

It is a common mistake in survey design to assume respondents know more than than they do, or have a memory as good as yours on your topic of interest.  Good surveys should leave no room for ambiguity or rely on prior knowledge.

Example 1:  Please look at our new logo. [insert image of new logo].  To what extent do you agree that our new logo is an improvement on our current logo?  [1- strongly disagree -5- strongly agree]

Alternative:  Please look at our new logo [insert image of new logo] and compare it with our existing logo [insert image of existing logo].  Please rate our new logo against our existing logo.

Example 2:  To what extent to you agree or disagree that there are enough arts and cultural activities in your local area? [1- strongly disagree- 5 strongly agree]

Alternative: To what extent to you agree or disagree that there are enough arts and cultural activities in your local area?  By ‘local area’ we mean within a 10 minute drive of your home?  [1- strongly disagree- 5 strongly agree]

Mistake #3:  Starting with demographics

I have seen many surveys start with a series of demographic questions (age, sex, income etc) but it is better to leave these questions to the end. Respondents can find these types of questions intrusive and aren’t always comfortable answering them on a whim at the beginning of a survey, and it is likely they will not complete the whole thing.  They are more likely to volunteer this type of information once they are sure the survey is legitimate and have taken the time to fill it out completely.

Mistake #4: Asking two questions in one

Two questions are often asked in one because the survey designer may not see them as mutually exclusive ideas.  But respondents may have a different view and will not know how to answer the question.

Example: How satisfied are you with the colours and style of our new website?  [1- very unsatisfied – 5- very satisfied]

Alternative: How satisfied are you with the colours of our new website:  [1- very unsatisfied- 5- very satisfied].  How satisfied are you with the style of our new website?  [1- very unsatisfied- 5- very satisfied].

Mistake #5: Inadequate response options

Without paying enough attention to detail, it is possible to ask a question of respondents, yet not give a full range of possible responses.

Example: How long have you been working in this job?   [1-2 years,  2-5 years, 5 or more years].

Alternative: How long have you been working in this job?  [Less than a year, 1-2 years,  2-5 years, 5 or more years].

Mistake #6: Grammatical inconsistencies

When you are asking a number of questions of a similar nature such as a rating scale, it is easy to overlook grammatical inconsistencies.

Example: How happy have you been with the support you have received from your manager?  [1- very unsatisfied- 5- very satisfied].

Alternative: How happy have you been with the support you have received from your manager?  [1- very unhappy- 5- very happy].

Mistake #7: Using negative question wording

Negative question wording make respondents do a double take when trying to respond.  They usually include the word ‘not’ in the question itself, then they ask respondents to agree or disagree with the position or statement. Designing your survey questions in positive language gives you one less source of error and bias to worry about, and that’s not a bad thing!

Example: Do you agree or disagree that the arts are not recognised as a valuable tool in helping to strengthen communities? [Agree/ Disagree/ Don’t know]

Alternative: Do you think the arts are recognised as a valuable tool in helping to strengthen communities?  [Yes/ No/ Don’t know]

Mistake #8: Too many open-ended questions

Very often, when you’re writing a survey, you realise there are opportunities to discover new things or you are unsure which answer options to offer.  You may think the solution is to offer open-ended questions.  You might include a ‘please explain’ after each question. Having two or three of these can be fine, but if there are more than that it becomes a turn-off.  This will result in a lot of missing data.  So, choose wisely and use open-ended questions judiciously.

Mistake #9: Asking too much of respondents

Surveys that are too long, too complex and too confusing will result in respondents getting annoyed, skipping questions or not finishing the survey.  It is not always possible to ask respondents everything you have ever wanted to know.  You need to prioritise and respect the time and effort respondents are making to answer your questionnaire.

Mistake #10: Unnecessary questions

It is not uncommon for survey designers to throw in every possible question they can think of and try to justify a reason for the question later.  The most apposite (and amusing) example of this was brought to my attention in a self-completion survey at Lincoln Castle in the UK in 2011. The survey included a number of standard questions that related to satisfaction with the visit.  The final two questions in the Lincoln Castle survey were:

What is your sexual orientation?  [bisexual, gay, heterosexual, lesbian, prefer not to say].

Is your gender the same as the gender you were assigned at birth? [yes, no, prefer not to say]

Is sexual orientation likely to be relevant to the visitor experience in Lincoln Castle?  And as for gender reassignment…?!  I  suppose it is possible that they are planning a festival or event onsite targeted at gay visitors who have had a sex change. But I suspect not.

The fact is, it’s easy these days to write a survey with lots of ill-considered questions, and send it out to a broad group of people.  It is better to consult someone like a market research or evaluation consultant to put the effort into writing a survey that will pay larger dividends when using the data.

Back to All Posts