Salsa Blog

5 Tips for Creating Online Surveys

creating online surveysThis was originally posted on the Whole Whale.

I was recently asked what is the best tool for creating online surveys. The question struck me, because whether it's SurveyMonkey, Wufoo, Zoho, or some custom thing – the tool doesn't really matter if you mess up the questions and execution of a survey.

As these tools have dropped to $0, the diligence behind the survey process seems to have followed. Here are five tips for anyone creating an online survey:

1. Beware sampling bias. Calculate the sample size you need based on population to make sure you have significant results within an appropriate confidence interval.

2. Create unbiased questions. Be careful about the content, perception, context, and order of questions. Don't make open ended questions – unless you like drawing conclusions that could be found in the comments section.

When people were asked whether they would “favor or oppose taking military action in Iraq to end Saddam Hussein’s rule,” 68% said they favored military action while 25% said they opposed military action. However, when asked whether they would “favor or oppose taking military action in Iraq to end Saddam Hussein’s rule even if it meant that U.S. forces might suffer thousands of casualties,” responses were dramatically different; only 43% said they favored military action while 48% said they opposed it. (From a January 2003 Pew Research survey.)

3. Test the survey on real people. Imagine that you have the results already – what conclusions could you make? What could you say to the press? This is an interesting way to re-examine your questions and something that would’ve really helped me in the past.

4. A variable incentive works and doesn’t skew the results as long as the prize doesn’t conceptually conflict with the intention of the survey. For example, giving a cash award for a survey of homeless people about how they spend money could be a bad idea.

5. Using “Average” to summarize questions is easy, tempting, and – on average – misleading. For example, if Bill Gates reads this article, I could easily say that the average reader of my blog is a millionaire.

In conclusion...

Did you hear about the statistician who had his head in an oven and his feet in a bucket of ice? When asked how he felt, he replied, “On the average, I feel just fine." (From every geeky stat professor out there.)

More resources:
Learn how to create simple surveys and questionnaires in Salsa
Learn statistics
Online survey tips from SurveyMonkey
Full survey design process
Black Swan
Thinking Fast and Slow

Topics: Strategy