« The woman in red says: Break the rules | Main | Hey, Mr. Paulson! What does this mean to me? »
September 17, 2008
Survey fatigue
Let’s start with the good news: The International Association of Business Communicators (IABC) is conducting a member survey about the organization’s print publication, Communication World, and its electronic newsletter, CW Bulletin.
Feedback is a good thing.
Here’s the bad news: The survey is so long, so extensive, so open-ended, that it’s exhausting to complete.
It contains 29 questions, a lot for an online survey. Even worse, 16 of those questions are open-ended (also known as write-in) questions. And they’re not the usual run-of-the-mill “do-you-have-any-suggestions?” questions; these require deep thought. Here are just a few examples:
“Please list up to five issues or trends in branding and marketing that you would like us to address in CW.” (There are actually six questions just like this, asking for topic ideas in employee communication, PR, measurement, change communication and skills development.)
“What do you like best about CW?”
“What other industry-related e-newsletters do you subscribe to?
“What kinds of products or services would you like to see advertised in CW or CW Bulletin?”
Whew! Only the most dedicated reader (or member) would take the time to answer every question. Most people would either do what I did—answered just a few questions and skipped the others—or get to a certain point and jump ship. In research parlance, that’s called “noncompletion.”
What’s the root cause of this problem? There are at least two: First, what the survey creators wanted to do was explore issues in an open-ended ways. But the wrong tool for this job is a survey, which is a quantitative, closed-ended, data-producing instrument. The right method is qualitative, either focus groups or interviews.
Second, the creators didn’t consider the experience of survey respondents. If the creators had tested the survey, they would have realized that completing it would take at least 15 and probably up to 30 minutes—waaaayyy too long for an online survey.
Before you are tempted to include open-ended questions in your next survey, consider the fatigue factor.
Posted by Alison Davis at September 17, 2008 11:48 AM
Comments
Your points are well taken, Alison. Your experience speaks volumes, and we appreciate your advice. I thought it might be helpful to give a bit of background on our thinking in developing the survey. This might help others who are wrestling with the same issues.
We certainly could have (and still can) include a focus group component to our research. And you are quite right: Without the open-ended questions, the survey would have been much shorter. But here’s the dilemma. One of our biggest challenges (and biggest opportunities) is the diversity of our audience: IABC members are located all over the globe, and have a wide range of experience levels and job responsibilities. It's pretty tough to get a truly representative group to meet -- even online. And we wanted to give everyone an equal opportunity to comment.
I'm sure you noticed that none of the questions were mandatory. We wanted to make sure that people could skip questions and focus only on those that they felt were most important to them. And for the most part, that’s what they did.
We send this survey out every other year, and we typically include only a few open-ended questions. However, we needed to get a deeper level of input this year. We just weren’t getting the richness of data that we felt we needed from our audience to help us continue to improve CW.
We haven’t closed the survey yet, but I can tell you that the response rate has been high and the comments are incredibly helpful – just what we’d hoped to get.
Thanks again for your suggestions. We’ll be thinking about your advice when we send the next CW survey out -- in 2010.
Posted by: Natasha Nicholson at September 17, 2008 07:50 PM