Open-ended survey questions: examples, costs, and when an interview is better

An open-ended survey question lets people answer in their own words instead of picking from a list. Use it when you cannot predict the answers, when you want the words customers use, or to catch the issue your closed questions missed. Use few of them: they are skipped far more often than multiple-choice questions and every answer has to be read. If most of what you want to learn needs a follow-up question, run interviews instead.

What open-ended questions are good at

The Nielsen Norman Group's article on open-ended vs. closed questions puts the main benefit simply: open-ended questions let you find more than you anticipated. Closed questions return only the options you thought to offer.

Pew Research Center's guide to writing survey questions shows how large that gap can be. After the 2008 U.S. presidential election, 58% of respondents picked the economy as their most important issue when it appeared on a list, but only 35% volunteered it when asked the open-ended version. And 43% of open-ended answers named something that was not on the closed list at all. The list shaped what people said.

Open-ended survey question examples

For customer and market surveys, the useful ones are specific and point at a recent event:

• "What were you using before you started using us?"
• "What almost stopped you from signing up?"
• "Describe the last time an order from a supplier went wrong. What happened?"
• "What is one thing you would change about how your team tracks shipments today?"
• "What else did you consider before choosing your current tool?"
• "Why did you choose that rating?" (after a key rating question)
• "Is there anything else you'd like to tell us?" (at the end)

And the ones that rarely work in a survey:

• "What features would you like to see?" People describe solutions, not problems, and without a follow-up you cannot ask what they would give up for them.
• "What do you think of our product?" Too broad; answers range from one word to an essay and are hard to compare.
• "How could we improve?" Answered mainly by people who already had something to say.

The costs: skipped answers and reading time

Pew's analysis of nonresponse on open-ended questions in its American Trends Panel found that the average nonresponse rate for closed-ended questions was between 1% and 2%, while open-ended questions averaged around 18%, ranging from 3% to just over 50% depending on the question. Questions asking for several sentences had a higher median nonresponse rate (18%) than those asking for a word or phrase (13%). The strongest factor was how much thought the question required. Nonresponse was also consistently higher on mobile devices.

NN/g adds that a survey with many open-ended questions will usually have a lower completion rate, and AAPOR's best practices recommend limiting the number of difficult or open-ended questions to keep respondent burden low. AAPOR also notes that open-ended answers must be coded before you can count them.

A hypothetical illustration of what that means for a small team: you send a survey to logistics coordinators and get 300 responses. If your skip rate matched Pew's panel average of about 18%, you would have roughly 300 × 0.82 = 246 written answers per open-ended question. Three open-ended questions means around 740 answers to read and tag. That is a real afternoon or more, and it should be planned before you add the third text box.

How to write open-ended questions people answer

Based on the Pew findings above:

• Ask for a short answer when a short answer will do. "In a word or two, what did you use before us?" is easier than "Describe your previous workflow."
• Lower the effort. Point at something the person already knows, like a recent event, rather than asking them to form a new opinion on the spot.
• Make it clear what kind of answer you want. Pew says it should be evident to respondents that they can answer in their own words and what type of answer to give: a problem, a tool name, a number of days.
• Place them carefully. Pew generally asks open-ended questions about topics such as national problems near the beginning of a questionnaire, because closed-ended questions on the same topic placed before them make respondents more likely to repeat those concepts. NN/g recommends one broad optional question at the end for anything the survey missed.
• Make them optional. A forced text box gets filled with "n/a" or causes people to quit.

How to analyze the answers

1. Read 30 to 50 answers before creating any categories.
2. Write a short list of categories in plain words, including "Other."
3. Tag every answer; one answer can have more than one tag.
4. Count the tags, and keep two or three representative quotes for each.
5. Check that someone else would tag a few answers the same way.

The counts from this process are a rough guide, not a precise measure. People who bother to write are not a random subset of respondents.

When a 30-minute interview beats a free-text box

A text box gets one answer. An interview gets the answer and the next five questions. NN/g notes that if your questionnaire relies heavily on open-ended questions, that may be a sign an interview would serve your aims better.

Choose interviews when:

• You need the story. "What went wrong with the last shipment?" in a survey gets "carrier was late." In a call you can ask what happened next, who noticed, what it cost and what they changed.
• You need to follow up on surprises. A survey cannot ask "What do you mean by that?"
• You need to understand a decision. Who was involved, what alternatives were considered, what nearly stopped it. That rarely fits in a text box.
• Your audience is small. If you can reach only 40 people in the role, eight conversations may teach you more than 40 short text answers.

A common sequence works well: a few interviews to learn the words and options, a survey with mostly closed questions to count them, then a few follow-up calls with people whose answers surprised you. Qualitative research interview questions covers how to write the guide for those calls, and types of survey questions covers the closed formats.

Your next step

Look at each open-ended question in your draft and decide which of three it is: a short answer you need (keep it, make it easy), a catch-all at the end (keep one), or a question that needs a conversation (move it to interviews).

For the interviews, Instant Expert can find people who match a description you write, such as operations managers at freight brokerages. You review who it finds, it sends your invitations, and you pay only for calls that get booked. The directory page for operations professionals in logistics technology shows the kind of people you can reach.