Biased survey questions: leading, loaded and double-barreled examples...

A biased survey question nudges people toward one answer. The nudge can come from the wording ("How much did you enjoy..."), from an assumption built into the question, from two questions squeezed into one, from answer options that lean positive, or from the questions that came before it. The fix is almost always the same: remove the hint, ask one thing, and give every reasonable answer an equal chance.

This matters more than it looks. In a Pew Research Center experiment described in its guide to writing survey questions, 68% of respondents favored military action in Iraq when asked plainly, but only 43% favored it when the question added "even if it meant that U.S. forces might suffer thousands of casualties." Same topic, one added clause, a 25-point swing. Your pricing or feature survey is not immune to the same effect.

Six kinds of biased question and how to fix them

Type	Biased example	Neutral rewrite
Leading: the question suggests the answer	"How much time does our scheduling tool save you each week?"	"How, if at all, has the scheduling tool changed the time you spend on bookings?"
Loaded: the question assumes something that may be false	"What do you find most frustrating about your current billing software?"	"Do you use software for billing? (Yes / No)" then, for Yes: "What, if anything, would you change about it?"
Double-barreled: two things, one answer	"How satisfied are you with our pricing and support?"	Two questions: one about pricing, one about support
Agree-disagree statement	"Our product is easy to use. (Agree / Disagree)"	"How easy or difficult is the product to use? Very easy / Somewhat easy / Somewhat difficult / Very difficult"
Unbalanced scale	Excellent / Very good / Good / Poor	Excellent / Good / Fair / Poor, or a scale with equal positive and negative options
Pleading context	"We're aiming for five stars. How would you rate us?"	"How would you rate your experience?"

Some notes on why each fix works:

Leading questions. The Nielsen Norman Group's article on leading questions defines them as questions that include or imply the desired answer. Its suggested repair is to name the topic without naming a feeling: "How might this affect your efficiency, if at all?" instead of "How well would this save time for you?"
Agree-disagree statements. Pew notes that some respondents tend to agree with whatever statement they are shown, which it calls acquiescence bias, and recommends offering a choice between alternative statements instead. AAPOR's best practices make the same point: respondents may lean toward "yes" or "agree" to please the person asking, even unconsciously.
Double-barreled questions. Pew and NN/g both recommend splitting them. NN/g's survey best practices uses "How easy and intuitive was this website to use?" as its example: respondents either pick one word to answer or average the two, and you cannot tell which.
Unbalanced scales. NN/g points out that a scale with three positive options and two negative ones pushes people toward a positive answer. Keep the same number of options on each side.
Pleading context. NN/g's example is a question that opens with "We are committed to achieving a 5-star satisfaction rating." Anyone who had a mediocre experience now feels rude saying so. Delete the sentence.

Loaded words and one-sided framing

Word choice alone can move results. Pew reports that 51% of respondents favored "making it legal for doctors to give terminally ill patients the means to end their lives," but only 44% favored "making it legal for doctors to assist terminally ill patients in committing suicide." It also notes repeated experiments showing more support for "assistance to the poor" than for "welfare."

In business surveys the loaded words are usually quieter: "outdated," "clunky," "manual," "seamless," "AI-powered." If a word carries an opinion, replace it with a plain description. "Paper forms" is a description. "Outdated paper forms" is an argument.

AAPOR also warns against questions that present only one side of an issue. "Would you pay more for faster delivery?" presents one side. "Which matters more to you: lower delivery fees or faster delivery?" presents both.

Bias from question order

Earlier questions change how people read later ones. Pew found that 88% of people said they were dissatisfied with the way things were going in the country when that question came right after one about approval of the president, compared with 78% when it came without that context.

The business version: if you ask five questions about problems with a competitor and then ask how satisfied people are with their current tool, you have primed them to think about problems. AAPOR recommends putting general questions before specific ones on the same topic. Ask overall satisfaction first, then the specific problems.

A worked example: rewriting a dental clinic survey

This example is hypothetical. A founder building appointment-reminder software for dental clinics drafts four questions for office managers:

"How much revenue do no-shows cost your practice each month?"
"How satisfied are you with your current reminder and scheduling system?"
"Automated text reminders would reduce our no-shows. (Strongly agree to Strongly disagree)"
"Would you consider switching to a better reminder tool?"

Every question tilts. The first assumes no-shows cost meaningful revenue and asks for a number most managers have not calculated. The second is double-barreled. The third is an agree statement about the founder's own solution. The fourth describes the alternative as "better."

A neutral version:

"In the past four weeks, roughly how many scheduled appointments were missed without notice?" None / 1-5 / 6-15 / 16-30 / More than 30 / Don't know
"How do you currently remind patients about appointments?" (options from a few earlier conversations, plus "Other")
"How well does your current reminder method work for your practice?" Very well / Somewhat well / Not very well / Not at all well
"In the past year, have you looked at other ways of reminding patients?" Yes / No / Don't know

The second draft cannot tell the founder that dentists want the product. It can tell them how common missed appointments are and whether anyone is looking for a change, which is the more useful thing to know.

A ten-minute bias check before you send

Read each question and ask:

Does it contain an adjective that carries an opinion?
Does it assume something about the respondent that might be false?
Does it contain "and" or "or" joining two things to be rated?
Could someone who dislikes the product answer it comfortably?
Are there as many negative options as positive ones, plus "Don't know" or "Not applicable" where needed?
Would an earlier question change how someone reads this one?

Then do what AAPOR recommends: pretest with a few people like your respondents and ask them to explain, out loud, how they understood each question. Bias you cannot see in your own draft often becomes obvious when someone else reads it.

Wording is one source of bias; who answers is another. A perfectly neutral question sent only to your happiest customers still gives a skewed picture. Market research survey covers sampling, and types of survey questions covers which format to use for each question.

Your next step

Put your draft through the six-type table above and rewrite anything that matches. If you keep finding that the neutral version of a question is "why?", a survey may be the wrong tool; interviews vs. surveys explains how to choose. For those conversations, Instant Expert can find people who match a description you write, such as office managers at independent dental practices. You review who it finds, it sends your invitations, and you pay only for calls that get booked. The directory page for operations professionals in dental services is one place to start.