Concept testing: rank ideas on what buyers say, then test what they do
Concept testing works for comparing product ideas and finding confusion, not for forecasting sales. Write each idea the same way, then follow up with a behavior test.
Concept testing is best used to compare a few product ideas and find what confuses buyers, never to forecast sales. You show people in your target market a short description of each idea, sometimes with a sketch and a price, and measure whether they understand it, find it relevant and would consider buying it.
The trap is stated interest. Jakob Nielsen of Nielsen Norman Group puts the rule simply: watch what people actually do, put little faith in what they say they do, and even less in what they predict they will do (NN/g). So use the scores to rank ideas, then test behavior.
Concept testing sits between problem interviews and building
| Method | Question it answers |
|---|---|
| Problem interviews | Is the problem real, frequent and costly? |
| Concept test | Do buyers understand this idea, and does it appeal more than the alternatives? |
| Fake door test | Will people act on it when it appears to be available? |
| Usability test | Can people use what we designed? |
| Willingness-to-pay research | What will they pay, and how do they decide? |
If you have not confirmed the problem yet, start with problem interviews. Otherwise you are measuring reactions to a fix for a problem you have not confirmed.
A made-up example: three ideas for dental practices
Here is a made-up example used throughout. A team building software for independent dental practices already knows front-desk staff lose hours to two problems: checking patients' insurance before appointments and filling last-minute cancellations. They have three ideas:
- A. Automatic insurance eligibility checks for every patient two days before their appointment.
- B. A texting assistant that offers cancelled slots to patients who asked for an earlier time and books the first one who replies.
- C. A monthly report comparing the practice's collections with similar practices.
1. Write every concept in the same format
Use one short format for every idea, so people compare ideas instead of writing. Include:
- The problem, in the buyer's words.
- What the product does, in one or two sentences.
- How it differs from what they do today.
- A simple picture or mockup, if the idea depends on it.
- A price, if price is part of the decision.
For idea B: "When a patient cancels the day before, your front desk spends an hour calling the waiting list. This assistant texts patients who asked for an earlier appointment and books the first one who replies. It works with your current scheduling system. $149 a month per location."
Write A and C to the same length. The easiest way to rig a concept test is to describe your favorite idea more vividly than the others. (Everyone's favorite idea somehow gets the nicest adjectives.)
2. Start with interviews, then survey to compare
Concept interviews come first: a few people per buyer type, one concept at a time, with time to ask why. They show what confuses and worries people.
Concept surveys compare ideas across more people. SurveyMonkey describes three designs (SurveyMonkey):
- Monadic: each respondent sees one concept.
- Sequential monadic: each respondent rates several, one after another.
- Forced choice: each respondent sees all of them and picks a favorite.
SurveyMonkey recommends monadic designs: they mirror real life, where a buyer sees one product, keep surveys short and avoid order effects. It cites studies showing people rate the first concept they see more favorably, and a strong first concept makes later ones look worse.
The cost is sample size, since each concept needs its own group. If respondents are scarce, rotate the order in a sequential design, though SurveyMonkey warns rotation may not balance out in a small sample.
3. Ask about understanding before appeal
In interviews, ask:
- "In your own words, what does this do, and who is it for?"
- "When did you last deal with this problem? What did you do?" This ties the concept to real behavior. The Mom Test is a short book on asking questions like this without inviting polite, biased feedback.
- "What would this replace, and what would you stop doing?"
- "What would worry you about using it?"
- "Who else would need to agree before the practice paid for it?"
- "What would you need to see before you tried it?" Then ask for a small next step, such as a follow-up demo or an introduction to the practice owner.
In a survey, ask the same things about every concept: what it does (scored for accuracy), relevance, difference from today, believability, purchase likelihood and what would stop them.
4. Read the scores as a ranking, never a forecast
- Check comprehension first. If many people describe a concept wrongly, fix the description before judging appeal.
- Compare concepts on the same measures, from the same test. "Likely to buy" is a ranking. Nielsen gives a hypothetical: if half of survey respondents say they would buy more from sites with 3-D product views, all it shows is that 3-D sounds cool.
- Read the open answers for objections that repeat.
- Look at who responded most strongly. In the example, practices with three or more dentists might prefer idea B while solo practices prefer A. That tells you which segment to test next.
- Then test behavior. Give buyers a way to act on the leading idea: a waitlist, a fake door or a request for a paid pilot. Market validation covers how to ask for a commitment.
5. Avoid the four mistakes that inflate scores
- Testing with the wrong people. Screen respondents on the job they do; research screener questions covers how.
- Leading questions. "How much would this help your practice?" assumes it would. Biased survey questions has fixes.
- Asking only fans. Customers who already like you will be kind to every concept.
- Stopping at stated interest. A high score earns a behavior test. Building comes after that.
Your next step
Write each idea as a concept statement in the same format, with a price if price matters. Show them to five people who match your target buyer, one at a time, and ask each to explain the concept back to you before you ask what they think of it.
If you don't know people in the field, describe them to Instant Expert, such as "office managers at independent dental practices with two to five dentists." The explain-it-back step works as a written request: put one concept statement in your note and ask each person to describe, in their own words, what it does and who it's for. They answer in text or by voice note, and you're charged only for the answers you get. Use a 15 or 30 minute call when you want to ask follow-up questions. You set what each person receives, at least $5, and our fee is added on top: a $100 offer costs you $125. The directory pages for operations professionals in dental services and dental consultants are one place to start.