MaxDiff analysis: set design, sample size and scoring

MaxDiff analysis ranks a list of items by showing people small sets of them, four or five at a time, and asking which one matters most and which matters least. Also called best-worst scaling, MaxDiff analysis works best when you have a dozen or more standalone items to prioritize, can show each item to each person about three times, and can reach at least 200 respondents. Run it when a rating scale would tell you everything is "very important", and skip it when your items only make sense combined into a product with a price.

The trick is that nobody gets to say everything matters. Every screen forces a trade-off, and enough trade-offs add up to a ranking with clear gaps between items.

1. Use MaxDiff when you need a ranking, because rating scales flatten everything

Ask people to rate 20 features from 1 to 5 and you will get a lot of 4s and 5s. Bryan Orme of Sawtooth Software, which sells MaxDiff software, describes this straight-lining as a core weakness of rating scales, along with scale use bias: people use the scale differently, especially across cultures (Sawtooth Software blog, 2019). If you have written a Likert scale question and gotten back a wall of "agree", you have met this problem.

MaxDiff removes it by asking people to choose instead of rate. Sawtooth's summary of Orme's 2018 paper "How Good Is Best-Worst Scaling?" says it separates items better than rating scales or constant-sum questions and finds more statistically significant differences between groups (Sawtooth MaxDiff technical papers). Keep in mind that this evidence comes from the vendor.

The method is credited to Jordan Louviere, who developed it in 1987 at the University of Alberta, according to the introduction to his 2015 Cambridge book with Terry Flynn and A. A. J. Marley (Cambridge University Press).

2. Pick MaxDiff over conjoint when the items stand alone

MaxDiff ranks a single list of things: features, benefits, messages, pain points, brand claims. Conjoint analysis does a different job. It shows whole product profiles that mix several attributes and a price, then estimates what each attribute level is worth.

So the rule is simple. If the question is "which of these 20 things do buyers care about most?", use MaxDiff. If the question is "would buyers pay $20 more a month for priority support plus the mobile app?", use conjoint. The Louviere book reflects this split: its chapters separate the object case (ranking a list of items) from the profile and multi-profile cases, which deal with whole profiles built from attribute levels (Cambridge University Press).

A good pairing: run MaxDiff first to cut 30 candidate features to the five that matter, then put those five into a conjoint study with price, so the conjoint only has to handle a short list.

3. Show four or five items per set, and never more than half the list

Sawtooth's design manual recommends four or five items per set. It adds a cap: no more than half the items in your study, so a list of eight gets at most four per set (Sawtooth Lighthouse Studio manual). The same page says research with synthetic data found little precision gain past about five items per set.

The reason four or five works so well is arithmetic. One best pick and one worst pick from four items tell you about five of the six possible pairs. From five items, you learn about seven of ten pairs (Wikipedia: MaxDiff). Go much larger and, as the manual puts it, the small statistical gains may be offset by respondent fatigue or confusion.

Write the items as short, parallel phrases at the same level of detail. "Faster payroll runs" next to "SOC 2 Type II compliance with annual third-party audit" is a fight the long one wins on word count alone. The guidance in how to write survey questions applies to each item.

4. Show each item about three times per person

The number of sets comes from a formula. Sawtooth's manual says to use at least 3K/k sets, where K is the number of items and k is the number per set, so each item appears to each respondent about three to five times (Sawtooth Lighthouse Studio manual). That minimum assumes you will score with hierarchical Bayes (more on that below), and the software warns if any item would appear fewer than two times.

Here is a made-up example used through the rest of this post. Dana is a product manager at a payroll app for restaurants with fewer than 20 staff, and she has 20 feature ideas in a spreadsheet tab called ROADMAP_REAL. With four items per set, 3 x 20 / 4 gives 15 sets, and each feature shows up 3 times per person. Fifteen quick screens is a reasonable ask. (Sixty would be a hostage situation.)

Let the software build the sets. Sawtooth's MaxDiff calculator page describes the goals of a good design: every item shown equally often, every pair of items shown together equally often, and every item placed in each position equally often to limit order effects. That is hard to do by hand and easy to get wrong.

Short version: four items per set, 3K/k sets, and a generated design.

5. Plan for 200 respondents, and 200 per group you want to compare

Sawtooth's calculator page says most MaxDiff surveys have at least 200 respondents, and that you should plan for at least 200 in each subgroup you want to compare, such as small versus large restaurants (Sawtooth MaxDiff calculator). The same page notes that a well-built design showing each item three or more times can give stable scores at much smaller samples when you score by simple counting, though it gives no exact floor.

Long lists change the math. When a list is too long to show each item three times, Sawtooth describes sparse MaxDiff: each person sees each item once or even less, and the scores are pooled across respondents at the group level. Its rule of thumb is that each item should be seen at least 500 times by the group you care about, and preferably 1,000 times (Sawtooth Lighthouse Studio manual). If Dana's list had 60 items, 12 sets of 5 would put each item in front of each person once, so 500 respondents would give each item its 500 views.

If you sell to B2B buyers, finding 200 qualified people is the hard part. A smaller sample scored by counting still gives you a ranking. Treat it as a strong hint that tells you which five items to discuss in follow-up calls.

6. Start with counts, then use hierarchical Bayes for individual scores

There are two common ways to score the data. Counting is the simple one: for each item, take the share of times it was picked best and subtract the share of times it was picked worst (Orme, 2009). In Dana's study, a tip-pooling feature picked best in two of its three appearances and never picked worst scores 67% minus 0%, or 67.

Counting has a granularity problem. When each item appears three times to a person, only seven scores are possible, from -100 to 100, so individual results are full of ties (Orme, 2009). In the paper's test dataset (287 respondents, 15 items, 15 sets of 5), 22% of respondents had a tie for their top item.

Hierarchical Bayes, usually shortened to HB, fits a choice model to each person while borrowing information from the whole sample. Orme calls it the current gold standard for individual-level MaxDiff scores. In that same dataset, HB predicted held-out choices correctly 64.2% of the time and counting 63.0%, against 20% by chance, and the average scores from both methods lined up closely (Orme, 2009).

So use counts to read the overall ranking. Use HB when you need per-person scores, for example to segment respondents or compare groups. Sawtooth's tool reports HB results as positive scores that sum to 100, where an item scoring 10 is twice as preferred as one scoring 5 (Sawtooth Discover help).

7. Remember the scores are relative, and anchor them if you need "good enough"

MaxDiff tells you which items beat which. It cannot tell you whether any of them matter at all. Orme puts it plainly in his 2009 work on anchored scaling: standard MaxDiff gives relative scores, and a respondent has no way to say all items are important or none are (Sawtooth MaxDiff technical papers).

If Dana's 20 features could all be things restaurants shrug at, the ranking will still crown a winner. Add an anchor question when that risk is real. The dual-response approach, which Orme describes and credits to Louviere, adds a question about whether the items are important or not, which places an important/not important threshold on the scale (Sawtooth MaxDiff technical papers). In anchored results, the anchor sits at 100, so an item at 200 is twice as preferred as the threshold (Sawtooth Discover help).

The other gap is why. A MaxDiff score says tip pooling beats shift swapping. It does not say whether that is because of tax rules or one bad Saturday. The Kano model asks how people feel when a feature is missing, and a few interviews cover the rest.

8. Skip MaxDiff in three situations

Skip it when you have fewer than about eight items. A short list is easier to rank directly, and the half-the-list cap leaves you with tiny sets.

Skip it when the decision involves price or bundles of attributes. That is conjoint analysis territory, and for a quick read on price, willingness to pay methods are cheaper.

Skip it when you do not yet know what the items should be. MaxDiff ranks a list you already have. If the list came from a brainstorm, validate the feature requests behind it first, because a ranked list of the wrong ideas is still the wrong list.

9. Follow up with the people behind the top and bottom scores

Once the ranking is in, talk to a handful of respondents whose top picks surprised you. Ask what they do today without the feature and what it would replace. The score says what won, and the call says why.

If the people you want to call back are busy professionals, you can use Instant Expert to invite specific people to a short paid call or a written answer, and you pay only if they book or answer. For a second opinion on the design or the results, consumer insights experts, user research experts and product marketing experts are the people to ask.

Write your item list this week, cut it to 20, and build a 15-set design with four items per set.