Price testing: survey first, A/B test last (and only with the traffic for it)

Price testing means showing buyers different prices and measuring what they actually do, and for most small companies the right order is a survey first, then a fake door test for self-serve products and a quote test for sales-led ones, and a randomized A/B price test only once you have thousands of buyers per price. Most price testing advice assumes an online store with heavy traffic. If you sell to a few hundred businesses a year, a checkout A/B test will take longer than your runway and still not give you a clear answer.

This post walks through the methods from cheapest to riskiest, says how many buyers each one needs, and covers the fairness problem that comes with charging two people different prices for the same thing.

1. Pick the method by how many buyers you see each month

The volume of real buying decisions you can observe decides which method can work. Price2Spy splits price testing into two families: experiments that change real prices, and surveys that ask people about prices without changing anything they pay. Here is how the main options compare.

Treat the volume column as a rough starting point, not a benchmark. Only the Gabor-Granger figure comes from a source (SurveyMonkey, below), and the A/B figure is worked out in section 5. The surveys sit at the top because they are cheap and nobody gets charged the wrong price. The A/B test sits at the bottom because it is the only one where a real customer pays more than their neighbor.

2. Start with a survey, because it costs you nothing but a few questions

Run a short pricing survey before you touch a live price. It narrows the range you test later, and it tells you whether your offer even makes sense to buyers at the prices you have in mind.

The Van Westendorp price sensitivity meter asks four questions about when a price feels too cheap, cheap, expensive and too expensive, and gives you a band of acceptable prices. The Gabor-Granger method, developed in the 1960s by economists Andre Gabor and Clive Granger according to SurveyMonkey, shows one price at a time and asks whether the person would buy at that price.

SurveyMonkey's guide recommends five evenly spaced prices, from one most people would accept to one most would reject, with each price on its own screen. You then multiply each price by the share who would buy at it to get a revenue index, and the peak of that curve is your candidate price. The same guide suggests at least 100 completed answers per segment you want to report on, and warns that the method works poorly for genuinely new categories, since people have no reference point.

Treat every survey number as a ceiling. People are generous with money they are not spending, which is why willingness to pay answers run high. A survey tells you where to aim, not where to land.

3. Use a fake door price page to see who clicks at each price

A fake door test puts a real-looking priced offer in front of visitors and counts who tries to buy before the product or plan exists. For price testing, you show different visitors the same plan at different prices and compare how many click "Buy" or "Start trial." Our guide to the fake door test covers the setup.

The click is the signal, so stop there. Do not take a card number for a price you cannot honor. Show a short page that says the plan is not ready yet, and offer to tell them when it is.

This method suits self-serve products with steady traffic. It does not fit enterprise sales, where nobody clicks "Buy" on a $60,000 contract (anyone who did deserves a phone call first).

4. In sales-led B2B, test prices in the quote, one batch at a time

If your deals close through conversations, the quote is your price test. Change the number you put in proposals for a batch of new prospects, keep everything else the same, and compare how those deals go.

Here is a made-up example. Priya sells inventory software to craft breweries. Her current quote is $400 a month. She quotes the next 15 qualified breweries at $550 a month, using the same demo, the same proposal template and the same rep. She tracks three things for each deal: whether it closed, whether the buyer asked for a discount, and how many days it took. If roughly as many deals close at $550, she has learned more than any survey could tell her.

Write the test price into the proposal template before the batch starts. Reps who believe the new price is too high will quietly discount it back down, and then you are testing your rep's nerve instead of the price.

The tradeoff is speed. Fifteen quotes might take a quarter, and 15 deals will only show you a large difference, never a subtle one. Use quote tests to find out whether a price is clearly too low, which is the more common mistake.

5. Run a randomized A/B price test only if you can reach the sample size

A true A/B price test randomly splits visitors and shows each group a different price at checkout. It gives the cleanest answer, and it needs far more buyers than most small companies have.

Evan Miller gives a rule of thumb for the sample size of each group: 16 times the variance divided by the square of the smallest difference you want to detect. For a conversion rate p, the variance is p times (1 minus p).

Here is a made-up example. Your checkout converts 3.0% of visitors at $40. You want to try $50. At $50, the price only breaks even on revenue per visitor if conversion stays above 2.4%, because $40 times 3.0% and $50 times 2.4% both come to $1.20 per visitor. So you need to detect a drop of 0.6 percentage points. The rule gives 16 times 0.03 times 0.97, divided by 0.006 squared, which is about 12,900 visitors per price. With 2,000 checkout visitors a month, that is about 13 months for two prices.

Fix the sample size before you start and do not stop early when the result looks good. Miller shows that checking results repeatedly and stopping at the first significant reading inflates false positives: peeking ten times at a reported 1% level gives you a true 5% level. Price2Spy adds two more limits: demand can shift within days for reasons unrelated to your test, and an A/B test only tells you which of two prices did better, not whether either one is the best price.

If you have under a few thousand checkout visitors a month, skip this method. Go back to quote tests and cohort tests, which accept a noisier answer in exchange for finishing this year.

6. Decide before launch what you will do for people who paid more

The biggest risk in a live price test is a customer finding out they paid more than someone else. Plan for it in advance, because they usually do find out.

The well-known case is Amazon in September 2000. According to Amazon's own statement, it ran a random price test on 68 DVD titles over five days, with discounts from 20% to 40%. After customers noticed, it refunded 6,896 customers an average of $3.10 each, and said future test buyers would automatically get the lowest test price when the test ended.

Adopt that rule from day one: when the test ends, everyone who bought gets the lowest price shown. It makes the test cheaper to defend and costs you only the gap between prices for the test period.

There is also a legal angle. The Robinson-Patman Act, as explained by the FTC, bars sellers from charging competing buyers different prices for the same commodity when that can harm competition. It covers goods, not services, and price differences can be justified by real cost differences or by meeting a competitor's price. Our reading of that page, which is not legal advice, is that a software company testing plan prices is probably outside the Act, while a company selling physical goods to competing retailers faces more risk. If you sell goods to resellers who compete with each other, ask a lawyer before you test wholesale prices on them.

7. Prefer a cohort test to a geo test when you change a live price

A cohort test changes the price for new customers from a set date and compares them with the customers who signed up just before. Existing customers keep their old price. Nobody sees two prices at the same moment, which removes most of the fairness problem.

The cost is noise. A cohort that starts in December is a different crowd from one that started in October, and Price2Spy's point about shifting demand applies in full. Run the new price for at least as long as your normal sales cycle, and compare the same weeks from last year if you have them.

A geo test charges different prices in different countries or regions at the same time. It works when the regions really are separate markets, such as different currencies and different competitors. It fits badly when buyers in one region can see the other region's price, which on the internet is most of the time.

Neither method suits a product where existing customers talk to new ones daily, such as a tight professional community. Someone will compare notes, and you are back to section 6.

8. Ask the people who set prices for a living before you run the expensive tests

The cheapest price test is a conversation with buyers and practitioners who have seen the same decision many times. A former pricing and packaging lead at a company like yours can tell you which price points buyers in your category anchor on, and a pricing strategy consultant can tell you which test your volume can support.

Our guides to pricing research and how to price a product cover how to frame those conversations, and B2B pricing management experts are a good place to start the list. You can use Instant Expert to reach the specific buyers or pricing practitioners you name, and you pay only if they book a call or answer.

Start this week with a five-price Gabor-Granger survey of real buyers. It costs nothing in refunds, and it tells you which two prices deserve a live test.