Quantitative market research: surveys, sample sizes and what the numbers mean

Quantitative market research measures how many people do, think or would buy something, and how much. The common methods are surveys, your own sales and product data, experiments such as A/B tests or preorders, and public statistics. The results are numbers you can compare, but they are only as good as the questions you asked and the people who answered.

Two things decide whether the numbers mean anything. First, you need to know what you are counting, which usually comes from interviews done beforehand. Second, you need to know who answered and how they were chosen. Sample size matters less than most people expect once those two are wrong.

A worked example

Suppose you sell maintenance-request software to property management companies. Earlier interviews showed you three ways companies handle after-hours repair requests: an answering service, an on-call staff member, or a line that goes to voicemail until morning. Now you want to know how common each approach is before deciding which one to build for first. This example is hypothetical.

You have a list of 1,500 property management companies from a trade event you attended, and you plan to email them a short survey.

The main quantitative methods

Nielsen Norman Group's overview of research methods describes quantitative data as gathered indirectly, through an instrument such as a survey or analytics tool, and analyzed mathematically. It says quantitative methods do a better job than qualitative ones at answering how many and how much.

For the property management example, a survey fits. You want to count which of three known setups companies use, and you cannot see that in your own data because most of these companies are not customers yet.

Decide what you are counting before you write questions

The American Association for Public Opinion Research (AAPOR) best practices start by asking whether a survey is the right method at all, and whether existing data already answers the question. For question writing, AAPOR recommends asking about one concept at a time, using words your audience understands, and making closed answer options mutually exclusive and complete, with options like "don't know" where they fit.

This is where earlier interviews pay off. Because you already know the three after-hours setups, you can list them as answer options, plus "other" with a text box. Without the interviews, you would be guessing at the options, and a missing option quietly pushes people into the nearest wrong one.

AAPOR also recommends pretesting questions before sending a survey, typically through cognitive interviews: short conversations in which people similar to your respondents answer the questions aloud and explain how they understood them. A few of these can catch a question that property managers read differently from how you meant it, such as whether "after hours" includes weekends.

How many responses you need

The margin of error describes how far a survey result might be from the true value because you asked a sample instead of everyone. Pew Research Center explains that a margin of plus or minus 3 points at the 95% confidence level means that if the same survey were run 100 times, the result would be within 3 points of the true value about 95 of those times.

For a simple random sample, the 95% margin of error on a percentage is roughly 1.96 × √(p × (1 − p) ÷ n), where n is the number of responses and p is the result. It is largest when p is 50%. We calculated it for a few sample sizes using that worst case:

Two consequences matter in practice. Going from 100 to 400 responses roughly halves the margin, but going from 400 to 1,000 buys much less. And Pew notes that subgroups have larger margins, and that the margin for a difference between two groups is generally about twice the margin for each one. If 400 companies answer and you want to compare the 100 with fewer than 50 units against the rest, each of those small-company estimates carries a margin near ±10 points.

Who answered matters more than how many

The table above assumes a random sample, where everyone in the population had a known chance of being picked. AAPOR distinguishes these probability samples from nonprobability samples, such as opt-in panels or people recruited through social media or personal networks, and notes that analyzing nonprobability results often needs special statistical techniques and careful transparency about the method.

The trade-event list is a nonprobability sample. It over-represents companies that attend events, and the ones that reply may be the ones with an after-hours problem worth talking about. Pew also notes that sampling error is only one source of error: who can be reached, who responds and how questions are worded all affect results and are rarely included in the reported margin.

So if 400 of the 1,500 companies reply and 32% say they use an answering service, report it as "32% of the 400 companies that responded," not "32% of property managers." That calculation gives about ±4.6 points for a random sample, but it does not correct for who chose to answer. If the decision depends on the exact share, look for a second source, such as asking an answering-service provider or a trade association how common their setup is.

Report numbers so others can check them

AAPOR's transparency guidance lists what to disclose alongside survey results, including the sample size, the population studied, how the sample was built and recruited, the survey mode and the full question wording. For an internal decision, a short method note does the same job: who you asked, how many answered, the exact questions, and what you think is missing.

Go back to interviews for the reasons

Suppose the survey also shows that 40% of respondents let after-hours calls go to voicemail. The survey cannot tell you whether that is a deliberate cost decision or a problem nobody has had time to fix. For that, go back to a few of those companies and ask them to walk through the last after-hours request they handled. Interviews versus surveys and checking interviews against product data cover how to combine the two.

If you need property managers or other professionals for pretesting or follow-up calls, the procurement professionals in property management directory page shows the kinds of people you might invite. You can also search for people by describing who you need. Instant Expert finds people whose work matches your question, you review them, it sends your invitations, and you pay for each call that gets booked.