Delphi method: how to get experts to agree without putting them in a room

The Delphi method gets experts to a forecast through anonymous rounds with feedback. How to run one, a worked example with numbers, and where expert calls fit.

Instant Expert EditorialPublished Updated 6 min read

The Delphi method gets a group of experts to a forecast without putting them in a room together. Panelists answer the same questions anonymously over several rounds. After each round they see a summary of the group's answers and reasons, and can revise their own. It ends when answers converge or after a set number of rounds.

RAND developed the method in the 1950s to forecast the effect of technology on warfare, and lists four defining features: anonymity, iterative data collection, feedback to participants and a statistical group response (RAND: Delphi method, RAND commentary).

Use it when no data can answer the question yet and you need a judgment you can defend: when something will happen, how likely it is, or which priorities matter most. It is slower than a few phone calls and less suited to exploring a new topic, so pair them: interviews to learn what to ask, a panel for the answer, interviews again to test it.

Anonymity is the point of the method, and its cost

According to RAND's Dmitry Khodyakov, the first peer-reviewed article to mention Delphi by name, by Norman Dalkey and Olaf Helmer, appeared in Management Science in 1963. Khodyakov's team's bibliographic review found about 20,000 articles that used or discussed the method, nearly two-thirds of them in medical journals. It is used for forecasts, research priorities, performance measures and clinical guidelines (RAND commentary).

Because panelists never meet and do not know who said what, the commentary explains, the method reduces groupthink and the pull of dominant personalities. (The loudest person in the room gets one anonymous answer, like everyone else.) The cost is that panelists cannot question each other directly.

Modified versions add discussion back in. The RAND/UCLA Appropriateness Method puts a moderated discussion round, usually in person, between two rounds of ratings, and limits panels to 18 people so the discussion stays workable. RAND's ExpertLens runs the discussion online and anonymously.

A worked example: when will electric box trucks pay off?

Here is a made-up example; the company, panel and numbers are all hypothetical. A regional food distributor is planning its next truck purchases and asks: "In what year will a battery-electric box truck cost no more than a diesel one to own and run over seven years, on routes like ours?" It depends on truck prices, electricity rates, maintenance, depot upgrades and resale values, and nobody has seven years of results for current models.

The panel. Twelve people with direct experience: three fleet managers already running electric box trucks, two dealer service managers, two utility account managers who handle fleet charging, two charging installers, two fleet leasing specialists and one used-truck remarketing manager.

The rule, set in advance. Run at most three rounds. Call it consensus if at least 75% of panelists give a year within two years of the group median. The threshold is the team's choice; write yours down before you see any answers.

Round 1 (open). "What factors will decide the answer for a route of this kind?" The organizer groups the answers into factors and writes the Round 2 questions.

Round 2 (estimates). Each panelist gives a year and a short reason. The twelve answers, sorted, are 2026, 2027, 2028, 2029, 2029, 2030, 2030, 2031, 2033, 2034, 2035 and 2037. With twelve answers, the median is the average of the sixth and seventh, which are both 2030. Only six of the twelve (50%) fall within two years of 2030, so there is no consensus yet.

Feedback. Every panelist gets the median, the full spread and the reasons behind the earliest and latest estimates, with no names attached.

Round 3 (revision). Panelists can change their answer and must explain if they stay outside 2028 to 2032. The revised answers are 2028, 2028, 2029, 2029, 2029, 2030, 2030, 2030, 2031, 2031, 2032 and 2035. The median is still 2030, but now 11 of 12 (92%) are within two years of it, so the rule is met.

The most useful output may be the holdout. The panelist who stayed at 2035 explained that older depots need electrical service upgrades that would take years of fuel savings to pay back. That turns one forecast into a better question: which depots need an upgrade, and what would it cost?

How to run a Delphi study in five steps

  1. Write a question with a specific answer, such as a year, a probability or a rating, and define every term.
  2. Choose panelists for relevant experience and variety. RAND's methodological guidance describes putting the same questions to a hand-picked, anonymous group of experts more than once and showing each what the others answered. Pick people who see the question from different positions.
  3. Set the rules first: the number of rounds, what counts as consensus and what you will report if it is not reached. RAND describes the process as running until consensus or a pre-determined number of rounds.
  4. Give useful feedback: the median, the spread and anonymized reasons, especially from the extremes.
  5. Report the dissent: the final median and spread, how many people changed their answer and the reasons from those who did not.

Expert calls belong before and after the panel

Before the panel. Three to five calls with people who know the topic show which factors matter, which terms confuse people and who else belongs on the panel. How to prepare for an expert interview covers the brief.

During the panel, skip them. If one panelist can argue their case privately while others only see a summary, the feedback is no longer equal. To understand an outlier, ask for a written explanation and share it anonymously.

After the panel. Test the conclusion with people who were not on it. The distributor might call two facilities managers who have upgraded depot power for electric trucks and ask what it cost and how long it took.

Instead of a panel. If the question is how something works rather than what will happen, a handful of expert interviews will answer it faster. If experts give conflicting answers, what to do when experts disagree helps you find out why.

Five ways to waste a Delphi panel

  • A vague question. "When will electric trucks be viable?" invites twelve definitions of viable.
  • A panel that thinks alike. Agreement among people with the same background tells you little.
  • Changing the consensus rule after seeing the answers.
  • Treating consensus as fact. A panel can agree and still be wrong. Replace its judgment with data when data arrives.
  • Reporting only the median. The spread and the reasons are often worth more than the number.

Your next step

Write your question as one sentence with a specific answer. List the four or five kinds of people whose experience bears on it and aim for at least two of each. Before Round 1, hold three short calls to check that the question makes sense to people who would answer it.

For those calls and the follow-up, Instant Expert can find people who match a description such as "fleet managers at food distributors who run electric box trucks." It sends your invitations, and you pay only for calls that get booked. Start with fleet electrification experts and EV charging network operations experts.