Delphi method: how to run an expert panel forecast, with a worked example

The Delphi method is a structured way to get a group of experts to a forecast or judgment without putting them in a room together. Panelists answer the same questions anonymously over several rounds. After each round they see a summary of the group's answers and the reasons behind them, and they can revise their own. The process ends when answers converge or after a set number of rounds.

RAND, where the method was developed in the 1950s to forecast the effect of technology on warfare, lists four defining features: anonymity, iterative data collection, feedback to participants and a statistical group response. RAND: Delphi method RAND commentary

Use it when no data can answer the question yet and you need a judgment you can defend: when something is likely to happen, how likely it is, or which priorities matter most. It is slower than a few phone calls and less suited to exploring a new topic. The two work well together: interviews to learn what to ask, a Delphi panel to get a structured answer, and interviews again to test the result.

Where the method came from and why anonymity matters

According to RAND's Dmitry Khodyakov, the first peer-reviewed article to mention Delphi by name, by Norman Dalkey and Olaf Helmer, appeared in Management Science in 1963. His team's bibliographic review found about 20,000 articles that used or discussed the method, nearly two-thirds of them in medical journals. It is used to make forecasts, set research priorities, develop performance measures and create clinical guidelines. RAND commentary

The same commentary explains the reason for anonymity: because panelists never meet and do not know who said what, the method reduces groupthink and the pull of dominant personalities. The cost is that panelists cannot question each other directly. Modified versions add discussion back in. The RAND/UCLA Appropriateness Method puts a moderated discussion round, usually in person, between two rounds of ratings, and limits panels to no more than 18 people so the discussion stays workable. RAND's ExpertLens runs the discussion online and anonymously instead.

A worked example: when will electric box trucks pay off?

Suppose a regional food distributor is planning its next truck purchase cycle. The question it cannot answer from data is: "In what year will a battery-electric box truck cost no more than a diesel one to own and run over seven years, on routes like ours?" The answer depends on truck prices, electricity rates, maintenance, depot upgrades and resale values, and nobody has seven years of results for current models. The company, panel and numbers below are hypothetical.

The panel. Twelve people with direct experience: three fleet managers already running electric box trucks, two dealer service managers, two utility account managers who handle fleet charging, two charging installers, two fleet leasing specialists and one used-truck remarketing manager.

The rule, set in advance. Run at most three rounds. Call it consensus if at least 75% of panelists give a year within two years of the group median. This threshold is the team's own choice; write yours down before you see any answers.

Round 1 (open). "What factors will decide the answer for a route of this kind?" The organizer groups the answers into a list of factors and writes the questions for Round 2.

Round 2 (estimates). Each panelist gives a year and a short reason. The twelve answers, sorted, are 2026, 2027, 2028, 2029, 2029, 2030, 2030, 2031, 2033, 2034, 2035 and 2037. With twelve answers, the median is the average of the sixth and seventh, which are both 2030. Only six of the twelve (50%) fall within two years of 2030, so there is no consensus yet.

Feedback. Every panelist receives the median, the full spread and the reasons given by the earliest and latest estimates, with no names attached.

Round 3 (revision). Panelists can change their answer and must explain if they stay outside 2028 to 2032. The revised answers are 2028, 2028, 2029, 2029, 2029, 2030, 2030, 2030, 2031, 2031, 2032 and 2035. The median is still 2030, but now 11 of 12 (92%) are within two years of it, so the rule is met.

The most useful output may be the holdout. The panelist who stayed at 2035 explained that older depots need electrical service upgrades that would take years of fuel savings to pay back. That turns a single forecast into a better question: which of the distributor's depots need an upgrade, and what would it cost?

How to run a Delphi study

1. Write a question with a specific answer, such as a year, a probability or a rating, and define every term in it.
2. Choose panelists for relevant experience and variety. RAND's methodological guidance describes the method as putting the same questions to a hand-picked, anonymous group of experts more than once and showing each of them what the others answered. RAND methodological guidance Include people who see the question from different positions, not ten people from the same kind of company.
3. Set the rules first: the number of rounds, what counts as consensus and what you will report if it is not reached. RAND describes the process as continuing until consensus is reached or a pre-determined number of rounds is completed.
4. Give useful feedback: the median, the spread and anonymized reasons, especially from the extremes.
5. Report the dissent. Give the final median and spread, how many people changed their answer and the reasons from those who did not.

Where one-on-one expert calls fit

Before the panel. Three to five calls with people who know the topic will show you which factors matter, which terms confuse people and who else should be on the panel. How to prepare for an expert interview covers the brief.

During the panel. Avoid private calls with panelists between rounds. If one panelist can argue their case to the organizer while others only see a summary, the feedback is no longer equal. To understand an outlier, ask for a written explanation and share it anonymously with everyone.

After the panel. Test the conclusion with people who were not on it. In the example, the distributor might call two facilities managers who have upgraded depot power for electric trucks and ask what it cost and how long it took.

Instead of a panel. If your question is about how something works rather than what will happen, a Delphi study is more process than you need. A handful of expert interviews will answer it faster. If experts give conflicting answers, what to do when experts disagree helps you find out why.

Common mistakes

• A vague question. "When will electric trucks be viable?" invites twelve definitions of viable.
• A panel that thinks alike. Agreement among people with the same background tells you little.
• Changing the consensus rule after seeing the answers.
• Treating consensus as fact. A panel can agree and still be wrong. The result is a structured judgment, useful when data does not exist, and it should be replaced by data when data arrives.
• Reporting only the median. The spread and the reasons are often worth more than the number.

Your next step

Write your question as one sentence with a specific answer and define every term. List the four or five kinds of people whose experience bears on it and aim for at least two of each. Before Round 1, hold three short calls to check that the question makes sense to people who would answer it.

For those calls and the follow-up afterward, Instant Expert can find people who match a description you write, such as "fleet managers at food distributors who run electric box trucks." You review who it finds, it sends your invitations, and you pay only for calls that get booked. The directory pages for fleet electrification experts and EV charging network operations experts are one place to start.