Vendor selection: criteria, a weighted scorecard and reference calls

Vendor selection is choosing a supplier by agreeing your criteria and their weights before you talk to vendors, scoring each option the same way, and then checking the scores against what real customers say. The usual criteria are fit to your requirements, implementation and support, total cost over the contract, security and compliance, and the vendor's stability. A scorecard narrows the field. Reference calls, especially with customers the vendor did not pick for you, often decide it.

This page covers the criteria, a scorecard you can copy, and how to run reference calls. Deeper checks on a vendor's finances, legal standing and security controls are part of due diligence, a separate step that usually follows once you have a front-runner.

A worked example

Suppose a 60-person accounting firm is choosing a document management system. Three vendors are left after demos: A, B and C. The managing partner, the IT contractor and the office manager will make the decision together. The example is hypothetical, and the numbers below are invented for illustration.

Choose the criteria before you see the vendors

Write the criteria down and agree on them before the first demo. Once people have seen a polished demo, the criteria can drift toward whatever that vendor did well. In US federal buying, FAR 15.305 requires agencies to evaluate proposals solely on the factors stated in the solicitation. You are not bound by that rule, but the discipline is worth copying.

Common criteria, with the question each one answers:

• Fit to requirements: does it do the must-have jobs your team described, without workarounds?
• Implementation and support: who does the setup, how long will it take, and what happens when something breaks?
• Total cost: what will you pay over the whole contract, including setup, training, add-ons and the likely renewal price?
• Security and compliance: does it meet the requirements your data and clients impose?
• Vendor stability: will the vendor still be supporting the product through your contract?

CIPS's guidance on evaluating tenders covers similar ground: whether all costs are included in the price, whether any extra costs are identified, payment terms, technical capability, and the supplier's capacity and financial viability to deliver. It also recommends considering whole-life costs, including eventual decommissioning or disposal. For software, that means asking how you get your data out when you leave.

Build a weighted scorecard

Give each criterion a weight that reflects how much it matters, with the weights adding to 100. Score each vendor from 1 to 5 on each criterion. Multiply each score by its weight, add up the results and divide by 5 to get a score out of 100.

Hypothetical weights and scores for the accounting firm:

The arithmetic for Vendor A: (4×35 + 3×20 + 3×20 + 5×15 + 4×10) ÷ 5 = (140 + 60 + 60 + 75 + 40) ÷ 5 = 375 ÷ 5 = 75. Vendor B: (175 + 40 + 80 + 45 + 30) ÷ 5 = 74. Vendor C: (105 + 100 + 80 + 60 + 20) ÷ 5 = 73.

The totals are within two points of each other, and a one-point change in any single score could reorder them. That can happen, and it is useful: it tells you the scorecard has done its job of removing weak options and cannot pick the winner on its own. What separates these three is the detail behind the scores. B fits best but scored 2 on implementation. C is easy to implement but less stable. You need evidence on those two points.

Have each evaluator score separately first, then compare. CIPS recommends that evaluation be structured and documented, with decisions and reasons recorded. Large disagreements between evaluators often mean a criterion was not defined clearly enough.

Run reference calls that tell you something

Vendors will give you references. Those customers are real, but the vendor chose them, so expect them to be happy. FAR 15.305 treats past performance as one indicator of whether a supplier can do the work, and says to weigh how recent and relevant the information is, where it came from and the context. Apply the same tests to references.

Ask for references like you. Same size, same industry, similar setup, gone live recently. A reference who bought four years ago tells you less about today's implementation team.

Find at least one reference yourself. Look for customers the vendor did not choose: people in your industry who use the product, or people who used it at a previous company. They are more likely to mention problems.

Ask about events, not opinions. "How was the implementation?" invites a polite answer. Better questions:

• "Walk me through the first 90 days after you signed. What took longer than planned?"
• "Tell me about the last time something broke. How long until it was fixed, and who fixed it?"
• "What did you pay in the second year compared with the first?"
• "What do you do outside the product because it can't do it?"
• "If you were choosing again today, what would you check that you didn't?"

Check the weak scores. In the example, ask B's customers about implementation and C's customers about product updates and support responsiveness. Then revise the scores and write down why.

Make the decision and record it

Once the scores reflect the reference evidence, the decision is usually clear. If it still is not, name the one or two remaining questions and resolve them, for example with a paid pilot or a contract clause, rather than adding criteria. Record who decided, the final scores and the reasons. CIPS's guidance also suggests offering feedback to unsuccessful suppliers; it is a courtesy that helps them and keeps your reputation as a buyer.

If you are at the proposal stage, request for proposal covers writing the criteria into the RFP. The procurement process shows where vendor selection fits overall.

Your next step

Write your five criteria and their weights today, and agree on them with the other decision-makers before the next demo. Then list the two weakest scores for your front-runner and the question you would ask a customer about each.

When the vendor's references are all you have, Instant Expert can find people who match a description such as "office managers at accounting firms who use document management software." You review who it finds, it sends your invitations, and you pay only for calls that get booked. Choosing the right expert for a research call covers who to pick; the directory page for procurement professionals in enterprise SaaS is another place to look.