Vendor selection: the scorecard narrows it down, reference calls decide
Vendor selection works best when a weighted scorecard narrows the field and reference calls test the weakest scores. Here is a worked scorecard and the calls.
Good vendor selection uses a scorecard to cut the field and reference calls to pick the winner. Vendor selection means agreeing your criteria and weights before you talk to vendors, scoring each option the same way, then checking the scores against what real customers say. The usual criteria are fit to your requirements, implementation and support, total cost over the contract, security and compliance, and the vendor's stability.
Deeper checks on a vendor's finances, legal standing and security controls belong to due diligence, a separate step that usually follows once you have a front-runner.
The example: three vendors and three deciders
Here is a made-up example, with invented numbers. A 60-person accounting firm is choosing a document management system. Three vendors are left after demos: A, B and C. The managing partner, the IT contractor and the office manager will decide together.
1. Write the criteria down before the first demo
After a polished demo, criteria tend to drift toward whatever that vendor did well. In US federal buying, FAR 15.305 requires agencies to evaluate proposals solely on the factors stated in the solicitation. You are not bound by that rule, but copy the discipline.
- Fit to requirements: does it do the must-have jobs your team described, without workarounds?
- Implementation and support: who does the setup, how long will it take, and what happens when something breaks?
- Total cost: what will you pay over the whole contract, including setup, training, add-ons and the likely renewal price?
- Security and compliance: does it meet the requirements your data and clients impose?
- Vendor stability: will the vendor still support the product through your contract?
CIPS's guidance on evaluating tenders covers similar ground: whether the price includes all costs, whether extra costs are identified, payment terms, technical capability, and the supplier's capacity and financial viability. It also recommends counting whole-life costs, including eventual decommissioning or disposal. For software, ask how you get your data out when you leave.
2. Weight the criteria and score every vendor the same way
Weight each criterion so the weights add to 100, and score each vendor from 1 to 5 per criterion. Multiply each score by its weight, add the results and divide by 5 for a score out of 100.
The accounting firm's made-up weights and scores:
| Criterion | Weight | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|
| Fit to requirements | 35 | 4 | 5 | 3 |
| Implementation and support | 20 | 3 | 2 | 5 |
| Total cost over 3 years | 20 | 3 | 4 | 4 |
| Security and compliance | 15 | 5 | 3 | 4 |
| Vendor stability | 10 | 4 | 3 | 2 |
| Weighted score (out of 100) | 75 | 74 | 73 |
The arithmetic for Vendor A: (4×35 + 3×20 + 3×20 + 5×15 + 4×10) ÷ 5 = (140 + 60 + 60 + 75 + 40) ÷ 5 = 375 ÷ 5 = 75. Vendor B: (175 + 40 + 80 + 45 + 30) ÷ 5 = 74. Vendor C: (105 + 100 + 80 + 60 + 20) ÷ 5 = 73.
Three vendors within two points. A one-point change in any single score could reorder them. (All those demos, and the spreadsheet calls it a tie.)
That is useful: the scorecard removed the weak options, and the rest needs evidence. B fits best but scored 2 on implementation. C is easy to implement but less stable.
Have each evaluator score separately first, then compare. CIPS recommends that evaluation be structured and documented, with decisions and reasons recorded. A large disagreement between evaluators often means a criterion was defined loosely.
3. Use reference calls to test the weakest scores
The vendor chose its references, so expect them to be happy. FAR 15.305 treats past performance as one indicator of whether a supplier can do the work, weighing how recent and relevant the information is, where it came from and the context. Apply the same tests to references.
Ask for references like you. Same size, same industry, similar setup, gone live recently. A customer who bought four years ago knows less about today's implementation team.
Find at least one reference yourself. People in your industry who use the product, or used it at a previous company, are more likely to mention problems.
Ask about events, not opinions. "How was the implementation?" invites a polite answer. Better:
- "Walk me through the first 90 days after you signed. What took longer than planned?"
- "Tell me about the last time something broke. How long until it was fixed, and who fixed it?"
- "What did you pay in the second year compared with the first?"
- "What do you do outside the product because it can't do it?"
- "If you were choosing again today, what would you check that you didn't?"
Check the weak scores. In the example, ask B's customers about implementation and C's customers about product updates and support responsiveness. Then revise the scores and write down why.
4. Settle the last question with a pilot, not more criteria
Once the scores reflect the reference evidence, the decision is usually clear. If not, name the one or two open questions and run a paid pilot to answer them. A contract clause can cover what the pilot can't. Record who decided, the final scores and the reasons. CIPS also suggests offering feedback to unsuccessful suppliers, a courtesy that helps them and your reputation as a buyer.
At the proposal stage, request for proposal covers writing the criteria into the RFP. The procurement process shows where vendor selection fits overall.
Your next step
Write your five criteria and weights today, and agree on them with the other decision-makers before the next demo. Then list your front-runner's two weakest scores and the question you would ask a customer about each.
When the vendor's references are all you have, Instant Expert can find people who match a description such as "office managers at accounting firms who use document management software." You review who it finds, it sends your invitations, and you pay only for calls that get booked. Choosing the right expert for a research call covers who to pick; the directory page for procurement professionals in enterprise SaaS is another place to look.